Merge pull request #285 from mvanhorn/fix/output-contract-planner-breadth
fix: output contract + planner breadth + entity grounding (Hermes Agent Use Cases)
This commit is contained in:
@@ -120,7 +120,49 @@ These five rules dominate every other rule in this file. If you find yourself ab
|
|||||||
|
|
||||||
**Observed LAW 4 violation (2026-04-18, Peter Steinberger disaster #2):** the model emitted `Headline`, `What he is actually saying`, `Cross-source corroboration`, `Where evidence is thin`, `Bottom line` on a GENERAL query. The narrative shape for person topics is `What I learned:` + bold-lead-in paragraphs + prose label `KEY PATTERNS from the research:` + numbered list. No blog-post subheadings.
|
**Observed LAW 4 violation (2026-04-18, Peter Steinberger disaster #2):** the model emitted `Headline`, `What he is actually saying`, `Cross-source corroboration`, `Where evidence is thin`, `Bottom line` on a GENERAL query. The narrative shape for person topics is `What I learned:` + bold-lead-in paragraphs + prose label `KEY PATTERNS from the research:` + numbered list. No blog-post subheadings.
|
||||||
|
|
||||||
**LAW 5 - ENGINE FOOTER PASS-THROUGH. EVERY QUERY TYPE. EVERY RUN.** The engine output ends with a `✅ All agents reported back!` emoji-tree footer bounded by `---` lines. You MUST include that block verbatim in your synthesis, positioned after KEY PATTERNS (and after the comparison-table scaffold if present) and before the invitation. Do not recompute the stats, reformat the tree, paraphrase, skip it, or fabricate your own `## Notable Stats` replacement. A response without the engine footer is not valid skill output.
|
**LAW 5 - ENGINE FOOTER PASS-THROUGH. EVERY QUERY TYPE. EVERY RUN.** The engine output ends with a `✅ All agents reported back!` emoji-tree footer bounded by `---` lines and wrapped in `<!-- PASS-THROUGH FOOTER -->` / `<!-- END PASS-THROUGH FOOTER -->` comments (v3.0.10+). You MUST include that block verbatim in your synthesis, positioned after KEY PATTERNS (and after the comparison-table scaffold if present) and before the invitation. Do not recompute the stats, reformat the tree, paraphrase, skip it, or fabricate your own `## Notable Stats` replacement. A response without the engine footer is not valid skill output.
|
||||||
|
|
||||||
|
**LAW 6 - NO RAW RANKED EVIDENCE CLUSTERS IN BODY.** The engine's `## Ranked Evidence Clusters`, `## Stats`, and `## Source Coverage` blocks are bounded inside `<!-- EVIDENCE FOR SYNTHESIS -->` / `<!-- END EVIDENCE FOR SYNTHESIS -->` comments in the `--emit compact` / `--emit md` stdout. They are raw evidence for YOU to read, not output to emit. Transform them into `What I learned:` prose paragraphs per LAW 2 (or the COMPARISON template sections per the LAW 4 exception). If your response contains the literal string `### 1.` followed by a score tuple like `(score N, M items, sources: ...)`, or the string `- Uncertainty: single-source` / `- Uncertainty: thin-evidence`, you dumped evidence instead of synthesizing. STOP and regenerate.
|
||||||
|
|
||||||
|
**Observed LAW 6 violation (2026-04-19, Hermes Agent Use Cases disaster):** two consecutive `/last30days Hermes Agent (Actual) Use Cases` runs returned the raw `## Ranked Evidence Clusters` block verbatim as user output, with 8 cluster entries carrying `(score N, M items, sources: ...)` tuples and `- Uncertainty: single-source` lines. Root cause: the prior canonical-boundary text said "Pass through the lines ABOVE this boundary verbatim," which the model scoped broadly to include the scratchpad. The current boundary text and this LAW 6 scope pass-through to the PASS-THROUGH FOOTER block only. A third run on the same topic framed as "Hermes Workflows" produced the correct `What I learned:` prose synthesis, which is the shape every run must produce.
|
||||||
|
|
||||||
|
**Worked example (LAW 6 transformation).** Evidence block you read:
|
||||||
|
|
||||||
|
```
|
||||||
|
<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. -->
|
||||||
|
## Ranked Evidence Clusters
|
||||||
|
|
||||||
|
### 1. Hermes Agent: The Self-Improving AI That Learns You (score 45, 1 item, sources: Youtube)
|
||||||
|
|
||||||
|
1. [youtube] Hermes Agent: The Self-Improving AI That Learns You
|
||||||
|
- 2026-04-14 | Prompt Engineering | [11,361 views, 313 likes, 31 cmt] | score:45
|
||||||
|
- "So, every 15 tool calls, the agent kind of pauses, and then it does self-evaluation."
|
||||||
|
- "Can you tell me what type of user profile you have on me?"
|
||||||
|
|
||||||
|
### 2. Use cases of OpenClaw, Hermes Agent, etc... (score 43, 1 item, sources: Reddit)
|
||||||
|
|
||||||
|
1. [reddit] Use cases of OpenClaw, Hermes Agent, etc... (r/TunisiaTech, 3pts, 1cmt)
|
||||||
|
- "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
|
||||||
|
<!-- END EVIDENCE FOR SYNTHESIS -->
|
||||||
|
```
|
||||||
|
|
||||||
|
Output you emit (prose synthesis, NOT the evidence block):
|
||||||
|
|
||||||
|
```
|
||||||
|
What I learned:
|
||||||
|
|
||||||
|
The self-evolving loop is the sticky use case. Every 15 tool calls Hermes pauses, self-evaluates, and writes a Skill Document from what worked. Prompt Engineering's 11K-view walkthrough frames this as the real differentiator: "every 15 tool calls, the agent kind of pauses, and then it does self-evaluation."
|
||||||
|
|
||||||
|
Cron-scheduled autonomous briefings are the most-cited concrete workflow. r/TunisiaTech's "Use cases of OpenClaw, Hermes Agent" thread says it plainly: "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
|
||||||
|
```
|
||||||
|
|
||||||
|
**LAW 7 - YOU ARE THE PLANNER. `--plan` IS MANDATORY ON NAMED-ENTITY TOPICS.** If you are the reasoning model hosting this skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime that invoked `/last30days`), YOU generate the JSON query plan. You do not need an API key, "LLM provider" credentials, or an external planning service - you ARE the LLM. The `--plan` flag exists precisely so a reasoning model generates its own plan upstream and passes it to the engine. The engine's internal planner and deterministic fallback are headless/cron paths only; on any reasoning-model path, bypass them by passing `--plan '$JSON'`.
|
||||||
|
|
||||||
|
Named-entity topics (capitalized proper nouns, product names, person names, project names, or any topic that would benefit from handle resolution in Step 0.55) REQUIRE `--plan`. Your invocation of `scripts/last30days.py` MUST contain `--plan '$JSON'`. A bare `python3 scripts/last30days.py "$TOPIC" --emit=compact` on a named-entity topic is a LAW 7 violation. Before you invoke Bash, self-check: does my command contain `--plan`? If no, STOP and generate a plan first (see Step 0.75 for the schema).
|
||||||
|
|
||||||
|
**Observed LAW 7 violation (2026-04-19, Hermes Agent Use Cases Run 1):** the model called the engine bare with no `--plan`, no pre-flight handle resolution. The engine emitted a stderr warning ("No --plan and no LLM provider configured. Using deterministic fallback...") which the model read as a capability constraint ("I don't have a key, I can't do LLM stuff") instead of as what it actually was: a reminder that the reasoning model skipped its own planning step. The misread came from the word "provider" - the engine uses "provider" to mean "the key for the engine's INTERNAL planner," but the model parsed it as "I need a provider to plan at all." You do not. You ARE the provider. Run 2 of the same topic (2026-04-19, framed as "best workflows") with the same model and same cache generated the plan itself via `--plan` and produced clean results - the delta was this step.
|
||||||
|
|
||||||
|
**Self-check before Bash:** re-read your pending `scripts/last30days.py` command. Does it contain `--plan '$JSON'`? If no, and the topic is a named entity, STOP. Return to Step 0.75 and generate the plan. Do not interpret the word "provider" in any engine message as "you need credentials" - you are the provider.
|
||||||
|
|
||||||
End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail.
|
End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail.
|
||||||
|
|
||||||
|
|||||||
+34
-1
@@ -204,7 +204,7 @@ def run(
|
|||||||
plan = planner._sanitize_plan(
|
plan = planner._sanitize_plan(
|
||||||
external_plan, topic, available, requested_sources, depth,
|
external_plan, topic, available, requested_sources, depth,
|
||||||
)
|
)
|
||||||
print(f"[Planner] Using external plan ({len(plan.subqueries)} subqueries)", file=sys.stderr)
|
plan_source = "external"
|
||||||
else:
|
else:
|
||||||
plan = planner.plan_query(
|
plan = planner.plan_query(
|
||||||
topic=topic,
|
topic=topic,
|
||||||
@@ -215,6 +215,14 @@ def run(
|
|||||||
model=None if mock else runtime.planner_model,
|
model=None if mock else runtime.planner_model,
|
||||||
context=config.get("_auto_resolve_context", ""),
|
context=config.get("_auto_resolve_context", ""),
|
||||||
)
|
)
|
||||||
|
# Source labelling: the fallback path annotates notes with "fallback-plan"
|
||||||
|
# or "deterministic-comparison-plan"; anything else came from the LLM.
|
||||||
|
if any("fallback" in note or "deterministic" in note for note in (plan.notes or [])):
|
||||||
|
plan_source = "deterministic"
|
||||||
|
elif not mock and reasoning_provider and runtime.planner_model:
|
||||||
|
plan_source = "llm"
|
||||||
|
else:
|
||||||
|
plan_source = "deterministic"
|
||||||
|
|
||||||
# Safety net: ensure grounding appears in all subqueries even if the planner
|
# Safety net: ensure grounding appears in all subqueries even if the planner
|
||||||
# omits it. This is redundant when the planner includes grounding via
|
# omits it. This is redundant when the planner includes grounding via
|
||||||
@@ -224,7 +232,32 @@ def run(
|
|||||||
if "grounding" not in sq.sources:
|
if "grounding" not in sq.sources:
|
||||||
sq.sources.append("grounding")
|
sq.sources.append("grounding")
|
||||||
|
|
||||||
|
# Always-on planner trace. Emits one summary line plus one per subquery
|
||||||
|
# so retrieval-breadth failures like the 2026-04-19 Hermes Agent Use Cases
|
||||||
|
# disaster are visible without --debug. Stderr only; does not leak into
|
||||||
|
# the user-facing stdout synthesis.
|
||||||
|
print(
|
||||||
|
f"[Planner] Plan: intent={plan.intent}, freshness={plan.freshness_mode}, "
|
||||||
|
f"cluster_mode={plan.cluster_mode}, subqueries={len(plan.subqueries)}, "
|
||||||
|
f"source={plan_source}",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
if plan.subqueries:
|
||||||
|
for index, sq in enumerate(plan.subqueries, start=1):
|
||||||
|
sources_str = ",".join(sq.sources) if sq.sources else "(none)"
|
||||||
|
print(
|
||||||
|
f"[Planner] sq{index} label={sq.label} "
|
||||||
|
f'search="{sq.search_query}" sources=[{sources_str}]',
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
print("[Planner] (no subqueries in plan)", file=sys.stderr)
|
||||||
|
|
||||||
bundle = schema.RetrievalBundle(artifacts={"grounding": []})
|
bundle = schema.RetrievalBundle(artifacts={"grounding": []})
|
||||||
|
# Expose plan_source to the renderer so render_compact can emit the
|
||||||
|
# DEGRADED RUN banner when a named-entity topic was invoked bare
|
||||||
|
# (source=deterministic AND no pre-research flags). LAW 7 backstop.
|
||||||
|
bundle.artifacts["plan_source"] = plan_source
|
||||||
|
|
||||||
# Project-mode or person-mode GitHub: run once before the main subquery loop
|
# Project-mode or person-mode GitHub: run once before the main subquery loop
|
||||||
_github_custom_done = False
|
_github_custom_done = False
|
||||||
|
|||||||
+135
-7
@@ -113,6 +113,25 @@ def plan_query(
|
|||||||
topic, available_sources, requested_sources, depth,
|
topic, available_sources, requested_sources, depth,
|
||||||
note=f"fallback-plan (LLM error: {type(exc).__name__})",
|
note=f"fallback-plan (LLM error: {type(exc).__name__})",
|
||||||
)
|
)
|
||||||
|
# No --plan was passed and no engine-internal provider is configured.
|
||||||
|
# The deterministic fallback path produces weaker plans than either the
|
||||||
|
# --plan path or the engine-internal LLM planner path. Surface this so
|
||||||
|
# the HOSTING reasoning model (Claude Code, Codex, Hermes, Gemini, etc.)
|
||||||
|
# realizes it is supposed to generate the plan itself and pass --plan.
|
||||||
|
# The word "provider" in engine messages refers to the engine's INTERNAL
|
||||||
|
# planner credentials - NOT a prerequisite the caller needs. If you are
|
||||||
|
# the hosting reasoning model, YOU are the provider. LAW 7 / 2026-04-19
|
||||||
|
# Hermes Agent Use Cases failure mode.
|
||||||
|
import sys
|
||||||
|
print(
|
||||||
|
"[Planner] No --plan passed. If you are the reasoning model hosting "
|
||||||
|
"this skill (Claude Code, Codex, Hermes, Gemini, or any agent runtime), "
|
||||||
|
"YOU ARE the planner: generate a JSON query plan yourself and pass it "
|
||||||
|
"via --plan. You do not need an API key or credentials; you ARE the "
|
||||||
|
"LLM. The deterministic fallback below is the headless/cron path only. "
|
||||||
|
"See LAW 7 in SKILL.md and Step 0.75 for the plan schema.",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
return _fallback_plan(topic, available_sources, requested_sources, depth)
|
return _fallback_plan(topic, available_sources, requested_sources, depth)
|
||||||
|
|
||||||
|
|
||||||
@@ -151,7 +170,7 @@ Return JSON only with this shape:
|
|||||||
}}
|
}}
|
||||||
|
|
||||||
Rules:
|
Rules:
|
||||||
- emit 1 to 4 subqueries
|
- emit 1 to 5 subqueries (how_to/opinion/product/breaking_news intents benefit from 4-5; factual/concept from 2)
|
||||||
- every subquery must include both search_query and ranking_query
|
- every subquery must include both search_query and ranking_query
|
||||||
- sources must be drawn from Available sources only
|
- sources must be drawn from Available sources only
|
||||||
- use cluster_mode=none for factual or many how-to queries
|
- use cluster_mode=none for factual or many how-to queries
|
||||||
@@ -162,6 +181,8 @@ Rules:
|
|||||||
- preserve exact proper nouns and entity strings from the topic
|
- preserve exact proper nouns and entity strings from the topic
|
||||||
- NEVER include temporal phrases in search_query: no 'last 30 days', 'recent', month names, year numbers
|
- NEVER include temporal phrases in search_query: no 'last 30 days', 'recent', month names, year numbers
|
||||||
- NEVER include meta-research phrases: no 'news', 'updates', 'public appearances', 'latest developments'
|
- NEVER include meta-research phrases: no 'news', 'updates', 'public appearances', 'latest developments'
|
||||||
|
- INTENT-MODIFIER HANDLING: when the topic contains one of {{use cases, use case, workflows, workflow, examples, tutorial, tutorials, review, reviews, comparison, applications, in practice, production, production use, how i use}}, STRIP that phrase from every search_query (keep its meaning in ranking_query). Emit 4-5 paraphrased subqueries that each express the intent differently (e.g., 'production', 'workflow OR pipeline', 'review OR experience', 'vs COMPETITOR', 'community discussion'). Broad retrieval, narrow ranking. This was the 2026-04-19 Hermes Agent Use Cases failure mode: the planner echoed "hermes agent use cases" as a literal search string and returned near-zero results because nobody posts that exact phrase.
|
||||||
|
- DO NOT quote the user's full topic verbatim in search_query. Quote only multi-word proper nouns like "Hermes Agent", "Claude Code", "Nous Research". Bare keywords OR'd together retrieve more than exact-phrase searches.
|
||||||
- search_query should match how content is TITLED on platforms
|
- search_query should match how content is TITLED on platforms
|
||||||
- GitHub (Issues/PRs) is best for engineering, developer tools, and open source topics: 'kanye west bully' not 'kanye west album news March 2026'
|
- GitHub (Issues/PRs) is best for engineering, developer tools, and open source topics: 'kanye west bully' not 'kanye west album news March 2026'
|
||||||
""".strip()
|
""".strip()
|
||||||
@@ -204,7 +225,7 @@ def _sanitize_plan(
|
|||||||
source_weights = _normalize_weights(source_weights)
|
source_weights = _normalize_weights(source_weights)
|
||||||
|
|
||||||
subqueries: list[schema.SubQuery] = []
|
subqueries: list[schema.SubQuery] = []
|
||||||
for index, subquery in enumerate((raw.get("subqueries") or [])[:_max_subqueries(intent_hint)], start=1):
|
for index, subquery in enumerate((raw.get("subqueries") or [])[:_max_subqueries(intent_hint, topic)], start=1):
|
||||||
if not isinstance(subquery, dict):
|
if not isinstance(subquery, dict):
|
||||||
continue
|
continue
|
||||||
sources = [source for source in subquery.get("sources") or [] if source in source_weights]
|
sources = [source for source in subquery.get("sources") or [] if source in source_weights]
|
||||||
@@ -382,13 +403,22 @@ def _fallback_plan(
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Intent-modifier fanout: when topic contains a phrase like "use cases",
|
||||||
|
# "workflows", "examples", "review" (see _INTENT_MODIFIER_PATTERNS),
|
||||||
|
# paraphrase the intent across 3 extra subqueries rather than echoing
|
||||||
|
# the literal phrase. Fixes 2026-04-19 Hermes Agent Use Cases failure.
|
||||||
|
# Excluded for comparison/prediction since those already have dedicated
|
||||||
|
# fanout (entity-per-subquery / odds).
|
||||||
|
if depth != "quick" and intent not in {"comparison", "prediction"} and _has_intent_modifier(topic):
|
||||||
|
subqueries.extend(_intent_modifier_subqueries(topic, core, base_search, source_weights))
|
||||||
|
|
||||||
return schema.QueryPlan(
|
return schema.QueryPlan(
|
||||||
intent=intent,
|
intent=intent,
|
||||||
freshness_mode=_default_freshness(intent),
|
freshness_mode=_default_freshness(intent),
|
||||||
cluster_mode=_default_cluster_mode(intent),
|
cluster_mode=_default_cluster_mode(intent),
|
||||||
raw_topic=topic,
|
raw_topic=topic,
|
||||||
subqueries=_normalize_subquery_weights(
|
subqueries=_normalize_subquery_weights(
|
||||||
_trim_subqueries_for_depth(subqueries[:_max_subqueries(intent)], intent, depth, list(source_weights))
|
_trim_subqueries_for_depth(subqueries[:_max_subqueries(intent, topic)], intent, depth, list(source_weights))
|
||||||
),
|
),
|
||||||
source_weights=_normalize_weights(source_weights),
|
source_weights=_normalize_weights(source_weights),
|
||||||
notes=[note],
|
notes=[note],
|
||||||
@@ -418,7 +448,15 @@ def _infer_intent(topic: str) -> str:
|
|||||||
return "concept"
|
return "concept"
|
||||||
if re.search(r"\b(tournament|championship|playoffs|march madness|world cup|olympics|super bowl|final four|ceremony|awards|keynote)\b", text):
|
if re.search(r"\b(tournament|championship|playoffs|march madness|world cup|olympics|super bowl|final four|ceremony|awards|keynote)\b", text):
|
||||||
return "breaking_news"
|
return "breaking_news"
|
||||||
return "breaking_news"
|
# Recency signals take priority when nothing more specific matched.
|
||||||
|
if re.search(r"\b(trending|this week|right now|today|this month)\b", text):
|
||||||
|
return "breaking_news"
|
||||||
|
# Default changed from "breaking_news" to "concept" on 2026-04-19 after
|
||||||
|
# the Hermes Agent Use Cases failure: unclassified topics were getting
|
||||||
|
# strict_recent freshness, which over-weighted the last 7 days and
|
||||||
|
# under-weighted older relevant material. "concept" defaults to
|
||||||
|
# evergreen_ok freshness, a safer posture for unknown topics.
|
||||||
|
return "concept"
|
||||||
|
|
||||||
|
|
||||||
def _default_freshness(intent: str) -> str:
|
def _default_freshness(intent: str) -> str:
|
||||||
@@ -464,8 +502,26 @@ def _default_source_weights(intent: str, sources: list[str]) -> dict[str, float]
|
|||||||
|
|
||||||
|
|
||||||
def _keyword_query(topic: str, core: str) -> str:
|
def _keyword_query(topic: str, core: str) -> str:
|
||||||
|
"""Build a search_query string for the deterministic fallback.
|
||||||
|
|
||||||
|
Quote ONLY title-cased multi-word proper nouns ("Hermes Agent",
|
||||||
|
"Claude Code", "Nous Research") so platform search engines preserve the
|
||||||
|
name as a phrase. Hyphenated compounds and lowercase terms are left as
|
||||||
|
bare keywords, which broadens retrieval instead of narrowing it.
|
||||||
|
|
||||||
|
Prior behavior quoted the entire compound including the user's typed
|
||||||
|
topic, producing searches like `"Hermes Agent Actual Use Cases" hermes agent actual`
|
||||||
|
that returned near-zero matches on X and Reddit because nobody posts
|
||||||
|
that exact phrase. See 2026-04-19 Hermes Agent Use Cases failure.
|
||||||
|
"""
|
||||||
compounds = query.extract_compound_terms(topic)
|
compounds = query.extract_compound_terms(topic)
|
||||||
quoted = " ".join(f"\"{term}\"" for term in compounds[:2])
|
# Only quote title-cased proper nouns (multi-word names). Hyphenated
|
||||||
|
# compounds go unquoted so platform tokenizers can split and match.
|
||||||
|
title_cased = [
|
||||||
|
term for term in compounds
|
||||||
|
if re.match(r"^(?:[A-Z][a-z]+\s+){1,}[A-Z][a-z]+$", term)
|
||||||
|
]
|
||||||
|
quoted = " ".join(f'"{term}"' for term in title_cased[:2])
|
||||||
keywords = [quoted.strip(), core.strip() or topic.strip()]
|
keywords = [quoted.strip(), core.strip() or topic.strip()]
|
||||||
return " ".join(part for part in keywords if part).strip()
|
return " ".join(part for part in keywords if part).strip()
|
||||||
|
|
||||||
@@ -513,12 +569,84 @@ def _should_force_deterministic_plan(topic: str) -> bool:
|
|||||||
return _infer_intent(topic) == "comparison" and len(_comparison_entities(topic)) >= 2
|
return _infer_intent(topic) == "comparison" and len(_comparison_entities(topic)) >= 2
|
||||||
|
|
||||||
|
|
||||||
def _max_subqueries(intent: str) -> int:
|
_INTENT_MODIFIER_PATTERNS = (
|
||||||
|
"use cases", "use case", "workflows", "workflow",
|
||||||
|
"examples", "example", "tutorial", "tutorials",
|
||||||
|
"review", "reviews", "comparison", "applications",
|
||||||
|
"in practice", "production use", "production",
|
||||||
|
"how i use",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _has_intent_modifier(topic: str) -> bool:
|
||||||
|
"""Return True if the topic contains an intent modifier phrase.
|
||||||
|
|
||||||
|
See 2026-04-19 Hermes Agent Use Cases failure: a literal "Hermes Agent
|
||||||
|
use cases" search returns near-zero matches because nobody posts that
|
||||||
|
exact phrase. Intent modifiers should be stripped from search_query
|
||||||
|
and paraphrased across multiple subqueries.
|
||||||
|
"""
|
||||||
|
text = topic.lower()
|
||||||
|
return any(pattern in text for pattern in _INTENT_MODIFIER_PATTERNS)
|
||||||
|
|
||||||
|
|
||||||
|
def _intent_modifier_subqueries(
|
||||||
|
topic: str,
|
||||||
|
core: str,
|
||||||
|
base_search: str,
|
||||||
|
source_weights: dict[str, float],
|
||||||
|
) -> list[schema.SubQuery]:
|
||||||
|
"""Produce paraphrased subqueries for intent-modifier topics.
|
||||||
|
|
||||||
|
The deterministic fallback used to echo the user's literal phrase
|
||||||
|
(e.g., "hermes agent use cases") into every search_query. This helper
|
||||||
|
fans out 3 extra subqueries that each express the intent differently
|
||||||
|
so retrieval pulls a broader corpus for reranking.
|
||||||
|
"""
|
||||||
|
entity = core or topic.strip()
|
||||||
|
sources = list(source_weights)
|
||||||
|
return [
|
||||||
|
schema.SubQuery(
|
||||||
|
label="workflows",
|
||||||
|
search_query=f"{entity} workflow pipeline",
|
||||||
|
ranking_query=f"What real-world workflows or pipelines are people running with {entity}?",
|
||||||
|
sources=sources,
|
||||||
|
weight=0.6,
|
||||||
|
),
|
||||||
|
schema.SubQuery(
|
||||||
|
label="production",
|
||||||
|
search_query=f"{entity} production real-world",
|
||||||
|
ranking_query=f"What production deployments or real-world use cases of {entity} are people describing?",
|
||||||
|
sources=sources,
|
||||||
|
weight=0.55,
|
||||||
|
),
|
||||||
|
schema.SubQuery(
|
||||||
|
label="experience",
|
||||||
|
search_query=f"{entity} experience review",
|
||||||
|
ranking_query=f"What hands-on experience reports or reviews of {entity} exist in the last 30 days?",
|
||||||
|
sources=sources,
|
||||||
|
weight=0.5,
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _max_subqueries(intent: str, topic: str | None = None) -> int:
|
||||||
|
# how_to/opinion/product/breaking_news/prediction benefit from 4-5
|
||||||
|
# paraphrased subqueries when the topic carries an intent modifier
|
||||||
|
# (use cases, workflows, examples, review, etc.). See 2026-04-19
|
||||||
|
# Hermes Agent Use Cases failure: prior cap of 3 produced near-literal
|
||||||
|
# echoes of the topic instead of a paraphrase fanout.
|
||||||
if intent == "comparison":
|
if intent == "comparison":
|
||||||
return 4
|
return 4
|
||||||
|
# Intent-modifier topics get headroom for paraphrase fanout even when
|
||||||
|
# the intent itself is factual/concept. Without this, a "Hermes Agent
|
||||||
|
# use cases" query (classified "concept" after the 2026-04-19 default
|
||||||
|
# change) would be capped at 2 and drop the fanout.
|
||||||
|
if topic and _has_intent_modifier(topic):
|
||||||
|
return 5
|
||||||
if intent in {"factual", "concept"}:
|
if intent in {"factual", "concept"}:
|
||||||
return 2
|
return 2
|
||||||
return 3
|
return 5
|
||||||
|
|
||||||
|
|
||||||
def _default_sources_for_intent(intent: str, available_sources: list[str]) -> list[str]:
|
def _default_sources_for_intent(intent: str, available_sources: list[str]) -> list[str]:
|
||||||
|
|||||||
+99
-9
@@ -100,6 +100,24 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
|
|||||||
lines.extend(f"- {warning}" for warning in report.warnings)
|
lines.extend(f"- {warning}" for warning in report.warnings)
|
||||||
lines.append("")
|
lines.append("")
|
||||||
|
|
||||||
|
# LAW 7 backstop: emit the DEGRADED RUN WARNING block BEFORE the evidence
|
||||||
|
# envelope so the model's pass-through contract forces it into the user's
|
||||||
|
# response on bare named-entity calls. The stderr [Planner] warning is
|
||||||
|
# invisible to the user; this block is not.
|
||||||
|
degraded_warning = _render_degraded_run_warning(report)
|
||||||
|
if degraded_warning:
|
||||||
|
lines.extend(degraded_warning)
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
|
# Open EVIDENCE FOR SYNTHESIS envelope. The ## Ranked Evidence Clusters,
|
||||||
|
# ## Stats, and ## Source Coverage blocks inside this envelope are raw
|
||||||
|
# evidence for the model to READ, not output to emit. LAW 6 in SKILL.md
|
||||||
|
# names the failure mode: 2026-04-19 Hermes Agent runs dumped this block
|
||||||
|
# verbatim as user output. The envelope comments give the model an
|
||||||
|
# unambiguous scope for "pass through verbatim" (the PASS-THROUGH FOOTER
|
||||||
|
# block below) vs "synthesize from" (this block).
|
||||||
|
lines.append("<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. Transform into `What I learned:` prose per LAW 2. -->")
|
||||||
|
lines.append("")
|
||||||
lines.append("## Ranked Evidence Clusters")
|
lines.append("## Ranked Evidence Clusters")
|
||||||
lines.append("")
|
lines.append("")
|
||||||
candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
|
candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
|
||||||
@@ -126,6 +144,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
|
|||||||
lines.extend([""] + best_takes)
|
lines.extend([""] + best_takes)
|
||||||
|
|
||||||
lines.extend(_render_source_coverage(report))
|
lines.extend(_render_source_coverage(report))
|
||||||
|
# Close EVIDENCE FOR SYNTHESIS envelope before anything that passes through verbatim.
|
||||||
|
lines.append("")
|
||||||
|
lines.append("<!-- END EVIDENCE FOR SYNTHESIS -->")
|
||||||
|
|
||||||
pre_research_warning = _render_pre_research_warning(report)
|
pre_research_warning = _render_pre_research_warning(report)
|
||||||
if pre_research_warning:
|
if pre_research_warning:
|
||||||
@@ -140,7 +161,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
|
|||||||
footer = _render_emoji_footer(report, save_path)
|
footer = _render_emoji_footer(report, save_path)
|
||||||
if footer:
|
if footer:
|
||||||
lines.append("")
|
lines.append("")
|
||||||
|
lines.append("<!-- PASS-THROUGH FOOTER: emit verbatim in the model response per LAW 5. -->")
|
||||||
lines.extend(footer)
|
lines.extend(footer)
|
||||||
|
lines.append("<!-- END PASS-THROUGH FOOTER -->")
|
||||||
|
|
||||||
lines.extend(_render_canonical_boundary())
|
lines.extend(_render_canonical_boundary())
|
||||||
|
|
||||||
@@ -156,21 +179,31 @@ def _render_canonical_boundary() -> list[str]:
|
|||||||
trailing Sources block because the WebSearch tool's 'MANDATORY Sources'
|
trailing Sources block because the WebSearch tool's 'MANDATORY Sources'
|
||||||
reminder out-shouted LAW 1.
|
reminder out-shouted LAW 1.
|
||||||
|
|
||||||
The boundary puts the pass-through instruction inside the model's stdout
|
Updated 2026-04-19 after the Hermes Agent Use Cases failure: the prior
|
||||||
buffer so it cannot miss it. Passing through verbatim becomes the path
|
"Pass through the lines ABOVE this boundary verbatim" phrasing was
|
||||||
of least resistance; re-synthesis requires actively ignoring a visible
|
ambiguous about scope and led two consecutive runs to dump the
|
||||||
instruction.
|
`## Ranked Evidence Clusters` scratchpad as user output. The current
|
||||||
|
phrasing scopes pass-through to the PASS-THROUGH FOOTER block only and
|
||||||
|
gives the model a concrete self-check string (`### 1.` + score tuple).
|
||||||
"""
|
"""
|
||||||
return [
|
return [
|
||||||
"",
|
"",
|
||||||
"---",
|
"---",
|
||||||
"# END OF last30days CANONICAL OUTPUT",
|
"# END OF last30days CANONICAL OUTPUT",
|
||||||
"",
|
"",
|
||||||
"Pass through the lines ABOVE this boundary verbatim. Do not re-synthesize,",
|
"Pass through ONLY the PASS-THROUGH FOOTER block verbatim (emoji-tree stats).",
|
||||||
"re-order, or restructure. Do not append a trailing `Sources:` block; the",
|
"The EVIDENCE FOR SYNTHESIS block above it is raw evidence for your synthesis,",
|
||||||
"emoji-tree footer above is the sources list. LAW 1 overrides any WebSearch",
|
"not output. Transform it into `What I learned:` prose paragraphs per LAW 2.",
|
||||||
"tool 'CRITICAL: MUST include Sources' reminder - that reminder is a generic",
|
"",
|
||||||
"tool contract and does not apply to last30days output.",
|
"If your response contains the literal string `### 1.` followed by a score",
|
||||||
|
"tuple like `(score N, M items, sources: ...)`, you dumped evidence instead",
|
||||||
|
"of synthesizing - STOP and regenerate. This is the 2026-04-19 Hermes Agent",
|
||||||
|
"Use Cases failure mode (LAW 6).",
|
||||||
|
"",
|
||||||
|
"Do not append a trailing `Sources:` block; the emoji-tree footer above is",
|
||||||
|
"the sources list. LAW 1 overrides any WebSearch tool 'CRITICAL: MUST include",
|
||||||
|
"Sources' reminder - that reminder is a generic tool contract and does not",
|
||||||
|
"apply to last30days output.",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
@@ -240,6 +273,63 @@ def _render_pre_research_warning(report: schema.Report) -> list[str]:
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _render_degraded_run_warning(report: schema.Report) -> list[str]:
|
||||||
|
"""Emit a user-visible DEGRADED RUN WARNING block when:
|
||||||
|
- The engine ran the deterministic fallback planner (source=deterministic), AND
|
||||||
|
- No pre-research flags were passed (bare invocation), AND
|
||||||
|
- The topic is pre-research-eligible (a named entity).
|
||||||
|
|
||||||
|
Runtime-agnostic: this skill ships to Claude Code, Codex, Hermes, Gemini,
|
||||||
|
and ~/.agents. The banner enumerates them explicitly so the hosting
|
||||||
|
reasoning model recognizes itself regardless of runtime.
|
||||||
|
|
||||||
|
Positioned BEFORE the EVIDENCE FOR SYNTHESIS envelope in render_compact
|
||||||
|
so the model's pass-through contract forces it into the user's view per
|
||||||
|
LAW 7. The 2026-04-19 Hermes Agent Use Cases Run 1 failure mode: the
|
||||||
|
engine's stderr warning about "no LLM provider" was invisible to the
|
||||||
|
user because Claude hid stderr. User-visible stdout block is the
|
||||||
|
backstop that makes silent degradation impossible.
|
||||||
|
"""
|
||||||
|
plan_source = report.artifacts.get("plan_source", "unknown")
|
||||||
|
flags_present = bool(report.artifacts.get("pre_research_flags_present", False))
|
||||||
|
if plan_source != "deterministic":
|
||||||
|
return []
|
||||||
|
if flags_present:
|
||||||
|
return []
|
||||||
|
if not _is_pre_research_eligible(report.topic):
|
||||||
|
return []
|
||||||
|
|
||||||
|
return [
|
||||||
|
"<!-- USER-VISIBLE BANNER: emit verbatim before synthesis per LAW 5 / LAW 7. -->",
|
||||||
|
"## DEGRADED RUN WARNING",
|
||||||
|
"",
|
||||||
|
f"⚠️ This run was called BARE on a named-entity topic (\"{report.topic}\"):",
|
||||||
|
"no `--plan` JSON, no pre-flight handle resolution. The engine ran",
|
||||||
|
"deterministic fallback retrieval, which is the cron/headless path -",
|
||||||
|
"weaker than the path where the hosting reasoning model authors its own",
|
||||||
|
"plan upstream.",
|
||||||
|
"",
|
||||||
|
"**If you are the reasoning model hosting this skill** (Claude Code, Codex,",
|
||||||
|
"Hermes, Gemini, or any agent runtime that invoked `/last30days`): YOU ARE",
|
||||||
|
"THE LLM. You do not need an API key, provider credentials, or an external",
|
||||||
|
"planning service. The `--plan` flag exists precisely so a reasoning model",
|
||||||
|
"generates its own JSON plan upstream and passes it in. That is you.",
|
||||||
|
"Generate a plan with `intent` / `freshness_mode` / 3-5 paraphrased",
|
||||||
|
"subqueries and re-run with `--plan '$JSON'`. This is not a",
|
||||||
|
"missing-credentials problem; this is a skipped-LAW-7 problem.",
|
||||||
|
"",
|
||||||
|
"What went wrong: on a named-entity topic, the full contract is",
|
||||||
|
"(a) resolve X handles / GitHub repos / subreddits via your runtime's",
|
||||||
|
"web-search tool (Step 0.55) and (b) generate a JSON `--plan` yourself",
|
||||||
|
"and pass it via `--plan '$JSON'` (Step 0.75 / LAW 7). Both were skipped.",
|
||||||
|
"",
|
||||||
|
"**If you are a user reading this:** the assistant skipped its own",
|
||||||
|
"planning step. Ask it to regenerate following Step 0.55 and Step 0.75",
|
||||||
|
"of SKILL.md.",
|
||||||
|
"<!-- END USER-VISIBLE BANNER -->",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def _parse_comparison_entities(topic: str) -> list[str] | None:
|
def _parse_comparison_entities(topic: str) -> list[str] | None:
|
||||||
"""Return list of entity names if topic is a comparison query, else None.
|
"""Return list of entity names if topic is a comparison query, else None.
|
||||||
|
|
||||||
|
|||||||
+125
-11
@@ -3,8 +3,34 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
|
import re
|
||||||
|
|
||||||
from . import http, providers, schema
|
from . import http, providers, query, schema
|
||||||
|
|
||||||
|
|
||||||
|
# Penalty applied when a candidate does not mention the primary entity
|
||||||
|
# from the topic in its title or snippet. Picked empirically: a typical
|
||||||
|
# score spread in the shortlist is 30-70, so 25 points reliably pushes
|
||||||
|
# an off-topic candidate below on-topic ones without fully zeroing out
|
||||||
|
# marginal matches. See 2026-04-19 Hermes Agent Use Cases failure: a
|
||||||
|
# Nate Herk "Managed Agents" video scored 51 / ranked #2 with zero
|
||||||
|
# Hermes content.
|
||||||
|
ENTITY_MISS_PENALTY = 25.0
|
||||||
|
|
||||||
|
# Intent modifiers to strip before extracting the primary entity so that,
|
||||||
|
# for example, "Hermes Agent use cases" yields primary_entity="hermes agent"
|
||||||
|
# rather than "hermes agent use cases". Kept in sync with
|
||||||
|
# planner._INTENT_MODIFIER_PATTERNS.
|
||||||
|
_INTENT_MODIFIER_RE = re.compile(
|
||||||
|
r"\b("
|
||||||
|
r"use cases|use case|workflows|workflow|"
|
||||||
|
r"examples|example|tutorial|tutorials|"
|
||||||
|
r"review|reviews|comparison|applications|"
|
||||||
|
r"in practice|production use|production|"
|
||||||
|
r"how i use"
|
||||||
|
r")\b",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
INTENT_SCORING_HINTS: dict[str, str] = {
|
INTENT_SCORING_HINTS: dict[str, str] = {
|
||||||
"comparison": (
|
"comparison": (
|
||||||
@@ -60,20 +86,21 @@ def rerank_candidates(
|
|||||||
) -> list[schema.Candidate]:
|
) -> list[schema.Candidate]:
|
||||||
"""Rerank the fused shortlist, demoting candidates the reranker scored as irrelevant."""
|
"""Rerank the fused shortlist, demoting candidates the reranker scored as irrelevant."""
|
||||||
shortlisted = candidates[:shortlist_size]
|
shortlisted = candidates[:shortlist_size]
|
||||||
|
primary_entity = _primary_entity(topic)
|
||||||
if provider and model and shortlisted:
|
if provider and model and shortlisted:
|
||||||
try:
|
try:
|
||||||
response = provider.generate_json(model, _build_prompt(topic, plan, shortlisted))
|
response = provider.generate_json(model, _build_prompt(topic, plan, shortlisted, primary_entity))
|
||||||
_apply_llm_scores(shortlisted, response)
|
_apply_llm_scores(shortlisted, response)
|
||||||
except (ValueError, KeyError, json.JSONDecodeError, OSError, http.HTTPError) as exc:
|
except (ValueError, KeyError, json.JSONDecodeError, OSError, http.HTTPError) as exc:
|
||||||
import sys
|
import sys
|
||||||
print(f"[Rerank] LLM reranking failed, using local fallback: {type(exc).__name__}: {exc}", file=sys.stderr)
|
print(f"[Rerank] LLM reranking failed, using local fallback: {type(exc).__name__}: {exc}", file=sys.stderr)
|
||||||
_apply_fallback_scores(shortlisted)
|
_apply_fallback_scores(shortlisted, primary_entity=primary_entity)
|
||||||
else:
|
else:
|
||||||
_apply_fallback_scores(shortlisted)
|
_apply_fallback_scores(shortlisted, primary_entity=primary_entity)
|
||||||
|
|
||||||
if len(candidates) > shortlist_size:
|
if len(candidates) > shortlist_size:
|
||||||
tail = candidates[shortlist_size:]
|
tail = candidates[shortlist_size:]
|
||||||
_apply_fallback_scores(tail)
|
_apply_fallback_scores(tail, primary_entity=primary_entity)
|
||||||
|
|
||||||
return sorted(
|
return sorted(
|
||||||
candidates,
|
candidates,
|
||||||
@@ -103,7 +130,7 @@ def _fenced_untrusted_content(candidate_block: str) -> str:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _build_prompt(topic: str, plan: schema.QueryPlan, candidates: list[schema.Candidate]) -> str:
|
def _build_prompt(topic: str, plan: schema.QueryPlan, candidates: list[schema.Candidate], primary_entity: str = "") -> str:
|
||||||
ranking_queries = "\n".join(
|
ranking_queries = "\n".join(
|
||||||
f"- {subquery.label}: {subquery.ranking_query}"
|
f"- {subquery.label}: {subquery.ranking_query}"
|
||||||
for subquery in plan.subqueries
|
for subquery in plan.subqueries
|
||||||
@@ -121,6 +148,16 @@ def _build_prompt(topic: str, plan: schema.QueryPlan, candidates: list[schema.Ca
|
|||||||
)
|
)
|
||||||
for candidate in candidates
|
for candidate in candidates
|
||||||
)
|
)
|
||||||
|
grounding_hint = ""
|
||||||
|
if primary_entity:
|
||||||
|
grounding_hint = (
|
||||||
|
f"\nPrimary entity grounding: the user's primary entity is \"{primary_entity}\". "
|
||||||
|
"A candidate that does NOT mention this entity (or a clear synonym/abbreviation) "
|
||||||
|
"in its title or snippet should score no higher than 30, regardless of other "
|
||||||
|
"signals. Do not let a candidate match the topic vicinity without matching the "
|
||||||
|
"entity itself. 2026-04-19 Hermes Agent Use Cases failure: a Nate Herk video "
|
||||||
|
"about Claude's Managed Agents scored 51 with zero Hermes content.\n"
|
||||||
|
)
|
||||||
return f"""
|
return f"""
|
||||||
Judge search-result relevance for a last-30-days research pipeline.
|
Judge search-result relevance for a last-30-days research pipeline.
|
||||||
|
|
||||||
@@ -145,7 +182,7 @@ Scoring guidance:
|
|||||||
- 70 to 89: clearly relevant and useful
|
- 70 to 89: clearly relevant and useful
|
||||||
- 40 to 69: somewhat relevant but weaker
|
- 40 to 69: somewhat relevant but weaker
|
||||||
- 0 to 39: weak, redundant, or off-target
|
- 0 to 39: weak, redundant, or off-target
|
||||||
{_intent_hint_block(plan)}
|
{grounding_hint}{_intent_hint_block(plan)}
|
||||||
{_fenced_untrusted_content(candidate_block)}
|
{_fenced_untrusted_content(candidate_block)}
|
||||||
""".strip()
|
""".strip()
|
||||||
|
|
||||||
@@ -169,21 +206,93 @@ def _apply_llm_scores(candidates: list[schema.Candidate], payload: dict) -> None
|
|||||||
candidate.final_score = _final_score(candidate)
|
candidate.final_score = _final_score(candidate)
|
||||||
|
|
||||||
|
|
||||||
def _apply_fallback_scores(candidates: list[schema.Candidate]) -> None:
|
def _apply_fallback_scores(candidates: list[schema.Candidate], *, primary_entity: str = "") -> None:
|
||||||
for candidate in candidates:
|
for candidate in candidates:
|
||||||
rerank_score, reason = _fallback_tuple(candidate)
|
rerank_score, reason = _fallback_tuple(candidate, primary_entity=primary_entity)
|
||||||
candidate.rerank_score = rerank_score
|
candidate.rerank_score = rerank_score
|
||||||
candidate.explanation = reason
|
candidate.explanation = reason
|
||||||
candidate.final_score = _final_score(candidate)
|
candidate.final_score = _final_score(candidate)
|
||||||
|
|
||||||
|
|
||||||
def _fallback_tuple(candidate: schema.Candidate) -> tuple[float, str]:
|
def _candidate_haystack(candidate: schema.Candidate) -> str:
|
||||||
|
"""Build the lowercase text blob against which entity-grounding is checked.
|
||||||
|
|
||||||
|
Expanded 2026-04-19 to include transcript snippets, transcript highlights,
|
||||||
|
and top-comment text. The prior `title + snippet` check missed YouTube
|
||||||
|
videos whose entity mentions live in transcript content and Reddit posts
|
||||||
|
whose mentions are in top comments. Now checks all text surfaces a human
|
||||||
|
would see.
|
||||||
|
"""
|
||||||
|
parts: list[str] = [candidate.title or "", candidate.snippet or ""]
|
||||||
|
metadata = candidate.metadata or {}
|
||||||
|
|
||||||
|
transcript_snippet = metadata.get("transcript_snippet") or ""
|
||||||
|
if isinstance(transcript_snippet, str):
|
||||||
|
parts.append(transcript_snippet)
|
||||||
|
|
||||||
|
for hl in metadata.get("transcript_highlights") or []:
|
||||||
|
if isinstance(hl, str):
|
||||||
|
parts.append(hl)
|
||||||
|
|
||||||
|
for tc in metadata.get("top_comments") or []:
|
||||||
|
if isinstance(tc, dict):
|
||||||
|
parts.append(str(tc.get("excerpt", "") or tc.get("text", "") or ""))
|
||||||
|
elif isinstance(tc, str):
|
||||||
|
parts.append(tc)
|
||||||
|
|
||||||
|
for insight in metadata.get("comment_insights") or []:
|
||||||
|
if isinstance(insight, str):
|
||||||
|
parts.append(insight)
|
||||||
|
|
||||||
|
return " ".join(parts).lower()
|
||||||
|
|
||||||
|
|
||||||
|
def _fallback_tuple(candidate: schema.Candidate, *, primary_entity: str = "") -> tuple[float, str]:
|
||||||
score = (
|
score = (
|
||||||
(candidate.local_relevance * 100.0 * 0.7)
|
(candidate.local_relevance * 100.0 * 0.7)
|
||||||
+ (candidate.freshness * 0.2)
|
+ (candidate.freshness * 0.2)
|
||||||
+ (candidate.source_quality * 100.0 * 0.1)
|
+ (candidate.source_quality * 100.0 * 0.1)
|
||||||
)
|
)
|
||||||
return max(0.0, min(100.0, score)), "fallback-local-score"
|
reason = "fallback-local-score"
|
||||||
|
# Entity-grounding demotion: if the primary entity (topic minus intent
|
||||||
|
# modifier) is not present anywhere in the candidate's text surfaces
|
||||||
|
# (title, snippet, transcript, transcript highlights, top comments,
|
||||||
|
# insights), subtract ENTITY_MISS_PENALTY. Skip for candidates with
|
||||||
|
# NO text anywhere (e.g., image-only TikToks) to avoid penalizing
|
||||||
|
# thin-text sources unfairly. 2026-04-19 Nate Herk "Managed Agents"
|
||||||
|
# video ranked #2 on a Hermes query despite zero Hermes mentions
|
||||||
|
# because the old haystack only checked title + snippet.
|
||||||
|
if primary_entity:
|
||||||
|
haystack = _candidate_haystack(candidate)
|
||||||
|
if haystack.strip() and primary_entity.lower() not in haystack:
|
||||||
|
score -= ENTITY_MISS_PENALTY
|
||||||
|
reason = "fallback-local-score (entity-miss demotion)"
|
||||||
|
return max(0.0, min(100.0, score)), reason
|
||||||
|
|
||||||
|
|
||||||
|
def _primary_entity(topic: str) -> str:
|
||||||
|
"""Extract the primary entity from the topic for grounding checks.
|
||||||
|
|
||||||
|
Strips intent-modifier suffixes (see planner._INTENT_MODIFIER_PATTERNS),
|
||||||
|
trims trailing punctuation, collapses whitespace. Returns the empty
|
||||||
|
string for topics that are all intent modifier with no entity, so
|
||||||
|
callers can skip the grounding check.
|
||||||
|
"""
|
||||||
|
stripped = _INTENT_MODIFIER_RE.sub(" ", topic)
|
||||||
|
# Also collapse multiple spaces and strip punctuation.
|
||||||
|
stripped = re.sub(r"\s+", " ", stripped).strip(" \t\r\n?.,:;!")
|
||||||
|
return stripped
|
||||||
|
|
||||||
|
|
||||||
|
#: Secondary entity-miss penalty applied directly to final_score (not just
|
||||||
|
#: rerank_score). The -25 on rerank_score composes to only -15 on final_score
|
||||||
|
#: via the 0.60 weight, which engagement bonus partially offsets on
|
||||||
|
#: high-view YouTube items. This secondary penalty lands the full weight on
|
||||||
|
#: the composite signal the cluster-scoring layer consumes. 2026-04-19
|
||||||
|
#: Nate Herk "Managed Agents" video ranked at cluster #2 with score 51
|
||||||
|
#: despite the rerank_score demotion because engagement + freshness drowned
|
||||||
|
#: the dilute penalty. This backstop makes the demotion actually decisive.
|
||||||
|
ENTITY_MISS_FINAL_PENALTY = 20.0
|
||||||
|
|
||||||
|
|
||||||
def _final_score(candidate: schema.Candidate) -> float:
|
def _final_score(candidate: schema.Candidate) -> float:
|
||||||
@@ -204,6 +313,11 @@ def _final_score(candidate: schema.Candidate) -> float:
|
|||||||
)
|
)
|
||||||
if candidate.rerank_score is not None and candidate.rerank_score < 20.0:
|
if candidate.rerank_score is not None and candidate.rerank_score < 20.0:
|
||||||
base *= 0.3
|
base *= 0.3
|
||||||
|
# Secondary entity-grounding penalty: when the fallback path flagged
|
||||||
|
# entity-miss via candidate.explanation, apply an additional penalty
|
||||||
|
# at final_score level so engagement signal can't mask the demotion.
|
||||||
|
if candidate.explanation and "entity-miss" in candidate.explanation:
|
||||||
|
base = max(0.0, base - ENTITY_MISS_FINAL_PENALTY)
|
||||||
return base
|
return base
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -28,6 +28,30 @@ class PipelineV3Tests(unittest.TestCase):
|
|||||||
self.assertIn("grounding", report.items_by_source)
|
self.assertIn("grounding", report.items_by_source)
|
||||||
self.assertEqual("gemini", report.provider_runtime.reasoning_provider)
|
self.assertEqual("gemini", report.provider_runtime.reasoning_provider)
|
||||||
|
|
||||||
|
def test_planner_trace_always_fires_on_mock_run(self):
|
||||||
|
"""Unit 5: The unified planner trace emits one summary line plus one
|
||||||
|
line per subquery on every run, regardless of --debug. 2026-04-19
|
||||||
|
Hermes Agent Use Cases failure: retrieval-breadth issues were invisible
|
||||||
|
because the internal planner path logged nothing.
|
||||||
|
"""
|
||||||
|
import io
|
||||||
|
import contextlib
|
||||||
|
buf = io.StringIO()
|
||||||
|
with contextlib.redirect_stderr(buf):
|
||||||
|
pipeline.run(
|
||||||
|
topic="test topic",
|
||||||
|
config={"LAST30DAYS_REASONING_PROVIDER": "gemini"},
|
||||||
|
depth="quick",
|
||||||
|
requested_sources=["reddit", "x", "grounding"],
|
||||||
|
mock=True,
|
||||||
|
)
|
||||||
|
output = buf.getvalue()
|
||||||
|
self.assertIn("[Planner] Plan: intent=", output)
|
||||||
|
self.assertIn("subqueries=", output)
|
||||||
|
self.assertIn("source=", output)
|
||||||
|
# At least one per-subquery line.
|
||||||
|
self.assertIn("[Planner] sq1 label=", output)
|
||||||
|
|
||||||
|
|
||||||
class TestSourceFetchCap(unittest.TestCase):
|
class TestSourceFetchCap(unittest.TestCase):
|
||||||
"""X source fetch count must be capped by MAX_SOURCE_FETCHES."""
|
"""X source fetch count must be capped by MAX_SOURCE_FETCHES."""
|
||||||
|
|||||||
@@ -281,5 +281,177 @@ class PlannerV3Tests(unittest.TestCase):
|
|||||||
self.assertIn("instagram", all_sources)
|
self.assertIn("instagram", all_sources)
|
||||||
|
|
||||||
|
|
||||||
|
class IntentModifierBreadthTests(unittest.TestCase):
|
||||||
|
"""Unit 2: Topics with intent modifiers (use cases, workflows, examples,
|
||||||
|
review, comparison) must fan out across paraphrased subqueries rather
|
||||||
|
than echo the literal phrase. 2026-04-19 Hermes Agent Use Cases failure.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_max_subqueries_raised_to_5_for_how_to(self):
|
||||||
|
self.assertEqual(5, planner._max_subqueries("how_to"))
|
||||||
|
|
||||||
|
def test_max_subqueries_raised_to_5_for_opinion(self):
|
||||||
|
self.assertEqual(5, planner._max_subqueries("opinion"))
|
||||||
|
|
||||||
|
def test_max_subqueries_raised_to_5_for_product(self):
|
||||||
|
self.assertEqual(5, planner._max_subqueries("product"))
|
||||||
|
|
||||||
|
def test_max_subqueries_unchanged_for_comparison(self):
|
||||||
|
self.assertEqual(4, planner._max_subqueries("comparison"))
|
||||||
|
|
||||||
|
def test_max_subqueries_unchanged_for_factual_and_concept(self):
|
||||||
|
self.assertEqual(2, planner._max_subqueries("factual"))
|
||||||
|
self.assertEqual(2, planner._max_subqueries("concept"))
|
||||||
|
|
||||||
|
def test_has_intent_modifier_detects_use_cases(self):
|
||||||
|
self.assertTrue(planner._has_intent_modifier("Hermes Agent use cases"))
|
||||||
|
self.assertTrue(planner._has_intent_modifier("Hermes Agent Actual Use Cases"))
|
||||||
|
|
||||||
|
def test_has_intent_modifier_detects_workflows(self):
|
||||||
|
self.assertTrue(planner._has_intent_modifier("Claude Code workflows"))
|
||||||
|
|
||||||
|
def test_has_intent_modifier_detects_review_and_tutorial(self):
|
||||||
|
self.assertTrue(planner._has_intent_modifier("Ollama review"))
|
||||||
|
self.assertTrue(planner._has_intent_modifier("DSPy tutorial"))
|
||||||
|
|
||||||
|
def test_has_intent_modifier_false_for_bare_entity(self):
|
||||||
|
self.assertFalse(planner._has_intent_modifier("Kanye West"))
|
||||||
|
self.assertFalse(planner._has_intent_modifier("hermes agent"))
|
||||||
|
|
||||||
|
def test_fallback_fans_out_when_intent_modifier_present(self):
|
||||||
|
plan = planner.plan_query(
|
||||||
|
topic="Hermes Agent use cases",
|
||||||
|
available_sources=["reddit", "x", "youtube", "hackernews"],
|
||||||
|
requested_sources=None,
|
||||||
|
depth="default",
|
||||||
|
provider=None,
|
||||||
|
model=None,
|
||||||
|
)
|
||||||
|
# Expect at least 3 subqueries total (primary + fanout); cap is 5 for
|
||||||
|
# how_to/opinion/product/breaking_news. Label set should include at
|
||||||
|
# least one of the paraphrase labels.
|
||||||
|
labels = {sq.label for sq in plan.subqueries}
|
||||||
|
self.assertGreaterEqual(len(plan.subqueries), 3)
|
||||||
|
self.assertTrue(
|
||||||
|
labels & {"workflows", "production", "experience"},
|
||||||
|
f"Expected paraphrase labels in {labels}",
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_fallback_does_not_fan_out_for_bare_entity(self):
|
||||||
|
plan = planner.plan_query(
|
||||||
|
topic="Kanye West",
|
||||||
|
available_sources=["reddit", "x", "grounding"],
|
||||||
|
requested_sources=None,
|
||||||
|
depth="default",
|
||||||
|
provider=None,
|
||||||
|
model=None,
|
||||||
|
)
|
||||||
|
# Bare entity without intent modifier should not trigger the paraphrase
|
||||||
|
# fanout (those labels are not in the plan).
|
||||||
|
labels = {sq.label for sq in plan.subqueries}
|
||||||
|
self.assertFalse(labels & {"workflows", "production", "experience"})
|
||||||
|
|
||||||
|
def test_prompt_includes_intent_modifier_rule(self):
|
||||||
|
prompt = planner._build_prompt(
|
||||||
|
topic="Hermes Agent use cases",
|
||||||
|
available_sources=["reddit", "x", "youtube"],
|
||||||
|
requested_sources=None,
|
||||||
|
depth="default",
|
||||||
|
)
|
||||||
|
self.assertIn("INTENT-MODIFIER HANDLING", prompt)
|
||||||
|
self.assertIn("use cases", prompt)
|
||||||
|
self.assertIn("STRIP that phrase", prompt)
|
||||||
|
|
||||||
|
|
||||||
|
class FallbackDefaultsTests(unittest.TestCase):
|
||||||
|
"""Unit 3: Deterministic fallback defaults and keyword_query quoting.
|
||||||
|
2026-04-19 Hermes Agent Use Cases failure.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_unclassified_topic_defaults_to_concept_not_breaking_news(self):
|
||||||
|
# Prior default was "breaking_news" with strict_recent freshness,
|
||||||
|
# which biased against older relevant material on unfamiliar topics.
|
||||||
|
self.assertEqual("concept", planner._infer_intent("some unfamiliar topic"))
|
||||||
|
self.assertEqual("concept", planner._infer_intent("Hermes Agent"))
|
||||||
|
|
||||||
|
def test_recency_signals_still_break_out_to_breaking_news(self):
|
||||||
|
self.assertEqual("breaking_news", planner._infer_intent("trending AI tools"))
|
||||||
|
self.assertEqual("breaking_news", planner._infer_intent("what's happening today"))
|
||||||
|
self.assertEqual("breaking_news", planner._infer_intent("this week in AI"))
|
||||||
|
|
||||||
|
def test_specific_intents_still_classify_correctly(self):
|
||||||
|
# Regression: other regex branches still fire as before.
|
||||||
|
self.assertEqual("how_to", planner._infer_intent("how to deploy Docker"))
|
||||||
|
self.assertEqual("factual", planner._infer_intent("who acquired Wiz"))
|
||||||
|
self.assertEqual("opinion", planner._infer_intent("thoughts on OpenAI Codex"))
|
||||||
|
self.assertEqual("comparison", planner._infer_intent("Codex vs Claude Code"))
|
||||||
|
|
||||||
|
def test_keyword_query_quotes_only_title_cased_proper_nouns(self):
|
||||||
|
# "Hermes Agent" is a multi-word title-cased proper noun — keep quoted.
|
||||||
|
# "Use Cases" is also title-cased BUT we only quote the first 2
|
||||||
|
# title-cased compounds; the first extracted is "Hermes Agent".
|
||||||
|
search = planner._keyword_query("Hermes Agent use cases", "hermes agent")
|
||||||
|
self.assertIn('"Hermes Agent"', search)
|
||||||
|
# The old behavior quoted the entire typed topic; confirm it does not.
|
||||||
|
self.assertNotIn('"Hermes Agent Actual Use Cases"', search)
|
||||||
|
|
||||||
|
def test_keyword_query_does_not_quote_bare_lowercase_topic(self):
|
||||||
|
search = planner._keyword_query("kanye west bully", "kanye west bully")
|
||||||
|
# Lowercase topics have no title-cased compound to quote.
|
||||||
|
self.assertNotIn('"', search)
|
||||||
|
|
||||||
|
def test_fallback_logs_warning_when_no_provider(self):
|
||||||
|
import io
|
||||||
|
import contextlib
|
||||||
|
buf = io.StringIO()
|
||||||
|
with contextlib.redirect_stderr(buf):
|
||||||
|
planner.plan_query(
|
||||||
|
topic="Hermes Agent use cases",
|
||||||
|
available_sources=["reddit", "x"],
|
||||||
|
requested_sources=None,
|
||||||
|
depth="default",
|
||||||
|
provider=None,
|
||||||
|
model=None,
|
||||||
|
)
|
||||||
|
output = buf.getvalue()
|
||||||
|
# New language: "No --plan passed" + "YOU ARE the planner" +
|
||||||
|
# runtime enumeration. Unit 4 (2026-04-19) rewrite to stop the
|
||||||
|
# "no provider = no LLM = I need a key" misread.
|
||||||
|
self.assertIn("No --plan passed", output)
|
||||||
|
self.assertIn("YOU ARE the planner", output)
|
||||||
|
self.assertIn("you ARE the LLM", output)
|
||||||
|
# Runtime-agnostic: each supported runtime name should appear.
|
||||||
|
for runtime_name in ("Claude Code", "Codex", "Hermes", "Gemini"):
|
||||||
|
self.assertIn(runtime_name, output)
|
||||||
|
# The old misleading phrasing must NOT appear.
|
||||||
|
self.assertNotIn("No --plan and no LLM provider configured", output)
|
||||||
|
|
||||||
|
def test_fallback_does_not_log_new_warning_when_provider_present(self):
|
||||||
|
# When a provider is configured, the provider path runs; if it
|
||||||
|
# errors, we get the "LLM planning failed" message, NOT the
|
||||||
|
# "No --plan passed" guidance (which is specifically for the
|
||||||
|
# no-provider-no-plan caller path).
|
||||||
|
import io
|
||||||
|
import contextlib
|
||||||
|
buf = io.StringIO()
|
||||||
|
|
||||||
|
class _NoopProvider:
|
||||||
|
def generate_json(self, model, prompt):
|
||||||
|
raise ValueError("force fallback for test")
|
||||||
|
|
||||||
|
with contextlib.redirect_stderr(buf):
|
||||||
|
planner.plan_query(
|
||||||
|
topic="Kanye West",
|
||||||
|
available_sources=["reddit", "x"],
|
||||||
|
requested_sources=None,
|
||||||
|
depth="default",
|
||||||
|
provider=_NoopProvider(),
|
||||||
|
model="some-model",
|
||||||
|
)
|
||||||
|
output = buf.getvalue()
|
||||||
|
self.assertIn("LLM planning failed", output)
|
||||||
|
self.assertNotIn("No --plan passed", output)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
+127
-2
@@ -117,8 +117,72 @@ class RenderV3Tests(unittest.TestCase):
|
|||||||
report.errors_by_source = {"x": "HTTP 400: Bad Request"}
|
report.errors_by_source = {"x": "HTTP 400: Bad Request"}
|
||||||
text = render.render_compact(report)
|
text = render.render_compact(report)
|
||||||
self.assertIn("## Source Errors", text)
|
self.assertIn("## Source Errors", text)
|
||||||
self.assertIn("HTTP 400: Bad Request", text)
|
|
||||||
self.assertIn("X:", text)
|
|
||||||
|
class OutputEnvelopeTests(unittest.TestCase):
|
||||||
|
"""LAW 6 envelope comments: scope "pass through verbatim" unambiguously.
|
||||||
|
|
||||||
|
Added 2026-04-19 after the Hermes Agent Use Cases failure where two
|
||||||
|
consecutive runs dumped `## Ranked Evidence Clusters` as user output.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_evidence_for_synthesis_envelope_wraps_raw_evidence(self):
|
||||||
|
text = render.render_compact(sample_report())
|
||||||
|
self.assertIn("<!-- EVIDENCE FOR SYNTHESIS:", text)
|
||||||
|
self.assertIn("<!-- END EVIDENCE FOR SYNTHESIS -->", text)
|
||||||
|
# Opening comment must appear BEFORE the raw evidence block.
|
||||||
|
self.assertLess(
|
||||||
|
text.index("<!-- EVIDENCE FOR SYNTHESIS:"),
|
||||||
|
text.index("## Ranked Evidence Clusters"),
|
||||||
|
)
|
||||||
|
# Closing comment must appear AFTER Source Coverage.
|
||||||
|
self.assertGreater(
|
||||||
|
text.index("<!-- END EVIDENCE FOR SYNTHESIS -->"),
|
||||||
|
text.index("## Source Coverage"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_pass_through_footer_envelope_wraps_emoji_tree(self):
|
||||||
|
text = render.render_compact(sample_report())
|
||||||
|
self.assertIn("<!-- PASS-THROUGH FOOTER:", text)
|
||||||
|
self.assertIn("<!-- END PASS-THROUGH FOOTER -->", text)
|
||||||
|
# Emoji footer sits between the two markers.
|
||||||
|
open_idx = text.index("<!-- PASS-THROUGH FOOTER:")
|
||||||
|
close_idx = text.index("<!-- END PASS-THROUGH FOOTER -->")
|
||||||
|
self.assertIn("All agents reported back!", text[open_idx:close_idx])
|
||||||
|
|
||||||
|
def test_canonical_boundary_scopes_pass_through_to_footer(self):
|
||||||
|
text = render.render_compact(sample_report())
|
||||||
|
# New boundary text scopes verbatim to the PASS-THROUGH FOOTER block,
|
||||||
|
# not everything above.
|
||||||
|
self.assertIn("Pass through ONLY the PASS-THROUGH FOOTER block verbatim", text)
|
||||||
|
# Self-check string is present so the model has a concrete failure signal.
|
||||||
|
self.assertIn("### 1.", text)
|
||||||
|
self.assertIn("LAW 6", text)
|
||||||
|
# The prior ambiguous phrasing is gone.
|
||||||
|
self.assertNotIn("Pass through the lines ABOVE this boundary verbatim", text)
|
||||||
|
|
||||||
|
def test_envelopes_appear_in_md_emit_mode(self):
|
||||||
|
# --emit md and --emit compact both route to render_compact, so the
|
||||||
|
# same envelopes apply. Guard against future divergence.
|
||||||
|
text = render.render_compact(sample_report())
|
||||||
|
self.assertEqual(text.count("<!-- EVIDENCE FOR SYNTHESIS:"), 1)
|
||||||
|
self.assertEqual(text.count("<!-- END EVIDENCE FOR SYNTHESIS -->"), 1)
|
||||||
|
self.assertEqual(text.count("<!-- PASS-THROUGH FOOTER:"), 1)
|
||||||
|
self.assertEqual(text.count("<!-- END PASS-THROUGH FOOTER -->"), 1)
|
||||||
|
|
||||||
|
def test_no_dangling_envelope_open_without_close(self):
|
||||||
|
# Open/close counts must always match, even for empty clusters.
|
||||||
|
report = sample_report()
|
||||||
|
report.clusters = []
|
||||||
|
text = render.render_compact(report)
|
||||||
|
self.assertEqual(
|
||||||
|
text.count("<!-- EVIDENCE FOR SYNTHESIS:"),
|
||||||
|
text.count("<!-- END EVIDENCE FOR SYNTHESIS -->"),
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
text.count("<!-- PASS-THROUGH FOOTER:"),
|
||||||
|
text.count("<!-- END PASS-THROUGH FOOTER -->"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class RenderTopCommentsTests(unittest.TestCase):
|
class RenderTopCommentsTests(unittest.TestCase):
|
||||||
@@ -398,5 +462,66 @@ class RenderBestTakesCompactTests(unittest.TestCase):
|
|||||||
self.assertNotIn("## Best Takes", text)
|
self.assertNotIn("## Best Takes", text)
|
||||||
|
|
||||||
|
|
||||||
|
class DegradedRunBannerTests(unittest.TestCase):
|
||||||
|
"""Unit 1: DEGRADED RUN WARNING surfaces bare named-entity invocations
|
||||||
|
in user-visible stdout. LAW 7 backstop. 2026-04-19 Hermes Agent Use
|
||||||
|
Cases Run 1 failure mode.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def _bare_named_entity_report(self) -> schema.Report:
|
||||||
|
report = sample_report()
|
||||||
|
report.topic = "Hermes Agent"
|
||||||
|
report.artifacts["plan_source"] = "deterministic"
|
||||||
|
report.artifacts["pre_research_flags_present"] = False
|
||||||
|
return report
|
||||||
|
|
||||||
|
def test_banner_appears_on_bare_named_entity_deterministic_run(self):
|
||||||
|
text = render.render_compact(self._bare_named_entity_report())
|
||||||
|
self.assertIn("## DEGRADED RUN WARNING", text)
|
||||||
|
self.assertIn("<!-- USER-VISIBLE BANNER:", text)
|
||||||
|
self.assertIn("<!-- END USER-VISIBLE BANNER -->", text)
|
||||||
|
self.assertIn("YOU ARE", text)
|
||||||
|
# Runtime-agnostic enumeration: all host runtimes appear.
|
||||||
|
for runtime_name in ("Claude Code", "Codex", "Hermes", "Gemini"):
|
||||||
|
self.assertIn(runtime_name, text)
|
||||||
|
|
||||||
|
def test_banner_positioned_before_evidence_envelope(self):
|
||||||
|
text = render.render_compact(self._bare_named_entity_report())
|
||||||
|
banner_idx = text.index("## DEGRADED RUN WARNING")
|
||||||
|
envelope_idx = text.index("<!-- EVIDENCE FOR SYNTHESIS:")
|
||||||
|
self.assertLess(banner_idx, envelope_idx,
|
||||||
|
"DEGRADED RUN banner must appear BEFORE evidence envelope so pass-through catches it.")
|
||||||
|
|
||||||
|
def test_banner_suppressed_when_plan_source_external(self):
|
||||||
|
report = self._bare_named_entity_report()
|
||||||
|
report.artifacts["plan_source"] = "external"
|
||||||
|
text = render.render_compact(report)
|
||||||
|
self.assertNotIn("## DEGRADED RUN WARNING", text)
|
||||||
|
|
||||||
|
def test_banner_suppressed_when_plan_source_llm(self):
|
||||||
|
report = self._bare_named_entity_report()
|
||||||
|
report.artifacts["plan_source"] = "llm"
|
||||||
|
text = render.render_compact(report)
|
||||||
|
self.assertNotIn("## DEGRADED RUN WARNING", text)
|
||||||
|
|
||||||
|
def test_banner_suppressed_when_pre_research_flags_present(self):
|
||||||
|
report = self._bare_named_entity_report()
|
||||||
|
report.artifacts["pre_research_flags_present"] = True
|
||||||
|
text = render.render_compact(report)
|
||||||
|
self.assertNotIn("## DEGRADED RUN WARNING", text)
|
||||||
|
|
||||||
|
def test_banner_suppressed_on_non_eligible_abstract_topic(self):
|
||||||
|
report = self._bare_named_entity_report()
|
||||||
|
# Multi-word lowercase abstract phrase is NOT pre-research-eligible.
|
||||||
|
report.topic = "how to deploy containers in the cloud"
|
||||||
|
text = render.render_compact(report)
|
||||||
|
self.assertNotIn("## DEGRADED RUN WARNING", text)
|
||||||
|
|
||||||
|
def test_banner_mentions_law_7_and_plan_flag(self):
|
||||||
|
text = render.render_compact(self._bare_named_entity_report())
|
||||||
|
self.assertIn("LAW 7", text)
|
||||||
|
self.assertIn("--plan", text)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
+203
-1
@@ -178,9 +178,211 @@ class RerankV3Tests(unittest.TestCase):
|
|||||||
self.assertEqual("gemini-3.1-flash-lite-preview", provider.model)
|
self.assertEqual("gemini-3.1-flash-lite-preview", provider.model)
|
||||||
self.assertEqual(95.0, first.rerank_score)
|
self.assertEqual(95.0, first.rerank_score)
|
||||||
self.assertEqual("high fit", first.explanation)
|
self.assertEqual("high fit", first.explanation)
|
||||||
self.assertEqual("fallback-local-score", second.explanation)
|
# Tail is scored via the fallback (may or may not carry the entity-miss
|
||||||
|
# suffix depending on topic-title overlap; assert the base tag is present).
|
||||||
|
self.assertIn("fallback-local-score", second.explanation or "")
|
||||||
self.assertEqual(first.candidate_id, ranked[0].candidate_id)
|
self.assertEqual(first.candidate_id, ranked[0].candidate_id)
|
||||||
|
|
||||||
|
|
||||||
|
class EntityGroundingTests(unittest.TestCase):
|
||||||
|
"""Unit 4: Reranker entity-grounding demotion. 2026-04-19 Hermes Agent
|
||||||
|
Use Cases failure: an off-topic video about Claude Managed Agents
|
||||||
|
scored 51 and ranked #2 with zero Hermes content.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def _candidate(self, title: str, snippet: str = "") -> schema.Candidate:
|
||||||
|
return schema.Candidate(
|
||||||
|
candidate_id=f"c-{title[:10]}",
|
||||||
|
item_id="i1",
|
||||||
|
source="youtube",
|
||||||
|
title=title,
|
||||||
|
url="https://example.com",
|
||||||
|
snippet=snippet,
|
||||||
|
subquery_labels=["primary"],
|
||||||
|
native_ranks={"primary:youtube": 1},
|
||||||
|
local_relevance=0.8,
|
||||||
|
freshness=80,
|
||||||
|
engagement=50,
|
||||||
|
source_quality=0.7,
|
||||||
|
rrf_score=0.02,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_primary_entity_strips_intent_modifier(self):
|
||||||
|
self.assertEqual("Hermes Agent", rerank._primary_entity("Hermes Agent use cases"))
|
||||||
|
self.assertEqual("Hermes Agent Actual", rerank._primary_entity("Hermes Agent Actual Use Cases"))
|
||||||
|
self.assertEqual("Claude Code", rerank._primary_entity("Claude Code workflows"))
|
||||||
|
self.assertEqual("DSPy", rerank._primary_entity("DSPy tutorial"))
|
||||||
|
|
||||||
|
def test_primary_entity_leaves_bare_entity_unchanged(self):
|
||||||
|
self.assertEqual("Kanye West", rerank._primary_entity("Kanye West"))
|
||||||
|
self.assertEqual("Nous Research", rerank._primary_entity("Nous Research"))
|
||||||
|
|
||||||
|
def test_fallback_demotes_candidate_without_primary_entity(self):
|
||||||
|
on_topic = self._candidate("Hermes Agent: Self-Improving AI", "Nous Research Hermes walkthrough")
|
||||||
|
off_topic = self._candidate("I Tested Claude's Managed Agents", "What you need to know about Anthropic's new managed agents")
|
||||||
|
rerank._apply_fallback_scores([on_topic, off_topic], primary_entity="Hermes Agent")
|
||||||
|
self.assertGreater(on_topic.final_score, off_topic.final_score)
|
||||||
|
self.assertIn("entity-miss", off_topic.explanation or "")
|
||||||
|
self.assertEqual(on_topic.explanation, "fallback-local-score")
|
||||||
|
|
||||||
|
def test_fallback_match_is_case_insensitive(self):
|
||||||
|
on_topic = self._candidate("HERMES agent rocks", "some text")
|
||||||
|
rerank._apply_fallback_scores([on_topic], primary_entity="Hermes Agent")
|
||||||
|
self.assertEqual("fallback-local-score", on_topic.explanation)
|
||||||
|
|
||||||
|
def test_fallback_skips_demotion_for_empty_text_candidates(self):
|
||||||
|
empty = self._candidate("", "")
|
||||||
|
rerank._apply_fallback_scores([empty], primary_entity="Hermes Agent")
|
||||||
|
self.assertEqual("fallback-local-score", empty.explanation)
|
||||||
|
|
||||||
|
def test_fallback_skips_demotion_when_no_primary_entity(self):
|
||||||
|
off = self._candidate("Completely unrelated", "snippet")
|
||||||
|
rerank._apply_fallback_scores([off], primary_entity="")
|
||||||
|
self.assertEqual("fallback-local-score", off.explanation)
|
||||||
|
|
||||||
|
def test_llm_prompt_includes_primary_entity_grounding_hint(self):
|
||||||
|
candidate = self._candidate("Something", "snippet text")
|
||||||
|
plan = make_plan()
|
||||||
|
prompt = rerank._build_prompt(
|
||||||
|
"Hermes Agent use cases", plan, [candidate], primary_entity="Hermes Agent"
|
||||||
|
)
|
||||||
|
self.assertIn("Primary entity grounding", prompt)
|
||||||
|
self.assertIn("Hermes Agent", prompt)
|
||||||
|
|
||||||
|
def test_llm_prompt_omits_grounding_hint_when_no_primary_entity(self):
|
||||||
|
candidate = self._candidate("Something", "snippet text")
|
||||||
|
plan = make_plan()
|
||||||
|
prompt = rerank._build_prompt("", plan, [candidate], primary_entity="")
|
||||||
|
self.assertNotIn("Primary entity grounding", prompt)
|
||||||
|
|
||||||
|
|
||||||
|
class ExpandedHaystackTests(unittest.TestCase):
|
||||||
|
"""Unit 3: Entity-grounding haystack covers transcript snippets,
|
||||||
|
transcript highlights, top comments, and comment insights - not
|
||||||
|
just title + snippet.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def _youtube_candidate(self, title: str, transcript_snippet: str = "",
|
||||||
|
transcript_highlights: list[str] | None = None) -> schema.Candidate:
|
||||||
|
c = schema.Candidate(
|
||||||
|
candidate_id=f"c-{title[:10]}",
|
||||||
|
item_id="i1",
|
||||||
|
source="youtube",
|
||||||
|
title=title,
|
||||||
|
url="https://youtube.com/watch?v=x",
|
||||||
|
snippet="",
|
||||||
|
subquery_labels=["primary"],
|
||||||
|
native_ranks={"primary:youtube": 1},
|
||||||
|
local_relevance=0.8,
|
||||||
|
freshness=80,
|
||||||
|
engagement=50,
|
||||||
|
source_quality=0.7,
|
||||||
|
rrf_score=0.02,
|
||||||
|
)
|
||||||
|
c.metadata = {}
|
||||||
|
if transcript_snippet:
|
||||||
|
c.metadata["transcript_snippet"] = transcript_snippet
|
||||||
|
if transcript_highlights:
|
||||||
|
c.metadata["transcript_highlights"] = transcript_highlights
|
||||||
|
return c
|
||||||
|
|
||||||
|
def test_entity_found_in_transcript_snippet_avoids_demotion(self):
|
||||||
|
# Title + snippet miss the entity, but the transcript contains it.
|
||||||
|
c = self._youtube_candidate(
|
||||||
|
"Weekly roundup",
|
||||||
|
transcript_snippet="In this video I walk through using Hermes Agent in production.",
|
||||||
|
)
|
||||||
|
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
|
||||||
|
self.assertEqual("fallback-local-score", c.explanation)
|
||||||
|
|
||||||
|
def test_entity_found_in_transcript_highlights_avoids_demotion(self):
|
||||||
|
c = self._youtube_candidate(
|
||||||
|
"Some review",
|
||||||
|
transcript_highlights=[
|
||||||
|
"Today we're talking about Hermes Agent",
|
||||||
|
"Let's compare it to the alternatives",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
|
||||||
|
self.assertEqual("fallback-local-score", c.explanation)
|
||||||
|
|
||||||
|
def test_entity_missing_everywhere_still_demoted_for_video(self):
|
||||||
|
# Nate Herk "Managed Agents" case: no Hermes in title, snippet,
|
||||||
|
# or transcript - demotion fires.
|
||||||
|
c = self._youtube_candidate(
|
||||||
|
"I Tested Claude's New Managed Agents",
|
||||||
|
transcript_snippet="Managed agents are Anthropic's new product with ClickUp and cron...",
|
||||||
|
)
|
||||||
|
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
|
||||||
|
self.assertIn("entity-miss", c.explanation)
|
||||||
|
|
||||||
|
def test_entity_found_in_reddit_top_comments_avoids_demotion(self):
|
||||||
|
c = schema.Candidate(
|
||||||
|
candidate_id="r1",
|
||||||
|
item_id="i1",
|
||||||
|
source="reddit",
|
||||||
|
title="Best agent framework?",
|
||||||
|
url="https://reddit.com/r/x",
|
||||||
|
snippet="",
|
||||||
|
subquery_labels=["primary"],
|
||||||
|
native_ranks={"primary:reddit": 1},
|
||||||
|
local_relevance=0.8, freshness=80, engagement=50,
|
||||||
|
source_quality=0.7, rrf_score=0.02,
|
||||||
|
)
|
||||||
|
c.metadata = {
|
||||||
|
"top_comments": [
|
||||||
|
{"excerpt": "I've been using Hermes Agent for a month and it's great"},
|
||||||
|
{"text": "another comment"},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
|
||||||
|
self.assertEqual("fallback-local-score", c.explanation)
|
||||||
|
|
||||||
|
def test_entity_found_in_comment_insights_avoids_demotion(self):
|
||||||
|
c = schema.Candidate(
|
||||||
|
candidate_id="r2", item_id="i1", source="reddit",
|
||||||
|
title="AI tools", url="https://reddit.com/r/x", snippet="",
|
||||||
|
subquery_labels=["primary"],
|
||||||
|
native_ranks={"primary:reddit": 1},
|
||||||
|
local_relevance=0.8, freshness=80, engagement=50,
|
||||||
|
source_quality=0.7, rrf_score=0.02,
|
||||||
|
)
|
||||||
|
c.metadata = {
|
||||||
|
"comment_insights": ["Consensus: Hermes Agent handles long sessions best"],
|
||||||
|
}
|
||||||
|
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
|
||||||
|
self.assertEqual("fallback-local-score", c.explanation)
|
||||||
|
|
||||||
|
def test_truly_empty_candidate_still_skipped(self):
|
||||||
|
# Image-only TikTok with no text anywhere - do not penalize.
|
||||||
|
c = self._youtube_candidate("") # empty title
|
||||||
|
rerank._apply_fallback_scores([c], primary_entity="Hermes Agent")
|
||||||
|
self.assertEqual("fallback-local-score", c.explanation)
|
||||||
|
|
||||||
|
def test_final_score_secondary_penalty_applied_on_entity_miss(self):
|
||||||
|
# When fallback flags entity-miss, final_score gets an ADDITIONAL
|
||||||
|
# -20 penalty beyond the rerank_score reduction. Verify by
|
||||||
|
# comparing final_score for a demoted candidate vs an identical
|
||||||
|
# candidate that matched the entity.
|
||||||
|
off_topic = self._youtube_candidate("Managed Agents from Anthropic")
|
||||||
|
on_topic = self._youtube_candidate(
|
||||||
|
"Hermes Agent walkthrough",
|
||||||
|
transcript_snippet="Hermes Agent review",
|
||||||
|
)
|
||||||
|
rerank._apply_fallback_scores([off_topic, on_topic], primary_entity="Hermes Agent")
|
||||||
|
# Gap should be well above the rerank_score-only path's 0.60 * 25 = 15;
|
||||||
|
# with the secondary penalty it's 15 + 20 = 35 points.
|
||||||
|
gap = on_topic.final_score - off_topic.final_score
|
||||||
|
self.assertGreater(gap, 25.0,
|
||||||
|
f"entity-miss demotion gap only {gap:.1f}; secondary penalty may not be firing")
|
||||||
|
|
||||||
|
def test_secondary_penalty_not_applied_when_entity_match(self):
|
||||||
|
on_topic = self._youtube_candidate("Hermes Agent: use cases")
|
||||||
|
rerank._apply_fallback_scores([on_topic], primary_entity="Hermes Agent")
|
||||||
|
# Explanation does NOT contain entity-miss, so secondary penalty
|
||||||
|
# should not fire; final_score reflects only base signal.
|
||||||
|
self.assertNotIn("entity-miss", on_topic.explanation or "")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
Reference in New Issue
Block a user