fix: scope pass-through to footer only, add LAW 6 against raw cluster dumps
The engine's ## Ranked Evidence Clusters block is a scratchpad for the model to read, not user-facing output. Two consecutive /last30days runs on 2026-04-19 (Hermes Agent Use Cases) dumped it verbatim as user output because the prior canonical-boundary text (Pass through the lines ABOVE this boundary verbatim) was ambiguous about scope. Split render_compact stdout into two bounded blocks: - <!-- EVIDENCE FOR SYNTHESIS: ... --> wraps Ranked Evidence Clusters, Stats, and Source Coverage. Transform into prose per LAW 2. - <!-- PASS-THROUGH FOOTER: ... --> wraps the emoji-tree footer only. Emit verbatim per LAW 5. Rewrite _render_canonical_boundary to scope pass-through to the footer block explicitly and give the model a concrete self-check string (### 1. followed by a score tuple) as the named LAW 6 failure signal. Add LAW 6 to SKILL.md OUTPUT CONTRACT with the observed violation (2026-04-19 Hermes Agent Use Cases) and a worked transformation example.
This commit is contained in:
@@ -120,7 +120,41 @@ These five rules dominate every other rule in this file. If you find yourself ab
|
|||||||
|
|
||||||
**Observed LAW 4 violation (2026-04-18, Peter Steinberger disaster #2):** the model emitted `Headline`, `What he is actually saying`, `Cross-source corroboration`, `Where evidence is thin`, `Bottom line` on a GENERAL query. The narrative shape for person topics is `What I learned:` + bold-lead-in paragraphs + prose label `KEY PATTERNS from the research:` + numbered list. No blog-post subheadings.
|
**Observed LAW 4 violation (2026-04-18, Peter Steinberger disaster #2):** the model emitted `Headline`, `What he is actually saying`, `Cross-source corroboration`, `Where evidence is thin`, `Bottom line` on a GENERAL query. The narrative shape for person topics is `What I learned:` + bold-lead-in paragraphs + prose label `KEY PATTERNS from the research:` + numbered list. No blog-post subheadings.
|
||||||
|
|
||||||
**LAW 5 - ENGINE FOOTER PASS-THROUGH. EVERY QUERY TYPE. EVERY RUN.** The engine output ends with a `✅ All agents reported back!` emoji-tree footer bounded by `---` lines. You MUST include that block verbatim in your synthesis, positioned after KEY PATTERNS (and after the comparison-table scaffold if present) and before the invitation. Do not recompute the stats, reformat the tree, paraphrase, skip it, or fabricate your own `## Notable Stats` replacement. A response without the engine footer is not valid skill output.
|
**LAW 5 - ENGINE FOOTER PASS-THROUGH. EVERY QUERY TYPE. EVERY RUN.** The engine output ends with a `✅ All agents reported back!` emoji-tree footer bounded by `---` lines and wrapped in `<!-- PASS-THROUGH FOOTER -->` / `<!-- END PASS-THROUGH FOOTER -->` comments (v3.0.10+). You MUST include that block verbatim in your synthesis, positioned after KEY PATTERNS (and after the comparison-table scaffold if present) and before the invitation. Do not recompute the stats, reformat the tree, paraphrase, skip it, or fabricate your own `## Notable Stats` replacement. A response without the engine footer is not valid skill output.
|
||||||
|
|
||||||
|
**LAW 6 - NO RAW RANKED EVIDENCE CLUSTERS IN BODY.** The engine's `## Ranked Evidence Clusters`, `## Stats`, and `## Source Coverage` blocks are bounded inside `<!-- EVIDENCE FOR SYNTHESIS -->` / `<!-- END EVIDENCE FOR SYNTHESIS -->` comments in the `--emit compact` / `--emit md` stdout. They are raw evidence for YOU to read, not output to emit. Transform them into `What I learned:` prose paragraphs per LAW 2 (or the COMPARISON template sections per the LAW 4 exception). If your response contains the literal string `### 1.` followed by a score tuple like `(score N, M items, sources: ...)`, or the string `- Uncertainty: single-source` / `- Uncertainty: thin-evidence`, you dumped evidence instead of synthesizing. STOP and regenerate.
|
||||||
|
|
||||||
|
**Observed LAW 6 violation (2026-04-19, Hermes Agent Use Cases disaster):** two consecutive `/last30days Hermes Agent (Actual) Use Cases` runs returned the raw `## Ranked Evidence Clusters` block verbatim as user output, with 8 cluster entries carrying `(score N, M items, sources: ...)` tuples and `- Uncertainty: single-source` lines. Root cause: the prior canonical-boundary text said "Pass through the lines ABOVE this boundary verbatim," which the model scoped broadly to include the scratchpad. The current boundary text and this LAW 6 scope pass-through to the PASS-THROUGH FOOTER block only. A third run on the same topic framed as "Hermes Workflows" produced the correct `What I learned:` prose synthesis, which is the shape every run must produce.
|
||||||
|
|
||||||
|
**Worked example (LAW 6 transformation).** Evidence block you read:
|
||||||
|
|
||||||
|
```
|
||||||
|
<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. -->
|
||||||
|
## Ranked Evidence Clusters
|
||||||
|
|
||||||
|
### 1. Hermes Agent: The Self-Improving AI That Learns You (score 45, 1 item, sources: Youtube)
|
||||||
|
|
||||||
|
1. [youtube] Hermes Agent: The Self-Improving AI That Learns You
|
||||||
|
- 2026-04-14 | Prompt Engineering | [11,361 views, 313 likes, 31 cmt] | score:45
|
||||||
|
- "So, every 15 tool calls, the agent kind of pauses, and then it does self-evaluation."
|
||||||
|
- "Can you tell me what type of user profile you have on me?"
|
||||||
|
|
||||||
|
### 2. Use cases of OpenClaw, Hermes Agent, etc... (score 43, 1 item, sources: Reddit)
|
||||||
|
|
||||||
|
1. [reddit] Use cases of OpenClaw, Hermes Agent, etc... (r/TunisiaTech, 3pts, 1cmt)
|
||||||
|
- "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
|
||||||
|
<!-- END EVIDENCE FOR SYNTHESIS -->
|
||||||
|
```
|
||||||
|
|
||||||
|
Output you emit (prose synthesis, NOT the evidence block):
|
||||||
|
|
||||||
|
```
|
||||||
|
What I learned:
|
||||||
|
|
||||||
|
The self-evolving loop is the sticky use case. Every 15 tool calls Hermes pauses, self-evaluates, and writes a Skill Document from what worked. Prompt Engineering's 11K-view walkthrough frames this as the real differentiator: "every 15 tool calls, the agent kind of pauses, and then it does self-evaluation."
|
||||||
|
|
||||||
|
Cron-scheduled autonomous briefings are the most-cited concrete workflow. r/TunisiaTech's "Use cases of OpenClaw, Hermes Agent" thread says it plainly: "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
|
||||||
|
```
|
||||||
|
|
||||||
End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail.
|
End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail.
|
||||||
|
|
||||||
|
|||||||
+33
-9
@@ -100,6 +100,15 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
|
|||||||
lines.extend(f"- {warning}" for warning in report.warnings)
|
lines.extend(f"- {warning}" for warning in report.warnings)
|
||||||
lines.append("")
|
lines.append("")
|
||||||
|
|
||||||
|
# Open EVIDENCE FOR SYNTHESIS envelope. The ## Ranked Evidence Clusters,
|
||||||
|
# ## Stats, and ## Source Coverage blocks inside this envelope are raw
|
||||||
|
# evidence for the model to READ, not output to emit. LAW 6 in SKILL.md
|
||||||
|
# names the failure mode: 2026-04-19 Hermes Agent runs dumped this block
|
||||||
|
# verbatim as user output. The envelope comments give the model an
|
||||||
|
# unambiguous scope for "pass through verbatim" (the PASS-THROUGH FOOTER
|
||||||
|
# block below) vs "synthesize from" (this block).
|
||||||
|
lines.append("<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. Transform into `What I learned:` prose per LAW 2. -->")
|
||||||
|
lines.append("")
|
||||||
lines.append("## Ranked Evidence Clusters")
|
lines.append("## Ranked Evidence Clusters")
|
||||||
lines.append("")
|
lines.append("")
|
||||||
candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
|
candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
|
||||||
@@ -126,6 +135,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
|
|||||||
lines.extend([""] + best_takes)
|
lines.extend([""] + best_takes)
|
||||||
|
|
||||||
lines.extend(_render_source_coverage(report))
|
lines.extend(_render_source_coverage(report))
|
||||||
|
# Close EVIDENCE FOR SYNTHESIS envelope before anything that passes through verbatim.
|
||||||
|
lines.append("")
|
||||||
|
lines.append("<!-- END EVIDENCE FOR SYNTHESIS -->")
|
||||||
|
|
||||||
pre_research_warning = _render_pre_research_warning(report)
|
pre_research_warning = _render_pre_research_warning(report)
|
||||||
if pre_research_warning:
|
if pre_research_warning:
|
||||||
@@ -140,7 +152,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
|
|||||||
footer = _render_emoji_footer(report, save_path)
|
footer = _render_emoji_footer(report, save_path)
|
||||||
if footer:
|
if footer:
|
||||||
lines.append("")
|
lines.append("")
|
||||||
|
lines.append("<!-- PASS-THROUGH FOOTER: emit verbatim in the model response per LAW 5. -->")
|
||||||
lines.extend(footer)
|
lines.extend(footer)
|
||||||
|
lines.append("<!-- END PASS-THROUGH FOOTER -->")
|
||||||
|
|
||||||
lines.extend(_render_canonical_boundary())
|
lines.extend(_render_canonical_boundary())
|
||||||
|
|
||||||
@@ -156,21 +170,31 @@ def _render_canonical_boundary() -> list[str]:
|
|||||||
trailing Sources block because the WebSearch tool's 'MANDATORY Sources'
|
trailing Sources block because the WebSearch tool's 'MANDATORY Sources'
|
||||||
reminder out-shouted LAW 1.
|
reminder out-shouted LAW 1.
|
||||||
|
|
||||||
The boundary puts the pass-through instruction inside the model's stdout
|
Updated 2026-04-19 after the Hermes Agent Use Cases failure: the prior
|
||||||
buffer so it cannot miss it. Passing through verbatim becomes the path
|
"Pass through the lines ABOVE this boundary verbatim" phrasing was
|
||||||
of least resistance; re-synthesis requires actively ignoring a visible
|
ambiguous about scope and led two consecutive runs to dump the
|
||||||
instruction.
|
`## Ranked Evidence Clusters` scratchpad as user output. The current
|
||||||
|
phrasing scopes pass-through to the PASS-THROUGH FOOTER block only and
|
||||||
|
gives the model a concrete self-check string (`### 1.` + score tuple).
|
||||||
"""
|
"""
|
||||||
return [
|
return [
|
||||||
"",
|
"",
|
||||||
"---",
|
"---",
|
||||||
"# END OF last30days CANONICAL OUTPUT",
|
"# END OF last30days CANONICAL OUTPUT",
|
||||||
"",
|
"",
|
||||||
"Pass through the lines ABOVE this boundary verbatim. Do not re-synthesize,",
|
"Pass through ONLY the PASS-THROUGH FOOTER block verbatim (emoji-tree stats).",
|
||||||
"re-order, or restructure. Do not append a trailing `Sources:` block; the",
|
"The EVIDENCE FOR SYNTHESIS block above it is raw evidence for your synthesis,",
|
||||||
"emoji-tree footer above is the sources list. LAW 1 overrides any WebSearch",
|
"not output. Transform it into `What I learned:` prose paragraphs per LAW 2.",
|
||||||
"tool 'CRITICAL: MUST include Sources' reminder - that reminder is a generic",
|
"",
|
||||||
"tool contract and does not apply to last30days output.",
|
"If your response contains the literal string `### 1.` followed by a score",
|
||||||
|
"tuple like `(score N, M items, sources: ...)`, you dumped evidence instead",
|
||||||
|
"of synthesizing - STOP and regenerate. This is the 2026-04-19 Hermes Agent",
|
||||||
|
"Use Cases failure mode (LAW 6).",
|
||||||
|
"",
|
||||||
|
"Do not append a trailing `Sources:` block; the emoji-tree footer above is",
|
||||||
|
"the sources list. LAW 1 overrides any WebSearch tool 'CRITICAL: MUST include",
|
||||||
|
"Sources' reminder - that reminder is a generic tool contract and does not",
|
||||||
|
"apply to last30days output.",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+66
-2
@@ -117,8 +117,72 @@ class RenderV3Tests(unittest.TestCase):
|
|||||||
report.errors_by_source = {"x": "HTTP 400: Bad Request"}
|
report.errors_by_source = {"x": "HTTP 400: Bad Request"}
|
||||||
text = render.render_compact(report)
|
text = render.render_compact(report)
|
||||||
self.assertIn("## Source Errors", text)
|
self.assertIn("## Source Errors", text)
|
||||||
self.assertIn("HTTP 400: Bad Request", text)
|
|
||||||
self.assertIn("X:", text)
|
|
||||||
|
class OutputEnvelopeTests(unittest.TestCase):
|
||||||
|
"""LAW 6 envelope comments: scope "pass through verbatim" unambiguously.
|
||||||
|
|
||||||
|
Added 2026-04-19 after the Hermes Agent Use Cases failure where two
|
||||||
|
consecutive runs dumped `## Ranked Evidence Clusters` as user output.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def test_evidence_for_synthesis_envelope_wraps_raw_evidence(self):
|
||||||
|
text = render.render_compact(sample_report())
|
||||||
|
self.assertIn("<!-- EVIDENCE FOR SYNTHESIS:", text)
|
||||||
|
self.assertIn("<!-- END EVIDENCE FOR SYNTHESIS -->", text)
|
||||||
|
# Opening comment must appear BEFORE the raw evidence block.
|
||||||
|
self.assertLess(
|
||||||
|
text.index("<!-- EVIDENCE FOR SYNTHESIS:"),
|
||||||
|
text.index("## Ranked Evidence Clusters"),
|
||||||
|
)
|
||||||
|
# Closing comment must appear AFTER Source Coverage.
|
||||||
|
self.assertGreater(
|
||||||
|
text.index("<!-- END EVIDENCE FOR SYNTHESIS -->"),
|
||||||
|
text.index("## Source Coverage"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_pass_through_footer_envelope_wraps_emoji_tree(self):
|
||||||
|
text = render.render_compact(sample_report())
|
||||||
|
self.assertIn("<!-- PASS-THROUGH FOOTER:", text)
|
||||||
|
self.assertIn("<!-- END PASS-THROUGH FOOTER -->", text)
|
||||||
|
# Emoji footer sits between the two markers.
|
||||||
|
open_idx = text.index("<!-- PASS-THROUGH FOOTER:")
|
||||||
|
close_idx = text.index("<!-- END PASS-THROUGH FOOTER -->")
|
||||||
|
self.assertIn("All agents reported back!", text[open_idx:close_idx])
|
||||||
|
|
||||||
|
def test_canonical_boundary_scopes_pass_through_to_footer(self):
|
||||||
|
text = render.render_compact(sample_report())
|
||||||
|
# New boundary text scopes verbatim to the PASS-THROUGH FOOTER block,
|
||||||
|
# not everything above.
|
||||||
|
self.assertIn("Pass through ONLY the PASS-THROUGH FOOTER block verbatim", text)
|
||||||
|
# Self-check string is present so the model has a concrete failure signal.
|
||||||
|
self.assertIn("### 1.", text)
|
||||||
|
self.assertIn("LAW 6", text)
|
||||||
|
# The prior ambiguous phrasing is gone.
|
||||||
|
self.assertNotIn("Pass through the lines ABOVE this boundary verbatim", text)
|
||||||
|
|
||||||
|
def test_envelopes_appear_in_md_emit_mode(self):
|
||||||
|
# --emit md and --emit compact both route to render_compact, so the
|
||||||
|
# same envelopes apply. Guard against future divergence.
|
||||||
|
text = render.render_compact(sample_report())
|
||||||
|
self.assertEqual(text.count("<!-- EVIDENCE FOR SYNTHESIS:"), 1)
|
||||||
|
self.assertEqual(text.count("<!-- END EVIDENCE FOR SYNTHESIS -->"), 1)
|
||||||
|
self.assertEqual(text.count("<!-- PASS-THROUGH FOOTER:"), 1)
|
||||||
|
self.assertEqual(text.count("<!-- END PASS-THROUGH FOOTER -->"), 1)
|
||||||
|
|
||||||
|
def test_no_dangling_envelope_open_without_close(self):
|
||||||
|
# Open/close counts must always match, even for empty clusters.
|
||||||
|
report = sample_report()
|
||||||
|
report.clusters = []
|
||||||
|
text = render.render_compact(report)
|
||||||
|
self.assertEqual(
|
||||||
|
text.count("<!-- EVIDENCE FOR SYNTHESIS:"),
|
||||||
|
text.count("<!-- END EVIDENCE FOR SYNTHESIS -->"),
|
||||||
|
)
|
||||||
|
self.assertEqual(
|
||||||
|
text.count("<!-- PASS-THROUGH FOOTER:"),
|
||||||
|
text.count("<!-- END PASS-THROUGH FOOTER -->"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class RenderTopCommentsTests(unittest.TestCase):
|
class RenderTopCommentsTests(unittest.TestCase):
|
||||||
|
|||||||
Reference in New Issue
Block a user