fix: scope pass-through to footer only, add LAW 6 against raw cluster dumps

The engine's ## Ranked Evidence Clusters block is a scratchpad for the
model to read, not user-facing output. Two consecutive /last30days runs
on 2026-04-19 (Hermes Agent Use Cases) dumped it verbatim as user output
because the prior canonical-boundary text (Pass through the lines ABOVE
this boundary verbatim) was ambiguous about scope.

Split render_compact stdout into two bounded blocks:

- <!-- EVIDENCE FOR SYNTHESIS: ... --> wraps Ranked Evidence Clusters,
  Stats, and Source Coverage. Transform into prose per LAW 2.
- <!-- PASS-THROUGH FOOTER: ... --> wraps the emoji-tree footer only.
  Emit verbatim per LAW 5.

Rewrite _render_canonical_boundary to scope pass-through to the footer
block explicitly and give the model a concrete self-check string
(### 1. followed by a score tuple) as the named LAW 6 failure signal.

Add LAW 6 to SKILL.md OUTPUT CONTRACT with the observed violation
(2026-04-19 Hermes Agent Use Cases) and a worked transformation example.
This commit is contained in:
Matt Van Horn
2026-04-19 09:23:55 -07:00
parent f635f78e4a
commit 52fb0e50cb
3 changed files with 134 additions and 12 deletions
+35 -1
View File
@@ -120,7 +120,41 @@ These five rules dominate every other rule in this file. If you find yourself ab
**Observed LAW 4 violation (2026-04-18, Peter Steinberger disaster #2):** the model emitted `Headline`, `What he is actually saying`, `Cross-source corroboration`, `Where evidence is thin`, `Bottom line` on a GENERAL query. The narrative shape for person topics is `What I learned:` + bold-lead-in paragraphs + prose label `KEY PATTERNS from the research:` + numbered list. No blog-post subheadings. **Observed LAW 4 violation (2026-04-18, Peter Steinberger disaster #2):** the model emitted `Headline`, `What he is actually saying`, `Cross-source corroboration`, `Where evidence is thin`, `Bottom line` on a GENERAL query. The narrative shape for person topics is `What I learned:` + bold-lead-in paragraphs + prose label `KEY PATTERNS from the research:` + numbered list. No blog-post subheadings.
**LAW 5 - ENGINE FOOTER PASS-THROUGH. EVERY QUERY TYPE. EVERY RUN.** The engine output ends with a `✅ All agents reported back!` emoji-tree footer bounded by `---` lines. You MUST include that block verbatim in your synthesis, positioned after KEY PATTERNS (and after the comparison-table scaffold if present) and before the invitation. Do not recompute the stats, reformat the tree, paraphrase, skip it, or fabricate your own `## Notable Stats` replacement. A response without the engine footer is not valid skill output. **LAW 5 - ENGINE FOOTER PASS-THROUGH. EVERY QUERY TYPE. EVERY RUN.** The engine output ends with a `✅ All agents reported back!` emoji-tree footer bounded by `---` lines and wrapped in `<!-- PASS-THROUGH FOOTER -->` / `<!-- END PASS-THROUGH FOOTER -->` comments (v3.0.10+). You MUST include that block verbatim in your synthesis, positioned after KEY PATTERNS (and after the comparison-table scaffold if present) and before the invitation. Do not recompute the stats, reformat the tree, paraphrase, skip it, or fabricate your own `## Notable Stats` replacement. A response without the engine footer is not valid skill output.
**LAW 6 - NO RAW RANKED EVIDENCE CLUSTERS IN BODY.** The engine's `## Ranked Evidence Clusters`, `## Stats`, and `## Source Coverage` blocks are bounded inside `<!-- EVIDENCE FOR SYNTHESIS -->` / `<!-- END EVIDENCE FOR SYNTHESIS -->` comments in the `--emit compact` / `--emit md` stdout. They are raw evidence for YOU to read, not output to emit. Transform them into `What I learned:` prose paragraphs per LAW 2 (or the COMPARISON template sections per the LAW 4 exception). If your response contains the literal string `### 1.` followed by a score tuple like `(score N, M items, sources: ...)`, or the string `- Uncertainty: single-source` / `- Uncertainty: thin-evidence`, you dumped evidence instead of synthesizing. STOP and regenerate.
**Observed LAW 6 violation (2026-04-19, Hermes Agent Use Cases disaster):** two consecutive `/last30days Hermes Agent (Actual) Use Cases` runs returned the raw `## Ranked Evidence Clusters` block verbatim as user output, with 8 cluster entries carrying `(score N, M items, sources: ...)` tuples and `- Uncertainty: single-source` lines. Root cause: the prior canonical-boundary text said "Pass through the lines ABOVE this boundary verbatim," which the model scoped broadly to include the scratchpad. The current boundary text and this LAW 6 scope pass-through to the PASS-THROUGH FOOTER block only. A third run on the same topic framed as "Hermes Workflows" produced the correct `What I learned:` prose synthesis, which is the shape every run must produce.
**Worked example (LAW 6 transformation).** Evidence block you read:
```
<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. -->
## Ranked Evidence Clusters
### 1. Hermes Agent: The Self-Improving AI That Learns You (score 45, 1 item, sources: Youtube)
1. [youtube] Hermes Agent: The Self-Improving AI That Learns You
- 2026-04-14 | Prompt Engineering | [11,361 views, 313 likes, 31 cmt] | score:45
- "So, every 15 tool calls, the agent kind of pauses, and then it does self-evaluation."
- "Can you tell me what type of user profile you have on me?"
### 2. Use cases of OpenClaw, Hermes Agent, etc... (score 43, 1 item, sources: Reddit)
1. [reddit] Use cases of OpenClaw, Hermes Agent, etc... (r/TunisiaTech, 3pts, 1cmt)
- "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
<!-- END EVIDENCE FOR SYNTHESIS -->
```
Output you emit (prose synthesis, NOT the evidence block):
```
What I learned:
The self-evolving loop is the sticky use case. Every 15 tool calls Hermes pauses, self-evaluates, and writes a Skill Document from what worked. Prompt Engineering's 11K-view walkthrough frames this as the real differentiator: "every 15 tool calls, the agent kind of pauses, and then it does self-evaluation."
Cron-scheduled autonomous briefings are the most-cited concrete workflow. r/TunisiaTech's "Use cases of OpenClaw, Hermes Agent" thread says it plainly: "Currently I have daily cron jobs for news briefing, but I know there's much more I can do."
```
End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail. End of OUTPUT CONTRACT. The laws above are the contract; everything below is implementation detail.
+33 -9
View File
@@ -100,6 +100,15 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
lines.extend(f"- {warning}" for warning in report.warnings) lines.extend(f"- {warning}" for warning in report.warnings)
lines.append("") lines.append("")
# Open EVIDENCE FOR SYNTHESIS envelope. The ## Ranked Evidence Clusters,
# ## Stats, and ## Source Coverage blocks inside this envelope are raw
# evidence for the model to READ, not output to emit. LAW 6 in SKILL.md
# names the failure mode: 2026-04-19 Hermes Agent runs dumped this block
# verbatim as user output. The envelope comments give the model an
# unambiguous scope for "pass through verbatim" (the PASS-THROUGH FOOTER
# block below) vs "synthesize from" (this block).
lines.append("<!-- EVIDENCE FOR SYNTHESIS: read this, do not emit verbatim. Transform into `What I learned:` prose per LAW 2. -->")
lines.append("")
lines.append("## Ranked Evidence Clusters") lines.append("## Ranked Evidence Clusters")
lines.append("") lines.append("")
candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates} candidate_by_id = {candidate.candidate_id: candidate for candidate in report.ranked_candidates}
@@ -126,6 +135,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
lines.extend([""] + best_takes) lines.extend([""] + best_takes)
lines.extend(_render_source_coverage(report)) lines.extend(_render_source_coverage(report))
# Close EVIDENCE FOR SYNTHESIS envelope before anything that passes through verbatim.
lines.append("")
lines.append("<!-- END EVIDENCE FOR SYNTHESIS -->")
pre_research_warning = _render_pre_research_warning(report) pre_research_warning = _render_pre_research_warning(report)
if pre_research_warning: if pre_research_warning:
@@ -140,7 +152,9 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
footer = _render_emoji_footer(report, save_path) footer = _render_emoji_footer(report, save_path)
if footer: if footer:
lines.append("") lines.append("")
lines.append("<!-- PASS-THROUGH FOOTER: emit verbatim in the model response per LAW 5. -->")
lines.extend(footer) lines.extend(footer)
lines.append("<!-- END PASS-THROUGH FOOTER -->")
lines.extend(_render_canonical_boundary()) lines.extend(_render_canonical_boundary())
@@ -156,21 +170,31 @@ def _render_canonical_boundary() -> list[str]:
trailing Sources block because the WebSearch tool's 'MANDATORY Sources' trailing Sources block because the WebSearch tool's 'MANDATORY Sources'
reminder out-shouted LAW 1. reminder out-shouted LAW 1.
The boundary puts the pass-through instruction inside the model's stdout Updated 2026-04-19 after the Hermes Agent Use Cases failure: the prior
buffer so it cannot miss it. Passing through verbatim becomes the path "Pass through the lines ABOVE this boundary verbatim" phrasing was
of least resistance; re-synthesis requires actively ignoring a visible ambiguous about scope and led two consecutive runs to dump the
instruction. `## Ranked Evidence Clusters` scratchpad as user output. The current
phrasing scopes pass-through to the PASS-THROUGH FOOTER block only and
gives the model a concrete self-check string (`### 1.` + score tuple).
""" """
return [ return [
"", "",
"---", "---",
"# END OF last30days CANONICAL OUTPUT", "# END OF last30days CANONICAL OUTPUT",
"", "",
"Pass through the lines ABOVE this boundary verbatim. Do not re-synthesize,", "Pass through ONLY the PASS-THROUGH FOOTER block verbatim (emoji-tree stats).",
"re-order, or restructure. Do not append a trailing `Sources:` block; the", "The EVIDENCE FOR SYNTHESIS block above it is raw evidence for your synthesis,",
"emoji-tree footer above is the sources list. LAW 1 overrides any WebSearch", "not output. Transform it into `What I learned:` prose paragraphs per LAW 2.",
"tool 'CRITICAL: MUST include Sources' reminder - that reminder is a generic", "",
"tool contract and does not apply to last30days output.", "If your response contains the literal string `### 1.` followed by a score",
"tuple like `(score N, M items, sources: ...)`, you dumped evidence instead",
"of synthesizing - STOP and regenerate. This is the 2026-04-19 Hermes Agent",
"Use Cases failure mode (LAW 6).",
"",
"Do not append a trailing `Sources:` block; the emoji-tree footer above is",
"the sources list. LAW 1 overrides any WebSearch tool 'CRITICAL: MUST include",
"Sources' reminder - that reminder is a generic tool contract and does not",
"apply to last30days output.",
] ]
+66 -2
View File
@@ -117,8 +117,72 @@ class RenderV3Tests(unittest.TestCase):
report.errors_by_source = {"x": "HTTP 400: Bad Request"} report.errors_by_source = {"x": "HTTP 400: Bad Request"}
text = render.render_compact(report) text = render.render_compact(report)
self.assertIn("## Source Errors", text) self.assertIn("## Source Errors", text)
self.assertIn("HTTP 400: Bad Request", text)
self.assertIn("X:", text)
class OutputEnvelopeTests(unittest.TestCase):
"""LAW 6 envelope comments: scope "pass through verbatim" unambiguously.
Added 2026-04-19 after the Hermes Agent Use Cases failure where two
consecutive runs dumped `## Ranked Evidence Clusters` as user output.
"""
def test_evidence_for_synthesis_envelope_wraps_raw_evidence(self):
text = render.render_compact(sample_report())
self.assertIn("<!-- EVIDENCE FOR SYNTHESIS:", text)
self.assertIn("<!-- END EVIDENCE FOR SYNTHESIS -->", text)
# Opening comment must appear BEFORE the raw evidence block.
self.assertLess(
text.index("<!-- EVIDENCE FOR SYNTHESIS:"),
text.index("## Ranked Evidence Clusters"),
)
# Closing comment must appear AFTER Source Coverage.
self.assertGreater(
text.index("<!-- END EVIDENCE FOR SYNTHESIS -->"),
text.index("## Source Coverage"),
)
def test_pass_through_footer_envelope_wraps_emoji_tree(self):
text = render.render_compact(sample_report())
self.assertIn("<!-- PASS-THROUGH FOOTER:", text)
self.assertIn("<!-- END PASS-THROUGH FOOTER -->", text)
# Emoji footer sits between the two markers.
open_idx = text.index("<!-- PASS-THROUGH FOOTER:")
close_idx = text.index("<!-- END PASS-THROUGH FOOTER -->")
self.assertIn("All agents reported back!", text[open_idx:close_idx])
def test_canonical_boundary_scopes_pass_through_to_footer(self):
text = render.render_compact(sample_report())
# New boundary text scopes verbatim to the PASS-THROUGH FOOTER block,
# not everything above.
self.assertIn("Pass through ONLY the PASS-THROUGH FOOTER block verbatim", text)
# Self-check string is present so the model has a concrete failure signal.
self.assertIn("### 1.", text)
self.assertIn("LAW 6", text)
# The prior ambiguous phrasing is gone.
self.assertNotIn("Pass through the lines ABOVE this boundary verbatim", text)
def test_envelopes_appear_in_md_emit_mode(self):
# --emit md and --emit compact both route to render_compact, so the
# same envelopes apply. Guard against future divergence.
text = render.render_compact(sample_report())
self.assertEqual(text.count("<!-- EVIDENCE FOR SYNTHESIS:"), 1)
self.assertEqual(text.count("<!-- END EVIDENCE FOR SYNTHESIS -->"), 1)
self.assertEqual(text.count("<!-- PASS-THROUGH FOOTER:"), 1)
self.assertEqual(text.count("<!-- END PASS-THROUGH FOOTER -->"), 1)
def test_no_dangling_envelope_open_without_close(self):
# Open/close counts must always match, even for empty clusters.
report = sample_report()
report.clusters = []
text = render.render_compact(report)
self.assertEqual(
text.count("<!-- EVIDENCE FOR SYNTHESIS:"),
text.count("<!-- END EVIDENCE FOR SYNTHESIS -->"),
)
self.assertEqual(
text.count("<!-- PASS-THROUGH FOOTER:"),
text.count("<!-- END PASS-THROUGH FOOTER -->"),
)
class RenderTopCommentsTests(unittest.TestCase): class RenderTopCommentsTests(unittest.TestCase):