Files
last30days-skill/tests/test_render_v3.py
Matt Van Horn 6acfc9017e feat: move stats footer into Python engine, harden synthesis voice contract
DO NOT MERGE until local validation passes on 3+ golden topics.

Problem: the magic footer ( All agents reported back!, emoji tree,
Top voices, Raw results saved) was composed by the synthesizer model
following a "Copy this EXACTLY" template buried 1150 lines into SKILL.md.
Under context pressure, Opus 4.7 dropped it. Three recent /last30days
runs (Opus 4.7, programming language for AI agents, Kanye West)
produced clean prose with no footer and used AI-slop section headers
(## The launch, ## Where it disappoints) instead of flowing paragraphs.

Fix:

1. render.py: new _render_emoji_footer() emits the deterministic footer
   as the final block of every compact output. Zero-count sources are
   omitted. Tree characters (├─ / └─) computed from populated-line
   count. The model no longer assembles the tree from text instructions.

2. render.py: new _site_name_for_url() and _format_web_line_sources()
   map URLs to clean publication names (Later, Buffer, CNN, etc.) so
   the 🌐 Web line is pre-assembled by Python.

3. last30days.py: compute_save_path_display() turns the save path into
   a ~/-relative string that the engine puts in the footer. Signature
   change: emit_output() and render_compact() both accept save_path.

4. SKILL.md synthesis contract rewritten:
   - Footer template DELETED. Replaced with instruction to include the
     engine footer block verbatim.
   - URL-to-site-name sub-block DELETED. Engine does this.
   - "Calculate actual totals" paragraph DELETED. Engine does this.
   - All em-dashes in the synthesis section replaced with ` - ` (single
     hyphen with spaces). Em-dashes are the most reliable AI-slop tell.
   - New rules: no ## markdown section headers in response body, no
     invented title line like "{Topic}: last 30 days", no bold section
     labels acting as headers. Bold-lead-in paragraph shape stays.
   - SELF-CHECK updated to verify footer presence, no em-dashes, no
     body-level headers.

Tests: 15 new tests covering footer emission, zero-source omission,
tree character placement, save-path threading, URL-to-name helper,
Web line formatting, Top voices combination, Polymarket line.

All 127 tests pass across render, rerank, cluster, briefing, CLI,
internals, fun-scoring.

Plan: docs/plans/2026-04-17-003-feat-deterministic-footer-plan.md

Local validation protocol (blocks merge):
- Run /last30days in a fresh Claude Code window on 5 golden topics
- Verify each output contains the footer block verbatim
- Verify zero ## body headers, zero em-dashes/en-dashes, zero invented
  title lines
- Report 5x8 pass/fail matrix; all 40 cells must be green before merge

🤖 Generated with Claude Opus 4.7 (1M context) via [Claude Code](https://claude.com/claude-code) + Compound Engineering v2.63.1

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 09:15:55 -04:00

539 lines
23 KiB
Python

import sys
import unittest
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
from lib import render, schema
def sample_report() -> schema.Report:
primary_item = schema.SourceItem(
item_id="i1",
source="grounding",
title="Grounded result",
body="A grounded body with useful detail.",
url="https://example.com",
container="example.com",
published_at="2026-03-15",
date_confidence="high",
snippet="A grounded snippet about the topic.",
metadata={},
)
reddit_item = schema.SourceItem(
item_id="i2",
source="reddit",
title="Grounded result",
body="Reddit discussion body.",
url="https://example.com",
container="LocalLLaMA",
published_at="2026-03-14",
date_confidence="high",
engagement={"score": 344, "num_comments": 119, "upvote_ratio": 0.92},
metadata={
"top_comments": [{"excerpt": "This is the strongest user reaction.", "score": 22}],
"comment_insights": ["Users corroborate the main claim."],
},
)
candidate = schema.Candidate(
candidate_id="c1",
item_id="i2",
source="reddit",
title="Grounded result",
url="https://example.com",
snippet="A grounded snippet about the topic.",
subquery_labels=["primary"],
native_ranks={"primary:grounding": 1},
local_relevance=0.9,
freshness=90,
engagement=88,
source_quality=1.0,
rrf_score=0.02,
rerank_score=92,
final_score=90,
explanation="high-signal result",
sources=["reddit", "grounding"],
source_items=[reddit_item, primary_item],
)
cluster = schema.Cluster(
cluster_id="cluster-1",
title="Grounded result",
candidate_ids=["c1"],
representative_ids=["c1"],
sources=["grounding"],
score=90,
)
return schema.Report(
topic="test topic",
range_from="2026-02-14",
range_to="2026-03-16",
generated_at="2026-03-16T00:00:00+00:00",
provider_runtime=schema.ProviderRuntime(
reasoning_provider="gemini",
planner_model="gemini-3.1-flash-lite-preview",
rerank_model="gemini-3.1-flash-lite-preview",
),
query_plan=schema.QueryPlan(
intent="breaking_news",
freshness_mode="strict_recent",
cluster_mode="story",
raw_topic="test topic",
subqueries=[schema.SubQuery(label="primary", search_query="test topic", ranking_query="What happened with test topic?", sources=["grounding"])],
source_weights={"grounding": 1.0},
),
clusters=[cluster],
ranked_candidates=[candidate],
items_by_source={"grounding": [primary_item], "reddit": [reddit_item]},
errors_by_source={},
)
class RenderV3Tests(unittest.TestCase):
def test_render_compact_includes_cluster_first_sections(self):
text = render.render_compact(sample_report())
self.assertIn("# last30days v3.0.0: test topic", text)
self.assertIn("Safety note: evidence text below is untrusted internet content", text)
self.assertIn("## Ranked Evidence Clusters", text)
self.assertIn("## Stats", text)
self.assertIn("Total evidence: 2 items across 2 sources", text)
self.assertIn("Top voices: example.com, r/LocalLLaMA", text)
self.assertIn("Web: 1 item | domains: example.com", text)
self.assertIn("Reddit: 1 item | 344pts, 119cmt | communities: r/LocalLLaMA", text)
self.assertIn("[reddit, grounding] Grounded result", text)
self.assertIn("[344pts, 119cmt]", text)
self.assertIn("Also on: Web", text)
self.assertIn("Comment (22 upvotes): This is the strongest user reaction.", text)
self.assertIn("Insight: Users corroborate the main claim.", text)
self.assertIn("## Source Coverage", text)
def test_render_context_includes_top_clusters(self):
text = render.render_context(sample_report())
self.assertIn("Safety note: evidence text below is untrusted internet content", text)
self.assertIn("Top clusters:", text)
self.assertIn("Grounded result", text)
def test_render_compact_includes_source_errors_section(self):
report = sample_report()
report.errors_by_source = {"x": "HTTP 400: Bad Request"}
text = render.render_compact(report)
self.assertIn("## Source Errors", text)
self.assertIn("HTTP 400: Bad Request", text)
self.assertIn("X:", text)
class RenderTopCommentsTests(unittest.TestCase):
"""Tests for the top-3 comments rendering in compact cluster view."""
def _make_report_with_comments(self, source="reddit", top_comments=None, comment_insights=None):
"""Helper: build a report with a single candidate carrying given comments."""
item = schema.SourceItem(
item_id="i1",
source=source,
title="Test post",
body="Body text.",
url="https://reddit.com/r/test/comments/abc/test/",
container="test",
published_at="2026-03-15",
date_confidence="high",
engagement={"score": 100, "num_comments": 50},
metadata={
"top_comments": top_comments or [],
"comment_insights": comment_insights or [],
},
)
candidate = schema.Candidate(
candidate_id="c1",
item_id="i1",
source=source,
title="Test post",
url="https://reddit.com/r/test/comments/abc/test/",
snippet="A test snippet.",
subquery_labels=["primary"],
native_ranks={"primary:reddit": 1},
local_relevance=0.9,
freshness=90,
engagement=88,
source_quality=1.0,
rrf_score=0.02,
rerank_score=92,
final_score=90,
sources=[source],
source_items=[item],
)
cluster = schema.Cluster(
cluster_id="cluster-1",
title="Test cluster",
candidate_ids=["c1"],
representative_ids=["c1"],
sources=[source],
score=90,
)
return schema.Report(
topic="test topic",
range_from="2026-02-14",
range_to="2026-03-16",
generated_at="2026-03-16T00:00:00+00:00",
provider_runtime=schema.ProviderRuntime(
reasoning_provider="gemini",
planner_model="gemini-3.1-flash-lite-preview",
rerank_model="gemini-3.1-flash-lite-preview",
),
query_plan=schema.QueryPlan(
intent="breaking_news",
freshness_mode="strict_recent",
cluster_mode="story",
raw_topic="test topic",
subqueries=[schema.SubQuery(label="primary", search_query="test", ranking_query="test?", sources=[source])],
source_weights={source: 1.0},
),
clusters=[cluster],
ranked_candidates=[candidate],
items_by_source={source: [item]},
errors_by_source={},
)
def test_reddit_5_comments_renders_top_3(self):
"""Reddit candidate with 5 comments (scores 500, 200, 50, 8, 3) renders 3."""
comments = [
{"score": 500, "excerpt": "Comment with 500 upvotes", "author": "user1"},
{"score": 200, "excerpt": "Comment with 200 upvotes", "author": "user2"},
{"score": 50, "excerpt": "Comment with 50 upvotes", "author": "user3"},
{"score": 8, "excerpt": "Comment with 8 upvotes", "author": "user4"},
{"score": 3, "excerpt": "Comment with 3 upvotes", "author": "user5"},
]
report = self._make_report_with_comments(top_comments=comments)
text = render.render_compact(report)
self.assertIn("Comment (500 upvotes):", text)
self.assertIn("Comment (200 upvotes):", text)
self.assertIn("Comment (50 upvotes):", text)
self.assertNotIn("Comment (8 upvotes):", text)
self.assertNotIn("Comment (3 upvotes):", text)
def test_reddit_1_comment_renders_1(self):
"""Reddit candidate with 1 comment renders 1."""
comments = [{"score": 100, "excerpt": "Single comment", "author": "user1"}]
report = self._make_report_with_comments(top_comments=comments)
text = render.render_compact(report)
self.assertIn("Comment (100 upvotes): Single comment", text)
def test_reddit_0_comments_no_section(self):
"""Reddit candidate with 0 comments renders no comment section."""
report = self._make_report_with_comments(top_comments=[])
text = render.render_compact(report)
self.assertNotIn("Comment (", text)
self.assertNotIn("upvotes)", text)
def test_non_reddit_no_comments(self):
"""Non-Reddit candidate doesn't render comments when metadata has none."""
report = self._make_report_with_comments(source="grounding", top_comments=[])
text = render.render_compact(report)
self.assertNotIn("Comment (", text)
self.assertIn("Test cluster", text)
def test_all_comments_below_score_10_no_section(self):
"""All comments below score 10 renders no comment section."""
comments = [
{"score": 9, "excerpt": "Low score 1", "author": "user1"},
{"score": 5, "excerpt": "Low score 2", "author": "user2"},
{"score": 1, "excerpt": "Low score 3", "author": "user3"},
]
report = self._make_report_with_comments(top_comments=comments)
text = render.render_compact(report)
self.assertNotIn("Comment (", text)
self.assertNotIn("upvotes)", text)
def test_youtube_comments_use_likes_label_and_50_threshold(self):
comments = [
{"score": 120, "excerpt": "legit fire tutorial", "author": "alice"},
{"score": 60, "excerpt": "saved me hours", "author": "bob"},
{"score": 10, "excerpt": "below threshold", "author": "carol"},
]
report = self._make_report_with_comments(source="youtube", top_comments=comments)
text = render.render_compact(report)
self.assertIn("Comment (120 likes): legit fire tutorial", text)
self.assertIn("Comment (60 likes): saved me hours", text)
self.assertNotIn("Comment (10 likes)", text)
# Render must not silently label YT as upvotes.
self.assertNotIn("Comment (120 upvotes)", text)
def test_tiktok_comments_use_likes_label_and_500_threshold(self):
comments = [
{"score": 2000, "excerpt": "this aged well", "author": "a"},
{"score": 600, "excerpt": "so real", "author": "b"},
{"score": 400, "excerpt": "below tt threshold", "author": "c"},
{"score": 50, "excerpt": "way below", "author": "d"},
]
report = self._make_report_with_comments(source="tiktok", top_comments=comments)
text = render.render_compact(report)
self.assertIn("Comment (2000 likes): this aged well", text)
self.assertIn("Comment (600 likes): so real", text)
self.assertNotIn("Comment (400 likes)", text)
self.assertNotIn("Comment (50 likes)", text)
class RenderBestTakesCompactTests(unittest.TestCase):
"""Tests for Best Takes section in compact output and fun tags on candidates."""
def _make_candidate(self, cid, fun_score=None, fun_explanation=None, final_score=80):
"""Helper: build a candidate with a given fun_score."""
item = schema.SourceItem(
item_id=f"item-{cid}",
source="reddit",
title=f"Post {cid}",
body="Body text.",
url=f"https://reddit.com/r/test/comments/{cid}/",
container="test",
published_at="2026-03-15",
date_confidence="high",
engagement={"score": 200, "num_comments": 30},
metadata={
"top_comments": [{"excerpt": "Funny comment", "score": 50, "body": "lmao this is gold"}],
},
)
return schema.Candidate(
candidate_id=cid,
item_id=f"item-{cid}",
source="reddit",
title=f"Post {cid}",
url=f"https://reddit.com/r/test/comments/{cid}/",
snippet="A test snippet.",
subquery_labels=["primary"],
native_ranks={"primary:reddit": 1},
local_relevance=0.9,
freshness=90,
engagement=88,
source_quality=1.0,
rrf_score=0.02,
rerank_score=92,
final_score=final_score,
sources=["reddit"],
source_items=[item],
fun_score=fun_score,
fun_explanation=fun_explanation,
)
def _make_report_with_candidates(self, candidates):
"""Helper: build a report with given candidates."""
items = []
for c in candidates:
items.extend(c.source_items)
cluster = schema.Cluster(
cluster_id="cluster-1",
title="Test cluster",
candidate_ids=[c.candidate_id for c in candidates],
representative_ids=[c.candidate_id for c in candidates],
sources=["reddit"],
score=90,
)
return schema.Report(
topic="test topic",
range_from="2026-02-14",
range_to="2026-03-16",
generated_at="2026-03-16T00:00:00+00:00",
provider_runtime=schema.ProviderRuntime(
reasoning_provider="gemini",
planner_model="gemini-3.1-flash-lite-preview",
rerank_model="gemini-3.1-flash-lite-preview",
),
query_plan=schema.QueryPlan(
intent="breaking_news",
freshness_mode="strict_recent",
cluster_mode="story",
raw_topic="test topic",
subqueries=[schema.SubQuery(label="primary", search_query="test", ranking_query="test?", sources=["reddit"])],
source_weights={"reddit": 1.0},
),
clusters=[cluster],
ranked_candidates=candidates,
items_by_source={"reddit": items},
errors_by_source={},
)
def test_compact_includes_best_takes_with_2_high_fun_candidates(self):
"""Compact output includes Best Takes section when 2+ candidates score >= 70."""
candidates = [
self._make_candidate("c1", fun_score=85, fun_explanation="hilarious comment"),
self._make_candidate("c2", fun_score=75, fun_explanation="witty remark"),
self._make_candidate("c3", fun_score=40),
]
report = self._make_report_with_candidates(candidates)
text = render.render_compact(report)
self.assertIn("## Best Takes", text)
self.assertIn("(fun:85)", text)
self.assertIn("(fun:75)", text)
def test_candidate_with_fun_score_85_shows_fun_tag(self):
"""Candidate with fun_score=85 shows 'fun:85' in its detail line."""
candidates = [self._make_candidate("c1", fun_score=85)]
report = self._make_report_with_candidates(candidates)
text = render.render_compact(report)
self.assertIn("fun:85", text)
def test_candidate_with_fun_score_40_no_fun_tag(self):
"""Candidate with fun_score=40 does NOT show fun tag (below 50 threshold)."""
candidates = [self._make_candidate("c1", fun_score=40)]
report = self._make_report_with_candidates(candidates)
text = render.render_compact(report)
self.assertNotIn("fun:40", text)
self.assertNotIn("fun:", text)
def test_no_best_takes_with_0_high_fun_candidates(self):
"""No Best Takes section when 0 candidates above threshold."""
candidates = [
self._make_candidate("c1", fun_score=50),
self._make_candidate("c2", fun_score=40),
]
report = self._make_report_with_candidates(candidates)
text = render.render_compact(report)
self.assertNotIn("## Best Takes", text)
def test_no_best_takes_with_1_high_fun_candidate(self):
"""No Best Takes section when only 1 candidate above threshold."""
candidates = [
self._make_candidate("c1", fun_score=80),
self._make_candidate("c2", fun_score=50),
]
report = self._make_report_with_candidates(candidates)
text = render.render_compact(report)
self.assertNotIn("## Best Takes", text)
class EmojiFooterTests(unittest.TestCase):
"""Deterministic magic footer emitted by the Python engine."""
def _make_report(self, items_by_source):
return schema.Report(
topic="test topic",
range_from="2026-03-18",
range_to="2026-04-17",
generated_at="2026-04-17T00:00:00+00:00",
provider_runtime=schema.ProviderRuntime(reasoning_provider="n/a", planner_model="n/a", rerank_model="n/a"),
query_plan=schema.QueryPlan(
intent="news", freshness_mode="strict_recent", cluster_mode="story", raw_topic="test topic",
subqueries=[schema.SubQuery(label="p", search_query="x", ranking_query="x", sources=["reddit"])],
source_weights={"reddit": 1.0},
),
clusters=[], ranked_candidates=[],
items_by_source=items_by_source, errors_by_source={},
)
def _reddit(self, item_id="r1", score=300, comments=50, sub="test"):
return schema.SourceItem(
item_id=item_id, source="reddit", title="t", body="",
url=f"https://reddit.com/r/{sub}/{item_id}", container=sub,
engagement={"score": score, "num_comments": comments},
)
def _x(self, item_id="x1", author="user", likes=100, reposts=10):
return schema.SourceItem(
item_id=item_id, source="x", title="t", body="",
url=f"https://x.com/{author}/status/{item_id}", author=author,
engagement={"likes": likes, "reposts": reposts},
)
def _web(self, url, item_id=None):
return schema.SourceItem(
item_id=item_id or f"g-{url[:8]}", source="grounding", title="t", body="",
url=url, container=url.split("//")[-1].split("/")[0],
)
def test_footer_present_with_reddit_and_x(self):
report = self._make_report({"reddit": [self._reddit()], "x": [self._x()]})
out = render.render_compact(report, save_path="~/Documents/Last30Days/test-raw.md")
self.assertIn("✅ All agents reported back!", out)
self.assertIn("├─ 🟠 Reddit: 1 thread │ 300 upvotes │ 50 comments", out)
self.assertIn("🔵 X: 1 post │ 100 likes │ 10 reposts", out)
self.assertIn("└─ 📎 Raw results saved to ~/Documents/Last30Days/test-raw.md", out)
def test_footer_omits_zero_count_sources(self):
report = self._make_report({"reddit": [self._reddit()]})
out = render.render_compact(report, save_path="~/foo.md")
self.assertNotIn("YouTube:", out)
self.assertNotIn("TikTok:", out)
self.assertNotIn("Instagram:", out)
self.assertIn("🟠 Reddit:", out)
def test_footer_tree_ends_with_last_line(self):
report = self._make_report({"reddit": [self._reddit()]})
out = render.render_compact(report, save_path="~/foo.md")
self.assertIn("└─ 📎 Raw results saved", out)
for line in out.splitlines():
if "Raw results saved" in line:
self.assertTrue(line.startswith("└─"), f"Raw results line should start with └─, got: {line}")
def test_footer_absent_when_no_save_path(self):
report = self._make_report({"reddit": [self._reddit()]})
out = render.render_compact(report)
self.assertIn("🟠 Reddit:", out)
self.assertNotIn("Raw results saved", out)
def test_footer_absent_when_all_sources_empty(self):
report = self._make_report({})
out = render.render_compact(report, save_path="~/foo.md")
self.assertNotIn("✅ All agents reported back!", out)
def test_web_line_uses_clean_publication_names(self):
report = self._make_report({
"grounding": [
self._web("https://later.com/blog/x"),
self._web("https://buffer.com/resources/y"),
self._web("https://unknown.weirdsite.xyz/z"),
],
})
out = render.render_compact(report, save_path="~/foo.md")
self.assertIn("🌐 Web: 3 pages - Later, Buffer, unknown.weirdsite.xyz", out)
def test_top_voices_combines_handles_and_subreddits(self):
report = self._make_report({
"reddit": [self._reddit(sub="Anthropic"), self._reddit(item_id="r2", sub="ClaudeAI")],
"x": [self._x(author="boris_cherny"), self._x(item_id="x2", author="alexalbert__")],
})
out = render.render_compact(report, save_path="~/foo.md")
self.assertIn("🗣️ Top voices:", out)
for line in out.splitlines():
if "Top voices:" in line:
self.assertIn("@boris_cherny", line)
self.assertIn("r/", line)
def test_footer_renders_after_source_coverage(self):
report = self._make_report({"reddit": [self._reddit()]})
out = render.render_compact(report, save_path="~/foo.md")
source_coverage_pos = out.find("## Source Coverage")
footer_pos = out.find("✅ All agents reported back!")
self.assertLess(source_coverage_pos, footer_pos)
class SiteNameHelperTests(unittest.TestCase):
"""URL to publication name helper used by the Web footer line."""
def test_known_publication_returns_clean_name(self):
self.assertEqual(render._site_name_for_url("https://later.com/blog/x"), "Later")
self.assertEqual(render._site_name_for_url("https://www.cnn.com/2026/x"), "CNN")
self.assertEqual(render._site_name_for_url("https://buffer.com/y"), "Buffer")
def test_unknown_publication_falls_back_to_full_host(self):
self.assertEqual(render._site_name_for_url("https://unknown.xyz/abc"), "unknown.xyz")
self.assertEqual(render._site_name_for_url("https://sub.unknown.xyz/abc"), "sub.unknown.xyz")
def test_subdomain_stripped_when_apex_is_known(self):
self.assertEqual(render._site_name_for_url("https://eu.bloomberg.com/x"), "Bloomberg")
def test_empty_url_returns_empty(self):
self.assertEqual(render._site_name_for_url(""), "")
def test_url_without_scheme(self):
self.assertEqual(render._site_name_for_url("later.com/x"), "Later")
def test_format_web_line_dedupes(self):
items = [
schema.SourceItem(item_id="1", source="grounding", title="t", body="", url="https://later.com/a"),
schema.SourceItem(item_id="2", source="grounding", title="t", body="", url="https://later.com/b"),
schema.SourceItem(item_id="3", source="grounding", title="t", body="", url="https://buffer.com/c"),
]
result = render._format_web_line_sources(items)
self.assertEqual(result, "Later, Buffer")
if __name__ == "__main__":
unittest.main()