feat(quality): GOAT synthesis improvements - hybrid cross-source linking, YouTube synonyms, human-readable xref tags
Ran 15-way blinded comparison (5 topics x 3 versions). CROSS won all 5 topics (4.74/5.0 avg vs HN 4.10, Base 3.73). Then improved CROSS further: - dedupe.py: hybrid similarity (token+trigram Jaccard) at 0.40 threshold, cross-source links went from 3 to 13 items across 5 topics - render.py: [xref: HN5, HN4] -> [also on: HN, Reddit] for human-readable tags - youtube_yt.py: SYNONYMS dict so "hip hop" matches "rap" (0.33 -> 0.71 score) - SKILL.md: instruction #7 tells Claude to lead with cross-platform signals Validation: improved CROSS scores 4.38/5.0 vs original 3.98 (+0.40), wins 4/5 topics. Biggest gains in specificity (+0.8) and format compliance (+1.0). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
@@ -0,0 +1,471 @@
|
||||
---
|
||||
title: "feat: GOAT Synthesis Output - 15-Test Comparison and CROSS Improvement"
|
||||
type: feat
|
||||
status: completed
|
||||
date: 2026-02-25
|
||||
origin: docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md
|
||||
---
|
||||
|
||||
# GOAT Synthesis Output - 15-Test Comparison and CROSS Improvement
|
||||
|
||||
## Overview
|
||||
|
||||
The synthesis output - the "What I learned" / "I'm now an expert on X" narrative - is the ONLY thing that matters. All data collection, scoring, deduplication, and rendering exist solely to produce the best possible synthesis. This plan runs 15 synthesis comparisons (5 topics x 3 versions), judges which version produces the best final output, and creates concrete changes to make CROSS the GOAT.
|
||||
|
||||
## Problem Statement / Motivation
|
||||
|
||||
The previous 15-test comparison analyzed the data layer (JSON files, item counts, YouTube scores, cross-ref counts). That analysis showed CROSS wins on data quality metrics. But data quality != output quality. The user's exact words: "that's all that matters... all DATA/methods/etc. should be about making it the best fucking results ever."
|
||||
|
||||
The paper.design example shows what great output looks like:
|
||||
- "Paper Desktop + MCP is the big story" - a specific, grounded finding
|
||||
- "per @stephenhaney" / "per r/UXDesign" - actual source citations
|
||||
- "698 likes on X" - real engagement numbers woven into narrative
|
||||
- "KEY PATTERNS from the research:" - structured patterns, not vague summaries
|
||||
- "---I'm now an expert on paper.design" - confident, actionable invitation
|
||||
|
||||
We need to see what Base, HN, and CROSS actually produce as narratives, side-by-side, for the same topics. Then fix CROSS until its output is undeniably the best.
|
||||
|
||||
## Approach: Controlled Synthesis Comparison
|
||||
|
||||
### Why not re-run `claude --print "/last30days X"` 15 times?
|
||||
|
||||
Three problems:
|
||||
1. **WebSearch non-determinism** - Claude's WebSearch tool returns different results per run, confounding version comparison
|
||||
2. **QUERY_TYPE non-determinism** - Claude classifies "best rap songs 2026" as RECOMMENDATIONS one run and GENERAL the next, producing structurally different outputs
|
||||
3. **Reddit API non-determinism** - The existing JSON data shows Base got 4 Reddit items for Claude Code while CROSS got 10, purely from API timing
|
||||
|
||||
### The controlled approach
|
||||
|
||||
Reuse the 15 JSON files already captured (same-day, topic-sequential, verified clean). For each:
|
||||
|
||||
1. Convert JSON to compact markdown using each version's `render_compact()` (preserving version-specific rendering like CROSS's `[xref:]` tags)
|
||||
2. Feed the compact markdown to Claude with the synthesis instructions from each version's SKILL.md
|
||||
3. Hardcode QUERY_TYPE per topic to eliminate classification variance
|
||||
4. Exclude WebSearch to isolate Python pipeline differences
|
||||
5. Save all 15 synthesis outputs for side-by-side comparison
|
||||
|
||||
This gives us 15 synthesis narratives produced from identical underlying data, with only the version's rendering and synthesis instructions varying. The fairest possible comparison.
|
||||
|
||||
## The 5 Topics and Their QUERY_TYPEs
|
||||
|
||||
| # | Topic | QUERY_TYPE | Rationale |
|
||||
|---|-------|------------|-----------|
|
||||
| 1 | "Claude Code skills and MCP servers" | GENERAL | Broad understanding, not a ranked list |
|
||||
| 2 | "Seedance AI video generation" | NEWS | Recent product launch, what's happening |
|
||||
| 3 | "M4 MacBook Pro review" | RECOMMENDATIONS | "review" implies evaluation/ranking |
|
||||
| 4 | "best rap songs 2026" | RECOMMENDATIONS | "best" triggers ranked list |
|
||||
| 5 | "React vs Svelte 2026" | GENERAL | Framework debate, not ranked |
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Phase 1: Setup and Data Preservation
|
||||
|
||||
#### 1a. Copy JSON data to repo (CRITICAL - `/tmp` is ephemeral)
|
||||
|
||||
```bash
|
||||
mkdir -p docs/comparison-results/json
|
||||
cp /tmp/last30days-comparison/full/*.json docs/comparison-results/json/
|
||||
```
|
||||
|
||||
- [x] Copy all 15 JSON files to `docs/comparison-results/json/`
|
||||
- [x] Verify all 15 files are present and non-empty
|
||||
|
||||
#### 1b. Build JSON-to-compact converter
|
||||
|
||||
The render pipeline differs per version. We need each version's `render_compact()` to produce the compact markdown that version would actually show Claude.
|
||||
|
||||
**Script: `scripts/generate-synthesis-inputs.py`**
|
||||
|
||||
```python
|
||||
"""
|
||||
For each of the 15 JSON result files, render compact markdown
|
||||
using that version's render_compact() function.
|
||||
|
||||
The JSON files store raw data. Each version's render.py formats
|
||||
it differently (CROSS adds [xref:] tags, relevance scores, etc.).
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
sys.path.insert(0, 'scripts')
|
||||
from lib.render import render_compact
|
||||
from lib.schema import Report
|
||||
|
||||
for json_file in sorted(glob('docs/comparison-results/json/*.json')):
|
||||
data = json.load(open(json_file))
|
||||
report = Report.from_dict(data)
|
||||
compact = render_compact(report)
|
||||
# Save as .md alongside the JSON
|
||||
md_file = json_file.replace('/json/', '/compact/').replace('.json', '.md')
|
||||
open(md_file, 'w').write(compact)
|
||||
```
|
||||
|
||||
Problem: `render_compact()` differs across Base/HN/CROSS. Solution: run the converter 3 times, once per git checkout, only for that version's files.
|
||||
|
||||
- [x] Write `scripts/generate-synthesis-inputs.py`
|
||||
- [x] Run on Base checkout for `base-*.json` files -> `docs/comparison-results/compact/base-*.md`
|
||||
- [x] Run on HN checkout for `hn-*.json` files -> `docs/comparison-results/compact/hn-*.md`
|
||||
- [x] Run on CROSS checkout for `cross-*.json` files -> `docs/comparison-results/compact/cross-*.md`
|
||||
- [x] Clean `__pycache__` between every git checkout
|
||||
|
||||
#### 1c. Extract synthesis prompts from each version's SKILL.md
|
||||
|
||||
Each version's SKILL.md contains the instructions Claude uses to synthesize the compact data into the narrative. These differ across versions (CROSS has cross-ref instructions, HN has HN citation rules, etc.).
|
||||
|
||||
- [x] Check out Base (427a4e4), copy SKILL.md synthesis section (Judge Agent + Display phases) to `docs/comparison-results/prompts/base-synthesis-prompt.md`
|
||||
- [x] Check out HN (main), copy to `docs/comparison-results/prompts/hn-synthesis-prompt.md`
|
||||
- [x] Check out CROSS (feat/youtube-relevance-cross-source), copy to `docs/comparison-results/prompts/cross-synthesis-prompt.md`
|
||||
- [x] For each, prepend the hardcoded QUERY_TYPE and topic name
|
||||
|
||||
### Phase 2: Generate 15 Synthesis Outputs
|
||||
|
||||
#### 2a. Write the synthesis runner
|
||||
|
||||
**Script: `scripts/run-synthesis-comparison.py`**
|
||||
|
||||
For each of the 15 (topic, version) pairs:
|
||||
1. Read the compact markdown from `docs/comparison-results/compact/{version}-{n}-{slug}.md`
|
||||
2. Read the synthesis prompt from `docs/comparison-results/prompts/{version}-synthesis-prompt.md`
|
||||
3. Call the Anthropic API (Claude Sonnet 4.6 for cost/speed, or Opus 4.6 for max quality - configurable)
|
||||
4. Template: "You are running /last30days. The user asked about '{topic}'. QUERY_TYPE = {query_type}. Here is the research data:\n\n{compact_markdown}\n\n{synthesis_prompt}"
|
||||
5. Save response to `docs/comparison-results/synthesis/{version}-{n}-{slug}.md`
|
||||
|
||||
```python
|
||||
"""
|
||||
Generate synthesis narratives for all 15 test cases.
|
||||
Uses Anthropic API directly to control the synthesis environment.
|
||||
"""
|
||||
import anthropic
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
client = anthropic.Anthropic()
|
||||
|
||||
TOPICS = [
|
||||
(1, 'claude-code', 'Claude Code skills and MCP servers', 'GENERAL'),
|
||||
(2, 'seedance', 'Seedance AI video generation', 'NEWS'),
|
||||
(3, 'macbook', 'M4 MacBook Pro review', 'RECOMMENDATIONS'),
|
||||
(4, 'rap', 'best rap songs 2026', 'RECOMMENDATIONS'),
|
||||
(5, 'react-svelte', 'React vs Svelte 2026', 'GENERAL'),
|
||||
]
|
||||
VERSIONS = ['base', 'hn', 'cross']
|
||||
|
||||
for version in VERSIONS:
|
||||
prompt_text = Path(f'docs/comparison-results/prompts/{version}-synthesis-prompt.md').read_text()
|
||||
for num, slug, topic, qtype in TOPICS:
|
||||
compact = Path(f'docs/comparison-results/compact/{version}-{num}-{slug}.md').read_text()
|
||||
|
||||
user_msg = f"""You are the /last30days skill. The user asked: "{topic}"
|
||||
|
||||
Parsed intent:
|
||||
- TOPIC = {topic}
|
||||
- TARGET_TOOL = unknown
|
||||
- QUERY_TYPE = {qtype}
|
||||
|
||||
Here is the research output from the Python pipeline:
|
||||
|
||||
{compact}
|
||||
|
||||
Now synthesize this research into your expert narrative following these instructions:
|
||||
|
||||
{prompt_text}"""
|
||||
|
||||
response = client.messages.create(
|
||||
model=os.environ.get('SYNTHESIS_MODEL', 'claude-sonnet-4-6-20250514'),
|
||||
max_tokens=4096,
|
||||
messages=[{"role": "user", "content": user_msg}]
|
||||
)
|
||||
|
||||
output = response.content[0].text
|
||||
out_path = Path(f'docs/comparison-results/synthesis/{version}-{num}-{slug}.md')
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
out_path.write_text(output)
|
||||
print(f' {version}-{num}-{slug}: {len(output)} chars')
|
||||
```
|
||||
|
||||
- [x] Write `scripts/run-synthesis-comparison.py`
|
||||
- [x] Run it (15 API calls, ~2-3 min total)
|
||||
- [x] Verify all 15 synthesis files exist and are non-empty
|
||||
|
||||
#### 2b. Quality check synthesis outputs
|
||||
|
||||
Spot-check 3 outputs (one per version) for:
|
||||
- Did Claude actually produce a "What I learned" narrative?
|
||||
- Are there real citations (per @handle, per r/subreddit)?
|
||||
- Is the stats block present?
|
||||
- Is there an invitation at the end?
|
||||
|
||||
If any output is broken (e.g., Claude refused or produced meta-commentary instead of synthesis), adjust the prompt and re-run that one.
|
||||
|
||||
- [x] Spot-check `cross-1-claude-code.md`, `hn-2-seedance.md`, `base-4-rap.md`
|
||||
- [x] Fix any prompt issues and re-run failed outputs
|
||||
|
||||
### Phase 3: Evaluate - Which Version Produces the Best Output?
|
||||
|
||||
#### 3a. Define evaluation rubric (BEFORE reading any outputs)
|
||||
|
||||
Score each synthesis 1-5 on these dimensions:
|
||||
|
||||
| Dimension | Weight | 1 (Bad) | 3 (OK) | 5 (Great) |
|
||||
|-----------|--------|---------|--------|-----------|
|
||||
| **Groundedness** | 30% | Generic statements, no citations | Some citations but mixed with pre-trained knowledge | Every finding backed by specific source (per @handle, per r/sub, per channel) |
|
||||
| **Specificity** | 25% | Vague ("AI video tools are improving") | Some specifics but also filler | Named entities, exact numbers, product versions ("Seedance 2.0 added lip sync per @aifilmmaker") |
|
||||
| **Coverage** | 20% | Only mentions 1-2 sources | Mentions most sources but unevenly | Weaves findings from Reddit, X, YouTube, HN naturally into narrative |
|
||||
| **Actionability** | 15% | "This is interesting" with no next step | Generic suggestions | Specific, research-derived suggestions ("I can show you Seedance 2.0's lip sync workflow") |
|
||||
| **Format Compliance** | 10% | Missing stats block, no invitation | Partial stats, generic invitation | Perfect stats block with real counts, source-specific invitation |
|
||||
|
||||
Total: weighted average, 1.0 to 5.0
|
||||
|
||||
#### 3b. LLM-as-judge evaluation (blinded)
|
||||
|
||||
**Script: `scripts/evaluate-synthesis.py`**
|
||||
|
||||
For each topic (5 total), present all 3 versions to an evaluator LLM with version labels stripped. Ask it to score each on the rubric above.
|
||||
|
||||
```python
|
||||
"""
|
||||
Blinded LLM evaluation of synthesis outputs.
|
||||
Strips version labels, presents as Version A/B/C in random order.
|
||||
"""
|
||||
import anthropic
|
||||
import random
|
||||
|
||||
RUBRIC = """Score each version 1-5 on these dimensions:
|
||||
|
||||
1. GROUNDEDNESS (30%): Does the narrative cite specific sources?
|
||||
Look for: "per @handle", "per r/subreddit", "per [channel] on YouTube"
|
||||
1 = generic, no citations. 5 = every finding has a source.
|
||||
|
||||
2. SPECIFICITY (25%): Are findings specific or vague?
|
||||
1 = "AI video is trending". 5 = "Seedance 2.0 added lip sync, 698 likes per @paper"
|
||||
|
||||
3. COVERAGE (20%): Does it represent findings from all available sources?
|
||||
1 = only Reddit mentioned. 5 = Reddit, X, YouTube, HN woven naturally.
|
||||
|
||||
4. ACTIONABILITY (15%): Does the invitation give specific next steps based on research?
|
||||
1 = "let me know if you want more". 5 = "I can walk you through Seedance 2.0's workflow"
|
||||
|
||||
5. FORMAT COMPLIANCE (10%): Stats block present with real counts? Citation format correct?
|
||||
1 = missing stats. 5 = perfect stats block + source counts + top voices.
|
||||
|
||||
For each version, output:
|
||||
- Groundedness: X/5
|
||||
- Specificity: X/5
|
||||
- Coverage: X/5
|
||||
- Actionability: X/5
|
||||
- Format: X/5
|
||||
- Weighted Total: X.X/5.0
|
||||
- One sentence on what makes this version better or worse than the others.
|
||||
"""
|
||||
|
||||
# For each topic, randomly shuffle version order to prevent position bias
|
||||
for topic in TOPICS:
|
||||
versions = ['base', 'hn', 'cross']
|
||||
random.shuffle(versions)
|
||||
label_map = {v: chr(65+i) for i, v in enumerate(versions)} # A, B, C
|
||||
|
||||
# Present to evaluator
|
||||
prompt = f"Topic: {topic}\n\n"
|
||||
for v in versions:
|
||||
text = read_synthesis(v, topic)
|
||||
prompt += f"=== VERSION {label_map[v]} ===\n{text}\n\n"
|
||||
prompt += RUBRIC
|
||||
|
||||
# Call evaluator (use Opus for best judgment)
|
||||
response = evaluate(prompt)
|
||||
# Map labels back to versions
|
||||
save_evaluation(topic, response, label_map)
|
||||
```
|
||||
|
||||
- [x] Write `scripts/evaluate-synthesis.py`
|
||||
- [x] Run evaluation (5 API calls, one per topic)
|
||||
- [x] Collect scores into summary table
|
||||
|
||||
#### 3c. Human spot-check
|
||||
|
||||
Read 3 synthesis outputs yourself (one per version, same topic) and verify the LLM evaluation makes sense. If the LLM scores don't match your gut, investigate.
|
||||
|
||||
- [x] Read all 3 versions for topic 1 (Claude Code) side-by-side
|
||||
- [x] Read all 3 versions for topic 2 (Seedance) side-by-side
|
||||
- [x] Verify LLM scores align with human judgment
|
||||
|
||||
#### 3d. Compile verdict
|
||||
|
||||
| Topic | Base | HN | CROSS | Winner |
|
||||
|-------|------|-----|-------|--------|
|
||||
| Claude Code | X.X | X.X | X.X | ? |
|
||||
| Seedance | X.X | X.X | X.X | ? |
|
||||
| MacBook | X.X | X.X | X.X | ? |
|
||||
| Rap songs | X.X | X.X | X.X | ? |
|
||||
| React/Svelte | X.X | X.X | X.X | ? |
|
||||
| **Overall** | **X.X** | **X.X** | **X.X** | **?** |
|
||||
|
||||
Plus per-dimension breakdown:
|
||||
- Which version is best at Groundedness?
|
||||
- Which version is best at Specificity?
|
||||
- Which version is best at Coverage?
|
||||
- Which version is best at Actionability?
|
||||
- Which version is best at Format Compliance?
|
||||
|
||||
- [x] Build summary table with scores
|
||||
- [x] Identify per-dimension winners
|
||||
- [x] Write verdict paragraph
|
||||
|
||||
### Phase 4: Make CROSS the GOAT
|
||||
|
||||
Based on the evaluation, identify the specific synthesis weaknesses and fix them. Changes fall into two buckets:
|
||||
|
||||
#### Bucket A: Data pipeline changes (score.py, dedupe.py, render.py, youtube_yt.py)
|
||||
|
||||
These affect what compact markdown Claude sees. From the previous JSON analysis, known issues:
|
||||
|
||||
1. **Cross-source linking nearly broken** (3/178 items linked at 0.5 threshold)
|
||||
- Fix: hybrid similarity (token + trigram Jaccard) at 0.40 threshold
|
||||
- File: `scripts/lib/dedupe.py`
|
||||
- Expected impact: 26 items linked instead of 3
|
||||
|
||||
2. **Cross-ref rendering cryptic** (`[xref: HN5, HN4]` is meaningless)
|
||||
- Fix: Show `[also on: HN, Reddit]` with source names
|
||||
- File: `scripts/lib/render.py`
|
||||
- Expected impact: Claude can naturally say "discussed on both Reddit and HN"
|
||||
|
||||
3. **YouTube synonym gap** ("hip hop" != "rap" in relevance scoring)
|
||||
- Fix: SYNONYMS dict in youtube_yt.py
|
||||
- File: `scripts/lib/youtube_yt.py`
|
||||
- Expected impact: "Lit Hip Hop Mix 2026" gets 0.67+ instead of 0.33
|
||||
|
||||
4. **HN search too narrow** (React/Svelte returns 0 HN items)
|
||||
- Fix: Split multi-keyword topics into OR queries for HN Algolia
|
||||
- File: `scripts/lib/hackernews.py` (or wherever HN search lives)
|
||||
- Expected impact: Framework debate topics get HN coverage
|
||||
|
||||
#### Bucket B: Synthesis instruction changes (SKILL.md)
|
||||
|
||||
These affect how Claude interprets the data. Specific changes to discover from the 15-output comparison:
|
||||
|
||||
5. **Cross-source narrative instruction** - If CROSS data has `[also on: HN, Reddit]` tags but Claude ignores them, add explicit instruction: "When items appear across multiple platforms, lead with that cross-platform signal - it's the strongest evidence of importance."
|
||||
|
||||
6. **YouTube transcript utilization** - CROSS provides transcript snippets. If Claude's synthesis doesn't use them, add: "YouTube transcripts contain direct quotes and technical details. Weave 1-2 transcript quotes into your synthesis."
|
||||
|
||||
7. **Source weighting for synthesis** - Current: Reddit/X > YouTube > Web. Should HN be weighted higher for tech topics? Should YouTube transcript content elevate YouTube's synthesis weight?
|
||||
|
||||
8. **Citation density tuning** - The paper.design example has ~1 citation per paragraph. If any version over-cites (every sentence) or under-cites (no sources), adjust the citation frequency instruction.
|
||||
|
||||
9. **Stats block accuracy** - Verify each version's stats block matches actual data. If counts are wrong, the render output may need to include pre-computed stats that Claude can copy directly.
|
||||
|
||||
10. **Invitation quality** - The best invitations reference specific things from the research ("I can walk you through Seedance 2.0's lip sync workflow"). If a version produces generic invitations ("let me know if you want to know more"), strengthen the instruction.
|
||||
|
||||
Changes 5-10 are discovered from Phase 3 analysis. They may or may not apply.
|
||||
|
||||
- [x] Implement Bucket A changes (data pipeline fixes from JSON analysis)
|
||||
- [x] Analyze Phase 3 results to identify Bucket B changes needed
|
||||
- [x] Implement Bucket B changes (SKILL.md synthesis instruction improvements)
|
||||
- [x] Run `bash scripts/sync.sh` to deploy updated skill
|
||||
|
||||
### Phase 5: Validate
|
||||
|
||||
Re-run 5 synthesis outputs (one per topic, CROSS-after only) using the improved CROSS code, and compare against CROSS-before.
|
||||
|
||||
- [x] Generate 5 new compact markdowns from JSON data using improved render.py
|
||||
- [x] Generate 5 new synthesis outputs using improved SKILL.md
|
||||
- [x] Score with same rubric (LLM-as-judge, blinded against CROSS-before)
|
||||
- [x] Verify improvement on every dimension, or iterate
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
docs/comparison-results/
|
||||
json/ # 15 JSON files (already exist at /tmp, need copying)
|
||||
base-1-claude-code.json
|
||||
hn-1-claude-code.json
|
||||
cross-1-claude-code.json
|
||||
... (15 total)
|
||||
compact/ # 15 compact markdown files (rendered per-version)
|
||||
base-1-claude-code.md
|
||||
hn-1-claude-code.md
|
||||
cross-1-claude-code.md
|
||||
... (15 total)
|
||||
prompts/ # 3 synthesis prompts (extracted from each SKILL.md)
|
||||
base-synthesis-prompt.md
|
||||
hn-synthesis-prompt.md
|
||||
cross-synthesis-prompt.md
|
||||
synthesis/ # 15 synthesis narratives (the actual output!)
|
||||
base-1-claude-code.md
|
||||
hn-1-claude-code.md
|
||||
cross-1-claude-code.md
|
||||
... (15 total)
|
||||
evaluation/ # 5 evaluation results (one per topic, blinded)
|
||||
eval-1-claude-code.md
|
||||
eval-2-seedance.md
|
||||
eval-3-macbook.md
|
||||
eval-4-rap.md
|
||||
eval-5-react-svelte.md
|
||||
summary.md # Final verdict with all scores and improvement plan
|
||||
|
||||
scripts/
|
||||
generate-synthesis-inputs.py # JSON -> compact converter
|
||||
run-synthesis-comparison.py # Compact + prompt -> synthesis via Claude API
|
||||
evaluate-synthesis.py # Blinded LLM evaluation
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] All 15 JSON files preserved in `docs/comparison-results/json/`
|
||||
- [x] All 15 compact markdowns generated (version-specific rendering preserved)
|
||||
- [x] All 3 synthesis prompts extracted from version-specific SKILL.md files
|
||||
- [x] All 15 synthesis narratives generated and saved
|
||||
- [x] All 15 syntheses scored on 5-dimension rubric (blinded, LLM-as-judge)
|
||||
- [x] Summary table shows per-topic and overall winner
|
||||
- [x] At least 3 synthesis outputs human-spot-checked against LLM scores
|
||||
- [x] CROSS improvement plan includes both data pipeline (Bucket A) and synthesis instruction (Bucket B) changes
|
||||
- [x] Bucket A changes implemented with tests
|
||||
- [x] Bucket B changes implemented and deployed via sync.sh
|
||||
- [x] 5 validation synthesis outputs show improvement over CROSS-before
|
||||
- [x] Final summary.md documents everything: scores, verdict, changes, validation
|
||||
|
||||
## Dependencies & Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| `/tmp` data loss before copying | Phase 1a is the first action - copy immediately |
|
||||
| Claude API rate limits during synthesis | 15 calls is well under limits; add 2s sleep between calls |
|
||||
| LLM evaluator bias toward longer outputs | Rubric weights specificity and groundedness, not length |
|
||||
| QUERY_TYPE affecting output format | Hardcoded per topic in synthesis prompt |
|
||||
| Reddit item-count variance confounding comparison | Note caveat; flag topics where counts differ >30% |
|
||||
| render_compact() crash on JSON data | Test converter on 1 file before batch run |
|
||||
|
||||
## Technical Considerations
|
||||
|
||||
### Why Anthropic API instead of `claude --print`?
|
||||
|
||||
1. **Controlled environment** - Same model, same temperature, same max_tokens for all 15
|
||||
2. **No WebSearch confound** - API calls don't have tool access unless we grant it
|
||||
3. **Reproducible** - Can re-run with different models or prompts
|
||||
4. **Faster** - API call takes ~10s vs ~3min for full `claude --print` with tools
|
||||
|
||||
### Why reuse JSON data instead of re-running the pipeline?
|
||||
|
||||
1. **Eliminates temporal non-determinism** - Same Reddit/X/YouTube data for all comparisons
|
||||
2. **No rate limiting** - Zero API calls to source platforms
|
||||
3. **Already validated** - 15 files verified clean, zero errors
|
||||
4. **Still tests rendering differences** - Each version's render_compact() runs on its checkout
|
||||
|
||||
### What about the WebSearch step?
|
||||
|
||||
The full skill pipeline includes Claude doing WebSearch after the Python script. We deliberately exclude this because:
|
||||
1. WebSearch results vary per run (different web results each time)
|
||||
2. WebSearch is identical across all 3 versions (no version difference to test)
|
||||
3. Including it would confound the comparison with noise
|
||||
4. The Python pipeline data is where version differences live
|
||||
|
||||
## Sources & References
|
||||
|
||||
- Previous JSON analysis: [docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md](docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md) - data-layer comparison with scoring verdict
|
||||
- Previous test plan: [docs/plans/2026-02-25-feat-15-test-comparison-make-cross-goat-plan.md](docs/plans/2026-02-25-feat-15-test-comparison-make-cross-goat-plan.md) - execution protocol for 15 JSON test runs
|
||||
- SKILL.md synthesis instructions: `SKILL.md` lines 165-320 (Judge Agent + Display phases)
|
||||
- Scoring weights: `scripts/lib/score.py` - 45% relevance + 25% recency + 30% engagement
|
||||
- Render pipeline: `scripts/lib/render.py:57` - `render_compact()` function
|
||||
- Cross-source linking: `scripts/lib/dedupe.py:160` - `cross_source_link()` function
|
||||
- YouTube relevance: `scripts/lib/youtube_yt.py` - `_compute_relevance()` token overlap
|
||||
- Existing test harness: `scripts/test-v1-vs-v2.sh:104` - `claude --print` approach
|
||||
- YouTube display bug: `docs/plans/2026-02-15-fix-youtube-display-and-search-quality-plan.md`
|
||||
- Output formatting fixes: `docs/plans/2026-02-06-fix-last30days-v2-formatting-reddit-citations-plan.md`
|
||||
Reference in New Issue
Block a user