sawz/last30days-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3803f3df4bd8a4e37e1b7c9741681bad0be2103e
last30days-skill/docs/plans/2026-02-25-feat-15-test-comparison-make-cross-goat-plan.md
T
Matt Van Horn 32992834ee Merge PR #48: feat: add Xiaohongshu source + Reddit public fallback 
- Xiaohongshu search via local MCP service (opt-in, zero impact if service not running)
- Reddit public JSON fallback (works with zero API keys)
- Reddit priority: ScrapeCreators -> OpenAI -> public fallback
- Updated env.py: Reddit always available via public fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 16:11:35 -08:00

10 KiB
Raw Blame History

title, type, status, date, origin
title type status date origin
feat: 15-Test Side-by-Side Comparison - Make CROSS the GOAT feat active 2026-02-25 docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md

15-Test Side-by-Side Comparison - Make CROSS the GOAT

Overview

Run all 5 canonical topics through all 3 skill versions (base, HN, CROSS) for 15 total full last30days runs. Save every result. Analyze side-by-side. Judge which version is best. Then create an improvement plan to make CROSS the definitive next release.

Problem Statement / Motivation

The previous analysis only ran 5 tests on the CROSS branch - never comparing the same topics across all 3 versions. Without true side-by-side data, we can't judge whether CROSS is actually better or if the new features (dynamic YouTube scoring, cross-refs) introduce regressions. The user wants to see all 15 results, know which version wins, and get a concrete plan to make CROSS the GOAT before shipping.

The 3 Versions

Version Git State Sources YouTube Relevance Cross-Refs
Base commit 427a4e4 Reddit, X, YouTube, Web Hardcoded 0.7 No
HN main / f60a435 Reddit, X, YouTube, HN, Web Hardcoded 0.7 No
CROSS feat/youtube-relevance-cross-source / 0591f55 Reddit, X, YouTube, HN, Web Dynamic 0.1-1.0 Yes (Jaccard 0.5)

The 5 Topics

# Topic Category Why chosen
1 "Claude Code skills and MCP servers" Developer tools Core user base topic
2 "Seedance AI video generation" Creative AI Trending topic, cross-platform buzz
3 "M4 MacBook Pro review" Consumer tech Product review, mainstream
4 "best rap songs 2026" Pop culture Non-tech stress test
5 "React vs Svelte 2026" Framework debate Dev community, opinion-heavy

Test Matrix (15 Runs)

Run in topic-sequential order (all 3 versions of topic 1 back-to-back, then topic 2, etc.) to minimize temporal confounds on YouTube/X search results.

Run Topic Version Output File
1 Claude Code Base base-1-claude-code.json
2 Claude Code HN hn-1-claude-code.json
3 Claude Code CROSS cross-1-claude-code.json
4 Seedance Base base-2-seedance.json
5 Seedance HN hn-2-seedance.json
6 Seedance CROSS cross-2-seedance.json
7 MacBook Base base-3-macbook.json
8 MacBook HN hn-3-macbook.json
9 MacBook CROSS cross-3-macbook.json
10 Rap songs Base base-4-rap.json
11 Rap songs HN hn-4-rap.json
12 Rap songs CROSS cross-4-rap.json
13 React/Svelte Base base-5-react-svelte.json
14 React/Svelte HN hn-5-react-svelte.json
15 React/Svelte CROSS cross-5-react-svelte.json

Output directory: /tmp/last30days-comparison/full/

Execution Protocol

Pre-flight (once)

  • Clear model cache: rm -f ~/.cache/last30days/model_selection.json
  • Run --diagnose on current branch, save as diagnose-baseline.json
  • Verify all API keys active: Reddit (OPENAI_API_KEY), X (Bird cookies or XAI_API_KEY), YouTube (yt-dlp), HN (no key needed), Web (parallel AI or Brave)
  • Create output dir: mkdir -p /tmp/last30days-comparison/full
  • Stash any uncommitted changes: git stash (none needed - no uncommitted changes)

Per-topic loop (repeat 5 times)

For each topic, run all 3 versions back-to-back:

# CRITICAL: Clean __pycache__ between EVERY git checkout to prevent stale bytecode
cleanup() {
  find scripts -name '__pycache__' -exec rm -rf {} + 2>/dev/null
  find scripts -name '*.pyc' -delete 2>/dev/null
}

  • Step 1: Base version

    • cleanup && git checkout 427a4e4
    • python3 scripts/last30days.py --diagnose 2>/dev/null (verify sources match baseline)
    • python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/base-{N}-{slug}.json 2>/tmp/last30days-comparison/full/base-{N}-{slug}.log

  • Step 2: HN version

    • cleanup && git checkout main
    • python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/hn-{N}-{slug}.json 2>/tmp/last30days-comparison/full/hn-{N}-{slug}.log

  • Step 3: CROSS version

    • cleanup && git checkout feat/youtube-relevance-cross-source
    • python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/cross-{N}-{slug}.json 2>/tmp/last30days-comparison/full/cross-{N}-{slug}.log

Post-run

  • Return to feature branch: git checkout feat/youtube-relevance-cross-source
  • Verify all 15 JSON files exist and are non-empty
  • Check for *_error fields in any JSON output - flag but include (zero errors)

Analysis Dimensions

1. Source Coverage Table

For each of 15 runs, count items per source:

Topic Version Reddit X YouTube HN Web Total

Expected: Base has 0 HN items. HN and CROSS should have identical source counts (same search code). Any differences indicate API non-determinism.

2. YouTube Relevance Comparison

Matched-item analysis: Match videos by video_id across Base/CROSS runs of the same topic. For each matched video:

  • Base relevance: always 0.7
  • CROSS relevance: dynamic score
  • Delta and direction (did dynamic scoring promote or demote this video?)

Aggregate analysis: Distribution stats (min/avg/max/stddev) per version per topic.

3. Cross-Source Links (CROSS only)

  • Count of items with cross_refs per topic
  • Quality assessment: are the linked items actually about the same story?
  • Which source pairs link most often? (Reddit-HN? YouTube-HN? X-Reddit?)

4. Score Distribution and Rankings

  • Mean/median score of top 10 items per version per topic
  • Rank position changes: do the same items appear in different orders?
  • Does HN crowd out Reddit/X items in the top 10?

5. "Best Version" Judging Criteria

Define BEFORE analyzing to avoid bias:

Metric Weight How measured
Source diversity 25% Shannon entropy across source types in top 15 items
Score quality 25% Mean score of top 10 items
Relevance accuracy 25% YouTube: do high-relevance videos actually match the query? Manual spot-check of top 3 + bottom 3 per topic
Bonus features 25% HN value-add (unique info not in other sources) + cross-ref utility (do xrefs add value to the reader?)

6. HN Value-Add Assessment

For each topic where HN returns results:

  • How many HN items appear in the overall top 10?
  • Do HN items provide information not available from Reddit/X/YouTube?
  • Are HN comment insights (top_comments) genuinely useful?

Deliverables

A. Raw Results Archive

All 15 JSON files plus logs saved in /tmp/last30days-comparison/full/, also copied to docs/comparison-results/ for persistence.

B. Comparison Summary Table

A single markdown table showing all 15 runs with key metrics per cell.

C. Version Verdict

Clear judgment: which version is best overall, and best per topic category (tech, consumer, culture).

D. CROSS Improvement Plan

Concrete changes to make CROSS the GOAT:

Based on the previous analysis (see origin doc), likely improvements include:

  1. Fix cross-source linking - Switch from char-trigram Jaccard (0.5) to hybrid similarity (max of trigram + token Jaccard) at threshold 0.40. Previous modeling showed this goes from 1 link to 24 links across 5 tests.

  2. Cross-ref rendering - Change from cryptic [xref: X1, HN3] IDs to human-readable [also on: Reddit, HN] labels.

  3. HN search broadening - Investigate why React/Svelte returns 0 HN items.

  4. YouTube relevance - Already working well, may need minor threshold tuning based on 15-test data.

  5. Score normalization - Ensure HN items don't systematically crowd out other sources in the merged ranking.

New improvements to discover from 15-test data:

  • Regressions introduced by CROSS changes
  • Edge cases where Base or HN outperforms CROSS
  • Topic-specific tuning opportunities

Acceptance Criteria

  • All 15 JSON result files saved and non-empty
  • All 15 runs use identical source routing (verified via --diagnose)
  • pycache cleaned between every git checkout
  • Comparison summary table covers all 15 runs
  • YouTube matched-item analysis for at least 3 topics (all 5 done)
  • Cross-source link quality spot-check for CROSS runs
  • Clear version verdict with supporting data
  • CROSS improvement plan with specific code changes
  • All results appended to the analysis document for the user to review

Technical Considerations

  • pycache poisoning: Python caches bytecode. Switching git commits without clearing __pycache__ means old code runs even after checkout. MUST clean between every checkout.
  • API rate limiting: 15 runs hitting Reddit, X, YouTube, HN APIs. The --quick mode and natural ~30-90s per run provide spacing, but monitor for 429s in logs.
  • YouTube non-determinism: yt-dlp search results shift over time. Topic-sequential ordering minimizes this by running all 3 versions of the same topic within ~3-5 minutes.
  • X search session: Bird CLI depends on browser cookies. If session expires mid-run, X results degrade silently. Check X item counts across runs.
  • --quick limitation: Results reflect Phase 1 only (8-12 items per source). Full pipeline with --deep might show different patterns. Document this caveat.

Sources & References

Reference in New Issue View Git Blame Copy Permalink