sawz/last30days-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
32992834ee42e6fd0c433fcd70764e94a1296767
last30days-skill/docs/plans/2026-02-25-feat-15-test-comparison-make-cross-goat-plan.md
T
Matt Van Horn 32992834ee Merge PR #48: feat: add Xiaohongshu source + Reddit public fallback 
- Xiaohongshu search via local MCP service (opt-in, zero impact if service not running)
- Reddit public JSON fallback (works with zero API keys)
- Reddit priority: ScrapeCreators -> OpenAI -> public fallback
- Updated env.py: Reddit always available via public fallback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 16:11:35 -08:00

213 lines
10 KiB
Markdown
Raw Blame History

---
title: "feat: 15-Test Side-by-Side Comparison - Make CROSS the GOAT"
type: feat
status: active
date: 2026-02-25
origin: docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md
---
# 15-Test Side-by-Side Comparison - Make CROSS the GOAT
## Overview
Run all 5 canonical topics through all 3 skill versions (base, HN, CROSS) for 15 total full last30days runs. Save every result. Analyze side-by-side. Judge which version is best. Then create an improvement plan to make CROSS the definitive next release.
## Problem Statement / Motivation
The previous analysis only ran 5 tests on the CROSS branch - never comparing the same topics across all 3 versions. Without true side-by-side data, we can't judge whether CROSS is actually better or if the new features (dynamic YouTube scoring, cross-refs) introduce regressions. The user wants to see all 15 results, know which version wins, and get a concrete plan to make CROSS the GOAT before shipping.
## The 3 Versions
| Version | Git State | Sources | YouTube Relevance | Cross-Refs |
|---------|-----------|---------|-------------------|------------|
| **Base** | commit `427a4e4` | Reddit, X, YouTube, Web | Hardcoded 0.7 | No |
| **HN** | `main` / `f60a435` | Reddit, X, YouTube, **HN**, Web | Hardcoded 0.7 | No |
| **CROSS** | `feat/youtube-relevance-cross-source` / `0591f55` | Reddit, X, YouTube, HN, Web | Dynamic 0.1-1.0 | Yes (Jaccard 0.5) |
## The 5 Topics
| # | Topic | Category | Why chosen |
|---|-------|----------|------------|
| 1 | "Claude Code skills and MCP servers" | Developer tools | Core user base topic |
| 2 | "Seedance AI video generation" | Creative AI | Trending topic, cross-platform buzz |
| 3 | "M4 MacBook Pro review" | Consumer tech | Product review, mainstream |
| 4 | "best rap songs 2026" | Pop culture | Non-tech stress test |
| 5 | "React vs Svelte 2026" | Framework debate | Dev community, opinion-heavy |
## Test Matrix (15 Runs)
Run in **topic-sequential** order (all 3 versions of topic 1 back-to-back, then topic 2, etc.) to minimize temporal confounds on YouTube/X search results.
| Run | Topic | Version | Output File |
|-----|-------|---------|-------------|
| 1 | Claude Code | Base | `base-1-claude-code.json` |
| 2 | Claude Code | HN | `hn-1-claude-code.json` |
| 3 | Claude Code | CROSS | `cross-1-claude-code.json` |
| 4 | Seedance | Base | `base-2-seedance.json` |
| 5 | Seedance | HN | `hn-2-seedance.json` |
| 6 | Seedance | CROSS | `cross-2-seedance.json` |
| 7 | MacBook | Base | `base-3-macbook.json` |
| 8 | MacBook | HN | `hn-3-macbook.json` |
| 9 | MacBook | CROSS | `cross-3-macbook.json` |
| 10 | Rap songs | Base | `base-4-rap.json` |
| 11 | Rap songs | HN | `hn-4-rap.json` |
| 12 | Rap songs | CROSS | `cross-4-rap.json` |
| 13 | React/Svelte | Base | `base-5-react-svelte.json` |
| 14 | React/Svelte | HN | `hn-5-react-svelte.json` |
| 15 | React/Svelte | CROSS | `cross-5-react-svelte.json` |
**Output directory:** `/tmp/last30days-comparison/full/`
## Execution Protocol
### Pre-flight (once)
- [x] Clear model cache: `rm -f ~/.cache/last30days/model_selection.json`
- [x] Run `--diagnose` on current branch, save as `diagnose-baseline.json`
- [x] Verify all API keys active: Reddit (OPENAI_API_KEY), X (Bird cookies or XAI_API_KEY), YouTube (yt-dlp), HN (no key needed), Web (parallel AI or Brave)
- [x] Create output dir: `mkdir -p /tmp/last30days-comparison/full`
- [x] Stash any uncommitted changes: `git stash` (none needed - no uncommitted changes)
### Per-topic loop (repeat 5 times)
For each topic, run all 3 versions back-to-back:
```bash
# CRITICAL: Clean __pycache__ between EVERY git checkout to prevent stale bytecode
cleanup() {
find scripts -name '__pycache__' -exec rm -rf {} + 2>/dev/null
find scripts -name '*.pyc' -delete 2>/dev/null
}
```
- [ ] **Step 1: Base version**
- `cleanup && git checkout 427a4e4`
- `python3 scripts/last30days.py --diagnose 2>/dev/null` (verify sources match baseline)
- `python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/base-{N}-{slug}.json 2>/tmp/last30days-comparison/full/base-{N}-{slug}.log`
- [ ] **Step 2: HN version**
- `cleanup && git checkout main`
- `python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/hn-{N}-{slug}.json 2>/tmp/last30days-comparison/full/hn-{N}-{slug}.log`
- [ ] **Step 3: CROSS version**
- `cleanup && git checkout feat/youtube-relevance-cross-source`
- `python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/cross-{N}-{slug}.json 2>/tmp/last30days-comparison/full/cross-{N}-{slug}.log`
### Post-run
- [x] Return to feature branch: `git checkout feat/youtube-relevance-cross-source`
- [x] Verify all 15 JSON files exist and are non-empty
- [x] Check for `*_error` fields in any JSON output - flag but include (zero errors)
## Analysis Dimensions
### 1. Source Coverage Table
For each of 15 runs, count items per source:
| Topic | Version | Reddit | X | YouTube | HN | Web | Total |
|-------|---------|--------|---|---------|----|----|-------|
Expected: Base has 0 HN items. HN and CROSS should have identical source counts (same search code). Any differences indicate API non-determinism.
### 2. YouTube Relevance Comparison
**Matched-item analysis:** Match videos by `video_id` across Base/CROSS runs of the same topic. For each matched video:
- Base relevance: always 0.7
- CROSS relevance: dynamic score
- Delta and direction (did dynamic scoring promote or demote this video?)
**Aggregate analysis:** Distribution stats (min/avg/max/stddev) per version per topic.
### 3. Cross-Source Links (CROSS only)
- Count of items with cross_refs per topic
- Quality assessment: are the linked items actually about the same story?
- Which source pairs link most often? (Reddit-HN? YouTube-HN? X-Reddit?)
### 4. Score Distribution and Rankings
- Mean/median score of top 10 items per version per topic
- Rank position changes: do the same items appear in different orders?
- Does HN crowd out Reddit/X items in the top 10?
### 5. "Best Version" Judging Criteria
Define BEFORE analyzing to avoid bias:
| Metric | Weight | How measured |
|--------|--------|-------------|
| Source diversity | 25% | Shannon entropy across source types in top 15 items |
| Score quality | 25% | Mean score of top 10 items |
| Relevance accuracy | 25% | YouTube: do high-relevance videos actually match the query? Manual spot-check of top 3 + bottom 3 per topic |
| Bonus features | 25% | HN value-add (unique info not in other sources) + cross-ref utility (do xrefs add value to the reader?) |
### 6. HN Value-Add Assessment
For each topic where HN returns results:
- How many HN items appear in the overall top 10?
- Do HN items provide information not available from Reddit/X/YouTube?
- Are HN comment insights (top_comments) genuinely useful?
## Deliverables
### A. Raw Results Archive
All 15 JSON files plus logs saved in `/tmp/last30days-comparison/full/`, also copied to `docs/comparison-results/` for persistence.
### B. Comparison Summary Table
A single markdown table showing all 15 runs with key metrics per cell.
### C. Version Verdict
Clear judgment: which version is best overall, and best per topic category (tech, consumer, culture).
### D. CROSS Improvement Plan
Concrete changes to make CROSS the GOAT:
**Based on the previous analysis (see origin doc), likely improvements include:**
1. **Fix cross-source linking** - Switch from char-trigram Jaccard (0.5) to hybrid similarity (max of trigram + token Jaccard) at threshold 0.40. Previous modeling showed this goes from 1 link to 24 links across 5 tests.
2. **Cross-ref rendering** - Change from cryptic `[xref: X1, HN3]` IDs to human-readable `[also on: Reddit, HN]` labels.
3. **HN search broadening** - Investigate why React/Svelte returns 0 HN items.
4. **YouTube relevance** - Already working well, may need minor threshold tuning based on 15-test data.
5. **Score normalization** - Ensure HN items don't systematically crowd out other sources in the merged ranking.
**New improvements to discover from 15-test data:**
- Regressions introduced by CROSS changes
- Edge cases where Base or HN outperforms CROSS
- Topic-specific tuning opportunities
## Acceptance Criteria
- [x] All 15 JSON result files saved and non-empty
- [x] All 15 runs use identical source routing (verified via --diagnose)
- [x] __pycache__ cleaned between every git checkout
- [x] Comparison summary table covers all 15 runs
- [x] YouTube matched-item analysis for at least 3 topics (all 5 done)
- [x] Cross-source link quality spot-check for CROSS runs
- [x] Clear version verdict with supporting data
- [x] CROSS improvement plan with specific code changes
- [x] All results appended to the analysis document for the user to review
## Technical Considerations
- **__pycache__ poisoning**: Python caches bytecode. Switching git commits without clearing `__pycache__` means old code runs even after checkout. MUST clean between every checkout.
- **API rate limiting**: 15 runs hitting Reddit, X, YouTube, HN APIs. The `--quick` mode and natural ~30-90s per run provide spacing, but monitor for 429s in logs.
- **YouTube non-determinism**: yt-dlp search results shift over time. Topic-sequential ordering minimizes this by running all 3 versions of the same topic within ~3-5 minutes.
- **X search session**: Bird CLI depends on browser cookies. If session expires mid-run, X results degrade silently. Check X item counts across runs.
- **`--quick` limitation**: Results reflect Phase 1 only (8-12 items per source). Full pipeline with `--deep` might show different patterns. Document this caveat.
## Sources & References
- Previous analysis: [docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md](docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md)
- Implementation plan: [docs/plans/2026-02-25-feat-youtube-relevance-and-cross-source-linking-plan.md](docs/plans/2026-02-25-feat-youtube-relevance-and-cross-source-linking-plan.md)
- Existing comparison harness: [scripts/test-v1-vs-v2.sh](scripts/test-v1-vs-v2.sh)
- Scoring weights: [scripts/lib/score.py](scripts/lib/score.py) - 45% relevance + 25% recency + 30% engagement
Reference in New Issue View Git Blame Copy Permalink