sawz/last30days-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Harden local search evaluation harness

Browse Source 
Isolate eval subprocesses from local yt-dlp config and fix nDCG normalization against the judged pool.

Validation: uv run python -m unittest tests.test_evaluate_search_quality
This commit is contained in:
Jeffrey Sperling
2026-03-14 00:38:43 -07:00
parent 3a0f3d8b19
commit 8c1dce95e8
3 changed files with 78 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/search-quality-eval.md
+1
View File
@@ -43,5 +43,6 @@ Notes:
- The script forces a clean env-based auth path when it shells out to `last30days.py`.
- It passes `XAI_API_KEY`, `OPENAI_API_KEY`, and `SCRAPECREATORS_API_KEY`, but intentionally does not pass browser-cookie X auth. That keeps evaluation runs on the popup-free path.
- It also strips `node` from the eval `PATH` and wraps `yt-dlp` with `--ignore-config`, so older revisions do not inherit local browser-cookie config either.
- `Jaccard` and retention are regression guards, not truth metrics.
- `Precision@5` and `nDCG@5` are only as good as the judged pool. They help compare revisions, but they are not a substitute for a larger labeled benchmark.