Commit Graph

4 Commits

Author SHA1 Message Date
Dinakar Sarbada cfde1dbbe4 docs: fix stale script paths 2026-05-17 00:08:57 -07:00
Jeffrey Sperling 8c1dce95e8 Harden local search evaluation harness
Isolate eval subprocesses from local yt-dlp config and fix nDCG normalization against the judged pool.

Validation: uv run python -m unittest tests.test_evaluate_search_quality
2026-03-14 00:38:43 -07:00
Jeffrey Sperling 3a0f3d8b19 Accept GOOGLE_API_KEY for local Gemini eval
This workspace uses GOOGLE_API_KEY as the canonical Google credential. Accept it ahead of the Gemini-specific aliases so the local evaluation harness can run without a separate GEMINI_API_KEY export.

Validation: uv run python -m unittest tests.test_env_project tests.test_evaluate_search_quality and a one-shot keychain-backed resolution check.
2026-03-13 19:25:29 -07:00
Jeffrey Sperling 8eda5fad5c Add local search quality evaluation harness
Add an optional local evaluator that compares a baseline revision against a candidate checkout, computes deterministic stability metrics, and can call Gemini for judged ranking metrics when configured.

The harness isolates child runs with a temporary HOME and a node-free PATH so historical revisions cannot trigger Bird browser-cookie auth during evaluation.

Validation: uv run python -m unittest and local smoke/full deterministic eval runs.
2026-03-13 19:21:33 -07:00