Jeffrey Sperling
8eda5fad5c
Add an optional local evaluator that compares a baseline revision against a candidate checkout, computes deterministic stability metrics, and can call Gemini for judged ranking metrics when configured. The harness isolates child runs with a temporary HOME and a node-free PATH so historical revisions cannot trigger Bird browser-cookie auth during evaluation. Validation: uv run python -m unittest and local smoke/full deterministic eval runs.
1.6 KiB
1.6 KiB
Search Quality Eval
scripts/evaluate_search_quality.py is an optional local evaluation step for retrieval quality. It is not part of the user-facing runtime and does not need to run in CI by default.
What it does:
- runs a baseline revision (default
origin/main) against a candidate checkout - evaluates the fixed 5 reviewer topics by default
- computes deterministic stability metrics:
Jaccardoverlap vs baseline- retention vs baseline
- per-source counts and overlap
- optionally calls Gemini as a judge for graded relevance labels and then computes:
Precision@5nDCG@5- source-coverage recall across the judged union pool
Recommended usage:
uv run python scripts/evaluate_search_quality.py
Useful flags:
uv run python scripts/evaluate_search_quality.py \
--baseline-rev origin/main \
--candidate-rev HEAD \
--no-default-topics \
--topic "cursor IDE pricing" \
--per-source-limit 5
Gemini configuration:
- set
GEMINI_API_KEYto enable LLM judging - optional: set
GEMINI_MODEL - default model is
gemini-3-pro-previewfor the direct Gemini API
Notes:
- The script forces a clean env-based auth path when it shells out to
last30days.py. - It passes
XAI_API_KEY,OPENAI_API_KEY, andSCRAPECREATORS_API_KEY, but intentionally does not pass browser-cookie X auth. That keeps evaluation runs on the popup-free path. Jaccardand retention are regression guards, not truth metrics.Precision@5andnDCG@5are only as good as the judged pool. They help compare revisions, but they are not a substitute for a larger labeled benchmark.