Jeffrey Sperling
8eda5fad5c
Add an optional local evaluator that compares a baseline revision against a candidate checkout, computes deterministic stability metrics, and can call Gemini for judged ranking metrics when configured. The harness isolates child runs with a temporary HOME and a node-free PATH so historical revisions cannot trigger Bird browser-cookie auth during evaluation. Validation: uv run python -m unittest and local smoke/full deterministic eval runs.
47 lines
1.6 KiB
Markdown
47 lines
1.6 KiB
Markdown
# Search Quality Eval
|
|
|
|
`scripts/evaluate_search_quality.py` is an optional local evaluation step for retrieval quality. It is not part of the user-facing runtime and does not need to run in CI by default.
|
|
|
|
What it does:
|
|
|
|
- runs a baseline revision (default `origin/main`) against a candidate checkout
|
|
- evaluates the fixed 5 reviewer topics by default
|
|
- computes deterministic stability metrics:
|
|
- `Jaccard` overlap vs baseline
|
|
- retention vs baseline
|
|
- per-source counts and overlap
|
|
- optionally calls Gemini as a judge for graded relevance labels and then computes:
|
|
- `Precision@5`
|
|
- `nDCG@5`
|
|
- source-coverage recall across the judged union pool
|
|
|
|
Recommended usage:
|
|
|
|
```bash
|
|
uv run python scripts/evaluate_search_quality.py
|
|
```
|
|
|
|
Useful flags:
|
|
|
|
```bash
|
|
uv run python scripts/evaluate_search_quality.py \
|
|
--baseline-rev origin/main \
|
|
--candidate-rev HEAD \
|
|
--no-default-topics \
|
|
--topic "cursor IDE pricing" \
|
|
--per-source-limit 5
|
|
```
|
|
|
|
Gemini configuration:
|
|
|
|
- set `GEMINI_API_KEY` to enable LLM judging
|
|
- optional: set `GEMINI_MODEL`
|
|
- default model is `gemini-3-pro-preview` for the direct Gemini API
|
|
|
|
Notes:
|
|
|
|
- The script forces a clean env-based auth path when it shells out to `last30days.py`.
|
|
- It passes `XAI_API_KEY`, `OPENAI_API_KEY`, and `SCRAPECREATORS_API_KEY`, but intentionally does not pass browser-cookie X auth. That keeps evaluation runs on the popup-free path.
|
|
- `Jaccard` and retention are regression guards, not truth metrics.
|
|
- `Precision@5` and `nDCG@5` are only as good as the judged pool. They help compare revisions, but they are not a substitute for a larger labeled benchmark.
|