Add local search quality evaluation harness

Add an optional local evaluator that compares a baseline revision against a candidate checkout, computes deterministic stability metrics, and can call Gemini for judged ranking metrics when configured.

The harness isolates child runs with a temporary HOME and a node-free PATH so historical revisions cannot trigger Bird browser-cookie auth during evaluation.

Validation: uv run python -m unittest and local smoke/full deterministic eval runs.
This commit is contained in:
Jeffrey Sperling
2026-03-13 19:21:33 -07:00
parent 946af84f9a
commit 8eda5fad5c
5 changed files with 732 additions and 0 deletions
+2
View File
@@ -243,10 +243,12 @@ def get_config() -> Dict[str, Any]:
keys = [
('XAI_API_KEY', None),
('GEMINI_API_KEY', None),
('OPENROUTER_API_KEY', None),
('PARALLEL_API_KEY', None),
('BRAVE_API_KEY', None),
('XIAOHONGSHU_API_BASE', None),
('GEMINI_MODEL', None),
('OPENAI_MODEL_POLICY', 'auto'),
('OPENAI_MODEL_PIN', None),
('XAI_MODEL_POLICY', 'latest'),