Commit Graph
9 Commits
Author SHA1 Message Date
Jeffrey Sperling d667586597 Add shared query.py and relevance.py modules
Consolidate duplicated _extract_core_subject() (7 copies across bird_x,
reddit, youtube_yt, tiktok, instagram, bluesky, scrapecreators_x) into
query.extract_core_subject() with parameterized noise set, max_words,
and suffix stripping.

Consolidate duplicated _tokenize/_compute_relevance/STOPWORDS/SYNONYMS
(4 copies across youtube_yt, tiktok, instagram, scrapecreators_x) into
relevance.token_overlap_relevance() with hashtag-aware matching.

Integration into per-module imports follows in next commits.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling ce8e289692 Address review feedback: fix tiebreaker map, error handling, regex patterns
- Add BlueskyItem/TruthSocialItem to _ITEM_SOURCE_MAP (wrong tiebreaker)
- Add bluesky/truthsocial to _DEFAULT_TIEBREAKER
- Log HTTPError in select_openai_model instead of silent fallback
- Remove overly broad 'or.*for' from comparison regex (false positives)
- Remove bare 'will' from prediction regex (misclassifies feature queries)
- Narrow brave_search except clauses to ValueError/TypeError
- Fix stale comments: pricing table, docstrings, penalty descriptions
2026-03-11 18:32:37 -07:00
Jeffrey Sperling 588cff3e00 Optimize model selection for cost-efficiency on structured extraction
The task profile is search tool invocation + JSON extraction — not
reasoning or creative work. Mini models handle this equally well at
3-5x lower cost per call.

OpenAI changes:
- Rename is_mainline_openai_model -> is_search_capable_model
- Include mini variants (gpt-5-mini, gpt-4.1-mini) in candidate pool
- Exclude gpt-4o-mini (no domain filtering) and nano (no web_search)
- select_openai_model() now prefers mini within newest generation
- OPENAI_FALLBACK_MODELS: gpt-5-mini first, mainline as last resort
- MODEL_FALLBACK_ORDER: same mini-first ordering

xAI changes:
- Switch alias from grok-4-1-fast (reasoning) to
  grok-4-1-fast-non-reasoning — same token price, faster response,
  no wasted reasoning tokens for structured extraction

Cost per Reddit search call: ~$0.015 (gpt-5-mini) vs ~$0.044 (gpt-4.1)
2026-03-11 18:05:09 -07:00
Jeffrey Sperling 859f6c5829 Add Brave LLM Context endpoint as opt-in web search mode
Brave's /res/v1/llm/context returns pre-extracted text chunks
optimized for LLM consumption instead of URLs + short snippets.
Enable with BRAVE_LLM_CONTEXT=1 env var; same API key and pricing.

- Add _search_llm_context() and _normalize_llm_context() to brave_search.py
- Wire opt-in flag through _search_web() in last30days.py
- Update module docstring (free tier eliminated Feb 2026)
- Add 23 tests covering normalization, filtering, date parsing
2026-03-11 18:04:43 -07:00
Jeffrey Sperling 4fde52459d Filter Polymarket results to active events only
Add events_status=active and keep_closed_markets=0 to Gamma API
search params, filtering out resolved/closed markets that clutter results.
These params are confirmed in the Polymarket OpenAPI spec.
2026-03-11 18:04:43 -07:00
Jeffrey Sperling ef7c0f05dd Add query-type-aware source tiering and scoring
Detect query type (product/concept/opinion/how_to/comparison/breaking_news/
prediction) via lightweight regex patterns and use it for:

1. Source selection: each query type has tier-1 (always run) and tier-2
   (run if available) sources. Unlisted sources are opt-in only.
   Truth Social is always opt-in regardless of query type.

2. WebSearch penalty: varies by query type instead of flat -15pt.
   Concept queries get 0 penalty (web docs are authoritative),
   how_to gets 5pt, breaking_news gets 10pt, product/opinion get 15pt.

3. Tiebreaker ordering: source priority varies by query type.
   YouTube ranks first for how_to, Polymarket for prediction,
   HN for concept queries, X for breaking news.

All changes are backward-compatible: callers that don't pass query_type
get the original behavior (15pt penalty, Reddit > X > YouTube tiebreaker).
2026-03-11 18:04:43 -07:00
Jeffrey Sperling e568ef8af9 Revert MODEL_FALLBACK_ORDER to upstream values
Model optimization (mini-first fallback, is_search_capable_model) belongs
in PR #67. This PR stays focused on endpoint/API fixes only.

Also fixes pre-existing test bug where test asserted gpt-4o was first in
MODEL_FALLBACK_ORDER when it was actually gpt-4.1.
2026-03-11 18:02:33 -07:00
Jeffrey Sperling 3e9e2f632b Update stale API endpoints and model chains
- Instagram: migrate /v1/ to /v2/ ScrapeCreators endpoint (v1 deprecated Feb 2026)
- OpenAI: switch fallback chain to [gpt-5-mini, gpt-4.1-mini, gpt-4.1] (8x cheaper,
  gpt-5-mini is the first mini model supporting web_search with filters.allowed_domains)
- xAI: use explicit grok-4-1-fast-non-reasoning (bare name aliases to reasoning variant)
- xAI: pass from_date/to_date natively to x_search tool config instead of prompt-only
- Polymarket: correct rate limit comment (15K/10s, not 350/10s)
2026-03-11 16:42:49 -07:00
Jeffrey Sperling 9ca84e495e Fix stale test assertions and truthsocial pytest dependency
- test_models: update xAI model expectations to grok-4-1-fast (matching
  current XAI_POLICY_MAP)
- test_openai_reddit: update fallback order assertion to gpt-4.1 (matching
  current MODEL_FALLBACK_ORDER)
- test_codex_auth: expect 'reddit' not 'web' when no API keys (Reddit
  is available via public JSON fallback)
- test_truthsocial: convert from pytest-style classes to unittest.TestCase,
  fix import path to use sys.path.insert pattern (matching all other tests)
2026-03-11 16:35:55 -07:00