Commit Graph
13 Commits
Author SHA1 Message Date
Jeffrey Sperling 38caae3288 Add mise.toml for Python version pinning and implementation plan
Pin Python 3.12 via mise for consistent local development.
Add plan document for the query/relevance consolidation work.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 96948cc7c0 Deduplicate relevance code across youtube/tiktok/instagram/scrapecreators_x
Replace duplicated STOPWORDS, SYNONYMS, _tokenize, and _compute_relevance
in four modules with imports from the shared relevance.py module.

Existing tests pass unchanged since modules re-export the functions
under the same names via import aliases.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling dc88c215be Integrate shared query.py into per-source modules
Replace duplicated _extract_core_subject() in bird_x, reddit, youtube_yt,
tiktok, instagram, bluesky, and scrapecreators_x with thin wrappers that
delegate to query.extract_core_subject() with platform-specific noise sets.

Each module preserves its current behavior exactly:
- bird_x: max_words=5, strip_suffixes=True, full noise set
- youtube_yt: keeps tips/tricks/tutorial/guide/review (content types)
- reddit: preserves original smaller noise set
- tiktok/instagram: same small noise set
- bluesky/scrapecreators_x: minimal noise set

Existing tests pass without modification since _extract_core_subject()
still exists as a callable on each module.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling fa42a5d031 Add urllib fallback for TikTok/Instagram when requests unavailable
Previously tiktok.py and instagram.py returned an error when the
requests library was not installed. Reddit already had an http.get()
fallback using stdlib urllib. Apply the same pattern so all three
ScrapeCreators modules work without requests installed.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling d667586597 Add shared query.py and relevance.py modules
Consolidate duplicated _extract_core_subject() (7 copies across bird_x,
reddit, youtube_yt, tiktok, instagram, bluesky, scrapecreators_x) into
query.extract_core_subject() with parameterized noise set, max_words,
and suffix stripping.

Consolidate duplicated _tokenize/_compute_relevance/STOPWORDS/SYNONYMS
(4 copies across youtube_yt, tiktok, instagram, scrapecreators_x) into
relevance.token_overlap_relevance() with hashtag-aware matching.

Integration into per-module imports follows in next commits.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling ce8e289692 Address review feedback: fix tiebreaker map, error handling, regex patterns
- Add BlueskyItem/TruthSocialItem to _ITEM_SOURCE_MAP (wrong tiebreaker)
- Add bluesky/truthsocial to _DEFAULT_TIEBREAKER
- Log HTTPError in select_openai_model instead of silent fallback
- Remove overly broad 'or.*for' from comparison regex (false positives)
- Remove bare 'will' from prediction regex (misclassifies feature queries)
- Narrow brave_search except clauses to ValueError/TypeError
- Fix stale comments: pricing table, docstrings, penalty descriptions
2026-03-11 18:32:37 -07:00
Jeffrey Sperling 588cff3e00 Optimize model selection for cost-efficiency on structured extraction
The task profile is search tool invocation + JSON extraction — not
reasoning or creative work. Mini models handle this equally well at
3-5x lower cost per call.

OpenAI changes:
- Rename is_mainline_openai_model -> is_search_capable_model
- Include mini variants (gpt-5-mini, gpt-4.1-mini) in candidate pool
- Exclude gpt-4o-mini (no domain filtering) and nano (no web_search)
- select_openai_model() now prefers mini within newest generation
- OPENAI_FALLBACK_MODELS: gpt-5-mini first, mainline as last resort
- MODEL_FALLBACK_ORDER: same mini-first ordering

xAI changes:
- Switch alias from grok-4-1-fast (reasoning) to
  grok-4-1-fast-non-reasoning — same token price, faster response,
  no wasted reasoning tokens for structured extraction

Cost per Reddit search call: ~$0.015 (gpt-5-mini) vs ~$0.044 (gpt-4.1)
2026-03-11 18:05:09 -07:00
Jeffrey Sperling 859f6c5829 Add Brave LLM Context endpoint as opt-in web search mode
Brave's /res/v1/llm/context returns pre-extracted text chunks
optimized for LLM consumption instead of URLs + short snippets.
Enable with BRAVE_LLM_CONTEXT=1 env var; same API key and pricing.

- Add _search_llm_context() and _normalize_llm_context() to brave_search.py
- Wire opt-in flag through _search_web() in last30days.py
- Update module docstring (free tier eliminated Feb 2026)
- Add 23 tests covering normalization, filtering, date parsing
2026-03-11 18:04:43 -07:00
Jeffrey Sperling 4fde52459d Filter Polymarket results to active events only
Add events_status=active and keep_closed_markets=0 to Gamma API
search params, filtering out resolved/closed markets that clutter results.
These params are confirmed in the Polymarket OpenAPI spec.
2026-03-11 18:04:43 -07:00
Jeffrey Sperling ef7c0f05dd Add query-type-aware source tiering and scoring
Detect query type (product/concept/opinion/how_to/comparison/breaking_news/
prediction) via lightweight regex patterns and use it for:

1. Source selection: each query type has tier-1 (always run) and tier-2
   (run if available) sources. Unlisted sources are opt-in only.
   Truth Social is always opt-in regardless of query type.

2. WebSearch penalty: varies by query type instead of flat -15pt.
   Concept queries get 0 penalty (web docs are authoritative),
   how_to gets 5pt, breaking_news gets 10pt, product/opinion get 15pt.

3. Tiebreaker ordering: source priority varies by query type.
   YouTube ranks first for how_to, Polymarket for prediction,
   HN for concept queries, X for breaking news.

All changes are backward-compatible: callers that don't pass query_type
get the original behavior (15pt penalty, Reddit > X > YouTube tiebreaker).
2026-03-11 18:04:43 -07:00
Jeffrey Sperling e568ef8af9 Revert MODEL_FALLBACK_ORDER to upstream values
Model optimization (mini-first fallback, is_search_capable_model) belongs
in PR #67. This PR stays focused on endpoint/API fixes only.

Also fixes pre-existing test bug where test asserted gpt-4o was first in
MODEL_FALLBACK_ORDER when it was actually gpt-4.1.
2026-03-11 18:02:33 -07:00
Jeffrey Sperling 3e9e2f632b Update stale API endpoints and model chains
- Instagram: migrate /v1/ to /v2/ ScrapeCreators endpoint (v1 deprecated Feb 2026)
- OpenAI: switch fallback chain to [gpt-5-mini, gpt-4.1-mini, gpt-4.1] (8x cheaper,
  gpt-5-mini is the first mini model supporting web_search with filters.allowed_domains)
- xAI: use explicit grok-4-1-fast-non-reasoning (bare name aliases to reasoning variant)
- xAI: pass from_date/to_date natively to x_search tool config instead of prompt-only
- Polymarket: correct rate limit comment (15K/10s, not 350/10s)
2026-03-11 16:42:49 -07:00
Jeffrey Sperling 9ca84e495e Fix stale test assertions and truthsocial pytest dependency
- test_models: update xAI model expectations to grok-4-1-fast (matching
  current XAI_POLICY_MAP)
- test_openai_reddit: update fallback order assertion to gpt-4.1 (matching
  current MODEL_FALLBACK_ORDER)
- test_codex_auth: expect 'reddit' not 'web' when no API keys (Reddit
  is available via public JSON fallback)
- test_truthsocial: convert from pytest-style classes to unittest.TestCase,
  fix import path to use sys.path.insert pattern (matching all other tests)
2026-03-11 16:35:55 -07:00