Commit Graph
21 Commits
Author SHA1 Message Date
Jeffrey Sperling 946af84f9a Tighten relevance scoring and Polymarket ranking
Score against original user intent on Reddit, remove the artificial low-end relevance floor, and make Polymarket semantics dominate generic market quality signals.

Also apply the relevance filter to Polymarket and update the affected cross-source tests.

Validation: uv run python -m unittest
2026-03-13 19:21:25 -07:00
Jeffrey Sperling cbee987f65 Extract relevance_filter, add Bluesky/TruthSocial type hint + test coverage
- Extract _relevance_filter from last30days.py closure to score.relevance_filter()
  for testability
- Add BlueskyItem/TruthSocialItem to sort_items() type hint (was missing despite
  being in _ITEM_SOURCE_MAP)
- Add tests: Bluesky/TruthSocial engagement scoring, sort_items mixed sources,
  relevance_filter behavior (threshold, minimum-result guarantee, missing attr),
  select_openai_model HTTP 401/403 error paths
2026-03-11 19:09:06 -07:00
Jeffrey Sperling d8d2b97716 Gitignore mise.toml instead of removing it
Dev environment tool config is useful locally but shouldn't be tracked.
2026-03-11 19:01:33 -07:00
Jeffrey Sperling 036bcd2ae3 Address review feedback: deduplicate query_type, clean unused imports, fix defaults
- Remove duplicate detect_query_type from query.py (divergent 5-type version);
  canonical 7-type version lives in query_type.py
- Fix reddit.py import to use query_type.detect_query_type
- Clean unused STOPWORDS/SYNONYMS/tokenize imports from youtube_yt, instagram,
  tiktok, scrapecreators_x, bird_x after relevance consolidation
- Fix _relevance_filter default from 0.7 to 0.0 (items without relevance
  should not silently pass the filter)
- Remove --dateafter from yt-dlp (returns 0 results for evergreen topics)
- Remove restrictSearchableAttributes from HN search (misses Ask/Show HN)
- Lower HN points filter from >5 to >2 (avoids filtering niche posts)
- Add error logging to select_openai_model HTTP failures
- Remove mise.toml and internal planning doc from repo
- Update module docstrings to describe current purpose, not migration history
- Update tests to import from canonical relevance module
2026-03-11 18:40:07 -07:00
Jeffrey Sperling 6c402f66b7 Update plan to reflect single upstream PR strategy 2026-03-11 18:32:45 -07:00
Jeffrey Sperling 046795c4ae Add post-retrieval relevance filtering across all sources
Filter items with relevance < 0.3 per source after dedup, but only
when list has >3 items. Extends the Reddit-only minimum-result
guarantee to all sources: keeps top 3 by relevance if all filtered.

This works with the computed relevance scores from the previous commit
to actually remove off-topic results from the final report.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 1002f1f020 Add platform-specific query optimizations
- hackernews: use extract_core_subject instead of raw topic, add
  points>5 filter and restrictSearchableAttributes=title to reduce
  noise from URL-match and low-signal posts
- youtube: add --dateafter parameter to yt-dlp for server-side date
  filtering (Python soft filter still handles fallback)
- reddit: skip opinion/review query variant for how_to/comparison
  queries where it adds noise
- bird_x: add OR-group retry with compound terms before falling back
  to word-dropping (uses X OR operator for multi-concept queries)
- query.py: add detect_query_type() and extract_compound_terms()
2026-03-11 18:32:45 -07:00
Jeffrey Sperling c5be117701 Replace hardcoded 0.7 relevance with computed token-overlap scores
- bird_x: parse_bird_response now accepts query param and computes
  token_overlap_relevance against tweet text
- reddit: _normalize_post computes relevance from query vs title+selftext
- hackernews: blends 60% Algolia rank + 40% token overlap + engagement

This makes the 45%-weight relevance factor in score.py actually
differentiate results instead of being a constant.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 38caae3288 Add mise.toml for Python version pinning and implementation plan
Pin Python 3.12 via mise for consistent local development.
Add plan document for the query/relevance consolidation work.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling 96948cc7c0 Deduplicate relevance code across youtube/tiktok/instagram/scrapecreators_x
Replace duplicated STOPWORDS, SYNONYMS, _tokenize, and _compute_relevance
in four modules with imports from the shared relevance.py module.

Existing tests pass unchanged since modules re-export the functions
under the same names via import aliases.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling dc88c215be Integrate shared query.py into per-source modules
Replace duplicated _extract_core_subject() in bird_x, reddit, youtube_yt,
tiktok, instagram, bluesky, and scrapecreators_x with thin wrappers that
delegate to query.extract_core_subject() with platform-specific noise sets.

Each module preserves its current behavior exactly:
- bird_x: max_words=5, strip_suffixes=True, full noise set
- youtube_yt: keeps tips/tricks/tutorial/guide/review (content types)
- reddit: preserves original smaller noise set
- tiktok/instagram: same small noise set
- bluesky/scrapecreators_x: minimal noise set

Existing tests pass without modification since _extract_core_subject()
still exists as a callable on each module.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling fa42a5d031 Add urllib fallback for TikTok/Instagram when requests unavailable
Previously tiktok.py and instagram.py returned an error when the
requests library was not installed. Reddit already had an http.get()
fallback using stdlib urllib. Apply the same pattern so all three
ScrapeCreators modules work without requests installed.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling d667586597 Add shared query.py and relevance.py modules
Consolidate duplicated _extract_core_subject() (7 copies across bird_x,
reddit, youtube_yt, tiktok, instagram, bluesky, scrapecreators_x) into
query.extract_core_subject() with parameterized noise set, max_words,
and suffix stripping.

Consolidate duplicated _tokenize/_compute_relevance/STOPWORDS/SYNONYMS
(4 copies across youtube_yt, tiktok, instagram, scrapecreators_x) into
relevance.token_overlap_relevance() with hashtag-aware matching.

Integration into per-module imports follows in next commits.
2026-03-11 18:32:45 -07:00
Jeffrey Sperling ce8e289692 Address review feedback: fix tiebreaker map, error handling, regex patterns
- Add BlueskyItem/TruthSocialItem to _ITEM_SOURCE_MAP (wrong tiebreaker)
- Add bluesky/truthsocial to _DEFAULT_TIEBREAKER
- Log HTTPError in select_openai_model instead of silent fallback
- Remove overly broad 'or.*for' from comparison regex (false positives)
- Remove bare 'will' from prediction regex (misclassifies feature queries)
- Narrow brave_search except clauses to ValueError/TypeError
- Fix stale comments: pricing table, docstrings, penalty descriptions
2026-03-11 18:32:37 -07:00
Jeffrey Sperling 588cff3e00 Optimize model selection for cost-efficiency on structured extraction
The task profile is search tool invocation + JSON extraction — not
reasoning or creative work. Mini models handle this equally well at
3-5x lower cost per call.

OpenAI changes:
- Rename is_mainline_openai_model -> is_search_capable_model
- Include mini variants (gpt-5-mini, gpt-4.1-mini) in candidate pool
- Exclude gpt-4o-mini (no domain filtering) and nano (no web_search)
- select_openai_model() now prefers mini within newest generation
- OPENAI_FALLBACK_MODELS: gpt-5-mini first, mainline as last resort
- MODEL_FALLBACK_ORDER: same mini-first ordering

xAI changes:
- Switch alias from grok-4-1-fast (reasoning) to
  grok-4-1-fast-non-reasoning — same token price, faster response,
  no wasted reasoning tokens for structured extraction

Cost per Reddit search call: ~$0.015 (gpt-5-mini) vs ~$0.044 (gpt-4.1)
2026-03-11 18:05:09 -07:00
Jeffrey Sperling 859f6c5829 Add Brave LLM Context endpoint as opt-in web search mode
Brave's /res/v1/llm/context returns pre-extracted text chunks
optimized for LLM consumption instead of URLs + short snippets.
Enable with BRAVE_LLM_CONTEXT=1 env var; same API key and pricing.

- Add _search_llm_context() and _normalize_llm_context() to brave_search.py
- Wire opt-in flag through _search_web() in last30days.py
- Update module docstring (free tier eliminated Feb 2026)
- Add 23 tests covering normalization, filtering, date parsing
2026-03-11 18:04:43 -07:00
Jeffrey Sperling 4fde52459d Filter Polymarket results to active events only
Add events_status=active and keep_closed_markets=0 to Gamma API
search params, filtering out resolved/closed markets that clutter results.
These params are confirmed in the Polymarket OpenAPI spec.
2026-03-11 18:04:43 -07:00
Jeffrey Sperling ef7c0f05dd Add query-type-aware source tiering and scoring
Detect query type (product/concept/opinion/how_to/comparison/breaking_news/
prediction) via lightweight regex patterns and use it for:

1. Source selection: each query type has tier-1 (always run) and tier-2
   (run if available) sources. Unlisted sources are opt-in only.
   Truth Social is always opt-in regardless of query type.

2. WebSearch penalty: varies by query type instead of flat -15pt.
   Concept queries get 0 penalty (web docs are authoritative),
   how_to gets 5pt, breaking_news gets 10pt, product/opinion get 15pt.

3. Tiebreaker ordering: source priority varies by query type.
   YouTube ranks first for how_to, Polymarket for prediction,
   HN for concept queries, X for breaking news.

All changes are backward-compatible: callers that don't pass query_type
get the original behavior (15pt penalty, Reddit > X > YouTube tiebreaker).
2026-03-11 18:04:43 -07:00
Jeffrey Sperling e568ef8af9 Revert MODEL_FALLBACK_ORDER to upstream values
Model optimization (mini-first fallback, is_search_capable_model) belongs
in PR #67. This PR stays focused on endpoint/API fixes only.

Also fixes pre-existing test bug where test asserted gpt-4o was first in
MODEL_FALLBACK_ORDER when it was actually gpt-4.1.
2026-03-11 18:02:33 -07:00
Jeffrey Sperling 3e9e2f632b Update stale API endpoints and model chains
- Instagram: migrate /v1/ to /v2/ ScrapeCreators endpoint (v1 deprecated Feb 2026)
- OpenAI: switch fallback chain to [gpt-5-mini, gpt-4.1-mini, gpt-4.1] (8x cheaper,
  gpt-5-mini is the first mini model supporting web_search with filters.allowed_domains)
- xAI: use explicit grok-4-1-fast-non-reasoning (bare name aliases to reasoning variant)
- xAI: pass from_date/to_date natively to x_search tool config instead of prompt-only
- Polymarket: correct rate limit comment (15K/10s, not 350/10s)
2026-03-11 16:42:49 -07:00
Jeffrey Sperling 9ca84e495e Fix stale test assertions and truthsocial pytest dependency
- test_models: update xAI model expectations to grok-4-1-fast (matching
  current XAI_POLICY_MAP)
- test_openai_reddit: update fallback order assertion to gpt-4.1 (matching
  current MODEL_FALLBACK_ORDER)
- test_codex_auth: expect 'reddit' not 'web' when no API keys (Reddit
  is available via public JSON fallback)
- test_truthsocial: convert from pytest-style classes to unittest.TestCase,
  fix import path to use sys.path.insert pattern (matching all other tests)
2026-03-11 16:35:55 -07:00