diff --git a/docs/plans/2026-03-11-refactor-query-relevance-consolidation-plan.md b/docs/plans/2026-03-11-refactor-query-relevance-consolidation-plan.md new file mode 100644 index 0000000..e82d684 --- /dev/null +++ b/docs/plans/2026-03-11-refactor-query-relevance-consolidation-plan.md @@ -0,0 +1,80 @@ +# Search Pipeline: Query & Relevance Consolidation + +## Strategy: Two-phase delivery + +**Phase 1 — Upstream PR** (`refactor/query-relevance-consolidation` -> `mvanhorn/last30days-skill:main`) +Pure refactors and bug fixes anyone would want. No behavior changes. + +**Phase 2 — Fork-only** (`feat/search-quality` on `j-sperling/last30days-skill`) +Opinionated behavior changes: computed relevance scores, platform-specific query optimization, post-retrieval filtering. + +--- + +## Phase 1: Upstream PR (refactors + bug fixes) + +### Step 1: New `query.py` — shared query utilities +- Consolidate 7 duplicated `_extract_core_subject()` (bird_x, reddit, youtube_yt, tiktok, instagram, bluesky, scrapecreators_x) into one parameterized function +- `extract_core_subject(topic, noise=None, max_words=None, strip_suffixes=False)` — platform modules pass their own noise set and options to preserve current behavior +- Shared `PREFIXES` list (identical across all 7), shared `NOISE_WORDS` base set +- Each platform imports `extract_core_subject` and calls with its own overrides (e.g. bird_x passes `max_words=5, strip_suffixes=True`; youtube keeps tips/tricks/tutorial in its noise exclusion) +- Fix reddit.py prefix-loop missing `break` (apply all matching prefixes vs only first) +- Skip polymarket.py (too different — handles "last N days", preserves title case) +- Tests: `tests/test_query.py` + +### Step 2: New `relevance.py` — shared relevance scoring +- Consolidate `_tokenize`, `_compute_relevance`, `STOPWORDS`, `SYNONYMS` from youtube_yt/tiktok/instagram +- `token_overlap_relevance(query, text, hashtags=None) -> float` — zero-dep, superset of all three implementations (hashtag substring matching from tiktok/instagram, synonym expansion from youtube) +- Unified `SYNONYMS` dict (youtube superset: includes svelte/vue entries missing from tiktok/instagram) +- Tests: `tests/test_relevance.py` (migrate from `test_youtube_relevance.py` + new hashtag tests) + +### Step 6: urllib fallback for TikTok/Instagram (independent bug fix) +- `tiktok.py`: add `http.get()`/`http.post()` fallback when `_requests is None` +- `instagram.py`: same pattern +- Copies pattern from reddit.py's existing fallback + +### Step 3: Integrate `query.py` into per-source modules (pure refactor) +- `bird_x.py`: replace lines 52-106 with import, call `extract_core_subject(topic, max_words=5, strip_suffixes=True, noise=BIRD_NOISE)` +- `reddit.py`: replace `NOISE_WORDS` + `_extract_core_subject` with query import; `expand_reddit_queries` imports from query.py too +- `youtube_yt.py`: replace `_extract_core_subject` with import, pass youtube-specific noise set (keeps tips/tricks/tutorial/guide/review) +- `tiktok.py`, `instagram.py`, `bluesky.py`: same replacement with their noise sets +- Update tests: 12+ test methods across 6 test files reference `module._extract_core_subject()` — either re-export from original modules or update test imports + +### Step 8: Deduplicate relevance code in youtube/tiktok/instagram (pure refactor) +- `youtube_yt.py`: remove `STOPWORDS`, `SYNONYMS`, `_tokenize`, `_compute_relevance`; import from `relevance.py` +- `tiktok.py`: same +- `instagram.py`: same +- Update tests: `test_youtube_relevance.py`, `test_tiktok.py`, `test_instagram_sc.py`, `test_scrapecreators_x.py` reference `module._tokenize`/`module._compute_relevance` — re-export or update imports + +### Commit order: 1 → 2 → 6 → 3 → 8 + +--- + +## Phase 2: Fork-only (behavior changes) + +### Step 4: Replace hardcoded relevance with computed scores +- `bird_x.py:471` — `"relevance": 0.7` → `token_overlap_relevance(core_topic, text)` +- `reddit.py:223` — `"relevance": 0.7` → `token_overlap_relevance(core, title + " " + selftext)` +- `hackernews.py:139-141` — blend: `0.6 * rank_score + 0.4 * token_overlap` + +### Step 5: Platform-specific query optimization +- `detect_query_type(topic)` — heuristic classifier (product/concept/opinion/how_to/comparison), added here not Phase 1 +- `extract_compound_terms(topic)` — detect hyphenated/title-case terms, return quoted +- `bird_x.py`: OR-group construction for multi-concept queries, OR-based retry before word-dropping fallback +- `reddit.py`: conditional opinion/review suffix only for product/opinion queries (uses `detect_query_type`) +- `hackernews.py`: add `numericFilters: points>5`, `restrictSearchableAttributes=title`, use `extract_core_subject()` instead of raw topic +- `youtube_yt.py`: add `--dateafter YYYYMMDD` (from_date already in signature) + +### Step 7: Post-retrieval relevance filtering in orchestrator +- `last30days.py` (after dedup): filter items with `relevance < 0.3` per source (only when list has >3 items) +- Extend fallback guarantee to all sources: keep top 3 by relevance if all filtered +- `rerank_with_embeddings()` — optional, env-var gated (`OPENAI_API_KEY` or `GOOGLE_API_KEY`), uses existing `http.py`, graceful fallback to token overlap + +### Commit order: 4 → 5 → 7 + +--- + +## Verify + +```bash +cd ~/projects/last30days-skill && python3 -m unittest discover -s tests -v +``` diff --git a/mise.toml b/mise.toml new file mode 100644 index 0000000..a190abb --- /dev/null +++ b/mise.toml @@ -0,0 +1,2 @@ +[tools] +python = "3.12"