diff --git a/docs/plans/2026-02-07-feat-smart-supplemental-search-plan.md b/docs/plans/2026-02-07-feat-smart-supplemental-search-plan.md new file mode 100644 index 0000000..b49117b --- /dev/null +++ b/docs/plans/2026-02-07-feat-smart-supplemental-search-plan.md @@ -0,0 +1,263 @@ +--- +title: "feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X" +type: feat +date: 2026-02-07 +--- + +# feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X + +## Overview + +Add an intelligent "discover → drill down" second pass to both Reddit and X searches. After the initial broad search, extract entities (handles, subreddits, hashtags) from results and run targeted secondary searches to surface content the broad pass missed. This supplements — does not replace — the existing search pipeline. + +## Problem Statement / Motivation + +The current search pipeline does a single broad pass per source (with Reddit having 2 fallbacks for low-result scenarios). This works well for general topics, but misses content that lives in: + +- **Niche subreddits** that don't rank for generic queries (e.g., searching "Nano Banana Pro" finds r/generativeAI but misses r/nanobanana, r/localLLaMA) +- **Key accounts on X** that are the authorities on a topic but whose individual posts don't rank for broad keyword search (e.g., @steipete for Open Claw, @karpathy for AI training) +- **Conversation threads** where the most valuable discussion happens in replies, not the original tweet + +The product works great today. This is about squeezing 20-30% more high-quality results from sources we already have access to. + +## Proposed Solution + +### Architecture: Two-Phase Search + +``` +CURRENT (Phase 1 — unchanged): + Broad topic search → Reddit results + X results + ↓ +NEW (Phase 2 — supplemental): + Extract entities from Phase 1 results + ↓ ↓ + [SUBREDDITS] [@HANDLES + #HASHTAGS] + ↓ ↓ + Targeted Reddit Targeted X searches + searches per sub per handle/hashtag + ↓ ↓ + Merge + dedupe with Phase 1 results +``` + +Phase 2 only runs if Phase 1 returned results (entities need to come from somewhere). Phase 2 results are merged and deduped against Phase 1 — the existing `dedupe.py` handles this. + +### Feature 1: Entity Extraction Module (NEW FILE) + +**File: `scripts/lib/entity_extract.py`** + +A lightweight module that parses Phase 1 results and extracts: + +**From X results:** +- `@handles` — from `author_handle` field + any @mentions in post text +- `#hashtags` — from post text +- Rank by frequency: handles that appear 2+ times are "key voices" + +**From Reddit results:** +- `subreddit` names — from the `subreddit` field on each result +- Cross-referenced subreddits — from enriched comment text mentioning "r/othersub" +- Rank by frequency: subreddits with 2+ threads are "core communities" + +**Output:** +```python +{ + "x_handles": ["steipete", "openclaw", "karpathy"], # ranked by frequency + "x_hashtags": ["#openclaw", "#aitools"], + "reddit_subreddits": ["generativeAI", "localLLaMA", "nanobanana"], + "reddit_cross_refs": ["singularity", "MachineLearning"], # mentioned in comments +} +``` + +**Rules:** +- No hardcoded entities — everything discovered dynamically from Phase 1 +- Cap at top 5 handles, top 3 hashtags, top 5 subreddits +- Skip generic handles (@elonmusk, @OpenAI) that appear everywhere — maintain a small exclusion list of "too common" handles (< 20 entries) +- Skip the original topic's "obvious" subreddit if it was already searched + +### Feature 2: Supplemental X Search (Bird) + +**File: modify `scripts/lib/bird_x.py`** + +Add a `search_handles()` function: + +```python +def search_handles(handles: list[str], topic: str, from_date: str, count_per: int = 5) -> list: + """Search top handles for topic-related content.""" + results = [] + for handle in handles[:5]: + # Uses Bird's support for X search operators + query = f"from:{handle} {topic} since:{from_date}" + cmd = ["bird", "search", query, "-n", str(count_per), "--json"] + # ... parse results, add to list + return results +``` + +**Why Bird, not xAI:** Bird is free (uses your X login). Running 5 secondary searches via xAI would cost ~$0.025 per run, which adds up. Bird costs nothing. + +**xAI alternative for users without Bird:** If Bird is not available but xAI is, use `allowed_x_handles` parameter: + +```python +# xAI supports filtering to specific handles (max 10) +tools = [{ + "type": "x_search", + "x_handles": {"allowed_x_handles": top_handles[:10]} +}] +``` + +### Feature 3: Supplemental Reddit Search + +**File: modify `scripts/lib/openai_reddit.py`** + +Add a `search_subreddits()` function: + +```python +def search_subreddits(subreddits: list[str], topic: str, ...) -> list: + """Search discovered subreddits for topic-related content.""" + # Build multi-subreddit query for the OpenAI web_search prompt + sub_query = " OR ".join(f"r/{sub}" for sub in subreddits[:5]) + prompt = f"Search Reddit for threads about {topic} in these communities: {sub_query}" + # ... single OpenAI API call, same pattern as existing search +``` + +**Alternative approach — Reddit JSON API (free, no API key):** + +```python +def search_subreddit_json(subreddit: str, topic: str) -> list: + """Search a specific subreddit via Reddit's free JSON endpoint.""" + url = f"https://www.reddit.com/r/{subreddit}/search/.json" + params = {"q": topic, "restrict_sr": "on", "sort": "new", "limit": 10} + # ... parse JSON response +``` + +This is free, requires no API key, and gives us structured data. The `.json` endpoint trick is well-documented and widely used. + +### Feature 4: Orchestration Changes + +**File: modify `scripts/last30days.py`** + +After Phase 1 completes and enrichment is done, run Phase 2: + +```python +# Phase 1 (existing — unchanged) +reddit_items, x_items = run_parallel_search(...) + +# Phase 2 (new — supplemental) +if reddit_items or x_items: + entities = entity_extract.extract(reddit_items, x_items) + + supplemental_reddit = [] + supplemental_x = [] + + # Run supplemental searches in parallel + with ThreadPoolExecutor(max_workers=2) as executor: + if entities["reddit_subreddits"]: + reddit_future = executor.submit( + openai_reddit.search_subreddits, + entities["reddit_subreddits"], topic, ... + ) + if entities["x_handles"] and bird_available: + x_future = executor.submit( + bird_x.search_handles, + entities["x_handles"], topic, from_date, ... + ) + + # Merge with Phase 1 + all_reddit = reddit_items + supplemental_reddit + all_x = x_items + supplemental_x + + # Dedupe handles the rest +``` + +**Depth-dependent behavior:** +| Depth | Phase 2 behavior | +|---|---| +| `--quick` | Skip Phase 2 entirely (speed matters) | +| default | Run Phase 2 with caps: 3 handles, 3 subreddits, 3 results each | +| `--deep` | Run Phase 2 with caps: 5 handles, 5 subreddits, 5 results each | + +### Feature 5: Thread Expansion for High-Engagement Posts (stretch goal) + +**File: modify `scripts/lib/bird_x.py`** + +For X posts with very high engagement (top 1-2 by likes), expand the conversation thread: + +```python +def expand_thread(tweet_id: str) -> list: + """Fetch full thread for a high-engagement tweet.""" + cmd = ["bird", "thread", tweet_id, "--json"] + # ... parse thread, extract key replies +``` + +This surfaces the discussion around viral posts — often more valuable than the original tweet. Only trigger for posts with 100+ likes to avoid noise. + +## Technical Considerations + +### Performance +- Phase 2 adds 2-5 seconds for Bird (5 subprocess calls) and 3-8 seconds for Reddit subreddit search (1 API call) +- On `--quick` mode, Phase 2 is skipped entirely — zero performance impact +- Phase 2 runs AFTER Phase 1, not in parallel with it (needs Phase 1 results for entity extraction) + +### Cost +- Reddit subreddit search: 1 additional OpenAI API call (~$0.005) OR free via `.json` endpoint +- X handle search via Bird: Free (uses your X login) +- X handle search via xAI (fallback): 1 additional API call (~$0.005) +- Thread expansion: Free via Bird + +### No New Dependencies +- Entity extraction is string parsing — no NLP libraries needed +- Reddit `.json` endpoint uses existing `http.py` transport +- Bird CLI calls use existing subprocess pattern from `bird_x.py` + +### Backward Compatibility +- Phase 2 is purely additive — all existing behavior unchanged +- If Phase 2 finds nothing, output is identical to current +- Deduplication handles any overlap between Phase 1 and Phase 2 + +## Acceptance Criteria + +- [x] Entity extraction module correctly parses handles, hashtags, and subreddits from search results +- [x] Supplemental X searches via Bird find additional content from key handles +- [x] Supplemental Reddit searches find content in discovered subreddits +- [x] Phase 2 results are properly merged and deduped with Phase 1 +- [x] `--quick` mode skips Phase 2 entirely +- [x] `--deep` mode searches more handles/subreddits with higher per-query limits +- [x] No performance regression on `--quick` mode +- [ ] Default mode adds < 10 seconds of latency +- [x] Works with Bird-only, xAI-only, and both-available configurations +- [x] Output format unchanged (Phase 2 results look identical to Phase 1 results) + +## Implementation Order + +1. `scripts/lib/entity_extract.py` — Entity extraction from results (new file) +2. `scripts/lib/bird_x.py` — Add `search_handles()` function +3. `scripts/lib/openai_reddit.py` — Add `search_subreddits()` function +4. `scripts/last30days.py` — Orchestration: Phase 2 after Phase 1 +5. Test with real queries: "Open Claw", "Nano Banana Pro", "kanye west" +6. (Stretch) Thread expansion for high-engagement posts + +## Research Sources + +### Reddit Search Techniques +- [reddit-research-mcp](https://github.com/king-of-the-grackles/reddit-research-mcp) — MCP server with semantic subreddit discovery via 20K+ pre-indexed communities +- [anvaka/sayit](https://github.com/anvaka/sayit) — Subreddit similarity graph via collaborative filtering (Jaccard similarity on user overlap) +- [YARS](https://github.com/datavorous/yars) — No-API-key Reddit scraper using `.json` endpoint trick +- Reddit's free JSON search endpoint: `reddit.com/r/{sub}/search/.json?q=QUERY&restrict_sr=on` — no auth needed +- Reddit search operators: `subreddit:`, `title:`, `selftext:`, `author:`, `flair:` (Lucene-style) + +### X/Twitter Search Techniques +- [igorbrigadir/twitter-advanced-search](https://github.com/igorbrigadir/twitter-advanced-search) — Canonical reference of all X search operators +- Bird CLI supports all X operators: `from:`, `to:`, `conversation_id:`, `min_retweets:`, `#hashtag`, `list:` +- xAI x_search `allowed_x_handles` parameter — filter to max 10 specific handles +- xAI x_search semantic search — finds conceptually related content without exact keyword matches +- [Bellingcat OSINT Toolkit](https://bellingcat.gitbook.io/toolkit) — Multi-pass handle discovery methodology + +### Key Insight +The biggest gap in the current implementation is that **neither X nor Reddit search does entity extraction from initial results to inform follow-up queries.** Every tool/project researched that achieves better-than-basic results does some form of "discover entities → search entities" two-pass strategy. + +## What We're NOT Doing + +- **Not adding new API dependencies** — everything uses existing OpenAI, xAI, or Bird infrastructure +- **Not adding NLP/ML libraries** — entity extraction is simple string parsing +- **Not changing the output format** — Phase 2 results merge seamlessly +- **Not hardcoding any entities** — all discovery is dynamic from search results +- **Not slowing down `--quick` mode** — Phase 2 is skipped entirely +- **Not replacing the current search** — Phase 2 supplements Phase 1 diff --git a/scripts/last30days.py b/scripts/last30days.py index 2066a79..94fb0cd 100644 --- a/scripts/last30days.py +++ b/scripts/last30days.py @@ -30,6 +30,7 @@ from lib import ( bird_x, dates, dedupe, + entity_extract, env, http, models, @@ -205,6 +206,129 @@ def _search_x( return x_items, raw_response, x_error +def _run_supplemental( + topic: str, + reddit_items: list, + x_items: list, + from_date: str, + to_date: str, + depth: str, + x_source: str, + progress: ui.ProgressDisplay = None, +) -> tuple: + """Run Phase 2 supplemental searches based on entities from Phase 1. + + Extracts handles/subreddits from initial results, then runs targeted + searches to find additional content the broad search missed. + + Args: + topic: Original search topic + reddit_items: Phase 1 Reddit items (raw dicts) + x_items: Phase 1 X items (raw dicts) + from_date: Start date + to_date: End date + depth: Research depth + x_source: 'bird' or 'xai' + progress: Optional progress display + + Returns: + Tuple of (supplemental_reddit, supplemental_x) + """ + # Depth-dependent caps + if depth == "default": + max_handles = 3 + max_subs = 3 + count_per = 3 + else: # deep + max_handles = 5 + max_subs = 5 + count_per = 5 + + # Extract entities from Phase 1 results + entities = entity_extract.extract_entities( + reddit_items, x_items, + max_handles=max_handles, + max_subreddits=max_subs, + ) + + has_handles = entities["x_handles"] and x_source == "bird" + has_subs = entities["reddit_subreddits"] + + if not has_handles and not has_subs: + return [], [] + + parts = [] + if has_handles: + parts.append(f"@{', @'.join(entities['x_handles'][:3])}") + if has_subs: + parts.append(f"r/{', r/'.join(entities['reddit_subreddits'][:3])}") + sys.stderr.write(f"[Phase 2] Drilling into {' + '.join(parts)}\n") + sys.stderr.flush() + + supplemental_reddit = [] + supplemental_x = [] + + # Collect existing URLs to avoid adding duplicates before dedupe + existing_urls = set() + for item in reddit_items: + existing_urls.add(item.get("url", "")) + for item in x_items: + existing_urls.add(item.get("url", "")) + + # Run supplemental searches in parallel + reddit_future = None + x_future = None + + with ThreadPoolExecutor(max_workers=2) as executor: + if has_subs: + reddit_future = executor.submit( + openai_reddit.search_subreddits, + entities["reddit_subreddits"], + topic, + from_date, + to_date, + count_per, + ) + + if has_handles: + x_future = executor.submit( + bird_x.search_handles, + entities["x_handles"], + topic, + from_date, + count_per, + ) + + if reddit_future: + try: + raw_reddit = reddit_future.result() + # Filter out URLs already found in Phase 1 + supplemental_reddit = [ + item for item in raw_reddit + if item.get("url", "") not in existing_urls + ] + except Exception as e: + sys.stderr.write(f"[Phase 2] Supplemental Reddit error: {e}\n") + + if x_future: + try: + raw_x = x_future.result() + supplemental_x = [ + item for item in raw_x + if item.get("url", "") not in existing_urls + ] + except Exception as e: + sys.stderr.write(f"[Phase 2] Supplemental X error: {e}\n") + + if supplemental_reddit or supplemental_x: + sys.stderr.write( + f"[Phase 2] +{len(supplemental_reddit)} Reddit, +{len(supplemental_x)} X\n" + ) + sys.stderr.flush() + + return supplemental_reddit, supplemental_x + + def run_research( topic: str, sources: str, @@ -319,6 +443,18 @@ def run_research( if progress: progress.end_reddit_enrich() + # Phase 2: Supplemental search based on entities from Phase 1 + # Skip on --quick (speed matters) and mock mode + if depth != "quick" and not mock and (reddit_items or x_items): + sup_reddit, sup_x = _run_supplemental( + topic, reddit_items, x_items, + from_date, to_date, depth, x_source, progress, + ) + if sup_reddit: + reddit_items.extend(sup_reddit) + if sup_x: + x_items.extend(sup_x) + return reddit_items, x_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error diff --git a/scripts/lib/bird_x.py b/scripts/lib/bird_x.py index e9e1982..70ac1e8 100644 --- a/scripts/lib/bird_x.py +++ b/scripts/lib/bird_x.py @@ -190,6 +190,70 @@ def search_x( return {"error": str(e), "items": []} +def search_handles( + handles: List[str], + topic: str, + from_date: str, + count_per: int = 5, +) -> List[Dict[str, Any]]: + """Search specific X handles for topic-related content. + + Runs targeted Bird searches using `from:handle topic` syntax. + Used in Phase 2 supplemental search after entity extraction. + + Args: + handles: List of X handles to search (without @) + topic: Search topic (core subject, not full verbose query) + from_date: Start date (YYYY-MM-DD) + count_per: Results to request per handle + + Returns: + List of raw item dicts (same format as parse_bird_response output). + """ + all_items = [] + core_topic = _extract_core_subject(topic) + + for handle in handles: + handle = handle.lstrip("@") + query = f"from:{handle} {core_topic} since:{from_date}" + + cmd = [ + "bird", "search", + query, + "-n", str(count_per), + "--json", + ] + + try: + result = subprocess.run( + cmd, + capture_output=True, + text=True, + timeout=15, # Short timeout per handle + ) + + if result.returncode != 0: + _log(f"Handle search failed for @{handle}: {result.stderr.strip()}") + continue + + output = result.stdout.strip() + if not output: + continue + + response = json.loads(output) + items = parse_bird_response(response) + all_items.extend(items) + + except subprocess.TimeoutExpired: + _log(f"Handle search timed out for @{handle}") + except json.JSONDecodeError: + _log(f"Invalid JSON from handle search for @{handle}") + except Exception as e: + _log(f"Handle search error for @{handle}: {e}") + + return all_items + + def parse_bird_response(response: Dict[str, Any]) -> List[Dict[str, Any]]: """Parse Bird response to match xai_x output format. diff --git a/scripts/lib/entity_extract.py b/scripts/lib/entity_extract.py new file mode 100644 index 0000000..2ab6531 --- /dev/null +++ b/scripts/lib/entity_extract.py @@ -0,0 +1,127 @@ +"""Entity extraction from Phase 1 search results for supplemental searches.""" + +import re +from collections import Counter +from typing import Any, Dict, List + +# Handles that appear too frequently to be useful for targeted search. +# These are generic/platform accounts, not topic-specific voices. +GENERIC_HANDLES = { + "elonmusk", "openai", "google", "microsoft", "apple", "meta", + "github", "youtube", "x", "twitter", "reddit", "wikipedia", + "nytimes", "washingtonpost", "cnn", "bbc", "reuters", + "verified", "jack", "sundarpichai", +} + + +def extract_entities( + reddit_items: List[Dict[str, Any]], + x_items: List[Dict[str, Any]], + max_handles: int = 5, + max_hashtags: int = 3, + max_subreddits: int = 5, +) -> Dict[str, List[str]]: + """Extract key entities from Phase 1 results for supplemental searches. + + Parses X results for @handles and #hashtags, Reddit results for subreddit + names and cross-referenced communities. + + Args: + reddit_items: Raw Reddit item dicts from Phase 1 + x_items: Raw X item dicts from Phase 1 + max_handles: Maximum handles to return + max_hashtags: Maximum hashtags to return + max_subreddits: Maximum subreddits to return + + Returns: + Dict with keys: x_handles, x_hashtags, reddit_subreddits + """ + handles = _extract_x_handles(x_items) + hashtags = _extract_x_hashtags(x_items) + subreddits = _extract_subreddits(reddit_items) + + return { + "x_handles": handles[:max_handles], + "x_hashtags": hashtags[:max_hashtags], + "reddit_subreddits": subreddits[:max_subreddits], + } + + +def _extract_x_handles(x_items: List[Dict[str, Any]]) -> List[str]: + """Extract and rank @handles from X results. + + Sources handles from: + 1. author_handle field (who posted) + 2. @mentions in post text (who they're talking about/to) + + Returns handles ranked by frequency, filtered for generic accounts. + """ + handle_counts = Counter() + + for item in x_items: + # Author handle + author = item.get("author_handle", "").strip().lstrip("@").lower() + if author and author not in GENERIC_HANDLES: + handle_counts[author] += 1 + + # @mentions in text + text = item.get("text", "") + mentions = re.findall(r'@(\w{1,15})', text) + for mention in mentions: + mention_lower = mention.lower() + if mention_lower not in GENERIC_HANDLES: + handle_counts[mention_lower] += 1 + + # Return all handles ranked by frequency + return [h for h, _ in handle_counts.most_common()] + + +def _extract_x_hashtags(x_items: List[Dict[str, Any]]) -> List[str]: + """Extract and rank #hashtags from X results. + + Returns hashtags ranked by frequency. + """ + hashtag_counts = Counter() + + for item in x_items: + text = item.get("text", "") + tags = re.findall(r'#(\w{2,30})', text) + for tag in tags: + hashtag_counts[tag.lower()] += 1 + + # Return all hashtags ranked by frequency + return [f"#{t}" for t, _ in hashtag_counts.most_common()] + + +def _extract_subreddits(reddit_items: List[Dict[str, Any]]) -> List[str]: + """Extract and rank subreddits from Reddit results. + + Sources from: + 1. subreddit field on each result + 2. Cross-references in comment text (e.g., "check out r/localLLaMA") + + Returns subreddits ranked by frequency. + """ + sub_counts = Counter() + + for item in reddit_items: + # Primary subreddit + sub = item.get("subreddit", "").strip().lstrip("r/") + if sub: + sub_counts[sub] += 1 + + # Cross-references in comment insights + for insight in item.get("comment_insights", []): + cross_refs = re.findall(r'r/(\w{2,30})', insight) + for ref in cross_refs: + sub_counts[ref] += 1 + + # Cross-references in top comments + for comment in item.get("top_comments", []): + excerpt = comment.get("excerpt", "") + cross_refs = re.findall(r'r/(\w{2,30})', excerpt) + for ref in cross_refs: + sub_counts[ref] += 1 + + # Return subreddits ranked by frequency + return [sub for sub, _ in sub_counts.most_common()] diff --git a/scripts/lib/openai_reddit.py b/scripts/lib/openai_reddit.py index 8067a30..811bacd 100644 --- a/scripts/lib/openai_reddit.py +++ b/scripts/lib/openai_reddit.py @@ -198,6 +198,87 @@ def search_reddit( raise http.HTTPError("No models available") +def search_subreddits( + subreddits: List[str], + topic: str, + from_date: str, + to_date: str, + count_per: int = 5, +) -> List[Dict[str, Any]]: + """Search specific subreddits via Reddit's free JSON endpoint. + + No API key needed. Uses reddit.com/r/{sub}/search/.json endpoint. + Used in Phase 2 supplemental search after entity extraction. + + Args: + subreddits: List of subreddit names (without r/) + topic: Search topic + from_date: Start date (YYYY-MM-DD) + to_date: End date (YYYY-MM-DD) + count_per: Results to request per subreddit + + Returns: + List of raw item dicts (same format as parse_reddit_response output). + """ + all_items = [] + core = _extract_core_subject(topic) + + for sub in subreddits: + sub = sub.lstrip("r/") + try: + url = f"https://www.reddit.com/r/{sub}/search/.json" + params = f"q={_url_encode(core)}&restrict_sr=on&sort=new&limit={count_per}&raw_json=1" + full_url = f"{url}?{params}" + + headers = { + "User-Agent": http.USER_AGENT, + "Accept": "application/json", + } + + data = http.get(full_url, headers=headers, timeout=15) + + # Reddit search returns {"data": {"children": [...]}} + children = data.get("data", {}).get("children", []) + for i, child in enumerate(children): + if child.get("kind") != "t3": # t3 = link/submission + continue + post = child.get("data", {}) + permalink = post.get("permalink", "") + if not permalink: + continue + + item = { + "id": f"RS{len(all_items)+1}", + "title": str(post.get("title", "")).strip(), + "url": f"https://www.reddit.com{permalink}", + "subreddit": str(post.get("subreddit", sub)).strip(), + "date": None, + "why_relevant": f"Found in r/{sub} supplemental search", + "relevance": 0.65, # Slightly lower default for supplemental + } + + # Parse date from created_utc + created_utc = post.get("created_utc") + if created_utc: + from . import dates as dates_mod + item["date"] = dates_mod.timestamp_to_date(created_utc) + + all_items.append(item) + + except http.HTTPError as e: + _log_info(f"Subreddit search failed for r/{sub}: {e}") + except Exception as e: + _log_info(f"Subreddit search error for r/{sub}: {e}") + + return all_items + + +def _url_encode(text: str) -> str: + """Simple URL encoding for query parameters.""" + import urllib.parse + return urllib.parse.quote_plus(text) + + def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]: """Parse OpenAI response to extract Reddit items.