feat: Smart supplemental search — Phase 2 entity-aware drill-down

After the initial broad search (Phase 1), extract key entities from results
and run targeted secondary searches to surface content the broad pass missed:

- New entity_extract.py: parses @handles, #hashtags, subreddits from results
- bird_x.py: search_handles() does targeted from:handle searches via Bird CLI
- openai_reddit.py: search_subreddits() uses Reddit's free .json search endpoint
- last30days.py: Phase 2 orchestration runs after enrichment, merges + dedupes

Tested with "kanye west" (+9 Reddit, +1 X) and "claude code skills" (+6 Reddit, +1 X).
Phase 2 is skipped on --quick mode. Default caps at 3 handles/subs, deep at 5.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
2026-02-07 10:00:28 -08:00
parent 650aa1100b
commit 1ae7a16c75
5 changed files with 671 additions and 0 deletions
@@ -0,0 +1,263 @@
---
title: "feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X"
type: feat
date: 2026-02-07
---
# feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X
## Overview
Add an intelligent "discover → drill down" second pass to both Reddit and X searches. After the initial broad search, extract entities (handles, subreddits, hashtags) from results and run targeted secondary searches to surface content the broad pass missed. This supplements — does not replace — the existing search pipeline.
## Problem Statement / Motivation
The current search pipeline does a single broad pass per source (with Reddit having 2 fallbacks for low-result scenarios). This works well for general topics, but misses content that lives in:
- **Niche subreddits** that don't rank for generic queries (e.g., searching "Nano Banana Pro" finds r/generativeAI but misses r/nanobanana, r/localLLaMA)
- **Key accounts on X** that are the authorities on a topic but whose individual posts don't rank for broad keyword search (e.g., @steipete for Open Claw, @karpathy for AI training)
- **Conversation threads** where the most valuable discussion happens in replies, not the original tweet
The product works great today. This is about squeezing 20-30% more high-quality results from sources we already have access to.
## Proposed Solution
### Architecture: Two-Phase Search
```
CURRENT (Phase 1 — unchanged):
Broad topic search → Reddit results + X results
NEW (Phase 2 — supplemental):
Extract entities from Phase 1 results
↓ ↓
[SUBREDDITS] [@HANDLES + #HASHTAGS]
↓ ↓
Targeted Reddit Targeted X searches
searches per sub per handle/hashtag
↓ ↓
Merge + dedupe with Phase 1 results
```
Phase 2 only runs if Phase 1 returned results (entities need to come from somewhere). Phase 2 results are merged and deduped against Phase 1 — the existing `dedupe.py` handles this.
### Feature 1: Entity Extraction Module (NEW FILE)
**File: `scripts/lib/entity_extract.py`**
A lightweight module that parses Phase 1 results and extracts:
**From X results:**
- `@handles` — from `author_handle` field + any @mentions in post text
- `#hashtags` — from post text
- Rank by frequency: handles that appear 2+ times are "key voices"
**From Reddit results:**
- `subreddit` names — from the `subreddit` field on each result
- Cross-referenced subreddits — from enriched comment text mentioning "r/othersub"
- Rank by frequency: subreddits with 2+ threads are "core communities"
**Output:**
```python
{
"x_handles": ["steipete", "openclaw", "karpathy"], # ranked by frequency
"x_hashtags": ["#openclaw", "#aitools"],
"reddit_subreddits": ["generativeAI", "localLLaMA", "nanobanana"],
"reddit_cross_refs": ["singularity", "MachineLearning"], # mentioned in comments
}
```
**Rules:**
- No hardcoded entities — everything discovered dynamically from Phase 1
- Cap at top 5 handles, top 3 hashtags, top 5 subreddits
- Skip generic handles (@elonmusk, @OpenAI) that appear everywhere — maintain a small exclusion list of "too common" handles (< 20 entries)
- Skip the original topic's "obvious" subreddit if it was already searched
### Feature 2: Supplemental X Search (Bird)
**File: modify `scripts/lib/bird_x.py`**
Add a `search_handles()` function:
```python
def search_handles(handles: list[str], topic: str, from_date: str, count_per: int = 5) -> list:
"""Search top handles for topic-related content."""
results = []
for handle in handles[:5]:
# Uses Bird's support for X search operators
query = f"from:{handle} {topic} since:{from_date}"
cmd = ["bird", "search", query, "-n", str(count_per), "--json"]
# ... parse results, add to list
return results
```
**Why Bird, not xAI:** Bird is free (uses your X login). Running 5 secondary searches via xAI would cost ~$0.025 per run, which adds up. Bird costs nothing.
**xAI alternative for users without Bird:** If Bird is not available but xAI is, use `allowed_x_handles` parameter:
```python
# xAI supports filtering to specific handles (max 10)
tools = [{
"type": "x_search",
"x_handles": {"allowed_x_handles": top_handles[:10]}
}]
```
### Feature 3: Supplemental Reddit Search
**File: modify `scripts/lib/openai_reddit.py`**
Add a `search_subreddits()` function:
```python
def search_subreddits(subreddits: list[str], topic: str, ...) -> list:
"""Search discovered subreddits for topic-related content."""
# Build multi-subreddit query for the OpenAI web_search prompt
sub_query = " OR ".join(f"r/{sub}" for sub in subreddits[:5])
prompt = f"Search Reddit for threads about {topic} in these communities: {sub_query}"
# ... single OpenAI API call, same pattern as existing search
```
**Alternative approach — Reddit JSON API (free, no API key):**
```python
def search_subreddit_json(subreddit: str, topic: str) -> list:
"""Search a specific subreddit via Reddit's free JSON endpoint."""
url = f"https://www.reddit.com/r/{subreddit}/search/.json"
params = {"q": topic, "restrict_sr": "on", "sort": "new", "limit": 10}
# ... parse JSON response
```
This is free, requires no API key, and gives us structured data. The `.json` endpoint trick is well-documented and widely used.
### Feature 4: Orchestration Changes
**File: modify `scripts/last30days.py`**
After Phase 1 completes and enrichment is done, run Phase 2:
```python
# Phase 1 (existing — unchanged)
reddit_items, x_items = run_parallel_search(...)
# Phase 2 (new — supplemental)
if reddit_items or x_items:
entities = entity_extract.extract(reddit_items, x_items)
supplemental_reddit = []
supplemental_x = []
# Run supplemental searches in parallel
with ThreadPoolExecutor(max_workers=2) as executor:
if entities["reddit_subreddits"]:
reddit_future = executor.submit(
openai_reddit.search_subreddits,
entities["reddit_subreddits"], topic, ...
)
if entities["x_handles"] and bird_available:
x_future = executor.submit(
bird_x.search_handles,
entities["x_handles"], topic, from_date, ...
)
# Merge with Phase 1
all_reddit = reddit_items + supplemental_reddit
all_x = x_items + supplemental_x
# Dedupe handles the rest
```
**Depth-dependent behavior:**
| Depth | Phase 2 behavior |
|---|---|
| `--quick` | Skip Phase 2 entirely (speed matters) |
| default | Run Phase 2 with caps: 3 handles, 3 subreddits, 3 results each |
| `--deep` | Run Phase 2 with caps: 5 handles, 5 subreddits, 5 results each |
### Feature 5: Thread Expansion for High-Engagement Posts (stretch goal)
**File: modify `scripts/lib/bird_x.py`**
For X posts with very high engagement (top 1-2 by likes), expand the conversation thread:
```python
def expand_thread(tweet_id: str) -> list:
"""Fetch full thread for a high-engagement tweet."""
cmd = ["bird", "thread", tweet_id, "--json"]
# ... parse thread, extract key replies
```
This surfaces the discussion around viral posts — often more valuable than the original tweet. Only trigger for posts with 100+ likes to avoid noise.
## Technical Considerations
### Performance
- Phase 2 adds 2-5 seconds for Bird (5 subprocess calls) and 3-8 seconds for Reddit subreddit search (1 API call)
- On `--quick` mode, Phase 2 is skipped entirely — zero performance impact
- Phase 2 runs AFTER Phase 1, not in parallel with it (needs Phase 1 results for entity extraction)
### Cost
- Reddit subreddit search: 1 additional OpenAI API call (~$0.005) OR free via `.json` endpoint
- X handle search via Bird: Free (uses your X login)
- X handle search via xAI (fallback): 1 additional API call (~$0.005)
- Thread expansion: Free via Bird
### No New Dependencies
- Entity extraction is string parsing — no NLP libraries needed
- Reddit `.json` endpoint uses existing `http.py` transport
- Bird CLI calls use existing subprocess pattern from `bird_x.py`
### Backward Compatibility
- Phase 2 is purely additive — all existing behavior unchanged
- If Phase 2 finds nothing, output is identical to current
- Deduplication handles any overlap between Phase 1 and Phase 2
## Acceptance Criteria
- [x] Entity extraction module correctly parses handles, hashtags, and subreddits from search results
- [x] Supplemental X searches via Bird find additional content from key handles
- [x] Supplemental Reddit searches find content in discovered subreddits
- [x] Phase 2 results are properly merged and deduped with Phase 1
- [x] `--quick` mode skips Phase 2 entirely
- [x] `--deep` mode searches more handles/subreddits with higher per-query limits
- [x] No performance regression on `--quick` mode
- [ ] Default mode adds < 10 seconds of latency
- [x] Works with Bird-only, xAI-only, and both-available configurations
- [x] Output format unchanged (Phase 2 results look identical to Phase 1 results)
## Implementation Order
1. `scripts/lib/entity_extract.py` — Entity extraction from results (new file)
2. `scripts/lib/bird_x.py` — Add `search_handles()` function
3. `scripts/lib/openai_reddit.py` — Add `search_subreddits()` function
4. `scripts/last30days.py` — Orchestration: Phase 2 after Phase 1
5. Test with real queries: "Open Claw", "Nano Banana Pro", "kanye west"
6. (Stretch) Thread expansion for high-engagement posts
## Research Sources
### Reddit Search Techniques
- [reddit-research-mcp](https://github.com/king-of-the-grackles/reddit-research-mcp) — MCP server with semantic subreddit discovery via 20K+ pre-indexed communities
- [anvaka/sayit](https://github.com/anvaka/sayit) — Subreddit similarity graph via collaborative filtering (Jaccard similarity on user overlap)
- [YARS](https://github.com/datavorous/yars) — No-API-key Reddit scraper using `.json` endpoint trick
- Reddit's free JSON search endpoint: `reddit.com/r/{sub}/search/.json?q=QUERY&restrict_sr=on` — no auth needed
- Reddit search operators: `subreddit:`, `title:`, `selftext:`, `author:`, `flair:` (Lucene-style)
### X/Twitter Search Techniques
- [igorbrigadir/twitter-advanced-search](https://github.com/igorbrigadir/twitter-advanced-search) — Canonical reference of all X search operators
- Bird CLI supports all X operators: `from:`, `to:`, `conversation_id:`, `min_retweets:`, `#hashtag`, `list:`
- xAI x_search `allowed_x_handles` parameter — filter to max 10 specific handles
- xAI x_search semantic search — finds conceptually related content without exact keyword matches
- [Bellingcat OSINT Toolkit](https://bellingcat.gitbook.io/toolkit) — Multi-pass handle discovery methodology
### Key Insight
The biggest gap in the current implementation is that **neither X nor Reddit search does entity extraction from initial results to inform follow-up queries.** Every tool/project researched that achieves better-than-basic results does some form of "discover entities → search entities" two-pass strategy.
## What We're NOT Doing
- **Not adding new API dependencies** — everything uses existing OpenAI, xAI, or Bird infrastructure
- **Not adding NLP/ML libraries** — entity extraction is simple string parsing
- **Not changing the output format** — Phase 2 results merge seamlessly
- **Not hardcoding any entities** — all discovery is dynamic from search results
- **Not slowing down `--quick` mode** — Phase 2 is skipped entirely
- **Not replacing the current search** — Phase 2 supplements Phase 1
+136
View File
@@ -30,6 +30,7 @@ from lib import (
bird_x,
dates,
dedupe,
entity_extract,
env,
http,
models,
@@ -205,6 +206,129 @@ def _search_x(
return x_items, raw_response, x_error
def _run_supplemental(
topic: str,
reddit_items: list,
x_items: list,
from_date: str,
to_date: str,
depth: str,
x_source: str,
progress: ui.ProgressDisplay = None,
) -> tuple:
"""Run Phase 2 supplemental searches based on entities from Phase 1.
Extracts handles/subreddits from initial results, then runs targeted
searches to find additional content the broad search missed.
Args:
topic: Original search topic
reddit_items: Phase 1 Reddit items (raw dicts)
x_items: Phase 1 X items (raw dicts)
from_date: Start date
to_date: End date
depth: Research depth
x_source: 'bird' or 'xai'
progress: Optional progress display
Returns:
Tuple of (supplemental_reddit, supplemental_x)
"""
# Depth-dependent caps
if depth == "default":
max_handles = 3
max_subs = 3
count_per = 3
else: # deep
max_handles = 5
max_subs = 5
count_per = 5
# Extract entities from Phase 1 results
entities = entity_extract.extract_entities(
reddit_items, x_items,
max_handles=max_handles,
max_subreddits=max_subs,
)
has_handles = entities["x_handles"] and x_source == "bird"
has_subs = entities["reddit_subreddits"]
if not has_handles and not has_subs:
return [], []
parts = []
if has_handles:
parts.append(f"@{', @'.join(entities['x_handles'][:3])}")
if has_subs:
parts.append(f"r/{', r/'.join(entities['reddit_subreddits'][:3])}")
sys.stderr.write(f"[Phase 2] Drilling into {' + '.join(parts)}\n")
sys.stderr.flush()
supplemental_reddit = []
supplemental_x = []
# Collect existing URLs to avoid adding duplicates before dedupe
existing_urls = set()
for item in reddit_items:
existing_urls.add(item.get("url", ""))
for item in x_items:
existing_urls.add(item.get("url", ""))
# Run supplemental searches in parallel
reddit_future = None
x_future = None
with ThreadPoolExecutor(max_workers=2) as executor:
if has_subs:
reddit_future = executor.submit(
openai_reddit.search_subreddits,
entities["reddit_subreddits"],
topic,
from_date,
to_date,
count_per,
)
if has_handles:
x_future = executor.submit(
bird_x.search_handles,
entities["x_handles"],
topic,
from_date,
count_per,
)
if reddit_future:
try:
raw_reddit = reddit_future.result()
# Filter out URLs already found in Phase 1
supplemental_reddit = [
item for item in raw_reddit
if item.get("url", "") not in existing_urls
]
except Exception as e:
sys.stderr.write(f"[Phase 2] Supplemental Reddit error: {e}\n")
if x_future:
try:
raw_x = x_future.result()
supplemental_x = [
item for item in raw_x
if item.get("url", "") not in existing_urls
]
except Exception as e:
sys.stderr.write(f"[Phase 2] Supplemental X error: {e}\n")
if supplemental_reddit or supplemental_x:
sys.stderr.write(
f"[Phase 2] +{len(supplemental_reddit)} Reddit, +{len(supplemental_x)} X\n"
)
sys.stderr.flush()
return supplemental_reddit, supplemental_x
def run_research(
topic: str,
sources: str,
@@ -319,6 +443,18 @@ def run_research(
if progress:
progress.end_reddit_enrich()
# Phase 2: Supplemental search based on entities from Phase 1
# Skip on --quick (speed matters) and mock mode
if depth != "quick" and not mock and (reddit_items or x_items):
sup_reddit, sup_x = _run_supplemental(
topic, reddit_items, x_items,
from_date, to_date, depth, x_source, progress,
)
if sup_reddit:
reddit_items.extend(sup_reddit)
if sup_x:
x_items.extend(sup_x)
return reddit_items, x_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error
+64
View File
@@ -190,6 +190,70 @@ def search_x(
return {"error": str(e), "items": []}
def search_handles(
handles: List[str],
topic: str,
from_date: str,
count_per: int = 5,
) -> List[Dict[str, Any]]:
"""Search specific X handles for topic-related content.
Runs targeted Bird searches using `from:handle topic` syntax.
Used in Phase 2 supplemental search after entity extraction.
Args:
handles: List of X handles to search (without @)
topic: Search topic (core subject, not full verbose query)
from_date: Start date (YYYY-MM-DD)
count_per: Results to request per handle
Returns:
List of raw item dicts (same format as parse_bird_response output).
"""
all_items = []
core_topic = _extract_core_subject(topic)
for handle in handles:
handle = handle.lstrip("@")
query = f"from:{handle} {core_topic} since:{from_date}"
cmd = [
"bird", "search",
query,
"-n", str(count_per),
"--json",
]
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=15, # Short timeout per handle
)
if result.returncode != 0:
_log(f"Handle search failed for @{handle}: {result.stderr.strip()}")
continue
output = result.stdout.strip()
if not output:
continue
response = json.loads(output)
items = parse_bird_response(response)
all_items.extend(items)
except subprocess.TimeoutExpired:
_log(f"Handle search timed out for @{handle}")
except json.JSONDecodeError:
_log(f"Invalid JSON from handle search for @{handle}")
except Exception as e:
_log(f"Handle search error for @{handle}: {e}")
return all_items
def parse_bird_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Bird response to match xai_x output format.
+127
View File
@@ -0,0 +1,127 @@
"""Entity extraction from Phase 1 search results for supplemental searches."""
import re
from collections import Counter
from typing import Any, Dict, List
# Handles that appear too frequently to be useful for targeted search.
# These are generic/platform accounts, not topic-specific voices.
GENERIC_HANDLES = {
"elonmusk", "openai", "google", "microsoft", "apple", "meta",
"github", "youtube", "x", "twitter", "reddit", "wikipedia",
"nytimes", "washingtonpost", "cnn", "bbc", "reuters",
"verified", "jack", "sundarpichai",
}
def extract_entities(
reddit_items: List[Dict[str, Any]],
x_items: List[Dict[str, Any]],
max_handles: int = 5,
max_hashtags: int = 3,
max_subreddits: int = 5,
) -> Dict[str, List[str]]:
"""Extract key entities from Phase 1 results for supplemental searches.
Parses X results for @handles and #hashtags, Reddit results for subreddit
names and cross-referenced communities.
Args:
reddit_items: Raw Reddit item dicts from Phase 1
x_items: Raw X item dicts from Phase 1
max_handles: Maximum handles to return
max_hashtags: Maximum hashtags to return
max_subreddits: Maximum subreddits to return
Returns:
Dict with keys: x_handles, x_hashtags, reddit_subreddits
"""
handles = _extract_x_handles(x_items)
hashtags = _extract_x_hashtags(x_items)
subreddits = _extract_subreddits(reddit_items)
return {
"x_handles": handles[:max_handles],
"x_hashtags": hashtags[:max_hashtags],
"reddit_subreddits": subreddits[:max_subreddits],
}
def _extract_x_handles(x_items: List[Dict[str, Any]]) -> List[str]:
"""Extract and rank @handles from X results.
Sources handles from:
1. author_handle field (who posted)
2. @mentions in post text (who they're talking about/to)
Returns handles ranked by frequency, filtered for generic accounts.
"""
handle_counts = Counter()
for item in x_items:
# Author handle
author = item.get("author_handle", "").strip().lstrip("@").lower()
if author and author not in GENERIC_HANDLES:
handle_counts[author] += 1
# @mentions in text
text = item.get("text", "")
mentions = re.findall(r'@(\w{1,15})', text)
for mention in mentions:
mention_lower = mention.lower()
if mention_lower not in GENERIC_HANDLES:
handle_counts[mention_lower] += 1
# Return all handles ranked by frequency
return [h for h, _ in handle_counts.most_common()]
def _extract_x_hashtags(x_items: List[Dict[str, Any]]) -> List[str]:
"""Extract and rank #hashtags from X results.
Returns hashtags ranked by frequency.
"""
hashtag_counts = Counter()
for item in x_items:
text = item.get("text", "")
tags = re.findall(r'#(\w{2,30})', text)
for tag in tags:
hashtag_counts[tag.lower()] += 1
# Return all hashtags ranked by frequency
return [f"#{t}" for t, _ in hashtag_counts.most_common()]
def _extract_subreddits(reddit_items: List[Dict[str, Any]]) -> List[str]:
"""Extract and rank subreddits from Reddit results.
Sources from:
1. subreddit field on each result
2. Cross-references in comment text (e.g., "check out r/localLLaMA")
Returns subreddits ranked by frequency.
"""
sub_counts = Counter()
for item in reddit_items:
# Primary subreddit
sub = item.get("subreddit", "").strip().lstrip("r/")
if sub:
sub_counts[sub] += 1
# Cross-references in comment insights
for insight in item.get("comment_insights", []):
cross_refs = re.findall(r'r/(\w{2,30})', insight)
for ref in cross_refs:
sub_counts[ref] += 1
# Cross-references in top comments
for comment in item.get("top_comments", []):
excerpt = comment.get("excerpt", "")
cross_refs = re.findall(r'r/(\w{2,30})', excerpt)
for ref in cross_refs:
sub_counts[ref] += 1
# Return subreddits ranked by frequency
return [sub for sub, _ in sub_counts.most_common()]
+81
View File
@@ -198,6 +198,87 @@ def search_reddit(
raise http.HTTPError("No models available")
def search_subreddits(
subreddits: List[str],
topic: str,
from_date: str,
to_date: str,
count_per: int = 5,
) -> List[Dict[str, Any]]:
"""Search specific subreddits via Reddit's free JSON endpoint.
No API key needed. Uses reddit.com/r/{sub}/search/.json endpoint.
Used in Phase 2 supplemental search after entity extraction.
Args:
subreddits: List of subreddit names (without r/)
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
count_per: Results to request per subreddit
Returns:
List of raw item dicts (same format as parse_reddit_response output).
"""
all_items = []
core = _extract_core_subject(topic)
for sub in subreddits:
sub = sub.lstrip("r/")
try:
url = f"https://www.reddit.com/r/{sub}/search/.json"
params = f"q={_url_encode(core)}&restrict_sr=on&sort=new&limit={count_per}&raw_json=1"
full_url = f"{url}?{params}"
headers = {
"User-Agent": http.USER_AGENT,
"Accept": "application/json",
}
data = http.get(full_url, headers=headers, timeout=15)
# Reddit search returns {"data": {"children": [...]}}
children = data.get("data", {}).get("children", [])
for i, child in enumerate(children):
if child.get("kind") != "t3": # t3 = link/submission
continue
post = child.get("data", {})
permalink = post.get("permalink", "")
if not permalink:
continue
item = {
"id": f"RS{len(all_items)+1}",
"title": str(post.get("title", "")).strip(),
"url": f"https://www.reddit.com{permalink}",
"subreddit": str(post.get("subreddit", sub)).strip(),
"date": None,
"why_relevant": f"Found in r/{sub} supplemental search",
"relevance": 0.65, # Slightly lower default for supplemental
}
# Parse date from created_utc
created_utc = post.get("created_utc")
if created_utc:
from . import dates as dates_mod
item["date"] = dates_mod.timestamp_to_date(created_utc)
all_items.append(item)
except http.HTTPError as e:
_log_info(f"Subreddit search failed for r/{sub}: {e}")
except Exception as e:
_log_info(f"Subreddit search error for r/{sub}: {e}")
return all_items
def _url_encode(text: str) -> str:
"""Simple URL encoding for query parameters."""
import urllib.parse
return urllib.parse.quote_plus(text)
def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse OpenAI response to extract Reddit items.