feat: Smart supplemental search — Phase 2 entity-aware drill-down
After the initial broad search (Phase 1), extract key entities from results and run targeted secondary searches to surface content the broad pass missed: - New entity_extract.py: parses @handles, #hashtags, subreddits from results - bird_x.py: search_handles() does targeted from:handle searches via Bird CLI - openai_reddit.py: search_subreddits() uses Reddit's free .json search endpoint - last30days.py: Phase 2 orchestration runs after enrichment, merges + dedupes Tested with "kanye west" (+9 Reddit, +1 X) and "claude code skills" (+6 Reddit, +1 X). Phase 2 is skipped on --quick mode. Default caps at 3 handles/subs, deep at 5. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,263 @@
|
|||||||
|
---
|
||||||
|
title: "feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X"
|
||||||
|
type: feat
|
||||||
|
date: 2026-02-07
|
||||||
|
---
|
||||||
|
|
||||||
|
# feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Add an intelligent "discover → drill down" second pass to both Reddit and X searches. After the initial broad search, extract entities (handles, subreddits, hashtags) from results and run targeted secondary searches to surface content the broad pass missed. This supplements — does not replace — the existing search pipeline.
|
||||||
|
|
||||||
|
## Problem Statement / Motivation
|
||||||
|
|
||||||
|
The current search pipeline does a single broad pass per source (with Reddit having 2 fallbacks for low-result scenarios). This works well for general topics, but misses content that lives in:
|
||||||
|
|
||||||
|
- **Niche subreddits** that don't rank for generic queries (e.g., searching "Nano Banana Pro" finds r/generativeAI but misses r/nanobanana, r/localLLaMA)
|
||||||
|
- **Key accounts on X** that are the authorities on a topic but whose individual posts don't rank for broad keyword search (e.g., @steipete for Open Claw, @karpathy for AI training)
|
||||||
|
- **Conversation threads** where the most valuable discussion happens in replies, not the original tweet
|
||||||
|
|
||||||
|
The product works great today. This is about squeezing 20-30% more high-quality results from sources we already have access to.
|
||||||
|
|
||||||
|
## Proposed Solution
|
||||||
|
|
||||||
|
### Architecture: Two-Phase Search
|
||||||
|
|
||||||
|
```
|
||||||
|
CURRENT (Phase 1 — unchanged):
|
||||||
|
Broad topic search → Reddit results + X results
|
||||||
|
↓
|
||||||
|
NEW (Phase 2 — supplemental):
|
||||||
|
Extract entities from Phase 1 results
|
||||||
|
↓ ↓
|
||||||
|
[SUBREDDITS] [@HANDLES + #HASHTAGS]
|
||||||
|
↓ ↓
|
||||||
|
Targeted Reddit Targeted X searches
|
||||||
|
searches per sub per handle/hashtag
|
||||||
|
↓ ↓
|
||||||
|
Merge + dedupe with Phase 1 results
|
||||||
|
```
|
||||||
|
|
||||||
|
Phase 2 only runs if Phase 1 returned results (entities need to come from somewhere). Phase 2 results are merged and deduped against Phase 1 — the existing `dedupe.py` handles this.
|
||||||
|
|
||||||
|
### Feature 1: Entity Extraction Module (NEW FILE)
|
||||||
|
|
||||||
|
**File: `scripts/lib/entity_extract.py`**
|
||||||
|
|
||||||
|
A lightweight module that parses Phase 1 results and extracts:
|
||||||
|
|
||||||
|
**From X results:**
|
||||||
|
- `@handles` — from `author_handle` field + any @mentions in post text
|
||||||
|
- `#hashtags` — from post text
|
||||||
|
- Rank by frequency: handles that appear 2+ times are "key voices"
|
||||||
|
|
||||||
|
**From Reddit results:**
|
||||||
|
- `subreddit` names — from the `subreddit` field on each result
|
||||||
|
- Cross-referenced subreddits — from enriched comment text mentioning "r/othersub"
|
||||||
|
- Rank by frequency: subreddits with 2+ threads are "core communities"
|
||||||
|
|
||||||
|
**Output:**
|
||||||
|
```python
|
||||||
|
{
|
||||||
|
"x_handles": ["steipete", "openclaw", "karpathy"], # ranked by frequency
|
||||||
|
"x_hashtags": ["#openclaw", "#aitools"],
|
||||||
|
"reddit_subreddits": ["generativeAI", "localLLaMA", "nanobanana"],
|
||||||
|
"reddit_cross_refs": ["singularity", "MachineLearning"], # mentioned in comments
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rules:**
|
||||||
|
- No hardcoded entities — everything discovered dynamically from Phase 1
|
||||||
|
- Cap at top 5 handles, top 3 hashtags, top 5 subreddits
|
||||||
|
- Skip generic handles (@elonmusk, @OpenAI) that appear everywhere — maintain a small exclusion list of "too common" handles (< 20 entries)
|
||||||
|
- Skip the original topic's "obvious" subreddit if it was already searched
|
||||||
|
|
||||||
|
### Feature 2: Supplemental X Search (Bird)
|
||||||
|
|
||||||
|
**File: modify `scripts/lib/bird_x.py`**
|
||||||
|
|
||||||
|
Add a `search_handles()` function:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def search_handles(handles: list[str], topic: str, from_date: str, count_per: int = 5) -> list:
|
||||||
|
"""Search top handles for topic-related content."""
|
||||||
|
results = []
|
||||||
|
for handle in handles[:5]:
|
||||||
|
# Uses Bird's support for X search operators
|
||||||
|
query = f"from:{handle} {topic} since:{from_date}"
|
||||||
|
cmd = ["bird", "search", query, "-n", str(count_per), "--json"]
|
||||||
|
# ... parse results, add to list
|
||||||
|
return results
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why Bird, not xAI:** Bird is free (uses your X login). Running 5 secondary searches via xAI would cost ~$0.025 per run, which adds up. Bird costs nothing.
|
||||||
|
|
||||||
|
**xAI alternative for users without Bird:** If Bird is not available but xAI is, use `allowed_x_handles` parameter:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# xAI supports filtering to specific handles (max 10)
|
||||||
|
tools = [{
|
||||||
|
"type": "x_search",
|
||||||
|
"x_handles": {"allowed_x_handles": top_handles[:10]}
|
||||||
|
}]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Feature 3: Supplemental Reddit Search
|
||||||
|
|
||||||
|
**File: modify `scripts/lib/openai_reddit.py`**
|
||||||
|
|
||||||
|
Add a `search_subreddits()` function:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def search_subreddits(subreddits: list[str], topic: str, ...) -> list:
|
||||||
|
"""Search discovered subreddits for topic-related content."""
|
||||||
|
# Build multi-subreddit query for the OpenAI web_search prompt
|
||||||
|
sub_query = " OR ".join(f"r/{sub}" for sub in subreddits[:5])
|
||||||
|
prompt = f"Search Reddit for threads about {topic} in these communities: {sub_query}"
|
||||||
|
# ... single OpenAI API call, same pattern as existing search
|
||||||
|
```
|
||||||
|
|
||||||
|
**Alternative approach — Reddit JSON API (free, no API key):**
|
||||||
|
|
||||||
|
```python
|
||||||
|
def search_subreddit_json(subreddit: str, topic: str) -> list:
|
||||||
|
"""Search a specific subreddit via Reddit's free JSON endpoint."""
|
||||||
|
url = f"https://www.reddit.com/r/{subreddit}/search/.json"
|
||||||
|
params = {"q": topic, "restrict_sr": "on", "sort": "new", "limit": 10}
|
||||||
|
# ... parse JSON response
|
||||||
|
```
|
||||||
|
|
||||||
|
This is free, requires no API key, and gives us structured data. The `.json` endpoint trick is well-documented and widely used.
|
||||||
|
|
||||||
|
### Feature 4: Orchestration Changes
|
||||||
|
|
||||||
|
**File: modify `scripts/last30days.py`**
|
||||||
|
|
||||||
|
After Phase 1 completes and enrichment is done, run Phase 2:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Phase 1 (existing — unchanged)
|
||||||
|
reddit_items, x_items = run_parallel_search(...)
|
||||||
|
|
||||||
|
# Phase 2 (new — supplemental)
|
||||||
|
if reddit_items or x_items:
|
||||||
|
entities = entity_extract.extract(reddit_items, x_items)
|
||||||
|
|
||||||
|
supplemental_reddit = []
|
||||||
|
supplemental_x = []
|
||||||
|
|
||||||
|
# Run supplemental searches in parallel
|
||||||
|
with ThreadPoolExecutor(max_workers=2) as executor:
|
||||||
|
if entities["reddit_subreddits"]:
|
||||||
|
reddit_future = executor.submit(
|
||||||
|
openai_reddit.search_subreddits,
|
||||||
|
entities["reddit_subreddits"], topic, ...
|
||||||
|
)
|
||||||
|
if entities["x_handles"] and bird_available:
|
||||||
|
x_future = executor.submit(
|
||||||
|
bird_x.search_handles,
|
||||||
|
entities["x_handles"], topic, from_date, ...
|
||||||
|
)
|
||||||
|
|
||||||
|
# Merge with Phase 1
|
||||||
|
all_reddit = reddit_items + supplemental_reddit
|
||||||
|
all_x = x_items + supplemental_x
|
||||||
|
|
||||||
|
# Dedupe handles the rest
|
||||||
|
```
|
||||||
|
|
||||||
|
**Depth-dependent behavior:**
|
||||||
|
| Depth | Phase 2 behavior |
|
||||||
|
|---|---|
|
||||||
|
| `--quick` | Skip Phase 2 entirely (speed matters) |
|
||||||
|
| default | Run Phase 2 with caps: 3 handles, 3 subreddits, 3 results each |
|
||||||
|
| `--deep` | Run Phase 2 with caps: 5 handles, 5 subreddits, 5 results each |
|
||||||
|
|
||||||
|
### Feature 5: Thread Expansion for High-Engagement Posts (stretch goal)
|
||||||
|
|
||||||
|
**File: modify `scripts/lib/bird_x.py`**
|
||||||
|
|
||||||
|
For X posts with very high engagement (top 1-2 by likes), expand the conversation thread:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def expand_thread(tweet_id: str) -> list:
|
||||||
|
"""Fetch full thread for a high-engagement tweet."""
|
||||||
|
cmd = ["bird", "thread", tweet_id, "--json"]
|
||||||
|
# ... parse thread, extract key replies
|
||||||
|
```
|
||||||
|
|
||||||
|
This surfaces the discussion around viral posts — often more valuable than the original tweet. Only trigger for posts with 100+ likes to avoid noise.
|
||||||
|
|
||||||
|
## Technical Considerations
|
||||||
|
|
||||||
|
### Performance
|
||||||
|
- Phase 2 adds 2-5 seconds for Bird (5 subprocess calls) and 3-8 seconds for Reddit subreddit search (1 API call)
|
||||||
|
- On `--quick` mode, Phase 2 is skipped entirely — zero performance impact
|
||||||
|
- Phase 2 runs AFTER Phase 1, not in parallel with it (needs Phase 1 results for entity extraction)
|
||||||
|
|
||||||
|
### Cost
|
||||||
|
- Reddit subreddit search: 1 additional OpenAI API call (~$0.005) OR free via `.json` endpoint
|
||||||
|
- X handle search via Bird: Free (uses your X login)
|
||||||
|
- X handle search via xAI (fallback): 1 additional API call (~$0.005)
|
||||||
|
- Thread expansion: Free via Bird
|
||||||
|
|
||||||
|
### No New Dependencies
|
||||||
|
- Entity extraction is string parsing — no NLP libraries needed
|
||||||
|
- Reddit `.json` endpoint uses existing `http.py` transport
|
||||||
|
- Bird CLI calls use existing subprocess pattern from `bird_x.py`
|
||||||
|
|
||||||
|
### Backward Compatibility
|
||||||
|
- Phase 2 is purely additive — all existing behavior unchanged
|
||||||
|
- If Phase 2 finds nothing, output is identical to current
|
||||||
|
- Deduplication handles any overlap between Phase 1 and Phase 2
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- [x] Entity extraction module correctly parses handles, hashtags, and subreddits from search results
|
||||||
|
- [x] Supplemental X searches via Bird find additional content from key handles
|
||||||
|
- [x] Supplemental Reddit searches find content in discovered subreddits
|
||||||
|
- [x] Phase 2 results are properly merged and deduped with Phase 1
|
||||||
|
- [x] `--quick` mode skips Phase 2 entirely
|
||||||
|
- [x] `--deep` mode searches more handles/subreddits with higher per-query limits
|
||||||
|
- [x] No performance regression on `--quick` mode
|
||||||
|
- [ ] Default mode adds < 10 seconds of latency
|
||||||
|
- [x] Works with Bird-only, xAI-only, and both-available configurations
|
||||||
|
- [x] Output format unchanged (Phase 2 results look identical to Phase 1 results)
|
||||||
|
|
||||||
|
## Implementation Order
|
||||||
|
|
||||||
|
1. `scripts/lib/entity_extract.py` — Entity extraction from results (new file)
|
||||||
|
2. `scripts/lib/bird_x.py` — Add `search_handles()` function
|
||||||
|
3. `scripts/lib/openai_reddit.py` — Add `search_subreddits()` function
|
||||||
|
4. `scripts/last30days.py` — Orchestration: Phase 2 after Phase 1
|
||||||
|
5. Test with real queries: "Open Claw", "Nano Banana Pro", "kanye west"
|
||||||
|
6. (Stretch) Thread expansion for high-engagement posts
|
||||||
|
|
||||||
|
## Research Sources
|
||||||
|
|
||||||
|
### Reddit Search Techniques
|
||||||
|
- [reddit-research-mcp](https://github.com/king-of-the-grackles/reddit-research-mcp) — MCP server with semantic subreddit discovery via 20K+ pre-indexed communities
|
||||||
|
- [anvaka/sayit](https://github.com/anvaka/sayit) — Subreddit similarity graph via collaborative filtering (Jaccard similarity on user overlap)
|
||||||
|
- [YARS](https://github.com/datavorous/yars) — No-API-key Reddit scraper using `.json` endpoint trick
|
||||||
|
- Reddit's free JSON search endpoint: `reddit.com/r/{sub}/search/.json?q=QUERY&restrict_sr=on` — no auth needed
|
||||||
|
- Reddit search operators: `subreddit:`, `title:`, `selftext:`, `author:`, `flair:` (Lucene-style)
|
||||||
|
|
||||||
|
### X/Twitter Search Techniques
|
||||||
|
- [igorbrigadir/twitter-advanced-search](https://github.com/igorbrigadir/twitter-advanced-search) — Canonical reference of all X search operators
|
||||||
|
- Bird CLI supports all X operators: `from:`, `to:`, `conversation_id:`, `min_retweets:`, `#hashtag`, `list:`
|
||||||
|
- xAI x_search `allowed_x_handles` parameter — filter to max 10 specific handles
|
||||||
|
- xAI x_search semantic search — finds conceptually related content without exact keyword matches
|
||||||
|
- [Bellingcat OSINT Toolkit](https://bellingcat.gitbook.io/toolkit) — Multi-pass handle discovery methodology
|
||||||
|
|
||||||
|
### Key Insight
|
||||||
|
The biggest gap in the current implementation is that **neither X nor Reddit search does entity extraction from initial results to inform follow-up queries.** Every tool/project researched that achieves better-than-basic results does some form of "discover entities → search entities" two-pass strategy.
|
||||||
|
|
||||||
|
## What We're NOT Doing
|
||||||
|
|
||||||
|
- **Not adding new API dependencies** — everything uses existing OpenAI, xAI, or Bird infrastructure
|
||||||
|
- **Not adding NLP/ML libraries** — entity extraction is simple string parsing
|
||||||
|
- **Not changing the output format** — Phase 2 results merge seamlessly
|
||||||
|
- **Not hardcoding any entities** — all discovery is dynamic from search results
|
||||||
|
- **Not slowing down `--quick` mode** — Phase 2 is skipped entirely
|
||||||
|
- **Not replacing the current search** — Phase 2 supplements Phase 1
|
||||||
@@ -30,6 +30,7 @@ from lib import (
|
|||||||
bird_x,
|
bird_x,
|
||||||
dates,
|
dates,
|
||||||
dedupe,
|
dedupe,
|
||||||
|
entity_extract,
|
||||||
env,
|
env,
|
||||||
http,
|
http,
|
||||||
models,
|
models,
|
||||||
@@ -205,6 +206,129 @@ def _search_x(
|
|||||||
return x_items, raw_response, x_error
|
return x_items, raw_response, x_error
|
||||||
|
|
||||||
|
|
||||||
|
def _run_supplemental(
|
||||||
|
topic: str,
|
||||||
|
reddit_items: list,
|
||||||
|
x_items: list,
|
||||||
|
from_date: str,
|
||||||
|
to_date: str,
|
||||||
|
depth: str,
|
||||||
|
x_source: str,
|
||||||
|
progress: ui.ProgressDisplay = None,
|
||||||
|
) -> tuple:
|
||||||
|
"""Run Phase 2 supplemental searches based on entities from Phase 1.
|
||||||
|
|
||||||
|
Extracts handles/subreddits from initial results, then runs targeted
|
||||||
|
searches to find additional content the broad search missed.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
topic: Original search topic
|
||||||
|
reddit_items: Phase 1 Reddit items (raw dicts)
|
||||||
|
x_items: Phase 1 X items (raw dicts)
|
||||||
|
from_date: Start date
|
||||||
|
to_date: End date
|
||||||
|
depth: Research depth
|
||||||
|
x_source: 'bird' or 'xai'
|
||||||
|
progress: Optional progress display
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (supplemental_reddit, supplemental_x)
|
||||||
|
"""
|
||||||
|
# Depth-dependent caps
|
||||||
|
if depth == "default":
|
||||||
|
max_handles = 3
|
||||||
|
max_subs = 3
|
||||||
|
count_per = 3
|
||||||
|
else: # deep
|
||||||
|
max_handles = 5
|
||||||
|
max_subs = 5
|
||||||
|
count_per = 5
|
||||||
|
|
||||||
|
# Extract entities from Phase 1 results
|
||||||
|
entities = entity_extract.extract_entities(
|
||||||
|
reddit_items, x_items,
|
||||||
|
max_handles=max_handles,
|
||||||
|
max_subreddits=max_subs,
|
||||||
|
)
|
||||||
|
|
||||||
|
has_handles = entities["x_handles"] and x_source == "bird"
|
||||||
|
has_subs = entities["reddit_subreddits"]
|
||||||
|
|
||||||
|
if not has_handles and not has_subs:
|
||||||
|
return [], []
|
||||||
|
|
||||||
|
parts = []
|
||||||
|
if has_handles:
|
||||||
|
parts.append(f"@{', @'.join(entities['x_handles'][:3])}")
|
||||||
|
if has_subs:
|
||||||
|
parts.append(f"r/{', r/'.join(entities['reddit_subreddits'][:3])}")
|
||||||
|
sys.stderr.write(f"[Phase 2] Drilling into {' + '.join(parts)}\n")
|
||||||
|
sys.stderr.flush()
|
||||||
|
|
||||||
|
supplemental_reddit = []
|
||||||
|
supplemental_x = []
|
||||||
|
|
||||||
|
# Collect existing URLs to avoid adding duplicates before dedupe
|
||||||
|
existing_urls = set()
|
||||||
|
for item in reddit_items:
|
||||||
|
existing_urls.add(item.get("url", ""))
|
||||||
|
for item in x_items:
|
||||||
|
existing_urls.add(item.get("url", ""))
|
||||||
|
|
||||||
|
# Run supplemental searches in parallel
|
||||||
|
reddit_future = None
|
||||||
|
x_future = None
|
||||||
|
|
||||||
|
with ThreadPoolExecutor(max_workers=2) as executor:
|
||||||
|
if has_subs:
|
||||||
|
reddit_future = executor.submit(
|
||||||
|
openai_reddit.search_subreddits,
|
||||||
|
entities["reddit_subreddits"],
|
||||||
|
topic,
|
||||||
|
from_date,
|
||||||
|
to_date,
|
||||||
|
count_per,
|
||||||
|
)
|
||||||
|
|
||||||
|
if has_handles:
|
||||||
|
x_future = executor.submit(
|
||||||
|
bird_x.search_handles,
|
||||||
|
entities["x_handles"],
|
||||||
|
topic,
|
||||||
|
from_date,
|
||||||
|
count_per,
|
||||||
|
)
|
||||||
|
|
||||||
|
if reddit_future:
|
||||||
|
try:
|
||||||
|
raw_reddit = reddit_future.result()
|
||||||
|
# Filter out URLs already found in Phase 1
|
||||||
|
supplemental_reddit = [
|
||||||
|
item for item in raw_reddit
|
||||||
|
if item.get("url", "") not in existing_urls
|
||||||
|
]
|
||||||
|
except Exception as e:
|
||||||
|
sys.stderr.write(f"[Phase 2] Supplemental Reddit error: {e}\n")
|
||||||
|
|
||||||
|
if x_future:
|
||||||
|
try:
|
||||||
|
raw_x = x_future.result()
|
||||||
|
supplemental_x = [
|
||||||
|
item for item in raw_x
|
||||||
|
if item.get("url", "") not in existing_urls
|
||||||
|
]
|
||||||
|
except Exception as e:
|
||||||
|
sys.stderr.write(f"[Phase 2] Supplemental X error: {e}\n")
|
||||||
|
|
||||||
|
if supplemental_reddit or supplemental_x:
|
||||||
|
sys.stderr.write(
|
||||||
|
f"[Phase 2] +{len(supplemental_reddit)} Reddit, +{len(supplemental_x)} X\n"
|
||||||
|
)
|
||||||
|
sys.stderr.flush()
|
||||||
|
|
||||||
|
return supplemental_reddit, supplemental_x
|
||||||
|
|
||||||
|
|
||||||
def run_research(
|
def run_research(
|
||||||
topic: str,
|
topic: str,
|
||||||
sources: str,
|
sources: str,
|
||||||
@@ -319,6 +443,18 @@ def run_research(
|
|||||||
if progress:
|
if progress:
|
||||||
progress.end_reddit_enrich()
|
progress.end_reddit_enrich()
|
||||||
|
|
||||||
|
# Phase 2: Supplemental search based on entities from Phase 1
|
||||||
|
# Skip on --quick (speed matters) and mock mode
|
||||||
|
if depth != "quick" and not mock and (reddit_items or x_items):
|
||||||
|
sup_reddit, sup_x = _run_supplemental(
|
||||||
|
topic, reddit_items, x_items,
|
||||||
|
from_date, to_date, depth, x_source, progress,
|
||||||
|
)
|
||||||
|
if sup_reddit:
|
||||||
|
reddit_items.extend(sup_reddit)
|
||||||
|
if sup_x:
|
||||||
|
x_items.extend(sup_x)
|
||||||
|
|
||||||
return reddit_items, x_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error
|
return reddit_items, x_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -190,6 +190,70 @@ def search_x(
|
|||||||
return {"error": str(e), "items": []}
|
return {"error": str(e), "items": []}
|
||||||
|
|
||||||
|
|
||||||
|
def search_handles(
|
||||||
|
handles: List[str],
|
||||||
|
topic: str,
|
||||||
|
from_date: str,
|
||||||
|
count_per: int = 5,
|
||||||
|
) -> List[Dict[str, Any]]:
|
||||||
|
"""Search specific X handles for topic-related content.
|
||||||
|
|
||||||
|
Runs targeted Bird searches using `from:handle topic` syntax.
|
||||||
|
Used in Phase 2 supplemental search after entity extraction.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
handles: List of X handles to search (without @)
|
||||||
|
topic: Search topic (core subject, not full verbose query)
|
||||||
|
from_date: Start date (YYYY-MM-DD)
|
||||||
|
count_per: Results to request per handle
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of raw item dicts (same format as parse_bird_response output).
|
||||||
|
"""
|
||||||
|
all_items = []
|
||||||
|
core_topic = _extract_core_subject(topic)
|
||||||
|
|
||||||
|
for handle in handles:
|
||||||
|
handle = handle.lstrip("@")
|
||||||
|
query = f"from:{handle} {core_topic} since:{from_date}"
|
||||||
|
|
||||||
|
cmd = [
|
||||||
|
"bird", "search",
|
||||||
|
query,
|
||||||
|
"-n", str(count_per),
|
||||||
|
"--json",
|
||||||
|
]
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
cmd,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=15, # Short timeout per handle
|
||||||
|
)
|
||||||
|
|
||||||
|
if result.returncode != 0:
|
||||||
|
_log(f"Handle search failed for @{handle}: {result.stderr.strip()}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
output = result.stdout.strip()
|
||||||
|
if not output:
|
||||||
|
continue
|
||||||
|
|
||||||
|
response = json.loads(output)
|
||||||
|
items = parse_bird_response(response)
|
||||||
|
all_items.extend(items)
|
||||||
|
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
_log(f"Handle search timed out for @{handle}")
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
_log(f"Invalid JSON from handle search for @{handle}")
|
||||||
|
except Exception as e:
|
||||||
|
_log(f"Handle search error for @{handle}: {e}")
|
||||||
|
|
||||||
|
return all_items
|
||||||
|
|
||||||
|
|
||||||
def parse_bird_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
def parse_bird_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||||
"""Parse Bird response to match xai_x output format.
|
"""Parse Bird response to match xai_x output format.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,127 @@
|
|||||||
|
"""Entity extraction from Phase 1 search results for supplemental searches."""
|
||||||
|
|
||||||
|
import re
|
||||||
|
from collections import Counter
|
||||||
|
from typing import Any, Dict, List
|
||||||
|
|
||||||
|
# Handles that appear too frequently to be useful for targeted search.
|
||||||
|
# These are generic/platform accounts, not topic-specific voices.
|
||||||
|
GENERIC_HANDLES = {
|
||||||
|
"elonmusk", "openai", "google", "microsoft", "apple", "meta",
|
||||||
|
"github", "youtube", "x", "twitter", "reddit", "wikipedia",
|
||||||
|
"nytimes", "washingtonpost", "cnn", "bbc", "reuters",
|
||||||
|
"verified", "jack", "sundarpichai",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def extract_entities(
|
||||||
|
reddit_items: List[Dict[str, Any]],
|
||||||
|
x_items: List[Dict[str, Any]],
|
||||||
|
max_handles: int = 5,
|
||||||
|
max_hashtags: int = 3,
|
||||||
|
max_subreddits: int = 5,
|
||||||
|
) -> Dict[str, List[str]]:
|
||||||
|
"""Extract key entities from Phase 1 results for supplemental searches.
|
||||||
|
|
||||||
|
Parses X results for @handles and #hashtags, Reddit results for subreddit
|
||||||
|
names and cross-referenced communities.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
reddit_items: Raw Reddit item dicts from Phase 1
|
||||||
|
x_items: Raw X item dicts from Phase 1
|
||||||
|
max_handles: Maximum handles to return
|
||||||
|
max_hashtags: Maximum hashtags to return
|
||||||
|
max_subreddits: Maximum subreddits to return
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict with keys: x_handles, x_hashtags, reddit_subreddits
|
||||||
|
"""
|
||||||
|
handles = _extract_x_handles(x_items)
|
||||||
|
hashtags = _extract_x_hashtags(x_items)
|
||||||
|
subreddits = _extract_subreddits(reddit_items)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"x_handles": handles[:max_handles],
|
||||||
|
"x_hashtags": hashtags[:max_hashtags],
|
||||||
|
"reddit_subreddits": subreddits[:max_subreddits],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_x_handles(x_items: List[Dict[str, Any]]) -> List[str]:
|
||||||
|
"""Extract and rank @handles from X results.
|
||||||
|
|
||||||
|
Sources handles from:
|
||||||
|
1. author_handle field (who posted)
|
||||||
|
2. @mentions in post text (who they're talking about/to)
|
||||||
|
|
||||||
|
Returns handles ranked by frequency, filtered for generic accounts.
|
||||||
|
"""
|
||||||
|
handle_counts = Counter()
|
||||||
|
|
||||||
|
for item in x_items:
|
||||||
|
# Author handle
|
||||||
|
author = item.get("author_handle", "").strip().lstrip("@").lower()
|
||||||
|
if author and author not in GENERIC_HANDLES:
|
||||||
|
handle_counts[author] += 1
|
||||||
|
|
||||||
|
# @mentions in text
|
||||||
|
text = item.get("text", "")
|
||||||
|
mentions = re.findall(r'@(\w{1,15})', text)
|
||||||
|
for mention in mentions:
|
||||||
|
mention_lower = mention.lower()
|
||||||
|
if mention_lower not in GENERIC_HANDLES:
|
||||||
|
handle_counts[mention_lower] += 1
|
||||||
|
|
||||||
|
# Return all handles ranked by frequency
|
||||||
|
return [h for h, _ in handle_counts.most_common()]
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_x_hashtags(x_items: List[Dict[str, Any]]) -> List[str]:
|
||||||
|
"""Extract and rank #hashtags from X results.
|
||||||
|
|
||||||
|
Returns hashtags ranked by frequency.
|
||||||
|
"""
|
||||||
|
hashtag_counts = Counter()
|
||||||
|
|
||||||
|
for item in x_items:
|
||||||
|
text = item.get("text", "")
|
||||||
|
tags = re.findall(r'#(\w{2,30})', text)
|
||||||
|
for tag in tags:
|
||||||
|
hashtag_counts[tag.lower()] += 1
|
||||||
|
|
||||||
|
# Return all hashtags ranked by frequency
|
||||||
|
return [f"#{t}" for t, _ in hashtag_counts.most_common()]
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_subreddits(reddit_items: List[Dict[str, Any]]) -> List[str]:
|
||||||
|
"""Extract and rank subreddits from Reddit results.
|
||||||
|
|
||||||
|
Sources from:
|
||||||
|
1. subreddit field on each result
|
||||||
|
2. Cross-references in comment text (e.g., "check out r/localLLaMA")
|
||||||
|
|
||||||
|
Returns subreddits ranked by frequency.
|
||||||
|
"""
|
||||||
|
sub_counts = Counter()
|
||||||
|
|
||||||
|
for item in reddit_items:
|
||||||
|
# Primary subreddit
|
||||||
|
sub = item.get("subreddit", "").strip().lstrip("r/")
|
||||||
|
if sub:
|
||||||
|
sub_counts[sub] += 1
|
||||||
|
|
||||||
|
# Cross-references in comment insights
|
||||||
|
for insight in item.get("comment_insights", []):
|
||||||
|
cross_refs = re.findall(r'r/(\w{2,30})', insight)
|
||||||
|
for ref in cross_refs:
|
||||||
|
sub_counts[ref] += 1
|
||||||
|
|
||||||
|
# Cross-references in top comments
|
||||||
|
for comment in item.get("top_comments", []):
|
||||||
|
excerpt = comment.get("excerpt", "")
|
||||||
|
cross_refs = re.findall(r'r/(\w{2,30})', excerpt)
|
||||||
|
for ref in cross_refs:
|
||||||
|
sub_counts[ref] += 1
|
||||||
|
|
||||||
|
# Return subreddits ranked by frequency
|
||||||
|
return [sub for sub, _ in sub_counts.most_common()]
|
||||||
@@ -198,6 +198,87 @@ def search_reddit(
|
|||||||
raise http.HTTPError("No models available")
|
raise http.HTTPError("No models available")
|
||||||
|
|
||||||
|
|
||||||
|
def search_subreddits(
|
||||||
|
subreddits: List[str],
|
||||||
|
topic: str,
|
||||||
|
from_date: str,
|
||||||
|
to_date: str,
|
||||||
|
count_per: int = 5,
|
||||||
|
) -> List[Dict[str, Any]]:
|
||||||
|
"""Search specific subreddits via Reddit's free JSON endpoint.
|
||||||
|
|
||||||
|
No API key needed. Uses reddit.com/r/{sub}/search/.json endpoint.
|
||||||
|
Used in Phase 2 supplemental search after entity extraction.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
subreddits: List of subreddit names (without r/)
|
||||||
|
topic: Search topic
|
||||||
|
from_date: Start date (YYYY-MM-DD)
|
||||||
|
to_date: End date (YYYY-MM-DD)
|
||||||
|
count_per: Results to request per subreddit
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of raw item dicts (same format as parse_reddit_response output).
|
||||||
|
"""
|
||||||
|
all_items = []
|
||||||
|
core = _extract_core_subject(topic)
|
||||||
|
|
||||||
|
for sub in subreddits:
|
||||||
|
sub = sub.lstrip("r/")
|
||||||
|
try:
|
||||||
|
url = f"https://www.reddit.com/r/{sub}/search/.json"
|
||||||
|
params = f"q={_url_encode(core)}&restrict_sr=on&sort=new&limit={count_per}&raw_json=1"
|
||||||
|
full_url = f"{url}?{params}"
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
"User-Agent": http.USER_AGENT,
|
||||||
|
"Accept": "application/json",
|
||||||
|
}
|
||||||
|
|
||||||
|
data = http.get(full_url, headers=headers, timeout=15)
|
||||||
|
|
||||||
|
# Reddit search returns {"data": {"children": [...]}}
|
||||||
|
children = data.get("data", {}).get("children", [])
|
||||||
|
for i, child in enumerate(children):
|
||||||
|
if child.get("kind") != "t3": # t3 = link/submission
|
||||||
|
continue
|
||||||
|
post = child.get("data", {})
|
||||||
|
permalink = post.get("permalink", "")
|
||||||
|
if not permalink:
|
||||||
|
continue
|
||||||
|
|
||||||
|
item = {
|
||||||
|
"id": f"RS{len(all_items)+1}",
|
||||||
|
"title": str(post.get("title", "")).strip(),
|
||||||
|
"url": f"https://www.reddit.com{permalink}",
|
||||||
|
"subreddit": str(post.get("subreddit", sub)).strip(),
|
||||||
|
"date": None,
|
||||||
|
"why_relevant": f"Found in r/{sub} supplemental search",
|
||||||
|
"relevance": 0.65, # Slightly lower default for supplemental
|
||||||
|
}
|
||||||
|
|
||||||
|
# Parse date from created_utc
|
||||||
|
created_utc = post.get("created_utc")
|
||||||
|
if created_utc:
|
||||||
|
from . import dates as dates_mod
|
||||||
|
item["date"] = dates_mod.timestamp_to_date(created_utc)
|
||||||
|
|
||||||
|
all_items.append(item)
|
||||||
|
|
||||||
|
except http.HTTPError as e:
|
||||||
|
_log_info(f"Subreddit search failed for r/{sub}: {e}")
|
||||||
|
except Exception as e:
|
||||||
|
_log_info(f"Subreddit search error for r/{sub}: {e}")
|
||||||
|
|
||||||
|
return all_items
|
||||||
|
|
||||||
|
|
||||||
|
def _url_encode(text: str) -> str:
|
||||||
|
"""Simple URL encoding for query parameters."""
|
||||||
|
import urllib.parse
|
||||||
|
return urllib.parse.quote_plus(text)
|
||||||
|
|
||||||
|
|
||||||
def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||||
"""Parse OpenAI response to extract Reddit items.
|
"""Parse OpenAI response to extract Reddit items.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user