feat: Smart supplemental search — Phase 2 entity-aware drill-down
After the initial broad search (Phase 1), extract key entities from results and run targeted secondary searches to surface content the broad pass missed: - New entity_extract.py: parses @handles, #hashtags, subreddits from results - bird_x.py: search_handles() does targeted from:handle searches via Bird CLI - openai_reddit.py: search_subreddits() uses Reddit's free .json search endpoint - last30days.py: Phase 2 orchestration runs after enrichment, merges + dedupes Tested with "kanye west" (+9 Reddit, +1 X) and "claude code skills" (+6 Reddit, +1 X). Phase 2 is skipped on --quick mode. Default caps at 3 handles/subs, deep at 5. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,263 @@
|
||||
---
|
||||
title: "feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X"
|
||||
type: feat
|
||||
date: 2026-02-07
|
||||
---
|
||||
|
||||
# feat: Smart Supplemental Search — Entity-Aware Secondary Passes for Reddit & X
|
||||
|
||||
## Overview
|
||||
|
||||
Add an intelligent "discover → drill down" second pass to both Reddit and X searches. After the initial broad search, extract entities (handles, subreddits, hashtags) from results and run targeted secondary searches to surface content the broad pass missed. This supplements — does not replace — the existing search pipeline.
|
||||
|
||||
## Problem Statement / Motivation
|
||||
|
||||
The current search pipeline does a single broad pass per source (with Reddit having 2 fallbacks for low-result scenarios). This works well for general topics, but misses content that lives in:
|
||||
|
||||
- **Niche subreddits** that don't rank for generic queries (e.g., searching "Nano Banana Pro" finds r/generativeAI but misses r/nanobanana, r/localLLaMA)
|
||||
- **Key accounts on X** that are the authorities on a topic but whose individual posts don't rank for broad keyword search (e.g., @steipete for Open Claw, @karpathy for AI training)
|
||||
- **Conversation threads** where the most valuable discussion happens in replies, not the original tweet
|
||||
|
||||
The product works great today. This is about squeezing 20-30% more high-quality results from sources we already have access to.
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
### Architecture: Two-Phase Search
|
||||
|
||||
```
|
||||
CURRENT (Phase 1 — unchanged):
|
||||
Broad topic search → Reddit results + X results
|
||||
↓
|
||||
NEW (Phase 2 — supplemental):
|
||||
Extract entities from Phase 1 results
|
||||
↓ ↓
|
||||
[SUBREDDITS] [@HANDLES + #HASHTAGS]
|
||||
↓ ↓
|
||||
Targeted Reddit Targeted X searches
|
||||
searches per sub per handle/hashtag
|
||||
↓ ↓
|
||||
Merge + dedupe with Phase 1 results
|
||||
```
|
||||
|
||||
Phase 2 only runs if Phase 1 returned results (entities need to come from somewhere). Phase 2 results are merged and deduped against Phase 1 — the existing `dedupe.py` handles this.
|
||||
|
||||
### Feature 1: Entity Extraction Module (NEW FILE)
|
||||
|
||||
**File: `scripts/lib/entity_extract.py`**
|
||||
|
||||
A lightweight module that parses Phase 1 results and extracts:
|
||||
|
||||
**From X results:**
|
||||
- `@handles` — from `author_handle` field + any @mentions in post text
|
||||
- `#hashtags` — from post text
|
||||
- Rank by frequency: handles that appear 2+ times are "key voices"
|
||||
|
||||
**From Reddit results:**
|
||||
- `subreddit` names — from the `subreddit` field on each result
|
||||
- Cross-referenced subreddits — from enriched comment text mentioning "r/othersub"
|
||||
- Rank by frequency: subreddits with 2+ threads are "core communities"
|
||||
|
||||
**Output:**
|
||||
```python
|
||||
{
|
||||
"x_handles": ["steipete", "openclaw", "karpathy"], # ranked by frequency
|
||||
"x_hashtags": ["#openclaw", "#aitools"],
|
||||
"reddit_subreddits": ["generativeAI", "localLLaMA", "nanobanana"],
|
||||
"reddit_cross_refs": ["singularity", "MachineLearning"], # mentioned in comments
|
||||
}
|
||||
```
|
||||
|
||||
**Rules:**
|
||||
- No hardcoded entities — everything discovered dynamically from Phase 1
|
||||
- Cap at top 5 handles, top 3 hashtags, top 5 subreddits
|
||||
- Skip generic handles (@elonmusk, @OpenAI) that appear everywhere — maintain a small exclusion list of "too common" handles (< 20 entries)
|
||||
- Skip the original topic's "obvious" subreddit if it was already searched
|
||||
|
||||
### Feature 2: Supplemental X Search (Bird)
|
||||
|
||||
**File: modify `scripts/lib/bird_x.py`**
|
||||
|
||||
Add a `search_handles()` function:
|
||||
|
||||
```python
|
||||
def search_handles(handles: list[str], topic: str, from_date: str, count_per: int = 5) -> list:
|
||||
"""Search top handles for topic-related content."""
|
||||
results = []
|
||||
for handle in handles[:5]:
|
||||
# Uses Bird's support for X search operators
|
||||
query = f"from:{handle} {topic} since:{from_date}"
|
||||
cmd = ["bird", "search", query, "-n", str(count_per), "--json"]
|
||||
# ... parse results, add to list
|
||||
return results
|
||||
```
|
||||
|
||||
**Why Bird, not xAI:** Bird is free (uses your X login). Running 5 secondary searches via xAI would cost ~$0.025 per run, which adds up. Bird costs nothing.
|
||||
|
||||
**xAI alternative for users without Bird:** If Bird is not available but xAI is, use `allowed_x_handles` parameter:
|
||||
|
||||
```python
|
||||
# xAI supports filtering to specific handles (max 10)
|
||||
tools = [{
|
||||
"type": "x_search",
|
||||
"x_handles": {"allowed_x_handles": top_handles[:10]}
|
||||
}]
|
||||
```
|
||||
|
||||
### Feature 3: Supplemental Reddit Search
|
||||
|
||||
**File: modify `scripts/lib/openai_reddit.py`**
|
||||
|
||||
Add a `search_subreddits()` function:
|
||||
|
||||
```python
|
||||
def search_subreddits(subreddits: list[str], topic: str, ...) -> list:
|
||||
"""Search discovered subreddits for topic-related content."""
|
||||
# Build multi-subreddit query for the OpenAI web_search prompt
|
||||
sub_query = " OR ".join(f"r/{sub}" for sub in subreddits[:5])
|
||||
prompt = f"Search Reddit for threads about {topic} in these communities: {sub_query}"
|
||||
# ... single OpenAI API call, same pattern as existing search
|
||||
```
|
||||
|
||||
**Alternative approach — Reddit JSON API (free, no API key):**
|
||||
|
||||
```python
|
||||
def search_subreddit_json(subreddit: str, topic: str) -> list:
|
||||
"""Search a specific subreddit via Reddit's free JSON endpoint."""
|
||||
url = f"https://www.reddit.com/r/{subreddit}/search/.json"
|
||||
params = {"q": topic, "restrict_sr": "on", "sort": "new", "limit": 10}
|
||||
# ... parse JSON response
|
||||
```
|
||||
|
||||
This is free, requires no API key, and gives us structured data. The `.json` endpoint trick is well-documented and widely used.
|
||||
|
||||
### Feature 4: Orchestration Changes
|
||||
|
||||
**File: modify `scripts/last30days.py`**
|
||||
|
||||
After Phase 1 completes and enrichment is done, run Phase 2:
|
||||
|
||||
```python
|
||||
# Phase 1 (existing — unchanged)
|
||||
reddit_items, x_items = run_parallel_search(...)
|
||||
|
||||
# Phase 2 (new — supplemental)
|
||||
if reddit_items or x_items:
|
||||
entities = entity_extract.extract(reddit_items, x_items)
|
||||
|
||||
supplemental_reddit = []
|
||||
supplemental_x = []
|
||||
|
||||
# Run supplemental searches in parallel
|
||||
with ThreadPoolExecutor(max_workers=2) as executor:
|
||||
if entities["reddit_subreddits"]:
|
||||
reddit_future = executor.submit(
|
||||
openai_reddit.search_subreddits,
|
||||
entities["reddit_subreddits"], topic, ...
|
||||
)
|
||||
if entities["x_handles"] and bird_available:
|
||||
x_future = executor.submit(
|
||||
bird_x.search_handles,
|
||||
entities["x_handles"], topic, from_date, ...
|
||||
)
|
||||
|
||||
# Merge with Phase 1
|
||||
all_reddit = reddit_items + supplemental_reddit
|
||||
all_x = x_items + supplemental_x
|
||||
|
||||
# Dedupe handles the rest
|
||||
```
|
||||
|
||||
**Depth-dependent behavior:**
|
||||
| Depth | Phase 2 behavior |
|
||||
|---|---|
|
||||
| `--quick` | Skip Phase 2 entirely (speed matters) |
|
||||
| default | Run Phase 2 with caps: 3 handles, 3 subreddits, 3 results each |
|
||||
| `--deep` | Run Phase 2 with caps: 5 handles, 5 subreddits, 5 results each |
|
||||
|
||||
### Feature 5: Thread Expansion for High-Engagement Posts (stretch goal)
|
||||
|
||||
**File: modify `scripts/lib/bird_x.py`**
|
||||
|
||||
For X posts with very high engagement (top 1-2 by likes), expand the conversation thread:
|
||||
|
||||
```python
|
||||
def expand_thread(tweet_id: str) -> list:
|
||||
"""Fetch full thread for a high-engagement tweet."""
|
||||
cmd = ["bird", "thread", tweet_id, "--json"]
|
||||
# ... parse thread, extract key replies
|
||||
```
|
||||
|
||||
This surfaces the discussion around viral posts — often more valuable than the original tweet. Only trigger for posts with 100+ likes to avoid noise.
|
||||
|
||||
## Technical Considerations
|
||||
|
||||
### Performance
|
||||
- Phase 2 adds 2-5 seconds for Bird (5 subprocess calls) and 3-8 seconds for Reddit subreddit search (1 API call)
|
||||
- On `--quick` mode, Phase 2 is skipped entirely — zero performance impact
|
||||
- Phase 2 runs AFTER Phase 1, not in parallel with it (needs Phase 1 results for entity extraction)
|
||||
|
||||
### Cost
|
||||
- Reddit subreddit search: 1 additional OpenAI API call (~$0.005) OR free via `.json` endpoint
|
||||
- X handle search via Bird: Free (uses your X login)
|
||||
- X handle search via xAI (fallback): 1 additional API call (~$0.005)
|
||||
- Thread expansion: Free via Bird
|
||||
|
||||
### No New Dependencies
|
||||
- Entity extraction is string parsing — no NLP libraries needed
|
||||
- Reddit `.json` endpoint uses existing `http.py` transport
|
||||
- Bird CLI calls use existing subprocess pattern from `bird_x.py`
|
||||
|
||||
### Backward Compatibility
|
||||
- Phase 2 is purely additive — all existing behavior unchanged
|
||||
- If Phase 2 finds nothing, output is identical to current
|
||||
- Deduplication handles any overlap between Phase 1 and Phase 2
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] Entity extraction module correctly parses handles, hashtags, and subreddits from search results
|
||||
- [x] Supplemental X searches via Bird find additional content from key handles
|
||||
- [x] Supplemental Reddit searches find content in discovered subreddits
|
||||
- [x] Phase 2 results are properly merged and deduped with Phase 1
|
||||
- [x] `--quick` mode skips Phase 2 entirely
|
||||
- [x] `--deep` mode searches more handles/subreddits with higher per-query limits
|
||||
- [x] No performance regression on `--quick` mode
|
||||
- [ ] Default mode adds < 10 seconds of latency
|
||||
- [x] Works with Bird-only, xAI-only, and both-available configurations
|
||||
- [x] Output format unchanged (Phase 2 results look identical to Phase 1 results)
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. `scripts/lib/entity_extract.py` — Entity extraction from results (new file)
|
||||
2. `scripts/lib/bird_x.py` — Add `search_handles()` function
|
||||
3. `scripts/lib/openai_reddit.py` — Add `search_subreddits()` function
|
||||
4. `scripts/last30days.py` — Orchestration: Phase 2 after Phase 1
|
||||
5. Test with real queries: "Open Claw", "Nano Banana Pro", "kanye west"
|
||||
6. (Stretch) Thread expansion for high-engagement posts
|
||||
|
||||
## Research Sources
|
||||
|
||||
### Reddit Search Techniques
|
||||
- [reddit-research-mcp](https://github.com/king-of-the-grackles/reddit-research-mcp) — MCP server with semantic subreddit discovery via 20K+ pre-indexed communities
|
||||
- [anvaka/sayit](https://github.com/anvaka/sayit) — Subreddit similarity graph via collaborative filtering (Jaccard similarity on user overlap)
|
||||
- [YARS](https://github.com/datavorous/yars) — No-API-key Reddit scraper using `.json` endpoint trick
|
||||
- Reddit's free JSON search endpoint: `reddit.com/r/{sub}/search/.json?q=QUERY&restrict_sr=on` — no auth needed
|
||||
- Reddit search operators: `subreddit:`, `title:`, `selftext:`, `author:`, `flair:` (Lucene-style)
|
||||
|
||||
### X/Twitter Search Techniques
|
||||
- [igorbrigadir/twitter-advanced-search](https://github.com/igorbrigadir/twitter-advanced-search) — Canonical reference of all X search operators
|
||||
- Bird CLI supports all X operators: `from:`, `to:`, `conversation_id:`, `min_retweets:`, `#hashtag`, `list:`
|
||||
- xAI x_search `allowed_x_handles` parameter — filter to max 10 specific handles
|
||||
- xAI x_search semantic search — finds conceptually related content without exact keyword matches
|
||||
- [Bellingcat OSINT Toolkit](https://bellingcat.gitbook.io/toolkit) — Multi-pass handle discovery methodology
|
||||
|
||||
### Key Insight
|
||||
The biggest gap in the current implementation is that **neither X nor Reddit search does entity extraction from initial results to inform follow-up queries.** Every tool/project researched that achieves better-than-basic results does some form of "discover entities → search entities" two-pass strategy.
|
||||
|
||||
## What We're NOT Doing
|
||||
|
||||
- **Not adding new API dependencies** — everything uses existing OpenAI, xAI, or Bird infrastructure
|
||||
- **Not adding NLP/ML libraries** — entity extraction is simple string parsing
|
||||
- **Not changing the output format** — Phase 2 results merge seamlessly
|
||||
- **Not hardcoding any entities** — all discovery is dynamic from search results
|
||||
- **Not slowing down `--quick` mode** — Phase 2 is skipped entirely
|
||||
- **Not replacing the current search** — Phase 2 supplements Phase 1
|
||||
@@ -30,6 +30,7 @@ from lib import (
|
||||
bird_x,
|
||||
dates,
|
||||
dedupe,
|
||||
entity_extract,
|
||||
env,
|
||||
http,
|
||||
models,
|
||||
@@ -205,6 +206,129 @@ def _search_x(
|
||||
return x_items, raw_response, x_error
|
||||
|
||||
|
||||
def _run_supplemental(
|
||||
topic: str,
|
||||
reddit_items: list,
|
||||
x_items: list,
|
||||
from_date: str,
|
||||
to_date: str,
|
||||
depth: str,
|
||||
x_source: str,
|
||||
progress: ui.ProgressDisplay = None,
|
||||
) -> tuple:
|
||||
"""Run Phase 2 supplemental searches based on entities from Phase 1.
|
||||
|
||||
Extracts handles/subreddits from initial results, then runs targeted
|
||||
searches to find additional content the broad search missed.
|
||||
|
||||
Args:
|
||||
topic: Original search topic
|
||||
reddit_items: Phase 1 Reddit items (raw dicts)
|
||||
x_items: Phase 1 X items (raw dicts)
|
||||
from_date: Start date
|
||||
to_date: End date
|
||||
depth: Research depth
|
||||
x_source: 'bird' or 'xai'
|
||||
progress: Optional progress display
|
||||
|
||||
Returns:
|
||||
Tuple of (supplemental_reddit, supplemental_x)
|
||||
"""
|
||||
# Depth-dependent caps
|
||||
if depth == "default":
|
||||
max_handles = 3
|
||||
max_subs = 3
|
||||
count_per = 3
|
||||
else: # deep
|
||||
max_handles = 5
|
||||
max_subs = 5
|
||||
count_per = 5
|
||||
|
||||
# Extract entities from Phase 1 results
|
||||
entities = entity_extract.extract_entities(
|
||||
reddit_items, x_items,
|
||||
max_handles=max_handles,
|
||||
max_subreddits=max_subs,
|
||||
)
|
||||
|
||||
has_handles = entities["x_handles"] and x_source == "bird"
|
||||
has_subs = entities["reddit_subreddits"]
|
||||
|
||||
if not has_handles and not has_subs:
|
||||
return [], []
|
||||
|
||||
parts = []
|
||||
if has_handles:
|
||||
parts.append(f"@{', @'.join(entities['x_handles'][:3])}")
|
||||
if has_subs:
|
||||
parts.append(f"r/{', r/'.join(entities['reddit_subreddits'][:3])}")
|
||||
sys.stderr.write(f"[Phase 2] Drilling into {' + '.join(parts)}\n")
|
||||
sys.stderr.flush()
|
||||
|
||||
supplemental_reddit = []
|
||||
supplemental_x = []
|
||||
|
||||
# Collect existing URLs to avoid adding duplicates before dedupe
|
||||
existing_urls = set()
|
||||
for item in reddit_items:
|
||||
existing_urls.add(item.get("url", ""))
|
||||
for item in x_items:
|
||||
existing_urls.add(item.get("url", ""))
|
||||
|
||||
# Run supplemental searches in parallel
|
||||
reddit_future = None
|
||||
x_future = None
|
||||
|
||||
with ThreadPoolExecutor(max_workers=2) as executor:
|
||||
if has_subs:
|
||||
reddit_future = executor.submit(
|
||||
openai_reddit.search_subreddits,
|
||||
entities["reddit_subreddits"],
|
||||
topic,
|
||||
from_date,
|
||||
to_date,
|
||||
count_per,
|
||||
)
|
||||
|
||||
if has_handles:
|
||||
x_future = executor.submit(
|
||||
bird_x.search_handles,
|
||||
entities["x_handles"],
|
||||
topic,
|
||||
from_date,
|
||||
count_per,
|
||||
)
|
||||
|
||||
if reddit_future:
|
||||
try:
|
||||
raw_reddit = reddit_future.result()
|
||||
# Filter out URLs already found in Phase 1
|
||||
supplemental_reddit = [
|
||||
item for item in raw_reddit
|
||||
if item.get("url", "") not in existing_urls
|
||||
]
|
||||
except Exception as e:
|
||||
sys.stderr.write(f"[Phase 2] Supplemental Reddit error: {e}\n")
|
||||
|
||||
if x_future:
|
||||
try:
|
||||
raw_x = x_future.result()
|
||||
supplemental_x = [
|
||||
item for item in raw_x
|
||||
if item.get("url", "") not in existing_urls
|
||||
]
|
||||
except Exception as e:
|
||||
sys.stderr.write(f"[Phase 2] Supplemental X error: {e}\n")
|
||||
|
||||
if supplemental_reddit or supplemental_x:
|
||||
sys.stderr.write(
|
||||
f"[Phase 2] +{len(supplemental_reddit)} Reddit, +{len(supplemental_x)} X\n"
|
||||
)
|
||||
sys.stderr.flush()
|
||||
|
||||
return supplemental_reddit, supplemental_x
|
||||
|
||||
|
||||
def run_research(
|
||||
topic: str,
|
||||
sources: str,
|
||||
@@ -319,6 +443,18 @@ def run_research(
|
||||
if progress:
|
||||
progress.end_reddit_enrich()
|
||||
|
||||
# Phase 2: Supplemental search based on entities from Phase 1
|
||||
# Skip on --quick (speed matters) and mock mode
|
||||
if depth != "quick" and not mock and (reddit_items or x_items):
|
||||
sup_reddit, sup_x = _run_supplemental(
|
||||
topic, reddit_items, x_items,
|
||||
from_date, to_date, depth, x_source, progress,
|
||||
)
|
||||
if sup_reddit:
|
||||
reddit_items.extend(sup_reddit)
|
||||
if sup_x:
|
||||
x_items.extend(sup_x)
|
||||
|
||||
return reddit_items, x_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error
|
||||
|
||||
|
||||
|
||||
@@ -190,6 +190,70 @@ def search_x(
|
||||
return {"error": str(e), "items": []}
|
||||
|
||||
|
||||
def search_handles(
|
||||
handles: List[str],
|
||||
topic: str,
|
||||
from_date: str,
|
||||
count_per: int = 5,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Search specific X handles for topic-related content.
|
||||
|
||||
Runs targeted Bird searches using `from:handle topic` syntax.
|
||||
Used in Phase 2 supplemental search after entity extraction.
|
||||
|
||||
Args:
|
||||
handles: List of X handles to search (without @)
|
||||
topic: Search topic (core subject, not full verbose query)
|
||||
from_date: Start date (YYYY-MM-DD)
|
||||
count_per: Results to request per handle
|
||||
|
||||
Returns:
|
||||
List of raw item dicts (same format as parse_bird_response output).
|
||||
"""
|
||||
all_items = []
|
||||
core_topic = _extract_core_subject(topic)
|
||||
|
||||
for handle in handles:
|
||||
handle = handle.lstrip("@")
|
||||
query = f"from:{handle} {core_topic} since:{from_date}"
|
||||
|
||||
cmd = [
|
||||
"bird", "search",
|
||||
query,
|
||||
"-n", str(count_per),
|
||||
"--json",
|
||||
]
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=15, # Short timeout per handle
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
_log(f"Handle search failed for @{handle}: {result.stderr.strip()}")
|
||||
continue
|
||||
|
||||
output = result.stdout.strip()
|
||||
if not output:
|
||||
continue
|
||||
|
||||
response = json.loads(output)
|
||||
items = parse_bird_response(response)
|
||||
all_items.extend(items)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
_log(f"Handle search timed out for @{handle}")
|
||||
except json.JSONDecodeError:
|
||||
_log(f"Invalid JSON from handle search for @{handle}")
|
||||
except Exception as e:
|
||||
_log(f"Handle search error for @{handle}: {e}")
|
||||
|
||||
return all_items
|
||||
|
||||
|
||||
def parse_bird_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse Bird response to match xai_x output format.
|
||||
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
"""Entity extraction from Phase 1 search results for supplemental searches."""
|
||||
|
||||
import re
|
||||
from collections import Counter
|
||||
from typing import Any, Dict, List
|
||||
|
||||
# Handles that appear too frequently to be useful for targeted search.
|
||||
# These are generic/platform accounts, not topic-specific voices.
|
||||
GENERIC_HANDLES = {
|
||||
"elonmusk", "openai", "google", "microsoft", "apple", "meta",
|
||||
"github", "youtube", "x", "twitter", "reddit", "wikipedia",
|
||||
"nytimes", "washingtonpost", "cnn", "bbc", "reuters",
|
||||
"verified", "jack", "sundarpichai",
|
||||
}
|
||||
|
||||
|
||||
def extract_entities(
|
||||
reddit_items: List[Dict[str, Any]],
|
||||
x_items: List[Dict[str, Any]],
|
||||
max_handles: int = 5,
|
||||
max_hashtags: int = 3,
|
||||
max_subreddits: int = 5,
|
||||
) -> Dict[str, List[str]]:
|
||||
"""Extract key entities from Phase 1 results for supplemental searches.
|
||||
|
||||
Parses X results for @handles and #hashtags, Reddit results for subreddit
|
||||
names and cross-referenced communities.
|
||||
|
||||
Args:
|
||||
reddit_items: Raw Reddit item dicts from Phase 1
|
||||
x_items: Raw X item dicts from Phase 1
|
||||
max_handles: Maximum handles to return
|
||||
max_hashtags: Maximum hashtags to return
|
||||
max_subreddits: Maximum subreddits to return
|
||||
|
||||
Returns:
|
||||
Dict with keys: x_handles, x_hashtags, reddit_subreddits
|
||||
"""
|
||||
handles = _extract_x_handles(x_items)
|
||||
hashtags = _extract_x_hashtags(x_items)
|
||||
subreddits = _extract_subreddits(reddit_items)
|
||||
|
||||
return {
|
||||
"x_handles": handles[:max_handles],
|
||||
"x_hashtags": hashtags[:max_hashtags],
|
||||
"reddit_subreddits": subreddits[:max_subreddits],
|
||||
}
|
||||
|
||||
|
||||
def _extract_x_handles(x_items: List[Dict[str, Any]]) -> List[str]:
|
||||
"""Extract and rank @handles from X results.
|
||||
|
||||
Sources handles from:
|
||||
1. author_handle field (who posted)
|
||||
2. @mentions in post text (who they're talking about/to)
|
||||
|
||||
Returns handles ranked by frequency, filtered for generic accounts.
|
||||
"""
|
||||
handle_counts = Counter()
|
||||
|
||||
for item in x_items:
|
||||
# Author handle
|
||||
author = item.get("author_handle", "").strip().lstrip("@").lower()
|
||||
if author and author not in GENERIC_HANDLES:
|
||||
handle_counts[author] += 1
|
||||
|
||||
# @mentions in text
|
||||
text = item.get("text", "")
|
||||
mentions = re.findall(r'@(\w{1,15})', text)
|
||||
for mention in mentions:
|
||||
mention_lower = mention.lower()
|
||||
if mention_lower not in GENERIC_HANDLES:
|
||||
handle_counts[mention_lower] += 1
|
||||
|
||||
# Return all handles ranked by frequency
|
||||
return [h for h, _ in handle_counts.most_common()]
|
||||
|
||||
|
||||
def _extract_x_hashtags(x_items: List[Dict[str, Any]]) -> List[str]:
|
||||
"""Extract and rank #hashtags from X results.
|
||||
|
||||
Returns hashtags ranked by frequency.
|
||||
"""
|
||||
hashtag_counts = Counter()
|
||||
|
||||
for item in x_items:
|
||||
text = item.get("text", "")
|
||||
tags = re.findall(r'#(\w{2,30})', text)
|
||||
for tag in tags:
|
||||
hashtag_counts[tag.lower()] += 1
|
||||
|
||||
# Return all hashtags ranked by frequency
|
||||
return [f"#{t}" for t, _ in hashtag_counts.most_common()]
|
||||
|
||||
|
||||
def _extract_subreddits(reddit_items: List[Dict[str, Any]]) -> List[str]:
|
||||
"""Extract and rank subreddits from Reddit results.
|
||||
|
||||
Sources from:
|
||||
1. subreddit field on each result
|
||||
2. Cross-references in comment text (e.g., "check out r/localLLaMA")
|
||||
|
||||
Returns subreddits ranked by frequency.
|
||||
"""
|
||||
sub_counts = Counter()
|
||||
|
||||
for item in reddit_items:
|
||||
# Primary subreddit
|
||||
sub = item.get("subreddit", "").strip().lstrip("r/")
|
||||
if sub:
|
||||
sub_counts[sub] += 1
|
||||
|
||||
# Cross-references in comment insights
|
||||
for insight in item.get("comment_insights", []):
|
||||
cross_refs = re.findall(r'r/(\w{2,30})', insight)
|
||||
for ref in cross_refs:
|
||||
sub_counts[ref] += 1
|
||||
|
||||
# Cross-references in top comments
|
||||
for comment in item.get("top_comments", []):
|
||||
excerpt = comment.get("excerpt", "")
|
||||
cross_refs = re.findall(r'r/(\w{2,30})', excerpt)
|
||||
for ref in cross_refs:
|
||||
sub_counts[ref] += 1
|
||||
|
||||
# Return subreddits ranked by frequency
|
||||
return [sub for sub, _ in sub_counts.most_common()]
|
||||
@@ -198,6 +198,87 @@ def search_reddit(
|
||||
raise http.HTTPError("No models available")
|
||||
|
||||
|
||||
def search_subreddits(
|
||||
subreddits: List[str],
|
||||
topic: str,
|
||||
from_date: str,
|
||||
to_date: str,
|
||||
count_per: int = 5,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Search specific subreddits via Reddit's free JSON endpoint.
|
||||
|
||||
No API key needed. Uses reddit.com/r/{sub}/search/.json endpoint.
|
||||
Used in Phase 2 supplemental search after entity extraction.
|
||||
|
||||
Args:
|
||||
subreddits: List of subreddit names (without r/)
|
||||
topic: Search topic
|
||||
from_date: Start date (YYYY-MM-DD)
|
||||
to_date: End date (YYYY-MM-DD)
|
||||
count_per: Results to request per subreddit
|
||||
|
||||
Returns:
|
||||
List of raw item dicts (same format as parse_reddit_response output).
|
||||
"""
|
||||
all_items = []
|
||||
core = _extract_core_subject(topic)
|
||||
|
||||
for sub in subreddits:
|
||||
sub = sub.lstrip("r/")
|
||||
try:
|
||||
url = f"https://www.reddit.com/r/{sub}/search/.json"
|
||||
params = f"q={_url_encode(core)}&restrict_sr=on&sort=new&limit={count_per}&raw_json=1"
|
||||
full_url = f"{url}?{params}"
|
||||
|
||||
headers = {
|
||||
"User-Agent": http.USER_AGENT,
|
||||
"Accept": "application/json",
|
||||
}
|
||||
|
||||
data = http.get(full_url, headers=headers, timeout=15)
|
||||
|
||||
# Reddit search returns {"data": {"children": [...]}}
|
||||
children = data.get("data", {}).get("children", [])
|
||||
for i, child in enumerate(children):
|
||||
if child.get("kind") != "t3": # t3 = link/submission
|
||||
continue
|
||||
post = child.get("data", {})
|
||||
permalink = post.get("permalink", "")
|
||||
if not permalink:
|
||||
continue
|
||||
|
||||
item = {
|
||||
"id": f"RS{len(all_items)+1}",
|
||||
"title": str(post.get("title", "")).strip(),
|
||||
"url": f"https://www.reddit.com{permalink}",
|
||||
"subreddit": str(post.get("subreddit", sub)).strip(),
|
||||
"date": None,
|
||||
"why_relevant": f"Found in r/{sub} supplemental search",
|
||||
"relevance": 0.65, # Slightly lower default for supplemental
|
||||
}
|
||||
|
||||
# Parse date from created_utc
|
||||
created_utc = post.get("created_utc")
|
||||
if created_utc:
|
||||
from . import dates as dates_mod
|
||||
item["date"] = dates_mod.timestamp_to_date(created_utc)
|
||||
|
||||
all_items.append(item)
|
||||
|
||||
except http.HTTPError as e:
|
||||
_log_info(f"Subreddit search failed for r/{sub}: {e}")
|
||||
except Exception as e:
|
||||
_log_info(f"Subreddit search error for r/{sub}: {e}")
|
||||
|
||||
return all_items
|
||||
|
||||
|
||||
def _url_encode(text: str) -> str:
|
||||
"""Simple URL encoding for query parameters."""
|
||||
import urllib.parse
|
||||
return urllib.parse.quote_plus(text)
|
||||
|
||||
|
||||
def parse_reddit_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
"""Parse OpenAI response to extract Reddit items.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user