Merge PR #48: feat: add Xiaohongshu source + Reddit public fallback
- Xiaohongshu search via local MCP service (opt-in, zero impact if service not running) - Reddit public JSON fallback (works with zero API keys) - Reddit priority: ScrapeCreators -> OpenAI -> public fallback - Updated env.py: Reddit always available via public fallback Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
@@ -0,0 +1,200 @@
|
||||
# How Reddit & X Search Work in last30days
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
User: /last30days "kanye west"
|
||||
↓
|
||||
┌─────┴─────┐
|
||||
↓ ↓ (concurrent via ThreadPoolExecutor)
|
||||
[REDDIT] [X/TWITTER]
|
||||
↓ ↓
|
||||
OpenAI Bird CLI or
|
||||
API xAI API
|
||||
↓ ↓
|
||||
Parse Parse
|
||||
↓ ↓
|
||||
Enrich ───┘
|
||||
(fetch ↓
|
||||
actual [MERGE]
|
||||
upvotes) ↓
|
||||
↓ [NORMALIZE → FILTER → SCORE → DEDUPE]
|
||||
└───────────↓
|
||||
[OUTPUT to SKILL.md agent]
|
||||
```
|
||||
|
||||
Both searches run **in parallel** using Python's `ThreadPoolExecutor(max_workers=2)`.
|
||||
|
||||
---
|
||||
|
||||
## Reddit Search
|
||||
|
||||
### How it works
|
||||
|
||||
Reddit search uses the **OpenAI Responses API** with the `web_search` tool, domain-filtered to `reddit.com` only.
|
||||
|
||||
**API Call:**
|
||||
```
|
||||
POST https://api.openai.com/v1/responses
|
||||
Authorization: Bearer {OPENAI_API_KEY}
|
||||
```
|
||||
|
||||
**Payload:**
|
||||
```json
|
||||
{
|
||||
"model": "gpt-5.2",
|
||||
"tools": [{
|
||||
"type": "web_search",
|
||||
"filters": { "allowed_domains": ["reddit.com"] }
|
||||
}],
|
||||
"input": "Search Reddit for threads about {topic}..."
|
||||
}
|
||||
```
|
||||
|
||||
The prompt asks the model to:
|
||||
1. Extract core subject (strip noise words like "best", "tips", "top")
|
||||
2. Search 3 patterns: `"{topic} site:reddit.com"`, `"reddit {topic}"`, `"{topic} reddit"`
|
||||
3. Return JSON with `title`, `url`, `subreddit`, `date`, `relevance`
|
||||
4. URLs must contain `/r/` AND `/comments/` (real threads only)
|
||||
|
||||
**Model fallback chain:** `gpt-5.2 → gpt-5.1 → gpt-5 → gpt-4.1 → gpt-4o → gpt-4o-mini`
|
||||
Triggers on HTTP 400/403 with access error keywords.
|
||||
|
||||
### Enrichment (the secret sauce)
|
||||
|
||||
After search, each thread gets **enriched** by hitting Reddit's free JSON API:
|
||||
|
||||
```
|
||||
GET https://reddit.com/r/{sub}/comments/{id}/{slug}/.json
|
||||
```
|
||||
|
||||
No API key needed. This returns the actual thread data:
|
||||
|
||||
| Data Point | Source |
|
||||
|---|---|
|
||||
| Upvotes (score) | Reddit JSON API |
|
||||
| Comment count | Reddit JSON API |
|
||||
| Upvote ratio | Reddit JSON API |
|
||||
| Top 10 comments (text + score) | Reddit JSON API |
|
||||
| 7 key comment insights | Extracted via heuristics |
|
||||
| Actual post date | `created_utc` timestamp |
|
||||
|
||||
**This is why Reddit results have real engagement metrics** — the enrichment step fetches actual upvote/comment data, not AI estimates.
|
||||
|
||||
### Depth settings
|
||||
|
||||
| Depth | Threads requested | Timeout |
|
||||
|---|---|---|
|
||||
| `--quick` | 15-25 | 90s |
|
||||
| default | 30-50 | 120s |
|
||||
| `--deep` | 70-100 | 180s |
|
||||
|
||||
---
|
||||
|
||||
## X/Twitter Search
|
||||
|
||||
X search has **two backends** — the skill auto-detects which to use.
|
||||
|
||||
### Priority: Bird CLI (free) → xAI API (paid)
|
||||
|
||||
```python
|
||||
if bird_installed and bird_authenticated:
|
||||
use Bird CLI # Free, uses your X login
|
||||
elif XAI_API_KEY:
|
||||
use xAI API # Paid, uses grok-4-1-fast
|
||||
else:
|
||||
skip X entirely # No X results
|
||||
```
|
||||
|
||||
### Backend 1: xAI API
|
||||
|
||||
**API Call:**
|
||||
```
|
||||
POST https://api.x.ai/v1/responses
|
||||
Authorization: Bearer {XAI_API_KEY}
|
||||
```
|
||||
|
||||
**Payload:**
|
||||
```json
|
||||
{
|
||||
"model": "grok-4-1-fast",
|
||||
"tools": [{ "type": "x_search" }],
|
||||
"input": "Search X for posts about {topic} from {from_date} to {to_date}..."
|
||||
}
|
||||
```
|
||||
|
||||
The prompt asks grok to return JSON with:
|
||||
- `text`, `url`, `author_handle`, `date`
|
||||
- `engagement`: `{ likes, reposts, replies, quotes }`
|
||||
- `why_relevant`, `relevance` score
|
||||
|
||||
**Engagement data comes from grok's x_search tool** — it has direct access to X's data.
|
||||
|
||||
### Backend 2: Bird CLI (free alternative)
|
||||
|
||||
Bird is a CLI tool (`npm install -g @steipete/bird`) that uses your X login.
|
||||
|
||||
**Command:**
|
||||
```bash
|
||||
bird search "{topic} since:{from_date}" -n 30 --json
|
||||
```
|
||||
|
||||
**Bird returns raw X API data** — likes, reposts, replies are real engagement metrics from X's API, not estimates.
|
||||
|
||||
| Metric | Bird CLI | xAI API |
|
||||
|---|---|---|
|
||||
| Post text | Real | Real |
|
||||
| Likes/reposts | Real (X API) | Real (x_search tool) |
|
||||
| Replies/quotes | Real | Real |
|
||||
| Author handle | Real | Real |
|
||||
| Relevance score | Default 0.7 (re-ranked by score.py) | AI-assessed 0.0-1.0 |
|
||||
|
||||
### Depth settings
|
||||
|
||||
| Depth | xAI posts | Bird results | xAI timeout | Bird timeout |
|
||||
|---|---|---|---|---|
|
||||
| `--quick` | 8-12 | 12 | 90s | 30s |
|
||||
| default | 20-30 | 30 | 120s | 45s |
|
||||
| `--deep` | 40-60 | 60 | 180s | 60s |
|
||||
|
||||
---
|
||||
|
||||
## Post-Processing (both sources)
|
||||
|
||||
After both searches complete:
|
||||
|
||||
1. **Normalize** — consistent formatting, timezone handling
|
||||
2. **Date filter** — hard filter to requested date range
|
||||
3. **Score** — relevance scoring (engagement-weighted)
|
||||
4. **Sort** — highest scores first
|
||||
5. **Deduplicate** — remove duplicate URLs
|
||||
6. **Fallback** — if all items filtered out, keep top 3 by relevance
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Layer | Strategy |
|
||||
|---|---|
|
||||
| HTTP requests | 3 retries with exponential backoff (1s → 2s → 3s) |
|
||||
| Model access errors | Automatic fallback to next model in chain |
|
||||
| Reddit enrichment | Per-item try/catch; keeps unenriched item on failure |
|
||||
| X source detection | Silent fallback from Bird → xAI → skip |
|
||||
| Overall pipeline | Errors stored as `reddit_error`/`x_error`, shown to user |
|
||||
|
||||
---
|
||||
|
||||
## Key Files
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `scripts/last30days.py` | Main orchestrator, concurrent execution |
|
||||
| `scripts/lib/openai_reddit.py` | Reddit search via OpenAI Responses API |
|
||||
| `scripts/lib/reddit_enrich.py` | Fetch real engagement data from Reddit JSON API |
|
||||
| `scripts/lib/xai_x.py` | X search via xAI API |
|
||||
| `scripts/lib/bird_x.py` | X search via Bird CLI (free) |
|
||||
| `scripts/lib/models.py` | Auto-select best available model |
|
||||
| `scripts/lib/env.py` | API key loading, source detection |
|
||||
| `scripts/lib/http.py` | HTTP transport with retries |
|
||||
| `scripts/lib/score.py` | Relevance scoring |
|
||||
| `scripts/lib/dedupe.py` | URL-based deduplication |
|
||||
Reference in New Issue
Block a user