Merge PR #48: feat: add Xiaohongshu source + Reddit public fallback
- Xiaohongshu search via local MCP service (opt-in, zero impact if service not running) - Reddit public JSON fallback (works with zero API keys) - Reddit priority: ScrapeCreators -> OpenAI -> public fallback - Updated env.py: Reddit always available via public fallback Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
@@ -0,0 +1,281 @@
|
||||
---
|
||||
title: "feat: Automated v1 vs v2 test harness using claude --print"
|
||||
type: feat
|
||||
date: 2026-02-06
|
||||
---
|
||||
|
||||
# feat: Automated V1 vs V2 Test Harness
|
||||
|
||||
## Overview
|
||||
|
||||
Build a bash script that swaps SKILL.md between v1 (upstream) and v2 (current), runs `claude --print "/last30days [query]"` for all 17 test queries on each version, captures output to files, then generates a comparison doc with analysis.
|
||||
|
||||
## How It Works
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ test-v1-vs-v2.sh │
|
||||
│ │
|
||||
│ 1. Save current SKILL.md as .v2 backup │
|
||||
│ 2. Install v1 SKILL.md from upstream │
|
||||
│ 3. Loop 17 queries → claude --print → v1/*.txt │
|
||||
│ 4. Restore v2 SKILL.md │
|
||||
│ 5. Loop 17 queries → claude --print → v2/*.txt │
|
||||
│ 6. Generate comparison doc │
|
||||
└─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Each `claude --print` call:
|
||||
- Invokes the /last30days skill exactly as a user would
|
||||
- Runs the Python script (real API calls to OpenAI + xAI)
|
||||
- Runs WebSearch
|
||||
- Applies SKILL.md presentation instructions
|
||||
- Returns the full formatted output
|
||||
- Exits (no interactive session)
|
||||
|
||||
## Implementation
|
||||
|
||||
### File: `scripts/test-v1-vs-v2.sh`
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
# === Config ===
|
||||
SKILL_DIR="$HOME/.claude/skills/last30days"
|
||||
REPO_DIR="/Users/mvanhorn/last30days-skill-private"
|
||||
OUT_DIR="$REPO_DIR/docs/test-results/v1-vs-v2-$(date +%Y%m%d-%H%M%S)"
|
||||
V1_DIR="$OUT_DIR/v1"
|
||||
V2_DIR="$OUT_DIR/v2"
|
||||
|
||||
mkdir -p "$V1_DIR" "$V2_DIR"
|
||||
|
||||
# All 17 test queries (from README + plans)
|
||||
declare -a QUERIES=(
|
||||
"prompting techniques for chatgpt for legal questions"
|
||||
"best clawdbot use cases"
|
||||
"how to best setup clawdbot"
|
||||
"prompting tips for nano banana pro for ios designs"
|
||||
"top claude code skills"
|
||||
"using ChatGPT to make images of dogs"
|
||||
"research best practices for beautiful remotion animation videos in claude code"
|
||||
"photorealistic people in nano banana pro"
|
||||
"What are the best rap songs lately"
|
||||
"what are people saying about DeepSeek R1"
|
||||
"best practices for cursor rules files for Cursor"
|
||||
"prompt advice for using suno to make killer songs in simple mode"
|
||||
"how do I use Codex with Claude Code on same app to make it better"
|
||||
"kanye west"
|
||||
"howie.ai"
|
||||
"open claw"
|
||||
"nano banana pro prompting"
|
||||
)
|
||||
|
||||
declare -a TYPES=(
|
||||
"PROMPTING+TOOL"
|
||||
"RECOMMENDATIONS"
|
||||
"HOW-TO"
|
||||
"PROMPTING+TOOL"
|
||||
"RECOMMENDATIONS"
|
||||
"GENERAL"
|
||||
"PROMPTING"
|
||||
"PROMPTING"
|
||||
"RECOMMENDATIONS"
|
||||
"NEWS"
|
||||
"PROMPTING"
|
||||
"PROMPTING"
|
||||
"HOW-TO"
|
||||
"NEWS"
|
||||
"GENERAL"
|
||||
"GENERAL"
|
||||
"PROMPTING"
|
||||
)
|
||||
|
||||
slugify() {
|
||||
echo "$1" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | head -c 60
|
||||
}
|
||||
|
||||
run_version() {
|
||||
local version="$1"
|
||||
local outdir="$2"
|
||||
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo " Running $version — ${#QUERIES[@]} queries"
|
||||
echo "=========================================="
|
||||
|
||||
for i in "${!QUERIES[@]}"; do
|
||||
local query="${QUERIES[$i]}"
|
||||
local type="${TYPES[$i]}"
|
||||
local slug=$(slugify "$query")
|
||||
local num=$((i + 1))
|
||||
local outfile="$outdir/${num}-${slug}.txt"
|
||||
|
||||
echo ""
|
||||
echo "[$version] ($num/${#QUERIES[@]}) $query [$type]"
|
||||
echo " → $outfile"
|
||||
|
||||
# Run claude --print with the skill invocation
|
||||
# --no-session-persistence: don't save to session history
|
||||
# Timeout after 5 minutes per query (generous for slow API calls)
|
||||
if timeout 300 claude --print \
|
||||
"/last30days $query" \
|
||||
> "$outfile" 2>"$outdir/${num}-${slug}.stderr.txt"; then
|
||||
echo " ✅ Done ($(wc -l < "$outfile") lines)"
|
||||
else
|
||||
echo " ❌ Failed or timed out"
|
||||
echo "FAILED: timeout or error" >> "$outfile"
|
||||
fi
|
||||
|
||||
# Brief pause between queries to avoid rate limits
|
||||
sleep 2
|
||||
done
|
||||
}
|
||||
|
||||
# === Phase 1: Test V1 ===
|
||||
echo "📦 Backing up current SKILL.md..."
|
||||
cp "$SKILL_DIR/SKILL.md" "$SKILL_DIR/SKILL.md.v2.bak"
|
||||
|
||||
echo "📥 Installing V1 SKILL.md from upstream..."
|
||||
cd "$REPO_DIR"
|
||||
git show upstream/main:SKILL.md > "$SKILL_DIR/SKILL.md"
|
||||
|
||||
# Also save a copy for reference
|
||||
cp "$SKILL_DIR/SKILL.md" "$OUT_DIR/v1-SKILL.md"
|
||||
|
||||
run_version "V1" "$V1_DIR"
|
||||
|
||||
# === Phase 2: Test V2 ===
|
||||
echo ""
|
||||
echo "📥 Restoring V2 SKILL.md..."
|
||||
cp "$SKILL_DIR/SKILL.md.v2.bak" "$SKILL_DIR/SKILL.md"
|
||||
|
||||
# Also save a copy for reference
|
||||
cp "$SKILL_DIR/SKILL.md" "$OUT_DIR/v2-SKILL.md"
|
||||
|
||||
run_version "V2" "$V2_DIR"
|
||||
|
||||
# === Phase 3: Generate summary ===
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo " Generating comparison summary"
|
||||
echo "=========================================="
|
||||
|
||||
SUMMARY="$OUT_DIR/comparison-summary.md"
|
||||
|
||||
cat > "$SUMMARY" << 'HEADER'
|
||||
# V1 vs V2 Comparison Results
|
||||
|
||||
Generated: $(date)
|
||||
|
||||
## Output Files
|
||||
|
||||
| # | Query | Type | V1 Lines | V2 Lines |
|
||||
|---|-------|------|----------|----------|
|
||||
HEADER
|
||||
|
||||
# Replace the date placeholder
|
||||
sed -i '' "s/\$(date)/$(date)/" "$SUMMARY"
|
||||
|
||||
for i in "${!QUERIES[@]}"; do
|
||||
local query="${QUERIES[$i]}"
|
||||
local type="${TYPES[$i]}"
|
||||
local slug=$(slugify "$query")
|
||||
local num=$((i + 1))
|
||||
|
||||
local v1file="$V1_DIR/${num}-${slug}.txt"
|
||||
local v2file="$V2_DIR/${num}-${slug}.txt"
|
||||
|
||||
local v1lines=$(wc -l < "$v1file" 2>/dev/null || echo "0")
|
||||
local v2lines=$(wc -l < "$v2file" 2>/dev/null || echo "0")
|
||||
|
||||
echo "| $num | \`$query\` | $type | $v1lines | $v2lines |" >> "$SUMMARY"
|
||||
done
|
||||
|
||||
cat >> "$SUMMARY" << 'FOOTER'
|
||||
|
||||
## Scorecard Template
|
||||
|
||||
For each query, score both versions on:
|
||||
|
||||
| Dimension | V1 | V2 | Notes |
|
||||
|-----------|----|----|-------|
|
||||
| Query Parsing Display (1-5) | | | |
|
||||
| Source Coverage (1-5) | | | |
|
||||
| Citation Quality (1-5) | | | |
|
||||
| Summary Structure (1-5) | | | |
|
||||
| Stats Box Format (1-5) | | | |
|
||||
| Research Grounding (1-5) | | | |
|
||||
|
||||
## Next Step
|
||||
|
||||
Read each pair of output files and score them using the test plan at:
|
||||
`docs/plans/2026-02-06-test-v1-vs-v2-comparison-plan.md`
|
||||
FOOTER
|
||||
|
||||
echo ""
|
||||
echo "✅ All done!"
|
||||
echo "📁 Results: $OUT_DIR"
|
||||
echo "📊 Summary: $SUMMARY"
|
||||
echo ""
|
||||
echo "V1 outputs: $V1_DIR/"
|
||||
echo "V2 outputs: $V2_DIR/"
|
||||
echo ""
|
||||
echo "To review, run:"
|
||||
echo " open $OUT_DIR"
|
||||
|
||||
# Cleanup backup
|
||||
rm -f "$SKILL_DIR/SKILL.md.v2.bak"
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Script runs all 17 queries on v1 SKILL.md
|
||||
- [ ] Script runs all 17 queries on v2 SKILL.md
|
||||
- [ ] Each query output saved to a separate .txt file
|
||||
- [ ] Comparison summary generated with line counts
|
||||
- [ ] SKILL.md restored to v2 after testing
|
||||
- [ ] Both SKILL.md versions saved in output dir for reference
|
||||
- [ ] Script handles timeouts gracefully (5 min per query)
|
||||
- [ ] Brief pause between queries to avoid rate limits
|
||||
|
||||
## Cost Estimate
|
||||
|
||||
- 34 total `claude --print` invocations
|
||||
- Each invocation: ~1 Python script run (OpenAI + xAI API) + 2-3 WebSearches + Claude response
|
||||
- Estimated: ~$0.10-0.30 per invocation for API calls
|
||||
- **Total estimate: $3-10 for the full run**
|
||||
|
||||
## Time Estimate
|
||||
|
||||
- Each query: ~1-3 minutes (script + WebSearch + synthesis)
|
||||
- 17 queries × 2 versions = 34 runs
|
||||
- **Total: ~45-90 minutes** (could run in background)
|
||||
|
||||
## How to Run
|
||||
|
||||
```bash
|
||||
cd /Users/mvanhorn/last30days-skill-private
|
||||
chmod +x scripts/test-v1-vs-v2.sh
|
||||
./scripts/test-v1-vs-v2.sh
|
||||
```
|
||||
|
||||
Or run in background:
|
||||
```bash
|
||||
nohup ./scripts/test-v1-vs-v2.sh > test-run.log 2>&1 &
|
||||
tail -f test-run.log
|
||||
```
|
||||
|
||||
## After the Run
|
||||
|
||||
Once all outputs are captured, Claude can read every file pair and generate the scored comparison doc with analysis — that's the part where I score each dimension 1-5 and write the final report.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `scripts/test-v1-vs-v2.sh` | The test harness script |
|
||||
| `docs/test-results/v1-vs-v2-*/` | Output directory (timestamped) |
|
||||
| `docs/test-results/v1-vs-v2-*/v1/*.txt` | V1 outputs |
|
||||
| `docs/test-results/v1-vs-v2-*/v2/*.txt` | V2 outputs |
|
||||
| `docs/test-results/v1-vs-v2-*/comparison-summary.md` | Auto-generated summary |
|
||||
@@ -0,0 +1,352 @@
|
||||
---
|
||||
title: "feat: Free Reddit via MCP — Remove OpenAI Key Requirement"
|
||||
type: feat
|
||||
date: 2026-02-06
|
||||
---
|
||||
|
||||
# feat: Free Reddit via MCP — Remove OpenAI Key Requirement
|
||||
|
||||
## Overview
|
||||
|
||||
Replace the OpenAI Responses API (paid) for Reddit searching with a **free, zero-config Reddit MCP server**, eliminating the need for an `OPENAI_API_KEY` to get Reddit results in the last30days skill. This work happens in a **new forked repo** to keep the current release branch clean.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Today, the last30days skill requires an OpenAI API key (`OPENAI_API_KEY`) to search Reddit. This is the most expensive dependency in the stack — OpenAI charges per-call for the Responses API with web search. Users without an OpenAI key get zero Reddit results and fall back to WebSearch-only mode, which loses the engagement metrics (upvotes, comments) that make last30days uniquely valuable.
|
||||
|
||||
**Goal:** Make Reddit search completely free with no API key registration required.
|
||||
|
||||
## Research Findings
|
||||
|
||||
### Option Analysis
|
||||
|
||||
Five approaches were evaluated. Two stand out:
|
||||
|
||||
| Option | Free? | Search? | Engagement? | Setup | Notes |
|
||||
|--------|-------|---------|-------------|-------|-------|
|
||||
| **reddit-mcp-buddy** (Node.js) | Yes (anonymous mode) | Yes | Yes | One CLI command | 371 stars, 3-tier auth, most popular |
|
||||
| **mcp-server-reddit** (Python) | Yes (redditwarp) | **No** | Yes | pip install | Recommended by ClaudeLog, but browse-only |
|
||||
| **.json URL trick** (curl) | Yes | Yes | Yes | Zero | ~10 req/min, no dependencies |
|
||||
| Reddit OAuth (PRAW) | Free but needs registration | Yes | Yes | App registration | 100 req/min, most reliable |
|
||||
| RSS feeds | Yes | Basic | **No** | Zero | Least useful, no metrics |
|
||||
|
||||
### Top Contender: reddit-mcp-buddy
|
||||
|
||||
**GitHub:** [karanb192/reddit-mcp-buddy](https://github.com/karanb192/reddit-mcp-buddy)
|
||||
- 371 stars, actively maintained (last update: Jan 29, 2026)
|
||||
- **Anonymous mode** — zero credentials, ~10 req/min
|
||||
- **MCP tools:** `search_reddit`, `browse_subreddit`, `get_post_details`, `user_analysis`, `reddit_explain`
|
||||
- **Install:** `claude mcp add --transport stdio reddit-mcp-buddy -s user -- npx -y reddit-mcp-buddy`
|
||||
- Returns full engagement metrics (score, num_comments, upvote_ratio)
|
||||
- TypeScript/Node.js (npx, no Python dependency)
|
||||
|
||||
### Strong Alternative: .json URL trick
|
||||
|
||||
- Append `.json` to any Reddit URL → full JSON response
|
||||
- Search: `https://www.reddit.com/search.json?q=TOPIC&sort=relevance&t=month&limit=100`
|
||||
- Zero dependencies, zero auth, zero setup
|
||||
- Returns same data as official API (score, num_comments, created_utc, upvote_ratio)
|
||||
- ~10 req/min rate limit (sufficient for skill use)
|
||||
- Can be called via `curl` or Python `urllib`
|
||||
|
||||
### Also Considered
|
||||
|
||||
- **Hawstein/mcp-server-reddit** (134 stars, Python, no auth) — **No search tool**, only browse subreddits. Cannot replace OpenAI's keyword search capability.
|
||||
- **Arindam200/reddit-mcp** (262 stars, PRAW) — Has search but **requires Reddit OAuth credentials**. Not truly zero-config.
|
||||
- **adhikasp/mcp-reddit** (348 stars) — Hot threads only, no search. Last updated Dec 2024.
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
### Approach A: MCP-First with .json Fallback (Recommended)
|
||||
|
||||
Add a new module `mcp_reddit.py` that:
|
||||
|
||||
1. **Detects** if a Reddit MCP server is configured in the user's Claude Code session
|
||||
2. **If MCP available:** Calls MCP `search_reddit` tool via the skill's `allowed-tools` (the skill already allows tools — MCP tools become available when configured)
|
||||
3. **If no MCP:** Falls back to Reddit's `.json` URL endpoints via `urllib` (stdlib, no dependencies)
|
||||
4. **Normalize** MCP/JSON results to the existing `RedditItem` schema
|
||||
5. **Enrich** with `reddit_enrich.py` (already works — fetches real thread data)
|
||||
|
||||
This gives users three tiers of Reddit access:
|
||||
- **Tier 1 (best):** Reddit MCP installed → full search + engagement via MCP tools
|
||||
- **Tier 2 (good):** No MCP, no keys → `.json` URL search + enrichment
|
||||
- **Tier 3 (existing):** OpenAI key present → existing `openai_reddit.py` still works (backward compat)
|
||||
|
||||
### Approach B: .json-Only (Simpler)
|
||||
|
||||
Skip MCP entirely. Add a `reddit_json.py` module that uses `https://www.reddit.com/search.json` directly. Simpler but misses the MCP ecosystem integration.
|
||||
|
||||
### Approach C: MCP-Only (Cleaner)
|
||||
|
||||
Require MCP setup. Simpler code but adds a user setup step (`claude mcp add ...`).
|
||||
|
||||
**Recommendation: Approach A** — MCP-first with .json fallback gives zero-config Reddit search for everyone while rewarding users who set up MCP with a better experience.
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
User invokes /last30days "topic"
|
||||
│
|
||||
├─ env.py detects sources:
|
||||
│ ├─ MCP reddit available? → mcp_reddit.py
|
||||
│ ├─ No MCP, no key? → reddit_json.py (.json URL trick)
|
||||
│ └─ OPENAI_API_KEY set? → openai_reddit.py (existing, backward compat)
|
||||
│
|
||||
├─ X search (Bird CLI / xAI — unchanged)
|
||||
│
|
||||
└─ Normalize → Score → Dedupe → Render (unchanged)
|
||||
```
|
||||
|
||||
### New Files
|
||||
|
||||
#### `scripts/lib/reddit_json.py`
|
||||
- Uses `urllib.request` (stdlib) to call Reddit's `.json` search endpoint
|
||||
- URL: `https://www.reddit.com/search.json?q={topic}&sort=relevance&t=month&limit=100`
|
||||
- Custom `User-Agent` header (required by Reddit)
|
||||
- Parses response into the same format as `openai_reddit.py` output
|
||||
- Handles pagination via `after` token if depth=deep (multiple requests)
|
||||
- Rate limiting: 1 second delay between requests
|
||||
|
||||
#### `scripts/lib/mcp_reddit.py`
|
||||
- Detects MCP availability (check if Reddit MCP tools exist in session)
|
||||
- Formats MCP tool calls for `search_reddit` with topic + date filters
|
||||
- Normalizes MCP response to match existing `RedditItem` schema
|
||||
- Falls back to `reddit_json.py` if MCP unavailable
|
||||
|
||||
#### Modified Files
|
||||
|
||||
- **`scripts/lib/env.py`** — Add Reddit source detection: MCP → .json → OpenAI
|
||||
- **`scripts/last30days.py`** — Route Reddit search through new priority chain
|
||||
- **`scripts/lib/normalize.py`** — Add normalizer for `.json` endpoint response format
|
||||
- **`SKILL.md`** — Document MCP setup as optional enhancement
|
||||
|
||||
### MCP Integration Design
|
||||
|
||||
The MCP approach has a subtle challenge: the last30days skill runs as a **Python subprocess** (`python3 last30days.py`), but MCP tools are available to **Claude's session**, not to subprocesses.
|
||||
|
||||
**Two paths to solve this:**
|
||||
|
||||
**Path 1: SKILL.md orchestration** — The SKILL.md workflow calls the MCP `search_reddit` tool directly (before or in parallel with the Python script), saves the MCP response to a temp file, and the Python script reads it:
|
||||
|
||||
```
|
||||
SKILL.md workflow:
|
||||
1. Call MCP search_reddit → save to /tmp/last30days_mcp_reddit.json
|
||||
2. Call python3 last30days.py --reddit-from=/tmp/last30days_mcp_reddit.json --emit=compact
|
||||
3. Python script reads pre-fetched Reddit data instead of calling OpenAI
|
||||
```
|
||||
|
||||
**Path 2: .json only for subprocess** — The Python script uses `.json` URLs directly (no MCP needed in subprocess). MCP is a bonus for users who want Claude to also browse specific threads interactively.
|
||||
|
||||
**Recommendation: Path 2 for MVP, Path 1 as enhancement.** The `.json` approach is self-contained, testable, and doesn't require SKILL.md workflow changes. MCP can be layered on later.
|
||||
|
||||
### Source Priority Chain (updated env.py)
|
||||
|
||||
```python
|
||||
def get_reddit_source(config: dict) -> str:
|
||||
"""Returns: 'mcp', 'json', 'openai', or 'none'"""
|
||||
# 1. Check for pre-fetched MCP data (from SKILL.md)
|
||||
if os.path.exists(MCP_REDDIT_CACHE_PATH):
|
||||
return 'mcp'
|
||||
# 2. Always available — no key needed
|
||||
# (reddit .json endpoints are free)
|
||||
return 'json'
|
||||
# 3. OpenAI key present → legacy path
|
||||
# if config.get('OPENAI_API_KEY'):
|
||||
# return 'openai'
|
||||
```
|
||||
|
||||
For MVP, the `.json` path is **always available** so it becomes the default. OpenAI path remains as opt-in for users who want higher rate limits.
|
||||
|
||||
### Data Mapping: .json → RedditItem
|
||||
|
||||
Reddit `.json` search returns `data.children[].data` with:
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "Post title",
|
||||
"permalink": "/r/subreddit/comments/abc123/...",
|
||||
"subreddit": "ClaudeAI",
|
||||
"score": 42,
|
||||
"num_comments": 15,
|
||||
"upvote_ratio": 0.95,
|
||||
"created_utc": 1738800000,
|
||||
"selftext": "Post body...",
|
||||
"url": "https://...",
|
||||
"author": "username"
|
||||
}
|
||||
```
|
||||
|
||||
Maps cleanly to existing `RedditItem`:
|
||||
|
||||
| .json field | RedditItem field | Notes |
|
||||
|-------------|------------------|-------|
|
||||
| `title` | `title` | Direct |
|
||||
| `permalink` | `url` | Prepend `https://www.reddit.com` |
|
||||
| `subreddit` | `subreddit` | Direct |
|
||||
| `score` | `engagement.score` | Direct |
|
||||
| `num_comments` | `engagement.num_comments` | Direct |
|
||||
| `upvote_ratio` | `engagement.upvote_ratio` | Direct |
|
||||
| `created_utc` | `date` | Convert epoch → YYYY-MM-DD |
|
||||
| `selftext` | (used for relevance) | AI relevance scoring needed |
|
||||
| `author` | (not in current schema) | Ignore for now |
|
||||
|
||||
### Relevance Scoring Without AI
|
||||
|
||||
The current `openai_reddit.py` gets relevance scores **from OpenAI** (the AI judges how relevant each result is). With `.json` endpoints, we lose that AI relevance judgment.
|
||||
|
||||
**Options:**
|
||||
1. **Keyword matching** — Score based on how many query terms appear in title + selftext. Simple but effective.
|
||||
2. **TF-IDF-like** — Weight rarer query terms higher. More accurate but more code.
|
||||
3. **Let Claude judge** — Pass results to Claude in SKILL.md and have Claude score relevance. Most accurate but changes the workflow.
|
||||
4. **Skip relevance, rely on Reddit's sort** — Reddit's `sort=relevance` already ranks by relevance. Trust it and use position-based scoring (first result = 1.0, last = 0.5).
|
||||
|
||||
**Recommendation: Option 4 for MVP.** Reddit's search relevance ranking is already good. Use position-based relevance (1.0 → 0.5 linear decay over result set) combined with the existing engagement-based scoring. This requires zero external dependencies and no AI calls.
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 0: Fork Repository
|
||||
|
||||
- [ ] Create new repo `last30days-free-reddit` (or branch in private repo)
|
||||
- [ ] `git clone /Users/mvanhorn/last30days-skill-private /Users/mvanhorn/last30days-free-reddit`
|
||||
- [ ] Create feature branch: `feat/free-reddit`
|
||||
- [ ] Verify all existing tests pass on the new branch
|
||||
|
||||
### Phase 1: reddit_json.py — Core .json Search
|
||||
|
||||
- [ ] Create `scripts/lib/reddit_json.py`
|
||||
- `search_reddit_json(topic: str, depth: str, date_from: str, date_to: str) -> list[dict]`
|
||||
- Build search URL with `q`, `sort=relevance`, `t=month`, `limit` (25/50/100 by depth)
|
||||
- Custom `User-Agent: last30days-skill/2.0 (by /u/last30days-bot)`
|
||||
- Parse `data.children[].data` → list of raw dicts
|
||||
- Handle pagination for deep mode (follow `after` token, max 3 pages)
|
||||
- Rate limit: `time.sleep(1.0)` between requests
|
||||
- Error handling: 429 (rate limit), 403 (blocked), network errors → return empty + error msg
|
||||
- [ ] Create `tests/test_reddit_json.py`
|
||||
- Mock HTTP responses using fixture files
|
||||
- Test: basic search parsing, pagination, rate limit handling, error cases
|
||||
- [ ] Create `fixtures/reddit_json_search_sample.json`
|
||||
- Real `.json` search response (sanitized)
|
||||
|
||||
### Phase 2: Normalize + Score .json Results
|
||||
|
||||
- [ ] Add `normalize_reddit_json()` to `scripts/lib/normalize.py`
|
||||
- Convert `.json` response format → `RedditItem` schema
|
||||
- `created_utc` (epoch) → `YYYY-MM-DD` string
|
||||
- `permalink` → full URL
|
||||
- Position-based relevance: `1.0 - (index / total * 0.5)` → range [1.0, 0.5]
|
||||
- Date confidence: `"high"` (Reddit provides exact timestamps)
|
||||
- [ ] Update `scripts/lib/score.py` if needed
|
||||
- `.json` results already have real engagement metrics — existing scoring formula works as-is
|
||||
- No penalty needed (unlike WebSearch which lacks engagement)
|
||||
- [ ] Add tests for normalization + scoring of `.json` data
|
||||
|
||||
### Phase 3: Integration into Orchestrator
|
||||
|
||||
- [ ] Update `scripts/lib/env.py`
|
||||
- Add `get_reddit_source(config) -> str` returning `'json'`, `'openai'`, or `'none'`
|
||||
- Priority: `.json` always available (default), `'openai'` if key present and user prefers
|
||||
- Add `REDDIT_SOURCE` config option: `auto` (default), `json`, `openai`
|
||||
- Update `get_available_sources()` to always include Reddit (since `.json` is free)
|
||||
- Update `get_missing_keys()` — Reddit no longer shows as "missing"
|
||||
- [ ] Update `scripts/last30days.py`
|
||||
- Add `_search_reddit_json()` function alongside existing `_search_reddit()`
|
||||
- Route based on `get_reddit_source()`: json → `_search_reddit_json()`, openai → `_search_reddit()`
|
||||
- Skip `reddit_enrich.py` for `.json` results (already have real engagement metrics!)
|
||||
- Update progress/stats output to show source: "Reddit (free)" vs "Reddit (OpenAI)"
|
||||
- [ ] Update SKILL.md promo messaging
|
||||
- Remove "Add OPENAI_API_KEY for Reddit" messaging
|
||||
- Instead: "Reddit search included free! Add OpenAI key for AI-enhanced relevance scoring."
|
||||
- [ ] Integration tests: full pipeline with `.json` mock data
|
||||
|
||||
### Phase 4: Testing & Polish
|
||||
|
||||
- [ ] Run all existing tests — ensure backward compatibility
|
||||
- [ ] Manual test: invoke `/last30days` with NO API keys → should get Reddit + WebSearch results
|
||||
- [ ] Manual test: invoke with OPENAI_API_KEY → should still use OpenAI path (backward compat)
|
||||
- [ ] Manual test: compare result quality — `.json` vs OpenAI for same topic
|
||||
- [ ] Update README.md — document free Reddit access
|
||||
- [ ] Update SPEC.md — document new source priority chain
|
||||
|
||||
### Phase 5 (Future): MCP Enhancement Layer
|
||||
|
||||
- [ ] Detect Reddit MCP in Claude's session
|
||||
- [ ] SKILL.md pre-fetches via MCP `search_reddit` → saves to temp file
|
||||
- [ ] Python script reads pre-fetched MCP data via `--reddit-from=` flag
|
||||
- [ ] MCP results get AI-judged relevance (since Claude sees them)
|
||||
- [ ] Better than `.json` position-based relevance
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### Functional
|
||||
- [ ] `/last30days "any topic"` returns Reddit results with **zero API keys configured**
|
||||
- [ ] Reddit results include real engagement metrics (score, comments, upvote_ratio)
|
||||
- [ ] Results are properly scored, deduped, and rendered (same quality as OpenAI path)
|
||||
- [ ] Existing OpenAI path still works when key is present
|
||||
- [ ] Existing X search (Bird CLI / xAI) is unchanged
|
||||
- [ ] Stats box shows correct source indicator ("Reddit (free)" or "Reddit (OpenAI)")
|
||||
|
||||
### Non-Functional
|
||||
- [ ] No external Python packages (stdlib only — `urllib.request`, `json`, `time`)
|
||||
- [ ] Respects Reddit rate limits (~10 req/min for unauthenticated)
|
||||
- [ ] Graceful degradation if Reddit blocks requests (429/403 → empty results + error msg)
|
||||
- [ ] All new code has unit tests with fixtures (no live API calls in tests)
|
||||
|
||||
### Quality Gates
|
||||
- [ ] All existing tests pass (zero regressions)
|
||||
- [ ] New tests cover: search parsing, normalization, scoring, error handling, pagination
|
||||
- [ ] Manual smoke test passes with zero API keys
|
||||
|
||||
## Risk Analysis
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| Reddit blocks `.json` endpoints | Low | High | Fall back to OpenAI path; monitor for 429s |
|
||||
| `.json` rate limit too restrictive | Medium | Medium | 1s delay between requests; cache results |
|
||||
| Relevance quality lower without AI scoring | Medium | Medium | Reddit's `sort=relevance` is decent; can add keyword scoring later |
|
||||
| Reddit changes `.json` response format | Low | Medium | Schema validation in normalize; fixture-based tests catch changes |
|
||||
| Node.js MCP server dependency conflicts | N/A (Phase 5) | Low | MCP is optional enhancement, not required |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **None** — this feature uses only Python stdlib and Reddit's public `.json` endpoints
|
||||
- Phase 5 (future) would add optional dependency on an MCP server
|
||||
|
||||
## Repository Setup
|
||||
|
||||
```bash
|
||||
# Fork into new working repo
|
||||
cp -r /Users/mvanhorn/last30days-skill-private /Users/mvanhorn/last30days-free-reddit
|
||||
cd /Users/mvanhorn/last30days-free-reddit
|
||||
|
||||
# Create feature branch
|
||||
git checkout -b feat/free-reddit
|
||||
|
||||
# Verify existing tests
|
||||
python3 -m pytest tests/ -v
|
||||
```
|
||||
|
||||
The new repo is disposable — once the feature is validated, changes merge back into `last30days-skill-private` via PR or cherry-pick.
|
||||
|
||||
## References
|
||||
|
||||
### Internal
|
||||
- `scripts/lib/openai_reddit.py` — Current Reddit search (to be replaced/supplemented)
|
||||
- `scripts/lib/env.py:get_available_sources()` — Source detection logic to update
|
||||
- `scripts/lib/normalize.py` — Add `.json` normalizer alongside existing
|
||||
- `scripts/lib/reddit_enrich.py` — May be skippable for `.json` results (already have engagement)
|
||||
- `scripts/lib/schema.py:RedditItem` — Target schema (unchanged)
|
||||
|
||||
### External
|
||||
- [Reddit .json search endpoint](https://www.reddit.com/search.json?q=test&sort=relevance&t=month&limit=25)
|
||||
- [Simon Willison — Scraping Reddit via JSON API](https://til.simonwillison.net/reddit/scraping-reddit-json)
|
||||
- [karanb192/reddit-mcp-buddy](https://github.com/karanb192/reddit-mcp-buddy) — Best MCP option for Phase 5
|
||||
- [Hawstein/mcp-server-reddit](https://github.com/Hawstein/mcp-server-reddit) — ClaudeLog-recommended MCP (no search though)
|
||||
- [Reddit API Rate Limits Guide](https://painonsocial.com/blog/reddit-api-rate-limits-guide)
|
||||
|
||||
### Research Sources
|
||||
- last30days skill output — community recommendations for Reddit search tools
|
||||
- GitHub search — 10+ Reddit MCP repos evaluated
|
||||
- npm/PyPI registries — package availability confirmed
|
||||
- ClaudeLog — [Reddit MCP reference](https://claudelog.com/claude-code-mcps/reddit-mcp/)
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
title: "release: Push V2 to public repo and launch"
|
||||
type: release
|
||||
date: 2026-02-07
|
||||
---
|
||||
|
||||
# release: Push V2 to public repo and launch
|
||||
|
||||
## Overview
|
||||
|
||||
Push all V2 changes from the private development repo to the public GitHub repo, then sync the installed skill. 52 commits need to go from `origin/main` (private) to `upstream/main` (public).
|
||||
|
||||
## Current State
|
||||
|
||||
- **Private repo:** `https://github.com/mvanhorn/last30days-skill-private` - 52 commits ahead of public
|
||||
- **Public repo:** `https://github.com/mvanhorn/last30days-skill` - last commit is V1 (`cc892d7`)
|
||||
- **Upstream remote:** Already configured in private repo
|
||||
- **Untracked files in private:** test logs, draft plans - these should NOT be pushed
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1: Clean up private repo
|
||||
|
||||
- [ ] Review untracked files - make sure nothing sensitive gets pushed
|
||||
- [ ] Decide: commit the untracked docs/plans or leave them out of the push
|
||||
|
||||
### Step 2: Push to public
|
||||
|
||||
```bash
|
||||
cd /Users/mvanhorn/last30days-skill-private
|
||||
git push upstream main
|
||||
```
|
||||
|
||||
This pushes all 52 commits from private `main` to public `main`. Since the private repo was forked from public, history is shared - this is a fast-forward push.
|
||||
|
||||
### Step 3: Sync installed skill
|
||||
|
||||
```bash
|
||||
# Update the installed copy that Claude Code actually uses
|
||||
cd ~/.claude/skills/last30days
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
### Step 4: Verify
|
||||
|
||||
- [ ] Check `https://github.com/mvanhorn/last30days-skill` shows V2 README with new examples
|
||||
- [ ] Check SKILL.md has the new argument-hint and timing disclaimer
|
||||
- [ ] Check bird_x.py has the fixed `_extract_core_subject()` and retry logic
|
||||
- [ ] Run a quick `/last30days` test to confirm installed skill works
|
||||
|
||||
## What Gets Published
|
||||
|
||||
Key files going public:
|
||||
- `README.md` - V2 features, 3 new examples (Nano Banana Pro, Kanye, Vibe Motion), speed tradeoff note
|
||||
- `SKILL.md` - New argument-hint, timing disclaimer, citation rules
|
||||
- `scripts/lib/bird_x.py` - Fixed query construction + retry logic
|
||||
- `scripts/lib/http.py` - USER_AGENT bumped to 2.0
|
||||
- `.claude-plugin/plugin.json` - Marketplace support
|
||||
- All Bird CLI integration code
|
||||
- Phase 2 supplemental search code
|
||||
- Model fallback chain
|
||||
|
||||
## Risks
|
||||
|
||||
- **Low risk:** This is a fast-forward push, no force push needed
|
||||
- **Public API keys:** Already confirmed - no `.env` files or secrets in the repo
|
||||
- **Untracked files:** Won't be pushed unless committed first
|
||||
@@ -0,0 +1,194 @@
|
||||
---
|
||||
title: ClawHub Scanner Compliance for last30days-official
|
||||
type: feat
|
||||
date: 2026-02-15
|
||||
---
|
||||
|
||||
# ClawHub Scanner Compliance for last30days-official
|
||||
|
||||
## Overview
|
||||
|
||||
Make the last30days skill pass ClawHub's security scanner (VirusTotal + Code Insight) so it can be published as `last30days-official`. The user's 8 other mvanhorn skills already pass - we replicate their exact pattern.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
ClawHub requires skills to pass a multi-layer security scan before publication:
|
||||
1. Metadata validation (frontmatter fields)
|
||||
2. VirusTotal automated scanning
|
||||
3. LLM-powered Code Insight analysis (checks if capabilities match documentation)
|
||||
4. Credential handling review
|
||||
|
||||
The current last30days SKILL.md has basic `metadata.clawdbot` but is missing fields the scanner checks: `emoji`, `user-invocable`, `disable-model-invocation`, `files` declaration. There's no `## Security & Permissions` section (required pattern from passing skills). README has no security/privacy documentation.
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
Follow the exact pattern from `clawdbot-skill-xai` and `clawdbot-skill-search-x` (both pass the scanner). Three files need changes.
|
||||
|
||||
### Fix 1: SKILL.md Frontmatter
|
||||
|
||||
Add missing scanner fields to the existing frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: last30days
|
||||
version: "2.1"
|
||||
description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, web."
|
||||
argument-hint: 'last30 AI video tools, last30 best project management tools'
|
||||
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
|
||||
homepage: https://github.com/mvanhorn/last30days-skill
|
||||
user-invocable: true
|
||||
disable-model-invocation: true
|
||||
metadata:
|
||||
clawdbot:
|
||||
emoji: "📰"
|
||||
requires:
|
||||
env:
|
||||
- OPENAI_API_KEY
|
||||
bins:
|
||||
- node
|
||||
- python3
|
||||
primaryEnv: OPENAI_API_KEY
|
||||
files:
|
||||
- "scripts/*"
|
||||
homepage: https://github.com/mvanhorn/last30days-skill
|
||||
tags:
|
||||
- research
|
||||
- reddit
|
||||
- x
|
||||
- youtube
|
||||
- trends
|
||||
- prompts
|
||||
---
|
||||
```
|
||||
|
||||
Key additions:
|
||||
- `user-invocable: true` - human must trigger it
|
||||
- `disable-model-invocation: true` - agent cannot self-trigger
|
||||
- `emoji: "📰"` - required display field
|
||||
- `files: ["scripts/*"]` - prevents false "instruction-only but has scripts" flag
|
||||
|
||||
### Fix 2: Security & Permissions Section in SKILL.md
|
||||
|
||||
Add to the bottom of SKILL.md (matches xai/search-x pattern exactly):
|
||||
|
||||
```markdown
|
||||
## Security & Permissions
|
||||
|
||||
**What this skill does:**
|
||||
- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery
|
||||
- Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
|
||||
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
|
||||
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
|
||||
- Fetches public Reddit thread data from `reddit.com` for engagement metrics
|
||||
- Stores research findings in local SQLite database (watchlist mode only)
|
||||
|
||||
**What this skill does NOT do:**
|
||||
- Does not post, like, or modify content on any platform
|
||||
- Does not access your Reddit, X, or YouTube accounts
|
||||
- Does not share API keys between providers (OpenAI key only goes to api.openai.com, etc.)
|
||||
- Does not log, cache, or write API keys to output files
|
||||
- Does not send data to any endpoint not listed above
|
||||
- Cannot be invoked autonomously by the agent (`disable-model-invocation: true`)
|
||||
|
||||
**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
|
||||
|
||||
Review scripts before first use to verify behavior.
|
||||
```
|
||||
|
||||
### Fix 3: Security & Privacy Section in README.md
|
||||
|
||||
Add after the "How It Works" section:
|
||||
|
||||
```markdown
|
||||
## Security & Privacy
|
||||
|
||||
### Data that leaves your machine
|
||||
|
||||
| Destination | Data Sent | API Key Required |
|
||||
|------------|-----------|-----------------|
|
||||
| `api.openai.com` | Search query (topic string) | OPENAI_API_KEY |
|
||||
| `reddit.com` | Thread URLs for enrichment | None (public JSON) |
|
||||
| Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY |
|
||||
| `youtube.com` (via yt-dlp) | Search query | None (public search) |
|
||||
| `api.search.brave.com` | Search query (optional) | BRAVE_API_KEY |
|
||||
| `api.parallel.ai` | Search query (optional) | PARALLEL_API_KEY |
|
||||
| `openrouter.ai` | Search query (optional) | OPENROUTER_API_KEY |
|
||||
|
||||
Your research topic is included in all outbound API requests. If you research sensitive topics, be aware that query strings are transmitted to the API providers listed above.
|
||||
|
||||
### Data stored locally
|
||||
|
||||
- API keys: `~/.config/last30days/.env` (chmod 600 recommended)
|
||||
- Watchlist database: `~/.local/share/last30days/research.db` (SQLite)
|
||||
- Briefings: `~/.local/share/last30days/briefs/`
|
||||
|
||||
### API key isolation
|
||||
|
||||
Each API key is transmitted only to its respective endpoint. Your OpenAI key is never sent to xAI, Brave, or any other provider. Browser cookies for X are read locally and used only for Twitter GraphQL requests.
|
||||
```
|
||||
|
||||
### Fix 4: Update .claude-plugin Files
|
||||
|
||||
**plugin.json** - bump version, add youtube keyword:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "last30days",
|
||||
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
|
||||
"version": "2.1.0",
|
||||
"author": {"name": "mvanhorn"},
|
||||
"repository": "https://github.com/mvanhorn/last30days-skill",
|
||||
"license": "MIT",
|
||||
"keywords": ["research", "reddit", "twitter", "x", "youtube", "trends", "prompts"],
|
||||
"skills": ["./"]
|
||||
}
|
||||
```
|
||||
|
||||
**marketplace.json** - add version, update description:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "last30days",
|
||||
"owner": {"name": "mvanhorn", "url": "https://github.com/mvanhorn"},
|
||||
"metadata": {
|
||||
"description": "Research any topic from the last 30 days across Reddit, X, YouTube, and the web",
|
||||
"version": "2.1.0"
|
||||
},
|
||||
"plugins": [{"name": "last30days", "source": "."}]
|
||||
}
|
||||
```
|
||||
|
||||
## What We DON'T Need
|
||||
|
||||
Based on the audit of your 8 passing skills:
|
||||
- **Script-level security manifest headers** - your passing skills (xai, search-x, parallel) do NOT have these. The scanner relies on SKILL.md, not per-file headers.
|
||||
- **Separate SECURITY.md file** - not needed; README section + SKILL.md section is sufficient.
|
||||
- **Shell injection fixes** - already clean. All subprocess calls use list-form args, no `shell=True` anywhere.
|
||||
- **Credential leak fixes** - already clean. All keys loaded from env vars, none hardcoded or logged.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] SKILL.md frontmatter has `user-invocable`, `disable-model-invocation`, `emoji`, `files`
|
||||
- [x] SKILL.md has `## Security & Permissions` section with "does" and "does NOT do" lists
|
||||
- [x] README.md has `## Security & Privacy` section with endpoint table and key isolation docs
|
||||
- [x] plugin.json version bumped to 2.1.0, youtube keyword added
|
||||
- [x] marketplace.json version and description updated
|
||||
- [x] `python3 scripts/last30days.py --diagnose` still works after changes
|
||||
- [x] Synced to all installed skill locations
|
||||
- [ ] Published to ClawHub as `last30days-official` (when auth is fixed)
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `SKILL.md` | Add frontmatter fields + Security & Permissions section |
|
||||
| `README.md` | Add Security & Privacy section |
|
||||
| `.claude-plugin/plugin.json` | Bump version, add youtube keyword |
|
||||
| `.claude-plugin/marketplace.json` | Add version, update description |
|
||||
|
||||
## References
|
||||
|
||||
- [13-point ClawHub checklist](https://gist.github.com/adhishthite/0db995ecfe2f23e09d0b2d418491982c)
|
||||
- [ClawHub docs](https://docs.openclaw.ai/tools/clawhub)
|
||||
- [ClawHub Developer Guide 2026](https://www.digitalapplied.com/blog/clawhub-skills-marketplace-developer-guide-2026)
|
||||
- Your passing skills: `clawdbot-skill-xai`, `clawdbot-skill-search-x` (exact pattern replicated)
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: "fix: watchlist engagement null crash"
|
||||
type: fix
|
||||
date: 2026-02-15
|
||||
---
|
||||
|
||||
# fix: Watchlist crashes on `engagement: null` from X posts
|
||||
|
||||
## Problem
|
||||
|
||||
When X posts return `engagement: null` (JSON null) instead of `engagement: {}` (empty object), the watchlist `_run_topic` findings parser crashes. This is because Python's `dict.get("engagement", {})` returns `None` when the key **exists** with value `None` — the default `{}` only applies when the key is **missing**.
|
||||
|
||||
```python
|
||||
# CRASHES — .get() returns None, not {}
|
||||
item.get("engagement", {}).get("likes", 0)
|
||||
# AttributeError: 'NoneType' object has no attribute 'get'
|
||||
```
|
||||
|
||||
**Reporter:** Soft launch tester, 2026-02-15
|
||||
**Severity:** Medium — breaks watchlist `run-one` and `run-all` for any topic that pulls X posts with null engagement
|
||||
**One-shot research unaffected** — the main `last30days.py` pipeline uses `normalize.py` which has `isinstance(eng_raw, dict)` guards
|
||||
|
||||
## Root Cause
|
||||
|
||||
Two locations use the vulnerable `dict.get("key", {})` pattern:
|
||||
|
||||
1. **`scripts/watchlist.py:188`** — THE REPORTED BUG
|
||||
```python
|
||||
"engagement_score": item.get("engagement", {}).get("likes", 0),
|
||||
```
|
||||
|
||||
2. **`scripts/lib/normalize.py:177`** — YouTube normalizer (same pattern, latent)
|
||||
```python
|
||||
eng_raw = item.get("engagement", {})
|
||||
```
|
||||
|
||||
Three other locations are already safe — they use `isinstance(eng_raw, dict)`:
|
||||
- `scripts/lib/normalize.py:70` (Reddit)
|
||||
- `scripts/lib/normalize.py:130` (X)
|
||||
- `scripts/lib/xai_x.py:190`
|
||||
|
||||
## Fix
|
||||
|
||||
Apply the `or {}` idiom (as suggested by reporter):
|
||||
|
||||
### scripts/watchlist.py:188
|
||||
|
||||
```python
|
||||
# Before
|
||||
"engagement_score": item.get("engagement", {}).get("likes", 0),
|
||||
# After
|
||||
"engagement_score": (item.get("engagement") or {}).get("likes", 0),
|
||||
```
|
||||
|
||||
### scripts/lib/normalize.py:177
|
||||
|
||||
```python
|
||||
# Before
|
||||
eng_raw = item.get("engagement", {})
|
||||
# After
|
||||
eng_raw = item.get("engagement") or {}
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `watchlist.py run-one` handles X posts with `engagement: null` without crashing
|
||||
- [ ] `watchlist.py run-one` handles X posts with `engagement: {}` (empty object)
|
||||
- [ ] `watchlist.py run-one` handles X posts with no engagement key at all
|
||||
- [ ] YouTube normalizer handles `engagement: null` without crashing
|
||||
- [ ] Existing tests still pass
|
||||
|
||||
## Regarding the screenshot question
|
||||
|
||||
The tester asked "Did you use watchlist on open claw or Claude?" — watchlist is designed for the **open variant** (Open Claw). It works in Claude Code too since it's just Python + SQLite, but the SKILL.md routing for watchlist commands is only in `variants/open/SKILL.md`. The main `SKILL.md` is one-shot research only.
|
||||
|
||||
Answer to give tester: "Watchlist works in both — it's plain Python. But the skill routing that understands 'watch add topic' is in the open variant. In Claude Code you'd need to call the script directly or use the open variant SKILL.md."
|
||||
@@ -0,0 +1,221 @@
|
||||
---
|
||||
title: Fix YouTube Display and Search Quality
|
||||
type: fix
|
||||
date: 2026-02-15
|
||||
---
|
||||
|
||||
# Fix YouTube Display and Search Quality
|
||||
|
||||
## Overview
|
||||
|
||||
YouTube is the v2.1 headline feature but it's broken in two ways: results don't appear in Claude's synthesis (display bug), and search quality is worse than youtube.com (search bug). Both need fixing before launch.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
**Display bug:** YouTube data exists in the script output but Claude never sees it. Reproduced on 4/5 recent test runs (Kanye, Seedance 2, Peter Steinberger, YouTube thumbnails). The skill worked once — the earlier "YouTube thumbnails" and "OpenClaw" runs showed YouTube stats — but subsequent runs silently dropped it.
|
||||
|
||||
**Search quality bug:** User searched "how to get on seedance 2" on youtube.com and got multiple recent results. yt-dlp returned 10 videos for the same query, but they included old irrelevant content because date filtering is broken.
|
||||
|
||||
## Root Cause Analysis
|
||||
|
||||
### Display Bug — Three compounding causes
|
||||
|
||||
1. **`2>&1` in SKILL.md bash command** (line 79) merges stderr progress messages into stdout. When Claude Code receives this mixed output, the YouTube section (which renders LAST after Reddit + X) can get lost in the noise or hit the 30K char Bash output limit.
|
||||
|
||||
2. **Background execution** (partially fixed). The old SKILL.md said "DO WEBSEARCH WHILE SCRIPT RUNS" which caused Claude to background the bash command. Backgrounded commands return truncated output via Task Output. *Already fixed in this session — SKILL.md now says FOREGROUND with 5-minute timeout.*
|
||||
|
||||
3. **No explicit model instruction to look for YouTube.** The SKILL.md tells Claude to synthesize but doesn't emphasize that YouTube data is in the script output and must be included.
|
||||
|
||||
### Search Quality Bug — `--flat-playlist` breaks date filtering
|
||||
|
||||
The yt-dlp command in `youtube_yt.py:110-116`:
|
||||
```bash
|
||||
yt-dlp ytsearch{count}:{query} --dateafter {YYYYMMDD} --flat-playlist --dump-json
|
||||
```
|
||||
|
||||
**`--flat-playlist` causes three problems:**
|
||||
1. `--dateafter` is silently ignored (no video-level metadata to filter on)
|
||||
2. All items have `date: None` (upload_date not in flat-playlist JSON)
|
||||
3. Old content leaks in (e.g., "the greatest youtube thumbnails of all time" returned for a 30-day query)
|
||||
|
||||
**`_extract_core_subject()` over-strips useful YouTube terms:**
|
||||
- Strips "tips", "tutorial", "review" — but these ARE the content types people search for on YouTube
|
||||
- "youtube thumbnail tips" → "youtube thumbnail" loses the intent signal
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
### Phase 1: Fix Display (Critical — blocks launch)
|
||||
|
||||
#### 1a. Remove `2>&1` from SKILL.md bash command
|
||||
|
||||
**File:** `SKILL.md:79` (both `last30days-skill-private/SKILL.md` and `~/.claude/skills/last30days21/SKILL.md`)
|
||||
|
||||
```bash
|
||||
# Before:
|
||||
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact 2>&1
|
||||
|
||||
# After:
|
||||
python3 "${SKILL_ROOT}/scripts/last30days.py" "$ARGUMENTS" --emit=compact
|
||||
```
|
||||
|
||||
This removes ~1-5KB of progress spam from the model's input and ensures clean stdout-only output.
|
||||
|
||||
#### 1b. Add YouTube-specific synthesis instruction to SKILL.md
|
||||
|
||||
After the "Read the ENTIRE output" instruction, add:
|
||||
|
||||
```markdown
|
||||
**The script output has THREE sections: Reddit items, X items, and YouTube items (in that order).
|
||||
If you see YouTube items in the output, you MUST include them in your synthesis and stats block.
|
||||
YouTube items look like: `**{video_id}** (score:N) {channel} [N views, N likes]`**
|
||||
```
|
||||
|
||||
#### 1c. Verify fix with test run
|
||||
|
||||
Run `/last30days21 youtube thumbnail tips` and confirm YouTube appears in stats.
|
||||
|
||||
### Phase 2: Fix Search Quality (High — headline feature quality)
|
||||
|
||||
#### 2a. Remove `--flat-playlist` flag
|
||||
|
||||
**File:** `scripts/lib/youtube_yt.py:110-116`
|
||||
|
||||
```python
|
||||
# Before:
|
||||
cmd = [
|
||||
"yt-dlp",
|
||||
f"ytsearch{count}:{core_topic}",
|
||||
"--dateafter", date_filter,
|
||||
"--flat-playlist",
|
||||
"--dump-json",
|
||||
]
|
||||
|
||||
# After:
|
||||
cmd = [
|
||||
"yt-dlp",
|
||||
f"ytsearch{count}:{core_topic}",
|
||||
"--dateafter", date_filter,
|
||||
"--dump-json",
|
||||
"--no-warnings",
|
||||
"--no-download",
|
||||
]
|
||||
```
|
||||
|
||||
**Impact:** Slower (yt-dlp resolves each video page for metadata) but:
|
||||
- `--dateafter` actually works — filters old content
|
||||
- `upload_date` populated — items get real dates
|
||||
- Engagement metrics more accurate
|
||||
|
||||
**Risk:** Could increase search time from ~5s to ~30-60s for 20 videos. Mitigate by reducing default count or increasing timeout.
|
||||
|
||||
**Alternative if too slow:** Keep `--flat-playlist` but append year to search query:
|
||||
```python
|
||||
# Bias toward recent content since --dateafter doesn't work with flat-playlist
|
||||
search_query = f"{core_topic} {from_date[:4]}" # e.g., "youtube thumbnail 2026"
|
||||
```
|
||||
|
||||
#### 2b. Fix `_extract_core_subject()` for YouTube-relevant terms
|
||||
|
||||
**File:** `scripts/lib/youtube_yt.py:67-76`
|
||||
|
||||
Don't strip terms that are useful YouTube content type signals:
|
||||
|
||||
```python
|
||||
# YouTube-specific: keep 'tips', 'tutorial', 'review' etc.
|
||||
# These are stripped for Reddit/X search but are valuable for YouTube
|
||||
noise = {
|
||||
'best', 'top', 'good', 'great', 'awesome', 'killer',
|
||||
'latest', 'new', 'news', 'update', 'updates',
|
||||
'trending', 'hottest', 'popular', 'viral',
|
||||
'practices', 'features',
|
||||
'recommendations', 'advice',
|
||||
'prompt', 'prompts', 'prompting',
|
||||
'methods', 'strategies', 'approaches',
|
||||
}
|
||||
# NOTE: 'tips', 'tricks', 'tutorial', 'guide', 'review', 'reviews'
|
||||
# are intentionally KEPT — they're YouTube content types
|
||||
```
|
||||
|
||||
#### 2c. Clean question marks and trailing punctuation
|
||||
|
||||
**File:** `scripts/lib/youtube_yt.py:48-80`
|
||||
|
||||
```python
|
||||
# At the end of _extract_core_subject():
|
||||
result = ' '.join(filtered) if filtered else text
|
||||
return result.rstrip('?!.') # Clean trailing punctuation
|
||||
```
|
||||
|
||||
### Phase 3: Polish (Medium — nice to have before launch)
|
||||
|
||||
#### 3a. Increase subprocess timeout for non-flat-playlist mode
|
||||
|
||||
**File:** `scripts/lib/youtube_yt.py:119-121`
|
||||
|
||||
```python
|
||||
# Before:
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
|
||||
|
||||
# After — resolving video pages takes longer:
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
|
||||
```
|
||||
|
||||
#### 3b. Add year hint to search query for recency bias
|
||||
|
||||
Even with `--dateafter` working, YouTube's search algorithm ranks by relevance not recency. Adding the year helps:
|
||||
|
||||
```python
|
||||
# After core_topic extraction:
|
||||
import datetime
|
||||
current_year = datetime.datetime.now().year
|
||||
search_query = f"{core_topic} {current_year}"
|
||||
```
|
||||
|
||||
#### 3c. Reduce compact render item count for YouTube
|
||||
|
||||
The compact render currently shows up to 15 YouTube items. With transcripts, this is too much output. Reduce to 10:
|
||||
|
||||
**File:** `scripts/lib/render.py` — in `render_compact()`, add YouTube-specific limit or reduce the default.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `/last30days21 youtube thumbnail tips` shows YouTube in stats block
|
||||
- [ ] YouTube items have real dates (not `None`)
|
||||
- [ ] Old content (> 30 days) filtered out by `--dateafter`
|
||||
- [ ] "youtube thumbnail tips" search returns videos about thumbnail tips (not generic old content)
|
||||
- [ ] "How to access Seedance 2" returns recent Seedance 2 tutorials
|
||||
- [ ] Script completes within 5 minutes for default depth
|
||||
- [ ] No `2>&1` in SKILL.md bash command
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `SKILL.md` | Remove `2>&1`, add YouTube synthesis instruction |
|
||||
| `scripts/lib/youtube_yt.py` | Remove `--flat-playlist`, fix noise words, add year hint, increase timeout |
|
||||
| `scripts/lib/render.py` | Optional: reduce YouTube item limit in compact mode |
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Quick smoke test — should show YouTube items with dates
|
||||
cd ~/.claude/skills/last30days21
|
||||
python3 scripts/last30days.py "youtube thumbnail tips" --quick --emit=compact 2>/dev/null | grep -c "youtube.com"
|
||||
|
||||
# Date test — should show YYYY-MM-DD dates, not None
|
||||
python3 -c "
|
||||
from scripts.lib import youtube_yt
|
||||
r = youtube_yt.search_youtube('youtube thumbnail tips', '2026-01-16', '2026-02-15', depth='quick')
|
||||
for v in r['items'][:3]: print(v['date'], v['title'][:50])
|
||||
"
|
||||
|
||||
# Full integration — run the skill in Claude Code and verify YouTube in stats
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- SKILL.md bash command: `scripts/last30days.py` line 79
|
||||
- YouTube search: `scripts/lib/youtube_yt.py` lines 83-174
|
||||
- Core subject extraction: `scripts/lib/youtube_yt.py` lines 48-80
|
||||
- Compact render: `scripts/lib/render.py` lines 48-238
|
||||
- Prior YouTube plan: `docs/plans/2026-02-14-feat-youtube-transcript-search-plan.md`
|
||||
@@ -0,0 +1,102 @@
|
||||
---
|
||||
title: Fix YouTube Timeout and Reddit Resilience
|
||||
type: fix
|
||||
date: 2026-02-15
|
||||
---
|
||||
|
||||
# Fix YouTube Timeout and Reddit Resilience
|
||||
|
||||
## Overview
|
||||
|
||||
YouTube search+transcript fetching exceeds the 60s future timeout on popular topics (20 videos + 5 transcripts), discarding all results. Reddit 429 rate-limiting burns through the entire time budget with aggressive retries, causing global timeouts. Both issues cause the script to return incomplete results and sometimes crash.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
**YouTube timeout (blocks launch):** YouTube found 20 Seedance 2.0 videos and fetched 4/5 transcripts, but the 60s future timeout killed everything — `✗ Error: YouTube search timed out after 60s`. The data was there; the budget wasn't.
|
||||
|
||||
**Reddit 429 cascade (degrades reliability):** One enrichment item hitting 429 burns up to 93s (3 retries x 30s timeout + backoff) against a 45s budget. Phase 2 supplemental search then also 429s, burning another 30s. Total wasted: 75s on doomed requests, triggering the 180s global timeout.
|
||||
|
||||
Observed failures across 6 test runs:
|
||||
- `seedance 2 access`: YouTube timed out (60s), Reddit 0 threads
|
||||
- `kanye west bully`: Global timeout first run, needed --quick retry
|
||||
- `Peter Steinberger`: Global timeout during enrichment
|
||||
- `nano banana pro`: Reddit timed out, YouTube worked (5 videos in time)
|
||||
- `kanye west bully` (retry): Reddit 0 threads, YouTube 4 videos
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
### Fix 1: Bump YouTube future timeout (NOT YET DONE)
|
||||
|
||||
**File:** `scripts/last30days.py:40-44`
|
||||
|
||||
Give YouTube its own timeout, separate from the shared `future` timeout. YouTube inherently takes longer because it does search + parallel transcript fetching.
|
||||
|
||||
```python
|
||||
# Option A: YouTube-specific timeout key
|
||||
TIMEOUT_PROFILES = {
|
||||
"quick": {"global": 90, "future": 30, "youtube_future": 60, ...},
|
||||
"default": {"global": 180, "future": 60, "youtube_future": 90, ...},
|
||||
"deep": {"global": 300, "future": 90, "youtube_future": 120, ...},
|
||||
}
|
||||
|
||||
# Option B: Simpler — just bump default future to 90 for all sources
|
||||
TIMEOUT_PROFILES = {
|
||||
"quick": {"global": 90, "future": 45, ...},
|
||||
"default": {"global": 180, "future": 90, ...},
|
||||
"deep": {"global": 300, "future": 120, ...},
|
||||
}
|
||||
```
|
||||
|
||||
**Recommendation:** Option A (YouTube-specific key) — keeps Reddit/X futures tight while giving YouTube the breathing room it needs. Reddit/X finish in 20-40s; YouTube needs 60-90s for transcript fetching.
|
||||
|
||||
Then where YouTube future is collected (~line 646):
|
||||
```python
|
||||
youtube_timeout = timeouts.get("youtube_future", timeouts["future"])
|
||||
youtube_items, youtube_error = youtube_future.result(timeout=youtube_timeout)
|
||||
```
|
||||
|
||||
### Fix 2: Reddit 429 fail-fast (ALREADY DONE — needs commit)
|
||||
|
||||
**Status:** Implemented in working tree, uncommitted. Changes across 4 files:
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `scripts/lib/http.py` | `get_reddit_json()` accepts `timeout` and `retries` params (was hardcoded 30s/3) |
|
||||
| `scripts/lib/reddit_enrich.py` | `RedditRateLimitError` exception; `fetch_thread_data()` propagates 429; `enrich_reddit_item()` defaults to 10s timeout / 1 retry |
|
||||
| `scripts/lib/openai_reddit.py` | `search_subreddits()` reduced to 1 retry; breaks subreddit loop on first 429 |
|
||||
| `scripts/last30days.py` | Enrichment loop catches `RedditRateLimitError`, cancels futures, bails; `rate_limited` flag skips Phase 2 Reddit |
|
||||
|
||||
**Impact:** 429 scenario drops from ~75s wasted to ~12s wasted.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] YouTube `seedance 2 access` query completes with videos (was timing out at 60s)
|
||||
- [ ] YouTube `kanye west bully` query returns 4+ videos with transcripts
|
||||
- [ ] Reddit 429 detected within ~12s, remaining enrichment skipped
|
||||
- [ ] Phase 2 Reddit skipped when rate-limited
|
||||
- [ ] No global timeout on default depth for any of the 5 test queries
|
||||
- [ ] All changes committed and synced to `~/.claude/skills/last30days21/` and `~/.claude/skills/last30days/`
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Status | Changes |
|
||||
|------|--------|---------|
|
||||
| `scripts/last30days.py` | Modify (partially done) | Add `youtube_future` to TIMEOUT_PROFILES; use it for YouTube future collection |
|
||||
| `scripts/lib/http.py` | Done (uncommitted) | Parameterized `get_reddit_json()` timeout/retries |
|
||||
| `scripts/lib/reddit_enrich.py` | Done (uncommitted) | `RedditRateLimitError`, fail-fast enrichment |
|
||||
| `scripts/lib/openai_reddit.py` | Done (uncommitted) | 429 early-bail in `search_subreddits()` |
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Quick smoke test — YouTube should complete within 90s
|
||||
cd ~/.claude/skills/last30days21
|
||||
python3 scripts/last30days.py "seedance 2 access" --emit=compact 2>&1 | grep -E "YouTube|timeout"
|
||||
|
||||
# Verify YouTube items in output
|
||||
python3 scripts/last30days.py "kanye west bully" --quick --emit=compact 2>/dev/null | grep "youtube.com" | wc -l
|
||||
|
||||
# Full integration — run the skill in Claude Code
|
||||
# /last30days21 seedance 2 access
|
||||
# Verify YouTube appears in stats block
|
||||
```
|
||||
@@ -0,0 +1,255 @@
|
||||
---
|
||||
title: "feat: Create GitHub Release for v2.1"
|
||||
type: feat
|
||||
date: 2026-02-17
|
||||
---
|
||||
|
||||
# Create GitHub Release for v2.1
|
||||
|
||||
## Overview
|
||||
|
||||
The last30days skill is at v2.1 with 2,747 stars and 317 forks, but has **zero git tags and zero GitHub Releases**. The code is already live on the public repo (`upstream/main`). All marketing copy is written. This plan creates a proper v2.1.0 GitHub Release from existing materials.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
GitHub Releases provide:
|
||||
- A landing page for each version with formatted release notes
|
||||
- Discoverability (GitHub shows releases in the sidebar, feeds them to the Explore page)
|
||||
- A stable reference point for users (`git checkout v2.1.0`)
|
||||
- A "Full Changelog" diff link
|
||||
- RSS feed for watchers
|
||||
- New contributor callouts (community goodwill)
|
||||
|
||||
Currently users have no way to reference a specific version of the skill. The README says "v2.1" but there's nothing in git to anchor that.
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
Create the first GitHub Release (v2.1.0) on the **public** repo using existing marketing copy. Use an annotated tag (not lightweight) for proper `git describe` support and fork propagation.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Tag and Release
|
||||
|
||||
#### 1. Create CHANGELOG.md
|
||||
|
||||
Create `CHANGELOG.md` following [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format. Source material already exists:
|
||||
|
||||
- `README.md` "What's New in V2.1" and "What's New in V2" sections
|
||||
- `docs/v2.1-tweets.md` (raw verified test results, feature descriptions)
|
||||
- `docs/v2.1-launch-copy.md` (feature copy, social posts)
|
||||
- `docs/pr-credits.md` (contributor credits)
|
||||
- Git log (`git log --oneline` for commit references)
|
||||
|
||||
Structure:
|
||||
```markdown
|
||||
# Changelog
|
||||
|
||||
## [2.1.0] - 2026-02-15
|
||||
|
||||
### Highlights
|
||||
|
||||
30 days of research. 30 seconds of work. Four sources. Zero stale prompts.
|
||||
|
||||
Three headline features...
|
||||
|
||||
### Added
|
||||
- Open-class skill with watchlists (SQLite-backed, FTS5)
|
||||
- YouTube as 4th research source via yt-dlp (search + transcript extraction)
|
||||
- OpenAI Codex CLI compatibility ($last30days invocation)
|
||||
- Bundled X search (vendored Bird GraphQL client, no external CLI)
|
||||
- Native web search backends (Parallel AI, Brave, OpenRouter/Perplexity Sonar)
|
||||
- Briefing and history modes (open variant)
|
||||
- --diagnose flag for source status checking
|
||||
- --store flag for SQLite accumulation
|
||||
|
||||
### Changed
|
||||
- Smarter query construction (strips noise words, auto-retry)
|
||||
- Two-phase search (Phase 2 entity-aware drill-down)
|
||||
- Reddit JSON enrichment (real upvotes/comments from reddit.com/.json)
|
||||
- Engagement-weighted scoring (relevance 45%, recency 25%, engagement 30%)
|
||||
- Model auto-selection with 7-day cache
|
||||
|
||||
### Fixed
|
||||
- YouTube timeout increased to 90s
|
||||
- Reddit 429 rate limit fail-fast
|
||||
- YouTube soft date filter (keeps evergreen content)
|
||||
- Eager import crash in __init__.py (Codex compatibility)
|
||||
|
||||
### New Contributors
|
||||
- @JosephOIbrahim - Windows Unicode fix
|
||||
- @levineam - Model fallback for unverified orgs
|
||||
- @jonthebeef - --days=N configurable lookback flag
|
||||
|
||||
### Credits
|
||||
- @steipete - Bird CLI (vendored X search) and yt-dlp inspiration
|
||||
- @galligan - Marketplace plugin inspiration
|
||||
- @hutchins - Pushed for YouTube feature
|
||||
|
||||
## [1.0.0] - 2026-01-15
|
||||
|
||||
Initial public release. Reddit + X search via OpenAI and xAI APIs.
|
||||
```
|
||||
|
||||
#### 2. Create annotated tag on the public repo
|
||||
|
||||
```bash
|
||||
cd /Users/mvanhorn/last30days-skill-private
|
||||
|
||||
# Tag on the current HEAD (which matches upstream/main)
|
||||
git tag -a v2.1.0 -m "Release v2.1.0: Watchlists, YouTube transcripts, Codex CLI, bundled X search"
|
||||
|
||||
# Push to public repo
|
||||
git push upstream v2.1.0
|
||||
```
|
||||
|
||||
Use annotated tag (not lightweight) because:
|
||||
- Stores tagger metadata and date
|
||||
- Works with `git describe`
|
||||
- Propagates to forks (317 forks)
|
||||
- Supports GPG signing if desired later
|
||||
|
||||
#### 3. Craft release notes
|
||||
|
||||
Assemble from existing copy. The release body should follow this structure:
|
||||
|
||||
```markdown
|
||||
## Highlights
|
||||
|
||||
The AI world reinvents itself every month. This skill keeps you current.
|
||||
|
||||
`/last30days` researches your topic across **Reddit, X, YouTube, and the web**
|
||||
from the last 30 days, finds what the community is actually upvoting, sharing,
|
||||
and saying on camera, and writes you a prompt that works today.
|
||||
|
||||
**Three headline features in v2.1:**
|
||||
|
||||
1. **Open-class skill with watchlists** - Track competitors, people, topics on a schedule.
|
||||
Pair with Open Claw for automated briefings. SQLite-backed with FTS5 search.
|
||||
|
||||
2. **YouTube transcripts as a 4th source** - When yt-dlp is installed, searches YouTube,
|
||||
grabs view counts, and reads the actual transcripts. A 20-minute review has 10x the
|
||||
signal of one X post.
|
||||
|
||||
3. **Codex CLI compatibility** - Same skill, same engine, same four sources.
|
||||
Install to `~/.agents/skills/last30days` and invoke with `$last30days`.
|
||||
|
||||
Plus: **Bundled X search** - Vendored Bird GraphQL client. No external CLI needed.
|
||||
Just Node.js 22+ and browser cookies.
|
||||
|
||||
## Real Results (verified 2/15)
|
||||
|
||||
| Topic | Reddit | X | YouTube | Web |
|
||||
|-------|--------|---|---------|-----|
|
||||
| Nano Banana Pro | - | 32 posts, 164 likes | 5 videos, 98K views | - |
|
||||
| Seedance 2.0 | 21 threads | 33 posts | 20 videos | 5 pages |
|
||||
| OpenClaw use cases | 35 threads, 1,130 upvotes | 23 posts | 20 videos, 1.57M views | - |
|
||||
| YouTube thumbnails | 7 threads, 654 upvotes | 32 posts | 18 videos, 6.15M views | - |
|
||||
|
||||
## What's New
|
||||
|
||||
### Added
|
||||
- Open-class skill with watchlist, briefing, and history modes
|
||||
- YouTube search + transcript extraction via yt-dlp
|
||||
- OpenAI Codex CLI compatibility
|
||||
- Bundled Twitter/X search (vendored Bird GraphQL)
|
||||
- Native web search (Parallel AI, Brave, OpenRouter)
|
||||
- `--diagnose` and `--store` flags
|
||||
- Conversational first-run experience (NUX)
|
||||
|
||||
### Changed
|
||||
- Two-phase search architecture (entity-aware drill-down)
|
||||
- Reddit JSON enrichment for real engagement metrics
|
||||
- Smarter query construction with auto-retry
|
||||
- Engagement-weighted scoring algorithm
|
||||
|
||||
### Fixed
|
||||
- YouTube/Reddit timeout resilience
|
||||
- Reddit 429 rate limit fail-fast
|
||||
- Eager import crash in Codex environments
|
||||
|
||||
## New Contributors
|
||||
|
||||
- @JosephOIbrahim - Windows Unicode fix
|
||||
- @levineam - Model fallback for unverified orgs
|
||||
- @jonthebeef - `--days=N` configurable lookback
|
||||
|
||||
## Credits
|
||||
|
||||
- @steipete - Bird CLI and yt-dlp/summarize inspiration
|
||||
- @galligan - Marketplace plugin inspiration
|
||||
- @hutchins - Pushed for YouTube feature
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# Claude Code
|
||||
claude install-skill https://github.com/mvanhorn/last30days-skill
|
||||
|
||||
# Manual
|
||||
git clone https://github.com/mvanhorn/last30days-skill ~/.claude/skills/last30days
|
||||
```
|
||||
|
||||
**Full Changelog**: https://github.com/mvanhorn/last30days-skill/commits/v2.1.0
|
||||
```
|
||||
|
||||
#### 4. Create the GitHub Release
|
||||
|
||||
```bash
|
||||
gh release create v2.1.0 \
|
||||
--repo mvanhorn/last30days-skill \
|
||||
--verify-tag \
|
||||
--title "v2.1.0 - Watchlists, YouTube Transcripts, Codex CLI" \
|
||||
-F release-notes.md
|
||||
```
|
||||
|
||||
No binary assets needed - this is an interpreted skill, and GitHub auto-generates source archives.
|
||||
|
||||
### Phase 2: Future Automation (Optional)
|
||||
|
||||
#### 5. Add `.github/release.yml` for auto-generated notes on future releases
|
||||
|
||||
```yaml
|
||||
changelog:
|
||||
exclude:
|
||||
labels:
|
||||
- ignore-for-release
|
||||
categories:
|
||||
- title: "Breaking Changes"
|
||||
labels: [breaking-change]
|
||||
- title: "Features"
|
||||
labels: [enhancement, feature]
|
||||
- title: "Bug Fixes"
|
||||
labels: [bug, fix]
|
||||
- title: "Documentation"
|
||||
labels: [documentation]
|
||||
- title: "Other Changes"
|
||||
labels: ["*"]
|
||||
```
|
||||
|
||||
This enables `gh release create v2.2.0 --generate-notes` for future releases with automatic PR categorization.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] `CHANGELOG.md` exists in repo root following Keep a Changelog format
|
||||
- [x] Annotated tag `v2.1.0` exists on the public repo
|
||||
- [x] GitHub Release `v2.1.0` is published at `github.com/mvanhorn/last30days-skill/releases`
|
||||
- [x] Release notes include: highlights, real results table, feature list, contributor credits, install instructions
|
||||
- [x] Release appears in the repo sidebar on GitHub
|
||||
|
||||
## Key Decisions
|
||||
|
||||
1. **v2.1.0 only** - Don't retroactively create v1.0.0 or v2.0.0 tags. The git history doesn't have clean boundary commits for those versions, and retroactive tags add complexity without value.
|
||||
|
||||
2. **Release on the public repo** (`upstream`), not the private repo. Users see `mvanhorn/last30days-skill`.
|
||||
|
||||
3. **CHANGELOG.md as source of truth** - The release notes are derived from CHANGELOG.md, not the other way around. Future releases update CHANGELOG.md first, then create the release.
|
||||
|
||||
4. **Update star count** - The existing copy says "1.5K stars" but the repo now has 2,747. Update in release notes.
|
||||
|
||||
## References
|
||||
|
||||
- Existing copy: `docs/v2.1-launch-copy.md`, `docs/v2.1-tweets.md`
|
||||
- Contributors: `docs/pr-credits.md`
|
||||
- README features: `README.md` "What's New" sections
|
||||
- GitHub Releases docs: https://docs.github.com/en/repositories/releasing-projects-on-github
|
||||
- Keep a Changelog: https://keepachangelog.com/en/1.1.0/
|
||||
@@ -0,0 +1,274 @@
|
||||
---
|
||||
title: "feat: Last30Days.com - Automated Trending Topic Research"
|
||||
type: feat
|
||||
date: 2026-02-20
|
||||
---
|
||||
|
||||
# Last30Days.com - Automated Trending Topic Research
|
||||
|
||||
## Overview
|
||||
|
||||
A website at Last30Days.com that automatically shows daily trending topics researched by the /last30days engine. The core challenge: the skill is query-driven (you give it a topic), but a trending page needs to *discover* what topics to research. This plan covers how to source trending topics, run them through the engine, and publish results.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The /last30days skill has 2,747 GitHub stars but no public-facing showcase. Users have to install the skill and run it themselves. A website that automatically shows trending topic results would:
|
||||
|
||||
1. **Drive installs** - people see the quality and want it for their own topics
|
||||
2. **SEO surface area** - each topic page is indexable content
|
||||
3. **Demonstrate capability** - "here's what /last30days found about X today" with real stats
|
||||
4. **Content flywheel** - daily fresh content with zero manual curation
|
||||
|
||||
## Trending Topic Discovery: The Options
|
||||
|
||||
The skill has **no existing trending discovery mechanism** - it's entirely query-driven. Here are the available sources, ranked by practicality.
|
||||
|
||||
### Tier 1: Free, High Signal, Zero Friction
|
||||
|
||||
| Source | What It Returns | Auth | Cost | Best For |
|
||||
|--------|----------------|------|------|----------|
|
||||
| **Wikipedia Pageviews** | Top 100 most-viewed articles yesterday | None | Free | General public interest (news, culture, events) |
|
||||
| **Hacker News** | Top 30 stories with scores | None | Free | Tech/startup topics |
|
||||
| **Reddit r/all/hot + rising** | Hottest and rapidly rising posts | OAuth (free) | Free | Broad internet culture |
|
||||
| **Google News RSS** | Top headlines, algorithmically curated | None | Free | Mainstream news |
|
||||
| **YouTube mostPopular** | Top 50 trending videos by region | API key | Free (10K units/day) | Pop culture, entertainment |
|
||||
|
||||
### Tier 2: Very Cheap, High Value
|
||||
|
||||
| Source | What It Returns | Auth | Cost | Best For |
|
||||
|--------|----------------|------|------|----------|
|
||||
| **Perplexity Sonar API** | "What's trending today?" with sourced answers | API key | ~$1/month | Meta-aggregator that replaces multiple sources |
|
||||
| **Google Trends (pytrends)** | Daily trending Google searches | None | Free but flaky | What people are actually searching |
|
||||
| **Bird `getNews()`** | X Explore page trending topics | Browser cookies | Free | Real-time Twitter/X conversation |
|
||||
|
||||
### Tier 3: Paid, Skip for MVP
|
||||
|
||||
| Source | Cost | Why Skip |
|
||||
|--------|------|----------|
|
||||
| X/Twitter API | $200/month minimum | Too expensive; Bird `getNews()` is free |
|
||||
| Exploding Topics | $249/month | Overkill for daily trends |
|
||||
| TikTok | Gated, requires approval | Application process |
|
||||
|
||||
### Existing Codebase Hooks
|
||||
|
||||
Several pieces already exist in the codebase that could be leveraged:
|
||||
|
||||
1. **Bird `getNews()` is already on disk.** The vendored Bird library at `scripts/lib/vendor/bird-search/` includes `twitter-client-news.js` which fetches X's Explore page tabs (For You, Trending, News, Sports, Entertainment). Only needs a ~30-line `bird-news.mjs` wrapper to expose it. Zero API keys needed.
|
||||
|
||||
2. **`store.get_trending(days=7)`** at `scripts/store.py:559` ranks watchlist topics by recent finding activity. Could feed "what the skill has been researching" as a meta-signal.
|
||||
|
||||
3. **Scoring algorithm** at `scripts/lib/score.py` has engagement formulas for Reddit, X, and YouTube that could rank "buzz" by reweighting engagement >> relevance.
|
||||
|
||||
4. **Brave Trending API** exists but isn't implemented in `brave_search.py`. Could be added.
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
### Topic Discovery Pipeline (daily cron)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ STEP 1: Fetch trending signals (parallel, free) │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Wikipedia Pageviews ─┐ │
|
||||
│ Hacker News Top 30 ─┤ │
|
||||
│ Reddit r/all/hot ─┼─→ Raw topics + signals │
|
||||
│ Google News RSS ─┤ │
|
||||
│ YouTube mostPopular ─┤ │
|
||||
│ Bird getNews() ─┘ │
|
||||
└─────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ STEP 2: Cluster + rank │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Extract topic keywords from titles │
|
||||
│ Cluster by semantic similarity │
|
||||
│ Rank by cross-source frequency │
|
||||
│ ("Pope Leo XIV" on Wikipedia + Reddit + News │
|
||||
│ = high confidence trend) │
|
||||
│ Output: top 15-20 topics, ranked │
|
||||
└─────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ STEP 3: Run /last30days on top topics │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Top 3-5 topics: full research (--emit=json) │
|
||||
│ → Full "What I learned" synthesis articles │
|
||||
│ Remaining 10-15: quick research (--quick) │
|
||||
│ → Stats teasers (32 X posts, 5 YouTube...) │
|
||||
└─────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ STEP 4: Publish to website │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Generate static HTML/JSON │
|
||||
│ Deploy to Last30Days.com │
|
||||
│ RSS feed for subscribers │
|
||||
└─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Topic Clustering Strategy
|
||||
|
||||
The hardest part is deduplicating across sources. "Pope Francis Dies" (Google News), "Pope_Francis" (Wikipedia #1 viewed), and a r/worldnews post are all the same topic. Approaches:
|
||||
|
||||
**Option A: LLM clustering (recommended for MVP)**
|
||||
Feed all raw titles to an LLM and ask it to cluster into distinct topics with a representative label. ~$0.01 per day via a fast model. Simple, accurate, handles edge cases.
|
||||
|
||||
**Option B: TF-IDF + cosine similarity**
|
||||
Extract keywords, compute pairwise similarity, agglomerative clustering. No API cost but worse on paraphrased titles.
|
||||
|
||||
**Option C: Embedding similarity**
|
||||
Embed all titles, cluster by cosine distance. Better than TF-IDF, costs a few cents per day.
|
||||
|
||||
### Website Architecture
|
||||
|
||||
**Option A: Static site (recommended for MVP)**
|
||||
- Daily cron generates JSON + static HTML
|
||||
- Host on GitHub Pages, Cloudflare Pages, or Vercel
|
||||
- Zero server cost, zero maintenance
|
||||
- Framework: plain HTML/CSS, or minimal Astro/11ty
|
||||
|
||||
**Option B: Next.js with ISR**
|
||||
- Incremental Static Regeneration rebuilds pages daily
|
||||
- More flexibility for future features (search, filtering, user accounts)
|
||||
- Hosting: Vercel free tier
|
||||
|
||||
**Option C: Full web app**
|
||||
- Database-backed, real-time updates, user accounts
|
||||
- Overkill for MVP
|
||||
|
||||
### Cost Estimate (daily operation)
|
||||
|
||||
| Item | Cost |
|
||||
|------|------|
|
||||
| Trending source APIs | $0 (all free tier) |
|
||||
| LLM clustering (fast model) | ~$0.01/day |
|
||||
| Full research on 3-5 topics | ~$0.10-0.50/day (OpenAI API for Reddit search) |
|
||||
| Quick research on 10-15 topics | ~$0.05-0.20/day |
|
||||
| Static hosting | $0 (GitHub/Cloudflare Pages) |
|
||||
| **Total** | **~$5-20/month** |
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Topic Discovery Script (MVP)
|
||||
|
||||
Build `scripts/discover_trending.py` that:
|
||||
- Fetches Wikipedia Pageviews, HN top stories, Reddit hot, Google News RSS in parallel
|
||||
- Filters out evergreen/non-topical Wikipedia pages (e.g., "Main Page", "ChatGPT" permanent traffic)
|
||||
- Uses a fast LLM to cluster raw titles into 15-20 distinct topics
|
||||
- Outputs ranked JSON: `[{topic, sources, confidence, category}]`
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] Fetches from at least 4 free trending sources in parallel
|
||||
- [ ] Clusters raw titles into deduplicated topics via LLM
|
||||
- [ ] Filters Wikipedia evergreen pages (maintain a blocklist)
|
||||
- [ ] Outputs ranked JSON with topic name, source count, category
|
||||
- [ ] Runs in < 60 seconds
|
||||
- [ ] No paid API keys required for discovery (only LLM clustering)
|
||||
|
||||
### Phase 2: Research Automation
|
||||
|
||||
Wire discovered topics into the existing /last30days engine:
|
||||
- Top 3-5 topics: `python3 scripts/last30days.py "$TOPIC" --emit=json --deep`
|
||||
- Remaining topics: `python3 scripts/last30days.py "$TOPIC" --emit=json --quick`
|
||||
- Store all results in `~/.local/share/last30days/trending/YYYY-MM-DD/`
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] Orchestration script runs discovery then research sequentially
|
||||
- [ ] Full research on top N topics, quick on the rest
|
||||
- [ ] Results stored as dated JSON files
|
||||
- [ ] Total daily runtime < 30 minutes
|
||||
- [ ] Handles timeouts/failures gracefully (skip topic, continue)
|
||||
|
||||
### Phase 3: Website Generation
|
||||
|
||||
Build a static site generator that reads the daily JSON and produces Last30Days.com:
|
||||
- Homepage: today's trending topics grid (title, category, key stat, source badges)
|
||||
- Topic pages: full synthesis for showcase topics, stats teaser for others
|
||||
- Archive: previous days accessible by date
|
||||
- RSS feed
|
||||
- CTA: "Want to research your own topic? Install the skill"
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] Static HTML generated from daily JSON
|
||||
- [ ] Homepage shows today's 15-20 trending topics
|
||||
- [ ] 3-5 showcase topic pages with full synthesis
|
||||
- [ ] Remaining topics show stats teasers + install CTA
|
||||
- [ ] Deploys to Last30Days.com (Cloudflare Pages or similar)
|
||||
- [ ] RSS feed for daily updates
|
||||
- [ ] Mobile responsive
|
||||
|
||||
### Phase 4: Bird Trending Integration (bonus)
|
||||
|
||||
Wire up the already-vendored Bird `getNews()` for X trending:
|
||||
- Create `scripts/lib/vendor/bird-search/bird-news.mjs` (~30 lines)
|
||||
- Add X Explore trending data as a 5th discovery source
|
||||
- X trends are the fastest-moving signal and fill the "what's happening right now" gap
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] `bird-news.mjs` wrapper exposes X Explore trending topics
|
||||
- [ ] Integrated into discovery pipeline as an additional source
|
||||
- [ ] Falls back gracefully if no X session cookies available
|
||||
|
||||
## Alternative Approaches Considered
|
||||
|
||||
### Perplexity-only approach
|
||||
Just ask Perplexity Sonar "what are the top 20 trending topics today?" daily. Simpler, but:
|
||||
- Single point of failure
|
||||
- Less transparent (can't show "sourced from Reddit, Wikipedia, HN")
|
||||
- Model may hallucinate or miss niche topics
|
||||
- **Verdict:** Good fallback, not primary.
|
||||
|
||||
### Curated topics (manual)
|
||||
Manually pick topics each day. Defeats the purpose of automation. Could supplement for "editorial picks."
|
||||
|
||||
### Social listening tools (Brandwatch, Sprout Social, etc.)
|
||||
Expensive ($500+/month), enterprise-focused, overkill.
|
||||
|
||||
## Technical Considerations
|
||||
|
||||
- **Rate limits:** Wikipedia, HN, and RSS have no practical limits for 1 daily call. Reddit's 100 QPM is generous. YouTube's 10K units/day allows ~3,000 `mostPopular` calls.
|
||||
- **Wikipedia filtering:** The top Wikipedia pages are always "Main Page", "Special:Search", etc. Need a blocklist of ~50 evergreen pages plus heuristics (skip pages under 1,000 characters, skip disambiguation pages).
|
||||
- **Cron timing:** Run discovery at ~2am UTC (after Wikipedia pageviews finalize for previous day). Run research at ~3am UTC. Deploy site by ~5am UTC.
|
||||
- **Cost control:** The /last30days engine uses OpenAI API for Reddit search. Full research on 5 topics * $0.05-0.10 each = ~$0.25-0.50/day. Quick research is cheaper.
|
||||
- **Domain:** Last30Days.com needs to be registered (check availability).
|
||||
|
||||
## Dependencies & Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| pytrends breaks (Google changes) | pytrends is a bonus source, not required. Other 4+ sources sufficient. |
|
||||
| Wikipedia pageviews delayed | Fall back to Perplexity Sonar for day's topics |
|
||||
| OpenAI API cost spikes | Cap at 5 full + 15 quick topics per day; use --quick for most |
|
||||
| Bird cookie auth stops working | X trending is a bonus source; discovery works without it |
|
||||
| Domain not available | Check availability; alternatives: last30.day, last30days.app |
|
||||
|
||||
## Success Metrics
|
||||
|
||||
- Daily automated publishing with zero manual intervention
|
||||
- 15-20 trending topics surfaced daily
|
||||
- 3-5 full synthesis articles per day
|
||||
- Site loads in < 2 seconds (static)
|
||||
- Drives measurable GitHub star growth / skill installs
|
||||
|
||||
## References
|
||||
|
||||
### Trending APIs (free)
|
||||
- Wikipedia Pageviews: `wikimedia.org/api/rest_v1/metrics/pageviews/top/{project}/{access}/{year}/{month}/{day}`
|
||||
- Hacker News: `hacker-news.firebaseio.com/v0/topstories.json`
|
||||
- Reddit: `oauth.reddit.com/r/all/hot` (or `reddit.com/r/all/hot.json` unauthenticated)
|
||||
- Google News RSS: `news.google.com/rss`
|
||||
- YouTube: `googleapis.com/youtube/v3/videos?chart=mostPopular`
|
||||
|
||||
### Existing codebase hooks
|
||||
- Bird `getNews()`: `scripts/lib/vendor/bird-search/vendor/package/dist/lib/twitter-client-news.js`
|
||||
- Store trending: `scripts/store.py:559` (`get_trending()`)
|
||||
- Scoring: `scripts/lib/score.py` (engagement formulas)
|
||||
- Brave Trending API: not implemented, available in Brave docs
|
||||
|
||||
### Inspiration
|
||||
- [Keep a Changelog](https://keepachangelog.com/) - clean daily update format
|
||||
- [Hacker News front page](https://news.ycombinator.com/) - minimal trending UI
|
||||
- [Exploding Topics](https://explodingtopics.com/) - trending topic showcase (paid, $249/mo)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,212 @@
|
||||
---
|
||||
title: "feat: 15-Test Side-by-Side Comparison - Make CROSS the GOAT"
|
||||
type: feat
|
||||
status: active
|
||||
date: 2026-02-25
|
||||
origin: docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md
|
||||
---
|
||||
|
||||
# 15-Test Side-by-Side Comparison - Make CROSS the GOAT
|
||||
|
||||
## Overview
|
||||
|
||||
Run all 5 canonical topics through all 3 skill versions (base, HN, CROSS) for 15 total full last30days runs. Save every result. Analyze side-by-side. Judge which version is best. Then create an improvement plan to make CROSS the definitive next release.
|
||||
|
||||
## Problem Statement / Motivation
|
||||
|
||||
The previous analysis only ran 5 tests on the CROSS branch - never comparing the same topics across all 3 versions. Without true side-by-side data, we can't judge whether CROSS is actually better or if the new features (dynamic YouTube scoring, cross-refs) introduce regressions. The user wants to see all 15 results, know which version wins, and get a concrete plan to make CROSS the GOAT before shipping.
|
||||
|
||||
## The 3 Versions
|
||||
|
||||
| Version | Git State | Sources | YouTube Relevance | Cross-Refs |
|
||||
|---------|-----------|---------|-------------------|------------|
|
||||
| **Base** | commit `427a4e4` | Reddit, X, YouTube, Web | Hardcoded 0.7 | No |
|
||||
| **HN** | `main` / `f60a435` | Reddit, X, YouTube, **HN**, Web | Hardcoded 0.7 | No |
|
||||
| **CROSS** | `feat/youtube-relevance-cross-source` / `0591f55` | Reddit, X, YouTube, HN, Web | Dynamic 0.1-1.0 | Yes (Jaccard 0.5) |
|
||||
|
||||
## The 5 Topics
|
||||
|
||||
| # | Topic | Category | Why chosen |
|
||||
|---|-------|----------|------------|
|
||||
| 1 | "Claude Code skills and MCP servers" | Developer tools | Core user base topic |
|
||||
| 2 | "Seedance AI video generation" | Creative AI | Trending topic, cross-platform buzz |
|
||||
| 3 | "M4 MacBook Pro review" | Consumer tech | Product review, mainstream |
|
||||
| 4 | "best rap songs 2026" | Pop culture | Non-tech stress test |
|
||||
| 5 | "React vs Svelte 2026" | Framework debate | Dev community, opinion-heavy |
|
||||
|
||||
## Test Matrix (15 Runs)
|
||||
|
||||
Run in **topic-sequential** order (all 3 versions of topic 1 back-to-back, then topic 2, etc.) to minimize temporal confounds on YouTube/X search results.
|
||||
|
||||
| Run | Topic | Version | Output File |
|
||||
|-----|-------|---------|-------------|
|
||||
| 1 | Claude Code | Base | `base-1-claude-code.json` |
|
||||
| 2 | Claude Code | HN | `hn-1-claude-code.json` |
|
||||
| 3 | Claude Code | CROSS | `cross-1-claude-code.json` |
|
||||
| 4 | Seedance | Base | `base-2-seedance.json` |
|
||||
| 5 | Seedance | HN | `hn-2-seedance.json` |
|
||||
| 6 | Seedance | CROSS | `cross-2-seedance.json` |
|
||||
| 7 | MacBook | Base | `base-3-macbook.json` |
|
||||
| 8 | MacBook | HN | `hn-3-macbook.json` |
|
||||
| 9 | MacBook | CROSS | `cross-3-macbook.json` |
|
||||
| 10 | Rap songs | Base | `base-4-rap.json` |
|
||||
| 11 | Rap songs | HN | `hn-4-rap.json` |
|
||||
| 12 | Rap songs | CROSS | `cross-4-rap.json` |
|
||||
| 13 | React/Svelte | Base | `base-5-react-svelte.json` |
|
||||
| 14 | React/Svelte | HN | `hn-5-react-svelte.json` |
|
||||
| 15 | React/Svelte | CROSS | `cross-5-react-svelte.json` |
|
||||
|
||||
**Output directory:** `/tmp/last30days-comparison/full/`
|
||||
|
||||
## Execution Protocol
|
||||
|
||||
### Pre-flight (once)
|
||||
|
||||
- [x] Clear model cache: `rm -f ~/.cache/last30days/model_selection.json`
|
||||
- [x] Run `--diagnose` on current branch, save as `diagnose-baseline.json`
|
||||
- [x] Verify all API keys active: Reddit (OPENAI_API_KEY), X (Bird cookies or XAI_API_KEY), YouTube (yt-dlp), HN (no key needed), Web (parallel AI or Brave)
|
||||
- [x] Create output dir: `mkdir -p /tmp/last30days-comparison/full`
|
||||
- [x] Stash any uncommitted changes: `git stash` (none needed - no uncommitted changes)
|
||||
|
||||
### Per-topic loop (repeat 5 times)
|
||||
|
||||
For each topic, run all 3 versions back-to-back:
|
||||
|
||||
```bash
|
||||
# CRITICAL: Clean __pycache__ between EVERY git checkout to prevent stale bytecode
|
||||
cleanup() {
|
||||
find scripts -name '__pycache__' -exec rm -rf {} + 2>/dev/null
|
||||
find scripts -name '*.pyc' -delete 2>/dev/null
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 1: Base version**
|
||||
- `cleanup && git checkout 427a4e4`
|
||||
- `python3 scripts/last30days.py --diagnose 2>/dev/null` (verify sources match baseline)
|
||||
- `python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/base-{N}-{slug}.json 2>/tmp/last30days-comparison/full/base-{N}-{slug}.log`
|
||||
|
||||
- [ ] **Step 2: HN version**
|
||||
- `cleanup && git checkout main`
|
||||
- `python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/hn-{N}-{slug}.json 2>/tmp/last30days-comparison/full/hn-{N}-{slug}.log`
|
||||
|
||||
- [ ] **Step 3: CROSS version**
|
||||
- `cleanup && git checkout feat/youtube-relevance-cross-source`
|
||||
- `python3 scripts/last30days.py "<topic>" --quick --emit=json > /tmp/last30days-comparison/full/cross-{N}-{slug}.json 2>/tmp/last30days-comparison/full/cross-{N}-{slug}.log`
|
||||
|
||||
### Post-run
|
||||
|
||||
- [x] Return to feature branch: `git checkout feat/youtube-relevance-cross-source`
|
||||
- [x] Verify all 15 JSON files exist and are non-empty
|
||||
- [x] Check for `*_error` fields in any JSON output - flag but include (zero errors)
|
||||
|
||||
## Analysis Dimensions
|
||||
|
||||
### 1. Source Coverage Table
|
||||
|
||||
For each of 15 runs, count items per source:
|
||||
|
||||
| Topic | Version | Reddit | X | YouTube | HN | Web | Total |
|
||||
|-------|---------|--------|---|---------|----|----|-------|
|
||||
|
||||
Expected: Base has 0 HN items. HN and CROSS should have identical source counts (same search code). Any differences indicate API non-determinism.
|
||||
|
||||
### 2. YouTube Relevance Comparison
|
||||
|
||||
**Matched-item analysis:** Match videos by `video_id` across Base/CROSS runs of the same topic. For each matched video:
|
||||
- Base relevance: always 0.7
|
||||
- CROSS relevance: dynamic score
|
||||
- Delta and direction (did dynamic scoring promote or demote this video?)
|
||||
|
||||
**Aggregate analysis:** Distribution stats (min/avg/max/stddev) per version per topic.
|
||||
|
||||
### 3. Cross-Source Links (CROSS only)
|
||||
|
||||
- Count of items with cross_refs per topic
|
||||
- Quality assessment: are the linked items actually about the same story?
|
||||
- Which source pairs link most often? (Reddit-HN? YouTube-HN? X-Reddit?)
|
||||
|
||||
### 4. Score Distribution and Rankings
|
||||
|
||||
- Mean/median score of top 10 items per version per topic
|
||||
- Rank position changes: do the same items appear in different orders?
|
||||
- Does HN crowd out Reddit/X items in the top 10?
|
||||
|
||||
### 5. "Best Version" Judging Criteria
|
||||
|
||||
Define BEFORE analyzing to avoid bias:
|
||||
|
||||
| Metric | Weight | How measured |
|
||||
|--------|--------|-------------|
|
||||
| Source diversity | 25% | Shannon entropy across source types in top 15 items |
|
||||
| Score quality | 25% | Mean score of top 10 items |
|
||||
| Relevance accuracy | 25% | YouTube: do high-relevance videos actually match the query? Manual spot-check of top 3 + bottom 3 per topic |
|
||||
| Bonus features | 25% | HN value-add (unique info not in other sources) + cross-ref utility (do xrefs add value to the reader?) |
|
||||
|
||||
### 6. HN Value-Add Assessment
|
||||
|
||||
For each topic where HN returns results:
|
||||
- How many HN items appear in the overall top 10?
|
||||
- Do HN items provide information not available from Reddit/X/YouTube?
|
||||
- Are HN comment insights (top_comments) genuinely useful?
|
||||
|
||||
## Deliverables
|
||||
|
||||
### A. Raw Results Archive
|
||||
|
||||
All 15 JSON files plus logs saved in `/tmp/last30days-comparison/full/`, also copied to `docs/comparison-results/` for persistence.
|
||||
|
||||
### B. Comparison Summary Table
|
||||
|
||||
A single markdown table showing all 15 runs with key metrics per cell.
|
||||
|
||||
### C. Version Verdict
|
||||
|
||||
Clear judgment: which version is best overall, and best per topic category (tech, consumer, culture).
|
||||
|
||||
### D. CROSS Improvement Plan
|
||||
|
||||
Concrete changes to make CROSS the GOAT:
|
||||
|
||||
**Based on the previous analysis (see origin doc), likely improvements include:**
|
||||
|
||||
1. **Fix cross-source linking** - Switch from char-trigram Jaccard (0.5) to hybrid similarity (max of trigram + token Jaccard) at threshold 0.40. Previous modeling showed this goes from 1 link to 24 links across 5 tests.
|
||||
|
||||
2. **Cross-ref rendering** - Change from cryptic `[xref: X1, HN3]` IDs to human-readable `[also on: Reddit, HN]` labels.
|
||||
|
||||
3. **HN search broadening** - Investigate why React/Svelte returns 0 HN items.
|
||||
|
||||
4. **YouTube relevance** - Already working well, may need minor threshold tuning based on 15-test data.
|
||||
|
||||
5. **Score normalization** - Ensure HN items don't systematically crowd out other sources in the merged ranking.
|
||||
|
||||
**New improvements to discover from 15-test data:**
|
||||
- Regressions introduced by CROSS changes
|
||||
- Edge cases where Base or HN outperforms CROSS
|
||||
- Topic-specific tuning opportunities
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] All 15 JSON result files saved and non-empty
|
||||
- [x] All 15 runs use identical source routing (verified via --diagnose)
|
||||
- [x] __pycache__ cleaned between every git checkout
|
||||
- [x] Comparison summary table covers all 15 runs
|
||||
- [x] YouTube matched-item analysis for at least 3 topics (all 5 done)
|
||||
- [x] Cross-source link quality spot-check for CROSS runs
|
||||
- [x] Clear version verdict with supporting data
|
||||
- [x] CROSS improvement plan with specific code changes
|
||||
- [x] All results appended to the analysis document for the user to review
|
||||
|
||||
## Technical Considerations
|
||||
|
||||
- **__pycache__ poisoning**: Python caches bytecode. Switching git commits without clearing `__pycache__` means old code runs even after checkout. MUST clean between every checkout.
|
||||
- **API rate limiting**: 15 runs hitting Reddit, X, YouTube, HN APIs. The `--quick` mode and natural ~30-90s per run provide spacing, but monitor for 429s in logs.
|
||||
- **YouTube non-determinism**: yt-dlp search results shift over time. Topic-sequential ordering minimizes this by running all 3 versions of the same topic within ~3-5 minutes.
|
||||
- **X search session**: Bird CLI depends on browser cookies. If session expires mid-run, X results degrade silently. Check X item counts across runs.
|
||||
- **`--quick` limitation**: Results reflect Phase 1 only (8-12 items per source). Full pipeline with `--deep` might show different patterns. Document this caveat.
|
||||
|
||||
## Sources & References
|
||||
|
||||
- Previous analysis: [docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md](docs/plans/2026-02-25-analysis-cross-source-comparison-plan.md)
|
||||
- Implementation plan: [docs/plans/2026-02-25-feat-youtube-relevance-and-cross-source-linking-plan.md](docs/plans/2026-02-25-feat-youtube-relevance-and-cross-source-linking-plan.md)
|
||||
- Existing comparison harness: [scripts/test-v1-vs-v2.sh](scripts/test-v1-vs-v2.sh)
|
||||
- Scoring weights: [scripts/lib/score.py](scripts/lib/score.py) - 45% relevance + 25% recency + 30% engagement
|
||||
@@ -0,0 +1,201 @@
|
||||
---
|
||||
title: "chore: open PR triage - build or close decision for all 5 open PRs"
|
||||
type: chore
|
||||
status: active
|
||||
date: 2026-03-02
|
||||
---
|
||||
|
||||
# Open PR Triage
|
||||
|
||||
Decision record for each of the 5 open PRs on `mvanhorn/last30days-skill`.
|
||||
For each: build/merge, cherry-pick, or close with explanation.
|
||||
|
||||
---
|
||||
|
||||
## PR #26 - Add HN, YouTube, and Product Hunt sources + --search flag
|
||||
**Author:** wkbaran | **Size:** +3402/-49 | **Date:** 2026-02-15
|
||||
|
||||
### What it adds
|
||||
- Hacker News source (Algolia API, no auth)
|
||||
- YouTube source (YouTube Data API v3, optional key)
|
||||
- Product Hunt source (GraphQL API, optional key)
|
||||
- `--search=SOURCES` flag (e.g. `--search=reddit,hn,yt`)
|
||||
- 58 new tests
|
||||
|
||||
### What's already on main
|
||||
- **HN:** ✅ already merged (`scripts/lib/hackernews.py`)
|
||||
- **YouTube:** ✅ already merged (`scripts/lib/youtube_yt.py`)
|
||||
- **Product Hunt:** ❌ NOT on main
|
||||
- **`--search` flag:** ❌ NOT on main
|
||||
|
||||
### Decision: CHERRY-PICK (`--search` flag only, skip Product Hunt)
|
||||
|
||||
The HN and YouTube modules in this PR are earlier versions than what's on
|
||||
main. We should NOT take those. Product Hunt is not a priority source.
|
||||
The one missing piece worth landing:
|
||||
|
||||
**`--search=SOURCES` flag** - lets callers (including agent mode) specify
|
||||
which sources to run. Useful for `--search=hn,polymarket` or
|
||||
`--search=reddit,x` focus runs.
|
||||
|
||||
### Plan
|
||||
- Cherry-pick just `tests/test_search_flag.py` and the `--search` argument
|
||||
wiring in `scripts/last30days.py`
|
||||
- Skip `scripts/lib/producthunt.py` and all PH-related code entirely
|
||||
- Do NOT take their `hackernews.py`, `youtube.py`, `render.py`, `normalize.py`,
|
||||
`schema.py` - our versions are newer
|
||||
- Update SKILL.md to document `--search` flag
|
||||
- Comment on PR thanking wkbaran - his HN/YouTube work was an inspiration
|
||||
for the sources we ended up building. Acknowledge that specifically.
|
||||
|
||||
### Acceptance Criteria
|
||||
- [ ] `python3 scripts/last30days.py "AI coding tools" --search=hn,reddit` runs only HN + Reddit
|
||||
- [ ] `python3 scripts/last30days.py "test" --search=x,web` skips Reddit
|
||||
- [ ] `test_search_flag.py` passes (adapt to our test patterns, no PH references)
|
||||
- [ ] SKILL.md updated with `--search` flag docs
|
||||
- [ ] Sync via `bash scripts/sync.sh`
|
||||
|
||||
---
|
||||
|
||||
## PR #24 - Adding Codex compatibility
|
||||
**Author:** el-analista | **Size:** +228/-29 | **Date:** 2026-02-11
|
||||
|
||||
### What it adds
|
||||
- `agents/openai.yaml` Codex discovery metadata
|
||||
- Portable script path resolution in SKILL.md (multi-install-path loop)
|
||||
- Platform-neutral UI text (`assistant` instead of Claude-specific wording)
|
||||
- `LAST30DAYS_CACHE_DIR` env override in `scripts/lib/cache.py`
|
||||
- `LAST30DAYS_OUTPUT_DIR` env override in `scripts/lib/render.py`
|
||||
- X/Bird query noise stripping + last-chance retry in `scripts/lib/bird_x.py`
|
||||
- New tests: `test_bird_x.py`, `test_cache.py`, `test_render.py`
|
||||
|
||||
### What's already on main
|
||||
- **Codex auth:** ✅ already merged (via PR #38)
|
||||
- **Multi-path script resolution:** ✅ already in SKILL.md (the `for dir in` loop)
|
||||
- **Platform-neutral wording:** ❌ NOT systematically applied
|
||||
- **Cache/output dir overrides:** ❌ NOT on main
|
||||
- **Bird X improvements:** ✅ partially (we have other bird_x fixes but may be missing noise stripping + retry)
|
||||
|
||||
### Decision: REVIEW AND PARTIAL CHERRY-PICK
|
||||
|
||||
The env var overrides (`LAST30DAYS_CACHE_DIR`, `LAST30DAYS_OUTPUT_DIR`) are
|
||||
genuinely useful for sandboxed/containerized Codex environments. The bird_x
|
||||
noise stripping and last-chance retry may improve X search quality and are
|
||||
low-risk additions.
|
||||
|
||||
The platform-neutral wording change ("assistant" instead of Claude references)
|
||||
should be skipped — SKILL.md is Claude-specific by design.
|
||||
|
||||
### Plan
|
||||
- Read the diff carefully against our current `cache.py`, `render.py`, `bird_x.py`
|
||||
- Cherry-pick the `LAST30DAYS_CACHE_DIR` and `LAST30DAYS_OUTPUT_DIR` env overrides
|
||||
- Cherry-pick the bird_x noise stripping + retry logic if it doesn't conflict
|
||||
with our existing bird_x changes
|
||||
- Skip `agents/openai.yaml` — we have our own multi-path resolution
|
||||
- Skip platform-neutral wording changes
|
||||
- Comment on PR thanking contributor and explaining what landed
|
||||
|
||||
### Acceptance Criteria
|
||||
- [ ] `LAST30DAYS_CACHE_DIR=/tmp/test python3 scripts/last30days.py "test" --mock` writes cache to /tmp/test
|
||||
- [ ] `LAST30DAYS_OUTPUT_DIR=/tmp/out python3 scripts/last30days.py "test" --mock` writes output to /tmp/out
|
||||
- [ ] Existing tests pass after cherry-pick
|
||||
- [ ] `test_cache.py` and `test_render.py` adapted and passing
|
||||
|
||||
---
|
||||
|
||||
## PR #14 - Simplify to WebSearch-first, make API keys optional
|
||||
**Author:** thangman1 | **Size:** +97/-168 | **Date:** 2026-02-01
|
||||
|
||||
### What it does
|
||||
Removes the Reddit/X Python search engine as the primary data source.
|
||||
Repositions Claude Code's built-in WebSearch as the "default" mode.
|
||||
Strips engagement metrics (upvotes, likes, repost counts) from output.
|
||||
Removes mode detection logic (Full Mode / Partial Mode / Web-Only Mode).
|
||||
|
||||
### Decision: CLOSE - do not merge
|
||||
|
||||
This PR inverts the core value proposition of last30days. The skill's
|
||||
differentiation is **real engagement data** from Reddit threads and X posts —
|
||||
upvotes, likes, reposts — that WebSearch cannot provide. Stripping that out
|
||||
produces a worse tool than just asking Claude to search the web, which anyone
|
||||
can already do.
|
||||
|
||||
The author's intent (lower barrier to entry, no API key required) is valid,
|
||||
but the right solution is making HN + Polymarket work without any API key
|
||||
(they already do), and making OPENAI_API_KEY easier to obtain — not removing
|
||||
the Reddit/X engine.
|
||||
|
||||
### Action
|
||||
- Comment on PR explaining why we're closing it
|
||||
- Acknowledge the valid friction point (API key setup) and point to HN +
|
||||
Polymarket as the zero-config sources
|
||||
- Close PR
|
||||
|
||||
---
|
||||
|
||||
## PR #10 - OpenRouter API integration
|
||||
**Author:** thetechreviewer | **Size:** +1029/-16 | **Date:** 2026-01-28
|
||||
|
||||
### What it adds
|
||||
OpenRouter as an alternative to OpenAI for the Reddit discovery search.
|
||||
Allows using any model available on OpenRouter instead of just OpenAI models.
|
||||
|
||||
### What's already on main
|
||||
- **`scripts/lib/openrouter_search.py`:** ✅ ALREADY ON MAIN
|
||||
- **Wired into `scripts/last30days.py`:** ✅ ALREADY ON MAIN (the `backend == "openrouter"` path)
|
||||
|
||||
### Decision: CLOSE - already merged
|
||||
|
||||
The OpenRouter integration that this PR introduced is already on main. It
|
||||
arrived via internal work that post-dated this PR. The PR is stale.
|
||||
|
||||
### Action
|
||||
- Comment on PR: "Thanks for this — OpenRouter support is already on main
|
||||
(landed via internal work). Closing as incorporated."
|
||||
- Close PR
|
||||
|
||||
---
|
||||
|
||||
## PR #5 - Add support for Codex auth with OpenAI Responses API
|
||||
**Author:** jblwilliams | **Size:** +358/-66 | **Date:** 2026-01-27
|
||||
|
||||
### What it adds
|
||||
- JWT-based Codex auth with `chatgpt_account_id`
|
||||
- Codex endpoint routing (`https://chatgpt.com/backend-api/codex/responses`)
|
||||
- SSE handling for streaming Codex responses
|
||||
- Typed auth status/source dataclass
|
||||
- Codex fallback model chain
|
||||
|
||||
### What's already on main
|
||||
**All of this is already on main.** The Codex auth system landed via PR #37
|
||||
(iliaal:codex-auth-merged), which in turn came in with PR #38. This PR (#5)
|
||||
predates that work and covers the same ground.
|
||||
|
||||
### Decision: CLOSE - already incorporated
|
||||
|
||||
### Action
|
||||
- Comment on PR: "Thanks for this early work on Codex auth! The same feature
|
||||
landed on main via PR #37 (from a separate contributor who built on similar
|
||||
ideas). The JWT decode, Codex endpoint routing, SSE parsing, and typed auth
|
||||
dataclass are all live. Closing as incorporated."
|
||||
- Close PR
|
||||
|
||||
---
|
||||
|
||||
## Summary Table
|
||||
|
||||
| PR | Author | Decision | Reason |
|
||||
|----|--------|----------|--------|
|
||||
| #26 | wkbaran | **Cherry-pick** | `--search` flag not on main; HN/YouTube already there; skip Product Hunt |
|
||||
| #24 | el-analista | **Cherry-pick** | Cache dir overrides + bird_x improvements worth landing; Codex auth already there |
|
||||
| #14 | thangman1 | **Close** | Removes engagement data, inverts core value proposition |
|
||||
| #10 | thetechreviewer | **Close** | OpenRouter already on main |
|
||||
| #5 | jblwilliams | **Close** | Codex auth already on main via PR #37/38 |
|
||||
|
||||
## Implementation Order (if proceeding)
|
||||
|
||||
1. Close PR #5, #10, #14 with comments (no code changes needed)
|
||||
2. Cherry-pick PR #24 pieces (bird_x + cache/render env overrides)
|
||||
3. Cherry-pick PR #26 pieces (Product Hunt + `--search` flag)
|
||||
4. Sync and test
|
||||
5. Push to upstream
|
||||
@@ -0,0 +1,49 @@
|
||||
---
|
||||
title: Close Commented GitHub Issues and PRs
|
||||
type: fix
|
||||
status: active
|
||||
date: 2026-03-03
|
||||
origin: docs/plans/2026-03-03-fix-triage-all-open-github-issues-plan.md
|
||||
---
|
||||
|
||||
# Close Commented GitHub Issues and PRs
|
||||
|
||||
All 9 open issues and 2 open PRs on `mvanhorn/last30days-skill` were commented on (2026-03-03) with fix confirmations or responses, but none were actually closed on GitHub.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Close 6 issues where fixes were confirmed (#40, #39, #32, #30, #29, #4)
|
||||
- [ ] Close PR #26 (superseded - features landed on main)
|
||||
- [ ] Verify 3 issues remain open (#36, #31, #22) - acknowledged but unresolved
|
||||
- [ ] Verify PR #24 remains open - explicitly left open per comment
|
||||
|
||||
## Actions
|
||||
|
||||
### Close as fixed (6 issues)
|
||||
|
||||
```bash
|
||||
gh issue close 40 --repo mvanhorn/last30days-skill --reason completed
|
||||
gh issue close 39 --repo mvanhorn/last30days-skill --reason completed
|
||||
gh issue close 32 --repo mvanhorn/last30days-skill --reason completed
|
||||
gh issue close 30 --repo mvanhorn/last30days-skill --reason completed
|
||||
gh issue close 29 --repo mvanhorn/last30days-skill --reason completed
|
||||
gh issue close 4 --repo mvanhorn/last30days-skill --reason completed
|
||||
```
|
||||
|
||||
### Close superseded PR (#26)
|
||||
|
||||
```bash
|
||||
gh pr close 26 --repo mvanhorn/last30days-skill
|
||||
```
|
||||
|
||||
### Keep open (no action needed)
|
||||
|
||||
- #36 - SKILL.md flag forwarding (investigating)
|
||||
- #31 - skills.sh audit scores (needs review)
|
||||
- #22 - Bird feature requests (backlog)
|
||||
- PR #24 - Codex compatibility (intentionally left open)
|
||||
|
||||
## Sources
|
||||
|
||||
- **Origin plan:** [docs/plans/2026-03-03-fix-triage-all-open-github-issues-plan.md](2026-03-03-fix-triage-all-open-github-issues-plan.md)
|
||||
- Commit: `5e5d586` - fix: triage all 16 open GitHub issues
|
||||
@@ -0,0 +1,375 @@
|
||||
---
|
||||
title: "feat: Paperclip Marketing Automation for last30days"
|
||||
type: feat
|
||||
status: active
|
||||
date: 2026-03-07
|
||||
---
|
||||
|
||||
# Paperclip Marketing Automation for last30days
|
||||
|
||||
## Overview
|
||||
|
||||
Set up a Paperclip "company" that auto-runs marketing for the last30days open-source skill (3,800+ stars). Four agent roles handle daily demo showcases, release announcements, community engagement, and analytics - all using a draft-then-approve workflow through Paperclip's built-in approval gates.
|
||||
|
||||
The killer angle: **last30days markets itself by running itself.** The Content Creator agent runs `/last30days [trending topic] --agent` on hot topics daily, then drafts X threads showing the results. Every post is a live demo.
|
||||
|
||||
## Problem Statement / Motivation
|
||||
|
||||
- All marketing is currently manual - ~60KB of pre-drafted X threads sit in `docs/` unposted
|
||||
- No automated community monitoring (GitHub issues, contributor shoutouts)
|
||||
- No metrics tracking (star growth, fork trends, social engagement)
|
||||
- Solo entrepreneur can't sustain daily content + community management + development
|
||||
- The tool's best ad is itself running on interesting topics, but that requires daily effort
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
A Paperclip company called **"last30days Marketing"** with 4 agent roles, all draft-then-approve.
|
||||
|
||||
### Company Structure
|
||||
|
||||
```
|
||||
last30days Marketing (Company)
|
||||
Mission: "Grow last30days to 10K GitHub stars through daily demo content,
|
||||
release marketing, and community engagement"
|
||||
|
||||
Marketing Director (Claude Code agent)
|
||||
- Sets daily topic priorities
|
||||
- Reviews draft quality before surfacing to human
|
||||
- Coordinates cross-agent work
|
||||
|
||||
Content Creator (Python script agent)
|
||||
- Runs last30days on trending topics daily
|
||||
- Drafts X showcase threads from the results
|
||||
- Heartbeat: daily at 8 AM PT
|
||||
|
||||
Release Manager (Bash + Python agent)
|
||||
- Watches for new git tags on upstream
|
||||
- Drafts release announcement threads
|
||||
- Heartbeat: every 6 hours (tag check is cheap)
|
||||
|
||||
Community Manager (Python script agent)
|
||||
- Monitors GitHub issues/PRs via gh CLI
|
||||
- Drafts welcome messages for new contributors
|
||||
- Surfaces popular feature requests
|
||||
- Heartbeat: every 4 hours
|
||||
|
||||
Analytics Analyst (Python script agent)
|
||||
- Tracks GitHub stars, forks, traffic
|
||||
- Tracks X engagement (@slashlast30days)
|
||||
- Generates weekly digest
|
||||
- Heartbeat: daily at 11 PM PT (collect), weekly Monday 9 AM (digest)
|
||||
```
|
||||
|
||||
### Draft-Then-Approve Flow
|
||||
|
||||
```
|
||||
Agent creates draft
|
||||
-> Saved to ~/Documents/Last30Days/drafts/{agent}/{date}-{slug}.md
|
||||
-> Paperclip approval gate triggers
|
||||
-> Matt reviews in Paperclip UI (approve / reject / edit)
|
||||
-> On approve: Python script posts to X API via tweepy
|
||||
-> Audit log records everything
|
||||
```
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Phase 1: Infrastructure (Day 1-2)
|
||||
|
||||
Set up Paperclip and external API access.
|
||||
|
||||
**Files to create:**
|
||||
|
||||
- `marketing/paperclip-config.yaml` - Company definition, org chart, budgets
|
||||
- `marketing/scripts/post_to_x.py` - X API v2 posting via tweepy (draft queue -> X)
|
||||
- `marketing/scripts/github_monitor.py` - GitHub event monitoring via gh CLI
|
||||
- `marketing/scripts/metrics_collector.py` - Star/fork/engagement tracking
|
||||
- `marketing/.env.example` - Required API keys template
|
||||
|
||||
**Setup steps:**
|
||||
|
||||
1. Install Paperclip locally:
|
||||
```bash
|
||||
git clone https://github.com/paperclipai/paperclip
|
||||
cd paperclip && pnpm install && pnpm dev
|
||||
```
|
||||
|
||||
2. Get X API v2 credentials (developer.x.com) for @slashlast30days
|
||||
- Need: API key, API secret, Access token, Access token secret
|
||||
- Permissions: Read + Write (posting)
|
||||
|
||||
3. GitHub token for monitoring (gh auth already configured)
|
||||
|
||||
4. Create the company in Paperclip UI at localhost:3100:
|
||||
- Company name: "last30days Marketing"
|
||||
- Mission: "Grow last30days to 10K GitHub stars"
|
||||
- Monthly budget cap: $50 (mostly API token costs)
|
||||
|
||||
### Phase 2: Content Creator Pipeline (Day 3-5)
|
||||
|
||||
The core marketing engine. Uses last30days to market itself.
|
||||
|
||||
**How it works:**
|
||||
|
||||
1. **Topic Discovery** - `marketing/scripts/discover_topics.py`
|
||||
- Scrapes trending topics from: Wikipedia pageviews API, HN front page, Reddit r/all
|
||||
- Filters for topics that would make compelling demos (tech, culture, sports, geopolitics)
|
||||
- Outputs ranked topic list to `~/Documents/Last30Days/topics-queue.json`
|
||||
|
||||
2. **Research & Draft** - `marketing/scripts/create_showcase.py`
|
||||
- Picks top topic from queue
|
||||
- Runs: `python3 scripts/last30days.py "{topic}" --agent --emit=compact --save-dir=~/Documents/Last30Days`
|
||||
- Reads the saved research output
|
||||
- Drafts a 1-2 tweet thread in the established style (see Style Guide below)
|
||||
- Saves draft to `~/Documents/Last30Days/drafts/content/{date}-{slug}.md`
|
||||
|
||||
3. **Approval Gate** - Paperclip surfaces draft for review
|
||||
- Matt approves/edits in Paperclip UI
|
||||
- On approve: triggers `post_to_x.py` with the draft content
|
||||
|
||||
**Style Guide (extracted from existing launch threads):**
|
||||
|
||||
```
|
||||
Format: Stats-first hook + key finding + tool credit
|
||||
|
||||
Example (from v2.5 launch):
|
||||
"/last30days Anthropic Pete Hegseth"
|
||||
|
||||
14 Reddit threads. 29 X posts (11,559 likes). 20 YouTube videos (739K views).
|
||||
5 HN stories. 9 Polymarket markets.
|
||||
|
||||
[Key finding in 2-3 sentences]
|
||||
|
||||
Polymarket: [relevant odds with specific numbers]
|
||||
|
||||
[One-liner showing the tool's value]
|
||||
|
||||
github.com/mvanhorn/last30days-skill
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Always lead with the /last30days command that was run
|
||||
- Always include the stats line (thread/post/video counts)
|
||||
- Always end with the GitHub link
|
||||
- Pick topics people care about RIGHT NOW
|
||||
- Never use em dashes - use hyphens instead
|
||||
- Keep threads to 1-2 tweets max for daily showcases
|
||||
- Save longer threads (3-6 tweets) for releases
|
||||
|
||||
### Phase 3: Release Manager Pipeline (Day 5-6)
|
||||
|
||||
**How it works:**
|
||||
|
||||
1. **Tag Watcher** - `marketing/scripts/watch_releases.py`
|
||||
- Runs `git -C /Users/mvanhorn/last30days-skill-private fetch upstream --tags` every 6 hours
|
||||
- Compares local tags vs upstream tags
|
||||
- On new tag: reads CHANGELOG.md diff since last tag
|
||||
|
||||
2. **Thread Drafter** - `marketing/scripts/draft_release_thread.py`
|
||||
- Reads changelog diff + release-notes.md
|
||||
- Drafts a 3-6 tweet thread following the v2.5 launch thread style
|
||||
- Includes: version number, headline features, demo queries, contributor shoutouts
|
||||
- Saves to `~/Documents/Last30Days/drafts/releases/{tag}.md`
|
||||
|
||||
3. **Approval Gate** - Same flow as content pipeline
|
||||
|
||||
**Template (from existing docs/v2.5-launch-tweets.md):**
|
||||
|
||||
```
|
||||
Tweet 1: Announcement + 3 headline features + GitHub link
|
||||
Tweet 2-4: One demo per tweet (command + stats + finding)
|
||||
Tweet 5: Contributor shoutouts
|
||||
Tweet 6: Install instructions
|
||||
```
|
||||
|
||||
### Phase 4: Community Manager Pipeline (Day 6-7)
|
||||
|
||||
**How it works:**
|
||||
|
||||
1. **GitHub Monitor** - `marketing/scripts/github_monitor.py`
|
||||
- Runs `gh issue list --repo mvanhorn/last30days-skill --state open --json number,title,author,createdAt,labels`
|
||||
- Runs `gh pr list --repo mvanhorn/last30days-skill --state open --json number,title,author,createdAt`
|
||||
- Compares against `~/Documents/Last30Days/community/seen.json` to detect new items
|
||||
- Categorizes: bug report, feature request, question, PR
|
||||
|
||||
2. **Response Drafter** - `marketing/scripts/draft_community_response.py`
|
||||
- For new issues: drafts a welcome + triage response
|
||||
- For new PRs: drafts a thank-you + initial review comment
|
||||
- For merged PRs: drafts a contributor shoutout tweet
|
||||
- Saves to `~/Documents/Last30Days/drafts/community/{type}-{number}.md`
|
||||
|
||||
3. **Approval Gate** - GitHub responses go through same Paperclip approval
|
||||
- Approved responses posted via `gh issue comment` or `gh pr comment`
|
||||
- Shoutout tweets posted via `post_to_x.py`
|
||||
|
||||
**Response templates:**
|
||||
|
||||
```markdown
|
||||
# New Issue (bug)
|
||||
Thanks for the report! I'll look into this. Can you share:
|
||||
- Your OS and Python version
|
||||
- The exact command you ran
|
||||
- Whether you have SCRAPECREATORS_API_KEY set
|
||||
|
||||
# New Issue (feature request)
|
||||
Interesting idea! [1-2 sentences acknowledging the value].
|
||||
Adding this to the backlog for consideration.
|
||||
|
||||
# New PR
|
||||
Thanks for the contribution, @{author}! I'll review this shortly.
|
||||
[If first-time contributor: Welcome to the project!]
|
||||
|
||||
# Merged PR (tweet)
|
||||
Shoutout to @{author} for [what they did] in last30days v{version}!
|
||||
[Brief description of the change and why it matters]
|
||||
github.com/mvanhorn/last30days-skill/pull/{number}
|
||||
```
|
||||
|
||||
### Phase 5: Analytics Pipeline (Day 7-8)
|
||||
|
||||
**How it works:**
|
||||
|
||||
1. **Metrics Collector** - `marketing/scripts/metrics_collector.py`
|
||||
- Daily: GitHub stars, forks, open issues, open PRs (via `gh api`)
|
||||
- Daily: X followers, tweet impressions for @slashlast30days (via X API v2)
|
||||
- Stores in SQLite at `~/Documents/Last30Days/analytics.db`
|
||||
|
||||
2. **Weekly Digest** - `marketing/scripts/weekly_digest.py`
|
||||
- Runs Monday 9 AM PT
|
||||
- Generates markdown report with week-over-week changes:
|
||||
- Star growth (absolute + rate)
|
||||
- New forks
|
||||
- Issues opened/closed
|
||||
- PRs merged
|
||||
- Top-performing tweets
|
||||
- Notable community interactions
|
||||
- Saves to `~/Documents/Last30Days/digests/{date}-weekly.md`
|
||||
- Optionally sends via email (SendGrid) or Slack webhook
|
||||
|
||||
**Schema for analytics.db:**
|
||||
|
||||
```sql
|
||||
CREATE TABLE daily_metrics (
|
||||
date TEXT PRIMARY KEY,
|
||||
github_stars INTEGER,
|
||||
github_forks INTEGER,
|
||||
github_open_issues INTEGER,
|
||||
github_open_prs INTEGER,
|
||||
x_followers INTEGER,
|
||||
x_impressions INTEGER,
|
||||
x_engagement_rate REAL
|
||||
);
|
||||
|
||||
CREATE TABLE tweet_performance (
|
||||
tweet_id TEXT PRIMARY KEY,
|
||||
posted_at TEXT,
|
||||
type TEXT, -- 'showcase', 'release', 'shoutout'
|
||||
topic TEXT,
|
||||
impressions INTEGER,
|
||||
likes INTEGER,
|
||||
retweets INTEGER,
|
||||
replies INTEGER,
|
||||
link_clicks INTEGER
|
||||
);
|
||||
```
|
||||
|
||||
## Technical Considerations
|
||||
|
||||
### API Costs
|
||||
|
||||
| Service | Usage | Est. Monthly Cost |
|
||||
|---------|-------|-------------------|
|
||||
| Paperclip | Self-hosted | $0 |
|
||||
| X API v2 | Free tier (posting) | $0 |
|
||||
| ScrapeCreators | ~30 daily research runs | ~$15 |
|
||||
| GitHub API | gh CLI, already authed | $0 |
|
||||
| Claude API | Marketing Director agent | ~$20 |
|
||||
| **Total** | | **~$35/month** |
|
||||
|
||||
### Security
|
||||
|
||||
- API keys stored in `marketing/.env` (gitignored, never committed)
|
||||
- X API tokens scoped to @slashlast30days only (not personal account)
|
||||
- Paperclip budget cap prevents runaway spend
|
||||
- All posts go through human approval gate - no autonomous posting
|
||||
- GitHub token uses existing `gh auth` session
|
||||
|
||||
### Failure Modes
|
||||
|
||||
- **last30days script fails** - Content Creator skips that day, logs error, tries again tomorrow with new topic
|
||||
- **X API rate limit** - Queue drafts and retry on next heartbeat cycle
|
||||
- **Paperclip goes down** - Drafts accumulate in filesystem, nothing posts (safe failure)
|
||||
- **Bad topic selection** - Marketing Director agent filters topics before research (no politics, no NSFW)
|
||||
- **Stale approval queue** - If drafts pile up >3 days unapproved, send a nudge notification
|
||||
|
||||
### Topic Selection Criteria
|
||||
|
||||
The discover_topics.py script should filter for topics that:
|
||||
1. Are trending NOW (Wikipedia pageview spike, HN front page, Reddit r/all)
|
||||
2. Would produce interesting multi-source results (not too niche, not too broad)
|
||||
3. Are safe for a developer tool brand (tech, sports, culture, science - avoid divisive politics)
|
||||
4. Haven't been covered in the last 7 days (dedup against previous showcases)
|
||||
5. Span different categories to show the tool's versatility (not all tech, not all sports)
|
||||
|
||||
Good examples (from existing launch threads): "Anthropic Pete Hegseth", "Seedance prompting", "Arizona basketball", "Iran war"
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Paperclip running locally with "last30days Marketing" company created
|
||||
- [ ] Content Creator agent runs last30days daily on a trending topic and produces a draft
|
||||
- [ ] Release Manager agent detects new git tags and drafts announcement threads
|
||||
- [ ] Community Manager agent detects new GitHub issues/PRs and drafts responses
|
||||
- [ ] Analytics agent collects daily metrics and generates weekly digest
|
||||
- [ ] All drafts go through Paperclip approval gate before posting
|
||||
- [ ] X API integration posts approved drafts to @slashlast30days
|
||||
- [ ] GitHub responses posted via gh CLI after approval
|
||||
- [ ] Monthly budget stays under $50
|
||||
- [ ] Style of generated tweets matches existing launch thread tone
|
||||
|
||||
## Success Metrics
|
||||
|
||||
- **Content velocity**: 5-7 showcase tweets/week (up from ~0 currently)
|
||||
- **Star growth**: Track week-over-week acceleration after content starts
|
||||
- **Time savings**: <5 min/day reviewing drafts vs 30-60 min/day manual marketing
|
||||
- **Content quality**: Drafts require minimal editing before approval (>80% approved as-is within 2 weeks)
|
||||
|
||||
## Dependencies & Risks
|
||||
|
||||
| Dependency | Risk | Mitigation |
|
||||
|-----------|------|------------|
|
||||
| Paperclip stability | Early-stage project, may have bugs | Pin to specific commit, keep simple config |
|
||||
| X API free tier | May get rate-limited or deprecated | Queue-based posting, daily limits |
|
||||
| Topic discovery quality | Bad topics = bad demos | Human review in approval gate, topic blocklist |
|
||||
| Claude API for Marketing Director | Cost could spike | Budget cap in Paperclip, simple prompts |
|
||||
| ScrapeCreators API | PAYG costs scale with usage | Cap at 1 research run/day for showcases |
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
last30days-skill-private/
|
||||
marketing/
|
||||
README.md # Setup instructions
|
||||
.env.example # Required API keys
|
||||
paperclip-config.yaml # Company definition
|
||||
scripts/
|
||||
discover_topics.py # Trending topic discovery
|
||||
create_showcase.py # Research + draft showcase tweet
|
||||
draft_release_thread.py # Release announcement drafter
|
||||
github_monitor.py # GitHub issue/PR monitor
|
||||
draft_community_response.py # Community response drafter
|
||||
post_to_x.py # X API posting (after approval)
|
||||
metrics_collector.py # Daily metrics collection
|
||||
weekly_digest.py # Weekly analytics digest
|
||||
templates/
|
||||
showcase.md # Tweet template for daily demos
|
||||
release.md # Thread template for releases
|
||||
community_responses.md # Response templates by type
|
||||
```
|
||||
|
||||
## Sources & References
|
||||
|
||||
- Paperclip GitHub: github.com/paperclipai/paperclip
|
||||
- Paperclip docs: paperclip.ing
|
||||
- Existing launch threads: `docs/v2.5-launch-tweets.md`, `docs/v2.1-tweets.md`
|
||||
- Existing launch copy: `docs/v2.1-launch-copy.md`
|
||||
- Planned Last30Days.com: `docs/plans/2026-02-20-feat-last30days-com-trending-topics-plan.md`
|
||||
- last30days `--agent` mode: SKILL.md line 115-142 (non-interactive output for automation)
|
||||
@@ -0,0 +1,261 @@
|
||||
---
|
||||
title: Triage Open Issues and PRs
|
||||
type: refactor
|
||||
status: active
|
||||
date: 2026-03-07
|
||||
---
|
||||
|
||||
# Triage Open Issues and PRs - last30days-skill
|
||||
|
||||
Consolidated review and action plan for all 6 open items (2 issues, 4 PRs) as of 2026-03-07.
|
||||
|
||||
---
|
||||
|
||||
## 1. PR #52 - Fix missing metadata files in skill upload bundle
|
||||
|
||||
**Author:** 04cb | **Size:** +0/-1 | **Fixes:** #46
|
||||
**File changed:** `.clawhubignore` (removes `*.json` glob)
|
||||
|
||||
### What it does
|
||||
|
||||
The `*.json` pattern in `.clawhubignore` was excluding `.claude-plugin/marketplace.json` and `.claude-plugin/plugin.json` - the metadata files required for ClawHub skill upload. This caused the "Zip file contains path with invalid characters" error reported in #46.
|
||||
|
||||
### Risk analysis
|
||||
|
||||
- Removing `*.json` also includes test fixtures (`fixtures/*.json`) and vendored `package.json` in the bundle. None are sensitive or harmful.
|
||||
- No `package-lock.json`, `skills-lock.json`, or credential files exist as `.json` in the repo.
|
||||
- One-line change with zero production code impact.
|
||||
|
||||
### Recommendation: MERGE IMMEDIATELY
|
||||
|
||||
- [x] Merge PR #52
|
||||
- [x] Close issue #46 (auto-closes via "Fixes #46")
|
||||
- [ ] Verify upload works post-merge
|
||||
|
||||
---
|
||||
|
||||
## 2. Issue #46 - "Claude will not upload as it says thread is invalid"
|
||||
|
||||
**Author:** johnuppard | **Error:** "Zip file contains path with invalid characters"
|
||||
|
||||
### Recommendation: CLOSES WITH PR #52
|
||||
|
||||
No standalone action needed. PR #52 is the fix. After merge, comment on #46 confirming resolution and ask reporter to verify.
|
||||
|
||||
---
|
||||
|
||||
## 3. PR #50 - test: add tests for entity_extract module
|
||||
|
||||
**Author:** mark-c4r | **Size:** +167/-0 (tests only)
|
||||
|
||||
### What it does
|
||||
|
||||
Adds `tests/test_entity_extract.py` with 23 test cases covering all 4 public/private functions in `scripts/lib/entity_extract.py`:
|
||||
|
||||
| Function | Cases | Coverage |
|
||||
|----------|-------|----------|
|
||||
| `_extract_x_handles` | 8 | author handles, @mentions, generic filtering, case normalization, frequency ranking, edge cases |
|
||||
| `_extract_x_hashtags` | 5 | basic extraction, multiple tags, frequency ranking, short tag filtering, empty input |
|
||||
| `_extract_subreddits` | 6 | field extraction, cross-refs in comments, frequency ranking, r/ stripping |
|
||||
| `extract_entities` | 4 | integration, max limits, empty inputs, return key validation |
|
||||
|
||||
### Code quality assessment
|
||||
|
||||
- All function signatures match the actual module implementation exactly
|
||||
- All 23 tests validated against the real module - they pass
|
||||
- Import pattern matches existing tests (`sys.path.insert` + `from lib import`)
|
||||
- Uses `unittest.TestCase` consistent with all other test files
|
||||
- No external dependencies, no API calls, no mocking needed (pure functions)
|
||||
- Test data uses inline dicts matching real Reddit/X response structure
|
||||
|
||||
### Recommendation: MERGE AFTER LOCAL VERIFICATION
|
||||
|
||||
- [x] Pull branch and run `python3 -m pytest tests/test_entity_extract.py -v` (23/23 passed)
|
||||
- [x] Run full suite `python3 -m pytest tests/` (317/320 passed - 3 pre-existing failures in test_models.py, unrelated)
|
||||
- [x] Merge
|
||||
|
||||
This is a clean, well-structured community contribution that adds coverage to a previously untested module.
|
||||
|
||||
---
|
||||
|
||||
## 4. PR #48 - feat: add Xiaohongshu source + Reddit public fallback
|
||||
|
||||
**Author:** YJLi-new | **Size:** +523/-75 | **Files:** 5 modified
|
||||
|
||||
### What it does
|
||||
|
||||
Two features bundled in one PR:
|
||||
|
||||
**A. Xiaohongshu (Little Red Book) source:**
|
||||
- New `scripts/lib/xiaohongshu_api.py` module for searching via xiaohongshu-mcp HTTP API
|
||||
- Source alias: `--search xiaohongshu` or `--search xhs`
|
||||
- Health/login availability checks in `env.py`
|
||||
- Handles Chinese numeric suffixes (wan/yi) in engagement parsing
|
||||
- Default API base: `http://host.docker.internal:18060` (Docker-hosted)
|
||||
|
||||
**B. Reddit public JSON fallback:**
|
||||
- `search_reddit_public()` added to `openai_reddit.py`
|
||||
- Uses `reddit.com/search/.json` endpoint (no API key required)
|
||||
- Multiple query strategy (topic, core subject, quoted core)
|
||||
- Engagement-based relevance heuristic (60% score, 40% comments)
|
||||
- Supplemental retries correctly gated - only fire when OpenAI auth is present
|
||||
|
||||
### Architecture compliance
|
||||
|
||||
| Pattern | Status |
|
||||
|---------|--------|
|
||||
| Three-function module pattern (search/parse/enrich) | Follows existing patterns |
|
||||
| `DEPTH_CONFIG` with quick/default/deep | Present |
|
||||
| `_log()` gated on stderr.isatty() | Present |
|
||||
| env.py availability check | Present with retry logic |
|
||||
| ThreadPoolExecutor dispatch | Correctly wired with +1 worker |
|
||||
| Render/UI integration | Consistent with existing sources |
|
||||
|
||||
### Issues found
|
||||
|
||||
1. **Health check duplication** - `is_xiaohongshu_available()` in env.py AND `search_feeds()` both probe login status. Redundant but harmless.
|
||||
2. **`from_date`/`to_date` accepted but unused** for Xiaohongshu - relies on API-side time bucketing. Acceptable given API limitation.
|
||||
3. **Over-defensive `setdefault()`** in normalization (id and source_domain already set by search_feeds). Harmless.
|
||||
4. **Error message format inconsistency** between OpenAI and public Reddit paths. Minor.
|
||||
5. **`get_available_sources()` docstring** not updated to reflect Reddit always being available now.
|
||||
6. **No tests included** for new Xiaohongshu module or Reddit public fallback.
|
||||
|
||||
### Security
|
||||
|
||||
- No hardcoded credentials
|
||||
- xsec_token comes from API response (not user input)
|
||||
- Docker `host.docker.internal` default is safe for containerized environments
|
||||
- All URLs use `https://` for public Xiaohongshu
|
||||
|
||||
### Recommendation: MERGE WITH CONDITIONS
|
||||
|
||||
The Reddit public fallback alone makes this worth merging - it makes Reddit work with zero API keys. Xiaohongshu adds value for Chinese-market research.
|
||||
|
||||
**Before merge:**
|
||||
- [ ] Run full test suite to confirm no regressions
|
||||
- [ ] Test Reddit public fallback locally: unset OPENAI_API_KEY, run `python3 scripts/last30days.py "test topic" --search reddit --emit=compact`
|
||||
- [ ] Verify Xiaohongshu gracefully skips when API is unavailable (should show skip message, not crash)
|
||||
- [ ] Update `get_available_sources()` docstring to reflect Reddit always-available change
|
||||
|
||||
**After merge (follow-up):**
|
||||
- [ ] Add `tests/test_xiaohongshu.py` with fixture-based tests
|
||||
- [ ] Add `tests/test_reddit_public.py` for the fallback path
|
||||
- [ ] Consider extracting health check duplication
|
||||
|
||||
---
|
||||
|
||||
## 5. PR #47 - feat: add Apify as unified API provider
|
||||
|
||||
**Author:** lapolazzati | **Size:** +1484/-73 | **Files:** 6 new + orchestrator changes
|
||||
|
||||
### What it does
|
||||
|
||||
Adds Apify as a single-token alternative (`APIFY_API_TOKEN`) covering Reddit, X, TikTok, and Instagram. Existing per-source keys take priority; Apify is a transparent fallback.
|
||||
|
||||
**New modules:**
|
||||
- `apify_client.py` (163 lines) - shared HTTP client for Apify run-sync API
|
||||
- `apify_reddit.py` (195 lines) - Reddit via `trudax/reddit-scraper` actor
|
||||
- `apify_x.py` (221 lines) - X via `apidojo/tweet-scraper` actor
|
||||
- `apify_tiktok.py` (284 lines) - TikTok via `clockworks/tiktok-scraper` actor
|
||||
- `apify_instagram.py` (288 lines) - Instagram via `apify/instagram-reel-scraper` actor
|
||||
|
||||
**Source routing priority:**
|
||||
```
|
||||
Reddit: OpenAI -> Apify -> None
|
||||
X: Bird -> xAI -> Apify -> None
|
||||
TikTok: ScrapeCreators -> Apify -> None
|
||||
Instagram: ScrapeCreators -> Apify -> None
|
||||
```
|
||||
|
||||
### Critical issues
|
||||
|
||||
1. **Actor ID mismatch (BLOCKER):** Code uses different Apify actors than documented in README and plan.md.
|
||||
|
||||
| Source | In Code | In Docs |
|
||||
|--------|---------|---------|
|
||||
| Reddit | `trudax/reddit-scraper` | `automation-lab/reddit-scraper` |
|
||||
| X | `apidojo/tweet-scraper` | `scraper_one/x-posts-search` |
|
||||
| TikTok | `clockworks/tiktok-scraper` | `epctex/tiktok-search-scraper` |
|
||||
| Instagram | `apify/instagram-reel-scraper` | matches |
|
||||
|
||||
Users following README instructions will hit wrong actors.
|
||||
|
||||
2. **No tests (BLOCKER):** 1484 lines of new code with zero test coverage. Each Apify module has its own date parsing, relevance scoring, and response normalization - all untested.
|
||||
|
||||
3. **Significant code duplication:** `apify_tiktok.py` and `apify_instagram.py` share nearly identical:
|
||||
- `_tokenize()` (7 lines)
|
||||
- `_compute_relevance()` (14 lines)
|
||||
- `STOPWORDS` set
|
||||
- `SYNONYMS` dict
|
||||
- `_extract_core_subject()` (~20 lines)
|
||||
|
||||
4. **Version mismatch:** README says v2.9 but git history shows v2.9.4 on main. Version should be higher.
|
||||
|
||||
5. **plan.md included in commit** - implementation planning doc shouldn't be in the final merge.
|
||||
|
||||
### What's good
|
||||
|
||||
- Source routing logic in env.py is clean and backward-compatible
|
||||
- Orchestrator dispatch uses consistent `_source` parameter pattern
|
||||
- Existing source paths are completely untouched
|
||||
- Timeout scaling is appropriate (90s quick, 150s default, 240s deep)
|
||||
- Token handling follows security best practices (parameter passing, Bearer auth)
|
||||
|
||||
### Recommendation: REQUEST CHANGES
|
||||
|
||||
This PR is too large and has too many issues for a clean merge. Request the contributor to:
|
||||
|
||||
**Must fix before any merge:**
|
||||
- [ ] Resolve actor ID mismatches (verify which actors actually work, update code OR docs)
|
||||
- [ ] Add unit tests for all 5 new modules (at minimum: response normalization, date parsing)
|
||||
- [ ] Fix version number
|
||||
- [ ] Remove plan.md from commit
|
||||
|
||||
**Should fix:**
|
||||
- [ ] Extract shared code from apify_tiktok/apify_instagram into `apify_common.py`
|
||||
- [ ] Consider splitting into 2 PRs:
|
||||
- PR A: `apify_client.py` + env.py routing (foundation)
|
||||
- PR B: Individual source modules + tests (features)
|
||||
|
||||
**Comment template for PR:**
|
||||
> Thanks for this contribution - the single-token approach is a great idea for simplifying setup. A few things need fixing before we can merge:
|
||||
>
|
||||
> 1. The Apify actor IDs in the code don't match the README/plan docs. Can you verify which actors are correct and align code + docs?
|
||||
> 2. We need test coverage for the new modules - at minimum normalization tests with sample actor responses.
|
||||
> 3. There's significant duplication between apify_tiktok.py and apify_instagram.py (_tokenize, _compute_relevance, STOPWORDS, SYNONYMS, _extract_core_subject). Could you extract shared code to an apify_common.py?
|
||||
> 4. Version in README should be higher than v2.9.4 (current main).
|
||||
> 5. Please remove plan.md from the commit.
|
||||
|
||||
---
|
||||
|
||||
## 6. Issue #45 - Add support for Gemini CLI
|
||||
|
||||
**Author:** alexferrari88 | **Request:** "Would it be possible to add support for Gemini CLI?"
|
||||
|
||||
### Assessment
|
||||
|
||||
This is a feature request to support Gemini CLI as a runtime alongside Claude Code and Codex. Key considerations:
|
||||
|
||||
- **Scope:** The skill is currently packaged for Claude Code (SKILL.md format) and Codex (~/.codex/skills/). Gemini CLI uses a different skill/extension format.
|
||||
- **Effort:** Would require understanding Gemini CLI's plugin system, creating a compatible manifest, and potentially adapting the Python execution model.
|
||||
- **Priority:** Low - Claude Code and Codex are the primary targets and where the user base is.
|
||||
- **Community:** If the requester wants to contribute, they'd be best positioned to understand Gemini CLI's requirements.
|
||||
|
||||
### Recommendation: ACKNOWLEDGE AND BACKLOG
|
||||
|
||||
- [ ] Respond with: "Thanks for the suggestion! Adding to the backlog. If you're familiar with Gemini CLI's skill/extension format and want to take a crack at it, PRs are welcome. The core Python scripts in `scripts/` are runtime-agnostic - the main work would be creating a Gemini-compatible manifest and deployment path."
|
||||
- [ ] Add a `help wanted` label
|
||||
- [ ] Keep open as a backlog item
|
||||
|
||||
---
|
||||
|
||||
## Priority Summary
|
||||
|
||||
| Priority | Item | Action | Risk |
|
||||
|----------|------|--------|------|
|
||||
| 1 | PR #52 (metadata fix) | Merge now | None |
|
||||
| 2 | Issue #46 (upload error) | Auto-closes with PR #52 | None |
|
||||
| 3 | PR #50 (entity tests) | Run tests, merge | None |
|
||||
| 4 | PR #48 (Xiaohongshu + Reddit fallback) | Test locally, merge with minor conditions | Low |
|
||||
| 5 | Issue #45 (Gemini CLI) | Acknowledge, backlog, label | None |
|
||||
| 6 | PR #47 (Apify unified) | Request changes (blockers found) | Medium-High |
|
||||
Reference in New Issue
Block a user