last30days-skill/docs/plans/2026-02-06-feat-free-reddit-mcp-integration-plan.md

title, type, date

title	type	date
feat: Free Reddit via MCP — Remove OpenAI Key Requirement	feat	2026-02-06

feat: Free Reddit via MCP — Remove OpenAI Key Requirement

Overview

Replace the OpenAI Responses API (paid) for Reddit searching with a free, zero-config Reddit MCP server, eliminating the need for an OPENAI_API_KEY to get Reddit results in the last30days skill. This work happens in a new forked repo to keep the current release branch clean.

Problem Statement

Today, the last30days skill requires an OpenAI API key (OPENAI_API_KEY) to search Reddit. This is the most expensive dependency in the stack — OpenAI charges per-call for the Responses API with web search. Users without an OpenAI key get zero Reddit results and fall back to WebSearch-only mode, which loses the engagement metrics (upvotes, comments) that make last30days uniquely valuable.

Goal: Make Reddit search completely free with no API key registration required.

Research Findings

Option Analysis

Five approaches were evaluated. Two stand out:

Option	Free?	Search?	Engagement?	Setup	Notes
reddit-mcp-buddy (Node.js)	Yes (anonymous mode)	Yes	Yes	One CLI command	371 stars, 3-tier auth, most popular
mcp-server-reddit (Python)	Yes (redditwarp)	No	Yes	pip install	Recommended by ClaudeLog, but browse-only
.json URL trick (curl)	Yes	Yes	Yes	Zero	~10 req/min, no dependencies
Reddit OAuth (PRAW)	Free but needs registration	Yes	Yes	App registration	100 req/min, most reliable
RSS feeds	Yes	Basic	No	Zero	Least useful, no metrics

Top Contender: reddit-mcp-buddy

GitHub: karanb192/reddit-mcp-buddy

• 371 stars, actively maintained (last update: Jan 29, 2026)
• Anonymous mode — zero credentials, ~10 req/min
• MCP tools: search_reddit, browse_subreddit, get_post_details, user_analysis, reddit_explain
• Install: claude mcp add --transport stdio reddit-mcp-buddy -s user -- npx -y reddit-mcp-buddy
• Returns full engagement metrics (score, num_comments, upvote_ratio)
• TypeScript/Node.js (npx, no Python dependency)

Strong Alternative: .json URL trick

• Append .json to any Reddit URL → full JSON response
• Search: https://www.reddit.com/search.json?q=TOPIC&sort=relevance&t=month&limit=100
• Zero dependencies, zero auth, zero setup
• Returns same data as official API (score, num_comments, created_utc, upvote_ratio)
• ~10 req/min rate limit (sufficient for skill use)
• Can be called via curl or Python urllib

Also Considered

• Hawstein/mcp-server-reddit (134 stars, Python, no auth) — No search tool, only browse subreddits. Cannot replace OpenAI's keyword search capability.
• Arindam200/reddit-mcp (262 stars, PRAW) — Has search but requires Reddit OAuth credentials. Not truly zero-config.
• adhikasp/mcp-reddit (348 stars) — Hot threads only, no search. Last updated Dec 2024.

Proposed Solution

Approach A: MCP-First with .json Fallback (Recommended)

Add a new module mcp_reddit.py that:

1. Detects if a Reddit MCP server is configured in the user's Claude Code session
2. If MCP available: Calls MCP search_reddit tool via the skill's allowed-tools (the skill already allows tools — MCP tools become available when configured)
3. If no MCP: Falls back to Reddit's .json URL endpoints via urllib (stdlib, no dependencies)
4. Normalize MCP/JSON results to the existing RedditItem schema
5. Enrich with reddit_enrich.py (already works — fetches real thread data)

This gives users three tiers of Reddit access:

• Tier 1 (best): Reddit MCP installed → full search + engagement via MCP tools
• Tier 2 (good): No MCP, no keys → .json URL search + enrichment
• Tier 3 (existing): OpenAI key present → existing openai_reddit.py still works (backward compat)

Approach B: .json-Only (Simpler)

Skip MCP entirely. Add a reddit_json.py module that uses https://www.reddit.com/search.json directly. Simpler but misses the MCP ecosystem integration.

Approach C: MCP-Only (Cleaner)

Require MCP setup. Simpler code but adds a user setup step (claude mcp add ...).

Recommendation: Approach A — MCP-first with .json fallback gives zero-config Reddit search for everyone while rewarding users who set up MCP with a better experience.

Technical Approach

Architecture

User invokes /last30days "topic"
  │
  ├─ env.py detects sources:
  │   ├─ MCP reddit available? → mcp_reddit.py
  │   ├─ No MCP, no key? → reddit_json.py (.json URL trick)
  │   └─ OPENAI_API_KEY set? → openai_reddit.py (existing, backward compat)
  │
  ├─ X search (Bird CLI / xAI — unchanged)
  │
  └─ Normalize → Score → Dedupe → Render (unchanged)

New Files

scripts/lib/reddit_json.py

• Uses urllib.request (stdlib) to call Reddit's .json search endpoint
• URL: https://www.reddit.com/search.json?q={topic}&sort=relevance&t=month&limit=100
• Custom User-Agent header (required by Reddit)
• Parses response into the same format as openai_reddit.py output
• Handles pagination via after token if depth=deep (multiple requests)
• Rate limiting: 1 second delay between requests

scripts/lib/mcp_reddit.py

• Detects MCP availability (check if Reddit MCP tools exist in session)
• Formats MCP tool calls for search_reddit with topic + date filters
• Normalizes MCP response to match existing RedditItem schema
• Falls back to reddit_json.py if MCP unavailable

Modified Files

• scripts/lib/env.py — Add Reddit source detection: MCP → .json → OpenAI
• scripts/last30days.py — Route Reddit search through new priority chain
• scripts/lib/normalize.py — Add normalizer for .json endpoint response format
• SKILL.md — Document MCP setup as optional enhancement

MCP Integration Design

The MCP approach has a subtle challenge: the last30days skill runs as a Python subprocess (python3 last30days.py), but MCP tools are available to Claude's session, not to subprocesses.

Two paths to solve this:

Path 1: SKILL.md orchestration — The SKILL.md workflow calls the MCP search_reddit tool directly (before or in parallel with the Python script), saves the MCP response to a temp file, and the Python script reads it:

SKILL.md workflow:
1. Call MCP search_reddit → save to /tmp/last30days_mcp_reddit.json
2. Call python3 last30days.py --reddit-from=/tmp/last30days_mcp_reddit.json --emit=compact
3. Python script reads pre-fetched Reddit data instead of calling OpenAI

Path 2: .json only for subprocess — The Python script uses .json URLs directly (no MCP needed in subprocess). MCP is a bonus for users who want Claude to also browse specific threads interactively.

Recommendation: Path 2 for MVP, Path 1 as enhancement. The .json approach is self-contained, testable, and doesn't require SKILL.md workflow changes. MCP can be layered on later.

Source Priority Chain (updated env.py)

def get_reddit_source(config: dict) -> str:
    """Returns: 'mcp', 'json', 'openai', or 'none'"""
    # 1. Check for pre-fetched MCP data (from SKILL.md)
    if os.path.exists(MCP_REDDIT_CACHE_PATH):
        return 'mcp'
    # 2. Always available — no key needed
    # (reddit .json endpoints are free)
    return 'json'
    # 3. OpenAI key present → legacy path
    # if config.get('OPENAI_API_KEY'):
    #     return 'openai'

For MVP, the .json path is always available so it becomes the default. OpenAI path remains as opt-in for users who want higher rate limits.

Data Mapping: .json → RedditItem

Reddit .json search returns data.children[].data with:

{
  "title": "Post title",
  "permalink": "/r/subreddit/comments/abc123/...",
  "subreddit": "ClaudeAI",
  "score": 42,
  "num_comments": 15,
  "upvote_ratio": 0.95,
  "created_utc": 1738800000,
  "selftext": "Post body...",
  "url": "https://...",
  "author": "username"
}

Maps cleanly to existing RedditItem:

.json field	RedditItem field	Notes
title	title	Direct
permalink	url	Prepend https://www.reddit.com
subreddit	subreddit	Direct
score	engagement.score	Direct
num_comments	engagement.num_comments	Direct
upvote_ratio	engagement.upvote_ratio	Direct
created_utc	date	Convert epoch → YYYY-MM-DD
selftext	(used for relevance)	AI relevance scoring needed
author	(not in current schema)	Ignore for now

Relevance Scoring Without AI

The current openai_reddit.py gets relevance scores from OpenAI (the AI judges how relevant each result is). With .json endpoints, we lose that AI relevance judgment.

Options:

1. Keyword matching — Score based on how many query terms appear in title + selftext. Simple but effective.
2. TF-IDF-like — Weight rarer query terms higher. More accurate but more code.
3. Let Claude judge — Pass results to Claude in SKILL.md and have Claude score relevance. Most accurate but changes the workflow.
4. Skip relevance, rely on Reddit's sort — Reddit's sort=relevance already ranks by relevance. Trust it and use position-based scoring (first result = 1.0, last = 0.5).

Recommendation: Option 4 for MVP. Reddit's search relevance ranking is already good. Use position-based relevance (1.0 → 0.5 linear decay over result set) combined with the existing engagement-based scoring. This requires zero external dependencies and no AI calls.

Implementation Phases

Phase 0: Fork Repository

• Create new repo last30days-free-reddit (or branch in private repo)
• git clone /Users/mvanhorn/last30days-skill-private /Users/mvanhorn/last30days-free-reddit
• Create feature branch: feat/free-reddit
• Verify all existing tests pass on the new branch

Phase 1: reddit_json.py — Core .json Search

• Create scripts/lib/reddit_json.py
  • search_reddit_json(topic: str, depth: str, date_from: str, date_to: str) -> list[dict]
  • Build search URL with q, sort=relevance, t=month, limit (25/50/100 by depth)
  • Custom User-Agent: last30days-skill/2.0 (by /u/last30days-bot)
  • Parse data.children[].data → list of raw dicts
  • Handle pagination for deep mode (follow after token, max 3 pages)
  • Rate limit: time.sleep(1.0) between requests
  • Error handling: 429 (rate limit), 403 (blocked), network errors → return empty + error msg
• Create tests/test_reddit_json.py
  • Mock HTTP responses using fixture files
  • Test: basic search parsing, pagination, rate limit handling, error cases
• Create fixtures/reddit_json_search_sample.json
  • Real .json search response (sanitized)

Phase 2: Normalize + Score .json Results

• Add normalize_reddit_json() to scripts/lib/normalize.py
  • Convert .json response format → RedditItem schema
  • created_utc (epoch) → YYYY-MM-DD string
  • permalink → full URL
  • Position-based relevance: 1.0 - (index / total * 0.5) → range [1.0, 0.5]
  • Date confidence: "high" (Reddit provides exact timestamps)
• Update scripts/lib/score.py if needed
  • .json results already have real engagement metrics — existing scoring formula works as-is
  • No penalty needed (unlike WebSearch which lacks engagement)
• Add tests for normalization + scoring of .json data

Phase 3: Integration into Orchestrator

• Update scripts/lib/env.py
  • Add get_reddit_source(config) -> str returning 'json', 'openai', or 'none'
  • Priority: .json always available (default), 'openai' if key present and user prefers
  • Add REDDIT_SOURCE config option: auto (default), json, openai
  • Update get_available_sources() to always include Reddit (since .json is free)
  • Update get_missing_keys() — Reddit no longer shows as "missing"
• Update scripts/last30days.py
  • Add _search_reddit_json() function alongside existing _search_reddit()
  • Route based on get_reddit_source(): json → _search_reddit_json(), openai → _search_reddit()
  • Skip reddit_enrich.py for .json results (already have real engagement metrics!)
  • Update progress/stats output to show source: "Reddit (free)" vs "Reddit (OpenAI)"
• Update SKILL.md promo messaging
  • Remove "Add OPENAI_API_KEY for Reddit" messaging
  • Instead: "Reddit search included free! Add OpenAI key for AI-enhanced relevance scoring."
• Integration tests: full pipeline with .json mock data

Phase 4: Testing & Polish

• Run all existing tests — ensure backward compatibility
• Manual test: invoke /last30days with NO API keys → should get Reddit + WebSearch results
• Manual test: invoke with OPENAI_API_KEY → should still use OpenAI path (backward compat)
• Manual test: compare result quality — .json vs OpenAI for same topic
• Update README.md — document free Reddit access
• Update SPEC.md — document new source priority chain

Phase 5 (Future): MCP Enhancement Layer

• Detect Reddit MCP in Claude's session
• SKILL.md pre-fetches via MCP search_reddit → saves to temp file
• Python script reads pre-fetched MCP data via --reddit-from= flag
• MCP results get AI-judged relevance (since Claude sees them)
• Better than .json position-based relevance

Acceptance Criteria

Functional

• /last30days "any topic" returns Reddit results with zero API keys configured
• Reddit results include real engagement metrics (score, comments, upvote_ratio)
• Results are properly scored, deduped, and rendered (same quality as OpenAI path)
• Existing OpenAI path still works when key is present
• Existing X search (Bird CLI / xAI) is unchanged
• Stats box shows correct source indicator ("Reddit (free)" or "Reddit (OpenAI)")

Non-Functional

• No external Python packages (stdlib only — urllib.request, json, time)
• Respects Reddit rate limits (~10 req/min for unauthenticated)
• Graceful degradation if Reddit blocks requests (429/403 → empty results + error msg)
• All new code has unit tests with fixtures (no live API calls in tests)

Quality Gates

• All existing tests pass (zero regressions)
• New tests cover: search parsing, normalization, scoring, error handling, pagination
• Manual smoke test passes with zero API keys

Risk Analysis

Risk	Likelihood	Impact	Mitigation
Reddit blocks .json endpoints	Low	High	Fall back to OpenAI path; monitor for 429s
.json rate limit too restrictive	Medium	Medium	1s delay between requests; cache results
Relevance quality lower without AI scoring	Medium	Medium	Reddit's sort=relevance is decent; can add keyword scoring later
Reddit changes .json response format	Low	Medium	Schema validation in normalize; fixture-based tests catch changes
Node.js MCP server dependency conflicts	N/A (Phase 5)	Low	MCP is optional enhancement, not required

Dependencies

• None — this feature uses only Python stdlib and Reddit's public .json endpoints
• Phase 5 (future) would add optional dependency on an MCP server

Repository Setup

# Fork into new working repo
cp -r /Users/mvanhorn/last30days-skill-private /Users/mvanhorn/last30days-free-reddit
cd /Users/mvanhorn/last30days-free-reddit

# Create feature branch
git checkout -b feat/free-reddit

# Verify existing tests
python3 -m pytest tests/ -v

The new repo is disposable — once the feature is validated, changes merge back into last30days-skill-private via PR or cherry-pick.

References

Internal

• scripts/lib/openai_reddit.py — Current Reddit search (to be replaced/supplemented)
• scripts/lib/env.py:get_available_sources() — Source detection logic to update
• scripts/lib/normalize.py — Add .json normalizer alongside existing
• scripts/lib/reddit_enrich.py — May be skippable for .json results (already have engagement)
• scripts/lib/schema.py:RedditItem — Target schema (unchanged)

External

• Reddit .json search endpoint
• Simon Willison — Scraping Reddit via JSON API
• karanb192/reddit-mcp-buddy — Best MCP option for Phase 5
• Hawstein/mcp-server-reddit — ClaudeLog-recommended MCP (no search though)
• Reddit API Rate Limits Guide

Research Sources

• last30days skill output — community recommendations for Reddit search tools
• GitHub search — 10+ Reddit MCP repos evaluated
• npm/PyPI registries — package availability confirmed
• ClaudeLog — Reddit MCP reference