Matt Van Horn
32992834ee
- Xiaohongshu search via local MCP service (opt-in, zero impact if service not running) - Reddit public JSON fallback (works with zero API keys) - Reddit priority: ScrapeCreators -> OpenAI -> public fallback - Updated env.py: Reddit always available via public fallback Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
275 lines
14 KiB
Markdown
275 lines
14 KiB
Markdown
---
|
|
title: "feat: Last30Days.com - Automated Trending Topic Research"
|
|
type: feat
|
|
date: 2026-02-20
|
|
---
|
|
|
|
# Last30Days.com - Automated Trending Topic Research
|
|
|
|
## Overview
|
|
|
|
A website at Last30Days.com that automatically shows daily trending topics researched by the /last30days engine. The core challenge: the skill is query-driven (you give it a topic), but a trending page needs to *discover* what topics to research. This plan covers how to source trending topics, run them through the engine, and publish results.
|
|
|
|
## Problem Statement
|
|
|
|
The /last30days skill has 2,747 GitHub stars but no public-facing showcase. Users have to install the skill and run it themselves. A website that automatically shows trending topic results would:
|
|
|
|
1. **Drive installs** - people see the quality and want it for their own topics
|
|
2. **SEO surface area** - each topic page is indexable content
|
|
3. **Demonstrate capability** - "here's what /last30days found about X today" with real stats
|
|
4. **Content flywheel** - daily fresh content with zero manual curation
|
|
|
|
## Trending Topic Discovery: The Options
|
|
|
|
The skill has **no existing trending discovery mechanism** - it's entirely query-driven. Here are the available sources, ranked by practicality.
|
|
|
|
### Tier 1: Free, High Signal, Zero Friction
|
|
|
|
| Source | What It Returns | Auth | Cost | Best For |
|
|
|--------|----------------|------|------|----------|
|
|
| **Wikipedia Pageviews** | Top 100 most-viewed articles yesterday | None | Free | General public interest (news, culture, events) |
|
|
| **Hacker News** | Top 30 stories with scores | None | Free | Tech/startup topics |
|
|
| **Reddit r/all/hot + rising** | Hottest and rapidly rising posts | OAuth (free) | Free | Broad internet culture |
|
|
| **Google News RSS** | Top headlines, algorithmically curated | None | Free | Mainstream news |
|
|
| **YouTube mostPopular** | Top 50 trending videos by region | API key | Free (10K units/day) | Pop culture, entertainment |
|
|
|
|
### Tier 2: Very Cheap, High Value
|
|
|
|
| Source | What It Returns | Auth | Cost | Best For |
|
|
|--------|----------------|------|------|----------|
|
|
| **Perplexity Sonar API** | "What's trending today?" with sourced answers | API key | ~$1/month | Meta-aggregator that replaces multiple sources |
|
|
| **Google Trends (pytrends)** | Daily trending Google searches | None | Free but flaky | What people are actually searching |
|
|
| **Bird `getNews()`** | X Explore page trending topics | Browser cookies | Free | Real-time Twitter/X conversation |
|
|
|
|
### Tier 3: Paid, Skip for MVP
|
|
|
|
| Source | Cost | Why Skip |
|
|
|--------|------|----------|
|
|
| X/Twitter API | $200/month minimum | Too expensive; Bird `getNews()` is free |
|
|
| Exploding Topics | $249/month | Overkill for daily trends |
|
|
| TikTok | Gated, requires approval | Application process |
|
|
|
|
### Existing Codebase Hooks
|
|
|
|
Several pieces already exist in the codebase that could be leveraged:
|
|
|
|
1. **Bird `getNews()` is already on disk.** The vendored Bird library at `scripts/lib/vendor/bird-search/` includes `twitter-client-news.js` which fetches X's Explore page tabs (For You, Trending, News, Sports, Entertainment). Only needs a ~30-line `bird-news.mjs` wrapper to expose it. Zero API keys needed.
|
|
|
|
2. **`store.get_trending(days=7)`** at `scripts/store.py:559` ranks watchlist topics by recent finding activity. Could feed "what the skill has been researching" as a meta-signal.
|
|
|
|
3. **Scoring algorithm** at `scripts/lib/score.py` has engagement formulas for Reddit, X, and YouTube that could rank "buzz" by reweighting engagement >> relevance.
|
|
|
|
4. **Brave Trending API** exists but isn't implemented in `brave_search.py`. Could be added.
|
|
|
|
## Proposed Architecture
|
|
|
|
### Topic Discovery Pipeline (daily cron)
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────┐
|
|
│ STEP 1: Fetch trending signals (parallel, free) │
|
|
├─────────────────────────────────────────────────┤
|
|
│ Wikipedia Pageviews ─┐ │
|
|
│ Hacker News Top 30 ─┤ │
|
|
│ Reddit r/all/hot ─┼─→ Raw topics + signals │
|
|
│ Google News RSS ─┤ │
|
|
│ YouTube mostPopular ─┤ │
|
|
│ Bird getNews() ─┘ │
|
|
└─────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────┐
|
|
│ STEP 2: Cluster + rank │
|
|
├─────────────────────────────────────────────────┤
|
|
│ Extract topic keywords from titles │
|
|
│ Cluster by semantic similarity │
|
|
│ Rank by cross-source frequency │
|
|
│ ("Pope Leo XIV" on Wikipedia + Reddit + News │
|
|
│ = high confidence trend) │
|
|
│ Output: top 15-20 topics, ranked │
|
|
└─────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────┐
|
|
│ STEP 3: Run /last30days on top topics │
|
|
├─────────────────────────────────────────────────┤
|
|
│ Top 3-5 topics: full research (--emit=json) │
|
|
│ → Full "What I learned" synthesis articles │
|
|
│ Remaining 10-15: quick research (--quick) │
|
|
│ → Stats teasers (32 X posts, 5 YouTube...) │
|
|
└─────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────┐
|
|
│ STEP 4: Publish to website │
|
|
├─────────────────────────────────────────────────┤
|
|
│ Generate static HTML/JSON │
|
|
│ Deploy to Last30Days.com │
|
|
│ RSS feed for subscribers │
|
|
└─────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Topic Clustering Strategy
|
|
|
|
The hardest part is deduplicating across sources. "Pope Francis Dies" (Google News), "Pope_Francis" (Wikipedia #1 viewed), and a r/worldnews post are all the same topic. Approaches:
|
|
|
|
**Option A: LLM clustering (recommended for MVP)**
|
|
Feed all raw titles to an LLM and ask it to cluster into distinct topics with a representative label. ~$0.01 per day via a fast model. Simple, accurate, handles edge cases.
|
|
|
|
**Option B: TF-IDF + cosine similarity**
|
|
Extract keywords, compute pairwise similarity, agglomerative clustering. No API cost but worse on paraphrased titles.
|
|
|
|
**Option C: Embedding similarity**
|
|
Embed all titles, cluster by cosine distance. Better than TF-IDF, costs a few cents per day.
|
|
|
|
### Website Architecture
|
|
|
|
**Option A: Static site (recommended for MVP)**
|
|
- Daily cron generates JSON + static HTML
|
|
- Host on GitHub Pages, Cloudflare Pages, or Vercel
|
|
- Zero server cost, zero maintenance
|
|
- Framework: plain HTML/CSS, or minimal Astro/11ty
|
|
|
|
**Option B: Next.js with ISR**
|
|
- Incremental Static Regeneration rebuilds pages daily
|
|
- More flexibility for future features (search, filtering, user accounts)
|
|
- Hosting: Vercel free tier
|
|
|
|
**Option C: Full web app**
|
|
- Database-backed, real-time updates, user accounts
|
|
- Overkill for MVP
|
|
|
|
### Cost Estimate (daily operation)
|
|
|
|
| Item | Cost |
|
|
|------|------|
|
|
| Trending source APIs | $0 (all free tier) |
|
|
| LLM clustering (fast model) | ~$0.01/day |
|
|
| Full research on 3-5 topics | ~$0.10-0.50/day (OpenAI API for Reddit search) |
|
|
| Quick research on 10-15 topics | ~$0.05-0.20/day |
|
|
| Static hosting | $0 (GitHub/Cloudflare Pages) |
|
|
| **Total** | **~$5-20/month** |
|
|
|
|
## Implementation Phases
|
|
|
|
### Phase 1: Topic Discovery Script (MVP)
|
|
|
|
Build `scripts/discover_trending.py` that:
|
|
- Fetches Wikipedia Pageviews, HN top stories, Reddit hot, Google News RSS in parallel
|
|
- Filters out evergreen/non-topical Wikipedia pages (e.g., "Main Page", "ChatGPT" permanent traffic)
|
|
- Uses a fast LLM to cluster raw titles into 15-20 distinct topics
|
|
- Outputs ranked JSON: `[{topic, sources, confidence, category}]`
|
|
|
|
**Acceptance criteria:**
|
|
- [ ] Fetches from at least 4 free trending sources in parallel
|
|
- [ ] Clusters raw titles into deduplicated topics via LLM
|
|
- [ ] Filters Wikipedia evergreen pages (maintain a blocklist)
|
|
- [ ] Outputs ranked JSON with topic name, source count, category
|
|
- [ ] Runs in < 60 seconds
|
|
- [ ] No paid API keys required for discovery (only LLM clustering)
|
|
|
|
### Phase 2: Research Automation
|
|
|
|
Wire discovered topics into the existing /last30days engine:
|
|
- Top 3-5 topics: `python3 scripts/last30days.py "$TOPIC" --emit=json --deep`
|
|
- Remaining topics: `python3 scripts/last30days.py "$TOPIC" --emit=json --quick`
|
|
- Store all results in `~/.local/share/last30days/trending/YYYY-MM-DD/`
|
|
|
|
**Acceptance criteria:**
|
|
- [ ] Orchestration script runs discovery then research sequentially
|
|
- [ ] Full research on top N topics, quick on the rest
|
|
- [ ] Results stored as dated JSON files
|
|
- [ ] Total daily runtime < 30 minutes
|
|
- [ ] Handles timeouts/failures gracefully (skip topic, continue)
|
|
|
|
### Phase 3: Website Generation
|
|
|
|
Build a static site generator that reads the daily JSON and produces Last30Days.com:
|
|
- Homepage: today's trending topics grid (title, category, key stat, source badges)
|
|
- Topic pages: full synthesis for showcase topics, stats teaser for others
|
|
- Archive: previous days accessible by date
|
|
- RSS feed
|
|
- CTA: "Want to research your own topic? Install the skill"
|
|
|
|
**Acceptance criteria:**
|
|
- [ ] Static HTML generated from daily JSON
|
|
- [ ] Homepage shows today's 15-20 trending topics
|
|
- [ ] 3-5 showcase topic pages with full synthesis
|
|
- [ ] Remaining topics show stats teasers + install CTA
|
|
- [ ] Deploys to Last30Days.com (Cloudflare Pages or similar)
|
|
- [ ] RSS feed for daily updates
|
|
- [ ] Mobile responsive
|
|
|
|
### Phase 4: Bird Trending Integration (bonus)
|
|
|
|
Wire up the already-vendored Bird `getNews()` for X trending:
|
|
- Create `scripts/lib/vendor/bird-search/bird-news.mjs` (~30 lines)
|
|
- Add X Explore trending data as a 5th discovery source
|
|
- X trends are the fastest-moving signal and fill the "what's happening right now" gap
|
|
|
|
**Acceptance criteria:**
|
|
- [ ] `bird-news.mjs` wrapper exposes X Explore trending topics
|
|
- [ ] Integrated into discovery pipeline as an additional source
|
|
- [ ] Falls back gracefully if no X session cookies available
|
|
|
|
## Alternative Approaches Considered
|
|
|
|
### Perplexity-only approach
|
|
Just ask Perplexity Sonar "what are the top 20 trending topics today?" daily. Simpler, but:
|
|
- Single point of failure
|
|
- Less transparent (can't show "sourced from Reddit, Wikipedia, HN")
|
|
- Model may hallucinate or miss niche topics
|
|
- **Verdict:** Good fallback, not primary.
|
|
|
|
### Curated topics (manual)
|
|
Manually pick topics each day. Defeats the purpose of automation. Could supplement for "editorial picks."
|
|
|
|
### Social listening tools (Brandwatch, Sprout Social, etc.)
|
|
Expensive ($500+/month), enterprise-focused, overkill.
|
|
|
|
## Technical Considerations
|
|
|
|
- **Rate limits:** Wikipedia, HN, and RSS have no practical limits for 1 daily call. Reddit's 100 QPM is generous. YouTube's 10K units/day allows ~3,000 `mostPopular` calls.
|
|
- **Wikipedia filtering:** The top Wikipedia pages are always "Main Page", "Special:Search", etc. Need a blocklist of ~50 evergreen pages plus heuristics (skip pages under 1,000 characters, skip disambiguation pages).
|
|
- **Cron timing:** Run discovery at ~2am UTC (after Wikipedia pageviews finalize for previous day). Run research at ~3am UTC. Deploy site by ~5am UTC.
|
|
- **Cost control:** The /last30days engine uses OpenAI API for Reddit search. Full research on 5 topics * $0.05-0.10 each = ~$0.25-0.50/day. Quick research is cheaper.
|
|
- **Domain:** Last30Days.com needs to be registered (check availability).
|
|
|
|
## Dependencies & Risks
|
|
|
|
| Risk | Mitigation |
|
|
|------|------------|
|
|
| pytrends breaks (Google changes) | pytrends is a bonus source, not required. Other 4+ sources sufficient. |
|
|
| Wikipedia pageviews delayed | Fall back to Perplexity Sonar for day's topics |
|
|
| OpenAI API cost spikes | Cap at 5 full + 15 quick topics per day; use --quick for most |
|
|
| Bird cookie auth stops working | X trending is a bonus source; discovery works without it |
|
|
| Domain not available | Check availability; alternatives: last30.day, last30days.app |
|
|
|
|
## Success Metrics
|
|
|
|
- Daily automated publishing with zero manual intervention
|
|
- 15-20 trending topics surfaced daily
|
|
- 3-5 full synthesis articles per day
|
|
- Site loads in < 2 seconds (static)
|
|
- Drives measurable GitHub star growth / skill installs
|
|
|
|
## References
|
|
|
|
### Trending APIs (free)
|
|
- Wikipedia Pageviews: `wikimedia.org/api/rest_v1/metrics/pageviews/top/{project}/{access}/{year}/{month}/{day}`
|
|
- Hacker News: `hacker-news.firebaseio.com/v0/topstories.json`
|
|
- Reddit: `oauth.reddit.com/r/all/hot` (or `reddit.com/r/all/hot.json` unauthenticated)
|
|
- Google News RSS: `news.google.com/rss`
|
|
- YouTube: `googleapis.com/youtube/v3/videos?chart=mostPopular`
|
|
|
|
### Existing codebase hooks
|
|
- Bird `getNews()`: `scripts/lib/vendor/bird-search/vendor/package/dist/lib/twitter-client-news.js`
|
|
- Store trending: `scripts/store.py:559` (`get_trending()`)
|
|
- Scoring: `scripts/lib/score.py` (engagement formulas)
|
|
- Brave Trending API: not implemented, available in Brave docs
|
|
|
|
### Inspiration
|
|
- [Keep a Changelog](https://keepachangelog.com/) - clean daily update format
|
|
- [Hacker News front page](https://news.ycombinator.com/) - minimal trending UI
|
|
- [Exploding Topics](https://explodingtopics.com/) - trending topic showcase (paid, $249/mo)
|