last30days-skill/docs/plans/2026-02-20-feat-last30days-com-trending-topics-plan.md

title, type, date

title	type	date
feat: Last30Days.com - Automated Trending Topic Research	feat	2026-02-20

Last30Days.com - Automated Trending Topic Research

Overview

A website at Last30Days.com that automatically shows daily trending topics researched by the /last30days engine. The core challenge: the skill is query-driven (you give it a topic), but a trending page needs to discover what topics to research. This plan covers how to source trending topics, run them through the engine, and publish results.

Problem Statement

The /last30days skill has 2,747 GitHub stars but no public-facing showcase. Users have to install the skill and run it themselves. A website that automatically shows trending topic results would:

1. Drive installs - people see the quality and want it for their own topics
2. SEO surface area - each topic page is indexable content
3. Demonstrate capability - "here's what /last30days found about X today" with real stats
4. Content flywheel - daily fresh content with zero manual curation

Trending Topic Discovery: The Options

The skill has no existing trending discovery mechanism - it's entirely query-driven. Here are the available sources, ranked by practicality.

Tier 1: Free, High Signal, Zero Friction

Source	What It Returns	Auth	Cost	Best For
Wikipedia Pageviews	Top 100 most-viewed articles yesterday	None	Free	General public interest (news, culture, events)
Hacker News	Top 30 stories with scores	None	Free	Tech/startup topics
Reddit r/all/hot + rising	Hottest and rapidly rising posts	OAuth (free)	Free	Broad internet culture
Google News RSS	Top headlines, algorithmically curated	None	Free	Mainstream news
YouTube mostPopular	Top 50 trending videos by region	API key	Free (10K units/day)	Pop culture, entertainment

Tier 2: Very Cheap, High Value

Source	What It Returns	Auth	Cost	Best For
Perplexity Sonar API	"What's trending today?" with sourced answers	API key	~$1/month	Meta-aggregator that replaces multiple sources
Google Trends (pytrends)	Daily trending Google searches	None	Free but flaky	What people are actually searching
Bird getNews()	X Explore page trending topics	Browser cookies	Free	Real-time Twitter/X conversation

Tier 3: Paid, Skip for MVP

Source	Cost	Why Skip
X/Twitter API	$200/month minimum	Too expensive; Bird getNews() is free
Exploding Topics	$249/month	Overkill for daily trends
TikTok	Gated, requires approval	Application process

Existing Codebase Hooks

Several pieces already exist in the codebase that could be leveraged:

1. Bird getNews() is already on disk. The vendored Bird library at scripts/lib/vendor/bird-search/ includes twitter-client-news.js which fetches X's Explore page tabs (For You, Trending, News, Sports, Entertainment). Only needs a ~30-line bird-news.mjs wrapper to expose it. Zero API keys needed.

2. store.get_trending(days=7) at scripts/store.py:559 ranks watchlist topics by recent finding activity. Could feed "what the skill has been researching" as a meta-signal.

3. Scoring algorithm at scripts/lib/score.py has engagement formulas for Reddit, X, and YouTube that could rank "buzz" by reweighting engagement >> relevance.

4. Brave Trending API exists but isn't implemented in brave_search.py. Could be added.

Proposed Architecture

Topic Discovery Pipeline (daily cron)

┌─────────────────────────────────────────────────┐
│ STEP 1: Fetch trending signals (parallel, free) │
├─────────────────────────────────────────────────┤
│ Wikipedia Pageviews  ─┐                         │
│ Hacker News Top 30   ─┤                         │
│ Reddit r/all/hot     ─┼─→ Raw topics + signals  │
│ Google News RSS      ─┤                         │
│ YouTube mostPopular  ─┤                         │
│ Bird getNews()       ─┘                         │
└─────────────────────────────────────────────────┘
                    │
                    ▼
┌─────────────────────────────────────────────────┐
│ STEP 2: Cluster + rank                          │
├─────────────────────────────────────────────────┤
│ Extract topic keywords from titles              │
│ Cluster by semantic similarity                  │
│ Rank by cross-source frequency                  │
│ ("Pope Leo XIV" on Wikipedia + Reddit + News    │
│   = high confidence trend)                      │
│ Output: top 15-20 topics, ranked                │
└─────────────────────────────────────────────────┘
                    │
                    ▼
┌─────────────────────────────────────────────────┐
│ STEP 3: Run /last30days on top topics           │
├─────────────────────────────────────────────────┤
│ Top 3-5 topics: full research (--emit=json)     │
│   → Full "What I learned" synthesis articles    │
│ Remaining 10-15: quick research (--quick)       │
│   → Stats teasers (32 X posts, 5 YouTube...)    │
└─────────────────────────────────────────────────┘
                    │
                    ▼
┌─────────────────────────────────────────────────┐
│ STEP 4: Publish to website                      │
├─────────────────────────────────────────────────┤
│ Generate static HTML/JSON                       │
│ Deploy to Last30Days.com                        │
│ RSS feed for subscribers                        │
└─────────────────────────────────────────────────┘

Topic Clustering Strategy

The hardest part is deduplicating across sources. "Pope Francis Dies" (Google News), "Pope_Francis" (Wikipedia #1 viewed), and a r/worldnews post are all the same topic. Approaches:

Option A: LLM clustering (recommended for MVP) Feed all raw titles to an LLM and ask it to cluster into distinct topics with a representative label. ~$0.01 per day via a fast model. Simple, accurate, handles edge cases.

Option B: TF-IDF + cosine similarity Extract keywords, compute pairwise similarity, agglomerative clustering. No API cost but worse on paraphrased titles.

Option C: Embedding similarity Embed all titles, cluster by cosine distance. Better than TF-IDF, costs a few cents per day.

Website Architecture

Option A: Static site (recommended for MVP)

• Daily cron generates JSON + static HTML
• Host on GitHub Pages, Cloudflare Pages, or Vercel
• Zero server cost, zero maintenance
• Framework: plain HTML/CSS, or minimal Astro/11ty

Option B: Next.js with ISR

• Incremental Static Regeneration rebuilds pages daily
• More flexibility for future features (search, filtering, user accounts)
• Hosting: Vercel free tier

Option C: Full web app

• Database-backed, real-time updates, user accounts
• Overkill for MVP

Cost Estimate (daily operation)

Item	Cost
Trending source APIs	$0 (all free tier)
LLM clustering (fast model)	~$0.01/day
Full research on 3-5 topics	~$0.10-0.50/day (OpenAI API for Reddit search)
Quick research on 10-15 topics	~$0.05-0.20/day
Static hosting	$0 (GitHub/Cloudflare Pages)
Total	~$5-20/month

Implementation Phases

Phase 1: Topic Discovery Script (MVP)

Build scripts/discover_trending.py that:

• Fetches Wikipedia Pageviews, HN top stories, Reddit hot, Google News RSS in parallel
• Filters out evergreen/non-topical Wikipedia pages (e.g., "Main Page", "ChatGPT" permanent traffic)
• Uses a fast LLM to cluster raw titles into 15-20 distinct topics
• Outputs ranked JSON: [{topic, sources, confidence, category}]

Acceptance criteria:

• Fetches from at least 4 free trending sources in parallel
• Clusters raw titles into deduplicated topics via LLM
• Filters Wikipedia evergreen pages (maintain a blocklist)
• Outputs ranked JSON with topic name, source count, category
• Runs in < 60 seconds
• No paid API keys required for discovery (only LLM clustering)

Phase 2: Research Automation

Wire discovered topics into the existing /last30days engine:

• Top 3-5 topics: python3 scripts/last30days.py "$TOPIC" --emit=json --deep
• Remaining topics: python3 scripts/last30days.py "$TOPIC" --emit=json --quick
• Store all results in ~/.local/share/last30days/trending/YYYY-MM-DD/

Acceptance criteria:

• Orchestration script runs discovery then research sequentially
• Full research on top N topics, quick on the rest
• Results stored as dated JSON files
• Total daily runtime < 30 minutes
• Handles timeouts/failures gracefully (skip topic, continue)

Phase 3: Website Generation

Build a static site generator that reads the daily JSON and produces Last30Days.com:

• Homepage: today's trending topics grid (title, category, key stat, source badges)
• Topic pages: full synthesis for showcase topics, stats teaser for others
• Archive: previous days accessible by date
• RSS feed
• CTA: "Want to research your own topic? Install the skill"

Acceptance criteria:

• Static HTML generated from daily JSON
• Homepage shows today's 15-20 trending topics
• 3-5 showcase topic pages with full synthesis
• Remaining topics show stats teasers + install CTA
• Deploys to Last30Days.com (Cloudflare Pages or similar)
• RSS feed for daily updates
• Mobile responsive

Phase 4: Bird Trending Integration (bonus)

Wire up the already-vendored Bird getNews() for X trending:

• Create scripts/lib/vendor/bird-search/bird-news.mjs (~30 lines)
• Add X Explore trending data as a 5th discovery source
• X trends are the fastest-moving signal and fill the "what's happening right now" gap

Acceptance criteria:

• bird-news.mjs wrapper exposes X Explore trending topics
• Integrated into discovery pipeline as an additional source
• Falls back gracefully if no X session cookies available

Alternative Approaches Considered

Perplexity-only approach

Just ask Perplexity Sonar "what are the top 20 trending topics today?" daily. Simpler, but:

• Single point of failure
• Less transparent (can't show "sourced from Reddit, Wikipedia, HN")
• Model may hallucinate or miss niche topics
• Verdict: Good fallback, not primary.

Curated topics (manual)

Manually pick topics each day. Defeats the purpose of automation. Could supplement for "editorial picks."

Social listening tools (Brandwatch, Sprout Social, etc.)

Expensive ($500+/month), enterprise-focused, overkill.

Technical Considerations

• Rate limits: Wikipedia, HN, and RSS have no practical limits for 1 daily call. Reddit's 100 QPM is generous. YouTube's 10K units/day allows ~3,000 mostPopular calls.
• Wikipedia filtering: The top Wikipedia pages are always "Main Page", "Special:Search", etc. Need a blocklist of ~50 evergreen pages plus heuristics (skip pages under 1,000 characters, skip disambiguation pages).
• Cron timing: Run discovery at ~2am UTC (after Wikipedia pageviews finalize for previous day). Run research at ~3am UTC. Deploy site by ~5am UTC.
• Cost control: The /last30days engine uses OpenAI API for Reddit search. Full research on 5 topics * $0.05-0.10 each = ~$0.25-0.50/day. Quick research is cheaper.
• Domain: Last30Days.com needs to be registered (check availability).

Dependencies & Risks

Risk	Mitigation
pytrends breaks (Google changes)	pytrends is a bonus source, not required. Other 4+ sources sufficient.
Wikipedia pageviews delayed	Fall back to Perplexity Sonar for day's topics
OpenAI API cost spikes	Cap at 5 full + 15 quick topics per day; use --quick for most
Bird cookie auth stops working	X trending is a bonus source; discovery works without it
Domain not available	Check availability; alternatives: last30.day, last30days.app

Success Metrics

• Daily automated publishing with zero manual intervention
• 15-20 trending topics surfaced daily
• 3-5 full synthesis articles per day
• Site loads in < 2 seconds (static)
• Drives measurable GitHub star growth / skill installs

References

Trending APIs (free)

• Wikipedia Pageviews: wikimedia.org/api/rest_v1/metrics/pageviews/top/{project}/{access}/{year}/{month}/{day}
• Hacker News: hacker-news.firebaseio.com/v0/topstories.json
• Reddit: oauth.reddit.com/r/all/hot (or reddit.com/r/all/hot.json unauthenticated)
• Google News RSS: news.google.com/rss
• YouTube: googleapis.com/youtube/v3/videos?chart=mostPopular

Existing codebase hooks

• Bird getNews(): scripts/lib/vendor/bird-search/vendor/package/dist/lib/twitter-client-news.js
• Store trending: scripts/store.py:559 (get_trending())
• Scoring: scripts/lib/score.py (engagement formulas)
• Brave Trending API: not implemented, available in Brave docs

Inspiration

• Keep a Changelog - clean daily update format
• Hacker News front page - minimal trending UI
• Exploding Topics - trending topic showcase (paid, $249/mo)