sawz/last30days-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(hackernews): add Hacker News as 5th research source

Browse Source 
Add HN search via free Algolia API (no key needed). Two-phase approach:
search for stories, then enrich top ones with comments. Integrated into
the full pipeline (normalize, score, dedupe, render) running in parallel
with Reddit/X/YouTube. Source priority: Reddit > X > HN > YouTube > Web.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
2026-02-24 18:33:31 -08:00
parent 427a4e453d
commit 38a7ea253e
12 changed files with 1119 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SKILL.md
+13 -7
View File
@@ -1,7 +1,7 @@
--- ---
name: last30days name: last30days
version: "2.1" version: "2.1"
description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, web. Become an expert and write copy-paste-ready prompts." description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, Hacker News, web. Become an expert and write copy-paste-ready prompts."
argument-hint: 'last30 AI video tools, last30 best project management tools' argument-hint: 'last30 AI video tools, last30 best project management tools'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
homepage: https://github.com/mvanhorn/last30days-skill homepage: https://github.com/mvanhorn/last30days-skill
@@ -25,13 +25,14 @@ metadata:
- reddit - reddit
- x - x
- youtube - youtube
- hackernews
- trends - trends
- prompts - prompts
--- ---
# last30days v2.1: Research Any Topic from the Last 30 Days # last30days v2.1: Research Any Topic from the Last 30 Days
Research ANY topic across Reddit, X, YouTube, and the web. Surface what people are actually discussing, recommending, and debating right now. Research ANY topic across Reddit, X, YouTube, Hacker News, and the web. Surface what people are actually discussing, recommending, and debating right now.
## CRITICAL: Parse User Intent ## CRITICAL: Parse User Intent
@@ -109,10 +110,10 @@ Use a **timeout of 300000** (5 minutes) on the Bash call. The script typically t
The script will automatically: The script will automatically:
- Detect available API keys - Detect available API keys
- Run Reddit/X/YouTube searches - Run Reddit/X/YouTube/Hacker News searches
- Output ALL results including YouTube transcripts - Output ALL results including YouTube transcripts and HN comments
**Read the ENTIRE output.** It contains THREE data sections in this order: Reddit items, X items, and YouTube items. If you miss the YouTube section, you will produce incomplete stats. **Read the ENTIRE output.** It contains FOUR data sections in this order: Reddit items, X items, Hacker News items, and YouTube items. If you miss sections, you will produce incomplete stats.
**YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, and optional transcript snippet. Count them and include them in your synthesis and stats block. **YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, and optional transcript snippet. Count them and include them in your synthesis and stats block.
@@ -253,7 +254,8 @@ CITATION PRIORITY (most to least preferred):
1. @handles from X — "per @handle" (these prove the tool's unique value) 1. @handles from X — "per @handle" (these prove the tool's unique value)
2. r/subreddits from Reddit — "per r/subreddit" 2. r/subreddits from Reddit — "per r/subreddit"
3. YouTube channels — "per [channel name] on YouTube" (transcript-backed insights) 3. YouTube channels — "per [channel name] on YouTube" (transcript-backed insights)
4. Web sources — ONLY when Reddit/X/YouTube don't cover that specific fact 4. HN discussions — "per HN" or "per hn/username" (developer community signal)
5. Web sources — ONLY when Reddit/X/YouTube/HN don't cover that specific fact
The tool's value is surfacing what PEOPLE are saying, not what journalists wrote. The tool's value is surfacing what PEOPLE are saying, not what journalists wrote.
When both a web article and an X post cover the same fact, cite the X post. When both a web article and an X post cover the same fact, cite the X post.
@@ -302,6 +304,7 @@ KEY PATTERNS from the research:
✅ All agents reported back! ✅ All agents reported back!
├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments ├─ 🟠 Reddit: {N} threads │ {N} upvotes │ {N} comments
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts ├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
├─ 🟡 HN: {N} stories │ {N} points │ {N} comments
├─ 🔴 YouTube: {N} videos │ {N} views │ {N} with transcripts ├─ 🔴 YouTube: {N} videos │ {N} views │ {N} with transcripts
├─ 🌐 Web: {N} pages (supplementary) ├─ 🌐 Web: {N} pages (supplementary)
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2} └─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
@@ -309,6 +312,7 @@ KEY PATTERNS from the research:
``` ```
If Reddit returned 0 threads, write: "├─ 🟠 Reddit: 0 threads (no results this cycle)" If Reddit returned 0 threads, write: "├─ 🟠 Reddit: 0 threads (no results this cycle)"
If HN returned 0 stories, write: "├─ 🟡 HN: 0 stories (no results this cycle)"
If YouTube returned 0 videos or yt-dlp is not installed, omit the YouTube line entirely. If YouTube returned 0 videos or yt-dlp is not installed, omit the YouTube line entirely.
NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji. NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
@@ -471,7 +475,7 @@ After delivering a prompt, end with:
``` ```
--- ---
📚 Expert in: {TOPIC} for {TARGET_TOOL} 📚 Expert in: {TOPIC} for {TARGET_TOOL}
📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} YouTube videos ({sum} views) + {n} web pages 📊 Based on: {n} Reddit threads ({sum} upvotes) + {n} X posts ({sum} likes) + {n} HN stories ({sum} points) + {n} YouTube videos ({sum} views) + {n} web pages
Want another prompt? Just tell me what you're creating next. Want another prompt? Just tell me what you're creating next.
``` ```
@@ -483,6 +487,7 @@ Want another prompt? Just tell me what you're creating next.
**What this skill does:** **What this skill does:**
- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery - Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery
- Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search - Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
- Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data) - Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search - Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
- Fetches public Reddit thread data from `reddit.com` for engagement metrics - Fetches public Reddit thread data from `reddit.com` for engagement metrics
@@ -494,6 +499,7 @@ Want another prompt? Just tell me what you're creating next.
- Does not share API keys between providers (OpenAI key only goes to api.openai.com, etc.) - Does not share API keys between providers (OpenAI key only goes to api.openai.com, etc.)
- Does not log, cache, or write API keys to output files - Does not log, cache, or write API keys to output files
- Does not send data to any endpoint not listed above - Does not send data to any endpoint not listed above
- Hacker News source is always available (no API key, no binary dependency)
- Cannot be invoked autonomously by the agent (`disable-model-invocation: true`) - Cannot be invoked autonomously by the agent (`disable-model-invocation: true`)
**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed) **Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
docs/plans/2026-02-24-feat-hacker-news-source-plan.md
+235
View File
@@ -0,0 +1,235 @@
---
title: "feat: Add Hacker News as a 5th research source"
type: feat
status: completed
date: 2026-02-24
---
# feat: Add Hacker News as a 5th Research Source
## Overview
Add Hacker News as a source to the last30days skill, using the free Algolia HN Search API (`hn.algolia.com/api/v1`). HN provides high-signal content from a technical audience — stories with high point counts and active comment threads are strong indicators of what the developer community actually cares about. No API key required.
## Problem Statement / Motivation
The skill currently covers Reddit, X, YouTube, and web. Hacker News is missing — and for technical topics, HN often surfaces discussions that don't appear on Reddit or X. HN's upvote system and comment culture produce high-quality signal: a 500-point story with 200 comments means the developer community is genuinely engaged. Community contributor @wkbaran proposed this in PR #26 alongside YouTube and Product Hunt. YouTube shipped in v2.1; now it's time for HN.
## Proposed Solution
Add `scripts/lib/hackernews.py` following the exact same pattern as `youtube_yt.py` (the simplest existing source — no API key, just HTTP calls). Use the Algolia HN Search API for discovery, then optionally fetch top comments from high-scoring stories for enrichment (like Reddit enrichment, but using the `/items/:id` endpoint instead of Reddit's JSON API).
### Two-phase approach (matches existing Reddit pattern):
1. **Phase 1 — Search**: Query Algolia for stories matching the topic within the date range. Get titles, URLs, points, comment counts.
2. **Phase 2 — Enrichment** (optional, top N stories): Fetch the `/items/:id` endpoint for the highest-scoring stories to get top-level comments. This gives "comment_insights" like Reddit enrichment does.
## Technical Approach
### Files to Create
#### `scripts/lib/hackernews.py`
The main source module. Pattern matches `youtube_yt.py` (simplest source).
```python
# scripts/lib/hackernews.py
"""Hacker News search via Algolia API (free, no auth required)."""
ALGOLIA_SEARCH_URL = "https://hn.algolia.com/api/v1/search"
ALGOLIA_SEARCH_BY_DATE_URL = "https://hn.algolia.com/api/v1/search_by_date"
ALGOLIA_ITEM_URL = "https://hn.algolia.com/api/v1/items"
DEPTH_CONFIG = {
"quick": 15,
"default": 30,
"deep": 60,
}
ENRICH_LIMITS = {
"quick": 3,
"default": 5,
"deep": 10,
}
```
Key functions:
- `search_hackernews(topic, from_date, to_date, depth="default") -> Dict[str, Any]`
- Calls `hn.algolia.com/api/v1/search?query={topic}&tags=story&numericFilters=created_at_i>{from_ts},created_at_i<{to_ts}&hitsPerPage={count}`
- Uses `http.get()` — stdlib only, matches existing pattern
- Returns raw Algolia response
- `parse_hackernews_response(response: Dict) -> List[Dict]`
- Extracts hits, maps to raw dicts with fields: `id` (prefix "HN"), `title`, `url`, `hn_url`, `author`, `date`, `engagement` (points, num_comments), `why_relevant`, `relevance`
- `relevance` estimated from Algolia rank + engagement boost (same pattern as Bill's PR)
- `enrich_top_stories(items, depth="default") -> List[Dict]`
- Fetches `/items/{objectID}` for top N stories (by points)
- Extracts top-level comments (author, text, points)
- Adds `top_comments` and `comment_insights` fields (same structure as Reddit enrichment)
- Uses `ThreadPoolExecutor` for parallel fetching
- `_date_to_unix(date_str: str) -> int` — Helper, converts YYYY-MM-DD to Unix timestamp
#### `tests/test_hackernews.py`
Standard unittest pattern matching existing tests.
- Test `parse_hackernews_response` with sample Algolia response
- Test `_date_to_unix` conversion
- Test empty response handling
- Test enrichment parsing
- Test score integration with `score.py`
### Files to Modify
#### `scripts/lib/schema.py`
- [x] Add `HackerNewsItem` dataclass:
```python
@dataclass
class HackerNewsItem:
id: str # "HN1", "HN2", ...
title: str
url: str # Original article URL
hn_url: str # news.ycombinator.com/item?id=...
author: str # HN username
date: Optional[str]
date_confidence: str # Always "high" (Algolia provides exact timestamps)
engagement: Optional[Engagement] # points + num_comments
top_comments: List[Comment] # From enrichment
comment_insights: List[str] # From enrichment
relevance: float
why_relevant: str
subs: SubScores
score: int
```
- [x] Add `hackernews: List[HackerNewsItem] = field(default_factory=list)` to `Report`
- [x] Add `hackernews_error: Optional[str] = None` to `Report`
- [x] Update `Report.to_dict()` and `Report.from_dict()`
#### `scripts/lib/normalize.py`
- [x] Add `normalize_hackernews_items(items: List[Dict], from_date, to_date) -> List[schema.HackerNewsItem]`
- Maps raw dicts to `HackerNewsItem` dataclass instances
- Sets `date_confidence = "high"` (Algolia provides `created_at_i` exact timestamps)
- Converts `engagement` dict to `schema.Engagement(score=points, num_comments=num_comments)`
#### `scripts/lib/score.py`
- [x] Add `compute_hackernews_engagement_raw(engagement) -> float`
- Formula: `0.55 * log1p(points) + 0.45 * log1p(num_comments)`
- Points are the primary signal on HN; comments indicate depth of discussion
- [x] Add `score_hackernews_items(items) -> List[schema.HackerNewsItem]`
- Uses standard 45/25/30 weights (relevance/recency/engagement) — same as Reddit/X/YouTube
- [x] Update `sort_items()` to handle `HackerNewsItem`
- Source priority: Reddit > X > **HN** > YouTube > WebSearch
- HN slots between X and YouTube: higher signal than YouTube (curated upvotes vs raw views), but X has real-time pulse
#### `scripts/lib/dedupe.py`
- [x] Add `dedupe_hackernews(items, threshold=0.7) -> List[schema.HackerNewsItem]`
- [x] Update `get_item_text()` to handle `HackerNewsItem` (return `title`)
- [x] Consider cross-source dedup: HN stories often link to the same URLs that appear in web search results. Dedupe by URL match across `hackernews` and `websearch` items.
#### `scripts/lib/render.py`
- [x] Add HN section to `render_compact()`:
```
### Hacker News Stories
**HN1** (score:85) hn/username (2026-02-15) [350pts, 127cmt]
Story title here
https://news.ycombinator.com/item?id=12345
*Why relevant*
```
- [x] Add to `render_source_status()`:
```
✅ HN: {N} stories
```
- [x] Add to `render_full_report()` and `render_context_snippet()`
- [x] Update `_assess_data_freshness()` to include HN items
#### `scripts/last30days.py`
- [x] Import: `from lib import hackernews`
- [x] Add `_search_hackernews(topic, from_date, to_date, depth) -> (items, error)` wrapper function
- Calls `hackernews.search_hackernews()`, then `hackernews.parse_hackernews_response()`
- Returns `(items, None)` or `([], error_string)`
- [x] Add to `TIMEOUT_PROFILES`: `"hackernews_future": 60` (default), `30` (quick), `90` (deep)
- [x] Add HN to `ThreadPoolExecutor` in `run_research()`:
```python
if do_hackernews:
hn_future = executor.submit(_search_hackernews, topic, from_date, to_date, depth)
```
- Increment `max_workers` by 1 when HN is enabled
- [x] Collect results: `hn_items, hn_error = hn_future.result(timeout=hn_timeout)`
- [x] Add HN enrichment phase (after Reddit enrichment, before Phase 2):
```python
if hn_items:
hn_items = hackernews.enrich_top_stories(hn_items, depth=depth)
```
- [x] Add to processing pipeline: normalize -> filter_by_date_range -> score -> sort -> dedupe
- [x] Assign to `report.hackernews` and `report.hackernews_error`
- [x] Update status UI: add `⏳ 🟡 HN Searching Hacker News...` and `✓ 🟡 HN Found {N} stories`
#### `scripts/lib/env.py`
- [x] HN is always available (no API key, no binary dependency)
- [x] Update `get_available_sources()` to include HN in the source list
- [x] Update `validate_sources()` to accept `hn` as a valid source name
- [x] Add `hn` to the `--search` flag documentation
#### `SKILL.md`
- [x] Update description: "Sources: Reddit, X, YouTube, **Hacker News**, and web"
- [x] Add HN to stats block template:
```
├─ 🟡 HN: {N} stories │ {N} points │ {N} comments
```
- [x] Update `metadata.clawdbot.tags` to include `hackernews`
- [x] Update Security section: add `hn.algolia.com` to endpoints list
- [x] Update citation priority to include HN: Reddit > X > YouTube > HN > Web
## Acceptance Criteria
- [x] `python3 scripts/last30days.py "AI coding agents" --emit=compact` shows HN section with stories, points, and comment counts
- [x] HN stories include `hn_url` linking to the HN discussion page (not just the article URL)
- [x] Top stories are enriched with top comments (like Reddit enrichment)
- [x] HN runs in parallel with Reddit/X/YouTube (no serial bottleneck)
- [x] Scoring uses standard 45/25/30 weights with engagement formula tuned for HN metrics
- [x] Stats block shows: `├─ 🟡 HN: {N} stories │ {N} points │ {N} comments`
- [x] `--search=hn` works to run HN only; `--search=reddit,hn` works for combos
- [x] `--quick`, `--deep` flags adjust HN result count (15/30/60)
- [x] All existing tests still pass
- [x] New `tests/test_hackernews.py` with tests for parse, normalize, score, enrichment
- [x] No API key required — works out of the box
- [x] Cross-source URL dedup: HN stories linking to same URL as web results get deduped
- [x] SKILL.md updated with HN in stats block, security section, and citation priority
## Dependencies & Risks
**Low risk:**
- Algolia HN API is free, public, no auth, well-established (used since 2014)
- No new dependencies — uses existing `http.py` (stdlib urllib)
- Pattern is identical to YouTube source (simplest existing source)
**Medium risk:**
- Algolia has no officially documented rate limit, but aggressive use could get throttled
- Mitigation: Existing `http.py` exponential backoff handles 429s
- Default depth only requests 30 items (1 API call for search + N for enrichment)
- Comment enrichment adds N API calls (one per story) which could slow down quick mode
- Mitigation: Limit enrichment to top 3/5/10 stories by depth; use ThreadPoolExecutor
**Compatibility:**
- HN always available — doesn't break anything when other sources are missing
- Existing `--search` flag needs extension but is backward-compatible
## Sources & References
- PR #26 by @wkbaran: [HN implementation reference](https://github.com/mvanhorn/last30days-skill/pull/26)
- Algolia HN API docs: `https://hn.algolia.com/api`
- Existing patterns: `scripts/lib/youtube_yt.py` (simplest source), `scripts/lib/openai_reddit.py` (enrichment pattern)
- Scoring reference: `scripts/lib/score.py:compute_reddit_engagement_raw()`
- Schema reference: `scripts/lib/schema.py:RedditItem` (closest analog to HN)
scripts/last30days.py
+93 -9
View File
@@ -38,9 +38,9 @@ _child_pids: set = set()
_child_pids_lock = threading.Lock() _child_pids_lock = threading.Lock()
TIMEOUT_PROFILES = { TIMEOUT_PROFILES = {
"quick": {"global": 90, "future": 30, "reddit_future": 60, "youtube_future": 60, "http": 15, "enrich_per": 8, "enrich_total": 30, "enrich_max_items": 10}, "quick": {"global": 90, "future": 30, "reddit_future": 60, "youtube_future": 60, "hackernews_future": 30, "http": 15, "enrich_per": 8, "enrich_total": 30, "enrich_max_items": 10},
"default": {"global": 180, "future": 60, "reddit_future": 90, "youtube_future": 90, "http": 30, "enrich_per": 15, "enrich_total": 45, "enrich_max_items": 15}, "default": {"global": 180, "future": 60, "reddit_future": 90, "youtube_future": 90, "hackernews_future": 60, "http": 30, "enrich_per": 15, "enrich_total": 45, "enrich_max_items": 15},
"deep": {"global": 300, "future": 90, "reddit_future": 120, "youtube_future": 120, "http": 30, "enrich_per": 15, "enrich_total": 60, "enrich_max_items": 25}, "deep": {"global": 300, "future": 90, "reddit_future": 120, "youtube_future": 120, "hackernews_future": 90, "http": 30, "enrich_per": 15, "enrich_total": 60, "enrich_max_items": 25},
} }
@@ -98,6 +98,7 @@ from lib import (
bird_x, bird_x,
dates, dates,
dedupe, dedupe,
hackernews,
entity_extract, entity_extract,
env, env,
http, http,
@@ -303,6 +304,34 @@ def _search_youtube(
return youtube_items, youtube_error return youtube_items, youtube_error
def _search_hackernews(
topic: str,
from_date: str,
to_date: str,
depth: str,
) -> tuple:
"""Search Hacker News via Algolia (runs in thread).
Returns:
Tuple of (hn_items, hn_error)
"""
hn_error = None
try:
response = hackernews.search_hackernews(
topic, from_date, to_date, depth=depth,
)
except Exception as e:
return [], f"{type(e).__name__}: {e}"
hn_items = hackernews.parse_hackernews_response(response)
if response.get("error"):
hn_error = response["error"]
return hn_items, hn_error
def _search_web( def _search_web(
topic: str, topic: str,
config: dict, config: dict,
@@ -516,6 +545,7 @@ def run_research(
reddit_items = [] reddit_items = []
x_items = [] x_items = []
youtube_items = [] youtube_items = []
hackernews_items = []
web_items = [] web_items = []
raw_openai = None raw_openai = None
raw_xai = None raw_xai = None
@@ -523,6 +553,7 @@ def run_research(
reddit_error = None reddit_error = None
x_error = None x_error = None
youtube_error = None youtube_error = None
hackernews_error = None
web_error = None web_error = None
# Determine web search mode # Determine web search mode
@@ -565,18 +596,20 @@ def run_research(
progress.show_error(f"YouTube error: {e}") progress.show_error(f"YouTube error: {e}")
if progress: if progress:
progress.end_youtube(len(youtube_items)) progress.end_youtube(len(youtube_items))
return reddit_items, x_items, youtube_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, web_error return reddit_items, x_items, youtube_items, hackernews_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, web_error
# Determine which searches to run # Determine which searches to run
do_reddit = sources in ("both", "reddit", "all", "reddit-web") do_reddit = sources in ("both", "reddit", "all", "reddit-web")
do_x = sources in ("both", "x", "all", "x-web") do_x = sources in ("both", "x", "all", "x-web")
do_hackernews = True # HN is always available (no API key)
# Run Reddit, X, YouTube, and Web searches in parallel # Run Reddit, X, YouTube, HN, and Web searches in parallel
reddit_future = None reddit_future = None
x_future = None x_future = None
youtube_future = None youtube_future = None
hackernews_future = None
web_future = None web_future = None
max_workers = 2 + (1 if run_youtube else 0) + (1 if web_backend else 0) max_workers = 2 + (1 if run_youtube else 0) + (1 if do_hackernews else 0) + (1 if web_backend else 0)
with ThreadPoolExecutor(max_workers=max_workers) as executor: with ThreadPoolExecutor(max_workers=max_workers) as executor:
# Submit searches # Submit searches
@@ -603,6 +636,13 @@ def run_research(
_search_youtube, topic, from_date, to_date, depth _search_youtube, topic, from_date, to_date, depth
) )
if do_hackernews:
if progress:
progress.start_hackernews()
hackernews_future = executor.submit(
_search_hackernews, topic, from_date, to_date, depth
)
if web_backend: if web_backend:
sys.stderr.write(f"[web] Searching via {web_backend}\n") sys.stderr.write(f"[web] Searching via {web_backend}\n")
sys.stderr.flush() sys.stderr.flush()
@@ -661,6 +701,23 @@ def run_research(
if progress: if progress:
progress.end_youtube(len(youtube_items)) progress.end_youtube(len(youtube_items))
if hackernews_future:
hn_timeout = timeouts.get("hackernews_future", future_timeout)
try:
hackernews_items, hackernews_error = hackernews_future.result(timeout=hn_timeout)
if hackernews_error and progress:
progress.show_error(f"HN error: {hackernews_error}")
except TimeoutError:
hackernews_error = f"HN search timed out after {hn_timeout}s"
if progress:
progress.show_error(hackernews_error)
except Exception as e:
hackernews_error = f"{type(e).__name__}: {e}"
if progress:
progress.show_error(f"HN error: {e}")
if progress:
progress.end_hackernews(len(hackernews_items))
if web_future: if web_future:
try: try:
web_items, web_error = web_future.result(timeout=future_timeout) web_items, web_error = web_future.result(timeout=future_timeout)
@@ -747,6 +804,14 @@ def run_research(
if progress: if progress:
progress.end_reddit_enrich() progress.end_reddit_enrich()
# Enrich HN stories with comments
if hackernews_items:
try:
hackernews_items = hackernews.enrich_top_stories(hackernews_items, depth=depth)
except Exception as e:
sys.stderr.write(f"[HN] Enrichment error: {e}\n")
sys.stderr.flush()
# Phase 2: Supplemental search based on entities from Phase 1 # Phase 2: Supplemental search based on entities from Phase 1
# Skip on --quick (speed matters), mock mode, or if Reddit is rate-limiting # Skip on --quick (speed matters), mock mode, or if Reddit is rate-limiting
if depth != "quick" and not mock and (reddit_items or x_items): if depth != "quick" and not mock and (reddit_items or x_items):
@@ -760,7 +825,7 @@ def run_research(
if sup_x: if sup_x:
x_items.extend(sup_x) x_items.extend(sup_x)
return reddit_items, x_items, youtube_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, web_error return reddit_items, x_items, youtube_items, hackernews_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, web_error
def main(): def main():
@@ -878,6 +943,7 @@ def main():
"bird_authenticated": x_source_status["bird_authenticated"], "bird_authenticated": x_source_status["bird_authenticated"],
"bird_username": x_source_status.get("bird_username"), "bird_username": x_source_status.get("bird_username"),
"youtube": has_ytdlp, "youtube": has_ytdlp,
"hackernews": True,
"web_search_backend": web_source, "web_search_backend": web_source,
"parallel_ai": bool(config.get("PARALLEL_API_KEY")), "parallel_ai": bool(config.get("PARALLEL_API_KEY")),
"brave": bool(config.get("BRAVE_API_KEY")), "brave": bool(config.get("BRAVE_API_KEY")),
@@ -905,6 +971,7 @@ def main():
"bird_authenticated": x_source_status["bird_authenticated"], "bird_authenticated": x_source_status["bird_authenticated"],
"bird_username": x_source_status.get("bird_username"), "bird_username": x_source_status.get("bird_username"),
"youtube": has_ytdlp, "youtube": has_ytdlp,
"hackernews": True,
"web_search_backend": web_source, "web_search_backend": web_source,
} }
ui.show_diagnostic_banner(diag) ui.show_diagnostic_banner(diag)
@@ -982,7 +1049,7 @@ def main():
mode = sources mode = sources
# Run research # Run research
reddit_items, x_items, youtube_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, web_error = run_research( reddit_items, x_items, youtube_items, hackernews_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, web_error = run_research(
args.topic, args.topic,
sources, sources,
config, config,
@@ -1004,6 +1071,7 @@ def main():
normalized_reddit = normalize.normalize_reddit_items(reddit_items, from_date, to_date) normalized_reddit = normalize.normalize_reddit_items(reddit_items, from_date, to_date)
normalized_x = normalize.normalize_x_items(x_items, from_date, to_date) normalized_x = normalize.normalize_x_items(x_items, from_date, to_date)
normalized_youtube = normalize.normalize_youtube_items(youtube_items, from_date, to_date) if youtube_items else [] normalized_youtube = normalize.normalize_youtube_items(youtube_items, from_date, to_date) if youtube_items else []
normalized_hn = normalize.normalize_hackernews_items(hackernews_items, from_date, to_date) if hackernews_items else []
normalized_web = websearch.normalize_websearch_items(web_items, from_date, to_date) if web_items else [] normalized_web = websearch.normalize_websearch_items(web_items, from_date, to_date) if web_items else []
# Hard date filter: exclude items with verified dates outside the range # Hard date filter: exclude items with verified dates outside the range
@@ -1014,24 +1082,28 @@ def main():
# that prefers recent videos but keeps older ones for evergreen topics. # that prefers recent videos but keeps older ones for evergreen topics.
# YouTube content has a longer shelf life than tweets/posts. # YouTube content has a longer shelf life than tweets/posts.
filtered_youtube = normalized_youtube filtered_youtube = normalized_youtube
filtered_hn = normalize.filter_by_date_range(normalized_hn, from_date, to_date) if normalized_hn else []
filtered_web = normalize.filter_by_date_range(normalized_web, from_date, to_date) if normalized_web else [] filtered_web = normalize.filter_by_date_range(normalized_web, from_date, to_date) if normalized_web else []
# Score items # Score items
scored_reddit = score.score_reddit_items(filtered_reddit) scored_reddit = score.score_reddit_items(filtered_reddit)
scored_x = score.score_x_items(filtered_x) scored_x = score.score_x_items(filtered_x)
scored_youtube = score.score_youtube_items(filtered_youtube) if filtered_youtube else [] scored_youtube = score.score_youtube_items(filtered_youtube) if filtered_youtube else []
scored_hn = score.score_hackernews_items(filtered_hn) if filtered_hn else []
scored_web = score.score_websearch_items(filtered_web) if filtered_web else [] scored_web = score.score_websearch_items(filtered_web) if filtered_web else []
# Sort items # Sort items
sorted_reddit = score.sort_items(scored_reddit) sorted_reddit = score.sort_items(scored_reddit)
sorted_x = score.sort_items(scored_x) sorted_x = score.sort_items(scored_x)
sorted_youtube = score.sort_items(scored_youtube) if scored_youtube else [] sorted_youtube = score.sort_items(scored_youtube) if scored_youtube else []
sorted_hn = score.sort_items(scored_hn) if scored_hn else []
sorted_web = score.sort_items(scored_web) if scored_web else [] sorted_web = score.sort_items(scored_web) if scored_web else []
# Dedupe items # Dedupe items
deduped_reddit = dedupe.dedupe_reddit(sorted_reddit) deduped_reddit = dedupe.dedupe_reddit(sorted_reddit)
deduped_x = dedupe.dedupe_x(sorted_x) deduped_x = dedupe.dedupe_x(sorted_x)
deduped_youtube = dedupe.dedupe_youtube(sorted_youtube) if sorted_youtube else [] deduped_youtube = dedupe.dedupe_youtube(sorted_youtube) if sorted_youtube else []
deduped_hn = dedupe.dedupe_hackernews(sorted_hn) if sorted_hn else []
deduped_web = websearch.dedupe_websearch(sorted_web) if sorted_web else [] deduped_web = websearch.dedupe_websearch(sorted_web) if sorted_web else []
# Minimum result guarantee: if all Reddit results were filtered out but # Minimum result guarantee: if all Reddit results were filtered out but
@@ -1055,10 +1127,12 @@ def main():
report.reddit = deduped_reddit report.reddit = deduped_reddit
report.x = deduped_x report.x = deduped_x
report.youtube = deduped_youtube report.youtube = deduped_youtube
report.hackernews = deduped_hn
report.web = deduped_web report.web = deduped_web
report.reddit_error = reddit_error report.reddit_error = reddit_error
report.x_error = x_error report.x_error = x_error
report.youtube_error = youtube_error report.youtube_error = youtube_error
report.hackernews_error = hackernews_error
report.web_error = web_error report.web_error = web_error
# Generate context snippet # Generate context snippet
@@ -1071,7 +1145,7 @@ def main():
if sources == "web": if sources == "web":
progress.show_web_only_complete() progress.show_web_only_complete()
else: else:
progress.show_complete(len(deduped_reddit), len(deduped_x), len(deduped_youtube)) progress.show_complete(len(deduped_reddit), len(deduped_x), len(deduped_youtube), len(deduped_hn))
# Build source info for status footer # Build source info for status footer
source_info = {} source_info = {}
@@ -1129,6 +1203,16 @@ def main():
"engagement_score": item.engagement.views if item.engagement and item.engagement.views else 0, "engagement_score": item.engagement.views if item.engagement and item.engagement.views else 0,
"relevance_score": item.relevance, "relevance_score": item.relevance,
}) })
for item in deduped_hn:
findings.append({
"source": "hackernews",
"url": item.hn_url,
"title": item.title,
"author": item.author,
"content": item.title,
"engagement_score": item.engagement.score if item.engagement else 0,
"relevance_score": item.relevance,
})
for item in deduped_web: for item in deduped_web:
findings.append({ findings.append({
"source": "web", "source": "web",
scripts/lib/dedupe.py
+11 -1
View File
@@ -36,10 +36,12 @@ def jaccard_similarity(set1: Set[str], set2: Set[str]) -> float:
return intersection / union if union > 0 else 0.0 return intersection / union if union > 0 else 0.0
def get_item_text(item: Union[schema.RedditItem, schema.XItem, schema.YouTubeItem]) -> str: def get_item_text(item: Union[schema.RedditItem, schema.XItem, schema.YouTubeItem, schema.HackerNewsItem]) -> str:
"""Get comparable text from an item.""" """Get comparable text from an item."""
if isinstance(item, schema.RedditItem): if isinstance(item, schema.RedditItem):
return item.title return item.title
elif isinstance(item, schema.HackerNewsItem):
return item.title
elif isinstance(item, schema.YouTubeItem): elif isinstance(item, schema.YouTubeItem):
return f"{item.title} {item.channel_name}" return f"{item.title} {item.channel_name}"
else: else:
@@ -128,3 +130,11 @@ def dedupe_youtube(
) -> List[schema.YouTubeItem]: ) -> List[schema.YouTubeItem]:
"""Dedupe YouTube items.""" """Dedupe YouTube items."""
return dedupe_items(items, threshold) return dedupe_items(items, threshold)
def dedupe_hackernews(
items: List[schema.HackerNewsItem],
threshold: float = 0.7,
) -> List[schema.HackerNewsItem]:
"""Dedupe Hacker News items."""
return dedupe_items(items, threshold)
scripts/lib/env.py
+8
View File
@@ -246,6 +246,14 @@ def is_ytdlp_available() -> bool:
return youtube_yt.is_ytdlp_installed() return youtube_yt.is_ytdlp_installed()
def is_hackernews_available() -> bool:
"""Check if Hacker News source is available.
Always returns True - HN uses free Algolia API, no key needed.
"""
return True
def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]: def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]:
"""Get detailed X source status for UI decisions. """Get detailed X source status for UI decisions.
scripts/lib/hackernews.py
+252
View File
@@ -0,0 +1,252 @@
"""Hacker News search via Algolia API (free, no auth required).
Uses hn.algolia.com/api/v1 for story discovery and comment enrichment.
No API key needed - just HTTP calls via stdlib urllib.
"""
import html
import math
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Any, Dict, List, Optional
from . import http
ALGOLIA_SEARCH_URL = "https://hn.algolia.com/api/v1/search"
ALGOLIA_SEARCH_BY_DATE_URL = "https://hn.algolia.com/api/v1/search_by_date"
ALGOLIA_ITEM_URL = "https://hn.algolia.com/api/v1/items"
DEPTH_CONFIG = {
"quick": 15,
"default": 30,
"deep": 60,
}
ENRICH_LIMITS = {
"quick": 3,
"default": 5,
"deep": 10,
}
def _log(msg: str):
"""Log to stderr."""
sys.stderr.write(f"[HN] {msg}\n")
sys.stderr.flush()
def _date_to_unix(date_str: str) -> int:
"""Convert YYYY-MM-DD to Unix timestamp (start of day UTC)."""
parts = date_str.split("-")
year, month, day = int(parts[0]), int(parts[1]), int(parts[2])
import calendar
import datetime
dt = datetime.datetime(year, month, day, tzinfo=datetime.timezone.utc)
return int(dt.timestamp())
def _unix_to_date(ts: int) -> str:
"""Convert Unix timestamp to YYYY-MM-DD."""
import datetime
dt = datetime.datetime.fromtimestamp(ts, tz=datetime.timezone.utc)
return dt.strftime("%Y-%m-%d")
def _strip_html(text: str) -> str:
"""Strip HTML tags and decode entities from HN comment text."""
import re
text = html.unescape(text)
text = re.sub(r'<p>', '\n', text)
text = re.sub(r'<[^>]+>', '', text)
return text.strip()
def search_hackernews(
topic: str,
from_date: str,
to_date: str,
depth: str = "default",
) -> Dict[str, Any]:
"""Search Hacker News via Algolia API.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: 'quick', 'default', or 'deep'
Returns:
Dict with Algolia response (contains 'hits' list).
"""
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
from_ts = _date_to_unix(from_date)
to_ts = _date_to_unix(to_date) + 86400 # Include the end date
_log(f"Searching for '{topic}' (since {from_date}, count={count})")
# Use relevance-sorted search (better for topic matching)
params = {
"query": topic,
"tags": "story",
"numericFilters": f"created_at_i>{from_ts},created_at_i<{to_ts}",
"hitsPerPage": str(count),
}
from urllib.parse import urlencode
url = f"{ALGOLIA_SEARCH_URL}?{urlencode(params)}"
try:
response = http.request("GET", url, timeout=30)
except http.HTTPError as e:
_log(f"Search failed: {e}")
return {"hits": [], "error": str(e)}
except Exception as e:
_log(f"Search failed: {e}")
return {"hits": [], "error": str(e)}
hits = response.get("hits", [])
_log(f"Found {len(hits)} stories")
return response
def parse_hackernews_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Parse Algolia response into normalized item dicts.
Returns:
List of item dicts ready for normalization.
"""
hits = response.get("hits", [])
items = []
for i, hit in enumerate(hits):
object_id = hit.get("objectID", "")
points = hit.get("points") or 0
num_comments = hit.get("num_comments") or 0
created_at_i = hit.get("created_at_i")
date_str = None
if created_at_i:
date_str = _unix_to_date(created_at_i)
# Article URL vs HN discussion URL
article_url = hit.get("url") or ""
hn_url = f"https://news.ycombinator.com/item?id={object_id}"
# Relevance: Algolia rank position gives a base, engagement boosts it
# Position 0 = most relevant from Algolia
rank_score = max(0.3, 1.0 - (i * 0.02)) # 1.0 -> 0.3 over 35 items
engagement_boost = min(0.2, math.log1p(points) / 40)
relevance = min(1.0, rank_score * 0.7 + engagement_boost + 0.1)
items.append({
"object_id": object_id,
"title": hit.get("title", ""),
"url": article_url,
"hn_url": hn_url,
"author": hit.get("author", ""),
"date": date_str,
"engagement": {
"points": points,
"num_comments": num_comments,
},
"relevance": round(relevance, 2),
"why_relevant": f"HN story about {hit.get('title', 'topic')[:60]}",
})
return items
def _fetch_item_comments(object_id: str, max_comments: int = 5) -> Dict[str, Any]:
"""Fetch top-level comments for a story from Algolia items endpoint.
Args:
object_id: HN story ID
max_comments: Max comments to return
Returns:
Dict with 'comments' list and 'comment_insights' list.
"""
url = f"{ALGOLIA_ITEM_URL}/{object_id}"
try:
data = http.request("GET", url, timeout=15)
except Exception as e:
_log(f"Failed to fetch comments for {object_id}: {e}")
return {"comments": [], "comment_insights": []}
children = data.get("children", [])
# Sort by points (highest first), filter to actual comments
real_comments = [
c for c in children
if c.get("text") and c.get("author")
]
real_comments.sort(key=lambda c: c.get("points") or 0, reverse=True)
comments = []
insights = []
for c in real_comments[:max_comments]:
text = _strip_html(c.get("text", ""))
excerpt = text[:300] + "..." if len(text) > 300 else text
comments.append({
"author": c.get("author", ""),
"text": excerpt,
"points": c.get("points") or 0,
})
# First sentence as insight
first_sentence = text.split(". ")[0].split("\n")[0][:200]
if first_sentence:
insights.append(first_sentence)
return {"comments": comments, "comment_insights": insights}
def enrich_top_stories(
items: List[Dict[str, Any]],
depth: str = "default",
) -> List[Dict[str, Any]]:
"""Fetch comments for top N stories by points.
Args:
items: Parsed HN items
depth: Research depth (controls how many to enrich)
Returns:
Items with top_comments and comment_insights added.
"""
if not items:
return items
limit = ENRICH_LIMITS.get(depth, ENRICH_LIMITS["default"])
# Sort by points to enrich the most popular stories
by_points = sorted(
range(len(items)),
key=lambda i: items[i].get("engagement", {}).get("points", 0),
reverse=True,
)
to_enrich = by_points[:limit]
_log(f"Enriching top {len(to_enrich)} stories with comments")
with ThreadPoolExecutor(max_workers=5) as executor:
futures = {
executor.submit(
_fetch_item_comments,
items[idx]["object_id"],
): idx
for idx in to_enrich
}
for future in as_completed(futures):
idx = futures[future]
try:
result = future.result(timeout=15)
items[idx]["top_comments"] = result["comments"]
items[idx]["comment_insights"] = result["comment_insights"]
except Exception:
items[idx]["top_comments"] = []
items[idx]["comment_insights"] = []
return items
scripts/lib/normalize.py
+58 -1
View File
@@ -4,7 +4,7 @@ from typing import Any, Dict, List, TypeVar, Union
from . import dates, schema from . import dates, schema
T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem) T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.HackerNewsItem)
def filter_by_date_range( def filter_by_date_range(
@@ -200,6 +200,63 @@ def normalize_youtube_items(
return normalized return normalized
def normalize_hackernews_items(
items: List[Dict[str, Any]],
from_date: str,
to_date: str,
) -> List[schema.HackerNewsItem]:
"""Normalize raw Hacker News items to schema.
Args:
items: Raw HN items from Algolia API
from_date: Start of date range
to_date: End of date range
Returns:
List of HackerNewsItem objects
"""
normalized = []
for i, item in enumerate(items):
# Parse engagement
eng_raw = item.get("engagement") or {}
engagement = schema.Engagement(
score=eng_raw.get("points"),
num_comments=eng_raw.get("num_comments"),
)
# Parse comments (from enrichment)
top_comments = []
for c in item.get("top_comments", []):
top_comments.append(schema.Comment(
score=c.get("points", 0),
date=None,
author=c.get("author", ""),
excerpt=c.get("text", ""),
url="",
))
# HN dates are always high confidence (exact timestamps from Algolia)
date_str = item.get("date")
normalized.append(schema.HackerNewsItem(
id=f"HN{i+1}",
title=item.get("title", ""),
url=item.get("url", ""),
hn_url=item.get("hn_url", ""),
author=item.get("author", ""),
date=date_str,
date_confidence="high",
engagement=engagement,
top_comments=top_comments,
comment_insights=item.get("comment_insights", []),
relevance=item.get("relevance", 0.5),
why_relevant=item.get("why_relevant", ""),
))
return normalized
def items_to_dicts(items: List) -> List[Dict[str, Any]]: def items_to_dicts(items: List) -> List[Dict[str, Any]]:
"""Convert schema items to dicts for JSON serialization.""" """Convert schema items to dicts for JSON serialization."""
return [item.to_dict() for item in items] return [item.to_dict() for item in items]
scripts/lib/render.py
+76 -2
View File
@@ -30,9 +30,10 @@ def _assess_data_freshness(report: schema.Report) -> dict:
reddit_recent = sum(1 for r in report.reddit if r.date and r.date >= report.range_from) reddit_recent = sum(1 for r in report.reddit if r.date and r.date >= report.range_from)
x_recent = sum(1 for x in report.x if x.date and x.date >= report.range_from) x_recent = sum(1 for x in report.x if x.date and x.date >= report.range_from)
web_recent = sum(1 for w in report.web if w.date and w.date >= report.range_from) web_recent = sum(1 for w in report.web if w.date and w.date >= report.range_from)
hn_recent = sum(1 for h in report.hackernews if h.date and h.date >= report.range_from)
total_recent = reddit_recent + x_recent + web_recent total_recent = reddit_recent + x_recent + web_recent + hn_recent
total_items = len(report.reddit) + len(report.x) + len(report.web) total_items = len(report.reddit) + len(report.x) + len(report.web) + len(report.hackernews)
return { return {
"reddit_recent": reddit_recent, "reddit_recent": reddit_recent,
@@ -215,6 +216,42 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
lines.append(f" *{item.why_relevant}*") lines.append(f" *{item.why_relevant}*")
lines.append("") lines.append("")
# Hacker News items
if report.hackernews_error:
lines.append("### Hacker News Stories")
lines.append("")
lines.append(f"**ERROR:** {report.hackernews_error}")
lines.append("")
elif report.hackernews:
lines.append("### Hacker News Stories")
lines.append("")
for item in report.hackernews[:limit]:
eng_str = ""
if item.engagement:
eng = item.engagement
parts = []
if eng.score is not None:
parts.append(f"{eng.score}pts")
if eng.num_comments is not None:
parts.append(f"{eng.num_comments}cmt")
if parts:
eng_str = f" [{', '.join(parts)}]"
date_str = f" ({item.date})" if item.date else ""
lines.append(f"**{item.id}** (score:{item.score}) hn/{item.author}{date_str}{eng_str}")
lines.append(f" {item.title}")
lines.append(f" {item.hn_url}")
lines.append(f" *{item.why_relevant}*")
# Comment insights
if item.comment_insights:
lines.append(f" Insights:")
for insight in item.comment_insights[:3]:
lines.append(f" - {insight}")
lines.append("")
# Web items (if any - populated by the assistant) # Web items (if any - populated by the assistant)
if report.web_error: if report.web_error:
lines.append("### Web Results") lines.append("### Web Results")
@@ -278,6 +315,14 @@ def render_source_status(report: schema.Report, source_info: dict = None) -> str
reason = source_info.get("x_skip_reason", "No Bird CLI or XAI_API_KEY") reason = source_info.get("x_skip_reason", "No Bird CLI or XAI_API_KEY")
lines.append(f" ⏭️ X: skipped — {reason}") lines.append(f" ⏭️ X: skipped — {reason}")
# Hacker News
if report.hackernews_error:
lines.append(f" ❌ HN: error - {report.hackernews_error}")
elif report.hackernews:
lines.append(f" ✅ HN: {len(report.hackernews)} stories")
else:
lines.append(" ⏭️ HN: 0 stories found")
# YouTube # YouTube
if report.youtube_error: if report.youtube_error:
lines.append(f" ❌ YouTube: error — {report.youtube_error}") lines.append(f" ❌ YouTube: error — {report.youtube_error}")
@@ -325,6 +370,8 @@ def render_context_snippet(report: schema.Report) -> str:
all_items.append((item.score, "Reddit", item.title, item.url)) all_items.append((item.score, "Reddit", item.title, item.url))
for item in report.x[:5]: for item in report.x[:5]:
all_items.append((item.score, "X", item.text[:50] + "...", item.url)) all_items.append((item.score, "X", item.text[:50] + "...", item.url))
for item in report.hackernews[:5]:
all_items.append((item.score, "HN", item.title[:50] + "...", item.hn_url))
for item in report.web[:5]: for item in report.web[:5]:
all_items.append((item.score, "Web", item.title[:50] + "...", item.url)) all_items.append((item.score, "Web", item.title[:50] + "...", item.url))
@@ -414,6 +461,33 @@ def render_full_report(report: schema.Report) -> str:
lines.append(f"> {item.text}") lines.append(f"> {item.text}")
lines.append("") lines.append("")
# HN section
if report.hackernews:
lines.append("## Hacker News Stories")
lines.append("")
for item in report.hackernews:
lines.append(f"### {item.id}: {item.title}")
lines.append("")
lines.append(f"- **Author:** {item.author}")
lines.append(f"- **HN URL:** {item.hn_url}")
if item.url:
lines.append(f"- **Article URL:** {item.url}")
lines.append(f"- **Date:** {item.date or 'Unknown'}")
lines.append(f"- **Score:** {item.score}/100")
lines.append(f"- **Relevance:** {item.why_relevant}")
if item.engagement:
eng = item.engagement
lines.append(f"- **Engagement:** {eng.score or '?'} points, {eng.num_comments or '?'} comments")
if item.comment_insights:
lines.append("")
lines.append("**Key Insights from Comments:**")
for insight in item.comment_insights:
lines.append(f"- {insight}")
lines.append("")
# Web section # Web section
if report.web: if report.web:
lines.append("## Web Results") lines.append("## Web Results")
scripts/lib/schema.py
+69
View File
@@ -207,6 +207,43 @@ class YouTubeItem:
} }
@dataclass
class HackerNewsItem:
"""Normalized Hacker News item."""
id: str # "HN1", "HN2", ...
title: str
url: str # Original article URL
hn_url: str # news.ycombinator.com/item?id=...
author: str # HN username
date: Optional[str] = None
date_confidence: str = "high" # Algolia provides exact timestamps
engagement: Optional[Engagement] = None # points + num_comments
top_comments: List[Comment] = field(default_factory=list)
comment_insights: List[str] = field(default_factory=list)
relevance: float = 0.5
why_relevant: str = ""
subs: SubScores = field(default_factory=SubScores)
score: int = 0
def to_dict(self) -> Dict[str, Any]:
return {
'id': self.id,
'title': self.title,
'url': self.url,
'hn_url': self.hn_url,
'author': self.author,
'date': self.date,
'date_confidence': self.date_confidence,
'engagement': self.engagement.to_dict() if self.engagement else None,
'top_comments': [c.to_dict() for c in self.top_comments],
'comment_insights': self.comment_insights,
'relevance': self.relevance,
'why_relevant': self.why_relevant,
'subs': self.subs.to_dict(),
'score': self.score,
}
@dataclass @dataclass
class Report: class Report:
"""Full research report.""" """Full research report."""
@@ -221,6 +258,7 @@ class Report:
x: List[XItem] = field(default_factory=list) x: List[XItem] = field(default_factory=list)
web: List[WebSearchItem] = field(default_factory=list) web: List[WebSearchItem] = field(default_factory=list)
youtube: List[YouTubeItem] = field(default_factory=list) youtube: List[YouTubeItem] = field(default_factory=list)
hackernews: List[HackerNewsItem] = field(default_factory=list)
best_practices: List[str] = field(default_factory=list) best_practices: List[str] = field(default_factory=list)
prompt_pack: List[str] = field(default_factory=list) prompt_pack: List[str] = field(default_factory=list)
context_snippet_md: str = "" context_snippet_md: str = ""
@@ -229,6 +267,7 @@ class Report:
x_error: Optional[str] = None x_error: Optional[str] = None
web_error: Optional[str] = None web_error: Optional[str] = None
youtube_error: Optional[str] = None youtube_error: Optional[str] = None
hackernews_error: Optional[str] = None
# Cache info # Cache info
from_cache: bool = False from_cache: bool = False
cache_age_hours: Optional[float] = None cache_age_hours: Optional[float] = None
@@ -248,6 +287,7 @@ class Report:
'x': [x.to_dict() for x in self.x], 'x': [x.to_dict() for x in self.x],
'web': [w.to_dict() for w in self.web], 'web': [w.to_dict() for w in self.web],
'youtube': [y.to_dict() for y in self.youtube], 'youtube': [y.to_dict() for y in self.youtube],
'hackernews': [h.to_dict() for h in self.hackernews],
'best_practices': self.best_practices, 'best_practices': self.best_practices,
'prompt_pack': self.prompt_pack, 'prompt_pack': self.prompt_pack,
'context_snippet_md': self.context_snippet_md, 'context_snippet_md': self.context_snippet_md,
@@ -260,6 +300,8 @@ class Report:
d['web_error'] = self.web_error d['web_error'] = self.web_error
if self.youtube_error: if self.youtube_error:
d['youtube_error'] = self.youtube_error d['youtube_error'] = self.youtube_error
if self.hackernews_error:
d['hackernews_error'] = self.hackernews_error
if self.from_cache: if self.from_cache:
d['from_cache'] = self.from_cache d['from_cache'] = self.from_cache
if self.cache_age_hours is not None: if self.cache_age_hours is not None:
@@ -359,6 +401,31 @@ class Report:
score=y.get('score', 0), score=y.get('score', 0),
)) ))
# Reconstruct HackerNews items
hn_items = []
for h in data.get('hackernews', []):
eng = None
if h.get('engagement'):
eng = Engagement(**h['engagement'])
comments = [Comment(**c) for c in h.get('top_comments', [])]
subs = SubScores(**h.get('subs', {})) if h.get('subs') else SubScores()
hn_items.append(HackerNewsItem(
id=h['id'],
title=h['title'],
url=h.get('url', ''),
hn_url=h.get('hn_url', ''),
author=h.get('author', ''),
date=h.get('date'),
date_confidence=h.get('date_confidence', 'high'),
engagement=eng,
top_comments=comments,
comment_insights=h.get('comment_insights', []),
relevance=h.get('relevance', 0.5),
why_relevant=h.get('why_relevant', ''),
subs=subs,
score=h.get('score', 0),
))
return cls( return cls(
topic=data['topic'], topic=data['topic'],
range_from=range_from, range_from=range_from,
@@ -371,6 +438,7 @@ class Report:
x=x_items, x=x_items,
web=web_items, web=web_items,
youtube=youtube_items, youtube=youtube_items,
hackernews=hn_items,
best_practices=data.get('best_practices', []), best_practices=data.get('best_practices', []),
prompt_pack=data.get('prompt_pack', []), prompt_pack=data.get('prompt_pack', []),
context_snippet_md=data.get('context_snippet_md', ''), context_snippet_md=data.get('context_snippet_md', ''),
@@ -378,6 +446,7 @@ class Report:
x_error=data.get('x_error'), x_error=data.get('x_error'),
web_error=data.get('web_error'), web_error=data.get('web_error'),
youtube_error=data.get('youtube_error'), youtube_error=data.get('youtube_error'),
hackernews_error=data.get('hackernews_error'),
from_cache=data.get('from_cache', False), from_cache=data.get('from_cache', False),
cache_age_hours=data.get('cache_age_hours'), cache_age_hours=data.get('cache_age_hours'),
) )
scripts/lib/score.py
+64 -4
View File
@@ -280,6 +280,64 @@ def score_youtube_items(items: List[schema.YouTubeItem]) -> List[schema.YouTubeI
return items return items
def compute_hackernews_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
"""Compute raw engagement score for Hacker News item.
Formula: 0.55*log1p(points) + 0.45*log1p(num_comments)
Points are the primary signal on HN; comments indicate depth of discussion.
"""
if engagement is None:
return None
if engagement.score is None and engagement.num_comments is None:
return None
points = log1p_safe(engagement.score)
comments = log1p_safe(engagement.num_comments)
return 0.55 * points + 0.45 * comments
def score_hackernews_items(items: List[schema.HackerNewsItem]) -> List[schema.HackerNewsItem]:
"""Compute scores for Hacker News items.
Uses same weight structure as Reddit/X (relevance + recency + engagement).
"""
if not items:
return items
eng_raw = [compute_hackernews_engagement_raw(item.engagement) for item in items]
eng_normalized = normalize_to_100(eng_raw)
for i, item in enumerate(items):
rel_score = int(item.relevance * 100)
rec_score = dates.recency_score(item.date)
if eng_normalized[i] is not None:
eng_score = int(eng_normalized[i])
else:
eng_score = DEFAULT_ENGAGEMENT
item.subs = schema.SubScores(
relevance=rel_score,
recency=rec_score,
engagement=eng_score,
)
overall = (
WEIGHT_RELEVANCE * rel_score +
WEIGHT_RECENCY * rec_score +
WEIGHT_ENGAGEMENT * eng_score
)
if eng_raw[i] is None:
overall -= UNKNOWN_ENGAGEMENT_PENALTY
item.score = max(0, min(100, int(overall)))
return items
def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]: def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]:
"""Compute scores for WebSearch items WITHOUT engagement metrics. """Compute scores for WebSearch items WITHOUT engagement metrics.
@@ -337,7 +395,7 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS
return items return items
def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem]]) -> List: def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.HackerNewsItem]]) -> List:
"""Sort items by score (descending), then date, then source priority. """Sort items by score (descending), then date, then source priority.
Args: Args:
@@ -354,15 +412,17 @@ def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSear
date = item.date or "0000-00-00" date = item.date or "0000-00-00"
date_key = -int(date.replace("-", "")) date_key = -int(date.replace("-", ""))
# Tertiary: source priority (Reddit > X > YouTube > WebSearch) # Tertiary: source priority (Reddit > X > HN > YouTube > WebSearch)
if isinstance(item, schema.RedditItem): if isinstance(item, schema.RedditItem):
source_priority = 0 source_priority = 0
elif isinstance(item, schema.XItem): elif isinstance(item, schema.XItem):
source_priority = 1 source_priority = 1
elif isinstance(item, schema.YouTubeItem): elif isinstance(item, schema.HackerNewsItem):
source_priority = 2 source_priority = 2
else: # WebSearchItem elif isinstance(item, schema.YouTubeItem):
source_priority = 3 source_priority = 3
else: # WebSearchItem
source_priority = 4
# Quaternary: title/text for stability # Quaternary: title/text for stability
text = getattr(item, "title", "") or getattr(item, "text", "") text = getattr(item, "title", "") or getattr(item, "text", "")
scripts/lib/ui.py
+21 -1
View File
@@ -72,6 +72,13 @@ YOUTUBE_MESSAGES = [
"Fetching transcripts...", "Fetching transcripts...",
] ]
HN_MESSAGES = [
"Searching Hacker News...",
"Scanning HN front page stories...",
"Finding technical discussions...",
"Discovering developer conversations...",
]
PROCESSING_MESSAGES = [ PROCESSING_MESSAGES = [
"Crunching the data...", "Crunching the data...",
"Scoring and ranking...", "Scoring and ranking...",
@@ -257,6 +264,15 @@ class ProgressDisplay:
if self.spinner: if self.spinner:
self.spinner.stop(f"{Colors.RED}YouTube{Colors.RESET} Found {count} videos") self.spinner.stop(f"{Colors.RED}YouTube{Colors.RESET} Found {count} videos")
def start_hackernews(self):
msg = random.choice(HN_MESSAGES)
self.spinner = Spinner(f"{Colors.YELLOW}HN{Colors.RESET} {msg}", Colors.YELLOW)
self.spinner.start()
def end_hackernews(self, count: int):
if self.spinner:
self.spinner.stop(f"{Colors.YELLOW}HN{Colors.RESET} Found {count} stories")
def start_processing(self): def start_processing(self):
msg = random.choice(PROCESSING_MESSAGES) msg = random.choice(PROCESSING_MESSAGES)
self.spinner = Spinner(f"{Colors.PURPLE}Processing{Colors.RESET} {msg}", Colors.PURPLE) self.spinner = Spinner(f"{Colors.PURPLE}Processing{Colors.RESET} {msg}", Colors.PURPLE)
@@ -266,18 +282,22 @@ class ProgressDisplay:
if self.spinner: if self.spinner:
self.spinner.stop() self.spinner.stop()
def show_complete(self, reddit_count: int, x_count: int, youtube_count: int = 0): def show_complete(self, reddit_count: int, x_count: int, youtube_count: int = 0, hn_count: int = 0):
elapsed = time.time() - self.start_time elapsed = time.time() - self.start_time
if IS_TTY: if IS_TTY:
sys.stderr.write(f"\n{Colors.GREEN}{Colors.BOLD}✓ Research complete{Colors.RESET} ") sys.stderr.write(f"\n{Colors.GREEN}{Colors.BOLD}✓ Research complete{Colors.RESET} ")
sys.stderr.write(f"{Colors.DIM}({elapsed:.1f}s){Colors.RESET}\n") sys.stderr.write(f"{Colors.DIM}({elapsed:.1f}s){Colors.RESET}\n")
sys.stderr.write(f" {Colors.YELLOW}Reddit:{Colors.RESET} {reddit_count} threads ") sys.stderr.write(f" {Colors.YELLOW}Reddit:{Colors.RESET} {reddit_count} threads ")
sys.stderr.write(f"{Colors.CYAN}X:{Colors.RESET} {x_count} posts") sys.stderr.write(f"{Colors.CYAN}X:{Colors.RESET} {x_count} posts")
if hn_count:
sys.stderr.write(f" {Colors.YELLOW}HN:{Colors.RESET} {hn_count} stories")
if youtube_count: if youtube_count:
sys.stderr.write(f" {Colors.RED}YouTube:{Colors.RESET} {youtube_count} videos") sys.stderr.write(f" {Colors.RED}YouTube:{Colors.RESET} {youtube_count} videos")
sys.stderr.write("\n\n") sys.stderr.write("\n\n")
else: else:
parts = [f"Reddit: {reddit_count} threads", f"X: {x_count} posts"] parts = [f"Reddit: {reddit_count} threads", f"X: {x_count} posts"]
if hn_count:
parts.append(f"HN: {hn_count} stories")
if youtube_count: if youtube_count:
parts.append(f"YouTube: {youtube_count} videos") parts.append(f"YouTube: {youtube_count} videos")
sys.stderr.write(f"✓ Research complete ({elapsed:.1f}s) - {', '.join(parts)}\n") sys.stderr.write(f"✓ Research complete ({elapsed:.1f}s) - {', '.join(parts)}\n")
tests/test_hackernews.py
+219
View File
@@ -0,0 +1,219 @@
"""Tests for Hacker News source module."""
import sys
import unittest
from pathlib import Path
# Add lib to path
sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
from lib import hackernews, normalize, schema, score
class TestDateToUnix(unittest.TestCase):
def test_known_date(self):
# 2026-01-01 00:00:00 UTC
result = hackernews._date_to_unix("2026-01-01")
self.assertIsInstance(result, int)
self.assertGreater(result, 0)
def test_roundtrip(self):
ts = hackernews._date_to_unix("2026-02-15")
back = hackernews._unix_to_date(ts)
self.assertEqual(back, "2026-02-15")
class TestStripHtml(unittest.TestCase):
def test_basic_html(self):
result = hackernews._strip_html("<p>Hello <b>world</b></p>")
self.assertIn("Hello", result)
self.assertIn("world", result)
self.assertNotIn("<", result)
def test_html_entities(self):
result = hackernews._strip_html("&amp; test")
self.assertIn("&", result)
self.assertIn("test", result)
def test_empty(self):
result = hackernews._strip_html("")
self.assertEqual(result, "")
class TestParseHackernewsResponse(unittest.TestCase):
SAMPLE_RESPONSE = {
"hits": [
{
"objectID": "12345",
"title": "Show HN: A new AI coding assistant",
"url": "https://example.com/article",
"author": "pg",
"points": 350,
"num_comments": 127,
"created_at_i": 1739836800, # 2025-02-18
},
{
"objectID": "12346",
"title": "Ask HN: Best practices for LLM apps?",
"url": "",
"author": "dang",
"points": 80,
"num_comments": 45,
"created_at_i": 1739750400, # 2025-02-17
},
],
}
def test_parses_hits(self):
items = hackernews.parse_hackernews_response(self.SAMPLE_RESPONSE)
self.assertEqual(len(items), 2)
def test_item_fields(self):
items = hackernews.parse_hackernews_response(self.SAMPLE_RESPONSE)
item = items[0]
self.assertEqual(item["object_id"], "12345")
self.assertEqual(item["title"], "Show HN: A new AI coding assistant")
self.assertEqual(item["url"], "https://example.com/article")
self.assertEqual(item["hn_url"], "https://news.ycombinator.com/item?id=12345")
self.assertEqual(item["author"], "pg")
self.assertEqual(item["engagement"]["points"], 350)
self.assertEqual(item["engagement"]["num_comments"], 127)
def test_date_conversion(self):
items = hackernews.parse_hackernews_response(self.SAMPLE_RESPONSE)
self.assertIsNotNone(items[0]["date"])
# Should be a valid YYYY-MM-DD string
self.assertRegex(items[0]["date"], r"^\d{4}-\d{2}-\d{2}$")
def test_hn_url_for_askhn(self):
"""Ask HN posts have no article URL, but should have hn_url."""
items = hackernews.parse_hackernews_response(self.SAMPLE_RESPONSE)
ask_hn = items[1]
self.assertEqual(ask_hn["url"], "")
self.assertIn("news.ycombinator.com", ask_hn["hn_url"])
def test_relevance_range(self):
items = hackernews.parse_hackernews_response(self.SAMPLE_RESPONSE)
for item in items:
self.assertGreaterEqual(item["relevance"], 0.0)
self.assertLessEqual(item["relevance"], 1.0)
def test_empty_response(self):
items = hackernews.parse_hackernews_response({"hits": []})
self.assertEqual(items, [])
def test_missing_hits(self):
items = hackernews.parse_hackernews_response({})
self.assertEqual(items, [])
class TestNormalizeHackernewsItems(unittest.TestCase):
def test_normalize(self):
raw_items = [
{
"object_id": "99999",
"title": "Test Story",
"url": "https://example.com",
"hn_url": "https://news.ycombinator.com/item?id=99999",
"author": "testuser",
"date": "2026-02-15",
"engagement": {"points": 100, "num_comments": 50},
"relevance": 0.8,
"why_relevant": "Test",
}
]
result = normalize.normalize_hackernews_items(raw_items, "2026-01-01", "2026-03-01")
self.assertEqual(len(result), 1)
self.assertIsInstance(result[0], schema.HackerNewsItem)
self.assertEqual(result[0].id, "HN1")
self.assertEqual(result[0].title, "Test Story")
self.assertEqual(result[0].date_confidence, "high")
self.assertEqual(result[0].engagement.score, 100)
self.assertEqual(result[0].engagement.num_comments, 50)
def test_normalize_with_comments(self):
raw_items = [
{
"object_id": "99999",
"title": "Test",
"url": "",
"hn_url": "",
"author": "user",
"date": "2026-02-15",
"engagement": {"points": 10, "num_comments": 5},
"relevance": 0.5,
"why_relevant": "Test",
"top_comments": [
{"author": "commenter", "text": "Great post!", "points": 5},
],
"comment_insights": ["Great post!"],
}
]
result = normalize.normalize_hackernews_items(raw_items, "2026-01-01", "2026-03-01")
self.assertEqual(len(result[0].top_comments), 1)
self.assertEqual(result[0].top_comments[0].author, "commenter")
self.assertEqual(len(result[0].comment_insights), 1)
class TestScoreHackernewsItems(unittest.TestCase):
def test_score_items(self):
items = [
schema.HackerNewsItem(
id="HN1", title="High engagement", url="", hn_url="",
author="user1", date="2026-02-20",
engagement=schema.Engagement(score=500, num_comments=200),
relevance=0.9,
),
schema.HackerNewsItem(
id="HN2", title="Low engagement", url="", hn_url="",
author="user2", date="2026-02-18",
engagement=schema.Engagement(score=10, num_comments=3),
relevance=0.5,
),
]
scored = score.score_hackernews_items(items)
self.assertEqual(len(scored), 2)
# High engagement + high relevance should score higher
self.assertGreater(scored[0].score, scored[1].score)
def test_score_empty(self):
result = score.score_hackernews_items([])
self.assertEqual(result, [])
def test_engagement_formula(self):
eng = schema.Engagement(score=100, num_comments=50)
result = score.compute_hackernews_engagement_raw(eng)
self.assertIsNotNone(result)
self.assertGreater(result, 0)
def test_engagement_none(self):
result = score.compute_hackernews_engagement_raw(None)
self.assertIsNone(result)
def test_engagement_empty(self):
eng = schema.Engagement()
result = score.compute_hackernews_engagement_raw(eng)
self.assertIsNone(result)
class TestSortItemsWithHN(unittest.TestCase):
def test_hn_priority_between_x_and_youtube(self):
"""HN should sort between X and YouTube at same score."""
x_item = schema.XItem(id="X1", text="test", url="", author_handle="user")
x_item.score = 50
hn_item = schema.HackerNewsItem(id="HN1", title="test", url="", hn_url="", author="user")
hn_item.score = 50
yt_item = schema.YouTubeItem(id="YT1", title="test", url="", channel_name="ch")
yt_item.score = 50
sorted_items = score.sort_items([yt_item, hn_item, x_item])
# Same score, so sorted by source priority: X > HN > YouTube
self.assertIsInstance(sorted_items[0], schema.XItem)
self.assertIsInstance(sorted_items[1], schema.HackerNewsItem)
self.assertIsInstance(sorted_items[2], schema.YouTubeItem)
if __name__ == "__main__":
unittest.main()