diff --git a/README.md b/README.md index 1580d5e..54f70d4 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,18 @@ # /last30days v2.5 -**The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, Hacker News, and the web from the last 30 days, finds what the community is actually upvoting, sharing, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know. +**The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, Hacker News, Polymarket, and the web from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know. **New in V2.5 - dramatically better results:** 1. **Smarter scoring across the board.** New relevance scoring with synonym expansion ("hip hop" matches "rap", "MacBook" matches "Mac"), cross-source linking that flags when the same story trends on multiple platforms simultaneously, and X handle resolution that finds viral posts keyword search completely misses. A blinded evaluation scored v2.5 at 4.38/5.0 vs 3.73/5.0 for v1 across 5 test topics. -2. **Hacker News as a 5th source.** HN stories, Show HN posts, and comment threads are now searched, scored, and synthesized alongside Reddit, X, YouTube, and the web. +2. **Hacker News and Polymarket as new sources.** HN stories, Show HN posts, and prediction market odds are now searched, scored, and synthesized alongside Reddit, X, YouTube, and the web. Polymarket surfaces what people are putting real money on - betting odds reflect conviction, not just opinions. 3. **X handle resolution.** Search "Dor Brothers" and the skill resolves their handle (@thedorbrothers), then searches their posts directly - finding their 5,600-like viral tweet that keyword search missed entirely. Works for people, brands, products, and tools. **New in V2.1:** Open-class skill with watchlists, YouTube transcripts as a source, works in OpenAI Codex CLI. [Full changelog below.](#whats-new-in-v21) **New in V2:** Smarter query construction, two-phase supplemental search, free X search via bundled Bird client, `--days=N` flag, automatic model fallback. [Full changelog below.](#whats-new-in-v2) -**The tradeoff:** /last30days finds a lot of content but takes 2-8 minutes depending on how niche your topic is. Five sources searched in parallel, results scored, deduplicated, and synthesized. We think the depth is worth the wait, but `--quick` mode is there if you need speed over thoroughness. +**The tradeoff:** /last30days finds a lot of content but takes 2-8 minutes depending on how niche your topic is. Six sources searched in parallel, results scored, deduplicated, and synthesized. We think the depth is worth the wait, but `--quick` mode is there if you need speed over thoroughness. **Best for prompt research**: discover what prompting techniques actually work for any tool (ChatGPT, Midjourney, Claude, Paper, etc.) by learning from real community discussions and best practices. @@ -122,7 +122,7 @@ Examples: ## What It Does -1. **Researches** - Scans Reddit, X, and YouTube for discussions from the last 30 days +1. **Researches** - Scans Reddit, X, YouTube, Hacker News, Polymarket, and the web for discussions from the last 30 days 2. **Synthesizes** - Identifies patterns, best practices, and what actually works 3. **Delivers** - Either writes copy-paste-ready prompts for your target tool, or gives you a curated expert-level answer @@ -857,7 +857,7 @@ This example shows /last30days discovering **emerging developer workflows** - re - **Node.js 22+** - For X search (bundled Twitter GraphQL client) - **X session** - Be logged into x.com in your browser, or set `AUTH_TOKEN`/`CT0` env vars - **xAI API key** (optional fallback) - If the bundled search can't authenticate, falls back to xAI's Grok API -- **yt-dlp** (optional) - For YouTube search + transcript extraction. Install via `brew install yt-dlp` or `pip install yt-dlp`. When present, automatically searches YouTube and extracts video transcripts as a 4th source. +- **yt-dlp** (optional) - For YouTube search + transcript extraction. Install via `brew install yt-dlp` or `pip install yt-dlp`. When present, automatically searches YouTube and extracts video transcripts as an additional source. At least one API key is required. X search works automatically if you're logged into x.com in your browser. YouTube search activates automatically when yt-dlp is in your PATH. @@ -869,6 +869,8 @@ At least one API key is required. X search works automatically if you're logged - OpenAI Responses API with `web_search` tool scoped to reddit.com - Vendored Twitter GraphQL search (or xAI API fallback) for X search - YouTube search + transcript extraction via yt-dlp (when installed) +- Hacker News search via Algolia API (free, no auth) +- Polymarket prediction market search via Gamma API (free, no auth) - WebSearch for blogs, news, docs, tutorials - Reddit JSON enrichment for real engagement metrics (upvotes, comments) - Scoring algorithm weighing recency, relevance, and engagement @@ -913,7 +915,7 @@ Ran a 15-way blinded comparison across 5 topics (Claude Code, Seedance, MacBook Scored on groundedness (30%), specificity (25%), coverage (20%), actionability (15%), format (10%). The relative ranking is meaningful; absolute numbers are LLM-grading-LLM and shouldn't be taken as objective quality scores. The biggest gains came from detecting where sources agree - not just finding more sources. -### Hacker News as a 5th source +### Hacker News as a source **The technical community's signal, captured automatically.** HN stories, Show HN posts, and Ask HN threads are searched, scored by points + comments, and synthesized alongside Reddit, X, YouTube, and the web. Comment insights are extracted from top threads to surface the technical community's actual take - not just headlines. @@ -921,6 +923,14 @@ HN items go through the same scoring pipeline as every other source and particip Inspired by community PRs from [@ARJ999](https://github.com/ARJ999) ([#12](https://github.com/mvanhorn/last30days-skill/pull/12)) and [@wkbaran](https://github.com/wkbaran) ([#26](https://github.com/mvanhorn/last30days-skill/pull/26)), with [@gbessoni](https://github.com/gbessoni) endorsing HN as the right addition. +### Polymarket prediction markets as a source + +**What people are putting real money on.** Polymarket prediction markets are searched for any topic, surfacing betting odds and price movements alongside what people are saying on Reddit/X/YouTube/HN. Search "Iran" and you'll find markets on US strikes, Khamenei's future, and nuclear negotiations - with live odds and volume. + +Uses smart multi-query expansion (same approach as YouTube synonym expansion and X handle resolution) to cast a wider net. "Arizona Basketball" finds markets on Big 12 title odds, NCAA tournament seeding, and March Madness outcomes - not just literal keyword matches. + +No API key required - uses Polymarket's free public Gamma API. Sources with zero results are automatically hidden from the stats output. + ### X handle resolution details The problem: when you search a topic on X, you find posts *about* it. But the topic's own account often doesn't mention its own name in tweets. Keyword search can't find those posts. @@ -1014,6 +1024,7 @@ Thanks to the contributors who helped shape V2: | Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY | | `youtube.com` (via yt-dlp) | Search query | None (public search) | | `hn.algolia.com` | Search query | None (public API) | +| `gamma-api.polymarket.com` | Search query | None (public API) | | `api.search.brave.com` | Search query (optional) | BRAVE_API_KEY | | `api.parallel.ai` | Search query (optional) | PARALLEL_API_KEY | | `openrouter.ai` | Search query (optional) | OPENROUTER_API_KEY | @@ -1032,6 +1043,6 @@ Each API key is transmitted only to its respective endpoint. Your OpenAI key is --- -*30 days of research. 30 seconds of work. Four sources. Zero stale prompts.* +*30 days of research. 30 seconds of work. Six sources. Zero stale prompts.* *Pair with [Open Claw](https://github.com/openclaw/openclaw) for automated watchlists and briefings. Reddit. X. YouTube. Web. - All synthesized into expert answers and copy-paste prompts.* diff --git a/SKILL.md b/SKILL.md index cd4aab2..ca2f294 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,7 +1,7 @@ --- name: last30days version: "2.1" -description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, Hacker News, web. Become an expert and write copy-paste-ready prompts." +description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, Hacker News, Polymarket, web. Become an expert and write copy-paste-ready prompts." argument-hint: 'last30 AI video tools, last30 best project management tools' allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch homepage: https://github.com/mvanhorn/last30days-skill @@ -32,7 +32,7 @@ metadata: # last30days v2.1: Research Any Topic from the Last 30 Days -Research ANY topic across Reddit, X, YouTube, Hacker News, and the web. Surface what people are actually discussing, recommending, and debating right now. +Research ANY topic across Reddit, X, YouTube, Hacker News, Polymarket, and the web. Surface what people are actually discussing, recommending, betting on, and debating right now. ## CRITICAL: Parse User Intent @@ -141,10 +141,10 @@ Use a **timeout of 300000** (5 minutes) on the Bash call. The script typically t The script will automatically: - Detect available API keys -- Run Reddit/X/YouTube/Hacker News searches -- Output ALL results including YouTube transcripts and HN comments +- Run Reddit/X/YouTube/Hacker News/Polymarket searches +- Output ALL results including YouTube transcripts, HN comments, and prediction market odds -**Read the ENTIRE output.** It contains FIVE data sections in this order: Reddit items, X items, YouTube items, Hacker News items, and WebSearch items. If you miss sections, you will produce incomplete stats. +**Read the ENTIRE output.** It contains SIX data sections in this order: Reddit items, X items, YouTube items, Hacker News items, Polymarket items, and WebSearch items. If you miss sections, you will produce incomplete stats. **YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, and optional transcript snippet. Count them and include them in your synthesis and stats block. @@ -339,14 +339,13 @@ KEY PATTERNS from the research: ├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts ├─ 🔴 YouTube: {N} videos │ {N} views │ {N} with transcripts ├─ 🟡 HN: {N} stories │ {N} points │ {N} comments +├─ 📈 Polymarket: {N} markets │ ${vol} volume │ top odds: {outcome} {pct}% ├─ 🌐 Web: {N} pages (supplementary) └─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2} --- ``` -If Reddit returned 0 threads, write: "├─ 🟠 Reddit: 0 threads (no results this cycle)" -If HN returned 0 stories, write: "├─ 🟡 HN: 0 stories (no results this cycle)" -If YouTube returned 0 videos or yt-dlp is not installed, omit the YouTube line entirely. +Omit any source line that returned 0 results. Do NOT show "0 threads" or "0 stories" lines. NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji. **SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If you catch yourself projecting your own knowledge instead of the research, rewrite it. @@ -521,6 +520,7 @@ Want another prompt? Just tell me what you're creating next. - Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery - Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search - Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth) +- Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth) - Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data) - Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search - Fetches public Reddit thread data from `reddit.com` for engagement metrics @@ -532,7 +532,7 @@ Want another prompt? Just tell me what you're creating next. - Does not share API keys between providers (OpenAI key only goes to api.openai.com, etc.) - Does not log, cache, or write API keys to output files - Does not send data to any endpoint not listed above -- Hacker News source is always available (no API key, no binary dependency) +- Hacker News and Polymarket sources are always available (no API key, no binary dependency) - Cannot be invoked autonomously by the agent (`disable-model-invocation: true`) **Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed) diff --git a/SPEC.md b/SPEC.md index a464c0b..e2cd212 100644 --- a/SPEC.md +++ b/SPEC.md @@ -18,6 +18,8 @@ The orchestrator (`last30days.py`) coordinates discovery, enrichment, normalizat - **openai_reddit.py**: OpenAI Responses API + web_search for Reddit - **xai_x.py**: xAI Responses API + x_search for X - **reddit_enrich.py**: Fetch Reddit thread JSON for real engagement metrics +- **hackernews.py**: Hacker News search via Algolia API (free, no auth) +- **polymarket.py**: Polymarket prediction market search via Gamma API (free, no auth) - **normalize.py**: Convert raw API responses to canonical schema - **score.py**: Compute popularity-aware scores (relevance + recency + engagement) - **dedupe.py**: Near-duplicate detection via text similarity diff --git a/docs/plans/2026-02-25-feat-polymarket-prediction-market-source-plan.md b/docs/plans/2026-02-25-feat-polymarket-prediction-market-source-plan.md new file mode 100644 index 0000000..7382cb3 --- /dev/null +++ b/docs/plans/2026-02-25-feat-polymarket-prediction-market-source-plan.md @@ -0,0 +1,254 @@ +--- +title: "feat: Add Polymarket prediction market search as 6th source" +type: feat +status: completed +date: 2026-02-25 +--- + +# feat: Add Polymarket Prediction Market Search + +## Overview + +Add Polymarket as a 6th research source to last30days. When researching any topic, search Polymarket's public API for relevant prediction markets - surfacing what people are putting real money on alongside what they're saying on Reddit/X/YouTube/HN/Web. + +Example: "/last30days Arizona Basketball" would find markets on tournament seeding, Big 12 title odds, and March Madness outcomes. The signal is unique - betting odds reflect conviction, not just opinions. + +## Proposed Solution + +Hit Polymarket's Gamma API with **smart multi-query search** - not just the raw topic, but expanded related keywords to find markets a human researcher would think of. Show price movement context ("down from 23.7% peak") using the API's built-in price change fields. No API key needed. + +Also: hide sources with zero results from the stats box (all sources, not just Polymarket). + +## Technical Approach + +### API Details + +**Endpoint:** `GET https://gamma-api.polymarket.com/public-search?q={topic}&limit={N}` + +- No authentication required (free, public, read-only) +- Rate limit: 350 req/10s (very generous) +- Returns `{"events": [...], "pagination": {...}}` +- Events contain nested `markets` arrays +- `outcomePrices` is a JSON-encoded string - must `json.loads()` it +- `volume` and `liquidity` are strings at market level, floats at event level +- **Price movement fields on every market:** `oneDayPriceChange`, `oneWeekPriceChange`, `oneMonthPriceChange` - these are free, no extra API calls + +### Intelligence Layer: Smart Query Expansion + +A single keyword search is dumb. "Iran" should find markets about Iran strikes, nuclear program, sanctions, oil prices, Khamenei. "Arizona Basketball" should find NCAA tournament odds, Big 12 title, March Madness seeding. + +**How it works:** + +1. **Extract core subject** using `_extract_core_subject(topic)` (already exists for YouTube/X) +2. **Generate 2-4 expanded queries** from the topic: + - The raw topic itself: `"Arizona Basketball"` + - Broader context: `"Arizona NCAA"`, `"Arizona Big 12"` + - For people/entities: add key associated terms (e.g., "Iran" -> "Iran strikes", "Khamenei", "Iran nuclear") +3. **Run all queries in parallel** against `/public-search` (rate limit is 350/10s, we're using 3-4) +4. **Merge and dedupe** results across queries (same event ID = same market) +5. **Score by relevance to original topic** - markets found by the raw topic query get a relevance boost over expanded-query matches + +**Query expansion strategy:** + +```python +def _expand_polymarket_queries(topic: str) -> List[str]: + """Generate 2-4 search queries to cast a wider net.""" + core = _extract_core_subject(topic) + queries = [core] # Always include the core topic + + # Split multi-word topics into component searches + words = core.split() + if len(words) >= 2: + # Try the first significant word alone (e.g., "Arizona" from "Arizona Basketball") + queries.append(words[0]) + + # Add the full topic if different from core + if topic.lower() != core.lower(): + queries.append(topic) + + return list(dict.fromkeys(queries))[:4] # Dedupe, cap at 4 +``` + +This is the same approach as YouTube synonym expansion and X handle resolution - cast a wider net, then score by relevance. + +### Price Movement Context + +The API gives us price change data for free. Use it to make the output actually useful: + +``` +Will Arizona win the NCAA Tournament? + Yes: 12% (down 11.7% this month) | No: 88% + $342K vol24h | $2.1M liquidity +``` + +**Fields available:** +- `oneDayPriceChange` - "up 3% today" +- `oneWeekPriceChange` - "down 5% this week" +- `oneMonthPriceChange` - "down 11.7% this month" + +Show the most significant movement (largest absolute change). Only show if change > 1% to avoid noise. + +### Key Design Decisions + +**Event-level granularity (not market-level).** Each event becomes one `PolymarketItem` showing its title and top 3 markets by volume. A "Trump" query returns 10 events, not 40+ individual markets. This keeps item counts manageable. + +**Smart multi-query search with dedup.** Don't rely on one keyword search. Expand the topic into 2-4 related queries, run them all, merge by event ID. This catches markets that use different terminology than the user's query. + +**Price movement context by default.** Show the most significant price change (day/week/month) inline with outcome prices. This makes prediction markets actually useful as a research signal - not just "Yes 65%" but "Yes 65%, up 12% this week." + +**Date filtering uses `active=true` + `volume1mo > 0`.** Markets can be created months ago but still actively traded. Filter by recent trading activity, not creation date. Set `date` to `updatedAt` for recency scoring. Set `date_confidence` to "high" (API provides exact timestamps). + +**Sort priority: below HN, above WebSearch.** Priority 4 (Reddit=0 > X=1 > YouTube=2 > HN=3 > Polymarket=4 > WebSearch=5). Prediction markets are supplementary signal. + +**Multi-outcome markets: show top 3 outcomes by price.** Binary markets show "Yes: 65%, No: 35%". Multi-candidate markets truncate to top 3 with "and N more". + +**Exclude closed/resolved markets for v1.** Only show `active=true, closed=false`. + +**Link to event pages** (not individual market pages) - shows all related markets in context. + +### Implementation Plan + +#### Phase 1: Hide zero-result sources (separate commit) + +- [x] `scripts/lib/render.py` - In `render_source_status()`, skip lines where count is 0 +- [x] `SKILL.md` - Remove "(no results this cycle)" instructions, replace with "omit sources with zero results" +- [x] `variants/open/references/research.md` - Same update +- [x] `~/.claude/skills/last30daysCROSS/SKILL.md` - Same update (via sync.sh) +- [x] Test: run a query where HN returns 0, verify it's hidden + +#### Phase 2: Polymarket source module + +- [x] `scripts/lib/polymarket.py` - New file: + - `DEPTH_CONFIG = {"quick": 5, "default": 10, "deep": 20}` (event count per query) + - `_expand_queries(topic)` - generate 2-4 search queries from topic: + - Raw topic: `"Arizona Basketball"` + - Core subject extracted: `"Arizona"` (via `_extract_core_subject`) + - Component words for multi-word topics: first significant word alone + - Cap at 4 queries, dedupe + - `search_polymarket(topic, from_date, to_date, depth)` - run all expanded queries against `/public-search` in parallel (ThreadPoolExecutor), merge results by event ID, dedupe + - `parse_polymarket_response(response)` - extract events, flatten top markets per event + - `_format_price_movement(market)` - pick most significant price change from `oneDayPriceChange`, `oneWeekPriceChange`, `oneMonthPriceChange`, format as "up/down X% this day/week/month", skip if < 1% + - URL-encode query params with `urllib.parse.urlencode` + - Filter: `active=true, closed=false, volume1mo > 0` + - Parse `outcomePrices` with `json.loads()`, handle malformed/missing gracefully + - Handle mixed types: `float(volume)` with try/except + - Relevance scoring: markets from raw topic query get 1.0 base, expanded queries get 0.7 base, then decay by position + - Return `{"items": [...]}` or `{"items": [], "error": "message"}` + +#### Phase 3: Schema + normalization + scoring + +- [x] `scripts/lib/schema.py` - Add `PolymarketItem` dataclass: + - `id`, `title` (event title), `question` (top market question), `url` (event URL) + - `outcome_prices: List[Tuple[str, float]]` (parsed, e.g. [("Yes", 0.65), ("No", 0.35)]) + - `price_movement: Optional[str]` (formatted, e.g. "down 11.7% this month") + - `volume24hr: float`, `liquidity: float`, `end_date: Optional[str]` + - Standard fields: `date`, `date_confidence`, `engagement`, `relevance`, `why_relevant`, `subs`, `score`, `cross_refs` + - Add `volume: Optional[float]` and `liquidity: Optional[float]` to `Engagement` dataclass + - Add `polymarket: List[PolymarketItem]` and `polymarket_error: Optional[str]` to `Report` + - Update `Report.to_dict()` and `Report.from_dict()` (handle missing key for backward compat) + +- [x] `scripts/lib/normalize.py` - Add `normalize_polymarket_items()`: + - Map raw API events to `PolymarketItem` instances + - Set `date` from `updatedAt`, `date_confidence = "high"` + - Build `Engagement(volume=volume24hr, liquidity=liquidity)` + - Update `TypeVar` to include `PolymarketItem` + +- [x] `scripts/lib/score.py` - Add scoring: + - `compute_polymarket_engagement_raw()`: `0.60 * log1p(volume24hr) + 0.40 * log1p(liquidity)` + - `score_polymarket_items()`: standard 45/25/30 weights (relevance/recency/engagement) + - Update `sort_items()`: Polymarket priority = 4 + +#### Phase 4: Dedupe + render + UI + +- [x] `scripts/lib/dedupe.py`: + - Add `PolymarketItem` to `AnyItem` union + - Add `dedupe_polymarket()` function + - Update `get_item_text()` for `PolymarketItem` (return `title + " " + question`) + - Update `_get_cross_source_text()` for cross-source linking + +- [x] `scripts/lib/render.py`: + - Add `_xref_tag()` prefix: `PM` -> `Polymarket` + - Add Polymarket section to `render_compact()`: + ``` + ### Prediction Markets (Polymarket) + + **PM1** (score:72) [$342K vol24h, $2.1M liquidity] + Will Arizona win the NCAA Tournament? + Yes: 12% (down 11.7% this month) | No: 88% + https://polymarket.com/event/arizona-ncaa-tournament + + **PM2** (score:65) [$89K vol24h, $450K liquidity] + Will Arizona win the Big 12 Tournament? + Yes: 28% (up 4% this week) | No: 72% + https://polymarket.com/event/arizona-big-12 + ``` + - Add to `render_source_status()` (with zero-result hiding from Phase 1) + - Add to `render_full_report()` and `render_context_snippet()` + - Add to `_assess_data_freshness()` + +- [x] `scripts/lib/ui.py`: + - Add `POLYMARKET_MESSAGES` list + - Add `start_polymarket()` / `end_polymarket()` methods + - Update `show_complete()` to accept `polymarket_count` + +#### Phase 5: Wire into main pipeline + +- [x] `scripts/last30days.py`: + - Import `from lib import polymarket` + - Add `_search_polymarket()` wrapper function + - Add `polymarket_future` to `TIMEOUT_PROFILES` (15s - API is fast) + - In `run_research()`: submit polymarket future to ThreadPoolExecutor, increment max_workers + - Collect results with timeout, add to processing pipeline (normalize -> filter -> score -> sort -> dedupe) + - Feed into `cross_source_link()` call + - Assign to `report.polymarket` / `report.polymarket_error` + - Update `--diagnose` output: `"polymarket": True` (always available) + - Update `--store` persistence loop for Polymarket items + - Update `show_complete()` call with `len(deduped_pm)` + +- [x] `scripts/lib/env.py` - Add `is_polymarket_available()` -> always `True` + +#### Phase 6: Documentation + deploy + +- [x] `SKILL.md` - Update "6 sources", add Polymarket to stats box template, update Security section with `gamma-api.polymarket.com` +- [x] `variants/open/references/research.md` - Same source count + stats updates +- [x] `README.md` - Add Polymarket to feature list, update source count references +- [x] `SPEC.md` - Add `polymarket.py` to architecture list +- [x] Run `bash scripts/sync.sh` to deploy + +#### Phase 7: Tests + +- [x] `tests/test_polymarket.py` - New file: + - `TestParsePolymarketResponse` - binary markets, multi-outcome, malformed outcomePrices, missing fields + - `TestNormalizePolymarketItems` - schema mapping, date handling, type coercion + - `TestScorePolymarketItems` - engagement formula, zero volume, high volume +- [x] `fixtures/polymarket_sample.json` - Representative API response with edge cases +- [x] `tests/test_cross_source.py` - Add Polymarket-to-Reddit linking test case + +## Acceptance Criteria + +- [x] `/last30days "Arizona Basketball"` shows relevant Polymarket prediction markets with odds and volume +- [x] `/last30days "best rap songs 2026"` gracefully returns no Polymarket results (and hides the PM line from stats) +- [x] Sources with zero results are hidden from stats box (all sources, not just PM) +- [x] `--diagnose` shows `"polymarket": true` +- [x] Multi-outcome markets show top 3 outcomes +- [x] Polymarket items participate in cross-source linking +- [x] No API key required - works out of the box for all users +- [x] `--quick`, `--deep` flags affect Polymarket result count +- [x] Older cached reports without `polymarket` key load without errors + +## Dependencies & Risks + +**No blockers.** The Gamma API is free, unauthenticated, and has generous rate limits (350 req/10s). No new dependencies needed - just stdlib `urllib` (already used by HN module). + +**Risk: API availability.** Polymarket could change or restrict their API. Mitigation: the source is optional and fails gracefully (returns empty list, hidden from stats). + +**Risk: Relevance mismatch.** Many topics won't have prediction markets. The hide-zero-sources change ensures this doesn't pollute the output. + +## Sources & References + +- Polymarket Gamma API docs: https://docs.polymarket.com/developers/gamma-markets-api/overview +- Polymarket CLI: https://github.com/Polymarket/polymarket-cli +- HN source plan (pattern to follow): `docs/plans/2026-02-24-feat-hacker-news-source-plan.md` +- HN source implementation (most recent similar feature): `scripts/lib/hackernews.py` +- Schema: `scripts/lib/schema.py` +- Main pipeline: `scripts/last30days.py` diff --git a/fixtures/polymarket_sample.json b/fixtures/polymarket_sample.json new file mode 100644 index 0000000..cfc68fd --- /dev/null +++ b/fixtures/polymarket_sample.json @@ -0,0 +1,150 @@ +{ + "events": [ + { + "id": "evt-arizona-big12", + "title": "Will Arizona win the Big 12 Championship?", + "slug": "arizona-big-12-championship", + "active": true, + "closed": false, + "updatedAt": "2026-02-24T18:30:00.000Z", + "markets": [ + { + "id": "mkt-arizona-big12-1", + "question": "Will Arizona win the Big 12 Championship?", + "active": true, + "closed": false, + "outcomes": "[\"Yes\", \"No\"]", + "outcomePrices": "[\"0.64\", \"0.36\"]", + "volume": "1500000", + "volume24hr": "342000", + "liquidity": "2100000", + "oneDayPriceChange": 0.03, + "oneWeekPriceChange": 0.225, + "oneMonthPriceChange": 0.15, + "endDate": "2026-03-15T00:00:00.000Z" + } + ] + }, + { + "id": "evt-arizona-ncaa", + "title": "Will Arizona win the NCAA Tournament?", + "slug": "arizona-ncaa-tournament", + "active": true, + "closed": false, + "updatedAt": "2026-02-23T12:00:00.000Z", + "markets": [ + { + "id": "mkt-arizona-ncaa-1", + "question": "Will Arizona win the NCAA Tournament?", + "active": true, + "closed": false, + "outcomes": "[\"Yes\", \"No\"]", + "outcomePrices": "[\"0.12\", \"0.88\"]", + "volume": "800000", + "volume24hr": "89000", + "liquidity": "450000", + "oneDayPriceChange": -0.005, + "oneWeekPriceChange": -0.02, + "oneMonthPriceChange": -0.117, + "endDate": "2026-04-07T00:00:00.000Z" + } + ] + }, + { + "id": "evt-closed-resolved", + "title": "Resolved Event (should be filtered)", + "slug": "resolved-event", + "active": true, + "closed": true, + "updatedAt": "2026-02-20T10:00:00.000Z", + "markets": [ + { + "id": "mkt-resolved-1", + "question": "Already resolved?", + "active": true, + "closed": true, + "outcomes": "[\"Yes\", \"No\"]", + "outcomePrices": "[\"1\", \"0\"]", + "volume": "500000", + "volume24hr": "0", + "liquidity": "0", + "oneDayPriceChange": 0, + "oneWeekPriceChange": 0, + "oneMonthPriceChange": 0 + } + ] + }, + { + "id": "evt-multi-outcome", + "title": "Who will win the Big 12 Tournament?", + "slug": "big-12-tournament-winner", + "active": true, + "closed": false, + "updatedAt": "2026-02-24T20:00:00.000Z", + "markets": [ + { + "id": "mkt-multi-1", + "question": "Who will win the Big 12 Tournament?", + "active": true, + "closed": false, + "outcomes": "[\"Arizona\", \"Kansas\", \"Houston\", \"Iowa State\", \"BYU\"]", + "outcomePrices": "[\"0.35\", \"0.22\", \"0.18\", \"0.15\", \"0.10\"]", + "volume": "2000000", + "volume24hr": "150000", + "liquidity": "1800000", + "oneDayPriceChange": 0.05, + "oneWeekPriceChange": 0.08, + "oneMonthPriceChange": 0.12 + } + ] + }, + { + "id": "evt-no-liquidity", + "title": "Dead market (no liquidity)", + "slug": "dead-market", + "active": true, + "closed": false, + "updatedAt": "2026-02-10T00:00:00.000Z", + "markets": [ + { + "id": "mkt-dead-1", + "question": "Dead market?", + "active": true, + "closed": false, + "outcomes": "[\"Yes\", \"No\"]", + "outcomePrices": "[\"0.50\", \"0.50\"]", + "volume": "100", + "volume24hr": "0", + "liquidity": "0", + "oneDayPriceChange": 0, + "oneWeekPriceChange": 0, + "oneMonthPriceChange": 0 + } + ] + }, + { + "id": "evt-malformed", + "title": "Malformed prices", + "slug": "malformed-prices", + "active": true, + "closed": false, + "updatedAt": "2026-02-22T08:00:00.000Z", + "markets": [ + { + "id": "mkt-malformed-1", + "question": "Malformed outcomes?", + "active": true, + "closed": false, + "outcomes": "not valid json", + "outcomePrices": "also not valid json", + "volume": "50000", + "volume24hr": "5000", + "liquidity": "30000", + "oneDayPriceChange": 0, + "oneWeekPriceChange": 0, + "oneMonthPriceChange": 0 + } + ] + } + ] +} diff --git a/scripts/last30days.py b/scripts/last30days.py index 1517417..096a38f 100644 --- a/scripts/last30days.py +++ b/scripts/last30days.py @@ -38,9 +38,9 @@ _child_pids: set = set() _child_pids_lock = threading.Lock() TIMEOUT_PROFILES = { - "quick": {"global": 90, "future": 30, "reddit_future": 60, "youtube_future": 60, "hackernews_future": 30, "http": 15, "enrich_per": 8, "enrich_total": 30, "enrich_max_items": 10}, - "default": {"global": 180, "future": 60, "reddit_future": 90, "youtube_future": 90, "hackernews_future": 60, "http": 30, "enrich_per": 15, "enrich_total": 45, "enrich_max_items": 15}, - "deep": {"global": 300, "future": 90, "reddit_future": 120, "youtube_future": 120, "hackernews_future": 90, "http": 30, "enrich_per": 15, "enrich_total": 60, "enrich_max_items": 25}, + "quick": {"global": 90, "future": 30, "reddit_future": 60, "youtube_future": 60, "hackernews_future": 30, "polymarket_future": 15, "http": 15, "enrich_per": 8, "enrich_total": 30, "enrich_max_items": 10}, + "default": {"global": 180, "future": 60, "reddit_future": 90, "youtube_future": 90, "hackernews_future": 60, "polymarket_future": 30, "http": 30, "enrich_per": 15, "enrich_total": 45, "enrich_max_items": 15}, + "deep": {"global": 300, "future": 90, "reddit_future": 120, "youtube_future": 120, "hackernews_future": 90, "polymarket_future": 45, "http": 30, "enrich_per": 15, "enrich_total": 60, "enrich_max_items": 25}, } @@ -99,6 +99,7 @@ from lib import ( dates, dedupe, hackernews, + polymarket, entity_extract, env, http, @@ -332,6 +333,34 @@ def _search_hackernews( return hn_items, hn_error +def _search_polymarket( + topic: str, + from_date: str, + to_date: str, + depth: str, +) -> tuple: + """Search Polymarket via Gamma API (runs in thread). + + Returns: + Tuple of (pm_items, pm_error) + """ + pm_error = None + + try: + response = polymarket.search_polymarket( + topic, from_date, to_date, depth=depth, + ) + except Exception as e: + return [], f"{type(e).__name__}: {e}" + + pm_items = polymarket.parse_polymarket_response(response, topic=topic) + + if response.get("error"): + pm_error = response["error"] + + return pm_items, pm_error + + def _search_web( topic: str, config: dict, @@ -588,6 +617,7 @@ def run_research( x_items = [] youtube_items = [] hackernews_items = [] + polymarket_items = [] web_items = [] raw_openai = None raw_xai = None @@ -596,6 +626,7 @@ def run_research( x_error = None youtube_error = None hackernews_error = None + polymarket_error = None web_error = None # Determine web search mode @@ -638,20 +669,22 @@ def run_research( progress.show_error(f"YouTube error: {e}") if progress: progress.end_youtube(len(youtube_items)) - return reddit_items, x_items, youtube_items, hackernews_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, web_error + return reddit_items, x_items, youtube_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, polymarket_error, web_error # Determine which searches to run do_reddit = sources in ("both", "reddit", "all", "reddit-web") do_x = sources in ("both", "x", "all", "x-web") do_hackernews = True # HN is always available (no API key) + do_polymarket = True # Polymarket is always available (no API key) - # Run Reddit, X, YouTube, HN, and Web searches in parallel + # Run Reddit, X, YouTube, HN, Polymarket, and Web searches in parallel reddit_future = None x_future = None youtube_future = None hackernews_future = None + polymarket_future = None web_future = None - max_workers = 2 + (1 if run_youtube else 0) + (1 if do_hackernews else 0) + (1 if web_backend else 0) + max_workers = 2 + (1 if run_youtube else 0) + (1 if do_hackernews else 0) + (1 if do_polymarket else 0) + (1 if web_backend else 0) with ThreadPoolExecutor(max_workers=max_workers) as executor: # Submit searches @@ -685,6 +718,13 @@ def run_research( _search_hackernews, topic, from_date, to_date, depth ) + if do_polymarket: + if progress: + progress.start_polymarket() + polymarket_future = executor.submit( + _search_polymarket, topic, from_date, to_date, depth + ) + if web_backend: sys.stderr.write(f"[web] Searching via {web_backend}\n") sys.stderr.flush() @@ -760,6 +800,23 @@ def run_research( if progress: progress.end_hackernews(len(hackernews_items)) + if polymarket_future: + pm_timeout = timeouts.get("polymarket_future", future_timeout) + try: + polymarket_items, polymarket_error = polymarket_future.result(timeout=pm_timeout) + if polymarket_error and progress: + progress.show_error(f"Polymarket error: {polymarket_error}") + except TimeoutError: + polymarket_error = f"Polymarket search timed out after {pm_timeout}s" + if progress: + progress.show_error(polymarket_error) + except Exception as e: + polymarket_error = f"{type(e).__name__}: {e}" + if progress: + progress.show_error(f"Polymarket error: {e}") + if progress: + progress.end_polymarket(len(polymarket_items)) + if web_future: try: web_items, web_error = web_future.result(timeout=future_timeout) @@ -868,7 +925,7 @@ def run_research( if sup_x: x_items.extend(sup_x) - return reddit_items, x_items, youtube_items, hackernews_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, web_error + return reddit_items, x_items, youtube_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, polymarket_error, web_error def main(): @@ -994,6 +1051,7 @@ def main(): "bird_username": x_source_status.get("bird_username"), "youtube": has_ytdlp, "hackernews": True, + "polymarket": True, "web_search_backend": web_source, "parallel_ai": bool(config.get("PARALLEL_API_KEY")), "brave": bool(config.get("BRAVE_API_KEY")), @@ -1022,6 +1080,7 @@ def main(): "bird_username": x_source_status.get("bird_username"), "youtube": has_ytdlp, "hackernews": True, + "polymarket": True, "web_search_backend": web_source, } ui.show_diagnostic_banner(diag) @@ -1099,7 +1158,7 @@ def main(): mode = sources # Run research - reddit_items, x_items, youtube_items, hackernews_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, web_error = run_research( + reddit_items, x_items, youtube_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, polymarket_error, web_error = run_research( args.topic, sources, config, @@ -1123,6 +1182,7 @@ def main(): normalized_x = normalize.normalize_x_items(x_items, from_date, to_date) normalized_youtube = normalize.normalize_youtube_items(youtube_items, from_date, to_date) if youtube_items else [] normalized_hn = normalize.normalize_hackernews_items(hackernews_items, from_date, to_date) if hackernews_items else [] + normalized_pm = normalize.normalize_polymarket_items(polymarket_items, from_date, to_date) if polymarket_items else [] normalized_web = websearch.normalize_websearch_items(web_items, from_date, to_date) if web_items else [] # Hard date filter: exclude items with verified dates outside the range @@ -1134,6 +1194,8 @@ def main(): # YouTube content has a longer shelf life than tweets/posts. filtered_youtube = normalized_youtube filtered_hn = normalize.filter_by_date_range(normalized_hn, from_date, to_date) if normalized_hn else [] + # Polymarket: skip hard date filter - markets are active/traded, updatedAt is fine + filtered_pm = normalized_pm filtered_web = normalize.filter_by_date_range(normalized_web, from_date, to_date) if normalized_web else [] # Score items @@ -1141,6 +1203,7 @@ def main(): scored_x = score.score_x_items(filtered_x) scored_youtube = score.score_youtube_items(filtered_youtube) if filtered_youtube else [] scored_hn = score.score_hackernews_items(filtered_hn) if filtered_hn else [] + scored_pm = score.score_polymarket_items(filtered_pm) if filtered_pm else [] scored_web = score.score_websearch_items(filtered_web) if filtered_web else [] # Sort items @@ -1148,6 +1211,7 @@ def main(): sorted_x = score.sort_items(scored_x) sorted_youtube = score.sort_items(scored_youtube) if scored_youtube else [] sorted_hn = score.sort_items(scored_hn) if scored_hn else [] + sorted_pm = score.sort_items(scored_pm) if scored_pm else [] sorted_web = score.sort_items(scored_web) if scored_web else [] # Dedupe items @@ -1155,6 +1219,7 @@ def main(): deduped_x = dedupe.dedupe_x(sorted_x) deduped_youtube = dedupe.dedupe_youtube(sorted_youtube) if sorted_youtube else [] deduped_hn = dedupe.dedupe_hackernews(sorted_hn) if sorted_hn else [] + deduped_pm = dedupe.dedupe_polymarket(sorted_pm) if sorted_pm else [] deduped_web = websearch.dedupe_websearch(sorted_web) if sorted_web else [] # Minimum result guarantee: if all Reddit results were filtered out but @@ -1166,7 +1231,7 @@ def main(): # Cross-source linking: annotate items that discuss the same story dedupe.cross_source_link( - deduped_reddit, deduped_x, deduped_youtube, deduped_hn, deduped_web, + deduped_reddit, deduped_x, deduped_youtube, deduped_hn, deduped_pm, deduped_web, ) progress.end_processing() @@ -1184,11 +1249,13 @@ def main(): report.x = deduped_x report.youtube = deduped_youtube report.hackernews = deduped_hn + report.polymarket = deduped_pm report.web = deduped_web report.reddit_error = reddit_error report.x_error = x_error report.youtube_error = youtube_error report.hackernews_error = hackernews_error + report.polymarket_error = polymarket_error report.web_error = web_error report.resolved_x_handle = args.x_handle @@ -1202,7 +1269,7 @@ def main(): if sources == "web": progress.show_web_only_complete() else: - progress.show_complete(len(deduped_reddit), len(deduped_x), len(deduped_youtube), len(deduped_hn)) + progress.show_complete(len(deduped_reddit), len(deduped_x), len(deduped_youtube), len(deduped_hn), len(deduped_pm)) # Build source info for status footer source_info = {} @@ -1270,6 +1337,16 @@ def main(): "engagement_score": item.engagement.score if item.engagement else 0, "relevance_score": item.relevance, }) + for item in deduped_pm: + findings.append({ + "source": "polymarket", + "url": item.url, + "title": item.question, + "author": "polymarket", + "content": item.title, + "engagement_score": item.engagement.volume if item.engagement and item.engagement.volume else 0, + "relevance_score": item.relevance, + }) for item in deduped_web: findings.append({ "source": "web", diff --git a/scripts/lib/dedupe.py b/scripts/lib/dedupe.py index 97b6779..1b65f36 100644 --- a/scripts/lib/dedupe.py +++ b/scripts/lib/dedupe.py @@ -46,7 +46,7 @@ def jaccard_similarity(set1: Set[str], set2: Set[str]) -> float: AnyItem = Union[schema.RedditItem, schema.XItem, schema.YouTubeItem, - schema.HackerNewsItem, schema.WebSearchItem] + schema.HackerNewsItem, schema.PolymarketItem, schema.WebSearchItem] def get_item_text(item: AnyItem) -> str: @@ -57,6 +57,8 @@ def get_item_text(item: AnyItem) -> str: return item.title elif isinstance(item, schema.YouTubeItem): return f"{item.title} {item.channel_name}" + elif isinstance(item, schema.PolymarketItem): + return f"{item.title} {item.question}" elif isinstance(item, schema.WebSearchItem): return item.title else: @@ -79,6 +81,8 @@ def _get_cross_source_text(item: AnyItem) -> str: elif title.startswith("Ask HN:"): title = title[7:].strip() return title + if isinstance(item, schema.PolymarketItem): + return item.title return get_item_text(item) @@ -198,6 +202,14 @@ def dedupe_hackernews( return dedupe_items(items, threshold) +def dedupe_polymarket( + items: List[schema.PolymarketItem], + threshold: float = 0.7, +) -> List[schema.PolymarketItem]: + """Dedupe Polymarket items.""" + return dedupe_items(items, threshold) + + def cross_source_link( *source_lists: List[AnyItem], threshold: float = 0.40, diff --git a/scripts/lib/env.py b/scripts/lib/env.py index 2c443c4..c04179a 100644 --- a/scripts/lib/env.py +++ b/scripts/lib/env.py @@ -254,6 +254,14 @@ def is_hackernews_available() -> bool: return True +def is_polymarket_available() -> bool: + """Check if Polymarket source is available. + + Always returns True - Gamma API is free, no key needed. + """ + return True + + def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]: """Get detailed X source status for UI decisions. diff --git a/scripts/lib/normalize.py b/scripts/lib/normalize.py index 2f356fc..d209f20 100644 --- a/scripts/lib/normalize.py +++ b/scripts/lib/normalize.py @@ -4,7 +4,7 @@ from typing import Any, Dict, List, TypeVar, Union from . import dates, schema -T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.HackerNewsItem) +T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.HackerNewsItem, schema.PolymarketItem) def filter_by_date_range( @@ -257,6 +257,50 @@ def normalize_hackernews_items( return normalized +def normalize_polymarket_items( + items: List[Dict[str, Any]], + from_date: str, + to_date: str, +) -> List[schema.PolymarketItem]: + """Normalize raw Polymarket items to schema. + + Args: + items: Raw Polymarket items from Gamma API + from_date: Start of date range + to_date: End of date range + + Returns: + List of PolymarketItem objects + """ + normalized = [] + + for i, item in enumerate(items): + engagement = schema.Engagement( + volume=item.get("volume24hr", 0.0), + liquidity=item.get("liquidity", 0.0), + ) + + date_str = item.get("date") + + normalized.append(schema.PolymarketItem( + id=f"PM{i+1}", + title=item.get("title", ""), + question=item.get("question", ""), + url=item.get("url", ""), + outcome_prices=item.get("outcome_prices", []), + outcomes_remaining=item.get("outcomes_remaining", 0), + price_movement=item.get("price_movement"), + date=date_str, + date_confidence="high", + engagement=engagement, + end_date=item.get("end_date"), + relevance=item.get("relevance", 0.5), + why_relevant=item.get("why_relevant", ""), + )) + + return normalized + + def items_to_dicts(items: List) -> List[Dict[str, Any]]: """Convert schema items to dicts for JSON serialization.""" return [item.to_dict() for item in items] diff --git a/scripts/lib/polymarket.py b/scripts/lib/polymarket.py new file mode 100644 index 0000000..879967a --- /dev/null +++ b/scripts/lib/polymarket.py @@ -0,0 +1,354 @@ +"""Polymarket prediction market search via Gamma API (free, no auth required). + +Uses gamma-api.polymarket.com for event/market discovery. +No API key needed - public read-only API with generous rate limits (350 req/10s). +""" + +import json +import math +import re +import sys +from concurrent.futures import ThreadPoolExecutor, as_completed +from typing import Any, Dict, List, Optional +from urllib.parse import quote_plus, urlencode + +from . import http + +GAMMA_SEARCH_URL = "https://gamma-api.polymarket.com/public-search" + +DEPTH_CONFIG = { + "quick": 5, + "default": 10, + "deep": 20, +} + + +def _log(msg: str): + """Log to stderr (only in TTY mode to avoid cluttering Claude Code output).""" + if sys.stderr.isatty(): + sys.stderr.write(f"[PM] {msg}\n") + sys.stderr.flush() + + +def _extract_core_subject(topic: str) -> str: + """Extract core subject from topic string. + + Strips common prefixes like 'last 7 days', 'what are people saying about', etc. + """ + topic = topic.strip() + # Remove common leading phrases + prefixes = [ + r"^last \d+ days?\s+", + r"^what(?:'s| is| are) (?:people saying about|happening with|going on with)\s+", + r"^how (?:is|are)\s+", + r"^tell me about\s+", + r"^research\s+", + ] + for pattern in prefixes: + topic = re.sub(pattern, "", topic, flags=re.IGNORECASE) + return topic.strip() + + +def _expand_queries(topic: str) -> List[str]: + """Generate 2-4 search queries to cast a wider net. + + Strategy: + - Always include the core subject + - Split multi-word topics into component searches + - Include the full topic if different from core + - Cap at 4 queries, dedupe + """ + core = _extract_core_subject(topic) + queries = [core] + + # Split multi-word topics into component searches + words = core.split() + if len(words) >= 2: + # Try the first significant word alone (e.g., "Arizona" from "Arizona Basketball") + queries.append(words[0]) + + # Add the full topic if different from core + if topic.lower().strip() != core.lower(): + queries.append(topic.strip()) + + # Dedupe while preserving order, cap at 4 + seen = set() + unique = [] + for q in queries: + q_lower = q.lower().strip() + if q_lower and q_lower not in seen: + seen.add(q_lower) + unique.append(q.strip()) + return unique[:4] + + +def _search_single_query(query: str, limit: int) -> Dict[str, Any]: + """Run a single search query against Gamma API.""" + params = { + "q": query, + "limit": str(limit), + } + url = f"{GAMMA_SEARCH_URL}?{urlencode(params)}" + + try: + response = http.request("GET", url, timeout=15, retries=2) + return response + except http.HTTPError as e: + _log(f"Search failed for '{query}': {e}") + return {"events": [], "error": str(e)} + except Exception as e: + _log(f"Search failed for '{query}': {e}") + return {"events": [], "error": str(e)} + + +def search_polymarket( + topic: str, + from_date: str, + to_date: str, + depth: str = "default", +) -> Dict[str, Any]: + """Search Polymarket via Gamma API with smart query expansion. + + Runs 2-4 expanded queries in parallel, merges and dedupes by event ID. + + Args: + topic: Search topic + from_date: Start date (YYYY-MM-DD) - used for activity filtering + to_date: End date (YYYY-MM-DD) + depth: 'quick', 'default', or 'deep' + + Returns: + Dict with 'events' list and optional 'error'. + """ + limit_per_query = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"]) + queries = _expand_queries(topic) + + _log(f"Searching for '{topic}' with queries: {queries} (limit={limit_per_query})") + + # Run all queries in parallel + all_events = {} # event_id -> (event_data, query_index) + errors = [] + + with ThreadPoolExecutor(max_workers=min(4, len(queries))) as executor: + futures = { + executor.submit(_search_single_query, q, limit_per_query): i + for i, q in enumerate(queries) + } + + for future in as_completed(futures): + query_idx = futures[future] + try: + response = future.result(timeout=15) + if response.get("error"): + errors.append(response["error"]) + + events = response.get("events", []) + for event in events: + event_id = event.get("id", "") + if not event_id: + continue + # Keep the first occurrence (from highest-priority query) + if event_id not in all_events: + all_events[event_id] = (event, query_idx) + elif query_idx < all_events[event_id][1]: + # Replace with higher-priority query result + all_events[event_id] = (event, query_idx) + except Exception as e: + errors.append(str(e)) + + # Sort by query priority, then by position + merged_events = [ev for ev, _ in sorted(all_events.values(), key=lambda x: x[1])] + _log(f"Found {len(merged_events)} unique events across {len(queries)} queries") + + result = {"events": merged_events} + if errors and not merged_events: + result["error"] = "; ".join(errors[:2]) + return result + + +def _format_price_movement(market: Dict[str, Any]) -> Optional[str]: + """Pick the most significant price change and format it. + + Returns string like 'down 11.7% this month' or None if no significant change. + """ + changes = [ + (abs(market.get("oneDayPriceChange") or 0), market.get("oneDayPriceChange"), "today"), + (abs(market.get("oneWeekPriceChange") or 0), market.get("oneWeekPriceChange"), "this week"), + (abs(market.get("oneMonthPriceChange") or 0), market.get("oneMonthPriceChange"), "this month"), + ] + + # Pick the largest absolute change + changes.sort(key=lambda x: x[0], reverse=True) + abs_change, raw_change, period = changes[0] + + # Skip if change is less than 1% (noise) + if abs_change < 0.01: + return None + + direction = "up" if raw_change > 0 else "down" + pct = abs_change * 100 + return f"{direction} {pct:.1f}% {period}" + + +def _parse_outcome_prices(market: Dict[str, Any]) -> List[tuple]: + """Parse outcomePrices JSON string into list of (outcome_name, price) tuples.""" + outcomes_raw = market.get("outcomes") or [] + prices_raw = market.get("outcomePrices") + + if not prices_raw: + return [] + + # Both outcomes and outcomePrices can be JSON-encoded strings + try: + if isinstance(outcomes_raw, str): + outcomes = json.loads(outcomes_raw) + else: + outcomes = outcomes_raw + except (json.JSONDecodeError, TypeError): + outcomes = [] + + try: + if isinstance(prices_raw, str): + prices = json.loads(prices_raw) + else: + prices = prices_raw + except (json.JSONDecodeError, TypeError): + return [] + + result = [] + for i, price in enumerate(prices): + try: + p = float(price) + except (ValueError, TypeError): + continue + name = outcomes[i] if i < len(outcomes) else f"Outcome {i+1}" + result.append((name, p)) + + return result + + +def parse_polymarket_response(response: Dict[str, Any], topic: str = "") -> List[Dict[str, Any]]: + """Parse Gamma API response into normalized item dicts. + + Each event becomes one item showing its title and top markets. + + Args: + response: Raw Gamma API response + topic: Original search topic (for relevance scoring) + + Returns: + List of item dicts ready for normalization. + """ + events = response.get("events", []) + items = [] + + for i, event in enumerate(events): + event_id = event.get("id", "") + title = event.get("title", "") + slug = event.get("slug", "") + + # Filter: skip closed/resolved events + if event.get("closed", False): + continue + if not event.get("active", True): + continue + + # Get markets for this event + markets = event.get("markets", []) + if not markets: + continue + + # Filter to active, open markets with liquidity (excludes resolved markets) + active_markets = [] + for m in markets: + if m.get("closed", False): + continue + if not m.get("active", True): + continue + # Must have liquidity (resolved markets have 0 or None) + try: + liq = float(m.get("liquidity", 0) or 0) + except (ValueError, TypeError): + liq = 0 + if liq > 0: + active_markets.append(m) + + if not active_markets: + continue + + # Sort markets by volume (most liquid first) + def market_volume(m): + try: + return float(m.get("volume", 0) or 0) + except (ValueError, TypeError): + return 0 + active_markets.sort(key=market_volume, reverse=True) + + # Take top market for the event + top_market = active_markets[0] + + # Parse outcome prices from top market + outcome_prices = _parse_outcome_prices(top_market) + + # Format price movement + price_movement = _format_price_movement(top_market) + + # Volume and liquidity + try: + volume24hr = float(top_market.get("volume24hr", 0) or 0) + except (ValueError, TypeError): + volume24hr = 0.0 + try: + liquidity = float(top_market.get("liquidity", 0) or 0) + except (ValueError, TypeError): + liquidity = 0.0 + + # Event URL + url = f"https://polymarket.com/event/{slug}" if slug else f"https://polymarket.com/event/{event_id}" + + # Date: use updatedAt from event + updated_at = event.get("updatedAt", "") + date_str = None + if updated_at: + # Parse ISO format: "2026-02-20T15:30:00.000Z" + try: + date_str = updated_at[:10] # YYYY-MM-DD + except (IndexError, TypeError): + pass + + # End date for the market + end_date = top_market.get("endDate") + if end_date: + try: + end_date = end_date[:10] + except (IndexError, TypeError): + end_date = None + + # Relevance: position-based decay + rank_score = max(0.3, 1.0 - (i * 0.03)) # 1.0 -> 0.3 over ~23 items + engagement_boost = min(0.15, math.log1p(volume24hr) / 60) + relevance = min(1.0, rank_score * 0.75 + engagement_boost + 0.1) + + # Top 3 outcomes for multi-outcome markets + top_outcomes = outcome_prices[:3] + remaining = len(outcome_prices) - 3 + if remaining < 0: + remaining = 0 + + items.append({ + "event_id": event_id, + "title": title, + "question": top_market.get("question", title), + "url": url, + "outcome_prices": top_outcomes, + "outcomes_remaining": remaining, + "price_movement": price_movement, + "volume24hr": volume24hr, + "liquidity": liquidity, + "date": date_str, + "end_date": end_date, + "relevance": round(relevance, 2), + "why_relevant": f"Prediction market: {title[:60]}", + }) + + return items diff --git a/scripts/lib/render.py b/scripts/lib/render.py index 7be613a..367c887 100644 --- a/scripts/lib/render.py +++ b/scripts/lib/render.py @@ -26,6 +26,8 @@ def _xref_tag(item) -> str: source_names.add('YouTube') elif ref_id.startswith('HN'): source_names.add('HN') + elif ref_id.startswith('PM'): + source_names.add('Polymarket') elif ref_id.startswith('W'): source_names.add('Web') if source_names: @@ -53,9 +55,10 @@ def _assess_data_freshness(report: schema.Report) -> dict: x_recent = sum(1 for x in report.x if x.date and x.date >= report.range_from) web_recent = sum(1 for w in report.web if w.date and w.date >= report.range_from) hn_recent = sum(1 for h in report.hackernews if h.date and h.date >= report.range_from) + pm_recent = sum(1 for p in report.polymarket if p.date and p.date >= report.range_from) - total_recent = reddit_recent + x_recent + web_recent + hn_recent - total_items = len(report.reddit) + len(report.x) + len(report.web) + len(report.hackernews) + total_recent = reddit_recent + x_recent + web_recent + hn_recent + pm_recent + total_items = len(report.reddit) + len(report.x) + len(report.web) + len(report.hackernews) + len(report.polymarket) return { "reddit_recent": reddit_recent, @@ -276,6 +279,59 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = " lines.append("") + # Polymarket items + if report.polymarket_error: + lines.append("### Prediction Markets (Polymarket)") + lines.append("") + lines.append(f"**ERROR:** {report.polymarket_error}") + lines.append("") + elif report.polymarket: + lines.append("### Prediction Markets (Polymarket)") + lines.append("") + for item in report.polymarket[:limit]: + eng_str = "" + if item.engagement: + eng = item.engagement + parts = [] + if eng.volume is not None: + if eng.volume >= 1_000_000: + parts.append(f"${eng.volume/1_000_000:.1f}M vol24h") + elif eng.volume >= 1_000: + parts.append(f"${eng.volume/1_000:.0f}K vol24h") + else: + parts.append(f"${eng.volume:.0f} vol24h") + if eng.liquidity is not None: + if eng.liquidity >= 1_000_000: + parts.append(f"${eng.liquidity/1_000_000:.1f}M liquidity") + elif eng.liquidity >= 1_000: + parts.append(f"${eng.liquidity/1_000:.0f}K liquidity") + else: + parts.append(f"${eng.liquidity:.0f} liquidity") + if parts: + eng_str = f" [{', '.join(parts)}]" + + date_str = f" ({item.date})" if item.date else "" + + lines.append(f"**{item.id}** (score:{item.score}){eng_str}{_xref_tag(item)}") + lines.append(f" {item.question}") + + # Outcome prices with price movement + if item.outcome_prices: + outcomes = [] + for name, price in item.outcome_prices: + pct = price * 100 + outcomes.append(f"{name}: {pct:.0f}%") + outcome_line = " | ".join(outcomes) + if item.outcomes_remaining > 0: + outcome_line += f" and {item.outcomes_remaining} more" + if item.price_movement: + outcome_line += f" ({item.price_movement})" + lines.append(f" {outcome_line}") + + lines.append(f" {item.url}") + lines.append(f" *{item.why_relevant}*") + lines.append("") + # Web items (if any - populated by the assistant) if report.web_error: lines.append("### Web Results") @@ -323,7 +379,7 @@ def render_source_status(report: schema.Report, source_info: dict = None) -> str elif report.reddit: lines.append(f" ✅ Reddit: {len(report.reddit)} threads") elif report.mode in ("both", "reddit-only", "all", "reddit-web"): - lines.append(" ⚠️ Reddit: 0 threads found") + pass # Hide zero-result sources else: reason = source_info.get("reddit_skip_reason", "not configured") lines.append(f" ⏭️ Reddit: skipped — {reason}") @@ -337,7 +393,7 @@ def render_source_status(report: schema.Report, source_info: dict = None) -> str x_line += f" (via @{report.resolved_x_handle} + keyword search)" lines.append(x_line) elif report.mode in ("both", "x-only", "all", "x-web"): - lines.append(" ⚠️ X: 0 posts found") + pass # Hide zero-result sources else: reason = source_info.get("x_skip_reason", "No Bird CLI or XAI_API_KEY") lines.append(f" ⏭️ X: skipped — {reason}") @@ -348,17 +404,21 @@ def render_source_status(report: schema.Report, source_info: dict = None) -> str elif report.youtube: with_transcripts = sum(1 for v in report.youtube if getattr(v, 'transcript_snippet', None)) lines.append(f" ✅ YouTube: {len(report.youtube)} videos ({with_transcripts} with transcripts)") - else: - reason = source_info.get("youtube_skip_reason", "yt-dlp not installed (brew install yt-dlp)") - lines.append(f" ⏭️ YouTube: skipped — {reason}") + # Hide when zero results (no skip reason line needed) # Hacker News if report.hackernews_error: lines.append(f" ❌ HN: error - {report.hackernews_error}") elif report.hackernews: lines.append(f" ✅ HN: {len(report.hackernews)} stories") - else: - lines.append(" ⏭️ HN: 0 stories found") + # Hide when zero results + + # Polymarket + if report.polymarket_error: + lines.append(f" ❌ Polymarket: error - {report.polymarket_error}") + elif report.polymarket: + lines.append(f" ✅ Polymarket: {len(report.polymarket)} markets") + # Hide when zero results # Web if report.web_error: @@ -399,6 +459,8 @@ def render_context_snippet(report: schema.Report) -> str: all_items.append((item.score, "X", item.text[:50] + "...", item.url)) for item in report.hackernews[:5]: all_items.append((item.score, "HN", item.title[:50] + "...", item.hn_url)) + for item in report.polymarket[:5]: + all_items.append((item.score, "Polymarket", item.question[:50] + "...", item.url)) for item in report.web[:5]: all_items.append((item.score, "Web", item.title[:50] + "...", item.url)) @@ -515,6 +577,29 @@ def render_full_report(report: schema.Report) -> str: lines.append("") + # Polymarket section + if report.polymarket: + lines.append("## Prediction Markets (Polymarket)") + lines.append("") + for item in report.polymarket: + lines.append(f"### {item.id}: {item.question}") + lines.append("") + lines.append(f"- **Event:** {item.title}") + lines.append(f"- **URL:** {item.url}") + lines.append(f"- **Date:** {item.date or 'Unknown'}") + lines.append(f"- **Score:** {item.score}/100") + + if item.outcome_prices: + outcomes = [f"{name}: {price*100:.0f}%" for name, price in item.outcome_prices] + lines.append(f"- **Outcomes:** {' | '.join(outcomes)}") + if item.price_movement: + lines.append(f"- **Trend:** {item.price_movement}") + if item.engagement: + eng = item.engagement + lines.append(f"- **Volume:** ${eng.volume or 0:,.0f} | Liquidity: ${eng.liquidity or 0:,.0f}") + + lines.append("") + # Web section if report.web: lines.append("## Web Results") diff --git a/scripts/lib/schema.py b/scripts/lib/schema.py index ee2912c..c990eef 100644 --- a/scripts/lib/schema.py +++ b/scripts/lib/schema.py @@ -22,6 +22,10 @@ class Engagement: # YouTube fields views: Optional[int] = None + # Polymarket fields + volume: Optional[float] = None + liquidity: Optional[float] = None + def to_dict(self) -> Dict[str, Any]: d = {} if self.score is not None: @@ -40,6 +44,10 @@ class Engagement: d['quotes'] = self.quotes if self.views is not None: d['views'] = self.views + if self.volume is not None: + d['volume'] = self.volume + if self.liquidity is not None: + d['liquidity'] = self.liquidity return d if d else None @@ -264,6 +272,49 @@ class HackerNewsItem: return d +@dataclass +class PolymarketItem: + """Normalized Polymarket prediction market item.""" + id: str # "PM1", "PM2", ... + title: str # Event title + question: str # Top market question + url: str # Event page URL + outcome_prices: List[tuple] = field(default_factory=list) # [(name, price), ...] + outcomes_remaining: int = 0 + price_movement: Optional[str] = None # "down 11.7% this month" + date: Optional[str] = None + date_confidence: str = "high" # API provides exact timestamps + engagement: Optional[Engagement] = None # volume + liquidity + end_date: Optional[str] = None + relevance: float = 0.5 + why_relevant: str = "" + subs: SubScores = field(default_factory=SubScores) + score: int = 0 + cross_refs: List[str] = field(default_factory=list) + + def to_dict(self) -> Dict[str, Any]: + d = { + 'id': self.id, + 'title': self.title, + 'question': self.question, + 'url': self.url, + 'outcome_prices': self.outcome_prices, + 'outcomes_remaining': self.outcomes_remaining, + 'price_movement': self.price_movement, + 'date': self.date, + 'date_confidence': self.date_confidence, + 'engagement': self.engagement.to_dict() if self.engagement else None, + 'end_date': self.end_date, + 'relevance': self.relevance, + 'why_relevant': self.why_relevant, + 'subs': self.subs.to_dict(), + 'score': self.score, + } + if self.cross_refs: + d['cross_refs'] = self.cross_refs + return d + + @dataclass class Report: """Full research report.""" @@ -279,6 +330,7 @@ class Report: web: List[WebSearchItem] = field(default_factory=list) youtube: List[YouTubeItem] = field(default_factory=list) hackernews: List[HackerNewsItem] = field(default_factory=list) + polymarket: List[PolymarketItem] = field(default_factory=list) best_practices: List[str] = field(default_factory=list) prompt_pack: List[str] = field(default_factory=list) context_snippet_md: str = "" @@ -288,6 +340,7 @@ class Report: web_error: Optional[str] = None youtube_error: Optional[str] = None hackernews_error: Optional[str] = None + polymarket_error: Optional[str] = None # Handle resolution resolved_x_handle: Optional[str] = None # Cache info @@ -310,6 +363,7 @@ class Report: 'web': [w.to_dict() for w in self.web], 'youtube': [y.to_dict() for y in self.youtube], 'hackernews': [h.to_dict() for h in self.hackernews], + 'polymarket': [p.to_dict() for p in self.polymarket], 'best_practices': self.best_practices, 'prompt_pack': self.prompt_pack, 'context_snippet_md': self.context_snippet_md, @@ -326,6 +380,8 @@ class Report: d['youtube_error'] = self.youtube_error if self.hackernews_error: d['hackernews_error'] = self.hackernews_error + if self.polymarket_error: + d['polymarket_error'] = self.polymarket_error if self.from_cache: d['from_cache'] = self.from_cache if self.cache_age_hours is not None: @@ -455,6 +511,32 @@ class Report: cross_refs=h.get('cross_refs', []), )) + # Reconstruct Polymarket items (backward compat: key may not exist) + pm_items = [] + for p in data.get('polymarket', []): + eng = None + if p.get('engagement'): + eng = Engagement(**p['engagement']) + subs = SubScores(**p.get('subs', {})) if p.get('subs') else SubScores() + pm_items.append(PolymarketItem( + id=p['id'], + title=p['title'], + question=p.get('question', ''), + url=p['url'], + outcome_prices=p.get('outcome_prices', []), + outcomes_remaining=p.get('outcomes_remaining', 0), + price_movement=p.get('price_movement'), + date=p.get('date'), + date_confidence=p.get('date_confidence', 'high'), + engagement=eng, + end_date=p.get('end_date'), + relevance=p.get('relevance', 0.5), + why_relevant=p.get('why_relevant', ''), + subs=subs, + score=p.get('score', 0), + cross_refs=p.get('cross_refs', []), + )) + return cls( topic=data['topic'], range_from=range_from, @@ -468,6 +550,7 @@ class Report: web=web_items, youtube=youtube_items, hackernews=hn_items, + polymarket=pm_items, best_practices=data.get('best_practices', []), prompt_pack=data.get('prompt_pack', []), context_snippet_md=data.get('context_snippet_md', ''), @@ -476,6 +559,7 @@ class Report: web_error=data.get('web_error'), youtube_error=data.get('youtube_error'), hackernews_error=data.get('hackernews_error'), + polymarket_error=data.get('polymarket_error'), resolved_x_handle=data.get('resolved_x_handle'), from_cache=data.get('from_cache', False), cache_age_hours=data.get('cache_age_hours'), diff --git a/scripts/lib/score.py b/scripts/lib/score.py index efca6d6..caae586 100644 --- a/scripts/lib/score.py +++ b/scripts/lib/score.py @@ -338,6 +338,64 @@ def score_hackernews_items(items: List[schema.HackerNewsItem]) -> List[schema.Ha return items +def compute_polymarket_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]: + """Compute raw engagement score for Polymarket item. + + Formula: 0.60*log1p(volume) + 0.40*log1p(liquidity) + Volume is the primary signal (money flowing); liquidity indicates market depth. + """ + if engagement is None: + return None + + if engagement.volume is None and engagement.liquidity is None: + return None + + volume = math.log1p(engagement.volume or 0) + liquidity = math.log1p(engagement.liquidity or 0) + + return 0.60 * volume + 0.40 * liquidity + + +def score_polymarket_items(items: List[schema.PolymarketItem]) -> List[schema.PolymarketItem]: + """Compute scores for Polymarket items. + + Uses same weight structure as Reddit/X (relevance + recency + engagement). + """ + if not items: + return items + + eng_raw = [compute_polymarket_engagement_raw(item.engagement) for item in items] + eng_normalized = normalize_to_100(eng_raw) + + for i, item in enumerate(items): + rel_score = int(item.relevance * 100) + rec_score = dates.recency_score(item.date) + + if eng_normalized[i] is not None: + eng_score = int(eng_normalized[i]) + else: + eng_score = DEFAULT_ENGAGEMENT + + item.subs = schema.SubScores( + relevance=rel_score, + recency=rec_score, + engagement=eng_score, + ) + + overall = ( + WEIGHT_RELEVANCE * rel_score + + WEIGHT_RECENCY * rec_score + + WEIGHT_ENGAGEMENT * eng_score + ) + + if eng_raw[i] is None: + overall -= UNKNOWN_ENGAGEMENT_PENALTY + + item.score = max(0, min(100, int(overall))) + + return items + + def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]: """Compute scores for WebSearch items WITHOUT engagement metrics. @@ -395,7 +453,7 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS return items -def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.HackerNewsItem]]) -> List: +def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.HackerNewsItem, schema.PolymarketItem]]) -> List: """Sort items by score (descending), then date, then source priority. Args: @@ -412,7 +470,7 @@ def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSear date = item.date or "0000-00-00" date_key = -int(date.replace("-", "")) - # Tertiary: source priority (Reddit > X > YouTube > HN > WebSearch) + # Tertiary: source priority (Reddit > X > YouTube > HN > Polymarket > WebSearch) if isinstance(item, schema.RedditItem): source_priority = 0 elif isinstance(item, schema.XItem): @@ -421,8 +479,10 @@ def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSear source_priority = 2 elif isinstance(item, schema.HackerNewsItem): source_priority = 3 - else: # WebSearchItem + elif isinstance(item, schema.PolymarketItem): source_priority = 4 + else: # WebSearchItem + source_priority = 5 # Quaternary: title/text for stability text = getattr(item, "title", "") or getattr(item, "text", "") diff --git a/scripts/lib/ui.py b/scripts/lib/ui.py index 3b9a209..ee88f9f 100644 --- a/scripts/lib/ui.py +++ b/scripts/lib/ui.py @@ -79,6 +79,13 @@ HN_MESSAGES = [ "Discovering developer conversations...", ] +POLYMARKET_MESSAGES = [ + "Checking prediction markets...", + "Finding what people are betting on...", + "Scanning Polymarket for odds...", + "Discovering prediction markets...", +] + PROCESSING_MESSAGES = [ "Crunching the data...", "Scoring and ranking...", @@ -274,6 +281,15 @@ class ProgressDisplay: if self.spinner: self.spinner.stop(f"{Colors.YELLOW}HN{Colors.RESET} Found {count} stories") + def start_polymarket(self): + msg = random.choice(POLYMARKET_MESSAGES) + self.spinner = Spinner(f"{Colors.GREEN}Polymarket{Colors.RESET} {msg}", Colors.GREEN, quiet=True) + self.spinner.start() + + def end_polymarket(self, count: int): + if self.spinner: + self.spinner.stop(f"{Colors.GREEN}Polymarket{Colors.RESET} Found {count} markets") + def start_processing(self): msg = random.choice(PROCESSING_MESSAGES) self.spinner = Spinner(f"{Colors.PURPLE}Processing{Colors.RESET} {msg}", Colors.PURPLE) @@ -283,7 +299,7 @@ class ProgressDisplay: if self.spinner: self.spinner.stop() - def show_complete(self, reddit_count: int, x_count: int, youtube_count: int = 0, hn_count: int = 0): + def show_complete(self, reddit_count: int, x_count: int, youtube_count: int = 0, hn_count: int = 0, pm_count: int = 0): elapsed = time.time() - self.start_time if IS_TTY: sys.stderr.write(f"\n{Colors.GREEN}{Colors.BOLD}✓ Research complete{Colors.RESET} ") @@ -294,6 +310,8 @@ class ProgressDisplay: sys.stderr.write(f" {Colors.RED}YouTube:{Colors.RESET} {youtube_count} videos") if hn_count: sys.stderr.write(f" {Colors.YELLOW}HN:{Colors.RESET} {hn_count} stories") + if pm_count: + sys.stderr.write(f" {Colors.GREEN}Polymarket:{Colors.RESET} {pm_count} markets") sys.stderr.write("\n\n") else: parts = [f"Reddit: {reddit_count} threads", f"X: {x_count} posts"] @@ -301,6 +319,8 @@ class ProgressDisplay: parts.append(f"YouTube: {youtube_count} videos") if hn_count: parts.append(f"HN: {hn_count} stories") + if pm_count: + parts.append(f"Polymarket: {pm_count} markets") sys.stderr.write(f"✓ Research complete ({elapsed:.1f}s) - {', '.join(parts)}\n") sys.stderr.flush() diff --git a/tests/test_cross_source.py b/tests/test_cross_source.py index 1e25805..70ecec7 100644 --- a/tests/test_cross_source.py +++ b/tests/test_cross_source.py @@ -116,6 +116,26 @@ class TestCrossSourceLink(unittest.TestCase): self.assertIn("HN1", web[0].cross_refs) self.assertIn("W1", hn[0].cross_refs) + def _make_pm(self, id, title, score=50): + item = schema.PolymarketItem(id=id, title=title, question="Q?", url="") + item.score = score + return item + + def test_polymarket_to_reddit_link(self): + reddit = [self._make_reddit("R1", "Will Arizona win the Big 12 Championship?")] + pm = [self._make_pm("PM1", "Will Arizona win the Big 12 Championship?")] + dedupe.cross_source_link(reddit, pm) + self.assertIn("PM1", reddit[0].cross_refs) + self.assertIn("R1", pm[0].cross_refs) + + def test_polymarket_multi_source(self): + reddit = [self._make_reddit("R1", "Iran nuclear deal prediction markets")] + hn = [self._make_hn("HN1", "Iran nuclear deal prediction markets")] + pm = [self._make_pm("PM1", "Iran nuclear deal prediction markets")] + dedupe.cross_source_link(reddit, hn, pm) + self.assertEqual(len(reddit[0].cross_refs), 2) + self.assertEqual(len(pm[0].cross_refs), 2) + class TestCrossRefsSchemaRoundTrip(unittest.TestCase): def test_reddit_roundtrip(self): diff --git a/tests/test_polymarket.py b/tests/test_polymarket.py new file mode 100644 index 0000000..f659cdb --- /dev/null +++ b/tests/test_polymarket.py @@ -0,0 +1,472 @@ +"""Tests for Polymarket prediction market source module.""" + +import json +import math +import sys +import unittest +from pathlib import Path + +# Add lib to path +sys.path.insert(0, str(Path(__file__).parent.parent / "scripts")) + +from lib import polymarket, normalize, schema, score + + +class TestExtractCoreSubject(unittest.TestCase): + def test_plain_topic(self): + self.assertEqual(polymarket._extract_core_subject("Arizona Basketball"), "Arizona Basketball") + + def test_strips_last_n_days(self): + self.assertEqual(polymarket._extract_core_subject("last 7 days Iran"), "Iran") + + def test_strips_what_are(self): + result = polymarket._extract_core_subject("what are people saying about Bitcoin") + self.assertEqual(result, "Bitcoin") + + def test_strips_tell_me_about(self): + result = polymarket._extract_core_subject("tell me about Ukraine") + self.assertEqual(result, "Ukraine") + + def test_strips_whitespace(self): + self.assertEqual(polymarket._extract_core_subject(" Iran "), "Iran") + + def test_empty_string(self): + self.assertEqual(polymarket._extract_core_subject(""), "") + + +class TestExpandQueries(unittest.TestCase): + def test_single_word(self): + queries = polymarket._expand_queries("Iran") + self.assertIn("Iran", queries) + # Single word: no split, just the core + self.assertEqual(len(queries), 1) + + def test_multi_word(self): + queries = polymarket._expand_queries("Arizona Basketball") + self.assertIn("Arizona Basketball", queries) + self.assertIn("Arizona", queries) + self.assertEqual(len(queries), 2) + + def test_with_prefix_stripped(self): + queries = polymarket._expand_queries("last 7 days Iran") + self.assertIn("Iran", queries) + # Full topic should also be included since it differs from core + self.assertIn("last 7 days Iran", queries) + + def test_deduplication(self): + queries = polymarket._expand_queries("Iran") + # Should not have duplicates + self.assertEqual(len(queries), len(set(q.lower() for q in queries))) + + def test_max_4_queries(self): + queries = polymarket._expand_queries("some really long topic with many words") + self.assertLessEqual(len(queries), 4) + + +class TestFormatPriceMovement(unittest.TestCase): + def test_significant_monthly_change(self): + market = { + "oneDayPriceChange": 0.005, + "oneWeekPriceChange": -0.02, + "oneMonthPriceChange": -0.117, + } + result = polymarket._format_price_movement(market) + self.assertEqual(result, "down 11.7% this month") + + def test_significant_weekly_change(self): + market = { + "oneDayPriceChange": 0.01, + "oneWeekPriceChange": 0.225, + "oneMonthPriceChange": 0.15, + } + result = polymarket._format_price_movement(market) + self.assertEqual(result, "up 22.5% this week") + + def test_significant_daily_change(self): + market = { + "oneDayPriceChange": -0.15, + "oneWeekPriceChange": 0.02, + "oneMonthPriceChange": 0.03, + } + result = polymarket._format_price_movement(market) + self.assertEqual(result, "down 15.0% today") + + def test_no_significant_change(self): + market = { + "oneDayPriceChange": 0.005, + "oneWeekPriceChange": -0.003, + "oneMonthPriceChange": 0.002, + } + result = polymarket._format_price_movement(market) + self.assertIsNone(result) + + def test_missing_fields(self): + result = polymarket._format_price_movement({}) + self.assertIsNone(result) + + def test_none_values(self): + market = { + "oneDayPriceChange": None, + "oneWeekPriceChange": None, + "oneMonthPriceChange": None, + } + result = polymarket._format_price_movement(market) + self.assertIsNone(result) + + +class TestParseOutcomePrices(unittest.TestCase): + def test_binary_market_json_strings(self): + market = { + "outcomes": '["Yes", "No"]', + "outcomePrices": '["0.65", "0.35"]', + } + result = polymarket._parse_outcome_prices(market) + self.assertEqual(len(result), 2) + self.assertEqual(result[0], ("Yes", 0.65)) + self.assertEqual(result[1], ("No", 0.35)) + + def test_list_inputs(self): + market = { + "outcomes": ["Yes", "No"], + "outcomePrices": ["0.70", "0.30"], + } + result = polymarket._parse_outcome_prices(market) + self.assertEqual(len(result), 2) + self.assertEqual(result[0], ("Yes", 0.70)) + + def test_multi_outcome(self): + market = { + "outcomes": '["Arizona", "Kansas", "Houston"]', + "outcomePrices": '["0.35", "0.22", "0.18"]', + } + result = polymarket._parse_outcome_prices(market) + self.assertEqual(len(result), 3) + self.assertEqual(result[0][0], "Arizona") + self.assertAlmostEqual(result[0][1], 0.35) + + def test_malformed_outcomes_json(self): + market = { + "outcomes": "not valid json", + "outcomePrices": '["0.50", "0.50"]', + } + result = polymarket._parse_outcome_prices(market) + # Should fall back to "Outcome N" names + self.assertEqual(len(result), 2) + self.assertEqual(result[0][0], "Outcome 1") + + def test_malformed_prices_json(self): + market = { + "outcomes": '["Yes", "No"]', + "outcomePrices": "not valid json", + } + result = polymarket._parse_outcome_prices(market) + self.assertEqual(result, []) + + def test_missing_prices(self): + market = {"outcomes": '["Yes", "No"]'} + result = polymarket._parse_outcome_prices(market) + self.assertEqual(result, []) + + def test_empty_outcomes(self): + market = { + "outcomes": "[]", + "outcomePrices": '["0.50", "0.50"]', + } + result = polymarket._parse_outcome_prices(market) + self.assertEqual(len(result), 2) + # Should use fallback names + self.assertEqual(result[0][0], "Outcome 1") + + +class TestParsePolymarketResponse(unittest.TestCase): + def setUp(self): + fixture_path = Path(__file__).parent.parent / "fixtures" / "polymarket_sample.json" + with open(fixture_path) as f: + self.sample = json.load(f) + + def test_parses_active_events(self): + items = polymarket.parse_polymarket_response(self.sample) + # Should include Arizona Big 12, Arizona NCAA, multi-outcome, and malformed + # Should exclude: closed/resolved event and no-liquidity event + titles = [item["title"] for item in items] + self.assertIn("Will Arizona win the Big 12 Championship?", titles) + self.assertIn("Will Arizona win the NCAA Tournament?", titles) + self.assertIn("Who will win the Big 12 Tournament?", titles) + + def test_filters_closed_events(self): + items = polymarket.parse_polymarket_response(self.sample) + titles = [item["title"] for item in items] + self.assertNotIn("Resolved Event (should be filtered)", titles) + + def test_filters_no_liquidity(self): + items = polymarket.parse_polymarket_response(self.sample) + titles = [item["title"] for item in items] + self.assertNotIn("Dead market (no liquidity)", titles) + + def test_item_fields(self): + items = polymarket.parse_polymarket_response(self.sample) + item = items[0] # Arizona Big 12 + self.assertEqual(item["event_id"], "evt-arizona-big12") + self.assertEqual(item["url"], "https://polymarket.com/event/arizona-big-12-championship") + self.assertIsNotNone(item["date"]) + self.assertIsNotNone(item["outcome_prices"]) + self.assertIsNotNone(item["volume24hr"]) + self.assertIsNotNone(item["liquidity"]) + + def test_outcome_prices_parsed(self): + items = polymarket.parse_polymarket_response(self.sample) + item = items[0] # Arizona Big 12 - binary + self.assertEqual(len(item["outcome_prices"]), 2) + self.assertEqual(item["outcome_prices"][0][0], "Yes") + self.assertAlmostEqual(item["outcome_prices"][0][1], 0.64) + + def test_multi_outcome_top3(self): + items = polymarket.parse_polymarket_response(self.sample) + multi = [i for i in items if i["title"] == "Who will win the Big 12 Tournament?"][0] + # Top 3 outcomes shown + self.assertEqual(len(multi["outcome_prices"]), 3) + # 5 total - 3 shown = 2 remaining + self.assertEqual(multi["outcomes_remaining"], 2) + + def test_price_movement(self): + items = polymarket.parse_polymarket_response(self.sample) + item = items[0] # Arizona Big 12 + # Weekly change (22.5%) is the most significant + self.assertEqual(item["price_movement"], "up 22.5% this week") + + def test_date_extraction(self): + items = polymarket.parse_polymarket_response(self.sample) + item = items[0] + self.assertEqual(item["date"], "2026-02-24") + + def test_relevance_range(self): + items = polymarket.parse_polymarket_response(self.sample) + for item in items: + self.assertGreaterEqual(item["relevance"], 0.0) + self.assertLessEqual(item["relevance"], 1.0) + + def test_empty_response(self): + items = polymarket.parse_polymarket_response({"events": []}) + self.assertEqual(items, []) + + def test_missing_events_key(self): + items = polymarket.parse_polymarket_response({}) + self.assertEqual(items, []) + + def test_malformed_prices_still_produces_item(self): + items = polymarket.parse_polymarket_response(self.sample) + malformed = [i for i in items if i["title"] == "Malformed prices"] + self.assertEqual(len(malformed), 1) + # Outcome prices should be empty due to malformed JSON + self.assertEqual(malformed[0]["outcome_prices"], []) + + def test_end_date_extraction(self): + items = polymarket.parse_polymarket_response(self.sample) + item = items[0] # Arizona Big 12 + self.assertEqual(item["end_date"], "2026-03-15") + + +class TestNormalizePolymarketItems(unittest.TestCase): + def test_normalize(self): + raw_items = [ + { + "event_id": "evt-1", + "title": "Test Market", + "question": "Will test pass?", + "url": "https://polymarket.com/event/test", + "outcome_prices": [("Yes", 0.75), ("No", 0.25)], + "outcomes_remaining": 0, + "price_movement": "up 5.0% this week", + "volume24hr": 100000.0, + "liquidity": 500000.0, + "date": "2026-02-20", + "end_date": "2026-03-01", + "relevance": 0.85, + "why_relevant": "Prediction market: Test Market", + } + ] + result = normalize.normalize_polymarket_items(raw_items, "2026-01-01", "2026-03-01") + self.assertEqual(len(result), 1) + self.assertIsInstance(result[0], schema.PolymarketItem) + self.assertEqual(result[0].id, "PM1") + self.assertEqual(result[0].title, "Test Market") + self.assertEqual(result[0].question, "Will test pass?") + self.assertEqual(result[0].date_confidence, "high") + self.assertEqual(result[0].engagement.volume, 100000.0) + self.assertEqual(result[0].engagement.liquidity, 500000.0) + self.assertEqual(result[0].price_movement, "up 5.0% this week") + + def test_normalize_multiple(self): + raw_items = [ + { + "event_id": f"evt-{i}", + "title": f"Market {i}", + "question": f"Question {i}?", + "url": f"https://polymarket.com/event/test-{i}", + "outcome_prices": [], + "outcomes_remaining": 0, + "volume24hr": 0.0, + "liquidity": 0.0, + "date": "2026-02-20", + "relevance": 0.5, + "why_relevant": f"Market {i}", + } + for i in range(3) + ] + result = normalize.normalize_polymarket_items(raw_items, "2026-01-01", "2026-03-01") + self.assertEqual(len(result), 3) + self.assertEqual(result[0].id, "PM1") + self.assertEqual(result[1].id, "PM2") + self.assertEqual(result[2].id, "PM3") + + +class TestScorePolymarketItems(unittest.TestCase): + def test_score_items(self): + items = [ + schema.PolymarketItem( + id="PM1", title="High volume", question="Q1?", url="", + date="2026-02-20", + engagement=schema.Engagement(volume=500000.0, liquidity=2000000.0), + relevance=0.9, + ), + schema.PolymarketItem( + id="PM2", title="Low volume", question="Q2?", url="", + date="2026-02-18", + engagement=schema.Engagement(volume=100.0, liquidity=500.0), + relevance=0.5, + ), + ] + scored = score.score_polymarket_items(items) + self.assertEqual(len(scored), 2) + # High engagement + high relevance should score higher + self.assertGreater(scored[0].score, scored[1].score) + + def test_score_empty(self): + result = score.score_polymarket_items([]) + self.assertEqual(result, []) + + def test_engagement_formula(self): + eng = schema.Engagement(volume=100000.0, liquidity=500000.0) + result = score.compute_polymarket_engagement_raw(eng) + self.assertIsNotNone(result) + self.assertGreater(result, 0) + # Verify formula: 0.60 * log1p(volume) + 0.40 * log1p(liquidity) + expected = 0.60 * math.log1p(100000.0) + 0.40 * math.log1p(500000.0) + self.assertAlmostEqual(result, expected) + + def test_engagement_none(self): + result = score.compute_polymarket_engagement_raw(None) + self.assertIsNone(result) + + def test_engagement_empty(self): + eng = schema.Engagement() + result = score.compute_polymarket_engagement_raw(eng) + self.assertIsNone(result) + + def test_zero_volume(self): + eng = schema.Engagement(volume=0.0, liquidity=0.0) + result = score.compute_polymarket_engagement_raw(eng) + self.assertIsNotNone(result) + self.assertEqual(result, 0.0) + + +class TestSortItemsWithPolymarket(unittest.TestCase): + def test_pm_priority_after_hn(self): + """Polymarket should sort after HN at same score.""" + hn_item = schema.HackerNewsItem(id="HN1", title="test", url="", hn_url="", author="user") + hn_item.score = 50 + + pm_item = schema.PolymarketItem(id="PM1", title="test", question="test?", url="") + pm_item.score = 50 + + web_item = schema.WebSearchItem(id="W1", title="test", url="", source_domain="example.com", snippet="") + web_item.score = 50 + + sorted_items = score.sort_items([web_item, pm_item, hn_item]) + # Same score, so sorted by priority: HN(3) > PM(4) > Web(5) + self.assertIsInstance(sorted_items[0], schema.HackerNewsItem) + self.assertIsInstance(sorted_items[1], schema.PolymarketItem) + self.assertIsInstance(sorted_items[2], schema.WebSearchItem) + + +class TestPolymarketSchemaRoundTrip(unittest.TestCase): + def test_to_dict_and_back(self): + item = schema.PolymarketItem( + id="PM1", + title="Test Event", + question="Will test pass?", + url="https://polymarket.com/event/test", + outcome_prices=[("Yes", 0.75), ("No", 0.25)], + outcomes_remaining=0, + price_movement="up 5.0% this week", + date="2026-02-20", + engagement=schema.Engagement(volume=100000.0, liquidity=500000.0), + end_date="2026-03-01", + relevance=0.85, + why_relevant="Test", + cross_refs=["R1", "HN2"], + ) + d = item.to_dict() + self.assertEqual(d["id"], "PM1") + self.assertEqual(d["title"], "Test Event") + self.assertEqual(d["cross_refs"], ["R1", "HN2"]) + + def test_empty_crossrefs_omitted(self): + item = schema.PolymarketItem(id="PM1", title="Test", question="Q?", url="") + d = item.to_dict() + self.assertNotIn("cross_refs", d) + + def test_report_roundtrip_with_polymarket(self): + report = schema.Report( + topic="test", range_from="2026-01-01", range_to="2026-02-01", + generated_at="2026-02-01T00:00:00Z", mode="both", + polymarket=[schema.PolymarketItem( + id="PM1", title="Test", question="Q?", url="", + outcome_prices=[("Yes", 0.5), ("No", 0.5)], + cross_refs=["R1"], + )], + ) + d = report.to_dict() + restored = schema.Report.from_dict(d) + self.assertEqual(len(restored.polymarket), 1) + self.assertEqual(restored.polymarket[0].id, "PM1") + self.assertEqual(restored.polymarket[0].cross_refs, ["R1"]) + + def test_report_backward_compat_no_polymarket(self): + """Old cached reports without polymarket key should load fine.""" + data = { + "topic": "test", + "range": {"from": "2026-01-01", "to": "2026-02-01"}, + "generated_at": "2026-02-01T00:00:00Z", + "mode": "both", + "reddit": [], + "x": [], + "web": [], + "youtube": [], + "hackernews": [], + # No "polymarket" key + } + report = schema.Report.from_dict(data) + self.assertEqual(report.polymarket, []) + self.assertIsNone(report.polymarket_error) + + def test_engagement_volume_liquidity(self): + eng = schema.Engagement(volume=342000.0, liquidity=2100000.0) + d = eng.to_dict() + self.assertEqual(d["volume"], 342000.0) + self.assertEqual(d["liquidity"], 2100000.0) + + +class TestDepthConfig(unittest.TestCase): + def test_quick_depth(self): + self.assertEqual(polymarket.DEPTH_CONFIG["quick"], 5) + + def test_default_depth(self): + self.assertEqual(polymarket.DEPTH_CONFIG["default"], 10) + + def test_deep_depth(self): + self.assertEqual(polymarket.DEPTH_CONFIG["deep"], 20) + + +if __name__ == "__main__": + unittest.main() diff --git a/variants/open/references/research.md b/variants/open/references/research.md index 74ff225..3a4d37f 100644 --- a/variants/open/references/research.md +++ b/variants/open/references/research.md @@ -1,6 +1,6 @@ # One-Shot Research Mode -Research ANY topic across Reddit, X, YouTube, and the web. Surface what people are actually discussing, recommending, and debating right now. +Research ANY topic across Reddit, X, YouTube, Polymarket, and the web. Surface what people are actually discussing, recommending, betting on, and debating right now. ## Parse User Intent @@ -19,7 +19,7 @@ Before doing anything, parse the user's input for: **Display your parsing** before calling tools: ``` -I'll research {TOPIC} across Reddit, X, YouTube, and the web. +I'll research {TOPIC} across Reddit, X, YouTube, Polymarket, and the web. Parsed intent: - TOPIC = {TOPIC} @@ -128,11 +128,14 @@ All agents reported back! |- Reddit: {N} threads | {N} upvotes | {N} comments |- X: {N} posts | {N} likes | {N} reposts |- YouTube: {N} videos | {N} views | {N} with transcripts +|- Polymarket: {N} markets | top odds: {outcome} {pct}% |- Web: {N} pages (supplementary) |- Top voices: @{handle1}, @{handle2} | r/{sub1}, r/{sub2} --- ``` +Omit any source line that returned 0 results. + **3. Invitation** with 2-3 specific follow-up suggestions based on research. ---