feat(polymarket): add Polymarket prediction markets as 6th research source
Search Polymarket's free Gamma API for relevant prediction markets on any
topic. Uses smart multi-query expansion to cast a wider net (e.g., "Arizona
Basketball" also searches "Arizona"), merges and dedupes by event ID, and
shows price movement context ("up 22.5% this week"). No API key required.
Also hides sources with zero results from the stats output (all sources).
54 new tests, all passing. Full pipeline integration with scoring, dedupe,
cross-source linking, and rendering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
@@ -1,18 +1,18 @@
|
|||||||
# /last30days v2.5
|
# /last30days v2.5
|
||||||
|
|
||||||
**The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, Hacker News, and the web from the last 30 days, finds what the community is actually upvoting, sharing, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know.
|
**The AI world reinvents itself every month. This skill keeps you current.** /last30days researches your topic across Reddit, X, YouTube, Hacker News, Polymarket, and the web from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations. Whether it's Seedance 2.0 access, paper.design prompts, or the latest Nano Banana Pro techniques, you'll know what people who are paying attention already know.
|
||||||
|
|
||||||
**New in V2.5 - dramatically better results:**
|
**New in V2.5 - dramatically better results:**
|
||||||
|
|
||||||
1. **Smarter scoring across the board.** New relevance scoring with synonym expansion ("hip hop" matches "rap", "MacBook" matches "Mac"), cross-source linking that flags when the same story trends on multiple platforms simultaneously, and X handle resolution that finds viral posts keyword search completely misses. A blinded evaluation scored v2.5 at 4.38/5.0 vs 3.73/5.0 for v1 across 5 test topics.
|
1. **Smarter scoring across the board.** New relevance scoring with synonym expansion ("hip hop" matches "rap", "MacBook" matches "Mac"), cross-source linking that flags when the same story trends on multiple platforms simultaneously, and X handle resolution that finds viral posts keyword search completely misses. A blinded evaluation scored v2.5 at 4.38/5.0 vs 3.73/5.0 for v1 across 5 test topics.
|
||||||
2. **Hacker News as a 5th source.** HN stories, Show HN posts, and comment threads are now searched, scored, and synthesized alongside Reddit, X, YouTube, and the web.
|
2. **Hacker News and Polymarket as new sources.** HN stories, Show HN posts, and prediction market odds are now searched, scored, and synthesized alongside Reddit, X, YouTube, and the web. Polymarket surfaces what people are putting real money on - betting odds reflect conviction, not just opinions.
|
||||||
3. **X handle resolution.** Search "Dor Brothers" and the skill resolves their handle (@thedorbrothers), then searches their posts directly - finding their 5,600-like viral tweet that keyword search missed entirely. Works for people, brands, products, and tools.
|
3. **X handle resolution.** Search "Dor Brothers" and the skill resolves their handle (@thedorbrothers), then searches their posts directly - finding their 5,600-like viral tweet that keyword search missed entirely. Works for people, brands, products, and tools.
|
||||||
|
|
||||||
**New in V2.1:** Open-class skill with watchlists, YouTube transcripts as a source, works in OpenAI Codex CLI. [Full changelog below.](#whats-new-in-v21)
|
**New in V2.1:** Open-class skill with watchlists, YouTube transcripts as a source, works in OpenAI Codex CLI. [Full changelog below.](#whats-new-in-v21)
|
||||||
|
|
||||||
**New in V2:** Smarter query construction, two-phase supplemental search, free X search via bundled Bird client, `--days=N` flag, automatic model fallback. [Full changelog below.](#whats-new-in-v2)
|
**New in V2:** Smarter query construction, two-phase supplemental search, free X search via bundled Bird client, `--days=N` flag, automatic model fallback. [Full changelog below.](#whats-new-in-v2)
|
||||||
|
|
||||||
**The tradeoff:** /last30days finds a lot of content but takes 2-8 minutes depending on how niche your topic is. Five sources searched in parallel, results scored, deduplicated, and synthesized. We think the depth is worth the wait, but `--quick` mode is there if you need speed over thoroughness.
|
**The tradeoff:** /last30days finds a lot of content but takes 2-8 minutes depending on how niche your topic is. Six sources searched in parallel, results scored, deduplicated, and synthesized. We think the depth is worth the wait, but `--quick` mode is there if you need speed over thoroughness.
|
||||||
|
|
||||||
**Best for prompt research**: discover what prompting techniques actually work for any tool (ChatGPT, Midjourney, Claude, Paper, etc.) by learning from real community discussions and best practices.
|
**Best for prompt research**: discover what prompting techniques actually work for any tool (ChatGPT, Midjourney, Claude, Paper, etc.) by learning from real community discussions and best practices.
|
||||||
|
|
||||||
@@ -122,7 +122,7 @@ Examples:
|
|||||||
|
|
||||||
## What It Does
|
## What It Does
|
||||||
|
|
||||||
1. **Researches** - Scans Reddit, X, and YouTube for discussions from the last 30 days
|
1. **Researches** - Scans Reddit, X, YouTube, Hacker News, Polymarket, and the web for discussions from the last 30 days
|
||||||
2. **Synthesizes** - Identifies patterns, best practices, and what actually works
|
2. **Synthesizes** - Identifies patterns, best practices, and what actually works
|
||||||
3. **Delivers** - Either writes copy-paste-ready prompts for your target tool, or gives you a curated expert-level answer
|
3. **Delivers** - Either writes copy-paste-ready prompts for your target tool, or gives you a curated expert-level answer
|
||||||
|
|
||||||
@@ -857,7 +857,7 @@ This example shows /last30days discovering **emerging developer workflows** - re
|
|||||||
- **Node.js 22+** - For X search (bundled Twitter GraphQL client)
|
- **Node.js 22+** - For X search (bundled Twitter GraphQL client)
|
||||||
- **X session** - Be logged into x.com in your browser, or set `AUTH_TOKEN`/`CT0` env vars
|
- **X session** - Be logged into x.com in your browser, or set `AUTH_TOKEN`/`CT0` env vars
|
||||||
- **xAI API key** (optional fallback) - If the bundled search can't authenticate, falls back to xAI's Grok API
|
- **xAI API key** (optional fallback) - If the bundled search can't authenticate, falls back to xAI's Grok API
|
||||||
- **yt-dlp** (optional) - For YouTube search + transcript extraction. Install via `brew install yt-dlp` or `pip install yt-dlp`. When present, automatically searches YouTube and extracts video transcripts as a 4th source.
|
- **yt-dlp** (optional) - For YouTube search + transcript extraction. Install via `brew install yt-dlp` or `pip install yt-dlp`. When present, automatically searches YouTube and extracts video transcripts as an additional source.
|
||||||
|
|
||||||
At least one API key is required. X search works automatically if you're logged into x.com in your browser. YouTube search activates automatically when yt-dlp is in your PATH.
|
At least one API key is required. X search works automatically if you're logged into x.com in your browser. YouTube search activates automatically when yt-dlp is in your PATH.
|
||||||
|
|
||||||
@@ -869,6 +869,8 @@ At least one API key is required. X search works automatically if you're logged
|
|||||||
- OpenAI Responses API with `web_search` tool scoped to reddit.com
|
- OpenAI Responses API with `web_search` tool scoped to reddit.com
|
||||||
- Vendored Twitter GraphQL search (or xAI API fallback) for X search
|
- Vendored Twitter GraphQL search (or xAI API fallback) for X search
|
||||||
- YouTube search + transcript extraction via yt-dlp (when installed)
|
- YouTube search + transcript extraction via yt-dlp (when installed)
|
||||||
|
- Hacker News search via Algolia API (free, no auth)
|
||||||
|
- Polymarket prediction market search via Gamma API (free, no auth)
|
||||||
- WebSearch for blogs, news, docs, tutorials
|
- WebSearch for blogs, news, docs, tutorials
|
||||||
- Reddit JSON enrichment for real engagement metrics (upvotes, comments)
|
- Reddit JSON enrichment for real engagement metrics (upvotes, comments)
|
||||||
- Scoring algorithm weighing recency, relevance, and engagement
|
- Scoring algorithm weighing recency, relevance, and engagement
|
||||||
@@ -913,7 +915,7 @@ Ran a 15-way blinded comparison across 5 topics (Claude Code, Seedance, MacBook
|
|||||||
|
|
||||||
Scored on groundedness (30%), specificity (25%), coverage (20%), actionability (15%), format (10%). The relative ranking is meaningful; absolute numbers are LLM-grading-LLM and shouldn't be taken as objective quality scores. The biggest gains came from detecting where sources agree - not just finding more sources.
|
Scored on groundedness (30%), specificity (25%), coverage (20%), actionability (15%), format (10%). The relative ranking is meaningful; absolute numbers are LLM-grading-LLM and shouldn't be taken as objective quality scores. The biggest gains came from detecting where sources agree - not just finding more sources.
|
||||||
|
|
||||||
### Hacker News as a 5th source
|
### Hacker News as a source
|
||||||
|
|
||||||
**The technical community's signal, captured automatically.** HN stories, Show HN posts, and Ask HN threads are searched, scored by points + comments, and synthesized alongside Reddit, X, YouTube, and the web. Comment insights are extracted from top threads to surface the technical community's actual take - not just headlines.
|
**The technical community's signal, captured automatically.** HN stories, Show HN posts, and Ask HN threads are searched, scored by points + comments, and synthesized alongside Reddit, X, YouTube, and the web. Comment insights are extracted from top threads to surface the technical community's actual take - not just headlines.
|
||||||
|
|
||||||
@@ -921,6 +923,14 @@ HN items go through the same scoring pipeline as every other source and particip
|
|||||||
|
|
||||||
Inspired by community PRs from [@ARJ999](https://github.com/ARJ999) ([#12](https://github.com/mvanhorn/last30days-skill/pull/12)) and [@wkbaran](https://github.com/wkbaran) ([#26](https://github.com/mvanhorn/last30days-skill/pull/26)), with [@gbessoni](https://github.com/gbessoni) endorsing HN as the right addition.
|
Inspired by community PRs from [@ARJ999](https://github.com/ARJ999) ([#12](https://github.com/mvanhorn/last30days-skill/pull/12)) and [@wkbaran](https://github.com/wkbaran) ([#26](https://github.com/mvanhorn/last30days-skill/pull/26)), with [@gbessoni](https://github.com/gbessoni) endorsing HN as the right addition.
|
||||||
|
|
||||||
|
### Polymarket prediction markets as a source
|
||||||
|
|
||||||
|
**What people are putting real money on.** Polymarket prediction markets are searched for any topic, surfacing betting odds and price movements alongside what people are saying on Reddit/X/YouTube/HN. Search "Iran" and you'll find markets on US strikes, Khamenei's future, and nuclear negotiations - with live odds and volume.
|
||||||
|
|
||||||
|
Uses smart multi-query expansion (same approach as YouTube synonym expansion and X handle resolution) to cast a wider net. "Arizona Basketball" finds markets on Big 12 title odds, NCAA tournament seeding, and March Madness outcomes - not just literal keyword matches.
|
||||||
|
|
||||||
|
No API key required - uses Polymarket's free public Gamma API. Sources with zero results are automatically hidden from the stats output.
|
||||||
|
|
||||||
### X handle resolution details
|
### X handle resolution details
|
||||||
|
|
||||||
The problem: when you search a topic on X, you find posts *about* it. But the topic's own account often doesn't mention its own name in tweets. Keyword search can't find those posts.
|
The problem: when you search a topic on X, you find posts *about* it. But the topic's own account often doesn't mention its own name in tweets. Keyword search can't find those posts.
|
||||||
@@ -1014,6 +1024,7 @@ Thanks to the contributors who helped shape V2:
|
|||||||
| Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY |
|
| Twitter GraphQL / `api.x.ai` | Search query | Browser cookies or XAI_API_KEY |
|
||||||
| `youtube.com` (via yt-dlp) | Search query | None (public search) |
|
| `youtube.com` (via yt-dlp) | Search query | None (public search) |
|
||||||
| `hn.algolia.com` | Search query | None (public API) |
|
| `hn.algolia.com` | Search query | None (public API) |
|
||||||
|
| `gamma-api.polymarket.com` | Search query | None (public API) |
|
||||||
| `api.search.brave.com` | Search query (optional) | BRAVE_API_KEY |
|
| `api.search.brave.com` | Search query (optional) | BRAVE_API_KEY |
|
||||||
| `api.parallel.ai` | Search query (optional) | PARALLEL_API_KEY |
|
| `api.parallel.ai` | Search query (optional) | PARALLEL_API_KEY |
|
||||||
| `openrouter.ai` | Search query (optional) | OPENROUTER_API_KEY |
|
| `openrouter.ai` | Search query (optional) | OPENROUTER_API_KEY |
|
||||||
@@ -1032,6 +1043,6 @@ Each API key is transmitted only to its respective endpoint. Your OpenAI key is
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
*30 days of research. 30 seconds of work. Four sources. Zero stale prompts.*
|
*30 days of research. 30 seconds of work. Six sources. Zero stale prompts.*
|
||||||
|
|
||||||
*Pair with [Open Claw](https://github.com/openclaw/openclaw) for automated watchlists and briefings. Reddit. X. YouTube. Web. - All synthesized into expert answers and copy-paste prompts.*
|
*Pair with [Open Claw](https://github.com/openclaw/openclaw) for automated watchlists and briefings. Reddit. X. YouTube. Web. - All synthesized into expert answers and copy-paste prompts.*
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
name: last30days
|
name: last30days
|
||||||
version: "2.1"
|
version: "2.1"
|
||||||
description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, Hacker News, web. Become an expert and write copy-paste-ready prompts."
|
description: "Research a topic from the last 30 days. Also triggered by 'last30'. Sources: Reddit, X, YouTube, Hacker News, Polymarket, web. Become an expert and write copy-paste-ready prompts."
|
||||||
argument-hint: 'last30 AI video tools, last30 best project management tools'
|
argument-hint: 'last30 AI video tools, last30 best project management tools'
|
||||||
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
|
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
|
||||||
homepage: https://github.com/mvanhorn/last30days-skill
|
homepage: https://github.com/mvanhorn/last30days-skill
|
||||||
@@ -32,7 +32,7 @@ metadata:
|
|||||||
|
|
||||||
# last30days v2.1: Research Any Topic from the Last 30 Days
|
# last30days v2.1: Research Any Topic from the Last 30 Days
|
||||||
|
|
||||||
Research ANY topic across Reddit, X, YouTube, Hacker News, and the web. Surface what people are actually discussing, recommending, and debating right now.
|
Research ANY topic across Reddit, X, YouTube, Hacker News, Polymarket, and the web. Surface what people are actually discussing, recommending, betting on, and debating right now.
|
||||||
|
|
||||||
## CRITICAL: Parse User Intent
|
## CRITICAL: Parse User Intent
|
||||||
|
|
||||||
@@ -141,10 +141,10 @@ Use a **timeout of 300000** (5 minutes) on the Bash call. The script typically t
|
|||||||
|
|
||||||
The script will automatically:
|
The script will automatically:
|
||||||
- Detect available API keys
|
- Detect available API keys
|
||||||
- Run Reddit/X/YouTube/Hacker News searches
|
- Run Reddit/X/YouTube/Hacker News/Polymarket searches
|
||||||
- Output ALL results including YouTube transcripts and HN comments
|
- Output ALL results including YouTube transcripts, HN comments, and prediction market odds
|
||||||
|
|
||||||
**Read the ENTIRE output.** It contains FIVE data sections in this order: Reddit items, X items, YouTube items, Hacker News items, and WebSearch items. If you miss sections, you will produce incomplete stats.
|
**Read the ENTIRE output.** It contains SIX data sections in this order: Reddit items, X items, YouTube items, Hacker News items, Polymarket items, and WebSearch items. If you miss sections, you will produce incomplete stats.
|
||||||
|
|
||||||
**YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, and optional transcript snippet. Count them and include them in your synthesis and stats block.
|
**YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, and optional transcript snippet. Count them and include them in your synthesis and stats block.
|
||||||
|
|
||||||
@@ -339,14 +339,13 @@ KEY PATTERNS from the research:
|
|||||||
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
|
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
|
||||||
├─ 🔴 YouTube: {N} videos │ {N} views │ {N} with transcripts
|
├─ 🔴 YouTube: {N} videos │ {N} views │ {N} with transcripts
|
||||||
├─ 🟡 HN: {N} stories │ {N} points │ {N} comments
|
├─ 🟡 HN: {N} stories │ {N} points │ {N} comments
|
||||||
|
├─ 📈 Polymarket: {N} markets │ ${vol} volume │ top odds: {outcome} {pct}%
|
||||||
├─ 🌐 Web: {N} pages (supplementary)
|
├─ 🌐 Web: {N} pages (supplementary)
|
||||||
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
|
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
|
||||||
---
|
---
|
||||||
```
|
```
|
||||||
|
|
||||||
If Reddit returned 0 threads, write: "├─ 🟠 Reddit: 0 threads (no results this cycle)"
|
Omit any source line that returned 0 results. Do NOT show "0 threads" or "0 stories" lines.
|
||||||
If HN returned 0 stories, write: "├─ 🟡 HN: 0 stories (no results this cycle)"
|
|
||||||
If YouTube returned 0 videos or yt-dlp is not installed, omit the YouTube line entirely.
|
|
||||||
NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
|
NEVER use plain text dashes (-) or pipe (|). ALWAYS use ├─ └─ │ and the emoji.
|
||||||
|
|
||||||
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If you catch yourself projecting your own knowledge instead of the research, rewrite it.
|
**SELF-CHECK before displaying**: Re-read your "What I learned" section. Does it match what the research ACTUALLY says? If you catch yourself projecting your own knowledge instead of the research, rewrite it.
|
||||||
@@ -521,6 +520,7 @@ Want another prompt? Just tell me what you're creating next.
|
|||||||
- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery
|
- Sends search queries to OpenAI's Responses API (`api.openai.com`) for Reddit discovery
|
||||||
- Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
|
- Sends search queries to Twitter's GraphQL API (via browser cookie auth) or xAI's API (`api.x.ai`) for X search
|
||||||
- Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)
|
- Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)
|
||||||
|
- Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth)
|
||||||
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
|
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
|
||||||
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
|
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
|
||||||
- Fetches public Reddit thread data from `reddit.com` for engagement metrics
|
- Fetches public Reddit thread data from `reddit.com` for engagement metrics
|
||||||
@@ -532,7 +532,7 @@ Want another prompt? Just tell me what you're creating next.
|
|||||||
- Does not share API keys between providers (OpenAI key only goes to api.openai.com, etc.)
|
- Does not share API keys between providers (OpenAI key only goes to api.openai.com, etc.)
|
||||||
- Does not log, cache, or write API keys to output files
|
- Does not log, cache, or write API keys to output files
|
||||||
- Does not send data to any endpoint not listed above
|
- Does not send data to any endpoint not listed above
|
||||||
- Hacker News source is always available (no API key, no binary dependency)
|
- Hacker News and Polymarket sources are always available (no API key, no binary dependency)
|
||||||
- Cannot be invoked autonomously by the agent (`disable-model-invocation: true`)
|
- Cannot be invoked autonomously by the agent (`disable-model-invocation: true`)
|
||||||
|
|
||||||
**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
|
**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
|
||||||
|
|||||||
@@ -18,6 +18,8 @@ The orchestrator (`last30days.py`) coordinates discovery, enrichment, normalizat
|
|||||||
- **openai_reddit.py**: OpenAI Responses API + web_search for Reddit
|
- **openai_reddit.py**: OpenAI Responses API + web_search for Reddit
|
||||||
- **xai_x.py**: xAI Responses API + x_search for X
|
- **xai_x.py**: xAI Responses API + x_search for X
|
||||||
- **reddit_enrich.py**: Fetch Reddit thread JSON for real engagement metrics
|
- **reddit_enrich.py**: Fetch Reddit thread JSON for real engagement metrics
|
||||||
|
- **hackernews.py**: Hacker News search via Algolia API (free, no auth)
|
||||||
|
- **polymarket.py**: Polymarket prediction market search via Gamma API (free, no auth)
|
||||||
- **normalize.py**: Convert raw API responses to canonical schema
|
- **normalize.py**: Convert raw API responses to canonical schema
|
||||||
- **score.py**: Compute popularity-aware scores (relevance + recency + engagement)
|
- **score.py**: Compute popularity-aware scores (relevance + recency + engagement)
|
||||||
- **dedupe.py**: Near-duplicate detection via text similarity
|
- **dedupe.py**: Near-duplicate detection via text similarity
|
||||||
|
|||||||
@@ -0,0 +1,254 @@
|
|||||||
|
---
|
||||||
|
title: "feat: Add Polymarket prediction market search as 6th source"
|
||||||
|
type: feat
|
||||||
|
status: completed
|
||||||
|
date: 2026-02-25
|
||||||
|
---
|
||||||
|
|
||||||
|
# feat: Add Polymarket Prediction Market Search
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Add Polymarket as a 6th research source to last30days. When researching any topic, search Polymarket's public API for relevant prediction markets - surfacing what people are putting real money on alongside what they're saying on Reddit/X/YouTube/HN/Web.
|
||||||
|
|
||||||
|
Example: "/last30days Arizona Basketball" would find markets on tournament seeding, Big 12 title odds, and March Madness outcomes. The signal is unique - betting odds reflect conviction, not just opinions.
|
||||||
|
|
||||||
|
## Proposed Solution
|
||||||
|
|
||||||
|
Hit Polymarket's Gamma API with **smart multi-query search** - not just the raw topic, but expanded related keywords to find markets a human researcher would think of. Show price movement context ("down from 23.7% peak") using the API's built-in price change fields. No API key needed.
|
||||||
|
|
||||||
|
Also: hide sources with zero results from the stats box (all sources, not just Polymarket).
|
||||||
|
|
||||||
|
## Technical Approach
|
||||||
|
|
||||||
|
### API Details
|
||||||
|
|
||||||
|
**Endpoint:** `GET https://gamma-api.polymarket.com/public-search?q={topic}&limit={N}`
|
||||||
|
|
||||||
|
- No authentication required (free, public, read-only)
|
||||||
|
- Rate limit: 350 req/10s (very generous)
|
||||||
|
- Returns `{"events": [...], "pagination": {...}}`
|
||||||
|
- Events contain nested `markets` arrays
|
||||||
|
- `outcomePrices` is a JSON-encoded string - must `json.loads()` it
|
||||||
|
- `volume` and `liquidity` are strings at market level, floats at event level
|
||||||
|
- **Price movement fields on every market:** `oneDayPriceChange`, `oneWeekPriceChange`, `oneMonthPriceChange` - these are free, no extra API calls
|
||||||
|
|
||||||
|
### Intelligence Layer: Smart Query Expansion
|
||||||
|
|
||||||
|
A single keyword search is dumb. "Iran" should find markets about Iran strikes, nuclear program, sanctions, oil prices, Khamenei. "Arizona Basketball" should find NCAA tournament odds, Big 12 title, March Madness seeding.
|
||||||
|
|
||||||
|
**How it works:**
|
||||||
|
|
||||||
|
1. **Extract core subject** using `_extract_core_subject(topic)` (already exists for YouTube/X)
|
||||||
|
2. **Generate 2-4 expanded queries** from the topic:
|
||||||
|
- The raw topic itself: `"Arizona Basketball"`
|
||||||
|
- Broader context: `"Arizona NCAA"`, `"Arizona Big 12"`
|
||||||
|
- For people/entities: add key associated terms (e.g., "Iran" -> "Iran strikes", "Khamenei", "Iran nuclear")
|
||||||
|
3. **Run all queries in parallel** against `/public-search` (rate limit is 350/10s, we're using 3-4)
|
||||||
|
4. **Merge and dedupe** results across queries (same event ID = same market)
|
||||||
|
5. **Score by relevance to original topic** - markets found by the raw topic query get a relevance boost over expanded-query matches
|
||||||
|
|
||||||
|
**Query expansion strategy:**
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _expand_polymarket_queries(topic: str) -> List[str]:
|
||||||
|
"""Generate 2-4 search queries to cast a wider net."""
|
||||||
|
core = _extract_core_subject(topic)
|
||||||
|
queries = [core] # Always include the core topic
|
||||||
|
|
||||||
|
# Split multi-word topics into component searches
|
||||||
|
words = core.split()
|
||||||
|
if len(words) >= 2:
|
||||||
|
# Try the first significant word alone (e.g., "Arizona" from "Arizona Basketball")
|
||||||
|
queries.append(words[0])
|
||||||
|
|
||||||
|
# Add the full topic if different from core
|
||||||
|
if topic.lower() != core.lower():
|
||||||
|
queries.append(topic)
|
||||||
|
|
||||||
|
return list(dict.fromkeys(queries))[:4] # Dedupe, cap at 4
|
||||||
|
```
|
||||||
|
|
||||||
|
This is the same approach as YouTube synonym expansion and X handle resolution - cast a wider net, then score by relevance.
|
||||||
|
|
||||||
|
### Price Movement Context
|
||||||
|
|
||||||
|
The API gives us price change data for free. Use it to make the output actually useful:
|
||||||
|
|
||||||
|
```
|
||||||
|
Will Arizona win the NCAA Tournament?
|
||||||
|
Yes: 12% (down 11.7% this month) | No: 88%
|
||||||
|
$342K vol24h | $2.1M liquidity
|
||||||
|
```
|
||||||
|
|
||||||
|
**Fields available:**
|
||||||
|
- `oneDayPriceChange` - "up 3% today"
|
||||||
|
- `oneWeekPriceChange` - "down 5% this week"
|
||||||
|
- `oneMonthPriceChange` - "down 11.7% this month"
|
||||||
|
|
||||||
|
Show the most significant movement (largest absolute change). Only show if change > 1% to avoid noise.
|
||||||
|
|
||||||
|
### Key Design Decisions
|
||||||
|
|
||||||
|
**Event-level granularity (not market-level).** Each event becomes one `PolymarketItem` showing its title and top 3 markets by volume. A "Trump" query returns 10 events, not 40+ individual markets. This keeps item counts manageable.
|
||||||
|
|
||||||
|
**Smart multi-query search with dedup.** Don't rely on one keyword search. Expand the topic into 2-4 related queries, run them all, merge by event ID. This catches markets that use different terminology than the user's query.
|
||||||
|
|
||||||
|
**Price movement context by default.** Show the most significant price change (day/week/month) inline with outcome prices. This makes prediction markets actually useful as a research signal - not just "Yes 65%" but "Yes 65%, up 12% this week."
|
||||||
|
|
||||||
|
**Date filtering uses `active=true` + `volume1mo > 0`.** Markets can be created months ago but still actively traded. Filter by recent trading activity, not creation date. Set `date` to `updatedAt` for recency scoring. Set `date_confidence` to "high" (API provides exact timestamps).
|
||||||
|
|
||||||
|
**Sort priority: below HN, above WebSearch.** Priority 4 (Reddit=0 > X=1 > YouTube=2 > HN=3 > Polymarket=4 > WebSearch=5). Prediction markets are supplementary signal.
|
||||||
|
|
||||||
|
**Multi-outcome markets: show top 3 outcomes by price.** Binary markets show "Yes: 65%, No: 35%". Multi-candidate markets truncate to top 3 with "and N more".
|
||||||
|
|
||||||
|
**Exclude closed/resolved markets for v1.** Only show `active=true, closed=false`.
|
||||||
|
|
||||||
|
**Link to event pages** (not individual market pages) - shows all related markets in context.
|
||||||
|
|
||||||
|
### Implementation Plan
|
||||||
|
|
||||||
|
#### Phase 1: Hide zero-result sources (separate commit)
|
||||||
|
|
||||||
|
- [x] `scripts/lib/render.py` - In `render_source_status()`, skip lines where count is 0
|
||||||
|
- [x] `SKILL.md` - Remove "(no results this cycle)" instructions, replace with "omit sources with zero results"
|
||||||
|
- [x] `variants/open/references/research.md` - Same update
|
||||||
|
- [x] `~/.claude/skills/last30daysCROSS/SKILL.md` - Same update (via sync.sh)
|
||||||
|
- [x] Test: run a query where HN returns 0, verify it's hidden
|
||||||
|
|
||||||
|
#### Phase 2: Polymarket source module
|
||||||
|
|
||||||
|
- [x] `scripts/lib/polymarket.py` - New file:
|
||||||
|
- `DEPTH_CONFIG = {"quick": 5, "default": 10, "deep": 20}` (event count per query)
|
||||||
|
- `_expand_queries(topic)` - generate 2-4 search queries from topic:
|
||||||
|
- Raw topic: `"Arizona Basketball"`
|
||||||
|
- Core subject extracted: `"Arizona"` (via `_extract_core_subject`)
|
||||||
|
- Component words for multi-word topics: first significant word alone
|
||||||
|
- Cap at 4 queries, dedupe
|
||||||
|
- `search_polymarket(topic, from_date, to_date, depth)` - run all expanded queries against `/public-search` in parallel (ThreadPoolExecutor), merge results by event ID, dedupe
|
||||||
|
- `parse_polymarket_response(response)` - extract events, flatten top markets per event
|
||||||
|
- `_format_price_movement(market)` - pick most significant price change from `oneDayPriceChange`, `oneWeekPriceChange`, `oneMonthPriceChange`, format as "up/down X% this day/week/month", skip if < 1%
|
||||||
|
- URL-encode query params with `urllib.parse.urlencode`
|
||||||
|
- Filter: `active=true, closed=false, volume1mo > 0`
|
||||||
|
- Parse `outcomePrices` with `json.loads()`, handle malformed/missing gracefully
|
||||||
|
- Handle mixed types: `float(volume)` with try/except
|
||||||
|
- Relevance scoring: markets from raw topic query get 1.0 base, expanded queries get 0.7 base, then decay by position
|
||||||
|
- Return `{"items": [...]}` or `{"items": [], "error": "message"}`
|
||||||
|
|
||||||
|
#### Phase 3: Schema + normalization + scoring
|
||||||
|
|
||||||
|
- [x] `scripts/lib/schema.py` - Add `PolymarketItem` dataclass:
|
||||||
|
- `id`, `title` (event title), `question` (top market question), `url` (event URL)
|
||||||
|
- `outcome_prices: List[Tuple[str, float]]` (parsed, e.g. [("Yes", 0.65), ("No", 0.35)])
|
||||||
|
- `price_movement: Optional[str]` (formatted, e.g. "down 11.7% this month")
|
||||||
|
- `volume24hr: float`, `liquidity: float`, `end_date: Optional[str]`
|
||||||
|
- Standard fields: `date`, `date_confidence`, `engagement`, `relevance`, `why_relevant`, `subs`, `score`, `cross_refs`
|
||||||
|
- Add `volume: Optional[float]` and `liquidity: Optional[float]` to `Engagement` dataclass
|
||||||
|
- Add `polymarket: List[PolymarketItem]` and `polymarket_error: Optional[str]` to `Report`
|
||||||
|
- Update `Report.to_dict()` and `Report.from_dict()` (handle missing key for backward compat)
|
||||||
|
|
||||||
|
- [x] `scripts/lib/normalize.py` - Add `normalize_polymarket_items()`:
|
||||||
|
- Map raw API events to `PolymarketItem` instances
|
||||||
|
- Set `date` from `updatedAt`, `date_confidence = "high"`
|
||||||
|
- Build `Engagement(volume=volume24hr, liquidity=liquidity)`
|
||||||
|
- Update `TypeVar` to include `PolymarketItem`
|
||||||
|
|
||||||
|
- [x] `scripts/lib/score.py` - Add scoring:
|
||||||
|
- `compute_polymarket_engagement_raw()`: `0.60 * log1p(volume24hr) + 0.40 * log1p(liquidity)`
|
||||||
|
- `score_polymarket_items()`: standard 45/25/30 weights (relevance/recency/engagement)
|
||||||
|
- Update `sort_items()`: Polymarket priority = 4
|
||||||
|
|
||||||
|
#### Phase 4: Dedupe + render + UI
|
||||||
|
|
||||||
|
- [x] `scripts/lib/dedupe.py`:
|
||||||
|
- Add `PolymarketItem` to `AnyItem` union
|
||||||
|
- Add `dedupe_polymarket()` function
|
||||||
|
- Update `get_item_text()` for `PolymarketItem` (return `title + " " + question`)
|
||||||
|
- Update `_get_cross_source_text()` for cross-source linking
|
||||||
|
|
||||||
|
- [x] `scripts/lib/render.py`:
|
||||||
|
- Add `_xref_tag()` prefix: `PM` -> `Polymarket`
|
||||||
|
- Add Polymarket section to `render_compact()`:
|
||||||
|
```
|
||||||
|
### Prediction Markets (Polymarket)
|
||||||
|
|
||||||
|
**PM1** (score:72) [$342K vol24h, $2.1M liquidity]
|
||||||
|
Will Arizona win the NCAA Tournament?
|
||||||
|
Yes: 12% (down 11.7% this month) | No: 88%
|
||||||
|
https://polymarket.com/event/arizona-ncaa-tournament
|
||||||
|
|
||||||
|
**PM2** (score:65) [$89K vol24h, $450K liquidity]
|
||||||
|
Will Arizona win the Big 12 Tournament?
|
||||||
|
Yes: 28% (up 4% this week) | No: 72%
|
||||||
|
https://polymarket.com/event/arizona-big-12
|
||||||
|
```
|
||||||
|
- Add to `render_source_status()` (with zero-result hiding from Phase 1)
|
||||||
|
- Add to `render_full_report()` and `render_context_snippet()`
|
||||||
|
- Add to `_assess_data_freshness()`
|
||||||
|
|
||||||
|
- [x] `scripts/lib/ui.py`:
|
||||||
|
- Add `POLYMARKET_MESSAGES` list
|
||||||
|
- Add `start_polymarket()` / `end_polymarket()` methods
|
||||||
|
- Update `show_complete()` to accept `polymarket_count`
|
||||||
|
|
||||||
|
#### Phase 5: Wire into main pipeline
|
||||||
|
|
||||||
|
- [x] `scripts/last30days.py`:
|
||||||
|
- Import `from lib import polymarket`
|
||||||
|
- Add `_search_polymarket()` wrapper function
|
||||||
|
- Add `polymarket_future` to `TIMEOUT_PROFILES` (15s - API is fast)
|
||||||
|
- In `run_research()`: submit polymarket future to ThreadPoolExecutor, increment max_workers
|
||||||
|
- Collect results with timeout, add to processing pipeline (normalize -> filter -> score -> sort -> dedupe)
|
||||||
|
- Feed into `cross_source_link()` call
|
||||||
|
- Assign to `report.polymarket` / `report.polymarket_error`
|
||||||
|
- Update `--diagnose` output: `"polymarket": True` (always available)
|
||||||
|
- Update `--store` persistence loop for Polymarket items
|
||||||
|
- Update `show_complete()` call with `len(deduped_pm)`
|
||||||
|
|
||||||
|
- [x] `scripts/lib/env.py` - Add `is_polymarket_available()` -> always `True`
|
||||||
|
|
||||||
|
#### Phase 6: Documentation + deploy
|
||||||
|
|
||||||
|
- [x] `SKILL.md` - Update "6 sources", add Polymarket to stats box template, update Security section with `gamma-api.polymarket.com`
|
||||||
|
- [x] `variants/open/references/research.md` - Same source count + stats updates
|
||||||
|
- [x] `README.md` - Add Polymarket to feature list, update source count references
|
||||||
|
- [x] `SPEC.md` - Add `polymarket.py` to architecture list
|
||||||
|
- [x] Run `bash scripts/sync.sh` to deploy
|
||||||
|
|
||||||
|
#### Phase 7: Tests
|
||||||
|
|
||||||
|
- [x] `tests/test_polymarket.py` - New file:
|
||||||
|
- `TestParsePolymarketResponse` - binary markets, multi-outcome, malformed outcomePrices, missing fields
|
||||||
|
- `TestNormalizePolymarketItems` - schema mapping, date handling, type coercion
|
||||||
|
- `TestScorePolymarketItems` - engagement formula, zero volume, high volume
|
||||||
|
- [x] `fixtures/polymarket_sample.json` - Representative API response with edge cases
|
||||||
|
- [x] `tests/test_cross_source.py` - Add Polymarket-to-Reddit linking test case
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- [x] `/last30days "Arizona Basketball"` shows relevant Polymarket prediction markets with odds and volume
|
||||||
|
- [x] `/last30days "best rap songs 2026"` gracefully returns no Polymarket results (and hides the PM line from stats)
|
||||||
|
- [x] Sources with zero results are hidden from stats box (all sources, not just PM)
|
||||||
|
- [x] `--diagnose` shows `"polymarket": true`
|
||||||
|
- [x] Multi-outcome markets show top 3 outcomes
|
||||||
|
- [x] Polymarket items participate in cross-source linking
|
||||||
|
- [x] No API key required - works out of the box for all users
|
||||||
|
- [x] `--quick`, `--deep` flags affect Polymarket result count
|
||||||
|
- [x] Older cached reports without `polymarket` key load without errors
|
||||||
|
|
||||||
|
## Dependencies & Risks
|
||||||
|
|
||||||
|
**No blockers.** The Gamma API is free, unauthenticated, and has generous rate limits (350 req/10s). No new dependencies needed - just stdlib `urllib` (already used by HN module).
|
||||||
|
|
||||||
|
**Risk: API availability.** Polymarket could change or restrict their API. Mitigation: the source is optional and fails gracefully (returns empty list, hidden from stats).
|
||||||
|
|
||||||
|
**Risk: Relevance mismatch.** Many topics won't have prediction markets. The hide-zero-sources change ensures this doesn't pollute the output.
|
||||||
|
|
||||||
|
## Sources & References
|
||||||
|
|
||||||
|
- Polymarket Gamma API docs: https://docs.polymarket.com/developers/gamma-markets-api/overview
|
||||||
|
- Polymarket CLI: https://github.com/Polymarket/polymarket-cli
|
||||||
|
- HN source plan (pattern to follow): `docs/plans/2026-02-24-feat-hacker-news-source-plan.md`
|
||||||
|
- HN source implementation (most recent similar feature): `scripts/lib/hackernews.py`
|
||||||
|
- Schema: `scripts/lib/schema.py`
|
||||||
|
- Main pipeline: `scripts/last30days.py`
|
||||||
@@ -0,0 +1,150 @@
|
|||||||
|
{
|
||||||
|
"events": [
|
||||||
|
{
|
||||||
|
"id": "evt-arizona-big12",
|
||||||
|
"title": "Will Arizona win the Big 12 Championship?",
|
||||||
|
"slug": "arizona-big-12-championship",
|
||||||
|
"active": true,
|
||||||
|
"closed": false,
|
||||||
|
"updatedAt": "2026-02-24T18:30:00.000Z",
|
||||||
|
"markets": [
|
||||||
|
{
|
||||||
|
"id": "mkt-arizona-big12-1",
|
||||||
|
"question": "Will Arizona win the Big 12 Championship?",
|
||||||
|
"active": true,
|
||||||
|
"closed": false,
|
||||||
|
"outcomes": "[\"Yes\", \"No\"]",
|
||||||
|
"outcomePrices": "[\"0.64\", \"0.36\"]",
|
||||||
|
"volume": "1500000",
|
||||||
|
"volume24hr": "342000",
|
||||||
|
"liquidity": "2100000",
|
||||||
|
"oneDayPriceChange": 0.03,
|
||||||
|
"oneWeekPriceChange": 0.225,
|
||||||
|
"oneMonthPriceChange": 0.15,
|
||||||
|
"endDate": "2026-03-15T00:00:00.000Z"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "evt-arizona-ncaa",
|
||||||
|
"title": "Will Arizona win the NCAA Tournament?",
|
||||||
|
"slug": "arizona-ncaa-tournament",
|
||||||
|
"active": true,
|
||||||
|
"closed": false,
|
||||||
|
"updatedAt": "2026-02-23T12:00:00.000Z",
|
||||||
|
"markets": [
|
||||||
|
{
|
||||||
|
"id": "mkt-arizona-ncaa-1",
|
||||||
|
"question": "Will Arizona win the NCAA Tournament?",
|
||||||
|
"active": true,
|
||||||
|
"closed": false,
|
||||||
|
"outcomes": "[\"Yes\", \"No\"]",
|
||||||
|
"outcomePrices": "[\"0.12\", \"0.88\"]",
|
||||||
|
"volume": "800000",
|
||||||
|
"volume24hr": "89000",
|
||||||
|
"liquidity": "450000",
|
||||||
|
"oneDayPriceChange": -0.005,
|
||||||
|
"oneWeekPriceChange": -0.02,
|
||||||
|
"oneMonthPriceChange": -0.117,
|
||||||
|
"endDate": "2026-04-07T00:00:00.000Z"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "evt-closed-resolved",
|
||||||
|
"title": "Resolved Event (should be filtered)",
|
||||||
|
"slug": "resolved-event",
|
||||||
|
"active": true,
|
||||||
|
"closed": true,
|
||||||
|
"updatedAt": "2026-02-20T10:00:00.000Z",
|
||||||
|
"markets": [
|
||||||
|
{
|
||||||
|
"id": "mkt-resolved-1",
|
||||||
|
"question": "Already resolved?",
|
||||||
|
"active": true,
|
||||||
|
"closed": true,
|
||||||
|
"outcomes": "[\"Yes\", \"No\"]",
|
||||||
|
"outcomePrices": "[\"1\", \"0\"]",
|
||||||
|
"volume": "500000",
|
||||||
|
"volume24hr": "0",
|
||||||
|
"liquidity": "0",
|
||||||
|
"oneDayPriceChange": 0,
|
||||||
|
"oneWeekPriceChange": 0,
|
||||||
|
"oneMonthPriceChange": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "evt-multi-outcome",
|
||||||
|
"title": "Who will win the Big 12 Tournament?",
|
||||||
|
"slug": "big-12-tournament-winner",
|
||||||
|
"active": true,
|
||||||
|
"closed": false,
|
||||||
|
"updatedAt": "2026-02-24T20:00:00.000Z",
|
||||||
|
"markets": [
|
||||||
|
{
|
||||||
|
"id": "mkt-multi-1",
|
||||||
|
"question": "Who will win the Big 12 Tournament?",
|
||||||
|
"active": true,
|
||||||
|
"closed": false,
|
||||||
|
"outcomes": "[\"Arizona\", \"Kansas\", \"Houston\", \"Iowa State\", \"BYU\"]",
|
||||||
|
"outcomePrices": "[\"0.35\", \"0.22\", \"0.18\", \"0.15\", \"0.10\"]",
|
||||||
|
"volume": "2000000",
|
||||||
|
"volume24hr": "150000",
|
||||||
|
"liquidity": "1800000",
|
||||||
|
"oneDayPriceChange": 0.05,
|
||||||
|
"oneWeekPriceChange": 0.08,
|
||||||
|
"oneMonthPriceChange": 0.12
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "evt-no-liquidity",
|
||||||
|
"title": "Dead market (no liquidity)",
|
||||||
|
"slug": "dead-market",
|
||||||
|
"active": true,
|
||||||
|
"closed": false,
|
||||||
|
"updatedAt": "2026-02-10T00:00:00.000Z",
|
||||||
|
"markets": [
|
||||||
|
{
|
||||||
|
"id": "mkt-dead-1",
|
||||||
|
"question": "Dead market?",
|
||||||
|
"active": true,
|
||||||
|
"closed": false,
|
||||||
|
"outcomes": "[\"Yes\", \"No\"]",
|
||||||
|
"outcomePrices": "[\"0.50\", \"0.50\"]",
|
||||||
|
"volume": "100",
|
||||||
|
"volume24hr": "0",
|
||||||
|
"liquidity": "0",
|
||||||
|
"oneDayPriceChange": 0,
|
||||||
|
"oneWeekPriceChange": 0,
|
||||||
|
"oneMonthPriceChange": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "evt-malformed",
|
||||||
|
"title": "Malformed prices",
|
||||||
|
"slug": "malformed-prices",
|
||||||
|
"active": true,
|
||||||
|
"closed": false,
|
||||||
|
"updatedAt": "2026-02-22T08:00:00.000Z",
|
||||||
|
"markets": [
|
||||||
|
{
|
||||||
|
"id": "mkt-malformed-1",
|
||||||
|
"question": "Malformed outcomes?",
|
||||||
|
"active": true,
|
||||||
|
"closed": false,
|
||||||
|
"outcomes": "not valid json",
|
||||||
|
"outcomePrices": "also not valid json",
|
||||||
|
"volume": "50000",
|
||||||
|
"volume24hr": "5000",
|
||||||
|
"liquidity": "30000",
|
||||||
|
"oneDayPriceChange": 0,
|
||||||
|
"oneWeekPriceChange": 0,
|
||||||
|
"oneMonthPriceChange": 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
+87
-10
@@ -38,9 +38,9 @@ _child_pids: set = set()
|
|||||||
_child_pids_lock = threading.Lock()
|
_child_pids_lock = threading.Lock()
|
||||||
|
|
||||||
TIMEOUT_PROFILES = {
|
TIMEOUT_PROFILES = {
|
||||||
"quick": {"global": 90, "future": 30, "reddit_future": 60, "youtube_future": 60, "hackernews_future": 30, "http": 15, "enrich_per": 8, "enrich_total": 30, "enrich_max_items": 10},
|
"quick": {"global": 90, "future": 30, "reddit_future": 60, "youtube_future": 60, "hackernews_future": 30, "polymarket_future": 15, "http": 15, "enrich_per": 8, "enrich_total": 30, "enrich_max_items": 10},
|
||||||
"default": {"global": 180, "future": 60, "reddit_future": 90, "youtube_future": 90, "hackernews_future": 60, "http": 30, "enrich_per": 15, "enrich_total": 45, "enrich_max_items": 15},
|
"default": {"global": 180, "future": 60, "reddit_future": 90, "youtube_future": 90, "hackernews_future": 60, "polymarket_future": 30, "http": 30, "enrich_per": 15, "enrich_total": 45, "enrich_max_items": 15},
|
||||||
"deep": {"global": 300, "future": 90, "reddit_future": 120, "youtube_future": 120, "hackernews_future": 90, "http": 30, "enrich_per": 15, "enrich_total": 60, "enrich_max_items": 25},
|
"deep": {"global": 300, "future": 90, "reddit_future": 120, "youtube_future": 120, "hackernews_future": 90, "polymarket_future": 45, "http": 30, "enrich_per": 15, "enrich_total": 60, "enrich_max_items": 25},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -99,6 +99,7 @@ from lib import (
|
|||||||
dates,
|
dates,
|
||||||
dedupe,
|
dedupe,
|
||||||
hackernews,
|
hackernews,
|
||||||
|
polymarket,
|
||||||
entity_extract,
|
entity_extract,
|
||||||
env,
|
env,
|
||||||
http,
|
http,
|
||||||
@@ -332,6 +333,34 @@ def _search_hackernews(
|
|||||||
return hn_items, hn_error
|
return hn_items, hn_error
|
||||||
|
|
||||||
|
|
||||||
|
def _search_polymarket(
|
||||||
|
topic: str,
|
||||||
|
from_date: str,
|
||||||
|
to_date: str,
|
||||||
|
depth: str,
|
||||||
|
) -> tuple:
|
||||||
|
"""Search Polymarket via Gamma API (runs in thread).
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (pm_items, pm_error)
|
||||||
|
"""
|
||||||
|
pm_error = None
|
||||||
|
|
||||||
|
try:
|
||||||
|
response = polymarket.search_polymarket(
|
||||||
|
topic, from_date, to_date, depth=depth,
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
return [], f"{type(e).__name__}: {e}"
|
||||||
|
|
||||||
|
pm_items = polymarket.parse_polymarket_response(response, topic=topic)
|
||||||
|
|
||||||
|
if response.get("error"):
|
||||||
|
pm_error = response["error"]
|
||||||
|
|
||||||
|
return pm_items, pm_error
|
||||||
|
|
||||||
|
|
||||||
def _search_web(
|
def _search_web(
|
||||||
topic: str,
|
topic: str,
|
||||||
config: dict,
|
config: dict,
|
||||||
@@ -588,6 +617,7 @@ def run_research(
|
|||||||
x_items = []
|
x_items = []
|
||||||
youtube_items = []
|
youtube_items = []
|
||||||
hackernews_items = []
|
hackernews_items = []
|
||||||
|
polymarket_items = []
|
||||||
web_items = []
|
web_items = []
|
||||||
raw_openai = None
|
raw_openai = None
|
||||||
raw_xai = None
|
raw_xai = None
|
||||||
@@ -596,6 +626,7 @@ def run_research(
|
|||||||
x_error = None
|
x_error = None
|
||||||
youtube_error = None
|
youtube_error = None
|
||||||
hackernews_error = None
|
hackernews_error = None
|
||||||
|
polymarket_error = None
|
||||||
web_error = None
|
web_error = None
|
||||||
|
|
||||||
# Determine web search mode
|
# Determine web search mode
|
||||||
@@ -638,20 +669,22 @@ def run_research(
|
|||||||
progress.show_error(f"YouTube error: {e}")
|
progress.show_error(f"YouTube error: {e}")
|
||||||
if progress:
|
if progress:
|
||||||
progress.end_youtube(len(youtube_items))
|
progress.end_youtube(len(youtube_items))
|
||||||
return reddit_items, x_items, youtube_items, hackernews_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, web_error
|
return reddit_items, x_items, youtube_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, polymarket_error, web_error
|
||||||
|
|
||||||
# Determine which searches to run
|
# Determine which searches to run
|
||||||
do_reddit = sources in ("both", "reddit", "all", "reddit-web")
|
do_reddit = sources in ("both", "reddit", "all", "reddit-web")
|
||||||
do_x = sources in ("both", "x", "all", "x-web")
|
do_x = sources in ("both", "x", "all", "x-web")
|
||||||
do_hackernews = True # HN is always available (no API key)
|
do_hackernews = True # HN is always available (no API key)
|
||||||
|
do_polymarket = True # Polymarket is always available (no API key)
|
||||||
|
|
||||||
# Run Reddit, X, YouTube, HN, and Web searches in parallel
|
# Run Reddit, X, YouTube, HN, Polymarket, and Web searches in parallel
|
||||||
reddit_future = None
|
reddit_future = None
|
||||||
x_future = None
|
x_future = None
|
||||||
youtube_future = None
|
youtube_future = None
|
||||||
hackernews_future = None
|
hackernews_future = None
|
||||||
|
polymarket_future = None
|
||||||
web_future = None
|
web_future = None
|
||||||
max_workers = 2 + (1 if run_youtube else 0) + (1 if do_hackernews else 0) + (1 if web_backend else 0)
|
max_workers = 2 + (1 if run_youtube else 0) + (1 if do_hackernews else 0) + (1 if do_polymarket else 0) + (1 if web_backend else 0)
|
||||||
|
|
||||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||||
# Submit searches
|
# Submit searches
|
||||||
@@ -685,6 +718,13 @@ def run_research(
|
|||||||
_search_hackernews, topic, from_date, to_date, depth
|
_search_hackernews, topic, from_date, to_date, depth
|
||||||
)
|
)
|
||||||
|
|
||||||
|
if do_polymarket:
|
||||||
|
if progress:
|
||||||
|
progress.start_polymarket()
|
||||||
|
polymarket_future = executor.submit(
|
||||||
|
_search_polymarket, topic, from_date, to_date, depth
|
||||||
|
)
|
||||||
|
|
||||||
if web_backend:
|
if web_backend:
|
||||||
sys.stderr.write(f"[web] Searching via {web_backend}\n")
|
sys.stderr.write(f"[web] Searching via {web_backend}\n")
|
||||||
sys.stderr.flush()
|
sys.stderr.flush()
|
||||||
@@ -760,6 +800,23 @@ def run_research(
|
|||||||
if progress:
|
if progress:
|
||||||
progress.end_hackernews(len(hackernews_items))
|
progress.end_hackernews(len(hackernews_items))
|
||||||
|
|
||||||
|
if polymarket_future:
|
||||||
|
pm_timeout = timeouts.get("polymarket_future", future_timeout)
|
||||||
|
try:
|
||||||
|
polymarket_items, polymarket_error = polymarket_future.result(timeout=pm_timeout)
|
||||||
|
if polymarket_error and progress:
|
||||||
|
progress.show_error(f"Polymarket error: {polymarket_error}")
|
||||||
|
except TimeoutError:
|
||||||
|
polymarket_error = f"Polymarket search timed out after {pm_timeout}s"
|
||||||
|
if progress:
|
||||||
|
progress.show_error(polymarket_error)
|
||||||
|
except Exception as e:
|
||||||
|
polymarket_error = f"{type(e).__name__}: {e}"
|
||||||
|
if progress:
|
||||||
|
progress.show_error(f"Polymarket error: {e}")
|
||||||
|
if progress:
|
||||||
|
progress.end_polymarket(len(polymarket_items))
|
||||||
|
|
||||||
if web_future:
|
if web_future:
|
||||||
try:
|
try:
|
||||||
web_items, web_error = web_future.result(timeout=future_timeout)
|
web_items, web_error = web_future.result(timeout=future_timeout)
|
||||||
@@ -868,7 +925,7 @@ def run_research(
|
|||||||
if sup_x:
|
if sup_x:
|
||||||
x_items.extend(sup_x)
|
x_items.extend(sup_x)
|
||||||
|
|
||||||
return reddit_items, x_items, youtube_items, hackernews_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, web_error
|
return reddit_items, x_items, youtube_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, polymarket_error, web_error
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
@@ -994,6 +1051,7 @@ def main():
|
|||||||
"bird_username": x_source_status.get("bird_username"),
|
"bird_username": x_source_status.get("bird_username"),
|
||||||
"youtube": has_ytdlp,
|
"youtube": has_ytdlp,
|
||||||
"hackernews": True,
|
"hackernews": True,
|
||||||
|
"polymarket": True,
|
||||||
"web_search_backend": web_source,
|
"web_search_backend": web_source,
|
||||||
"parallel_ai": bool(config.get("PARALLEL_API_KEY")),
|
"parallel_ai": bool(config.get("PARALLEL_API_KEY")),
|
||||||
"brave": bool(config.get("BRAVE_API_KEY")),
|
"brave": bool(config.get("BRAVE_API_KEY")),
|
||||||
@@ -1022,6 +1080,7 @@ def main():
|
|||||||
"bird_username": x_source_status.get("bird_username"),
|
"bird_username": x_source_status.get("bird_username"),
|
||||||
"youtube": has_ytdlp,
|
"youtube": has_ytdlp,
|
||||||
"hackernews": True,
|
"hackernews": True,
|
||||||
|
"polymarket": True,
|
||||||
"web_search_backend": web_source,
|
"web_search_backend": web_source,
|
||||||
}
|
}
|
||||||
ui.show_diagnostic_banner(diag)
|
ui.show_diagnostic_banner(diag)
|
||||||
@@ -1099,7 +1158,7 @@ def main():
|
|||||||
mode = sources
|
mode = sources
|
||||||
|
|
||||||
# Run research
|
# Run research
|
||||||
reddit_items, x_items, youtube_items, hackernews_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, web_error = run_research(
|
reddit_items, x_items, youtube_items, hackernews_items, polymarket_items, web_items, web_needed, raw_openai, raw_xai, raw_reddit_enriched, reddit_error, x_error, youtube_error, hackernews_error, polymarket_error, web_error = run_research(
|
||||||
args.topic,
|
args.topic,
|
||||||
sources,
|
sources,
|
||||||
config,
|
config,
|
||||||
@@ -1123,6 +1182,7 @@ def main():
|
|||||||
normalized_x = normalize.normalize_x_items(x_items, from_date, to_date)
|
normalized_x = normalize.normalize_x_items(x_items, from_date, to_date)
|
||||||
normalized_youtube = normalize.normalize_youtube_items(youtube_items, from_date, to_date) if youtube_items else []
|
normalized_youtube = normalize.normalize_youtube_items(youtube_items, from_date, to_date) if youtube_items else []
|
||||||
normalized_hn = normalize.normalize_hackernews_items(hackernews_items, from_date, to_date) if hackernews_items else []
|
normalized_hn = normalize.normalize_hackernews_items(hackernews_items, from_date, to_date) if hackernews_items else []
|
||||||
|
normalized_pm = normalize.normalize_polymarket_items(polymarket_items, from_date, to_date) if polymarket_items else []
|
||||||
normalized_web = websearch.normalize_websearch_items(web_items, from_date, to_date) if web_items else []
|
normalized_web = websearch.normalize_websearch_items(web_items, from_date, to_date) if web_items else []
|
||||||
|
|
||||||
# Hard date filter: exclude items with verified dates outside the range
|
# Hard date filter: exclude items with verified dates outside the range
|
||||||
@@ -1134,6 +1194,8 @@ def main():
|
|||||||
# YouTube content has a longer shelf life than tweets/posts.
|
# YouTube content has a longer shelf life than tweets/posts.
|
||||||
filtered_youtube = normalized_youtube
|
filtered_youtube = normalized_youtube
|
||||||
filtered_hn = normalize.filter_by_date_range(normalized_hn, from_date, to_date) if normalized_hn else []
|
filtered_hn = normalize.filter_by_date_range(normalized_hn, from_date, to_date) if normalized_hn else []
|
||||||
|
# Polymarket: skip hard date filter - markets are active/traded, updatedAt is fine
|
||||||
|
filtered_pm = normalized_pm
|
||||||
filtered_web = normalize.filter_by_date_range(normalized_web, from_date, to_date) if normalized_web else []
|
filtered_web = normalize.filter_by_date_range(normalized_web, from_date, to_date) if normalized_web else []
|
||||||
|
|
||||||
# Score items
|
# Score items
|
||||||
@@ -1141,6 +1203,7 @@ def main():
|
|||||||
scored_x = score.score_x_items(filtered_x)
|
scored_x = score.score_x_items(filtered_x)
|
||||||
scored_youtube = score.score_youtube_items(filtered_youtube) if filtered_youtube else []
|
scored_youtube = score.score_youtube_items(filtered_youtube) if filtered_youtube else []
|
||||||
scored_hn = score.score_hackernews_items(filtered_hn) if filtered_hn else []
|
scored_hn = score.score_hackernews_items(filtered_hn) if filtered_hn else []
|
||||||
|
scored_pm = score.score_polymarket_items(filtered_pm) if filtered_pm else []
|
||||||
scored_web = score.score_websearch_items(filtered_web) if filtered_web else []
|
scored_web = score.score_websearch_items(filtered_web) if filtered_web else []
|
||||||
|
|
||||||
# Sort items
|
# Sort items
|
||||||
@@ -1148,6 +1211,7 @@ def main():
|
|||||||
sorted_x = score.sort_items(scored_x)
|
sorted_x = score.sort_items(scored_x)
|
||||||
sorted_youtube = score.sort_items(scored_youtube) if scored_youtube else []
|
sorted_youtube = score.sort_items(scored_youtube) if scored_youtube else []
|
||||||
sorted_hn = score.sort_items(scored_hn) if scored_hn else []
|
sorted_hn = score.sort_items(scored_hn) if scored_hn else []
|
||||||
|
sorted_pm = score.sort_items(scored_pm) if scored_pm else []
|
||||||
sorted_web = score.sort_items(scored_web) if scored_web else []
|
sorted_web = score.sort_items(scored_web) if scored_web else []
|
||||||
|
|
||||||
# Dedupe items
|
# Dedupe items
|
||||||
@@ -1155,6 +1219,7 @@ def main():
|
|||||||
deduped_x = dedupe.dedupe_x(sorted_x)
|
deduped_x = dedupe.dedupe_x(sorted_x)
|
||||||
deduped_youtube = dedupe.dedupe_youtube(sorted_youtube) if sorted_youtube else []
|
deduped_youtube = dedupe.dedupe_youtube(sorted_youtube) if sorted_youtube else []
|
||||||
deduped_hn = dedupe.dedupe_hackernews(sorted_hn) if sorted_hn else []
|
deduped_hn = dedupe.dedupe_hackernews(sorted_hn) if sorted_hn else []
|
||||||
|
deduped_pm = dedupe.dedupe_polymarket(sorted_pm) if sorted_pm else []
|
||||||
deduped_web = websearch.dedupe_websearch(sorted_web) if sorted_web else []
|
deduped_web = websearch.dedupe_websearch(sorted_web) if sorted_web else []
|
||||||
|
|
||||||
# Minimum result guarantee: if all Reddit results were filtered out but
|
# Minimum result guarantee: if all Reddit results were filtered out but
|
||||||
@@ -1166,7 +1231,7 @@ def main():
|
|||||||
|
|
||||||
# Cross-source linking: annotate items that discuss the same story
|
# Cross-source linking: annotate items that discuss the same story
|
||||||
dedupe.cross_source_link(
|
dedupe.cross_source_link(
|
||||||
deduped_reddit, deduped_x, deduped_youtube, deduped_hn, deduped_web,
|
deduped_reddit, deduped_x, deduped_youtube, deduped_hn, deduped_pm, deduped_web,
|
||||||
)
|
)
|
||||||
|
|
||||||
progress.end_processing()
|
progress.end_processing()
|
||||||
@@ -1184,11 +1249,13 @@ def main():
|
|||||||
report.x = deduped_x
|
report.x = deduped_x
|
||||||
report.youtube = deduped_youtube
|
report.youtube = deduped_youtube
|
||||||
report.hackernews = deduped_hn
|
report.hackernews = deduped_hn
|
||||||
|
report.polymarket = deduped_pm
|
||||||
report.web = deduped_web
|
report.web = deduped_web
|
||||||
report.reddit_error = reddit_error
|
report.reddit_error = reddit_error
|
||||||
report.x_error = x_error
|
report.x_error = x_error
|
||||||
report.youtube_error = youtube_error
|
report.youtube_error = youtube_error
|
||||||
report.hackernews_error = hackernews_error
|
report.hackernews_error = hackernews_error
|
||||||
|
report.polymarket_error = polymarket_error
|
||||||
report.web_error = web_error
|
report.web_error = web_error
|
||||||
report.resolved_x_handle = args.x_handle
|
report.resolved_x_handle = args.x_handle
|
||||||
|
|
||||||
@@ -1202,7 +1269,7 @@ def main():
|
|||||||
if sources == "web":
|
if sources == "web":
|
||||||
progress.show_web_only_complete()
|
progress.show_web_only_complete()
|
||||||
else:
|
else:
|
||||||
progress.show_complete(len(deduped_reddit), len(deduped_x), len(deduped_youtube), len(deduped_hn))
|
progress.show_complete(len(deduped_reddit), len(deduped_x), len(deduped_youtube), len(deduped_hn), len(deduped_pm))
|
||||||
|
|
||||||
# Build source info for status footer
|
# Build source info for status footer
|
||||||
source_info = {}
|
source_info = {}
|
||||||
@@ -1270,6 +1337,16 @@ def main():
|
|||||||
"engagement_score": item.engagement.score if item.engagement else 0,
|
"engagement_score": item.engagement.score if item.engagement else 0,
|
||||||
"relevance_score": item.relevance,
|
"relevance_score": item.relevance,
|
||||||
})
|
})
|
||||||
|
for item in deduped_pm:
|
||||||
|
findings.append({
|
||||||
|
"source": "polymarket",
|
||||||
|
"url": item.url,
|
||||||
|
"title": item.question,
|
||||||
|
"author": "polymarket",
|
||||||
|
"content": item.title,
|
||||||
|
"engagement_score": item.engagement.volume if item.engagement and item.engagement.volume else 0,
|
||||||
|
"relevance_score": item.relevance,
|
||||||
|
})
|
||||||
for item in deduped_web:
|
for item in deduped_web:
|
||||||
findings.append({
|
findings.append({
|
||||||
"source": "web",
|
"source": "web",
|
||||||
|
|||||||
+13
-1
@@ -46,7 +46,7 @@ def jaccard_similarity(set1: Set[str], set2: Set[str]) -> float:
|
|||||||
|
|
||||||
|
|
||||||
AnyItem = Union[schema.RedditItem, schema.XItem, schema.YouTubeItem,
|
AnyItem = Union[schema.RedditItem, schema.XItem, schema.YouTubeItem,
|
||||||
schema.HackerNewsItem, schema.WebSearchItem]
|
schema.HackerNewsItem, schema.PolymarketItem, schema.WebSearchItem]
|
||||||
|
|
||||||
|
|
||||||
def get_item_text(item: AnyItem) -> str:
|
def get_item_text(item: AnyItem) -> str:
|
||||||
@@ -57,6 +57,8 @@ def get_item_text(item: AnyItem) -> str:
|
|||||||
return item.title
|
return item.title
|
||||||
elif isinstance(item, schema.YouTubeItem):
|
elif isinstance(item, schema.YouTubeItem):
|
||||||
return f"{item.title} {item.channel_name}"
|
return f"{item.title} {item.channel_name}"
|
||||||
|
elif isinstance(item, schema.PolymarketItem):
|
||||||
|
return f"{item.title} {item.question}"
|
||||||
elif isinstance(item, schema.WebSearchItem):
|
elif isinstance(item, schema.WebSearchItem):
|
||||||
return item.title
|
return item.title
|
||||||
else:
|
else:
|
||||||
@@ -79,6 +81,8 @@ def _get_cross_source_text(item: AnyItem) -> str:
|
|||||||
elif title.startswith("Ask HN:"):
|
elif title.startswith("Ask HN:"):
|
||||||
title = title[7:].strip()
|
title = title[7:].strip()
|
||||||
return title
|
return title
|
||||||
|
if isinstance(item, schema.PolymarketItem):
|
||||||
|
return item.title
|
||||||
return get_item_text(item)
|
return get_item_text(item)
|
||||||
|
|
||||||
|
|
||||||
@@ -198,6 +202,14 @@ def dedupe_hackernews(
|
|||||||
return dedupe_items(items, threshold)
|
return dedupe_items(items, threshold)
|
||||||
|
|
||||||
|
|
||||||
|
def dedupe_polymarket(
|
||||||
|
items: List[schema.PolymarketItem],
|
||||||
|
threshold: float = 0.7,
|
||||||
|
) -> List[schema.PolymarketItem]:
|
||||||
|
"""Dedupe Polymarket items."""
|
||||||
|
return dedupe_items(items, threshold)
|
||||||
|
|
||||||
|
|
||||||
def cross_source_link(
|
def cross_source_link(
|
||||||
*source_lists: List[AnyItem],
|
*source_lists: List[AnyItem],
|
||||||
threshold: float = 0.40,
|
threshold: float = 0.40,
|
||||||
|
|||||||
@@ -254,6 +254,14 @@ def is_hackernews_available() -> bool:
|
|||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def is_polymarket_available() -> bool:
|
||||||
|
"""Check if Polymarket source is available.
|
||||||
|
|
||||||
|
Always returns True - Gamma API is free, no key needed.
|
||||||
|
"""
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]:
|
def get_x_source_status(config: Dict[str, Any]) -> Dict[str, Any]:
|
||||||
"""Get detailed X source status for UI decisions.
|
"""Get detailed X source status for UI decisions.
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ from typing import Any, Dict, List, TypeVar, Union
|
|||||||
|
|
||||||
from . import dates, schema
|
from . import dates, schema
|
||||||
|
|
||||||
T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.HackerNewsItem)
|
T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.HackerNewsItem, schema.PolymarketItem)
|
||||||
|
|
||||||
|
|
||||||
def filter_by_date_range(
|
def filter_by_date_range(
|
||||||
@@ -257,6 +257,50 @@ def normalize_hackernews_items(
|
|||||||
return normalized
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_polymarket_items(
|
||||||
|
items: List[Dict[str, Any]],
|
||||||
|
from_date: str,
|
||||||
|
to_date: str,
|
||||||
|
) -> List[schema.PolymarketItem]:
|
||||||
|
"""Normalize raw Polymarket items to schema.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
items: Raw Polymarket items from Gamma API
|
||||||
|
from_date: Start of date range
|
||||||
|
to_date: End of date range
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of PolymarketItem objects
|
||||||
|
"""
|
||||||
|
normalized = []
|
||||||
|
|
||||||
|
for i, item in enumerate(items):
|
||||||
|
engagement = schema.Engagement(
|
||||||
|
volume=item.get("volume24hr", 0.0),
|
||||||
|
liquidity=item.get("liquidity", 0.0),
|
||||||
|
)
|
||||||
|
|
||||||
|
date_str = item.get("date")
|
||||||
|
|
||||||
|
normalized.append(schema.PolymarketItem(
|
||||||
|
id=f"PM{i+1}",
|
||||||
|
title=item.get("title", ""),
|
||||||
|
question=item.get("question", ""),
|
||||||
|
url=item.get("url", ""),
|
||||||
|
outcome_prices=item.get("outcome_prices", []),
|
||||||
|
outcomes_remaining=item.get("outcomes_remaining", 0),
|
||||||
|
price_movement=item.get("price_movement"),
|
||||||
|
date=date_str,
|
||||||
|
date_confidence="high",
|
||||||
|
engagement=engagement,
|
||||||
|
end_date=item.get("end_date"),
|
||||||
|
relevance=item.get("relevance", 0.5),
|
||||||
|
why_relevant=item.get("why_relevant", ""),
|
||||||
|
))
|
||||||
|
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
def items_to_dicts(items: List) -> List[Dict[str, Any]]:
|
def items_to_dicts(items: List) -> List[Dict[str, Any]]:
|
||||||
"""Convert schema items to dicts for JSON serialization."""
|
"""Convert schema items to dicts for JSON serialization."""
|
||||||
return [item.to_dict() for item in items]
|
return [item.to_dict() for item in items]
|
||||||
|
|||||||
@@ -0,0 +1,354 @@
|
|||||||
|
"""Polymarket prediction market search via Gamma API (free, no auth required).
|
||||||
|
|
||||||
|
Uses gamma-api.polymarket.com for event/market discovery.
|
||||||
|
No API key needed - public read-only API with generous rate limits (350 req/10s).
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||||
|
from typing import Any, Dict, List, Optional
|
||||||
|
from urllib.parse import quote_plus, urlencode
|
||||||
|
|
||||||
|
from . import http
|
||||||
|
|
||||||
|
GAMMA_SEARCH_URL = "https://gamma-api.polymarket.com/public-search"
|
||||||
|
|
||||||
|
DEPTH_CONFIG = {
|
||||||
|
"quick": 5,
|
||||||
|
"default": 10,
|
||||||
|
"deep": 20,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _log(msg: str):
|
||||||
|
"""Log to stderr (only in TTY mode to avoid cluttering Claude Code output)."""
|
||||||
|
if sys.stderr.isatty():
|
||||||
|
sys.stderr.write(f"[PM] {msg}\n")
|
||||||
|
sys.stderr.flush()
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_core_subject(topic: str) -> str:
|
||||||
|
"""Extract core subject from topic string.
|
||||||
|
|
||||||
|
Strips common prefixes like 'last 7 days', 'what are people saying about', etc.
|
||||||
|
"""
|
||||||
|
topic = topic.strip()
|
||||||
|
# Remove common leading phrases
|
||||||
|
prefixes = [
|
||||||
|
r"^last \d+ days?\s+",
|
||||||
|
r"^what(?:'s| is| are) (?:people saying about|happening with|going on with)\s+",
|
||||||
|
r"^how (?:is|are)\s+",
|
||||||
|
r"^tell me about\s+",
|
||||||
|
r"^research\s+",
|
||||||
|
]
|
||||||
|
for pattern in prefixes:
|
||||||
|
topic = re.sub(pattern, "", topic, flags=re.IGNORECASE)
|
||||||
|
return topic.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def _expand_queries(topic: str) -> List[str]:
|
||||||
|
"""Generate 2-4 search queries to cast a wider net.
|
||||||
|
|
||||||
|
Strategy:
|
||||||
|
- Always include the core subject
|
||||||
|
- Split multi-word topics into component searches
|
||||||
|
- Include the full topic if different from core
|
||||||
|
- Cap at 4 queries, dedupe
|
||||||
|
"""
|
||||||
|
core = _extract_core_subject(topic)
|
||||||
|
queries = [core]
|
||||||
|
|
||||||
|
# Split multi-word topics into component searches
|
||||||
|
words = core.split()
|
||||||
|
if len(words) >= 2:
|
||||||
|
# Try the first significant word alone (e.g., "Arizona" from "Arizona Basketball")
|
||||||
|
queries.append(words[0])
|
||||||
|
|
||||||
|
# Add the full topic if different from core
|
||||||
|
if topic.lower().strip() != core.lower():
|
||||||
|
queries.append(topic.strip())
|
||||||
|
|
||||||
|
# Dedupe while preserving order, cap at 4
|
||||||
|
seen = set()
|
||||||
|
unique = []
|
||||||
|
for q in queries:
|
||||||
|
q_lower = q.lower().strip()
|
||||||
|
if q_lower and q_lower not in seen:
|
||||||
|
seen.add(q_lower)
|
||||||
|
unique.append(q.strip())
|
||||||
|
return unique[:4]
|
||||||
|
|
||||||
|
|
||||||
|
def _search_single_query(query: str, limit: int) -> Dict[str, Any]:
|
||||||
|
"""Run a single search query against Gamma API."""
|
||||||
|
params = {
|
||||||
|
"q": query,
|
||||||
|
"limit": str(limit),
|
||||||
|
}
|
||||||
|
url = f"{GAMMA_SEARCH_URL}?{urlencode(params)}"
|
||||||
|
|
||||||
|
try:
|
||||||
|
response = http.request("GET", url, timeout=15, retries=2)
|
||||||
|
return response
|
||||||
|
except http.HTTPError as e:
|
||||||
|
_log(f"Search failed for '{query}': {e}")
|
||||||
|
return {"events": [], "error": str(e)}
|
||||||
|
except Exception as e:
|
||||||
|
_log(f"Search failed for '{query}': {e}")
|
||||||
|
return {"events": [], "error": str(e)}
|
||||||
|
|
||||||
|
|
||||||
|
def search_polymarket(
|
||||||
|
topic: str,
|
||||||
|
from_date: str,
|
||||||
|
to_date: str,
|
||||||
|
depth: str = "default",
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""Search Polymarket via Gamma API with smart query expansion.
|
||||||
|
|
||||||
|
Runs 2-4 expanded queries in parallel, merges and dedupes by event ID.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
topic: Search topic
|
||||||
|
from_date: Start date (YYYY-MM-DD) - used for activity filtering
|
||||||
|
to_date: End date (YYYY-MM-DD)
|
||||||
|
depth: 'quick', 'default', or 'deep'
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict with 'events' list and optional 'error'.
|
||||||
|
"""
|
||||||
|
limit_per_query = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
|
||||||
|
queries = _expand_queries(topic)
|
||||||
|
|
||||||
|
_log(f"Searching for '{topic}' with queries: {queries} (limit={limit_per_query})")
|
||||||
|
|
||||||
|
# Run all queries in parallel
|
||||||
|
all_events = {} # event_id -> (event_data, query_index)
|
||||||
|
errors = []
|
||||||
|
|
||||||
|
with ThreadPoolExecutor(max_workers=min(4, len(queries))) as executor:
|
||||||
|
futures = {
|
||||||
|
executor.submit(_search_single_query, q, limit_per_query): i
|
||||||
|
for i, q in enumerate(queries)
|
||||||
|
}
|
||||||
|
|
||||||
|
for future in as_completed(futures):
|
||||||
|
query_idx = futures[future]
|
||||||
|
try:
|
||||||
|
response = future.result(timeout=15)
|
||||||
|
if response.get("error"):
|
||||||
|
errors.append(response["error"])
|
||||||
|
|
||||||
|
events = response.get("events", [])
|
||||||
|
for event in events:
|
||||||
|
event_id = event.get("id", "")
|
||||||
|
if not event_id:
|
||||||
|
continue
|
||||||
|
# Keep the first occurrence (from highest-priority query)
|
||||||
|
if event_id not in all_events:
|
||||||
|
all_events[event_id] = (event, query_idx)
|
||||||
|
elif query_idx < all_events[event_id][1]:
|
||||||
|
# Replace with higher-priority query result
|
||||||
|
all_events[event_id] = (event, query_idx)
|
||||||
|
except Exception as e:
|
||||||
|
errors.append(str(e))
|
||||||
|
|
||||||
|
# Sort by query priority, then by position
|
||||||
|
merged_events = [ev for ev, _ in sorted(all_events.values(), key=lambda x: x[1])]
|
||||||
|
_log(f"Found {len(merged_events)} unique events across {len(queries)} queries")
|
||||||
|
|
||||||
|
result = {"events": merged_events}
|
||||||
|
if errors and not merged_events:
|
||||||
|
result["error"] = "; ".join(errors[:2])
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _format_price_movement(market: Dict[str, Any]) -> Optional[str]:
|
||||||
|
"""Pick the most significant price change and format it.
|
||||||
|
|
||||||
|
Returns string like 'down 11.7% this month' or None if no significant change.
|
||||||
|
"""
|
||||||
|
changes = [
|
||||||
|
(abs(market.get("oneDayPriceChange") or 0), market.get("oneDayPriceChange"), "today"),
|
||||||
|
(abs(market.get("oneWeekPriceChange") or 0), market.get("oneWeekPriceChange"), "this week"),
|
||||||
|
(abs(market.get("oneMonthPriceChange") or 0), market.get("oneMonthPriceChange"), "this month"),
|
||||||
|
]
|
||||||
|
|
||||||
|
# Pick the largest absolute change
|
||||||
|
changes.sort(key=lambda x: x[0], reverse=True)
|
||||||
|
abs_change, raw_change, period = changes[0]
|
||||||
|
|
||||||
|
# Skip if change is less than 1% (noise)
|
||||||
|
if abs_change < 0.01:
|
||||||
|
return None
|
||||||
|
|
||||||
|
direction = "up" if raw_change > 0 else "down"
|
||||||
|
pct = abs_change * 100
|
||||||
|
return f"{direction} {pct:.1f}% {period}"
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_outcome_prices(market: Dict[str, Any]) -> List[tuple]:
|
||||||
|
"""Parse outcomePrices JSON string into list of (outcome_name, price) tuples."""
|
||||||
|
outcomes_raw = market.get("outcomes") or []
|
||||||
|
prices_raw = market.get("outcomePrices")
|
||||||
|
|
||||||
|
if not prices_raw:
|
||||||
|
return []
|
||||||
|
|
||||||
|
# Both outcomes and outcomePrices can be JSON-encoded strings
|
||||||
|
try:
|
||||||
|
if isinstance(outcomes_raw, str):
|
||||||
|
outcomes = json.loads(outcomes_raw)
|
||||||
|
else:
|
||||||
|
outcomes = outcomes_raw
|
||||||
|
except (json.JSONDecodeError, TypeError):
|
||||||
|
outcomes = []
|
||||||
|
|
||||||
|
try:
|
||||||
|
if isinstance(prices_raw, str):
|
||||||
|
prices = json.loads(prices_raw)
|
||||||
|
else:
|
||||||
|
prices = prices_raw
|
||||||
|
except (json.JSONDecodeError, TypeError):
|
||||||
|
return []
|
||||||
|
|
||||||
|
result = []
|
||||||
|
for i, price in enumerate(prices):
|
||||||
|
try:
|
||||||
|
p = float(price)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
continue
|
||||||
|
name = outcomes[i] if i < len(outcomes) else f"Outcome {i+1}"
|
||||||
|
result.append((name, p))
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def parse_polymarket_response(response: Dict[str, Any], topic: str = "") -> List[Dict[str, Any]]:
|
||||||
|
"""Parse Gamma API response into normalized item dicts.
|
||||||
|
|
||||||
|
Each event becomes one item showing its title and top markets.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
response: Raw Gamma API response
|
||||||
|
topic: Original search topic (for relevance scoring)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of item dicts ready for normalization.
|
||||||
|
"""
|
||||||
|
events = response.get("events", [])
|
||||||
|
items = []
|
||||||
|
|
||||||
|
for i, event in enumerate(events):
|
||||||
|
event_id = event.get("id", "")
|
||||||
|
title = event.get("title", "")
|
||||||
|
slug = event.get("slug", "")
|
||||||
|
|
||||||
|
# Filter: skip closed/resolved events
|
||||||
|
if event.get("closed", False):
|
||||||
|
continue
|
||||||
|
if not event.get("active", True):
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Get markets for this event
|
||||||
|
markets = event.get("markets", [])
|
||||||
|
if not markets:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Filter to active, open markets with liquidity (excludes resolved markets)
|
||||||
|
active_markets = []
|
||||||
|
for m in markets:
|
||||||
|
if m.get("closed", False):
|
||||||
|
continue
|
||||||
|
if not m.get("active", True):
|
||||||
|
continue
|
||||||
|
# Must have liquidity (resolved markets have 0 or None)
|
||||||
|
try:
|
||||||
|
liq = float(m.get("liquidity", 0) or 0)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
liq = 0
|
||||||
|
if liq > 0:
|
||||||
|
active_markets.append(m)
|
||||||
|
|
||||||
|
if not active_markets:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Sort markets by volume (most liquid first)
|
||||||
|
def market_volume(m):
|
||||||
|
try:
|
||||||
|
return float(m.get("volume", 0) or 0)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
return 0
|
||||||
|
active_markets.sort(key=market_volume, reverse=True)
|
||||||
|
|
||||||
|
# Take top market for the event
|
||||||
|
top_market = active_markets[0]
|
||||||
|
|
||||||
|
# Parse outcome prices from top market
|
||||||
|
outcome_prices = _parse_outcome_prices(top_market)
|
||||||
|
|
||||||
|
# Format price movement
|
||||||
|
price_movement = _format_price_movement(top_market)
|
||||||
|
|
||||||
|
# Volume and liquidity
|
||||||
|
try:
|
||||||
|
volume24hr = float(top_market.get("volume24hr", 0) or 0)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
volume24hr = 0.0
|
||||||
|
try:
|
||||||
|
liquidity = float(top_market.get("liquidity", 0) or 0)
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
liquidity = 0.0
|
||||||
|
|
||||||
|
# Event URL
|
||||||
|
url = f"https://polymarket.com/event/{slug}" if slug else f"https://polymarket.com/event/{event_id}"
|
||||||
|
|
||||||
|
# Date: use updatedAt from event
|
||||||
|
updated_at = event.get("updatedAt", "")
|
||||||
|
date_str = None
|
||||||
|
if updated_at:
|
||||||
|
# Parse ISO format: "2026-02-20T15:30:00.000Z"
|
||||||
|
try:
|
||||||
|
date_str = updated_at[:10] # YYYY-MM-DD
|
||||||
|
except (IndexError, TypeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
# End date for the market
|
||||||
|
end_date = top_market.get("endDate")
|
||||||
|
if end_date:
|
||||||
|
try:
|
||||||
|
end_date = end_date[:10]
|
||||||
|
except (IndexError, TypeError):
|
||||||
|
end_date = None
|
||||||
|
|
||||||
|
# Relevance: position-based decay
|
||||||
|
rank_score = max(0.3, 1.0 - (i * 0.03)) # 1.0 -> 0.3 over ~23 items
|
||||||
|
engagement_boost = min(0.15, math.log1p(volume24hr) / 60)
|
||||||
|
relevance = min(1.0, rank_score * 0.75 + engagement_boost + 0.1)
|
||||||
|
|
||||||
|
# Top 3 outcomes for multi-outcome markets
|
||||||
|
top_outcomes = outcome_prices[:3]
|
||||||
|
remaining = len(outcome_prices) - 3
|
||||||
|
if remaining < 0:
|
||||||
|
remaining = 0
|
||||||
|
|
||||||
|
items.append({
|
||||||
|
"event_id": event_id,
|
||||||
|
"title": title,
|
||||||
|
"question": top_market.get("question", title),
|
||||||
|
"url": url,
|
||||||
|
"outcome_prices": top_outcomes,
|
||||||
|
"outcomes_remaining": remaining,
|
||||||
|
"price_movement": price_movement,
|
||||||
|
"volume24hr": volume24hr,
|
||||||
|
"liquidity": liquidity,
|
||||||
|
"date": date_str,
|
||||||
|
"end_date": end_date,
|
||||||
|
"relevance": round(relevance, 2),
|
||||||
|
"why_relevant": f"Prediction market: {title[:60]}",
|
||||||
|
})
|
||||||
|
|
||||||
|
return items
|
||||||
+94
-9
@@ -26,6 +26,8 @@ def _xref_tag(item) -> str:
|
|||||||
source_names.add('YouTube')
|
source_names.add('YouTube')
|
||||||
elif ref_id.startswith('HN'):
|
elif ref_id.startswith('HN'):
|
||||||
source_names.add('HN')
|
source_names.add('HN')
|
||||||
|
elif ref_id.startswith('PM'):
|
||||||
|
source_names.add('Polymarket')
|
||||||
elif ref_id.startswith('W'):
|
elif ref_id.startswith('W'):
|
||||||
source_names.add('Web')
|
source_names.add('Web')
|
||||||
if source_names:
|
if source_names:
|
||||||
@@ -53,9 +55,10 @@ def _assess_data_freshness(report: schema.Report) -> dict:
|
|||||||
x_recent = sum(1 for x in report.x if x.date and x.date >= report.range_from)
|
x_recent = sum(1 for x in report.x if x.date and x.date >= report.range_from)
|
||||||
web_recent = sum(1 for w in report.web if w.date and w.date >= report.range_from)
|
web_recent = sum(1 for w in report.web if w.date and w.date >= report.range_from)
|
||||||
hn_recent = sum(1 for h in report.hackernews if h.date and h.date >= report.range_from)
|
hn_recent = sum(1 for h in report.hackernews if h.date and h.date >= report.range_from)
|
||||||
|
pm_recent = sum(1 for p in report.polymarket if p.date and p.date >= report.range_from)
|
||||||
|
|
||||||
total_recent = reddit_recent + x_recent + web_recent + hn_recent
|
total_recent = reddit_recent + x_recent + web_recent + hn_recent + pm_recent
|
||||||
total_items = len(report.reddit) + len(report.x) + len(report.web) + len(report.hackernews)
|
total_items = len(report.reddit) + len(report.x) + len(report.web) + len(report.hackernews) + len(report.polymarket)
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"reddit_recent": reddit_recent,
|
"reddit_recent": reddit_recent,
|
||||||
@@ -276,6 +279,59 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
|
|||||||
|
|
||||||
lines.append("")
|
lines.append("")
|
||||||
|
|
||||||
|
# Polymarket items
|
||||||
|
if report.polymarket_error:
|
||||||
|
lines.append("### Prediction Markets (Polymarket)")
|
||||||
|
lines.append("")
|
||||||
|
lines.append(f"**ERROR:** {report.polymarket_error}")
|
||||||
|
lines.append("")
|
||||||
|
elif report.polymarket:
|
||||||
|
lines.append("### Prediction Markets (Polymarket)")
|
||||||
|
lines.append("")
|
||||||
|
for item in report.polymarket[:limit]:
|
||||||
|
eng_str = ""
|
||||||
|
if item.engagement:
|
||||||
|
eng = item.engagement
|
||||||
|
parts = []
|
||||||
|
if eng.volume is not None:
|
||||||
|
if eng.volume >= 1_000_000:
|
||||||
|
parts.append(f"${eng.volume/1_000_000:.1f}M vol24h")
|
||||||
|
elif eng.volume >= 1_000:
|
||||||
|
parts.append(f"${eng.volume/1_000:.0f}K vol24h")
|
||||||
|
else:
|
||||||
|
parts.append(f"${eng.volume:.0f} vol24h")
|
||||||
|
if eng.liquidity is not None:
|
||||||
|
if eng.liquidity >= 1_000_000:
|
||||||
|
parts.append(f"${eng.liquidity/1_000_000:.1f}M liquidity")
|
||||||
|
elif eng.liquidity >= 1_000:
|
||||||
|
parts.append(f"${eng.liquidity/1_000:.0f}K liquidity")
|
||||||
|
else:
|
||||||
|
parts.append(f"${eng.liquidity:.0f} liquidity")
|
||||||
|
if parts:
|
||||||
|
eng_str = f" [{', '.join(parts)}]"
|
||||||
|
|
||||||
|
date_str = f" ({item.date})" if item.date else ""
|
||||||
|
|
||||||
|
lines.append(f"**{item.id}** (score:{item.score}){eng_str}{_xref_tag(item)}")
|
||||||
|
lines.append(f" {item.question}")
|
||||||
|
|
||||||
|
# Outcome prices with price movement
|
||||||
|
if item.outcome_prices:
|
||||||
|
outcomes = []
|
||||||
|
for name, price in item.outcome_prices:
|
||||||
|
pct = price * 100
|
||||||
|
outcomes.append(f"{name}: {pct:.0f}%")
|
||||||
|
outcome_line = " | ".join(outcomes)
|
||||||
|
if item.outcomes_remaining > 0:
|
||||||
|
outcome_line += f" and {item.outcomes_remaining} more"
|
||||||
|
if item.price_movement:
|
||||||
|
outcome_line += f" ({item.price_movement})"
|
||||||
|
lines.append(f" {outcome_line}")
|
||||||
|
|
||||||
|
lines.append(f" {item.url}")
|
||||||
|
lines.append(f" *{item.why_relevant}*")
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
# Web items (if any - populated by the assistant)
|
# Web items (if any - populated by the assistant)
|
||||||
if report.web_error:
|
if report.web_error:
|
||||||
lines.append("### Web Results")
|
lines.append("### Web Results")
|
||||||
@@ -323,7 +379,7 @@ def render_source_status(report: schema.Report, source_info: dict = None) -> str
|
|||||||
elif report.reddit:
|
elif report.reddit:
|
||||||
lines.append(f" ✅ Reddit: {len(report.reddit)} threads")
|
lines.append(f" ✅ Reddit: {len(report.reddit)} threads")
|
||||||
elif report.mode in ("both", "reddit-only", "all", "reddit-web"):
|
elif report.mode in ("both", "reddit-only", "all", "reddit-web"):
|
||||||
lines.append(" ⚠️ Reddit: 0 threads found")
|
pass # Hide zero-result sources
|
||||||
else:
|
else:
|
||||||
reason = source_info.get("reddit_skip_reason", "not configured")
|
reason = source_info.get("reddit_skip_reason", "not configured")
|
||||||
lines.append(f" ⏭️ Reddit: skipped — {reason}")
|
lines.append(f" ⏭️ Reddit: skipped — {reason}")
|
||||||
@@ -337,7 +393,7 @@ def render_source_status(report: schema.Report, source_info: dict = None) -> str
|
|||||||
x_line += f" (via @{report.resolved_x_handle} + keyword search)"
|
x_line += f" (via @{report.resolved_x_handle} + keyword search)"
|
||||||
lines.append(x_line)
|
lines.append(x_line)
|
||||||
elif report.mode in ("both", "x-only", "all", "x-web"):
|
elif report.mode in ("both", "x-only", "all", "x-web"):
|
||||||
lines.append(" ⚠️ X: 0 posts found")
|
pass # Hide zero-result sources
|
||||||
else:
|
else:
|
||||||
reason = source_info.get("x_skip_reason", "No Bird CLI or XAI_API_KEY")
|
reason = source_info.get("x_skip_reason", "No Bird CLI or XAI_API_KEY")
|
||||||
lines.append(f" ⏭️ X: skipped — {reason}")
|
lines.append(f" ⏭️ X: skipped — {reason}")
|
||||||
@@ -348,17 +404,21 @@ def render_source_status(report: schema.Report, source_info: dict = None) -> str
|
|||||||
elif report.youtube:
|
elif report.youtube:
|
||||||
with_transcripts = sum(1 for v in report.youtube if getattr(v, 'transcript_snippet', None))
|
with_transcripts = sum(1 for v in report.youtube if getattr(v, 'transcript_snippet', None))
|
||||||
lines.append(f" ✅ YouTube: {len(report.youtube)} videos ({with_transcripts} with transcripts)")
|
lines.append(f" ✅ YouTube: {len(report.youtube)} videos ({with_transcripts} with transcripts)")
|
||||||
else:
|
# Hide when zero results (no skip reason line needed)
|
||||||
reason = source_info.get("youtube_skip_reason", "yt-dlp not installed (brew install yt-dlp)")
|
|
||||||
lines.append(f" ⏭️ YouTube: skipped — {reason}")
|
|
||||||
|
|
||||||
# Hacker News
|
# Hacker News
|
||||||
if report.hackernews_error:
|
if report.hackernews_error:
|
||||||
lines.append(f" ❌ HN: error - {report.hackernews_error}")
|
lines.append(f" ❌ HN: error - {report.hackernews_error}")
|
||||||
elif report.hackernews:
|
elif report.hackernews:
|
||||||
lines.append(f" ✅ HN: {len(report.hackernews)} stories")
|
lines.append(f" ✅ HN: {len(report.hackernews)} stories")
|
||||||
else:
|
# Hide when zero results
|
||||||
lines.append(" ⏭️ HN: 0 stories found")
|
|
||||||
|
# Polymarket
|
||||||
|
if report.polymarket_error:
|
||||||
|
lines.append(f" ❌ Polymarket: error - {report.polymarket_error}")
|
||||||
|
elif report.polymarket:
|
||||||
|
lines.append(f" ✅ Polymarket: {len(report.polymarket)} markets")
|
||||||
|
# Hide when zero results
|
||||||
|
|
||||||
# Web
|
# Web
|
||||||
if report.web_error:
|
if report.web_error:
|
||||||
@@ -399,6 +459,8 @@ def render_context_snippet(report: schema.Report) -> str:
|
|||||||
all_items.append((item.score, "X", item.text[:50] + "...", item.url))
|
all_items.append((item.score, "X", item.text[:50] + "...", item.url))
|
||||||
for item in report.hackernews[:5]:
|
for item in report.hackernews[:5]:
|
||||||
all_items.append((item.score, "HN", item.title[:50] + "...", item.hn_url))
|
all_items.append((item.score, "HN", item.title[:50] + "...", item.hn_url))
|
||||||
|
for item in report.polymarket[:5]:
|
||||||
|
all_items.append((item.score, "Polymarket", item.question[:50] + "...", item.url))
|
||||||
for item in report.web[:5]:
|
for item in report.web[:5]:
|
||||||
all_items.append((item.score, "Web", item.title[:50] + "...", item.url))
|
all_items.append((item.score, "Web", item.title[:50] + "...", item.url))
|
||||||
|
|
||||||
@@ -515,6 +577,29 @@ def render_full_report(report: schema.Report) -> str:
|
|||||||
|
|
||||||
lines.append("")
|
lines.append("")
|
||||||
|
|
||||||
|
# Polymarket section
|
||||||
|
if report.polymarket:
|
||||||
|
lines.append("## Prediction Markets (Polymarket)")
|
||||||
|
lines.append("")
|
||||||
|
for item in report.polymarket:
|
||||||
|
lines.append(f"### {item.id}: {item.question}")
|
||||||
|
lines.append("")
|
||||||
|
lines.append(f"- **Event:** {item.title}")
|
||||||
|
lines.append(f"- **URL:** {item.url}")
|
||||||
|
lines.append(f"- **Date:** {item.date or 'Unknown'}")
|
||||||
|
lines.append(f"- **Score:** {item.score}/100")
|
||||||
|
|
||||||
|
if item.outcome_prices:
|
||||||
|
outcomes = [f"{name}: {price*100:.0f}%" for name, price in item.outcome_prices]
|
||||||
|
lines.append(f"- **Outcomes:** {' | '.join(outcomes)}")
|
||||||
|
if item.price_movement:
|
||||||
|
lines.append(f"- **Trend:** {item.price_movement}")
|
||||||
|
if item.engagement:
|
||||||
|
eng = item.engagement
|
||||||
|
lines.append(f"- **Volume:** ${eng.volume or 0:,.0f} | Liquidity: ${eng.liquidity or 0:,.0f}")
|
||||||
|
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
# Web section
|
# Web section
|
||||||
if report.web:
|
if report.web:
|
||||||
lines.append("## Web Results")
|
lines.append("## Web Results")
|
||||||
|
|||||||
@@ -22,6 +22,10 @@ class Engagement:
|
|||||||
# YouTube fields
|
# YouTube fields
|
||||||
views: Optional[int] = None
|
views: Optional[int] = None
|
||||||
|
|
||||||
|
# Polymarket fields
|
||||||
|
volume: Optional[float] = None
|
||||||
|
liquidity: Optional[float] = None
|
||||||
|
|
||||||
def to_dict(self) -> Dict[str, Any]:
|
def to_dict(self) -> Dict[str, Any]:
|
||||||
d = {}
|
d = {}
|
||||||
if self.score is not None:
|
if self.score is not None:
|
||||||
@@ -40,6 +44,10 @@ class Engagement:
|
|||||||
d['quotes'] = self.quotes
|
d['quotes'] = self.quotes
|
||||||
if self.views is not None:
|
if self.views is not None:
|
||||||
d['views'] = self.views
|
d['views'] = self.views
|
||||||
|
if self.volume is not None:
|
||||||
|
d['volume'] = self.volume
|
||||||
|
if self.liquidity is not None:
|
||||||
|
d['liquidity'] = self.liquidity
|
||||||
return d if d else None
|
return d if d else None
|
||||||
|
|
||||||
|
|
||||||
@@ -264,6 +272,49 @@ class HackerNewsItem:
|
|||||||
return d
|
return d
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class PolymarketItem:
|
||||||
|
"""Normalized Polymarket prediction market item."""
|
||||||
|
id: str # "PM1", "PM2", ...
|
||||||
|
title: str # Event title
|
||||||
|
question: str # Top market question
|
||||||
|
url: str # Event page URL
|
||||||
|
outcome_prices: List[tuple] = field(default_factory=list) # [(name, price), ...]
|
||||||
|
outcomes_remaining: int = 0
|
||||||
|
price_movement: Optional[str] = None # "down 11.7% this month"
|
||||||
|
date: Optional[str] = None
|
||||||
|
date_confidence: str = "high" # API provides exact timestamps
|
||||||
|
engagement: Optional[Engagement] = None # volume + liquidity
|
||||||
|
end_date: Optional[str] = None
|
||||||
|
relevance: float = 0.5
|
||||||
|
why_relevant: str = ""
|
||||||
|
subs: SubScores = field(default_factory=SubScores)
|
||||||
|
score: int = 0
|
||||||
|
cross_refs: List[str] = field(default_factory=list)
|
||||||
|
|
||||||
|
def to_dict(self) -> Dict[str, Any]:
|
||||||
|
d = {
|
||||||
|
'id': self.id,
|
||||||
|
'title': self.title,
|
||||||
|
'question': self.question,
|
||||||
|
'url': self.url,
|
||||||
|
'outcome_prices': self.outcome_prices,
|
||||||
|
'outcomes_remaining': self.outcomes_remaining,
|
||||||
|
'price_movement': self.price_movement,
|
||||||
|
'date': self.date,
|
||||||
|
'date_confidence': self.date_confidence,
|
||||||
|
'engagement': self.engagement.to_dict() if self.engagement else None,
|
||||||
|
'end_date': self.end_date,
|
||||||
|
'relevance': self.relevance,
|
||||||
|
'why_relevant': self.why_relevant,
|
||||||
|
'subs': self.subs.to_dict(),
|
||||||
|
'score': self.score,
|
||||||
|
}
|
||||||
|
if self.cross_refs:
|
||||||
|
d['cross_refs'] = self.cross_refs
|
||||||
|
return d
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class Report:
|
class Report:
|
||||||
"""Full research report."""
|
"""Full research report."""
|
||||||
@@ -279,6 +330,7 @@ class Report:
|
|||||||
web: List[WebSearchItem] = field(default_factory=list)
|
web: List[WebSearchItem] = field(default_factory=list)
|
||||||
youtube: List[YouTubeItem] = field(default_factory=list)
|
youtube: List[YouTubeItem] = field(default_factory=list)
|
||||||
hackernews: List[HackerNewsItem] = field(default_factory=list)
|
hackernews: List[HackerNewsItem] = field(default_factory=list)
|
||||||
|
polymarket: List[PolymarketItem] = field(default_factory=list)
|
||||||
best_practices: List[str] = field(default_factory=list)
|
best_practices: List[str] = field(default_factory=list)
|
||||||
prompt_pack: List[str] = field(default_factory=list)
|
prompt_pack: List[str] = field(default_factory=list)
|
||||||
context_snippet_md: str = ""
|
context_snippet_md: str = ""
|
||||||
@@ -288,6 +340,7 @@ class Report:
|
|||||||
web_error: Optional[str] = None
|
web_error: Optional[str] = None
|
||||||
youtube_error: Optional[str] = None
|
youtube_error: Optional[str] = None
|
||||||
hackernews_error: Optional[str] = None
|
hackernews_error: Optional[str] = None
|
||||||
|
polymarket_error: Optional[str] = None
|
||||||
# Handle resolution
|
# Handle resolution
|
||||||
resolved_x_handle: Optional[str] = None
|
resolved_x_handle: Optional[str] = None
|
||||||
# Cache info
|
# Cache info
|
||||||
@@ -310,6 +363,7 @@ class Report:
|
|||||||
'web': [w.to_dict() for w in self.web],
|
'web': [w.to_dict() for w in self.web],
|
||||||
'youtube': [y.to_dict() for y in self.youtube],
|
'youtube': [y.to_dict() for y in self.youtube],
|
||||||
'hackernews': [h.to_dict() for h in self.hackernews],
|
'hackernews': [h.to_dict() for h in self.hackernews],
|
||||||
|
'polymarket': [p.to_dict() for p in self.polymarket],
|
||||||
'best_practices': self.best_practices,
|
'best_practices': self.best_practices,
|
||||||
'prompt_pack': self.prompt_pack,
|
'prompt_pack': self.prompt_pack,
|
||||||
'context_snippet_md': self.context_snippet_md,
|
'context_snippet_md': self.context_snippet_md,
|
||||||
@@ -326,6 +380,8 @@ class Report:
|
|||||||
d['youtube_error'] = self.youtube_error
|
d['youtube_error'] = self.youtube_error
|
||||||
if self.hackernews_error:
|
if self.hackernews_error:
|
||||||
d['hackernews_error'] = self.hackernews_error
|
d['hackernews_error'] = self.hackernews_error
|
||||||
|
if self.polymarket_error:
|
||||||
|
d['polymarket_error'] = self.polymarket_error
|
||||||
if self.from_cache:
|
if self.from_cache:
|
||||||
d['from_cache'] = self.from_cache
|
d['from_cache'] = self.from_cache
|
||||||
if self.cache_age_hours is not None:
|
if self.cache_age_hours is not None:
|
||||||
@@ -455,6 +511,32 @@ class Report:
|
|||||||
cross_refs=h.get('cross_refs', []),
|
cross_refs=h.get('cross_refs', []),
|
||||||
))
|
))
|
||||||
|
|
||||||
|
# Reconstruct Polymarket items (backward compat: key may not exist)
|
||||||
|
pm_items = []
|
||||||
|
for p in data.get('polymarket', []):
|
||||||
|
eng = None
|
||||||
|
if p.get('engagement'):
|
||||||
|
eng = Engagement(**p['engagement'])
|
||||||
|
subs = SubScores(**p.get('subs', {})) if p.get('subs') else SubScores()
|
||||||
|
pm_items.append(PolymarketItem(
|
||||||
|
id=p['id'],
|
||||||
|
title=p['title'],
|
||||||
|
question=p.get('question', ''),
|
||||||
|
url=p['url'],
|
||||||
|
outcome_prices=p.get('outcome_prices', []),
|
||||||
|
outcomes_remaining=p.get('outcomes_remaining', 0),
|
||||||
|
price_movement=p.get('price_movement'),
|
||||||
|
date=p.get('date'),
|
||||||
|
date_confidence=p.get('date_confidence', 'high'),
|
||||||
|
engagement=eng,
|
||||||
|
end_date=p.get('end_date'),
|
||||||
|
relevance=p.get('relevance', 0.5),
|
||||||
|
why_relevant=p.get('why_relevant', ''),
|
||||||
|
subs=subs,
|
||||||
|
score=p.get('score', 0),
|
||||||
|
cross_refs=p.get('cross_refs', []),
|
||||||
|
))
|
||||||
|
|
||||||
return cls(
|
return cls(
|
||||||
topic=data['topic'],
|
topic=data['topic'],
|
||||||
range_from=range_from,
|
range_from=range_from,
|
||||||
@@ -468,6 +550,7 @@ class Report:
|
|||||||
web=web_items,
|
web=web_items,
|
||||||
youtube=youtube_items,
|
youtube=youtube_items,
|
||||||
hackernews=hn_items,
|
hackernews=hn_items,
|
||||||
|
polymarket=pm_items,
|
||||||
best_practices=data.get('best_practices', []),
|
best_practices=data.get('best_practices', []),
|
||||||
prompt_pack=data.get('prompt_pack', []),
|
prompt_pack=data.get('prompt_pack', []),
|
||||||
context_snippet_md=data.get('context_snippet_md', ''),
|
context_snippet_md=data.get('context_snippet_md', ''),
|
||||||
@@ -476,6 +559,7 @@ class Report:
|
|||||||
web_error=data.get('web_error'),
|
web_error=data.get('web_error'),
|
||||||
youtube_error=data.get('youtube_error'),
|
youtube_error=data.get('youtube_error'),
|
||||||
hackernews_error=data.get('hackernews_error'),
|
hackernews_error=data.get('hackernews_error'),
|
||||||
|
polymarket_error=data.get('polymarket_error'),
|
||||||
resolved_x_handle=data.get('resolved_x_handle'),
|
resolved_x_handle=data.get('resolved_x_handle'),
|
||||||
from_cache=data.get('from_cache', False),
|
from_cache=data.get('from_cache', False),
|
||||||
cache_age_hours=data.get('cache_age_hours'),
|
cache_age_hours=data.get('cache_age_hours'),
|
||||||
|
|||||||
+63
-3
@@ -338,6 +338,64 @@ def score_hackernews_items(items: List[schema.HackerNewsItem]) -> List[schema.Ha
|
|||||||
return items
|
return items
|
||||||
|
|
||||||
|
|
||||||
|
def compute_polymarket_engagement_raw(engagement: Optional[schema.Engagement]) -> Optional[float]:
|
||||||
|
"""Compute raw engagement score for Polymarket item.
|
||||||
|
|
||||||
|
Formula: 0.60*log1p(volume) + 0.40*log1p(liquidity)
|
||||||
|
Volume is the primary signal (money flowing); liquidity indicates market depth.
|
||||||
|
"""
|
||||||
|
if engagement is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
if engagement.volume is None and engagement.liquidity is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
volume = math.log1p(engagement.volume or 0)
|
||||||
|
liquidity = math.log1p(engagement.liquidity or 0)
|
||||||
|
|
||||||
|
return 0.60 * volume + 0.40 * liquidity
|
||||||
|
|
||||||
|
|
||||||
|
def score_polymarket_items(items: List[schema.PolymarketItem]) -> List[schema.PolymarketItem]:
|
||||||
|
"""Compute scores for Polymarket items.
|
||||||
|
|
||||||
|
Uses same weight structure as Reddit/X (relevance + recency + engagement).
|
||||||
|
"""
|
||||||
|
if not items:
|
||||||
|
return items
|
||||||
|
|
||||||
|
eng_raw = [compute_polymarket_engagement_raw(item.engagement) for item in items]
|
||||||
|
eng_normalized = normalize_to_100(eng_raw)
|
||||||
|
|
||||||
|
for i, item in enumerate(items):
|
||||||
|
rel_score = int(item.relevance * 100)
|
||||||
|
rec_score = dates.recency_score(item.date)
|
||||||
|
|
||||||
|
if eng_normalized[i] is not None:
|
||||||
|
eng_score = int(eng_normalized[i])
|
||||||
|
else:
|
||||||
|
eng_score = DEFAULT_ENGAGEMENT
|
||||||
|
|
||||||
|
item.subs = schema.SubScores(
|
||||||
|
relevance=rel_score,
|
||||||
|
recency=rec_score,
|
||||||
|
engagement=eng_score,
|
||||||
|
)
|
||||||
|
|
||||||
|
overall = (
|
||||||
|
WEIGHT_RELEVANCE * rel_score +
|
||||||
|
WEIGHT_RECENCY * rec_score +
|
||||||
|
WEIGHT_ENGAGEMENT * eng_score
|
||||||
|
)
|
||||||
|
|
||||||
|
if eng_raw[i] is None:
|
||||||
|
overall -= UNKNOWN_ENGAGEMENT_PENALTY
|
||||||
|
|
||||||
|
item.score = max(0, min(100, int(overall)))
|
||||||
|
|
||||||
|
return items
|
||||||
|
|
||||||
|
|
||||||
def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]:
|
def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]:
|
||||||
"""Compute scores for WebSearch items WITHOUT engagement metrics.
|
"""Compute scores for WebSearch items WITHOUT engagement metrics.
|
||||||
|
|
||||||
@@ -395,7 +453,7 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS
|
|||||||
return items
|
return items
|
||||||
|
|
||||||
|
|
||||||
def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.HackerNewsItem]]) -> List:
|
def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSearchItem, schema.YouTubeItem, schema.HackerNewsItem, schema.PolymarketItem]]) -> List:
|
||||||
"""Sort items by score (descending), then date, then source priority.
|
"""Sort items by score (descending), then date, then source priority.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
@@ -412,7 +470,7 @@ def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSear
|
|||||||
date = item.date or "0000-00-00"
|
date = item.date or "0000-00-00"
|
||||||
date_key = -int(date.replace("-", ""))
|
date_key = -int(date.replace("-", ""))
|
||||||
|
|
||||||
# Tertiary: source priority (Reddit > X > YouTube > HN > WebSearch)
|
# Tertiary: source priority (Reddit > X > YouTube > HN > Polymarket > WebSearch)
|
||||||
if isinstance(item, schema.RedditItem):
|
if isinstance(item, schema.RedditItem):
|
||||||
source_priority = 0
|
source_priority = 0
|
||||||
elif isinstance(item, schema.XItem):
|
elif isinstance(item, schema.XItem):
|
||||||
@@ -421,8 +479,10 @@ def sort_items(items: List[Union[schema.RedditItem, schema.XItem, schema.WebSear
|
|||||||
source_priority = 2
|
source_priority = 2
|
||||||
elif isinstance(item, schema.HackerNewsItem):
|
elif isinstance(item, schema.HackerNewsItem):
|
||||||
source_priority = 3
|
source_priority = 3
|
||||||
else: # WebSearchItem
|
elif isinstance(item, schema.PolymarketItem):
|
||||||
source_priority = 4
|
source_priority = 4
|
||||||
|
else: # WebSearchItem
|
||||||
|
source_priority = 5
|
||||||
|
|
||||||
# Quaternary: title/text for stability
|
# Quaternary: title/text for stability
|
||||||
text = getattr(item, "title", "") or getattr(item, "text", "")
|
text = getattr(item, "title", "") or getattr(item, "text", "")
|
||||||
|
|||||||
+21
-1
@@ -79,6 +79,13 @@ HN_MESSAGES = [
|
|||||||
"Discovering developer conversations...",
|
"Discovering developer conversations...",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
POLYMARKET_MESSAGES = [
|
||||||
|
"Checking prediction markets...",
|
||||||
|
"Finding what people are betting on...",
|
||||||
|
"Scanning Polymarket for odds...",
|
||||||
|
"Discovering prediction markets...",
|
||||||
|
]
|
||||||
|
|
||||||
PROCESSING_MESSAGES = [
|
PROCESSING_MESSAGES = [
|
||||||
"Crunching the data...",
|
"Crunching the data...",
|
||||||
"Scoring and ranking...",
|
"Scoring and ranking...",
|
||||||
@@ -274,6 +281,15 @@ class ProgressDisplay:
|
|||||||
if self.spinner:
|
if self.spinner:
|
||||||
self.spinner.stop(f"{Colors.YELLOW}HN{Colors.RESET} Found {count} stories")
|
self.spinner.stop(f"{Colors.YELLOW}HN{Colors.RESET} Found {count} stories")
|
||||||
|
|
||||||
|
def start_polymarket(self):
|
||||||
|
msg = random.choice(POLYMARKET_MESSAGES)
|
||||||
|
self.spinner = Spinner(f"{Colors.GREEN}Polymarket{Colors.RESET} {msg}", Colors.GREEN, quiet=True)
|
||||||
|
self.spinner.start()
|
||||||
|
|
||||||
|
def end_polymarket(self, count: int):
|
||||||
|
if self.spinner:
|
||||||
|
self.spinner.stop(f"{Colors.GREEN}Polymarket{Colors.RESET} Found {count} markets")
|
||||||
|
|
||||||
def start_processing(self):
|
def start_processing(self):
|
||||||
msg = random.choice(PROCESSING_MESSAGES)
|
msg = random.choice(PROCESSING_MESSAGES)
|
||||||
self.spinner = Spinner(f"{Colors.PURPLE}Processing{Colors.RESET} {msg}", Colors.PURPLE)
|
self.spinner = Spinner(f"{Colors.PURPLE}Processing{Colors.RESET} {msg}", Colors.PURPLE)
|
||||||
@@ -283,7 +299,7 @@ class ProgressDisplay:
|
|||||||
if self.spinner:
|
if self.spinner:
|
||||||
self.spinner.stop()
|
self.spinner.stop()
|
||||||
|
|
||||||
def show_complete(self, reddit_count: int, x_count: int, youtube_count: int = 0, hn_count: int = 0):
|
def show_complete(self, reddit_count: int, x_count: int, youtube_count: int = 0, hn_count: int = 0, pm_count: int = 0):
|
||||||
elapsed = time.time() - self.start_time
|
elapsed = time.time() - self.start_time
|
||||||
if IS_TTY:
|
if IS_TTY:
|
||||||
sys.stderr.write(f"\n{Colors.GREEN}{Colors.BOLD}✓ Research complete{Colors.RESET} ")
|
sys.stderr.write(f"\n{Colors.GREEN}{Colors.BOLD}✓ Research complete{Colors.RESET} ")
|
||||||
@@ -294,6 +310,8 @@ class ProgressDisplay:
|
|||||||
sys.stderr.write(f" {Colors.RED}YouTube:{Colors.RESET} {youtube_count} videos")
|
sys.stderr.write(f" {Colors.RED}YouTube:{Colors.RESET} {youtube_count} videos")
|
||||||
if hn_count:
|
if hn_count:
|
||||||
sys.stderr.write(f" {Colors.YELLOW}HN:{Colors.RESET} {hn_count} stories")
|
sys.stderr.write(f" {Colors.YELLOW}HN:{Colors.RESET} {hn_count} stories")
|
||||||
|
if pm_count:
|
||||||
|
sys.stderr.write(f" {Colors.GREEN}Polymarket:{Colors.RESET} {pm_count} markets")
|
||||||
sys.stderr.write("\n\n")
|
sys.stderr.write("\n\n")
|
||||||
else:
|
else:
|
||||||
parts = [f"Reddit: {reddit_count} threads", f"X: {x_count} posts"]
|
parts = [f"Reddit: {reddit_count} threads", f"X: {x_count} posts"]
|
||||||
@@ -301,6 +319,8 @@ class ProgressDisplay:
|
|||||||
parts.append(f"YouTube: {youtube_count} videos")
|
parts.append(f"YouTube: {youtube_count} videos")
|
||||||
if hn_count:
|
if hn_count:
|
||||||
parts.append(f"HN: {hn_count} stories")
|
parts.append(f"HN: {hn_count} stories")
|
||||||
|
if pm_count:
|
||||||
|
parts.append(f"Polymarket: {pm_count} markets")
|
||||||
sys.stderr.write(f"✓ Research complete ({elapsed:.1f}s) - {', '.join(parts)}\n")
|
sys.stderr.write(f"✓ Research complete ({elapsed:.1f}s) - {', '.join(parts)}\n")
|
||||||
sys.stderr.flush()
|
sys.stderr.flush()
|
||||||
|
|
||||||
|
|||||||
@@ -116,6 +116,26 @@ class TestCrossSourceLink(unittest.TestCase):
|
|||||||
self.assertIn("HN1", web[0].cross_refs)
|
self.assertIn("HN1", web[0].cross_refs)
|
||||||
self.assertIn("W1", hn[0].cross_refs)
|
self.assertIn("W1", hn[0].cross_refs)
|
||||||
|
|
||||||
|
def _make_pm(self, id, title, score=50):
|
||||||
|
item = schema.PolymarketItem(id=id, title=title, question="Q?", url="")
|
||||||
|
item.score = score
|
||||||
|
return item
|
||||||
|
|
||||||
|
def test_polymarket_to_reddit_link(self):
|
||||||
|
reddit = [self._make_reddit("R1", "Will Arizona win the Big 12 Championship?")]
|
||||||
|
pm = [self._make_pm("PM1", "Will Arizona win the Big 12 Championship?")]
|
||||||
|
dedupe.cross_source_link(reddit, pm)
|
||||||
|
self.assertIn("PM1", reddit[0].cross_refs)
|
||||||
|
self.assertIn("R1", pm[0].cross_refs)
|
||||||
|
|
||||||
|
def test_polymarket_multi_source(self):
|
||||||
|
reddit = [self._make_reddit("R1", "Iran nuclear deal prediction markets")]
|
||||||
|
hn = [self._make_hn("HN1", "Iran nuclear deal prediction markets")]
|
||||||
|
pm = [self._make_pm("PM1", "Iran nuclear deal prediction markets")]
|
||||||
|
dedupe.cross_source_link(reddit, hn, pm)
|
||||||
|
self.assertEqual(len(reddit[0].cross_refs), 2)
|
||||||
|
self.assertEqual(len(pm[0].cross_refs), 2)
|
||||||
|
|
||||||
|
|
||||||
class TestCrossRefsSchemaRoundTrip(unittest.TestCase):
|
class TestCrossRefsSchemaRoundTrip(unittest.TestCase):
|
||||||
def test_reddit_roundtrip(self):
|
def test_reddit_roundtrip(self):
|
||||||
|
|||||||
@@ -0,0 +1,472 @@
|
|||||||
|
"""Tests for Polymarket prediction market source module."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import sys
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# Add lib to path
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
|
||||||
|
|
||||||
|
from lib import polymarket, normalize, schema, score
|
||||||
|
|
||||||
|
|
||||||
|
class TestExtractCoreSubject(unittest.TestCase):
|
||||||
|
def test_plain_topic(self):
|
||||||
|
self.assertEqual(polymarket._extract_core_subject("Arizona Basketball"), "Arizona Basketball")
|
||||||
|
|
||||||
|
def test_strips_last_n_days(self):
|
||||||
|
self.assertEqual(polymarket._extract_core_subject("last 7 days Iran"), "Iran")
|
||||||
|
|
||||||
|
def test_strips_what_are(self):
|
||||||
|
result = polymarket._extract_core_subject("what are people saying about Bitcoin")
|
||||||
|
self.assertEqual(result, "Bitcoin")
|
||||||
|
|
||||||
|
def test_strips_tell_me_about(self):
|
||||||
|
result = polymarket._extract_core_subject("tell me about Ukraine")
|
||||||
|
self.assertEqual(result, "Ukraine")
|
||||||
|
|
||||||
|
def test_strips_whitespace(self):
|
||||||
|
self.assertEqual(polymarket._extract_core_subject(" Iran "), "Iran")
|
||||||
|
|
||||||
|
def test_empty_string(self):
|
||||||
|
self.assertEqual(polymarket._extract_core_subject(""), "")
|
||||||
|
|
||||||
|
|
||||||
|
class TestExpandQueries(unittest.TestCase):
|
||||||
|
def test_single_word(self):
|
||||||
|
queries = polymarket._expand_queries("Iran")
|
||||||
|
self.assertIn("Iran", queries)
|
||||||
|
# Single word: no split, just the core
|
||||||
|
self.assertEqual(len(queries), 1)
|
||||||
|
|
||||||
|
def test_multi_word(self):
|
||||||
|
queries = polymarket._expand_queries("Arizona Basketball")
|
||||||
|
self.assertIn("Arizona Basketball", queries)
|
||||||
|
self.assertIn("Arizona", queries)
|
||||||
|
self.assertEqual(len(queries), 2)
|
||||||
|
|
||||||
|
def test_with_prefix_stripped(self):
|
||||||
|
queries = polymarket._expand_queries("last 7 days Iran")
|
||||||
|
self.assertIn("Iran", queries)
|
||||||
|
# Full topic should also be included since it differs from core
|
||||||
|
self.assertIn("last 7 days Iran", queries)
|
||||||
|
|
||||||
|
def test_deduplication(self):
|
||||||
|
queries = polymarket._expand_queries("Iran")
|
||||||
|
# Should not have duplicates
|
||||||
|
self.assertEqual(len(queries), len(set(q.lower() for q in queries)))
|
||||||
|
|
||||||
|
def test_max_4_queries(self):
|
||||||
|
queries = polymarket._expand_queries("some really long topic with many words")
|
||||||
|
self.assertLessEqual(len(queries), 4)
|
||||||
|
|
||||||
|
|
||||||
|
class TestFormatPriceMovement(unittest.TestCase):
|
||||||
|
def test_significant_monthly_change(self):
|
||||||
|
market = {
|
||||||
|
"oneDayPriceChange": 0.005,
|
||||||
|
"oneWeekPriceChange": -0.02,
|
||||||
|
"oneMonthPriceChange": -0.117,
|
||||||
|
}
|
||||||
|
result = polymarket._format_price_movement(market)
|
||||||
|
self.assertEqual(result, "down 11.7% this month")
|
||||||
|
|
||||||
|
def test_significant_weekly_change(self):
|
||||||
|
market = {
|
||||||
|
"oneDayPriceChange": 0.01,
|
||||||
|
"oneWeekPriceChange": 0.225,
|
||||||
|
"oneMonthPriceChange": 0.15,
|
||||||
|
}
|
||||||
|
result = polymarket._format_price_movement(market)
|
||||||
|
self.assertEqual(result, "up 22.5% this week")
|
||||||
|
|
||||||
|
def test_significant_daily_change(self):
|
||||||
|
market = {
|
||||||
|
"oneDayPriceChange": -0.15,
|
||||||
|
"oneWeekPriceChange": 0.02,
|
||||||
|
"oneMonthPriceChange": 0.03,
|
||||||
|
}
|
||||||
|
result = polymarket._format_price_movement(market)
|
||||||
|
self.assertEqual(result, "down 15.0% today")
|
||||||
|
|
||||||
|
def test_no_significant_change(self):
|
||||||
|
market = {
|
||||||
|
"oneDayPriceChange": 0.005,
|
||||||
|
"oneWeekPriceChange": -0.003,
|
||||||
|
"oneMonthPriceChange": 0.002,
|
||||||
|
}
|
||||||
|
result = polymarket._format_price_movement(market)
|
||||||
|
self.assertIsNone(result)
|
||||||
|
|
||||||
|
def test_missing_fields(self):
|
||||||
|
result = polymarket._format_price_movement({})
|
||||||
|
self.assertIsNone(result)
|
||||||
|
|
||||||
|
def test_none_values(self):
|
||||||
|
market = {
|
||||||
|
"oneDayPriceChange": None,
|
||||||
|
"oneWeekPriceChange": None,
|
||||||
|
"oneMonthPriceChange": None,
|
||||||
|
}
|
||||||
|
result = polymarket._format_price_movement(market)
|
||||||
|
self.assertIsNone(result)
|
||||||
|
|
||||||
|
|
||||||
|
class TestParseOutcomePrices(unittest.TestCase):
|
||||||
|
def test_binary_market_json_strings(self):
|
||||||
|
market = {
|
||||||
|
"outcomes": '["Yes", "No"]',
|
||||||
|
"outcomePrices": '["0.65", "0.35"]',
|
||||||
|
}
|
||||||
|
result = polymarket._parse_outcome_prices(market)
|
||||||
|
self.assertEqual(len(result), 2)
|
||||||
|
self.assertEqual(result[0], ("Yes", 0.65))
|
||||||
|
self.assertEqual(result[1], ("No", 0.35))
|
||||||
|
|
||||||
|
def test_list_inputs(self):
|
||||||
|
market = {
|
||||||
|
"outcomes": ["Yes", "No"],
|
||||||
|
"outcomePrices": ["0.70", "0.30"],
|
||||||
|
}
|
||||||
|
result = polymarket._parse_outcome_prices(market)
|
||||||
|
self.assertEqual(len(result), 2)
|
||||||
|
self.assertEqual(result[0], ("Yes", 0.70))
|
||||||
|
|
||||||
|
def test_multi_outcome(self):
|
||||||
|
market = {
|
||||||
|
"outcomes": '["Arizona", "Kansas", "Houston"]',
|
||||||
|
"outcomePrices": '["0.35", "0.22", "0.18"]',
|
||||||
|
}
|
||||||
|
result = polymarket._parse_outcome_prices(market)
|
||||||
|
self.assertEqual(len(result), 3)
|
||||||
|
self.assertEqual(result[0][0], "Arizona")
|
||||||
|
self.assertAlmostEqual(result[0][1], 0.35)
|
||||||
|
|
||||||
|
def test_malformed_outcomes_json(self):
|
||||||
|
market = {
|
||||||
|
"outcomes": "not valid json",
|
||||||
|
"outcomePrices": '["0.50", "0.50"]',
|
||||||
|
}
|
||||||
|
result = polymarket._parse_outcome_prices(market)
|
||||||
|
# Should fall back to "Outcome N" names
|
||||||
|
self.assertEqual(len(result), 2)
|
||||||
|
self.assertEqual(result[0][0], "Outcome 1")
|
||||||
|
|
||||||
|
def test_malformed_prices_json(self):
|
||||||
|
market = {
|
||||||
|
"outcomes": '["Yes", "No"]',
|
||||||
|
"outcomePrices": "not valid json",
|
||||||
|
}
|
||||||
|
result = polymarket._parse_outcome_prices(market)
|
||||||
|
self.assertEqual(result, [])
|
||||||
|
|
||||||
|
def test_missing_prices(self):
|
||||||
|
market = {"outcomes": '["Yes", "No"]'}
|
||||||
|
result = polymarket._parse_outcome_prices(market)
|
||||||
|
self.assertEqual(result, [])
|
||||||
|
|
||||||
|
def test_empty_outcomes(self):
|
||||||
|
market = {
|
||||||
|
"outcomes": "[]",
|
||||||
|
"outcomePrices": '["0.50", "0.50"]',
|
||||||
|
}
|
||||||
|
result = polymarket._parse_outcome_prices(market)
|
||||||
|
self.assertEqual(len(result), 2)
|
||||||
|
# Should use fallback names
|
||||||
|
self.assertEqual(result[0][0], "Outcome 1")
|
||||||
|
|
||||||
|
|
||||||
|
class TestParsePolymarketResponse(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
fixture_path = Path(__file__).parent.parent / "fixtures" / "polymarket_sample.json"
|
||||||
|
with open(fixture_path) as f:
|
||||||
|
self.sample = json.load(f)
|
||||||
|
|
||||||
|
def test_parses_active_events(self):
|
||||||
|
items = polymarket.parse_polymarket_response(self.sample)
|
||||||
|
# Should include Arizona Big 12, Arizona NCAA, multi-outcome, and malformed
|
||||||
|
# Should exclude: closed/resolved event and no-liquidity event
|
||||||
|
titles = [item["title"] for item in items]
|
||||||
|
self.assertIn("Will Arizona win the Big 12 Championship?", titles)
|
||||||
|
self.assertIn("Will Arizona win the NCAA Tournament?", titles)
|
||||||
|
self.assertIn("Who will win the Big 12 Tournament?", titles)
|
||||||
|
|
||||||
|
def test_filters_closed_events(self):
|
||||||
|
items = polymarket.parse_polymarket_response(self.sample)
|
||||||
|
titles = [item["title"] for item in items]
|
||||||
|
self.assertNotIn("Resolved Event (should be filtered)", titles)
|
||||||
|
|
||||||
|
def test_filters_no_liquidity(self):
|
||||||
|
items = polymarket.parse_polymarket_response(self.sample)
|
||||||
|
titles = [item["title"] for item in items]
|
||||||
|
self.assertNotIn("Dead market (no liquidity)", titles)
|
||||||
|
|
||||||
|
def test_item_fields(self):
|
||||||
|
items = polymarket.parse_polymarket_response(self.sample)
|
||||||
|
item = items[0] # Arizona Big 12
|
||||||
|
self.assertEqual(item["event_id"], "evt-arizona-big12")
|
||||||
|
self.assertEqual(item["url"], "https://polymarket.com/event/arizona-big-12-championship")
|
||||||
|
self.assertIsNotNone(item["date"])
|
||||||
|
self.assertIsNotNone(item["outcome_prices"])
|
||||||
|
self.assertIsNotNone(item["volume24hr"])
|
||||||
|
self.assertIsNotNone(item["liquidity"])
|
||||||
|
|
||||||
|
def test_outcome_prices_parsed(self):
|
||||||
|
items = polymarket.parse_polymarket_response(self.sample)
|
||||||
|
item = items[0] # Arizona Big 12 - binary
|
||||||
|
self.assertEqual(len(item["outcome_prices"]), 2)
|
||||||
|
self.assertEqual(item["outcome_prices"][0][0], "Yes")
|
||||||
|
self.assertAlmostEqual(item["outcome_prices"][0][1], 0.64)
|
||||||
|
|
||||||
|
def test_multi_outcome_top3(self):
|
||||||
|
items = polymarket.parse_polymarket_response(self.sample)
|
||||||
|
multi = [i for i in items if i["title"] == "Who will win the Big 12 Tournament?"][0]
|
||||||
|
# Top 3 outcomes shown
|
||||||
|
self.assertEqual(len(multi["outcome_prices"]), 3)
|
||||||
|
# 5 total - 3 shown = 2 remaining
|
||||||
|
self.assertEqual(multi["outcomes_remaining"], 2)
|
||||||
|
|
||||||
|
def test_price_movement(self):
|
||||||
|
items = polymarket.parse_polymarket_response(self.sample)
|
||||||
|
item = items[0] # Arizona Big 12
|
||||||
|
# Weekly change (22.5%) is the most significant
|
||||||
|
self.assertEqual(item["price_movement"], "up 22.5% this week")
|
||||||
|
|
||||||
|
def test_date_extraction(self):
|
||||||
|
items = polymarket.parse_polymarket_response(self.sample)
|
||||||
|
item = items[0]
|
||||||
|
self.assertEqual(item["date"], "2026-02-24")
|
||||||
|
|
||||||
|
def test_relevance_range(self):
|
||||||
|
items = polymarket.parse_polymarket_response(self.sample)
|
||||||
|
for item in items:
|
||||||
|
self.assertGreaterEqual(item["relevance"], 0.0)
|
||||||
|
self.assertLessEqual(item["relevance"], 1.0)
|
||||||
|
|
||||||
|
def test_empty_response(self):
|
||||||
|
items = polymarket.parse_polymarket_response({"events": []})
|
||||||
|
self.assertEqual(items, [])
|
||||||
|
|
||||||
|
def test_missing_events_key(self):
|
||||||
|
items = polymarket.parse_polymarket_response({})
|
||||||
|
self.assertEqual(items, [])
|
||||||
|
|
||||||
|
def test_malformed_prices_still_produces_item(self):
|
||||||
|
items = polymarket.parse_polymarket_response(self.sample)
|
||||||
|
malformed = [i for i in items if i["title"] == "Malformed prices"]
|
||||||
|
self.assertEqual(len(malformed), 1)
|
||||||
|
# Outcome prices should be empty due to malformed JSON
|
||||||
|
self.assertEqual(malformed[0]["outcome_prices"], [])
|
||||||
|
|
||||||
|
def test_end_date_extraction(self):
|
||||||
|
items = polymarket.parse_polymarket_response(self.sample)
|
||||||
|
item = items[0] # Arizona Big 12
|
||||||
|
self.assertEqual(item["end_date"], "2026-03-15")
|
||||||
|
|
||||||
|
|
||||||
|
class TestNormalizePolymarketItems(unittest.TestCase):
|
||||||
|
def test_normalize(self):
|
||||||
|
raw_items = [
|
||||||
|
{
|
||||||
|
"event_id": "evt-1",
|
||||||
|
"title": "Test Market",
|
||||||
|
"question": "Will test pass?",
|
||||||
|
"url": "https://polymarket.com/event/test",
|
||||||
|
"outcome_prices": [("Yes", 0.75), ("No", 0.25)],
|
||||||
|
"outcomes_remaining": 0,
|
||||||
|
"price_movement": "up 5.0% this week",
|
||||||
|
"volume24hr": 100000.0,
|
||||||
|
"liquidity": 500000.0,
|
||||||
|
"date": "2026-02-20",
|
||||||
|
"end_date": "2026-03-01",
|
||||||
|
"relevance": 0.85,
|
||||||
|
"why_relevant": "Prediction market: Test Market",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
result = normalize.normalize_polymarket_items(raw_items, "2026-01-01", "2026-03-01")
|
||||||
|
self.assertEqual(len(result), 1)
|
||||||
|
self.assertIsInstance(result[0], schema.PolymarketItem)
|
||||||
|
self.assertEqual(result[0].id, "PM1")
|
||||||
|
self.assertEqual(result[0].title, "Test Market")
|
||||||
|
self.assertEqual(result[0].question, "Will test pass?")
|
||||||
|
self.assertEqual(result[0].date_confidence, "high")
|
||||||
|
self.assertEqual(result[0].engagement.volume, 100000.0)
|
||||||
|
self.assertEqual(result[0].engagement.liquidity, 500000.0)
|
||||||
|
self.assertEqual(result[0].price_movement, "up 5.0% this week")
|
||||||
|
|
||||||
|
def test_normalize_multiple(self):
|
||||||
|
raw_items = [
|
||||||
|
{
|
||||||
|
"event_id": f"evt-{i}",
|
||||||
|
"title": f"Market {i}",
|
||||||
|
"question": f"Question {i}?",
|
||||||
|
"url": f"https://polymarket.com/event/test-{i}",
|
||||||
|
"outcome_prices": [],
|
||||||
|
"outcomes_remaining": 0,
|
||||||
|
"volume24hr": 0.0,
|
||||||
|
"liquidity": 0.0,
|
||||||
|
"date": "2026-02-20",
|
||||||
|
"relevance": 0.5,
|
||||||
|
"why_relevant": f"Market {i}",
|
||||||
|
}
|
||||||
|
for i in range(3)
|
||||||
|
]
|
||||||
|
result = normalize.normalize_polymarket_items(raw_items, "2026-01-01", "2026-03-01")
|
||||||
|
self.assertEqual(len(result), 3)
|
||||||
|
self.assertEqual(result[0].id, "PM1")
|
||||||
|
self.assertEqual(result[1].id, "PM2")
|
||||||
|
self.assertEqual(result[2].id, "PM3")
|
||||||
|
|
||||||
|
|
||||||
|
class TestScorePolymarketItems(unittest.TestCase):
|
||||||
|
def test_score_items(self):
|
||||||
|
items = [
|
||||||
|
schema.PolymarketItem(
|
||||||
|
id="PM1", title="High volume", question="Q1?", url="",
|
||||||
|
date="2026-02-20",
|
||||||
|
engagement=schema.Engagement(volume=500000.0, liquidity=2000000.0),
|
||||||
|
relevance=0.9,
|
||||||
|
),
|
||||||
|
schema.PolymarketItem(
|
||||||
|
id="PM2", title="Low volume", question="Q2?", url="",
|
||||||
|
date="2026-02-18",
|
||||||
|
engagement=schema.Engagement(volume=100.0, liquidity=500.0),
|
||||||
|
relevance=0.5,
|
||||||
|
),
|
||||||
|
]
|
||||||
|
scored = score.score_polymarket_items(items)
|
||||||
|
self.assertEqual(len(scored), 2)
|
||||||
|
# High engagement + high relevance should score higher
|
||||||
|
self.assertGreater(scored[0].score, scored[1].score)
|
||||||
|
|
||||||
|
def test_score_empty(self):
|
||||||
|
result = score.score_polymarket_items([])
|
||||||
|
self.assertEqual(result, [])
|
||||||
|
|
||||||
|
def test_engagement_formula(self):
|
||||||
|
eng = schema.Engagement(volume=100000.0, liquidity=500000.0)
|
||||||
|
result = score.compute_polymarket_engagement_raw(eng)
|
||||||
|
self.assertIsNotNone(result)
|
||||||
|
self.assertGreater(result, 0)
|
||||||
|
# Verify formula: 0.60 * log1p(volume) + 0.40 * log1p(liquidity)
|
||||||
|
expected = 0.60 * math.log1p(100000.0) + 0.40 * math.log1p(500000.0)
|
||||||
|
self.assertAlmostEqual(result, expected)
|
||||||
|
|
||||||
|
def test_engagement_none(self):
|
||||||
|
result = score.compute_polymarket_engagement_raw(None)
|
||||||
|
self.assertIsNone(result)
|
||||||
|
|
||||||
|
def test_engagement_empty(self):
|
||||||
|
eng = schema.Engagement()
|
||||||
|
result = score.compute_polymarket_engagement_raw(eng)
|
||||||
|
self.assertIsNone(result)
|
||||||
|
|
||||||
|
def test_zero_volume(self):
|
||||||
|
eng = schema.Engagement(volume=0.0, liquidity=0.0)
|
||||||
|
result = score.compute_polymarket_engagement_raw(eng)
|
||||||
|
self.assertIsNotNone(result)
|
||||||
|
self.assertEqual(result, 0.0)
|
||||||
|
|
||||||
|
|
||||||
|
class TestSortItemsWithPolymarket(unittest.TestCase):
|
||||||
|
def test_pm_priority_after_hn(self):
|
||||||
|
"""Polymarket should sort after HN at same score."""
|
||||||
|
hn_item = schema.HackerNewsItem(id="HN1", title="test", url="", hn_url="", author="user")
|
||||||
|
hn_item.score = 50
|
||||||
|
|
||||||
|
pm_item = schema.PolymarketItem(id="PM1", title="test", question="test?", url="")
|
||||||
|
pm_item.score = 50
|
||||||
|
|
||||||
|
web_item = schema.WebSearchItem(id="W1", title="test", url="", source_domain="example.com", snippet="")
|
||||||
|
web_item.score = 50
|
||||||
|
|
||||||
|
sorted_items = score.sort_items([web_item, pm_item, hn_item])
|
||||||
|
# Same score, so sorted by priority: HN(3) > PM(4) > Web(5)
|
||||||
|
self.assertIsInstance(sorted_items[0], schema.HackerNewsItem)
|
||||||
|
self.assertIsInstance(sorted_items[1], schema.PolymarketItem)
|
||||||
|
self.assertIsInstance(sorted_items[2], schema.WebSearchItem)
|
||||||
|
|
||||||
|
|
||||||
|
class TestPolymarketSchemaRoundTrip(unittest.TestCase):
|
||||||
|
def test_to_dict_and_back(self):
|
||||||
|
item = schema.PolymarketItem(
|
||||||
|
id="PM1",
|
||||||
|
title="Test Event",
|
||||||
|
question="Will test pass?",
|
||||||
|
url="https://polymarket.com/event/test",
|
||||||
|
outcome_prices=[("Yes", 0.75), ("No", 0.25)],
|
||||||
|
outcomes_remaining=0,
|
||||||
|
price_movement="up 5.0% this week",
|
||||||
|
date="2026-02-20",
|
||||||
|
engagement=schema.Engagement(volume=100000.0, liquidity=500000.0),
|
||||||
|
end_date="2026-03-01",
|
||||||
|
relevance=0.85,
|
||||||
|
why_relevant="Test",
|
||||||
|
cross_refs=["R1", "HN2"],
|
||||||
|
)
|
||||||
|
d = item.to_dict()
|
||||||
|
self.assertEqual(d["id"], "PM1")
|
||||||
|
self.assertEqual(d["title"], "Test Event")
|
||||||
|
self.assertEqual(d["cross_refs"], ["R1", "HN2"])
|
||||||
|
|
||||||
|
def test_empty_crossrefs_omitted(self):
|
||||||
|
item = schema.PolymarketItem(id="PM1", title="Test", question="Q?", url="")
|
||||||
|
d = item.to_dict()
|
||||||
|
self.assertNotIn("cross_refs", d)
|
||||||
|
|
||||||
|
def test_report_roundtrip_with_polymarket(self):
|
||||||
|
report = schema.Report(
|
||||||
|
topic="test", range_from="2026-01-01", range_to="2026-02-01",
|
||||||
|
generated_at="2026-02-01T00:00:00Z", mode="both",
|
||||||
|
polymarket=[schema.PolymarketItem(
|
||||||
|
id="PM1", title="Test", question="Q?", url="",
|
||||||
|
outcome_prices=[("Yes", 0.5), ("No", 0.5)],
|
||||||
|
cross_refs=["R1"],
|
||||||
|
)],
|
||||||
|
)
|
||||||
|
d = report.to_dict()
|
||||||
|
restored = schema.Report.from_dict(d)
|
||||||
|
self.assertEqual(len(restored.polymarket), 1)
|
||||||
|
self.assertEqual(restored.polymarket[0].id, "PM1")
|
||||||
|
self.assertEqual(restored.polymarket[0].cross_refs, ["R1"])
|
||||||
|
|
||||||
|
def test_report_backward_compat_no_polymarket(self):
|
||||||
|
"""Old cached reports without polymarket key should load fine."""
|
||||||
|
data = {
|
||||||
|
"topic": "test",
|
||||||
|
"range": {"from": "2026-01-01", "to": "2026-02-01"},
|
||||||
|
"generated_at": "2026-02-01T00:00:00Z",
|
||||||
|
"mode": "both",
|
||||||
|
"reddit": [],
|
||||||
|
"x": [],
|
||||||
|
"web": [],
|
||||||
|
"youtube": [],
|
||||||
|
"hackernews": [],
|
||||||
|
# No "polymarket" key
|
||||||
|
}
|
||||||
|
report = schema.Report.from_dict(data)
|
||||||
|
self.assertEqual(report.polymarket, [])
|
||||||
|
self.assertIsNone(report.polymarket_error)
|
||||||
|
|
||||||
|
def test_engagement_volume_liquidity(self):
|
||||||
|
eng = schema.Engagement(volume=342000.0, liquidity=2100000.0)
|
||||||
|
d = eng.to_dict()
|
||||||
|
self.assertEqual(d["volume"], 342000.0)
|
||||||
|
self.assertEqual(d["liquidity"], 2100000.0)
|
||||||
|
|
||||||
|
|
||||||
|
class TestDepthConfig(unittest.TestCase):
|
||||||
|
def test_quick_depth(self):
|
||||||
|
self.assertEqual(polymarket.DEPTH_CONFIG["quick"], 5)
|
||||||
|
|
||||||
|
def test_default_depth(self):
|
||||||
|
self.assertEqual(polymarket.DEPTH_CONFIG["default"], 10)
|
||||||
|
|
||||||
|
def test_deep_depth(self):
|
||||||
|
self.assertEqual(polymarket.DEPTH_CONFIG["deep"], 20)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# One-Shot Research Mode
|
# One-Shot Research Mode
|
||||||
|
|
||||||
Research ANY topic across Reddit, X, YouTube, and the web. Surface what people are actually discussing, recommending, and debating right now.
|
Research ANY topic across Reddit, X, YouTube, Polymarket, and the web. Surface what people are actually discussing, recommending, betting on, and debating right now.
|
||||||
|
|
||||||
## Parse User Intent
|
## Parse User Intent
|
||||||
|
|
||||||
@@ -19,7 +19,7 @@ Before doing anything, parse the user's input for:
|
|||||||
**Display your parsing** before calling tools:
|
**Display your parsing** before calling tools:
|
||||||
|
|
||||||
```
|
```
|
||||||
I'll research {TOPIC} across Reddit, X, YouTube, and the web.
|
I'll research {TOPIC} across Reddit, X, YouTube, Polymarket, and the web.
|
||||||
|
|
||||||
Parsed intent:
|
Parsed intent:
|
||||||
- TOPIC = {TOPIC}
|
- TOPIC = {TOPIC}
|
||||||
@@ -128,11 +128,14 @@ All agents reported back!
|
|||||||
|- Reddit: {N} threads | {N} upvotes | {N} comments
|
|- Reddit: {N} threads | {N} upvotes | {N} comments
|
||||||
|- X: {N} posts | {N} likes | {N} reposts
|
|- X: {N} posts | {N} likes | {N} reposts
|
||||||
|- YouTube: {N} videos | {N} views | {N} with transcripts
|
|- YouTube: {N} videos | {N} views | {N} with transcripts
|
||||||
|
|- Polymarket: {N} markets | top odds: {outcome} {pct}%
|
||||||
|- Web: {N} pages (supplementary)
|
|- Web: {N} pages (supplementary)
|
||||||
|- Top voices: @{handle1}, @{handle2} | r/{sub1}, r/{sub2}
|
|- Top voices: @{handle1}, @{handle2} | r/{sub1}, r/{sub2}
|
||||||
---
|
---
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Omit any source line that returned 0 results.
|
||||||
|
|
||||||
**3. Invitation** with 2-3 specific follow-up suggestions based on research.
|
**3. Invitation** with 2-3 specific follow-up suggestions based on research.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
Reference in New Issue
Block a user