feat(polymarket): outcome-aware scoring and synthesis instructions
- _compute_text_similarity() now checks outcome names with bidirectional substring matching (0.85) and token overlap (0.7), not just event titles - Collect outcomes from ALL active markets per event, filter to >1% price - Reorder outcome_prices to surface topic-matching outcome before top-3 truncation - Add SKILL.md "Prediction Markets" synthesis section with structural/long-term market preference, domain examples, citation format, and narrative weaving - Add Polymarket to citation priority list between HN and Web - Update stats box template to show up to 5 market odds - Fix render.py "vol24h" label to "volume" - Add NCAA seed fixture event for outcome-only matching tests - 82 polymarket tests pass (14 new) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
@@ -207,6 +207,28 @@ The Judge Agent must:
|
||||
|
||||
7. **Cross-platform signals are the strongest evidence.** When items have `[also on: Reddit, HN]` or similar tags, it means the same story appears across multiple platforms. Lead with these cross-platform findings - they're the most important signals in the research.
|
||||
|
||||
### Prediction Markets (Polymarket)
|
||||
|
||||
**CRITICAL: When Polymarket returns relevant markets, prediction market odds are among the highest-signal data points in your research.** Real money on outcomes cuts through opinion. Treat them as strong evidence, not an afterthought.
|
||||
|
||||
**How to interpret and synthesize Polymarket data:**
|
||||
|
||||
1. **Prefer structural/long-term markets over near-term deadlines.** Championship odds > regular season title. Regime change > near-term strike deadline. IPO/major milestone > incremental update. Presidency > individual state primary. When multiple markets exist, the bigger question is more interesting to the user.
|
||||
|
||||
2. **When the topic is an outcome in a multi-outcome market, call out that specific outcome's odds and movement.** Don't just say "Polymarket has a #1 seed market" - say "Arizona has a 28% chance of being the #1 overall seed, up 10% this month." The user cares about THEIR topic's position in the market.
|
||||
|
||||
3. **Weave odds into the narrative as supporting evidence.** Don't isolate Polymarket data in its own paragraph. Instead: "Final Four buzz is building - Polymarket gives Arizona a 12% chance to win the championship (up 3% this week), and 28% to earn a #1 seed."
|
||||
|
||||
4. **Citation format:** Always include specific odds AND movement. "Polymarket has Arizona at 28% for a #1 seed (up 10% this month)" - not just "per Polymarket."
|
||||
|
||||
5. **When multiple relevant markets exist, highlight 3-5 of the most interesting ones** in your synthesis, ordered by importance (structural > near-term). Don't just pick the highest-volume one.
|
||||
|
||||
**Domain examples of market importance ranking:**
|
||||
- **Sports:** Championship/tournament odds > conference title > regular season > weekly matchup
|
||||
- **Geopolitics:** Regime change/structural outcomes > near-term strike deadlines > sanctions
|
||||
- **Tech/Business:** IPO, major product launch, company milestones > incremental updates
|
||||
- **Elections:** Presidency > primary > individual state
|
||||
|
||||
**Do NOT display stats here - they come at the end, right before the invitation.**
|
||||
|
||||
---
|
||||
@@ -288,7 +310,8 @@ CITATION PRIORITY (most to least preferred):
|
||||
2. r/subreddits from Reddit — "per r/subreddit"
|
||||
3. YouTube channels — "per [channel name] on YouTube" (transcript-backed insights)
|
||||
4. HN discussions — "per HN" or "per hn/username" (developer community signal)
|
||||
5. Web sources — ONLY when Reddit/X/YouTube/HN don't cover that specific fact
|
||||
5. Polymarket — "Polymarket has X at Y% (up/down Z%)" with specific odds and movement
|
||||
6. Web sources — ONLY when Reddit/X/YouTube/HN/Polymarket don't cover that specific fact
|
||||
|
||||
The tool's value is surfacing what PEOPLE are saying, not what journalists wrote.
|
||||
When both a web article and an X post cover the same fact, cite the X post.
|
||||
@@ -339,7 +362,7 @@ KEY PATTERNS from the research:
|
||||
├─ 🔵 X: {N} posts │ {N} likes │ {N} reposts
|
||||
├─ 🔴 YouTube: {N} videos │ {N} views │ {N} with transcripts
|
||||
├─ 🟡 HN: {N} stories │ {N} points │ {N} comments
|
||||
├─ 📊 Polymarket: {N} markets ({short summary of top 2-3 market odds, e.g. "Big 12: 64% Yes, NCAA: 12% Yes"})
|
||||
├─ 📊 Polymarket: {N} markets │ {short summary of up to 5 most relevant market odds, e.g. "Championship: 12%, #1 Seed: 28%, Big 12: 64%, vs Kansas: 71%"}
|
||||
├─ 🌐 Web: {N} pages (supplementary)
|
||||
└─ 🗣️ Top voices: @{handle1} ({N} likes), @{handle2} │ r/{sub1}, r/{sub2}
|
||||
---
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
---
|
||||
title: "feat: Smarter Polymarket synthesis - surface the most interesting markets"
|
||||
type: feat
|
||||
status: completed
|
||||
date: 2026-02-26
|
||||
---
|
||||
|
||||
# feat: Smarter Polymarket Synthesis
|
||||
|
||||
## Overview
|
||||
|
||||
Polymarket data is one of the most powerful signals when relevant - real money on outcomes cuts through opinion. But the current system buries the most interesting markets and gives the LLM zero guidance on how to synthesize prediction market data.
|
||||
|
||||
For "Arizona Basketball", the skill found 2 markets but only highlighted "Big 12 title: 68%" in the stats and synthesis. The user cares MORE about:
|
||||
- NCAA Tournament championship odds (12%, up 3%)
|
||||
- #1 seed odds (85%)
|
||||
- Next game: Arizona vs Kansas (71% to win)
|
||||
|
||||
For "Iran War", the skill found 9 markets with $559M volume but only highlighted "strikes by Feb 28: 10% (down from 65%)". The user wanted regime change / Khamenei odds - the "bigger picture" structural question.
|
||||
|
||||
The problem has two layers: (1) the Python scoring penalizes multi-outcome markets where the topic is an outcome, and (2) the SKILL.md gives the LLM zero instructions for interpreting or highlighting prediction market data.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
### Layer 1: Scoring penalizes the most interesting markets
|
||||
|
||||
`_compute_text_similarity()` (polymarket.py:236) only compares the search topic against the **event title**. It never checks outcome names.
|
||||
|
||||
Example: Market titled "Who will be the #1 overall seed in the 2026 NCAA Tournament?" with outcomes ["Arizona", "Duke", "Houston", "Auburn"]:
|
||||
- Topic: "Arizona Basketball"
|
||||
- text_score: **0.0** (neither "arizona" nor "basketball" in event title)
|
||||
- 30% relevance penalty from this alone
|
||||
|
||||
This means the most contextually interesting markets (seeding, championship, matchup odds) get pushed below less interesting but title-matching markets (Big 12 regular season).
|
||||
|
||||
### Layer 2: SKILL.md has zero Polymarket synthesis guidance
|
||||
|
||||
The SKILL.md Judge Agent section tells the LLM how to weight Reddit (higher), YouTube (high), WebSearch (lower), but says **nothing** about:
|
||||
- How to interpret prediction market probabilities
|
||||
- Which markets are "most interesting" (championship > regular season)
|
||||
- When to lead with prediction market odds vs other sources
|
||||
- How to connect a specific outcome in a multi-outcome market to the user's topic
|
||||
- How to use odds as a signal alongside social media sentiment
|
||||
|
||||
### Layer 3: Stats box loses information
|
||||
|
||||
The Polymarket stats line only has room for 1-2 market highlights. When there are 5+ relevant markets, the user misses the most interesting ones.
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
### 1. Outcome-aware text similarity scoring
|
||||
|
||||
Update `_compute_text_similarity()` to check if the topic appears in any outcome name, with **bidirectional** substring matching. The check must work in both directions since the topic ("Arizona Basketball") is longer than the outcome name ("Arizona").
|
||||
|
||||
Collect outcome names from ALL active markets in the event (not just top market), since Gamma API can structure multi-outcome events as separate binary sub-markets.
|
||||
|
||||
Only match outcomes with probability > 1% to avoid noise from near-zero outcomes.
|
||||
|
||||
```python
|
||||
def _compute_text_similarity(topic: str, title: str, outcomes: list = None) -> float:
|
||||
core = _extract_core_subject(topic).lower()
|
||||
title_lower = title.lower()
|
||||
if not core:
|
||||
return 0.5
|
||||
|
||||
# Full substring match in title
|
||||
if core in title_lower:
|
||||
return 1.0
|
||||
|
||||
# Check if topic appears in any outcome name (bidirectional)
|
||||
if outcomes:
|
||||
core_tokens = set(core.split()) # Hoist outside loop
|
||||
best_outcome_score = 0.0
|
||||
for outcome_name in outcomes:
|
||||
outcome_lower = outcome_name.lower()
|
||||
# Bidirectional substring: "arizona" in "arizona basketball" OR "arizona basketball" in "arizona wildcats game"
|
||||
if core in outcome_lower or outcome_lower in core:
|
||||
best_outcome_score = max(best_outcome_score, 0.85)
|
||||
elif core_tokens & set(outcome_lower.split()):
|
||||
best_outcome_score = max(best_outcome_score, 0.7)
|
||||
if best_outcome_score > 0:
|
||||
return best_outcome_score
|
||||
|
||||
# Token overlap fallback against title
|
||||
topic_tokens = set(core.split())
|
||||
title_tokens = set(title_lower.split())
|
||||
if not topic_tokens:
|
||||
return 0.5
|
||||
overlap = len(topic_tokens & title_tokens)
|
||||
return overlap / len(topic_tokens)
|
||||
```
|
||||
|
||||
### 2. Surface the topic-matching outcome in display
|
||||
|
||||
When the topic matches an outcome name, reorder `outcome_prices` to put the matching outcome first before truncating to top 3. This ensures the LLM sees the user-relevant odds.
|
||||
|
||||
### 3. Add SKILL.md synthesis instructions for Polymarket
|
||||
|
||||
Add a dedicated section telling the LLM:
|
||||
- **Prediction markets are high-signal when relevant.** Real money on outcomes > opinions.
|
||||
- **Prefer markets that answer structural/long-term questions** (championships, regime changes, major milestones) over near-term deadline markets (weekly matchups, short-term event deadlines). When in doubt, the bigger question is more interesting.
|
||||
- **When the topic is an outcome in a multi-outcome market, call out that specific outcome's odds and movement.** Don't just say "Polymarket has a #1 seed market" - say "Arizona has 85% chance of a #1 seed, up from 72%."
|
||||
- **Weave odds into the "What I learned" narrative as supporting evidence.** "Final Four buzz is building - Polymarket gives Arizona a 12% chance to win the championship (up 3% this week), and 85% to earn a #1 seed."
|
||||
- **Citation format:** "Polymarket has Arizona at 85% for a #1 seed (up from 72%)" - include the specific odds and movement, not just "per Polymarket."
|
||||
- **Stats box:** Show up to 5 most relevant markets with odds. If more exist, show count.
|
||||
|
||||
Domain examples:
|
||||
- Sports: championship/tournament odds > regular season title > weekly matchup
|
||||
- Geopolitics: regime change > near-term strike deadline > sanctions
|
||||
- Tech: major milestones (IPO, product launch) > incremental updates
|
||||
- Elections: presidency > primary > individual state
|
||||
|
||||
### 4. Improve stats box template
|
||||
|
||||
Show up to 5 markets with odds, capped for readability:
|
||||
```
|
||||
├─ 📊 Polymarket: 5 markets (Championship: 12%, #1 Seed: 85%, Big 12: 68%, vs Kansas: 71%, NCAA: 12%)
|
||||
```
|
||||
|
||||
### 5. Fix render.py volume label
|
||||
|
||||
The render module labels volume as "vol24h" even when `volume1mo` is the actual data source. Fix to "vol/mo" when monthly volume is used.
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Implementation Plan
|
||||
|
||||
#### Phase 1: Fix text similarity to check outcomes
|
||||
|
||||
- [x] `scripts/lib/polymarket.py` - Update `_compute_text_similarity()` to accept optional `outcomes` parameter with bidirectional substring matching and token overlap
|
||||
- [x] `scripts/lib/polymarket.py` - In `parse_polymarket_response()`, collect outcome names from ALL active markets (not just top market), filter to outcomes with price > 1%, and pass to `_compute_text_similarity()`
|
||||
- [x] `scripts/lib/polymarket.py` - Reorder `outcome_prices` to surface the topic-matching outcome first before truncating to top 3
|
||||
- [x] `fixtures/polymarket_sample.json` - Add fixture event: "Who will be the #1 overall seed in the 2026 NCAA Tournament?" with outcomes ["Arizona", "Duke", "Houston", "Auburn"] where "Arizona" is NOT in the title
|
||||
- [x] `tests/test_polymarket.py` - Add tests for outcome-aware text similarity: bidirectional substring, token overlap, no-match, low-probability filtering
|
||||
- [x] `tests/test_polymarket.py` - Add test: multi-outcome market where topic is an outcome should rank higher than tangential title-match markets
|
||||
- [x] `tests/test_polymarket.py` - Add test: topic-matching outcome is surfaced to front of outcome_prices display
|
||||
|
||||
#### Phase 2: Add SKILL.md Polymarket synthesis instructions
|
||||
|
||||
- [x] `SKILL.md` - Add "Prediction Markets" subsection to the Judge Agent section with:
|
||||
- General heuristic: prefer structural/long-term markets over near-term deadlines
|
||||
- Domain examples (sports, geopolitics, tech, elections)
|
||||
- Citation format with specific odds and movement
|
||||
- Instruction to weave odds into "What I learned" narrative
|
||||
- [x] `SKILL.md` - Add Polymarket to citation priority list (between HN and Web) with format guidance
|
||||
- [x] `SKILL.md` - Update stats box template: show up to 5 markets with odds
|
||||
- [x] `variants/open/references/research.md` - Add condensed Polymarket synthesis guidance matching the open variant's style
|
||||
- [x] `scripts/lib/render.py` - Fix "vol24h" label to "vol/mo" when volume1mo is the data source (changed to "volume")
|
||||
|
||||
#### Phase 3: Tests and verification
|
||||
|
||||
- [x] Run full test suite (229 passed, 5 pre-existing failures unrelated to this change)
|
||||
- [ ] Manual test: `/last30days "Arizona Basketball"` - verify championship, seed, and matchup odds appear in synthesis
|
||||
- [ ] Manual test: `/last30days "Iran War"` - verify regime change and structural outcome markets appear
|
||||
- [x] Run `bash scripts/sync.sh` to deploy
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] Multi-outcome markets where the topic is an outcome (e.g. "Arizona" in a seeding market) get text_score >= 0.7, not 0.0
|
||||
- [x] Bidirectional matching works: "Arizona" as outcome matches topic "Arizona Basketball" (outcome_name in core)
|
||||
- [x] Topic-matching outcome is surfaced to front of outcome_prices display (not hidden in "and N more")
|
||||
- [x] Low-probability outcomes (< 1%) don't trigger outcome matching
|
||||
- [x] SKILL.md instructs the LLM to highlight structural/long-term markets over near-term ones
|
||||
- [x] SKILL.md provides citation format: "Polymarket has X at Y% (up/down Z%)"
|
||||
- [x] Stats box shows up to 5 markets with odds
|
||||
- [x] HN zero-result line is hidden (already fixed - verify on next run)
|
||||
- [x] All existing tests pass + new outcome-aware similarity tests pass
|
||||
- [x] render.py volume label is accurate
|
||||
|
||||
## Dependencies & Risks
|
||||
|
||||
**No blockers.** This is scoring improvements + SKILL.md instruction changes within the existing Polymarket module.
|
||||
|
||||
**Risk: Outcome name matching false positives.** A market "Will Arizona pass AI regulation?" would match "Arizona Basketball" on the word "Arizona" even though it's about the state, not the team. Mitigation: outcome match gets 0.7-0.85 (not 1.0), and volume/liquidity/movement signals still differentiate. A false positive at 0.85 text_score won't outrank a true title match at 1.0.
|
||||
|
||||
**Risk: Common-word false positives.** Words like "war," "AI," "US" could match generic outcomes. Mitigation: at 0.7 text_score (30% weight = 0.21 relevance), this is a small boost that won't override strong volume/liquidity signals from actually relevant markets. Monitor in testing.
|
||||
|
||||
**Risk: LLM still ignores synthesis instructions.** Mitigation: use CRITICAL formatting, specific do/don't examples, and concrete citation format.
|
||||
|
||||
## Sources & References
|
||||
|
||||
- Current text similarity: `scripts/lib/polymarket.py:236`
|
||||
- Render format: `scripts/lib/render.py:282`
|
||||
- SKILL.md synthesis: `SKILL.md:196` (Judge Agent section)
|
||||
- Previous quality ranking plan: `docs/plans/2026-02-25-feat-polymarket-quality-ranking-plan.md`
|
||||
@@ -182,6 +182,37 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "evt-ncaa-seed",
|
||||
"title": "Who will be the #1 overall seed in the 2026 NCAA Tournament?",
|
||||
"slug": "ncaa-1-seed-2026",
|
||||
"active": true,
|
||||
"closed": false,
|
||||
"updatedAt": "2026-02-25T14:00:00.000Z",
|
||||
"volume24hr": 200000,
|
||||
"volume1wk": 900000,
|
||||
"volume1mo": 2500000,
|
||||
"volume": 6000000,
|
||||
"liquidity": 1500000,
|
||||
"competitive": 0.88,
|
||||
"commentCount": 12,
|
||||
"markets": [
|
||||
{
|
||||
"id": "mkt-ncaa-seed-1",
|
||||
"question": "Who will be the #1 overall seed in the 2026 NCAA Tournament?",
|
||||
"active": true,
|
||||
"closed": false,
|
||||
"outcomes": "[\"Duke\", \"Arizona\", \"Houston\", \"Auburn\", \"Michigan\"]",
|
||||
"outcomePrices": "[\"0.30\", \"0.28\", \"0.20\", \"0.12\", \"0.10\"]",
|
||||
"volume": "2500000",
|
||||
"volume24hr": "200000",
|
||||
"liquidity": "1500000",
|
||||
"oneDayPriceChange": 0.02,
|
||||
"oneWeekPriceChange": 0.06,
|
||||
"oneMonthPriceChange": 0.10
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "evt-tangential",
|
||||
"title": "Will AI regulation pass in 2026?",
|
||||
|
||||
@@ -233,22 +233,36 @@ def _parse_outcome_prices(market: Dict[str, Any]) -> List[tuple]:
|
||||
return result
|
||||
|
||||
|
||||
def _compute_text_similarity(topic: str, title: str) -> float:
|
||||
"""Score how well the event title matches the search topic.
|
||||
def _compute_text_similarity(topic: str, title: str, outcomes: List[str] = None) -> float:
|
||||
"""Score how well the event title (or outcome names) match the search topic.
|
||||
|
||||
Returns 0.0-1.0. Substring containment gets full score,
|
||||
token overlap gets proportional score.
|
||||
Returns 0.0-1.0. Title substring match gets 1.0, outcome match gets 0.85/0.7,
|
||||
title token overlap gets proportional score.
|
||||
"""
|
||||
core = _extract_core_subject(topic).lower()
|
||||
title_lower = title.lower()
|
||||
if not core:
|
||||
return 0.5
|
||||
|
||||
# Full substring match
|
||||
# Full substring match in title
|
||||
if core in title_lower:
|
||||
return 1.0
|
||||
|
||||
# Token overlap fallback
|
||||
# Check if topic appears in any outcome name (bidirectional)
|
||||
if outcomes:
|
||||
core_tokens = set(core.split())
|
||||
best_outcome_score = 0.0
|
||||
for outcome_name in outcomes:
|
||||
outcome_lower = outcome_name.lower()
|
||||
# Bidirectional: "arizona" in "arizona basketball" OR "arizona basketball" contains "arizona"
|
||||
if core in outcome_lower or outcome_lower in core:
|
||||
best_outcome_score = max(best_outcome_score, 0.85)
|
||||
elif core_tokens & set(outcome_lower.split()):
|
||||
best_outcome_score = max(best_outcome_score, 0.7)
|
||||
if best_outcome_score > 0:
|
||||
return best_outcome_score
|
||||
|
||||
# Token overlap fallback against title
|
||||
topic_tokens = set(core.split())
|
||||
title_tokens = set(title_lower.split())
|
||||
if not topic_tokens:
|
||||
@@ -325,6 +339,14 @@ def parse_polymarket_response(response: Dict[str, Any], topic: str = "") -> List
|
||||
# Take top market for the event
|
||||
top_market = active_markets[0]
|
||||
|
||||
# Collect outcome names from ALL active markets (not just top) for similarity scoring
|
||||
# Filter to outcomes with price > 1% to avoid noise
|
||||
all_outcome_names = []
|
||||
for m in active_markets:
|
||||
for name, price in _parse_outcome_prices(m):
|
||||
if price > 0.01 and name not in all_outcome_names:
|
||||
all_outcome_names.append(name)
|
||||
|
||||
# Parse outcome prices from top market
|
||||
outcome_prices = _parse_outcome_prices(top_market)
|
||||
|
||||
@@ -360,7 +382,7 @@ def parse_polymarket_response(response: Dict[str, Any], topic: str = "") -> List
|
||||
end_date = None
|
||||
|
||||
# Quality-signal relevance (replaces position-based decay)
|
||||
text_score = _compute_text_similarity(topic, title) if topic else 0.5
|
||||
text_score = _compute_text_similarity(topic, title, all_outcome_names) if topic else 0.5
|
||||
|
||||
# Volume signal: log-scaled monthly volume (most stable signal)
|
||||
vol_raw = event_volume1mo or event_volume1wk or volume24hr
|
||||
@@ -387,6 +409,20 @@ def parse_polymarket_response(response: Dict[str, Any], topic: str = "") -> List
|
||||
0.10 * competitive_score
|
||||
))
|
||||
|
||||
# Surface the topic-matching outcome to the front before truncating
|
||||
if topic and outcome_prices:
|
||||
core = _extract_core_subject(topic).lower()
|
||||
reordered = []
|
||||
rest = []
|
||||
for pair in outcome_prices:
|
||||
name_lower = pair[0].lower()
|
||||
if core in name_lower or name_lower in core:
|
||||
reordered.append(pair)
|
||||
else:
|
||||
rest.append(pair)
|
||||
if reordered:
|
||||
outcome_prices = reordered + rest
|
||||
|
||||
# Top 3 outcomes for multi-outcome markets
|
||||
top_outcomes = outcome_prices[:3]
|
||||
remaining = len(outcome_prices) - 3
|
||||
|
||||
@@ -295,11 +295,11 @@ def render_compact(report: schema.Report, limit: int = 15, missing_keys: str = "
|
||||
parts = []
|
||||
if eng.volume is not None:
|
||||
if eng.volume >= 1_000_000:
|
||||
parts.append(f"${eng.volume/1_000_000:.1f}M vol24h")
|
||||
parts.append(f"${eng.volume/1_000_000:.1f}M volume")
|
||||
elif eng.volume >= 1_000:
|
||||
parts.append(f"${eng.volume/1_000:.0f}K vol24h")
|
||||
parts.append(f"${eng.volume/1_000:.0f}K volume")
|
||||
else:
|
||||
parts.append(f"${eng.volume:.0f} vol24h")
|
||||
parts.append(f"${eng.volume:.0f} volume")
|
||||
if eng.liquidity is not None:
|
||||
if eng.liquidity >= 1_000_000:
|
||||
parts.append(f"${eng.liquidity/1_000_000:.1f}M liquidity")
|
||||
|
||||
+105
-8
@@ -507,6 +507,79 @@ class TestTextSimilarity(unittest.TestCase):
|
||||
score = polymarket._compute_text_similarity("last 7 days Arizona", "Will Arizona win?")
|
||||
self.assertEqual(score, 1.0)
|
||||
|
||||
def test_outcome_substring_match(self):
|
||||
"""Topic 'Arizona' should match outcome 'Arizona' even when title has no overlap."""
|
||||
score = polymarket._compute_text_similarity(
|
||||
"Arizona",
|
||||
"Who will be the #1 overall seed?",
|
||||
outcomes=["Duke", "Arizona", "Houston"],
|
||||
)
|
||||
self.assertEqual(score, 0.85)
|
||||
|
||||
def test_outcome_bidirectional_match(self):
|
||||
"""Topic 'Arizona Basketball' should match outcome 'Arizona' (outcome in core)."""
|
||||
score = polymarket._compute_text_similarity(
|
||||
"Arizona Basketball",
|
||||
"Who will be the #1 overall seed?",
|
||||
outcomes=["Duke", "Arizona", "Houston"],
|
||||
)
|
||||
self.assertEqual(score, 0.85)
|
||||
|
||||
def test_outcome_token_overlap(self):
|
||||
"""Partial token overlap with outcome gets 0.7 when no substring match."""
|
||||
score = polymarket._compute_text_similarity(
|
||||
"Iran War",
|
||||
"Unrelated geopolitics title",
|
||||
outcomes=["War continues", "Peace deal"],
|
||||
)
|
||||
self.assertEqual(score, 0.7)
|
||||
|
||||
def test_outcome_no_match(self):
|
||||
"""No outcome match falls through to title token overlap."""
|
||||
score = polymarket._compute_text_similarity(
|
||||
"Arizona Basketball",
|
||||
"Will AI regulation pass in 2026?",
|
||||
outcomes=["Yes", "No"],
|
||||
)
|
||||
self.assertEqual(score, 0.0)
|
||||
|
||||
def test_outcome_low_price_filtered_by_caller(self):
|
||||
"""Outcomes with price <= 1% should be filtered by the caller, not this function."""
|
||||
# This function doesn't filter - it trusts the caller to pass only relevant outcomes
|
||||
score = polymarket._compute_text_similarity(
|
||||
"Arizona",
|
||||
"Unrelated title",
|
||||
outcomes=["Arizona"],
|
||||
)
|
||||
self.assertEqual(score, 0.85)
|
||||
|
||||
def test_title_match_still_beats_outcome(self):
|
||||
"""Title substring match (1.0) takes priority over outcome match (0.85)."""
|
||||
score = polymarket._compute_text_similarity(
|
||||
"Arizona",
|
||||
"Will Arizona win the tournament?",
|
||||
outcomes=["Arizona", "Duke"],
|
||||
)
|
||||
self.assertEqual(score, 1.0)
|
||||
|
||||
def test_empty_outcomes(self):
|
||||
"""Empty outcomes list falls through to title token overlap."""
|
||||
score = polymarket._compute_text_similarity(
|
||||
"Arizona Basketball",
|
||||
"Unrelated title",
|
||||
outcomes=[],
|
||||
)
|
||||
self.assertEqual(score, 0.0)
|
||||
|
||||
def test_none_outcomes(self):
|
||||
"""None outcomes falls through to title token overlap."""
|
||||
score = polymarket._compute_text_similarity(
|
||||
"Arizona Basketball",
|
||||
"Unrelated title",
|
||||
outcomes=None,
|
||||
)
|
||||
self.assertEqual(score, 0.0)
|
||||
|
||||
|
||||
class TestQualityRanking(unittest.TestCase):
|
||||
"""Verify quality-signal ranking: high-volume matching events rank above tangential ones."""
|
||||
@@ -520,22 +593,23 @@ class TestQualityRanking(unittest.TestCase):
|
||||
"""Arizona markets should rank above AI regulation when topic is 'Arizona Basketball'."""
|
||||
items = polymarket.parse_polymarket_response(self.sample, topic="Arizona Basketball")
|
||||
titles = [item["title"] for item in items]
|
||||
# Arizona events should come before tangential AI regulation event
|
||||
arizona_indices = [i for i, t in enumerate(titles) if "Arizona" in t or "Big 12" in t]
|
||||
# Arizona events (including outcome-matched ones like NCAA seed) should come before tangential
|
||||
arizona_indices = [i for i, t in enumerate(titles) if "Arizona" in t or "Big 12" in t or "NCAA" in t]
|
||||
tangential_indices = [i for i, t in enumerate(titles) if "AI regulation" in t]
|
||||
if tangential_indices:
|
||||
self.assertTrue(max(arizona_indices) < min(tangential_indices),
|
||||
f"Arizona markets should rank above tangential. Order: {titles}")
|
||||
|
||||
def test_high_volume_ranks_above_low_volume(self):
|
||||
"""Among matching events, higher volume should rank higher."""
|
||||
"""Among title-matched events, higher volume should rank higher."""
|
||||
items = polymarket.parse_polymarket_response(self.sample, topic="Arizona Basketball")
|
||||
# Arizona Big 12 has $3.5M monthly volume, Arizona NCAA has $800K
|
||||
# Arizona Big 12 Championship has $3.5M monthly volume, Arizona NCAA Tournament has $800K
|
||||
# Both have "Arizona" in the title (text_score=1.0), so volume breaks the tie
|
||||
big12 = [i for i, item in enumerate(items) if "Big 12 Championship" in item["title"]]
|
||||
ncaa = [i for i, item in enumerate(items) if "NCAA Tournament" in item["title"]]
|
||||
if big12 and ncaa:
|
||||
self.assertLess(big12[0], ncaa[0],
|
||||
"Higher volume Big 12 should rank above lower volume NCAA")
|
||||
ncaa_win = [i for i, item in enumerate(items) if item["title"] == "Will Arizona win the NCAA Tournament?"]
|
||||
if big12 and ncaa_win:
|
||||
self.assertLess(big12[0], ncaa_win[0],
|
||||
"Higher volume Big 12 Championship should rank above lower volume NCAA Tournament win")
|
||||
|
||||
def test_result_cap_applied(self):
|
||||
"""Parse should respect the _cap from search response."""
|
||||
@@ -558,6 +632,29 @@ class TestQualityRanking(unittest.TestCase):
|
||||
relevances = [item["relevance"] for item in items]
|
||||
self.assertEqual(relevances, sorted(relevances, reverse=True))
|
||||
|
||||
def test_ncaa_seed_found_via_outcome_matching(self):
|
||||
"""NCAA seed market should be found when Arizona is an outcome but not in title."""
|
||||
items = polymarket.parse_polymarket_response(self.sample, topic="Arizona Basketball")
|
||||
titles = [item["title"] for item in items]
|
||||
self.assertIn("Who will be the #1 overall seed in the 2026 NCAA Tournament?", titles)
|
||||
|
||||
def test_ncaa_seed_ranks_above_tangential(self):
|
||||
"""NCAA seed market (outcome match) should rank above AI regulation (no match)."""
|
||||
items = polymarket.parse_polymarket_response(self.sample, topic="Arizona Basketball")
|
||||
titles = [item["title"] for item in items]
|
||||
seed_idx = titles.index("Who will be the #1 overall seed in the 2026 NCAA Tournament?")
|
||||
tangential = [i for i, t in enumerate(titles) if "AI regulation" in t]
|
||||
if tangential:
|
||||
self.assertLess(seed_idx, tangential[0],
|
||||
f"NCAA seed should rank above tangential. Order: {titles}")
|
||||
|
||||
def test_outcome_reordering_surfaces_topic(self):
|
||||
"""Arizona should be surfaced to front of outcome_prices when topic matches."""
|
||||
items = polymarket.parse_polymarket_response(self.sample, topic="Arizona Basketball")
|
||||
seed_market = [i for i in items if "seed" in i["title"].lower()][0]
|
||||
# Arizona should be first in outcome_prices (reordered from position 2)
|
||||
self.assertEqual(seed_market["outcome_prices"][0][0], "Arizona")
|
||||
|
||||
|
||||
class TestNormalizePolymarketVolume1mo(unittest.TestCase):
|
||||
"""Verify normalization prefers volume1mo over volume24hr for engagement."""
|
||||
|
||||
@@ -85,6 +85,7 @@ Rules:
|
||||
3. Weight web LOWER (no engagement data)
|
||||
4. Identify cross-source patterns (strongest signals)
|
||||
5. Extract top 3-5 actionable insights
|
||||
6. **Prediction markets are high-signal when relevant** - real money on outcomes cuts through opinion. Prefer structural/long-term markets (championship > regular season, regime change > near-term deadline). When the topic is an outcome in a multi-outcome market, call out that specific outcome's odds and movement. Weave odds into narrative: "Polymarket has X at Y% (up/down Z%)"
|
||||
|
||||
**Ground synthesis in ACTUAL research, not pre-existing knowledge.**
|
||||
|
||||
@@ -128,7 +129,7 @@ All agents reported back!
|
||||
|- Reddit: {N} threads | {N} upvotes | {N} comments
|
||||
|- X: {N} posts | {N} likes | {N} reposts
|
||||
|- YouTube: {N} videos | {N} views | {N} with transcripts
|
||||
|- Polymarket: {N} markets ({short summary of top 2-3 market odds})
|
||||
|- Polymarket: {N} markets | {summary of up to 5 most relevant market odds}
|
||||
|- Web: {N} pages (supplementary)
|
||||
|- Top voices: @{handle1}, @{handle2} | r/{sub1}, r/{sub2}
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user