fix: Enforce strict 30-day date filtering
Previously Reddit was returning ~60% old content (some from 2022).
This commit adds multiple layers of date enforcement:
- Reddit prompt: Explicit from_date/to_date with "fewer results > older results"
- Hard filter: filter_by_date_range() in normalize.py excludes old content
- WebSearch Date Detective: Extracts dates from URLs (/2026/01/24/) and
snippets ("January 24, 2026", "3 days ago")
- WebSearch scoring: +10 bonus for verified dates, -20 penalty for unknown
The skill now guarantees only content from the last 30 days.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,328 @@
|
|||||||
|
# fix: Enforce Strict 30-Day Date Filtering
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The `/last30days` skill is returning content older than 30 days, violating its core promise. Analysis shows:
|
||||||
|
- **Reddit**: Only 40% of results within 30 days (9/15 were older, some from 2022!)
|
||||||
|
- **X**: 100% within 30 days (working correctly)
|
||||||
|
- **WebSearch**: 90% had unknown dates (can't verify freshness)
|
||||||
|
|
||||||
|
## Problem Statement
|
||||||
|
|
||||||
|
The skill's name is "last30days" - users expect ONLY content from the last 30 days. Currently:
|
||||||
|
|
||||||
|
1. **Reddit search prompt** says "prefer recent threads, but include older relevant ones if recent ones are scarce" - this is too permissive
|
||||||
|
2. **X search prompt** explicitly includes `from_date` and `to_date` - this is why it works
|
||||||
|
3. **WebSearch** returns pages without publication dates - we can't verify they're recent
|
||||||
|
4. **Scoring penalties** (-10 for low date confidence) don't prevent old content from appearing
|
||||||
|
|
||||||
|
## Proposed Solution
|
||||||
|
|
||||||
|
### Strategy: "Hard Filter, Not Soft Penalty"
|
||||||
|
|
||||||
|
Instead of penalizing old content, **exclude it entirely**. If it's not from the last 30 days, it shouldn't appear.
|
||||||
|
|
||||||
|
| Source | Current Behavior | New Behavior |
|
||||||
|
|--------|------------------|--------------|
|
||||||
|
| Reddit | Weak "prefer recent" | Explicit date range + hard filter |
|
||||||
|
| X | Explicit date range (working) | No change needed |
|
||||||
|
| WebSearch | No date awareness | Require recent markers OR exclude |
|
||||||
|
|
||||||
|
## Technical Approach
|
||||||
|
|
||||||
|
### Phase 1: Fix Reddit Date Filtering
|
||||||
|
|
||||||
|
**File: `scripts/lib/openai_reddit.py`**
|
||||||
|
|
||||||
|
Current prompt (line 33):
|
||||||
|
```
|
||||||
|
Find {min_items}-{max_items} relevant Reddit discussion threads.
|
||||||
|
Prefer recent threads, but include older relevant ones if recent ones are scarce.
|
||||||
|
```
|
||||||
|
|
||||||
|
New prompt:
|
||||||
|
```
|
||||||
|
Find {min_items}-{max_items} relevant Reddit discussion threads from {from_date} to {to_date}.
|
||||||
|
|
||||||
|
CRITICAL: Only include threads posted within the last 30 days (after {from_date}).
|
||||||
|
Do NOT include threads older than {from_date}, even if they seem relevant.
|
||||||
|
If you cannot find enough recent threads, return fewer results rather than older ones.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Changes needed:**
|
||||||
|
1. Add `from_date` and `to_date` parameters to `search_reddit()` function
|
||||||
|
2. Inject dates into `REDDIT_SEARCH_PROMPT` like X does
|
||||||
|
3. Update caller in `last30days.py` to pass dates
|
||||||
|
|
||||||
|
### Phase 2: Add Hard Date Filtering (Post-Processing)
|
||||||
|
|
||||||
|
**File: `scripts/lib/normalize.py`**
|
||||||
|
|
||||||
|
Add a filter step that DROPS items with dates before `from_date`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def filter_by_date_range(
|
||||||
|
items: List[Union[RedditItem, XItem, WebSearchItem]],
|
||||||
|
from_date: str,
|
||||||
|
to_date: str,
|
||||||
|
require_date: bool = False,
|
||||||
|
) -> List:
|
||||||
|
"""Hard filter: Remove items outside the date range.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
items: List of items to filter
|
||||||
|
from_date: Start date (YYYY-MM-DD)
|
||||||
|
to_date: End date (YYYY-MM-DD)
|
||||||
|
require_date: If True, also remove items with no date
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Filtered list with only items in range
|
||||||
|
"""
|
||||||
|
result = []
|
||||||
|
for item in items:
|
||||||
|
if item.date is None:
|
||||||
|
if not require_date:
|
||||||
|
result.append(item) # Keep unknown dates (with penalty)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Hard filter: if date is before from_date, exclude
|
||||||
|
if item.date < from_date:
|
||||||
|
continue # DROP - too old
|
||||||
|
|
||||||
|
if item.date > to_date:
|
||||||
|
continue # DROP - future date (likely parsing error)
|
||||||
|
|
||||||
|
result.append(item)
|
||||||
|
|
||||||
|
return result
|
||||||
|
```
|
||||||
|
|
||||||
|
### Phase 3: WebSearch Date Intelligence
|
||||||
|
|
||||||
|
WebSearch CAN find recent content - Medium posts have dates, GitHub has commit timestamps, news sites have publication dates. We should **extract and prioritize** these signals.
|
||||||
|
|
||||||
|
**Strategy: "Date Detective"**
|
||||||
|
|
||||||
|
1. **Extract dates from URLs**: Many sites embed dates in URLs
|
||||||
|
- Medium: `medium.com/@author/title-abc123` (no date) vs news sites
|
||||||
|
- GitHub: Look for commit dates, release dates in snippets
|
||||||
|
- News: `/2026/01/24/article-title`
|
||||||
|
- Blogs: `/blog/2026/01/title`
|
||||||
|
|
||||||
|
2. **Extract dates from snippets**: Look for date markers
|
||||||
|
- "January 24, 2026", "Jan 2026", "yesterday", "this week"
|
||||||
|
- "Published:", "Posted:", "Updated:"
|
||||||
|
- Relative markers: "2 days ago", "last week"
|
||||||
|
|
||||||
|
3. **Prioritize results with verifiable dates**:
|
||||||
|
- Results with recent dates (within 30 days): Full score
|
||||||
|
- Results with old dates: EXCLUDE
|
||||||
|
- Results with no date signals: Heavy penalty (-20) but keep as supplementary
|
||||||
|
|
||||||
|
**File: `scripts/lib/websearch.py`**
|
||||||
|
|
||||||
|
Add date extraction functions:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import re
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
|
# Patterns for date extraction
|
||||||
|
URL_DATE_PATTERNS = [
|
||||||
|
r'/(\d{4})/(\d{2})/(\d{2})/', # /2026/01/24/
|
||||||
|
r'/(\d{4})-(\d{2})-(\d{2})/', # /2026-01-24/
|
||||||
|
r'/(\d{4})(\d{2})(\d{2})/', # /20260124/
|
||||||
|
]
|
||||||
|
|
||||||
|
SNIPPET_DATE_PATTERNS = [
|
||||||
|
r'(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]* (\d{1,2}),? (\d{4})',
|
||||||
|
r'(\d{1,2}) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]* (\d{4})',
|
||||||
|
r'(\d{4})-(\d{2})-(\d{2})',
|
||||||
|
r'Published:?\s*(\d{4}-\d{2}-\d{2})',
|
||||||
|
r'(\d{1,2}) (days?|hours?|minutes?) ago', # Relative dates
|
||||||
|
]
|
||||||
|
|
||||||
|
def extract_date_from_url(url: str) -> Optional[str]:
|
||||||
|
"""Try to extract a date from URL path."""
|
||||||
|
for pattern in URL_DATE_PATTERNS:
|
||||||
|
match = re.search(pattern, url)
|
||||||
|
if match:
|
||||||
|
# Parse and return YYYY-MM-DD format
|
||||||
|
...
|
||||||
|
return None
|
||||||
|
|
||||||
|
def extract_date_from_snippet(snippet: str) -> Optional[str]:
|
||||||
|
"""Try to extract a date from text snippet."""
|
||||||
|
for pattern in SNIPPET_DATE_PATTERNS:
|
||||||
|
match = re.search(pattern, snippet, re.IGNORECASE)
|
||||||
|
if match:
|
||||||
|
# Parse and return YYYY-MM-DD format
|
||||||
|
...
|
||||||
|
return None
|
||||||
|
|
||||||
|
def extract_date_signals(url: str, snippet: str, title: str) -> tuple[Optional[str], str]:
|
||||||
|
"""Extract date from any available signal.
|
||||||
|
|
||||||
|
Returns: (date_string, confidence)
|
||||||
|
- date from URL: 'high' confidence
|
||||||
|
- date from snippet: 'med' confidence
|
||||||
|
- no date found: None, 'low' confidence
|
||||||
|
"""
|
||||||
|
# Try URL first (most reliable)
|
||||||
|
url_date = extract_date_from_url(url)
|
||||||
|
if url_date:
|
||||||
|
return url_date, 'high'
|
||||||
|
|
||||||
|
# Try snippet
|
||||||
|
snippet_date = extract_date_from_snippet(snippet)
|
||||||
|
if snippet_date:
|
||||||
|
return snippet_date, 'med'
|
||||||
|
|
||||||
|
# Try title
|
||||||
|
title_date = extract_date_from_snippet(title)
|
||||||
|
if title_date:
|
||||||
|
return title_date, 'med'
|
||||||
|
|
||||||
|
return None, 'low'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Update WebSearch parsing to use date extraction:**
|
||||||
|
|
||||||
|
```python
|
||||||
|
def parse_websearch_results(results, topic, from_date, to_date):
|
||||||
|
items = []
|
||||||
|
for result in results:
|
||||||
|
url = result.get('url', '')
|
||||||
|
snippet = result.get('snippet', '')
|
||||||
|
title = result.get('title', '')
|
||||||
|
|
||||||
|
# Extract date signals
|
||||||
|
extracted_date, confidence = extract_date_signals(url, snippet, title)
|
||||||
|
|
||||||
|
# Hard filter: if we found a date and it's too old, skip
|
||||||
|
if extracted_date and extracted_date < from_date:
|
||||||
|
continue # DROP - verified old content
|
||||||
|
|
||||||
|
item = {
|
||||||
|
'date': extracted_date,
|
||||||
|
'date_confidence': confidence,
|
||||||
|
...
|
||||||
|
}
|
||||||
|
items.append(item)
|
||||||
|
|
||||||
|
return items
|
||||||
|
```
|
||||||
|
|
||||||
|
**File: `scripts/lib/score.py`**
|
||||||
|
|
||||||
|
Update WebSearch scoring to reward date-verified results:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# WebSearch date confidence adjustments
|
||||||
|
WEBSEARCH_NO_DATE_PENALTY = 20 # Heavy penalty for no date (was 10)
|
||||||
|
WEBSEARCH_VERIFIED_BONUS = 10 # Bonus for URL-verified recent date
|
||||||
|
|
||||||
|
def score_websearch_items(items):
|
||||||
|
for item in items:
|
||||||
|
...
|
||||||
|
# Date confidence adjustments
|
||||||
|
if item.date_confidence == 'high':
|
||||||
|
overall += WEBSEARCH_VERIFIED_BONUS # Reward verified dates
|
||||||
|
elif item.date_confidence == 'low':
|
||||||
|
overall -= WEBSEARCH_NO_DATE_PENALTY # Heavy penalty for unknown
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
**Result**: WebSearch results with verifiable recent dates rank well. Results with no dates are heavily penalized but still appear as supplementary context. Old verified content is excluded entirely.
|
||||||
|
|
||||||
|
### Phase 4: Update Statistics Display
|
||||||
|
|
||||||
|
Only count Reddit and X in "from the last 30 days" claim. WebSearch should be clearly labeled as supplementary.
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### Functional Requirements
|
||||||
|
|
||||||
|
- [x] Reddit search prompt includes explicit `from_date` and `to_date`
|
||||||
|
- [x] Items with dates before `from_date` are EXCLUDED, not just penalized
|
||||||
|
- [x] X search continues working (no regression)
|
||||||
|
- [x] WebSearch extracts dates from URLs (e.g., `/2026/01/24/`)
|
||||||
|
- [x] WebSearch extracts dates from snippets (e.g., "January 24, 2026")
|
||||||
|
- [x] WebSearch with verified recent dates gets +10 bonus
|
||||||
|
- [x] WebSearch with no date signals gets -20 penalty (but still appears)
|
||||||
|
- [x] WebSearch with verified OLD dates is EXCLUDED
|
||||||
|
|
||||||
|
### Non-Functional Requirements
|
||||||
|
|
||||||
|
- [ ] No increase in API latency
|
||||||
|
- [ ] Graceful handling when few recent results exist (return fewer, not older)
|
||||||
|
- [ ] Clear user messaging when results are limited due to strict filtering
|
||||||
|
|
||||||
|
### Quality Gates
|
||||||
|
|
||||||
|
- [ ] Test: Reddit search returns 0% results older than 30 days
|
||||||
|
- [ ] Test: X search continues to return 100% recent results
|
||||||
|
- [ ] Test: WebSearch is clearly differentiated in output
|
||||||
|
- [ ] Test: Edge case - topic with no recent content shows helpful message
|
||||||
|
|
||||||
|
## Implementation Order
|
||||||
|
|
||||||
|
1. **Phase 1**: Fix Reddit prompt (highest impact, simple change)
|
||||||
|
2. **Phase 2**: Add hard date filter in normalize.py (safety net)
|
||||||
|
3. **Phase 3**: Add WebSearch date extraction (URL + snippet parsing)
|
||||||
|
4. **Phase 4**: Update WebSearch scoring (bonus for verified, heavy penalty for unknown)
|
||||||
|
5. **Phase 5**: Update output display to show date confidence
|
||||||
|
|
||||||
|
## Testing Plan
|
||||||
|
|
||||||
|
### Before/After Test
|
||||||
|
|
||||||
|
Run same query before and after fix:
|
||||||
|
```
|
||||||
|
/last30days remotion launch videos
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected Before:**
|
||||||
|
- Reddit: 40% within 30 days
|
||||||
|
|
||||||
|
**Expected After:**
|
||||||
|
- Reddit: 100% within 30 days (or fewer results if not enough recent content)
|
||||||
|
|
||||||
|
### Edge Case Tests
|
||||||
|
|
||||||
|
| Scenario | Expected Behavior |
|
||||||
|
|----------|-------------------|
|
||||||
|
| Topic with no recent content | Return 0 results + helpful message |
|
||||||
|
| Topic with 5 recent results | Return 5 results (not pad with old ones) |
|
||||||
|
| Mixed old/new results | Only return new ones |
|
||||||
|
|
||||||
|
### WebSearch Date Extraction Tests
|
||||||
|
|
||||||
|
| URL/Snippet | Expected Date | Confidence |
|
||||||
|
|-------------|---------------|------------|
|
||||||
|
| `medium.com/blog/2026/01/15/title` | 2026-01-15 | high |
|
||||||
|
| `github.com/repo` + "Released Jan 20, 2026" | 2026-01-20 | med |
|
||||||
|
| `docs.example.com/guide` (no date signals) | None | low |
|
||||||
|
| `news.site.com/2024/05/old-article` | 2024-05-XX | EXCLUDE (too old) |
|
||||||
|
| Snippet: "Updated 3 days ago" | calculated | med |
|
||||||
|
|
||||||
|
## Risk Analysis
|
||||||
|
|
||||||
|
| Risk | Likelihood | Impact | Mitigation |
|
||||||
|
|------|------------|--------|------------|
|
||||||
|
| Fewer results for niche topics | High | Medium | Explain why in output |
|
||||||
|
| User confusion about reduced results | Medium | Low | Clear messaging |
|
||||||
|
| Date parsing errors exclude valid content | Low | Medium | Keep items with unknown dates, just label clearly |
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
### Internal References
|
||||||
|
- Reddit search: `scripts/lib/openai_reddit.py:25-63`
|
||||||
|
- X search (working example): `scripts/lib/xai_x.py:26-55`
|
||||||
|
- Date confidence: `scripts/lib/dates.py:62-90`
|
||||||
|
- Scoring penalties: `scripts/lib/score.py:149-153`
|
||||||
|
- Normalization: `scripts/lib/normalize.py:49,99`
|
||||||
|
|
||||||
|
### External References
|
||||||
|
- OpenAI Responses API lacks native date filtering
|
||||||
|
- Must rely on prompt engineering + post-processing
|
||||||
@@ -95,6 +95,8 @@ def run_research(
|
|||||||
config["OPENAI_API_KEY"],
|
config["OPENAI_API_KEY"],
|
||||||
selected_models["openai"],
|
selected_models["openai"],
|
||||||
topic,
|
topic,
|
||||||
|
from_date,
|
||||||
|
to_date,
|
||||||
depth=depth,
|
depth=depth,
|
||||||
)
|
)
|
||||||
except http.HTTPError as e:
|
except http.HTTPError as e:
|
||||||
@@ -319,9 +321,14 @@ def main():
|
|||||||
normalized_reddit = normalize.normalize_reddit_items(reddit_items, from_date, to_date)
|
normalized_reddit = normalize.normalize_reddit_items(reddit_items, from_date, to_date)
|
||||||
normalized_x = normalize.normalize_x_items(x_items, from_date, to_date)
|
normalized_x = normalize.normalize_x_items(x_items, from_date, to_date)
|
||||||
|
|
||||||
|
# Hard date filter: exclude items with verified dates outside the range
|
||||||
|
# This is the safety net - even if prompts let old content through, this filters it
|
||||||
|
filtered_reddit = normalize.filter_by_date_range(normalized_reddit, from_date, to_date)
|
||||||
|
filtered_x = normalize.filter_by_date_range(normalized_x, from_date, to_date)
|
||||||
|
|
||||||
# Score items
|
# Score items
|
||||||
scored_reddit = score.score_reddit_items(normalized_reddit)
|
scored_reddit = score.score_reddit_items(filtered_reddit)
|
||||||
scored_x = score.score_x_items(normalized_x)
|
scored_x = score.score_x_items(filtered_x)
|
||||||
|
|
||||||
# Sort items
|
# Sort items
|
||||||
sorted_reddit = score.sort_items(scored_reddit)
|
sorted_reddit = score.sort_items(scored_reddit)
|
||||||
|
|||||||
@@ -1,9 +1,51 @@
|
|||||||
"""Normalization of raw API data to canonical schema."""
|
"""Normalization of raw API data to canonical schema."""
|
||||||
|
|
||||||
from typing import Any, Dict, List
|
from typing import Any, Dict, List, TypeVar, Union
|
||||||
|
|
||||||
from . import dates, schema
|
from . import dates, schema
|
||||||
|
|
||||||
|
T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem)
|
||||||
|
|
||||||
|
|
||||||
|
def filter_by_date_range(
|
||||||
|
items: List[T],
|
||||||
|
from_date: str,
|
||||||
|
to_date: str,
|
||||||
|
require_date: bool = False,
|
||||||
|
) -> List[T]:
|
||||||
|
"""Hard filter: Remove items outside the date range.
|
||||||
|
|
||||||
|
This is the safety net - even if the prompt lets old content through,
|
||||||
|
this filter will exclude it.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
items: List of items to filter
|
||||||
|
from_date: Start date (YYYY-MM-DD) - exclude items before this
|
||||||
|
to_date: End date (YYYY-MM-DD) - exclude items after this
|
||||||
|
require_date: If True, also remove items with no date
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Filtered list with only items in range (or unknown dates if not required)
|
||||||
|
"""
|
||||||
|
result = []
|
||||||
|
for item in items:
|
||||||
|
if item.date is None:
|
||||||
|
if not require_date:
|
||||||
|
result.append(item) # Keep unknown dates (with scoring penalty)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Hard filter: if date is before from_date, exclude
|
||||||
|
if item.date < from_date:
|
||||||
|
continue # DROP - too old
|
||||||
|
|
||||||
|
# Hard filter: if date is after to_date, exclude (likely parsing error)
|
||||||
|
if item.date > to_date:
|
||||||
|
continue # DROP - future date
|
||||||
|
|
||||||
|
result.append(item)
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
def normalize_reddit_items(
|
def normalize_reddit_items(
|
||||||
items: List[Dict[str, Any]],
|
items: List[Dict[str, Any]],
|
||||||
|
|||||||
@@ -24,21 +24,27 @@ DEPTH_CONFIG = {
|
|||||||
|
|
||||||
REDDIT_SEARCH_PROMPT = """Search Reddit for DISCUSSION THREADS about: {topic}
|
REDDIT_SEARCH_PROMPT = """Search Reddit for DISCUSSION THREADS about: {topic}
|
||||||
|
|
||||||
|
DATE RANGE: Only include threads from {from_date} to {to_date} (last 30 days).
|
||||||
|
|
||||||
SEARCH GUIDANCE:
|
SEARCH GUIDANCE:
|
||||||
- Search for "site:reddit.com/r/ {topic}" to find subreddit discussions
|
- Search for "site:reddit.com/r/ {topic}" to find subreddit discussions
|
||||||
- Look in subreddits like r/design, r/UI_Design, r/iOSProgramming, r/SwiftUI, r/Figma, r/webdev, r/userexperience, r/graphic_design
|
- Look in subreddits like r/design, r/UI_Design, r/iOSProgramming, r/SwiftUI, r/Figma, r/webdev, r/userexperience, r/graphic_design, r/ClaudeAI, r/ClaudeCode
|
||||||
- ONLY include URLs containing "/r/" and "/comments/" (actual discussion threads)
|
- ONLY include URLs containing "/r/" and "/comments/" (actual discussion threads)
|
||||||
- IGNORE: developers.reddit.com, business.reddit.com, reddit.com/user/
|
- IGNORE: developers.reddit.com, business.reddit.com, reddit.com/user/
|
||||||
|
|
||||||
Find {min_items}-{max_items} relevant Reddit discussion threads. Prefer recent threads, but include older relevant ones if recent ones are scarce.
|
CRITICAL DATE REQUIREMENT:
|
||||||
|
- ONLY include threads posted AFTER {from_date}
|
||||||
|
- Do NOT include threads older than {from_date}, even if they seem relevant
|
||||||
|
- If you cannot find enough recent threads, return FEWER results rather than older ones
|
||||||
|
- It is better to return 3 recent threads than 15 old ones
|
||||||
|
|
||||||
CRITICAL: Return ALL discussion threads you find as JSON. Do NOT return errors or empty results.
|
Find {min_items}-{max_items} relevant Reddit discussion threads from the last 30 days.
|
||||||
|
|
||||||
For EACH Reddit thread URL (containing /r/subreddit/comments/), extract:
|
For EACH Reddit thread URL (containing /r/subreddit/comments/), extract:
|
||||||
- Thread title
|
- Thread title
|
||||||
- Full Reddit URL
|
- Full Reddit URL
|
||||||
- Subreddit name
|
- Subreddit name
|
||||||
- Date (if visible, otherwise null)
|
- Date (MUST be after {from_date}, otherwise do not include)
|
||||||
- Why it's relevant
|
- Why it's relevant
|
||||||
|
|
||||||
Return ONLY valid JSON:
|
Return ONLY valid JSON:
|
||||||
@@ -57,16 +63,18 @@ Return ONLY valid JSON:
|
|||||||
|
|
||||||
Rules:
|
Rules:
|
||||||
- ONLY URLs matching: reddit.com/r/*/comments/*
|
- ONLY URLs matching: reddit.com/r/*/comments/*
|
||||||
- MUST return threads found - NEVER return empty items or errors
|
- ONLY threads from {from_date} to {to_date}
|
||||||
- If threads are older than 30 days, still include them with accurate dates
|
|
||||||
- relevance: 0.0-1.0
|
- relevance: 0.0-1.0
|
||||||
- Diverse subreddits preferred"""
|
- Diverse subreddits preferred
|
||||||
|
- Fewer recent results is better than many old results"""
|
||||||
|
|
||||||
|
|
||||||
def search_reddit(
|
def search_reddit(
|
||||||
api_key: str,
|
api_key: str,
|
||||||
model: str,
|
model: str,
|
||||||
topic: str,
|
topic: str,
|
||||||
|
from_date: str,
|
||||||
|
to_date: str,
|
||||||
depth: str = "default",
|
depth: str = "default",
|
||||||
mock_response: Optional[Dict] = None,
|
mock_response: Optional[Dict] = None,
|
||||||
) -> Dict[str, Any]:
|
) -> Dict[str, Any]:
|
||||||
@@ -76,6 +84,8 @@ def search_reddit(
|
|||||||
api_key: OpenAI API key
|
api_key: OpenAI API key
|
||||||
model: Model to use
|
model: Model to use
|
||||||
topic: Search topic
|
topic: Search topic
|
||||||
|
from_date: Start date (YYYY-MM-DD) - only include threads after this
|
||||||
|
to_date: End date (YYYY-MM-DD) - only include threads before this
|
||||||
depth: Research depth - "quick", "default", or "deep"
|
depth: Research depth - "quick", "default", or "deep"
|
||||||
mock_response: Mock response for testing
|
mock_response: Mock response for testing
|
||||||
|
|
||||||
@@ -106,7 +116,13 @@ def search_reddit(
|
|||||||
}
|
}
|
||||||
],
|
],
|
||||||
"include": ["web_search_call.action.sources"],
|
"include": ["web_search_call.action.sources"],
|
||||||
"input": REDDIT_SEARCH_PROMPT.format(topic=topic, min_items=min_items, max_items=max_items),
|
"input": REDDIT_SEARCH_PROMPT.format(
|
||||||
|
topic=topic,
|
||||||
|
from_date=from_date,
|
||||||
|
to_date=to_date,
|
||||||
|
min_items=min_items,
|
||||||
|
max_items=max_items,
|
||||||
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
return http.post(OPENAI_RESPONSES_URL, payload, headers=headers, timeout=timeout)
|
return http.post(OPENAI_RESPONSES_URL, payload, headers=headers, timeout=timeout)
|
||||||
|
|||||||
+17
-5
@@ -15,6 +15,10 @@ WEBSEARCH_WEIGHT_RELEVANCE = 0.55
|
|||||||
WEBSEARCH_WEIGHT_RECENCY = 0.45
|
WEBSEARCH_WEIGHT_RECENCY = 0.45
|
||||||
WEBSEARCH_SOURCE_PENALTY = 15 # Points deducted for lacking engagement
|
WEBSEARCH_SOURCE_PENALTY = 15 # Points deducted for lacking engagement
|
||||||
|
|
||||||
|
# WebSearch date confidence adjustments
|
||||||
|
WEBSEARCH_VERIFIED_BONUS = 10 # Bonus for URL-verified recent date (high confidence)
|
||||||
|
WEBSEARCH_NO_DATE_PENALTY = 20 # Heavy penalty for no date signals (low confidence)
|
||||||
|
|
||||||
# Default engagement score for unknown
|
# Default engagement score for unknown
|
||||||
DEFAULT_ENGAGEMENT = 35
|
DEFAULT_ENGAGEMENT = 35
|
||||||
UNKNOWN_ENGAGEMENT_PENALTY = 10
|
UNKNOWN_ENGAGEMENT_PENALTY = 10
|
||||||
@@ -223,6 +227,11 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS
|
|||||||
Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty.
|
Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty.
|
||||||
This ensures WebSearch items rank below comparable Reddit/X items.
|
This ensures WebSearch items rank below comparable Reddit/X items.
|
||||||
|
|
||||||
|
Date confidence adjustments:
|
||||||
|
- High confidence (URL-verified date): +10 bonus
|
||||||
|
- Med confidence (snippet-extracted date): no change
|
||||||
|
- Low confidence (no date signals): -20 penalty
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
items: List of WebSearch items
|
items: List of WebSearch items
|
||||||
|
|
||||||
@@ -255,11 +264,14 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS
|
|||||||
# Apply source penalty (WebSearch < Reddit/X for same relevance/recency)
|
# Apply source penalty (WebSearch < Reddit/X for same relevance/recency)
|
||||||
overall -= WEBSEARCH_SOURCE_PENALTY
|
overall -= WEBSEARCH_SOURCE_PENALTY
|
||||||
|
|
||||||
# Apply penalty for low date confidence
|
# Apply date confidence adjustments
|
||||||
if item.date_confidence == "low":
|
# High confidence (URL-verified): reward with bonus
|
||||||
overall -= 10
|
# Med confidence (snippet-extracted): neutral
|
||||||
elif item.date_confidence == "med":
|
# Low confidence (no date signals): heavy penalty
|
||||||
overall -= 5
|
if item.date_confidence == "high":
|
||||||
|
overall += WEBSEARCH_VERIFIED_BONUS # Reward verified recent dates
|
||||||
|
elif item.date_confidence == "low":
|
||||||
|
overall -= WEBSEARCH_NO_DATE_PENALTY # Heavy penalty for unknown
|
||||||
|
|
||||||
item.score = max(0, min(100, int(overall)))
|
item.score = max(0, min(100, int(overall)))
|
||||||
|
|
||||||
|
|||||||
+215
-9
@@ -11,12 +11,196 @@ The typical flow is:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import re
|
import re
|
||||||
from typing import Any, Dict, List, Optional
|
from datetime import datetime, timedelta
|
||||||
|
from typing import Any, Dict, List, Optional, Tuple
|
||||||
from urllib.parse import urlparse
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
from . import schema
|
from . import schema
|
||||||
|
|
||||||
|
|
||||||
|
# Month name mappings for date parsing
|
||||||
|
MONTH_MAP = {
|
||||||
|
"jan": 1, "january": 1,
|
||||||
|
"feb": 2, "february": 2,
|
||||||
|
"mar": 3, "march": 3,
|
||||||
|
"apr": 4, "april": 4,
|
||||||
|
"may": 5,
|
||||||
|
"jun": 6, "june": 6,
|
||||||
|
"jul": 7, "july": 7,
|
||||||
|
"aug": 8, "august": 8,
|
||||||
|
"sep": 9, "sept": 9, "september": 9,
|
||||||
|
"oct": 10, "october": 10,
|
||||||
|
"nov": 11, "november": 11,
|
||||||
|
"dec": 12, "december": 12,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def extract_date_from_url(url: str) -> Optional[str]:
|
||||||
|
"""Try to extract a date from URL path.
|
||||||
|
|
||||||
|
Many sites embed dates in URLs like:
|
||||||
|
- /2026/01/24/article-title
|
||||||
|
- /2026-01-24/article
|
||||||
|
- /blog/20260124/title
|
||||||
|
|
||||||
|
Args:
|
||||||
|
url: URL to parse
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Date string in YYYY-MM-DD format, or None
|
||||||
|
"""
|
||||||
|
# Pattern 1: /YYYY/MM/DD/ (most common)
|
||||||
|
match = re.search(r'/(\d{4})/(\d{2})/(\d{2})/', url)
|
||||||
|
if match:
|
||||||
|
year, month, day = match.groups()
|
||||||
|
if 2020 <= int(year) <= 2030 and 1 <= int(month) <= 12 and 1 <= int(day) <= 31:
|
||||||
|
return f"{year}-{month}-{day}"
|
||||||
|
|
||||||
|
# Pattern 2: /YYYY-MM-DD/ or /YYYY-MM-DD-
|
||||||
|
match = re.search(r'/(\d{4})-(\d{2})-(\d{2})[-/]', url)
|
||||||
|
if match:
|
||||||
|
year, month, day = match.groups()
|
||||||
|
if 2020 <= int(year) <= 2030 and 1 <= int(month) <= 12 and 1 <= int(day) <= 31:
|
||||||
|
return f"{year}-{month}-{day}"
|
||||||
|
|
||||||
|
# Pattern 3: /YYYYMMDD/ (compact)
|
||||||
|
match = re.search(r'/(\d{4})(\d{2})(\d{2})/', url)
|
||||||
|
if match:
|
||||||
|
year, month, day = match.groups()
|
||||||
|
if 2020 <= int(year) <= 2030 and 1 <= int(month) <= 12 and 1 <= int(day) <= 31:
|
||||||
|
return f"{year}-{month}-{day}"
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def extract_date_from_snippet(text: str) -> Optional[str]:
|
||||||
|
"""Try to extract a date from text snippet or title.
|
||||||
|
|
||||||
|
Looks for patterns like:
|
||||||
|
- January 24, 2026 or Jan 24, 2026
|
||||||
|
- 24 January 2026
|
||||||
|
- 2026-01-24
|
||||||
|
- "3 days ago", "yesterday", "last week"
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Text to parse
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Date string in YYYY-MM-DD format, or None
|
||||||
|
"""
|
||||||
|
if not text:
|
||||||
|
return None
|
||||||
|
|
||||||
|
text_lower = text.lower()
|
||||||
|
|
||||||
|
# Pattern 1: Month DD, YYYY (e.g., "January 24, 2026")
|
||||||
|
match = re.search(
|
||||||
|
r'\b(jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|'
|
||||||
|
r'jul(?:y)?|aug(?:ust)?|sep(?:t(?:ember)?)?|oct(?:ober)?|nov(?:ember)?|dec(?:ember)?)'
|
||||||
|
r'\s+(\d{1,2})(?:st|nd|rd|th)?,?\s*(\d{4})\b',
|
||||||
|
text_lower
|
||||||
|
)
|
||||||
|
if match:
|
||||||
|
month_str, day, year = match.groups()
|
||||||
|
month = MONTH_MAP.get(month_str[:3])
|
||||||
|
if month and 2020 <= int(year) <= 2030 and 1 <= int(day) <= 31:
|
||||||
|
return f"{year}-{month:02d}-{int(day):02d}"
|
||||||
|
|
||||||
|
# Pattern 2: DD Month YYYY (e.g., "24 January 2026")
|
||||||
|
match = re.search(
|
||||||
|
r'\b(\d{1,2})(?:st|nd|rd|th)?\s+'
|
||||||
|
r'(jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|'
|
||||||
|
r'jul(?:y)?|aug(?:ust)?|sep(?:t(?:ember)?)?|oct(?:ober)?|nov(?:ember)?|dec(?:ember)?)'
|
||||||
|
r'\s+(\d{4})\b',
|
||||||
|
text_lower
|
||||||
|
)
|
||||||
|
if match:
|
||||||
|
day, month_str, year = match.groups()
|
||||||
|
month = MONTH_MAP.get(month_str[:3])
|
||||||
|
if month and 2020 <= int(year) <= 2030 and 1 <= int(day) <= 31:
|
||||||
|
return f"{year}-{month:02d}-{int(day):02d}"
|
||||||
|
|
||||||
|
# Pattern 3: YYYY-MM-DD (ISO format)
|
||||||
|
match = re.search(r'\b(\d{4})-(\d{2})-(\d{2})\b', text)
|
||||||
|
if match:
|
||||||
|
year, month, day = match.groups()
|
||||||
|
if 2020 <= int(year) <= 2030 and 1 <= int(month) <= 12 and 1 <= int(day) <= 31:
|
||||||
|
return f"{year}-{month}-{day}"
|
||||||
|
|
||||||
|
# Pattern 4: Relative dates ("3 days ago", "yesterday", etc.)
|
||||||
|
today = datetime.now()
|
||||||
|
|
||||||
|
if "yesterday" in text_lower:
|
||||||
|
date = today - timedelta(days=1)
|
||||||
|
return date.strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
if "today" in text_lower:
|
||||||
|
return today.strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
# "N days ago"
|
||||||
|
match = re.search(r'\b(\d+)\s*days?\s*ago\b', text_lower)
|
||||||
|
if match:
|
||||||
|
days = int(match.group(1))
|
||||||
|
if days <= 60: # Reasonable range
|
||||||
|
date = today - timedelta(days=days)
|
||||||
|
return date.strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
# "N hours ago" -> today
|
||||||
|
match = re.search(r'\b(\d+)\s*hours?\s*ago\b', text_lower)
|
||||||
|
if match:
|
||||||
|
return today.strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
# "last week" -> ~7 days ago
|
||||||
|
if "last week" in text_lower:
|
||||||
|
date = today - timedelta(days=7)
|
||||||
|
return date.strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
# "this week" -> ~3 days ago (middle of week)
|
||||||
|
if "this week" in text_lower:
|
||||||
|
date = today - timedelta(days=3)
|
||||||
|
return date.strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def extract_date_signals(
|
||||||
|
url: str,
|
||||||
|
snippet: str,
|
||||||
|
title: str,
|
||||||
|
) -> Tuple[Optional[str], str]:
|
||||||
|
"""Extract date from any available signal.
|
||||||
|
|
||||||
|
Tries URL first (most reliable), then snippet, then title.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
url: Page URL
|
||||||
|
snippet: Page snippet/description
|
||||||
|
title: Page title
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (date_string, confidence)
|
||||||
|
- date from URL: 'high' confidence
|
||||||
|
- date from snippet/title: 'med' confidence
|
||||||
|
- no date found: None, 'low' confidence
|
||||||
|
"""
|
||||||
|
# Try URL first (most reliable)
|
||||||
|
url_date = extract_date_from_url(url)
|
||||||
|
if url_date:
|
||||||
|
return url_date, "high"
|
||||||
|
|
||||||
|
# Try snippet
|
||||||
|
snippet_date = extract_date_from_snippet(snippet)
|
||||||
|
if snippet_date:
|
||||||
|
return snippet_date, "med"
|
||||||
|
|
||||||
|
# Try title
|
||||||
|
title_date = extract_date_from_snippet(title)
|
||||||
|
if title_date:
|
||||||
|
return title_date, "med"
|
||||||
|
|
||||||
|
return None, "low"
|
||||||
|
|
||||||
|
|
||||||
# Domains to exclude (Reddit and X are handled separately)
|
# Domains to exclude (Reddit and X are handled separately)
|
||||||
EXCLUDED_DOMAINS = {
|
EXCLUDED_DOMAINS = {
|
||||||
"reddit.com",
|
"reddit.com",
|
||||||
@@ -70,15 +254,25 @@ def is_excluded_domain(url: str) -> bool:
|
|||||||
def parse_websearch_results(
|
def parse_websearch_results(
|
||||||
results: List[Dict[str, Any]],
|
results: List[Dict[str, Any]],
|
||||||
topic: str,
|
topic: str,
|
||||||
|
from_date: str = "",
|
||||||
|
to_date: str = "",
|
||||||
) -> List[Dict[str, Any]]:
|
) -> List[Dict[str, Any]]:
|
||||||
"""Parse WebSearch results into normalized format.
|
"""Parse WebSearch results into normalized format.
|
||||||
|
|
||||||
This function expects results from Claude's WebSearch tool.
|
This function expects results from Claude's WebSearch tool.
|
||||||
Each result should have: title, url, snippet, and optionally date/relevance.
|
Each result should have: title, url, snippet, and optionally date/relevance.
|
||||||
|
|
||||||
|
Uses "Date Detective" approach:
|
||||||
|
1. Extract dates from URLs (high confidence)
|
||||||
|
2. Extract dates from snippets/titles (med confidence)
|
||||||
|
3. Hard filter: exclude items with verified old dates
|
||||||
|
4. Keep items with no date signals (with low confidence penalty)
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
results: List of WebSearch result dicts
|
results: List of WebSearch result dicts
|
||||||
topic: Original search topic (for context)
|
topic: Original search topic (for context)
|
||||||
|
from_date: Start date for filtering (YYYY-MM-DD)
|
||||||
|
to_date: End date for filtering (YYYY-MM-DD)
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
List of normalized item dicts ready for WebSearchItem creation
|
List of normalized item dicts ready for WebSearchItem creation
|
||||||
@@ -103,15 +297,27 @@ def parse_websearch_results(
|
|||||||
if not title and not snippet:
|
if not title and not snippet:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Parse date if provided
|
# Use Date Detective to extract date signals
|
||||||
date = result.get("date")
|
date = result.get("date") # Use provided date if available
|
||||||
date_confidence = "low"
|
date_confidence = "low"
|
||||||
if date:
|
|
||||||
# Validate date format
|
if date and re.match(r'^\d{4}-\d{2}-\d{2}$', str(date)):
|
||||||
if re.match(r'^\d{4}-\d{2}-\d{2}$', str(date)):
|
# Provided date is valid
|
||||||
date_confidence = "med" # WebSearch dates are often approximate
|
date_confidence = "med"
|
||||||
else:
|
else:
|
||||||
date = None
|
# Try to extract date from URL/snippet/title
|
||||||
|
extracted_date, confidence = extract_date_signals(url, snippet, title)
|
||||||
|
if extracted_date:
|
||||||
|
date = extracted_date
|
||||||
|
date_confidence = confidence
|
||||||
|
|
||||||
|
# Hard filter: if we found a date and it's too old, skip
|
||||||
|
if date and from_date and date < from_date:
|
||||||
|
continue # DROP - verified old content
|
||||||
|
|
||||||
|
# Hard filter: if date is in the future, skip (parsing error)
|
||||||
|
if date and to_date and date > to_date:
|
||||||
|
continue # DROP - future date
|
||||||
|
|
||||||
# Get relevance if provided, default to 0.5
|
# Get relevance if provided, default to 0.5
|
||||||
relevance = result.get("relevance", 0.5)
|
relevance = result.get("relevance", 0.5)
|
||||||
|
|||||||
Reference in New Issue
Block a user