fix: Enforce strict 30-day date filtering

Previously Reddit was returning ~60% old content (some from 2022).
This commit adds multiple layers of date enforcement:

- Reddit prompt: Explicit from_date/to_date with "fewer results > older results"
- Hard filter: filter_by_date_range() in normalize.py excludes old content
- WebSearch Date Detective: Extracts dates from URLs (/2026/01/24/) and
  snippets ("January 24, 2026", "3 days ago")
- WebSearch scoring: +10 bonus for verified dates, -20 penalty for unknown

The skill now guarantees only content from the last 30 days.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
2026-01-24 12:23:34 -08:00
parent 354e8bad03
commit 18696e7b05
6 changed files with 636 additions and 25 deletions
+328
View File
@@ -0,0 +1,328 @@
# fix: Enforce Strict 30-Day Date Filtering
## Overview
The `/last30days` skill is returning content older than 30 days, violating its core promise. Analysis shows:
- **Reddit**: Only 40% of results within 30 days (9/15 were older, some from 2022!)
- **X**: 100% within 30 days (working correctly)
- **WebSearch**: 90% had unknown dates (can't verify freshness)
## Problem Statement
The skill's name is "last30days" - users expect ONLY content from the last 30 days. Currently:
1. **Reddit search prompt** says "prefer recent threads, but include older relevant ones if recent ones are scarce" - this is too permissive
2. **X search prompt** explicitly includes `from_date` and `to_date` - this is why it works
3. **WebSearch** returns pages without publication dates - we can't verify they're recent
4. **Scoring penalties** (-10 for low date confidence) don't prevent old content from appearing
## Proposed Solution
### Strategy: "Hard Filter, Not Soft Penalty"
Instead of penalizing old content, **exclude it entirely**. If it's not from the last 30 days, it shouldn't appear.
| Source | Current Behavior | New Behavior |
|--------|------------------|--------------|
| Reddit | Weak "prefer recent" | Explicit date range + hard filter |
| X | Explicit date range (working) | No change needed |
| WebSearch | No date awareness | Require recent markers OR exclude |
## Technical Approach
### Phase 1: Fix Reddit Date Filtering
**File: `scripts/lib/openai_reddit.py`**
Current prompt (line 33):
```
Find {min_items}-{max_items} relevant Reddit discussion threads.
Prefer recent threads, but include older relevant ones if recent ones are scarce.
```
New prompt:
```
Find {min_items}-{max_items} relevant Reddit discussion threads from {from_date} to {to_date}.
CRITICAL: Only include threads posted within the last 30 days (after {from_date}).
Do NOT include threads older than {from_date}, even if they seem relevant.
If you cannot find enough recent threads, return fewer results rather than older ones.
```
**Changes needed:**
1. Add `from_date` and `to_date` parameters to `search_reddit()` function
2. Inject dates into `REDDIT_SEARCH_PROMPT` like X does
3. Update caller in `last30days.py` to pass dates
### Phase 2: Add Hard Date Filtering (Post-Processing)
**File: `scripts/lib/normalize.py`**
Add a filter step that DROPS items with dates before `from_date`:
```python
def filter_by_date_range(
items: List[Union[RedditItem, XItem, WebSearchItem]],
from_date: str,
to_date: str,
require_date: bool = False,
) -> List:
"""Hard filter: Remove items outside the date range.
Args:
items: List of items to filter
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
require_date: If True, also remove items with no date
Returns:
Filtered list with only items in range
"""
result = []
for item in items:
if item.date is None:
if not require_date:
result.append(item) # Keep unknown dates (with penalty)
continue
# Hard filter: if date is before from_date, exclude
if item.date < from_date:
continue # DROP - too old
if item.date > to_date:
continue # DROP - future date (likely parsing error)
result.append(item)
return result
```
### Phase 3: WebSearch Date Intelligence
WebSearch CAN find recent content - Medium posts have dates, GitHub has commit timestamps, news sites have publication dates. We should **extract and prioritize** these signals.
**Strategy: "Date Detective"**
1. **Extract dates from URLs**: Many sites embed dates in URLs
- Medium: `medium.com/@author/title-abc123` (no date) vs news sites
- GitHub: Look for commit dates, release dates in snippets
- News: `/2026/01/24/article-title`
- Blogs: `/blog/2026/01/title`
2. **Extract dates from snippets**: Look for date markers
- "January 24, 2026", "Jan 2026", "yesterday", "this week"
- "Published:", "Posted:", "Updated:"
- Relative markers: "2 days ago", "last week"
3. **Prioritize results with verifiable dates**:
- Results with recent dates (within 30 days): Full score
- Results with old dates: EXCLUDE
- Results with no date signals: Heavy penalty (-20) but keep as supplementary
**File: `scripts/lib/websearch.py`**
Add date extraction functions:
```python
import re
from datetime import datetime, timedelta
# Patterns for date extraction
URL_DATE_PATTERNS = [
r'/(\d{4})/(\d{2})/(\d{2})/', # /2026/01/24/
r'/(\d{4})-(\d{2})-(\d{2})/', # /2026-01-24/
r'/(\d{4})(\d{2})(\d{2})/', # /20260124/
]
SNIPPET_DATE_PATTERNS = [
r'(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]* (\d{1,2}),? (\d{4})',
r'(\d{1,2}) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]* (\d{4})',
r'(\d{4})-(\d{2})-(\d{2})',
r'Published:?\s*(\d{4}-\d{2}-\d{2})',
r'(\d{1,2}) (days?|hours?|minutes?) ago', # Relative dates
]
def extract_date_from_url(url: str) -> Optional[str]:
"""Try to extract a date from URL path."""
for pattern in URL_DATE_PATTERNS:
match = re.search(pattern, url)
if match:
# Parse and return YYYY-MM-DD format
...
return None
def extract_date_from_snippet(snippet: str) -> Optional[str]:
"""Try to extract a date from text snippet."""
for pattern in SNIPPET_DATE_PATTERNS:
match = re.search(pattern, snippet, re.IGNORECASE)
if match:
# Parse and return YYYY-MM-DD format
...
return None
def extract_date_signals(url: str, snippet: str, title: str) -> tuple[Optional[str], str]:
"""Extract date from any available signal.
Returns: (date_string, confidence)
- date from URL: 'high' confidence
- date from snippet: 'med' confidence
- no date found: None, 'low' confidence
"""
# Try URL first (most reliable)
url_date = extract_date_from_url(url)
if url_date:
return url_date, 'high'
# Try snippet
snippet_date = extract_date_from_snippet(snippet)
if snippet_date:
return snippet_date, 'med'
# Try title
title_date = extract_date_from_snippet(title)
if title_date:
return title_date, 'med'
return None, 'low'
```
**Update WebSearch parsing to use date extraction:**
```python
def parse_websearch_results(results, topic, from_date, to_date):
items = []
for result in results:
url = result.get('url', '')
snippet = result.get('snippet', '')
title = result.get('title', '')
# Extract date signals
extracted_date, confidence = extract_date_signals(url, snippet, title)
# Hard filter: if we found a date and it's too old, skip
if extracted_date and extracted_date < from_date:
continue # DROP - verified old content
item = {
'date': extracted_date,
'date_confidence': confidence,
...
}
items.append(item)
return items
```
**File: `scripts/lib/score.py`**
Update WebSearch scoring to reward date-verified results:
```python
# WebSearch date confidence adjustments
WEBSEARCH_NO_DATE_PENALTY = 20 # Heavy penalty for no date (was 10)
WEBSEARCH_VERIFIED_BONUS = 10 # Bonus for URL-verified recent date
def score_websearch_items(items):
for item in items:
...
# Date confidence adjustments
if item.date_confidence == 'high':
overall += WEBSEARCH_VERIFIED_BONUS # Reward verified dates
elif item.date_confidence == 'low':
overall -= WEBSEARCH_NO_DATE_PENALTY # Heavy penalty for unknown
...
```
**Result**: WebSearch results with verifiable recent dates rank well. Results with no dates are heavily penalized but still appear as supplementary context. Old verified content is excluded entirely.
### Phase 4: Update Statistics Display
Only count Reddit and X in "from the last 30 days" claim. WebSearch should be clearly labeled as supplementary.
## Acceptance Criteria
### Functional Requirements
- [x] Reddit search prompt includes explicit `from_date` and `to_date`
- [x] Items with dates before `from_date` are EXCLUDED, not just penalized
- [x] X search continues working (no regression)
- [x] WebSearch extracts dates from URLs (e.g., `/2026/01/24/`)
- [x] WebSearch extracts dates from snippets (e.g., "January 24, 2026")
- [x] WebSearch with verified recent dates gets +10 bonus
- [x] WebSearch with no date signals gets -20 penalty (but still appears)
- [x] WebSearch with verified OLD dates is EXCLUDED
### Non-Functional Requirements
- [ ] No increase in API latency
- [ ] Graceful handling when few recent results exist (return fewer, not older)
- [ ] Clear user messaging when results are limited due to strict filtering
### Quality Gates
- [ ] Test: Reddit search returns 0% results older than 30 days
- [ ] Test: X search continues to return 100% recent results
- [ ] Test: WebSearch is clearly differentiated in output
- [ ] Test: Edge case - topic with no recent content shows helpful message
## Implementation Order
1. **Phase 1**: Fix Reddit prompt (highest impact, simple change)
2. **Phase 2**: Add hard date filter in normalize.py (safety net)
3. **Phase 3**: Add WebSearch date extraction (URL + snippet parsing)
4. **Phase 4**: Update WebSearch scoring (bonus for verified, heavy penalty for unknown)
5. **Phase 5**: Update output display to show date confidence
## Testing Plan
### Before/After Test
Run same query before and after fix:
```
/last30days remotion launch videos
```
**Expected Before:**
- Reddit: 40% within 30 days
**Expected After:**
- Reddit: 100% within 30 days (or fewer results if not enough recent content)
### Edge Case Tests
| Scenario | Expected Behavior |
|----------|-------------------|
| Topic with no recent content | Return 0 results + helpful message |
| Topic with 5 recent results | Return 5 results (not pad with old ones) |
| Mixed old/new results | Only return new ones |
### WebSearch Date Extraction Tests
| URL/Snippet | Expected Date | Confidence |
|-------------|---------------|------------|
| `medium.com/blog/2026/01/15/title` | 2026-01-15 | high |
| `github.com/repo` + "Released Jan 20, 2026" | 2026-01-20 | med |
| `docs.example.com/guide` (no date signals) | None | low |
| `news.site.com/2024/05/old-article` | 2024-05-XX | EXCLUDE (too old) |
| Snippet: "Updated 3 days ago" | calculated | med |
## Risk Analysis
| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| Fewer results for niche topics | High | Medium | Explain why in output |
| User confusion about reduced results | Medium | Low | Clear messaging |
| Date parsing errors exclude valid content | Low | Medium | Keep items with unknown dates, just label clearly |
## References
### Internal References
- Reddit search: `scripts/lib/openai_reddit.py:25-63`
- X search (working example): `scripts/lib/xai_x.py:26-55`
- Date confidence: `scripts/lib/dates.py:62-90`
- Scoring penalties: `scripts/lib/score.py:149-153`
- Normalization: `scripts/lib/normalize.py:49,99`
### External References
- OpenAI Responses API lacks native date filtering
- Must rely on prompt engineering + post-processing
+9 -2
View File
@@ -95,6 +95,8 @@ def run_research(
config["OPENAI_API_KEY"], config["OPENAI_API_KEY"],
selected_models["openai"], selected_models["openai"],
topic, topic,
from_date,
to_date,
depth=depth, depth=depth,
) )
except http.HTTPError as e: except http.HTTPError as e:
@@ -319,9 +321,14 @@ def main():
normalized_reddit = normalize.normalize_reddit_items(reddit_items, from_date, to_date) normalized_reddit = normalize.normalize_reddit_items(reddit_items, from_date, to_date)
normalized_x = normalize.normalize_x_items(x_items, from_date, to_date) normalized_x = normalize.normalize_x_items(x_items, from_date, to_date)
# Hard date filter: exclude items with verified dates outside the range
# This is the safety net - even if prompts let old content through, this filters it
filtered_reddit = normalize.filter_by_date_range(normalized_reddit, from_date, to_date)
filtered_x = normalize.filter_by_date_range(normalized_x, from_date, to_date)
# Score items # Score items
scored_reddit = score.score_reddit_items(normalized_reddit) scored_reddit = score.score_reddit_items(filtered_reddit)
scored_x = score.score_x_items(normalized_x) scored_x = score.score_x_items(filtered_x)
# Sort items # Sort items
sorted_reddit = score.sort_items(scored_reddit) sorted_reddit = score.sort_items(scored_reddit)
+43 -1
View File
@@ -1,9 +1,51 @@
"""Normalization of raw API data to canonical schema.""" """Normalization of raw API data to canonical schema."""
from typing import Any, Dict, List from typing import Any, Dict, List, TypeVar, Union
from . import dates, schema from . import dates, schema
T = TypeVar("T", schema.RedditItem, schema.XItem, schema.WebSearchItem)
def filter_by_date_range(
items: List[T],
from_date: str,
to_date: str,
require_date: bool = False,
) -> List[T]:
"""Hard filter: Remove items outside the date range.
This is the safety net - even if the prompt lets old content through,
this filter will exclude it.
Args:
items: List of items to filter
from_date: Start date (YYYY-MM-DD) - exclude items before this
to_date: End date (YYYY-MM-DD) - exclude items after this
require_date: If True, also remove items with no date
Returns:
Filtered list with only items in range (or unknown dates if not required)
"""
result = []
for item in items:
if item.date is None:
if not require_date:
result.append(item) # Keep unknown dates (with scoring penalty)
continue
# Hard filter: if date is before from_date, exclude
if item.date < from_date:
continue # DROP - too old
# Hard filter: if date is after to_date, exclude (likely parsing error)
if item.date > to_date:
continue # DROP - future date
result.append(item)
return result
def normalize_reddit_items( def normalize_reddit_items(
items: List[Dict[str, Any]], items: List[Dict[str, Any]],
+24 -8
View File
@@ -24,21 +24,27 @@ DEPTH_CONFIG = {
REDDIT_SEARCH_PROMPT = """Search Reddit for DISCUSSION THREADS about: {topic} REDDIT_SEARCH_PROMPT = """Search Reddit for DISCUSSION THREADS about: {topic}
DATE RANGE: Only include threads from {from_date} to {to_date} (last 30 days).
SEARCH GUIDANCE: SEARCH GUIDANCE:
- Search for "site:reddit.com/r/ {topic}" to find subreddit discussions - Search for "site:reddit.com/r/ {topic}" to find subreddit discussions
- Look in subreddits like r/design, r/UI_Design, r/iOSProgramming, r/SwiftUI, r/Figma, r/webdev, r/userexperience, r/graphic_design - Look in subreddits like r/design, r/UI_Design, r/iOSProgramming, r/SwiftUI, r/Figma, r/webdev, r/userexperience, r/graphic_design, r/ClaudeAI, r/ClaudeCode
- ONLY include URLs containing "/r/" and "/comments/" (actual discussion threads) - ONLY include URLs containing "/r/" and "/comments/" (actual discussion threads)
- IGNORE: developers.reddit.com, business.reddit.com, reddit.com/user/ - IGNORE: developers.reddit.com, business.reddit.com, reddit.com/user/
Find {min_items}-{max_items} relevant Reddit discussion threads. Prefer recent threads, but include older relevant ones if recent ones are scarce. CRITICAL DATE REQUIREMENT:
- ONLY include threads posted AFTER {from_date}
- Do NOT include threads older than {from_date}, even if they seem relevant
- If you cannot find enough recent threads, return FEWER results rather than older ones
- It is better to return 3 recent threads than 15 old ones
CRITICAL: Return ALL discussion threads you find as JSON. Do NOT return errors or empty results. Find {min_items}-{max_items} relevant Reddit discussion threads from the last 30 days.
For EACH Reddit thread URL (containing /r/subreddit/comments/), extract: For EACH Reddit thread URL (containing /r/subreddit/comments/), extract:
- Thread title - Thread title
- Full Reddit URL - Full Reddit URL
- Subreddit name - Subreddit name
- Date (if visible, otherwise null) - Date (MUST be after {from_date}, otherwise do not include)
- Why it's relevant - Why it's relevant
Return ONLY valid JSON: Return ONLY valid JSON:
@@ -57,16 +63,18 @@ Return ONLY valid JSON:
Rules: Rules:
- ONLY URLs matching: reddit.com/r/*/comments/* - ONLY URLs matching: reddit.com/r/*/comments/*
- MUST return threads found - NEVER return empty items or errors - ONLY threads from {from_date} to {to_date}
- If threads are older than 30 days, still include them with accurate dates
- relevance: 0.0-1.0 - relevance: 0.0-1.0
- Diverse subreddits preferred""" - Diverse subreddits preferred
- Fewer recent results is better than many old results"""
def search_reddit( def search_reddit(
api_key: str, api_key: str,
model: str, model: str,
topic: str, topic: str,
from_date: str,
to_date: str,
depth: str = "default", depth: str = "default",
mock_response: Optional[Dict] = None, mock_response: Optional[Dict] = None,
) -> Dict[str, Any]: ) -> Dict[str, Any]:
@@ -76,6 +84,8 @@ def search_reddit(
api_key: OpenAI API key api_key: OpenAI API key
model: Model to use model: Model to use
topic: Search topic topic: Search topic
from_date: Start date (YYYY-MM-DD) - only include threads after this
to_date: End date (YYYY-MM-DD) - only include threads before this
depth: Research depth - "quick", "default", or "deep" depth: Research depth - "quick", "default", or "deep"
mock_response: Mock response for testing mock_response: Mock response for testing
@@ -106,7 +116,13 @@ def search_reddit(
} }
], ],
"include": ["web_search_call.action.sources"], "include": ["web_search_call.action.sources"],
"input": REDDIT_SEARCH_PROMPT.format(topic=topic, min_items=min_items, max_items=max_items), "input": REDDIT_SEARCH_PROMPT.format(
topic=topic,
from_date=from_date,
to_date=to_date,
min_items=min_items,
max_items=max_items,
),
} }
return http.post(OPENAI_RESPONSES_URL, payload, headers=headers, timeout=timeout) return http.post(OPENAI_RESPONSES_URL, payload, headers=headers, timeout=timeout)
+17 -5
View File
@@ -15,6 +15,10 @@ WEBSEARCH_WEIGHT_RELEVANCE = 0.55
WEBSEARCH_WEIGHT_RECENCY = 0.45 WEBSEARCH_WEIGHT_RECENCY = 0.45
WEBSEARCH_SOURCE_PENALTY = 15 # Points deducted for lacking engagement WEBSEARCH_SOURCE_PENALTY = 15 # Points deducted for lacking engagement
# WebSearch date confidence adjustments
WEBSEARCH_VERIFIED_BONUS = 10 # Bonus for URL-verified recent date (high confidence)
WEBSEARCH_NO_DATE_PENALTY = 20 # Heavy penalty for no date signals (low confidence)
# Default engagement score for unknown # Default engagement score for unknown
DEFAULT_ENGAGEMENT = 35 DEFAULT_ENGAGEMENT = 35
UNKNOWN_ENGAGEMENT_PENALTY = 10 UNKNOWN_ENGAGEMENT_PENALTY = 10
@@ -223,6 +227,11 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS
Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty. Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty.
This ensures WebSearch items rank below comparable Reddit/X items. This ensures WebSearch items rank below comparable Reddit/X items.
Date confidence adjustments:
- High confidence (URL-verified date): +10 bonus
- Med confidence (snippet-extracted date): no change
- Low confidence (no date signals): -20 penalty
Args: Args:
items: List of WebSearch items items: List of WebSearch items
@@ -255,11 +264,14 @@ def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebS
# Apply source penalty (WebSearch < Reddit/X for same relevance/recency) # Apply source penalty (WebSearch < Reddit/X for same relevance/recency)
overall -= WEBSEARCH_SOURCE_PENALTY overall -= WEBSEARCH_SOURCE_PENALTY
# Apply penalty for low date confidence # Apply date confidence adjustments
if item.date_confidence == "low": # High confidence (URL-verified): reward with bonus
overall -= 10 # Med confidence (snippet-extracted): neutral
elif item.date_confidence == "med": # Low confidence (no date signals): heavy penalty
overall -= 5 if item.date_confidence == "high":
overall += WEBSEARCH_VERIFIED_BONUS # Reward verified recent dates
elif item.date_confidence == "low":
overall -= WEBSEARCH_NO_DATE_PENALTY # Heavy penalty for unknown
item.score = max(0, min(100, int(overall))) item.score = max(0, min(100, int(overall)))
+215 -9
View File
@@ -11,12 +11,196 @@ The typical flow is:
""" """
import re import re
from typing import Any, Dict, List, Optional from datetime import datetime, timedelta
from typing import Any, Dict, List, Optional, Tuple
from urllib.parse import urlparse from urllib.parse import urlparse
from . import schema from . import schema
# Month name mappings for date parsing
MONTH_MAP = {
"jan": 1, "january": 1,
"feb": 2, "february": 2,
"mar": 3, "march": 3,
"apr": 4, "april": 4,
"may": 5,
"jun": 6, "june": 6,
"jul": 7, "july": 7,
"aug": 8, "august": 8,
"sep": 9, "sept": 9, "september": 9,
"oct": 10, "october": 10,
"nov": 11, "november": 11,
"dec": 12, "december": 12,
}
def extract_date_from_url(url: str) -> Optional[str]:
"""Try to extract a date from URL path.
Many sites embed dates in URLs like:
- /2026/01/24/article-title
- /2026-01-24/article
- /blog/20260124/title
Args:
url: URL to parse
Returns:
Date string in YYYY-MM-DD format, or None
"""
# Pattern 1: /YYYY/MM/DD/ (most common)
match = re.search(r'/(\d{4})/(\d{2})/(\d{2})/', url)
if match:
year, month, day = match.groups()
if 2020 <= int(year) <= 2030 and 1 <= int(month) <= 12 and 1 <= int(day) <= 31:
return f"{year}-{month}-{day}"
# Pattern 2: /YYYY-MM-DD/ or /YYYY-MM-DD-
match = re.search(r'/(\d{4})-(\d{2})-(\d{2})[-/]', url)
if match:
year, month, day = match.groups()
if 2020 <= int(year) <= 2030 and 1 <= int(month) <= 12 and 1 <= int(day) <= 31:
return f"{year}-{month}-{day}"
# Pattern 3: /YYYYMMDD/ (compact)
match = re.search(r'/(\d{4})(\d{2})(\d{2})/', url)
if match:
year, month, day = match.groups()
if 2020 <= int(year) <= 2030 and 1 <= int(month) <= 12 and 1 <= int(day) <= 31:
return f"{year}-{month}-{day}"
return None
def extract_date_from_snippet(text: str) -> Optional[str]:
"""Try to extract a date from text snippet or title.
Looks for patterns like:
- January 24, 2026 or Jan 24, 2026
- 24 January 2026
- 2026-01-24
- "3 days ago", "yesterday", "last week"
Args:
text: Text to parse
Returns:
Date string in YYYY-MM-DD format, or None
"""
if not text:
return None
text_lower = text.lower()
# Pattern 1: Month DD, YYYY (e.g., "January 24, 2026")
match = re.search(
r'\b(jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|'
r'jul(?:y)?|aug(?:ust)?|sep(?:t(?:ember)?)?|oct(?:ober)?|nov(?:ember)?|dec(?:ember)?)'
r'\s+(\d{1,2})(?:st|nd|rd|th)?,?\s*(\d{4})\b',
text_lower
)
if match:
month_str, day, year = match.groups()
month = MONTH_MAP.get(month_str[:3])
if month and 2020 <= int(year) <= 2030 and 1 <= int(day) <= 31:
return f"{year}-{month:02d}-{int(day):02d}"
# Pattern 2: DD Month YYYY (e.g., "24 January 2026")
match = re.search(
r'\b(\d{1,2})(?:st|nd|rd|th)?\s+'
r'(jan(?:uary)?|feb(?:ruary)?|mar(?:ch)?|apr(?:il)?|may|jun(?:e)?|'
r'jul(?:y)?|aug(?:ust)?|sep(?:t(?:ember)?)?|oct(?:ober)?|nov(?:ember)?|dec(?:ember)?)'
r'\s+(\d{4})\b',
text_lower
)
if match:
day, month_str, year = match.groups()
month = MONTH_MAP.get(month_str[:3])
if month and 2020 <= int(year) <= 2030 and 1 <= int(day) <= 31:
return f"{year}-{month:02d}-{int(day):02d}"
# Pattern 3: YYYY-MM-DD (ISO format)
match = re.search(r'\b(\d{4})-(\d{2})-(\d{2})\b', text)
if match:
year, month, day = match.groups()
if 2020 <= int(year) <= 2030 and 1 <= int(month) <= 12 and 1 <= int(day) <= 31:
return f"{year}-{month}-{day}"
# Pattern 4: Relative dates ("3 days ago", "yesterday", etc.)
today = datetime.now()
if "yesterday" in text_lower:
date = today - timedelta(days=1)
return date.strftime("%Y-%m-%d")
if "today" in text_lower:
return today.strftime("%Y-%m-%d")
# "N days ago"
match = re.search(r'\b(\d+)\s*days?\s*ago\b', text_lower)
if match:
days = int(match.group(1))
if days <= 60: # Reasonable range
date = today - timedelta(days=days)
return date.strftime("%Y-%m-%d")
# "N hours ago" -> today
match = re.search(r'\b(\d+)\s*hours?\s*ago\b', text_lower)
if match:
return today.strftime("%Y-%m-%d")
# "last week" -> ~7 days ago
if "last week" in text_lower:
date = today - timedelta(days=7)
return date.strftime("%Y-%m-%d")
# "this week" -> ~3 days ago (middle of week)
if "this week" in text_lower:
date = today - timedelta(days=3)
return date.strftime("%Y-%m-%d")
return None
def extract_date_signals(
url: str,
snippet: str,
title: str,
) -> Tuple[Optional[str], str]:
"""Extract date from any available signal.
Tries URL first (most reliable), then snippet, then title.
Args:
url: Page URL
snippet: Page snippet/description
title: Page title
Returns:
Tuple of (date_string, confidence)
- date from URL: 'high' confidence
- date from snippet/title: 'med' confidence
- no date found: None, 'low' confidence
"""
# Try URL first (most reliable)
url_date = extract_date_from_url(url)
if url_date:
return url_date, "high"
# Try snippet
snippet_date = extract_date_from_snippet(snippet)
if snippet_date:
return snippet_date, "med"
# Try title
title_date = extract_date_from_snippet(title)
if title_date:
return title_date, "med"
return None, "low"
# Domains to exclude (Reddit and X are handled separately) # Domains to exclude (Reddit and X are handled separately)
EXCLUDED_DOMAINS = { EXCLUDED_DOMAINS = {
"reddit.com", "reddit.com",
@@ -70,15 +254,25 @@ def is_excluded_domain(url: str) -> bool:
def parse_websearch_results( def parse_websearch_results(
results: List[Dict[str, Any]], results: List[Dict[str, Any]],
topic: str, topic: str,
from_date: str = "",
to_date: str = "",
) -> List[Dict[str, Any]]: ) -> List[Dict[str, Any]]:
"""Parse WebSearch results into normalized format. """Parse WebSearch results into normalized format.
This function expects results from Claude's WebSearch tool. This function expects results from Claude's WebSearch tool.
Each result should have: title, url, snippet, and optionally date/relevance. Each result should have: title, url, snippet, and optionally date/relevance.
Uses "Date Detective" approach:
1. Extract dates from URLs (high confidence)
2. Extract dates from snippets/titles (med confidence)
3. Hard filter: exclude items with verified old dates
4. Keep items with no date signals (with low confidence penalty)
Args: Args:
results: List of WebSearch result dicts results: List of WebSearch result dicts
topic: Original search topic (for context) topic: Original search topic (for context)
from_date: Start date for filtering (YYYY-MM-DD)
to_date: End date for filtering (YYYY-MM-DD)
Returns: Returns:
List of normalized item dicts ready for WebSearchItem creation List of normalized item dicts ready for WebSearchItem creation
@@ -103,15 +297,27 @@ def parse_websearch_results(
if not title and not snippet: if not title and not snippet:
continue continue
# Parse date if provided # Use Date Detective to extract date signals
date = result.get("date") date = result.get("date") # Use provided date if available
date_confidence = "low" date_confidence = "low"
if date:
# Validate date format if date and re.match(r'^\d{4}-\d{2}-\d{2}$', str(date)):
if re.match(r'^\d{4}-\d{2}-\d{2}$', str(date)): # Provided date is valid
date_confidence = "med" # WebSearch dates are often approximate date_confidence = "med"
else: else:
date = None # Try to extract date from URL/snippet/title
extracted_date, confidence = extract_date_signals(url, snippet, title)
if extracted_date:
date = extracted_date
date_confidence = confidence
# Hard filter: if we found a date and it's too old, skip
if date and from_date and date < from_date:
continue # DROP - verified old content
# Hard filter: if date is in the future, skip (parsing error)
if date and to_date and date > to_date:
continue # DROP - future date
# Get relevance if provided, default to 0.5 # Get relevance if provided, default to 0.5
relevance = result.get("relevance", 0.5) relevance = result.get("relevance", 0.5)