fix(github): reject garbage in _parse_date; consolidate date parsing

github.py _parse_date used naive string slicing (return iso_str[:10])
which accepted any 10+ character string as a "date." For input
"hello world" it returned "hello worl". Now delegates to
dates.parse_date() which validates the format and returns None for
non-dates.

Also migrated reddit.py and threads.py _parse_date to the shared
dates.parse_date(). Both previously reimplemented ISO-with-trailing-
offset handling (the .replace("Z", "+00:00") dance) and reddit.py
also had its own Unix timestamp branch. dates.parse_date() already
handles all of this, including the +0000 no-colon variant Reddit emits.

Preserved reddit.py's original falsy-check so 0 still returns None
(epoch 0 would otherwise parse as "1970-01-01", breaking an existing
test and changing long-standing behavior).

Added 4 new github tests for garbage rejection and offset variants.
All 1026 existing tests pass (15 pre-existing failures unchanged).
This commit is contained in:
Ilia Alshanetsky
2026-04-10 07:39:07 -04:00
parent 86e1d77ad7
commit 65fcf6be65
4 changed files with 43 additions and 51 deletions
+10 -23
View File
@@ -9,10 +9,9 @@ API docs: https://scrapecreators.com/docs
import math
import re
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
from . import http, log
from . import dates, http, log
from .relevance import token_overlap_relevance as _compute_relevance
SCRAPECREATORS_BASE = "https://api.scrapecreators.com/v1/threads"
@@ -52,29 +51,17 @@ def _extract_core_subject(topic: str) -> str:
def _parse_date(item: Dict[str, Any]) -> Optional[str]:
"""Parse date from Threads item to YYYY-MM-DD.
Tries common timestamp fields: taken_at (unix), created_at (ISO),
and falls back to any date-like string field.
Tries common timestamp fields in order: taken_at and create_time
(unix timestamps in Meta APIs), then created_at, published_at, and
date (ISO 8601 strings). dates.parse_date() handles both.
"""
# Unix timestamp (taken_at is common in Meta APIs)
for key in ("taken_at", "create_time"):
ts = item.get(key)
if ts:
try:
from . import dates
return dates.timestamp_to_date(int(ts))
except (ValueError, TypeError):
pass
# ISO 8601 string
for key in ("created_at", "published_at", "date"):
for key in ("taken_at", "create_time", "created_at", "published_at", "date"):
val = item.get(key)
if val and isinstance(val, str):
try:
dt = datetime.fromisoformat(val.replace("Z", "+00:00"))
return dt.strftime("%Y-%m-%d")
except (ValueError, TypeError):
pass
if val is None:
continue
dt = dates.parse_date(str(val))
if dt:
return dt.strftime("%Y-%m-%d")
return None