Commit Graph

3 Commits

Author SHA1 Message Date
Ilia Alshanetsky 9dd3f21476 refactor: consolidate _sc_headers into http.scrapecreators_headers (#209)
Six source modules each defined an identical 8-line _sc_headers(token)
function returning {"x-api-key": token, "Content-Type": "application/json"}.
Moved it to http.scrapecreators_headers() and migrated all 33 call sites.

Affected files: reddit.py, threads.py, tiktok.py, instagram.py, pinterest.py,
youtube_yt.py. Zero per-source variation, zero behavior change.

Net: -40 lines. 1022 tests pass (15 pre-existing failures unchanged).
Live smoke test: reddit search returns 12 threads with full engagement.
2026-04-14 07:43:56 -04:00
Ilia Alshanetsky 65fcf6be65 fix(github): reject garbage in _parse_date; consolidate date parsing
github.py _parse_date used naive string slicing (return iso_str[:10])
which accepted any 10+ character string as a "date." For input
"hello world" it returned "hello worl". Now delegates to
dates.parse_date() which validates the format and returns None for
non-dates.

Also migrated reddit.py and threads.py _parse_date to the shared
dates.parse_date(). Both previously reimplemented ISO-with-trailing-
offset handling (the .replace("Z", "+00:00") dance) and reddit.py
also had its own Unix timestamp branch. dates.parse_date() already
handles all of this, including the +0000 no-colon variant Reddit emits.

Preserved reddit.py's original falsy-check so 0 still returns None
(epoch 0 would otherwise parse as "1970-01-01", breaking an existing
test and changing long-standing behavior).

Added 4 new github tests for garbage rejection and offset variants.
All 1026 existing tests pass (15 pre-existing failures unchanged).
2026-04-10 07:39:07 -04:00
Matt Van Horn 0a9ff16dfc feat: v3.0.0 - intelligent search, GitHub person/project mode, ELI5, 13+ sources
v3 rewrites the search engine from the ground up:

- Intelligent pre-research: resolves X handles, GitHub repos, subreddits,
  TikTok hashtags, and YouTube channels before searching
- GitHub person-mode: PR velocity, top repos by stars, release notes
- GitHub project-mode: live star counts, README, releases, top issues
- ELI5 mode: plain language synthesis, no jargon
- 13+ sources: Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket,
  GitHub, Threads, Pinterest, Perplexity, Bluesky, Web
- Free Reddit comments via public JSON (no API key needed)
- Fun judge v2: humor scoring baked into narrative
- Cookie consent before browser scanning
- 10,000 free ScrapeCreators calls
- 1,012 tests

Thank you to the community contributors whose issues and PRs shaped v3:
@uppinote20 (#143), @zerone0x (#134, #136), @thinkun (#116),
@thomasmktong (#124), @fanispoulinakisai-boop (#100), @pejmanjohn (#78),
@zl190 (#115), @hnshah (#84, #85, #86)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 10:52:23 -07:00