Commit Graph
7 Commits
Author SHA1 Message Date
Ilia Alshanetsky 18b5658674 chore: remove orphan test for deleted generate-synthesis-inputs script (#205)
tests/test_generate_synthesis_inputs_v3.py imported a script that no
longer exists in the repo. The test failed with FileNotFoundError on
every run.
2026-04-25 14:16:39 -07:00
Ilia Alshanetsky 9dd3f21476 refactor: consolidate _sc_headers into http.scrapecreators_headers (#209)
Six source modules each defined an identical 8-line _sc_headers(token)
function returning {"x-api-key": token, "Content-Type": "application/json"}.
Moved it to http.scrapecreators_headers() and migrated all 33 call sites.

Affected files: reddit.py, threads.py, tiktok.py, instagram.py, pinterest.py,
youtube_yt.py. Zero per-source variation, zero behavior change.

Net: -40 lines. 1022 tests pass (15 pre-existing failures unchanged).
Live smoke test: reddit search returns 12 threads with full engagement.
2026-04-14 07:43:56 -04:00
Ilia Alshanetsky 65fcf6be65 fix(github): reject garbage in _parse_date; consolidate date parsing
github.py _parse_date used naive string slicing (return iso_str[:10])
which accepted any 10+ character string as a "date." For input
"hello world" it returned "hello worl". Now delegates to
dates.parse_date() which validates the format and returns None for
non-dates.

Also migrated reddit.py and threads.py _parse_date to the shared
dates.parse_date(). Both previously reimplemented ISO-with-trailing-
offset handling (the .replace("Z", "+00:00") dance) and reddit.py
also had its own Unix timestamp branch. dates.parse_date() already
handles all of this, including the +0000 no-colon variant Reddit emits.

Preserved reddit.py's original falsy-check so 0 still returns None
(epoch 0 would otherwise parse as "1970-01-01", breaking an existing
test and changing long-standing behavior).

Added 4 new github tests for garbage rejection and offset variants.
All 1026 existing tests pass (15 pre-existing failures unchanged).
2026-04-10 07:39:07 -04:00
Ilia Alshanetsky 9ef9d38b90 refactor(reddit): migrate to http.get(params=...) helper
Added params kwarg to http.request()/http.get() that urlencodes a dict
into the query string. None values are dropped, ints and bools are
stringified, and params append correctly if the URL already has a
query string.

Migrated reddit.py to use this helper for all three ScrapeCreators
call sites (global search, subreddit search, post comments). Deleted
the try/import requests/except ImportError fallback and the paired
if not _requests: / else: branches. Six new http tests cover the
params-encoding behavior.

Net: reddit.py -70 lines. Behavior is identical - the existing http.py
urllib implementation already had retry logic, 429 handling, and
HTTPError types that are strictly better than the ad-hoc requests
branches we deleted.

99 reddit tests pass. Live smoke test on a real ScrapeCreators run
returned 12 threads with the same engagement data as before.
2026-04-10 07:25:26 -04:00
Ilia Alshanetsky eef3547c37 perf: optimize dedup, parallelize handle searches and enrichment
The dedup hot path recomputed normalize_text() 4 times per comparison
and recomputed item_text() on every inner-loop iteration. Pre-computing
n-gram sets and token sets into a _PreparedText cache cuts dedup time
by 6x (2.16s to 0.39s on 300 unique items).

Bird handle searches spawned one Node process per handle sequentially.
Now uses ThreadPoolExecutor so N handles run concurrently. Same pattern
applied to YouTube comment enrichment (was serial, Reddit was already
parallel) and the retry-thin-sources phase in the pipeline.

Clustering now pre-computes candidate text and uses prepared_similarity
for the O(n^2) grouping and MMR representative selection loops.

Minor: _is_wsl() cached with lru_cache, Bundle.add_items() uses
extend() instead of list concatenation.

End-to-end: 5.2s -> 3.7s (29% faster) on a typical 4-source query.
2026-04-09 19:00:30 -04:00
Ilia Alshanetsky 252c8222f1 feat: add WSL2 Windows Firefox cookie extraction for X auth
On WSL2, native Linux Firefox typically has no x.com cookies since users
browse in Windows. Chromium browsers (Edge, Chrome, Brave) encrypt cookies
with DPAPI/app-bound encryption, making them inaccessible without admin
privileges. Windows Firefox stores cookies unencrypted in SQLite, readable
directly through the /mnt/c mount.

The cookie extractor now detects WSL2 via /proc/version, locates Windows
Firefox profiles under /mnt/c/Users/*/AppData/Roaming/Mozilla/Firefox,
and falls back to them when Linux Firefox yields no results. Reports
source as "firefox-wsl" to distinguish from native.

Also fixes profile resolution priority: Install* sections (Firefox >= 67)
now take precedence over the legacy Default=1 flag, which could select a
stale profile on multi-profile installations.
2026-04-09 18:36:02 -04:00
Ilia AlshanetskyandJustin Williams d7bff81757 fix(bird_x): pass .env credentials to Node subprocesses for WSL2/headless auth
* chore: fix YAML error in argument-hint

* add codex auth support to responses API

* Use gpt-5.1-codex-mini as default model for Codex auth

Add CODEX_FALLBACK_MODELS chain (gpt-5.1-codex-mini → gpt-5.2) for
Codex endpoint which doesn't support standard OpenAI models like
gpt-4o-mini. Adds model fallback retry on 400 errors in the Codex
search path. Also adds test_codex_auth.py with 22 unit tests covering
JWT decoding, auth resolution, SSE parsing, and payload building.

* Pass .env credentials to Bird Node subprocesses for X auth

On platforms without browser cookie access (e.g. WSL2), Bird's
vendored Node.js module cannot read AUTH_TOKEN/CT0 from Firefox
or Chrome cookie stores. The .env config file already supports
these values, but they were only loaded into the Python config
dict — never exported to the environment of Node subprocesses.

- Add AUTH_TOKEN/CT0 to env.py config key loading
- Add set_credentials()/\_subprocess_env() to bird_x.py to inject
  credentials into the env dict passed to subprocess.run/Popen
- Call set_credentials() in main() before Bird auth detection

---------

Co-authored-by: Justin Williams <jblwilliams@gmail.com>
2026-03-02 23:24:59 -08:00