Six source modules each defined an identical 8-line _sc_headers(token)
function returning {"x-api-key": token, "Content-Type": "application/json"}.
Moved it to http.scrapecreators_headers() and migrated all 33 call sites.
Affected files: reddit.py, threads.py, tiktok.py, instagram.py, pinterest.py,
youtube_yt.py. Zero per-source variation, zero behavior change.
Net: -40 lines. 1022 tests pass (15 pre-existing failures unchanged).
Live smoke test: reddit search returns 12 threads with full engagement.
When a tweet has no engagement metrics, _first_of() returns None for
every key, producing {"likes": None, "reposts": None, ...}. This
all-None dict propagates to signals.py where it is treated as "data
exists but is zero" rather than "no data available." Return None
instead when every engagement field is missing.
Makes the `variants/open/` sync steps in `scripts/sync.sh` conditional on
the directory actually existing in the source tree. The script is shared
between the public and private repos of last30days-skill, but the OpenClaw
variant only lives in the private repo (it's sanitized via
`strip_for_openclaw.py` and published separately to ClawhHub). When the
script runs from a checkout of the public repo, the variants/open paths
don't exist and the unconditional `rsync` and `sync_target` calls error
out immediately.
Changes:
- `sync_target()` now only creates `variants/open/references` and rsyncs
`variants/open/` when `$SRC/variants/open` exists.
- The trailing `sync_target "$OPENCLAW_TARGET" ...` call is now gated by
the same check, with an explanatory skip message when the directory is
absent.
No behavior change when running from the private repo (which has
`variants/open/`). When running from the public repo, the script now
completes its COMMON_TARGETS loop without erroring.
This also closes out the confusion from PR #211, where a contributor saw
the broken `variants/open/` reference and tried to add the variant back
to the public repo. The real fix was making the script tolerate the
absence, not recreating the directory.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
github.py _parse_date used naive string slicing (return iso_str[:10])
which accepted any 10+ character string as a "date." For input
"hello world" it returned "hello worl". Now delegates to
dates.parse_date() which validates the format and returns None for
non-dates.
Also migrated reddit.py and threads.py _parse_date to the shared
dates.parse_date(). Both previously reimplemented ISO-with-trailing-
offset handling (the .replace("Z", "+00:00") dance) and reddit.py
also had its own Unix timestamp branch. dates.parse_date() already
handles all of this, including the +0000 no-colon variant Reddit emits.
Preserved reddit.py's original falsy-check so 0 still returns None
(epoch 0 would otherwise parse as "1970-01-01", breaking an existing
test and changing long-standing behavior).
Added 4 new github tests for garbage rejection and offset variants.
All 1026 existing tests pass (15 pre-existing failures unchanged).
Added params kwarg to http.request()/http.get() that urlencodes a dict
into the query string. None values are dropped, ints and bools are
stringified, and params append correctly if the URL already has a
query string.
Migrated reddit.py to use this helper for all three ScrapeCreators
call sites (global search, subreddit search, post comments). Deleted
the try/import requests/except ImportError fallback and the paired
if not _requests: / else: branches. Six new http tests cover the
params-encoding behavior.
Net: reddit.py -70 lines. Behavior is identical - the existing http.py
urllib implementation already had retry logic, 429 handling, and
HTTPError types that are strictly better than the ad-hoc requests
branches we deleted.
99 reddit tests pass. Live smoke test on a real ScrapeCreators run
returned 12 threads with the same engagement data as before.
The module-level _cached_token was set once and never refreshed. AT
Protocol tokens expire after ~2 hours, causing silent 401 errors in
long-running watchlist cron sessions. Adds a 90-minute expiry check
using time.monotonic() and logs re-authentication.
Fixes#92
Add column whitelists to prevent SQL injection via kwargs keys in
dynamic UPDATE queries. Values were already parameterized but column
names were string-interpolated directly from kwargs.
Fixes#90
The dedup hot path recomputed normalize_text() 4 times per comparison
and recomputed item_text() on every inner-loop iteration. Pre-computing
n-gram sets and token sets into a _PreparedText cache cuts dedup time
by 6x (2.16s to 0.39s on 300 unique items).
Bird handle searches spawned one Node process per handle sequentially.
Now uses ThreadPoolExecutor so N handles run concurrently. Same pattern
applied to YouTube comment enrichment (was serial, Reddit was already
parallel) and the retry-thin-sources phase in the pipeline.
Clustering now pre-computes candidate text and uses prepared_similarity
for the O(n^2) grouping and MMR representative selection loops.
Minor: _is_wsl() cached with lru_cache, Bundle.add_items() uses
extend() instead of list concatenation.
End-to-end: 5.2s -> 3.7s (29% faster) on a typical 4-source query.
On WSL2, native Linux Firefox typically has no x.com cookies since users
browse in Windows. Chromium browsers (Edge, Chrome, Brave) encrypt cookies
with DPAPI/app-bound encryption, making them inaccessible without admin
privileges. Windows Firefox stores cookies unencrypted in SQLite, readable
directly through the /mnt/c mount.
The cookie extractor now detects WSL2 via /proc/version, locates Windows
Firefox profiles under /mnt/c/Users/*/AppData/Roaming/Mozilla/Firefox,
and falls back to them when Linux Firefox yields no results. Reports
source as "firefox-wsl" to distinguish from native.
Also fixes profile resolution priority: Install* sections (Firefox >= 67)
now take precedence over the legacy Default=1 flag, which could select a
stale profile on multi-profile installations.
Add Xquik (xquik.com) as a new X/Twitter search source that uses a REST
API with full engagement metrics (likes, retweets, replies, quotes,
views, bookmarks). Uses stdlib urllib only -- no new dependencies.
- scripts/lib/xquik.py: source module with search, parse, query expansion
- tests/test_xquik.py: 32 unit tests covering all functions
- env.py: XQUIK_API_KEY config and availability check
- pipeline.py: source registration and retrieve dispatch
- normalize.py: reuses _normalize_x (same item format as Bird)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When Bird's JSON response is a raw array instead of an object,
json.loads returns a list. All callers use .get('items') which raises
AttributeError on lists. Wrap list responses in {"items": parsed} so
callers always receive a dict.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: INCLUDE_SOURCES config + TikTok/Instagram opt-in in NUX
- INCLUDE_SOURCES=tiktok,instagram in .env forces sources on for all
query types, bypassing the tier system
- NUX shows opt-in modal after ScrapeCreators key is saved: "Also
search TikTok and Instagram?" with honest call-usage warning
- Tier system preserved as default — override only when INCLUDE_SOURCES set
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: neutral call-usage copy — works for free and paid tiers
---------
Co-authored-by: Matt Van Horn <mvanhorn@MacBook-Pro.local>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add extract_transcript_highlights() that scores sentences by specificity
(numbers, proper nouns, topic relevance) and filters YouTube filler
(subscribe, welcome back, etc). Top 5 highlights shown as structured
bullets in compact output. Full transcript moved to collapsible <details>
block so the LLM reads highlights first, full text on demand.
SKILL.md updated to instruct the judge agent to quote highlights
directly in synthesis, same as Reddit top comments.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TRANSCRIPT_MAX_WORDS raised from 500 to 5000 so the LLM gets the full
content of most videos (up to ~25 minutes). Removed the second 200-char
truncation in render.py that was reducing transcripts to a single sentence
before the judge agent ever saw them.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When Cloudflare blocks requests to bsky.social or public.api.bsky.app
with a 403, the error was swallowed by a generic except clause and
reported as "Bluesky auth failed" - misleading users into thinking
their credentials were wrong.
Now _create_session() preserves the specific error in _session_error,
and search_bluesky() surfaces it. Cloudflare 403s get a clear message
about network-level blocks. Actual 401s say "Invalid credentials".
Closes#69
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Map prompt-oriented product searches and animation-oriented build searches away from the breaking-news default so source tiering and tiebreakers align with the benchmark topics.
Validation: uv run python -m unittest tests.test_query_type
Isolate eval subprocesses from local yt-dlp config and fix nDCG normalization against the judged pool.
Validation: uv run python -m unittest tests.test_evaluate_search_quality
This workspace uses GOOGLE_API_KEY as the canonical Google credential. Accept it ahead of the Gemini-specific aliases so the local evaluation harness can run without a separate GEMINI_API_KEY export.
Validation: uv run python -m unittest tests.test_env_project tests.test_evaluate_search_quality and a one-shot keychain-backed resolution check.
Add an optional local evaluator that compares a baseline revision against a candidate checkout, computes deterministic stability metrics, and can call Gemini for judged ranking metrics when configured.
The harness isolates child runs with a temporary HOME and a node-free PATH so historical revisions cannot trigger Bird browser-cookie auth during evaluation.
Validation: uv run python -m unittest and local smoke/full deterministic eval runs.
Score against original user intent on Reddit, remove the artificial low-end relevance floor, and make Polymarket semantics dominate generic market quality signals.
Also apply the relevance filter to Polymarket and update the affected cross-source tests.
Validation: uv run python -m unittest
Phase-2 Bird handle searches were still spawning Node without the injected AUTH_TOKEN/CT0 env. That left the search pipeline vulnerable to Chrome keychain prompts whenever a query drilled into X handles.
Pass the popup-safe subprocess env through those handle searches and cover it with a regression test.
Ilia Alshanetsky