Compare commits

..

84 Commits

Author SHA1 Message Date
Trevin Chow daca71f89e chore(release): v3.3.0
Release / build-and-release (push) Has been cancelled
~75 PRs merged since v3.2.0 plus 7 community fixes salvaged via PR triage.

Highlights:
- Install everywhere: npx skills add is canonical for Claude Code, Codex,
  Cursor, Gemini CLI, Copilot, Windsurf, and 50+ Agent Skills hosts.
- New emit mode: --emit=html for shareable HTML briefs.
- New source: Digg (auto-enabled when digg-pp-cli on PATH).
- New env vars: EXCLUDE_SOURCES, LAST30DAYS_YOUTUBE_SSH_HOST.
- New credential source: macOS Keychain.
- Reliability sweep: Reddit (4xx + URL prefix + multi-key auth), xAI
  error surfacing, Windows compatibility, YouTube/HN unblock,
  HTTP retries, planner gating, render fixes.
- Multi-harness reframe: AGENTS.md becomes canonical, CLAUDE.md points
  at it. SKILL_ROOT → SKILL_DIR substitution.

Breaking:
- .codex-plugin/plugin.json removed. Codex installs via npx skills add.
2026-05-17 09:25:46 -07:00
Kaustav Mishra d51e91ea26 fix(xai): surface API errors instead of silently returning empty results
parse_x_response was returning an empty items list whenever xAI returned
a 200 OK with a malformed payload — empty output text, missing "items"
key, or invalid JSON. The pipeline saw "successful response with zero
items" and quietly handed the user a degraded report with no indication
the API had failed. Now raise http.HTTPError on each of those branches
so _retrieve_stream's caller catches it and surfaces the failure in
errors_by_source, giving the user a visible signal that X didn't work.

Closes #155.

Co-authored-by: Kaustav Mishra <km.git007@gmail.com>
2026-05-17 09:20:37 -07:00
Trevin Chow 170b570cbc fix(reddit): re-raise HTTP 402 so fallback chain triggers
The ScrapeCreators 402 (payment required / credits exhausted) status
was being swallowed by the broad except Exception handlers in
_global_search, _subreddit_search, and fetch_post_comments, returning
[] instead of propagating. That caused users with exhausted credits
to silently get zero Reddit results instead of falling through to
the OpenAI / public Reddit JSON fallback chain in _search_reddit_thread.
Add 402 to the existing 401/403 re-raise list across all three
ScrapeCreators call paths. Closes #170.

Co-authored-by: Jonathan Oppenheim <no-reply@postquantum.space>
2026-05-17 09:20:37 -07:00
Trevin Chow 4bae05e7fa fix(reddit): use browser-like headers to fix HTTP 403 from urllib
Reddit's public JSON endpoint returns 403 to requests carrying the
generic User-Agent and minimal header set urllib defaults to, while
matching curl requests succeed. Switch to a current-Chrome User-Agent
and add Accept-Language / Accept-Encoding / Connection headers so the
fingerprint matches a normal browser. Reddit now serves gzip when
Accept-Encoding includes it, so decompress the body before JSON parse.
Update the user-agent assertion in tests/test_reddit_public.py to match
the new browser-like string. Closes #199.

Co-authored-by: Franco Carballar <francocarballar@gmail.com>
2026-05-17 09:20:37 -07:00
Trevin Chow a4f1f94802 fix(env): restore multi-key rotation for SCRAPECREATORS_API_KEY
Originally added in #268 to spread load across free-tier accounts when
SCRAPECREATORS_API_KEY is set to a comma-separated list. The 7-line block
was inadvertently dropped during the v3.0.6 consolidation (d14814a) even
though the changelog still advertised the feature. Re-apply the same
random.choice rotation in get_config() so user-facing behavior matches
the documented contract. Closes #287.

Co-authored-by: Eric Oberhofer <eric@oberhofer.io>
2026-05-17 09:20:37 -07:00
Trevin Chow 16ce073d0c fix(cli): keep child cleanup working on Windows
_cleanup_children() called os.killpg unconditionally — Windows doesn't
have killpg as an attribute on os, so the call raised AttributeError
(not caught by the existing OSError-family handler) and aborted cleanup.
Guard with hasattr(os, "killpg") and fall back to os.kill(pid, SIGTERM)
on platforms without process-group APIs. Closes #226. Refs #110.

Co-authored-by: gujishh <baiaoshh@163.com>
2026-05-17 09:20:02 -07:00
Trevin Chow 5994b4f76a fix(reddit): use removeprefix("r/") for subreddit names, not lstrip("r/")
str.lstrip("r/") treats its argument as a character set, stripping
leading r and / repeatedly. Subreddits starting with 'r' (e.g. r/robotics,
r/ruby) were silently mangled to 'obotics' / 'uby'. Replace with
str.removeprefix("r/") at all four call sites. Python 3.9+ pattern is
safe here — project requires 3.12. Closes #288.

Co-authored-by: Alex Key <alexanderkey0508@gmail.com>
2026-05-17 09:20:02 -07:00
Trevin Chow bb5e6efbf9 fix(scripts): replace hardcoded developer paths in test-v1-vs-v2.sh
REPO_DIR now derives from the script's location (with env-var override)
and the Claude binary is looked up via PATH (with CLAUDE env-var override)
instead of hardcoded to /Users/mvanhorn/.local/bin/claude. Works on any
checkout. Closes #297.

Co-authored-by: Dave Morin <dave@morin.com>
2026-05-17 09:20:02 -07:00
Trevin Chow 76b8df40d3 Merge pull request #318 from flyingnobita/fix/gemini-claude-polyglot-hooks
fix: make hooks.json polyglot for Gemini CLI and Claude Code compatibility
2026-05-17 01:05:30 -07:00
Trevin Chow eb2d8b55e0 Merge pull request #344 from dzivkovi/feat/config-enablement
feat: configuration enablement — env-var defaults + source resilience
2026-05-17 01:05:08 -07:00
Trevin Chow 1a8ffd4847 fix(quality_nudge): also guard Instagram silent-failure on INCLUDE_SOURCES allowlist 2026-05-17 01:03:55 -07:00
Trevin Chow 1814bb1967 fix(quality_nudge,bluesky): gate Instagram nudge on EXCLUDE_SOURCES + anchor bluesky tests at resolver 2026-05-17 01:03:55 -07:00
Trevin Chow f236cff86a chore(pr-344): adapt to rebased base — fix test imports + memory-dir doc style
Rebased onto current main where:
- instagram.py uses unified http.get (not _requests fallback); tests now
  mock http.get and assert params/timeout kwargs.
- quality_nudge tests use lib.* import path with sys.path setup.
- README/CONFIGURATION.md memory-dir lines say "defaults to" so they pass
  test_no_stray_hardcoded_memory_dir_paths.
2026-05-17 01:03:55 -07:00
Daniel Zivkovic 44971a6aae feat: configuration enablement — env-var defaults + source resilience
Six small additive changes that make the skill correctly understand its
configured sources, plus tests + docs.

User-visible benefits

- LAST30DAYS_STORE=1 in .env turns persistence default-on without
  remembering --store on every invocation. Mirrors LAST30DAYS_DEBUG /
  LAST30DAYS_SKIP_PREFLIGHT convention.
- SCRAPE_CREATORS_API_KEY (with underscore) accepted as alias for the
  canonical name. Matches the spelling used in the vendor's own example
  code (Adrian Horning's repo); saves the next user the same diagnostic
  rabbit hole.
- Bluesky search now hits api.bsky.app (canonical AppView) instead of
  public.api.bsky.app (BunnyCDN-blocked public mirror as of 2026-05-04).
  BSKY_SEARCH_HOST env var lets users self-rescue future host migrations
  without a code release. Pre-fix: silent 0 Bluesky posts on every run.
- App-password format validator emits a one-shot stderr warning when
  BSKY_APP_PASSWORD doesn't match xxxx-xxxx-xxxx-xxxx form. Detect-don't-
  gate: createSession still accepts main passwords; the warning helps
  users identify a hygiene issue without breaking existing setups.
- Instagram retry on multi-token 500. SC's v2 reels endpoint wraps
  Google Search and 500's frequently on multi-word queries; a hashtag-
  form retry runs once before bubbling up. Documented vendor instability.
- LAST30DAYS_TRANSCRIPT_TIMEOUT env var (default 30s, was hardcoded 15s).
  SC's transcript endpoint regularly takes >15s; the old default was
  clipping legitimate responses.
- Silent-failure visibility: new bonus_errored field in the quality
  nudge fires when SC is configured but Instagram returned 0 items.
  Users see "Bonus source silent: Instagram" instead of unexplained
  absence.
- YouTube degraded-ratio false-positive fixed. Captions-disabled videos
  can never produce a transcript regardless of yt-dlp version; they're
  now subtracted from the denominator so a single uploader-disabled
  video doesn't false-trigger the "stale yt-dlp" nudge.
- urllib retry path: status_code attribute typo fix. The Instagram
  500-retry was dead code on the urllib branch (getattr(e, 'status', ...)
  while http.HTTPError exposes status_code).

Docs

- README.md: added /plugin install last30days step after marketplace add
  in three places (the install was previously omitted in the docs).
- CONFIGURATION.md: documented LAST30DAYS_STORE env var, added
  BSKY_SEARCH_HOST + app-password format section, mentioned
  LAST30DAYS_TRANSCRIPT_TIMEOUT in the Instagram source row.

Test plan

- 43 new unit tests across test_bluesky.py, test_instagram_sc.py,
  test_quality_nudge.py, test_youtube_yt.py
- 141 total tests passing in target suite
- Verified end-to-end: /last30days "Toronto resale condo market" with
  all 11+ sources active stored 35 new + 5 updated findings, all builder-
  PR-style accounts absent (organic agent voice in Instagram + TikTok
  results)

Backward compatibility

All changes are strictly additive. Optional kwargs default to None.
New env vars are opt-in. Existing CLI flags untouched. Existing callers
of public functions unaffected.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 01:03:15 -07:00
Daniel Zivkovic a8e462c978 chore(gitignore): ignore /work and /print (personal artifacts)
Personal directories used by the /note slash command (work/) and PDF
print exports (print/) - these are local research artifacts, not
shipping content. Lives on daniel/personal to keep upstream/main and
PR branches free of personal noise.
2026-05-17 01:02:47 -07:00
Trevin Chow d9a0ac31f2 Merge pull request #323 from GAOJIAN-0106/fix/openrouter-default-model-id
fix: correct invalid OPENROUTER_DEFAULT model ID
2026-05-17 01:01:54 -07:00
Trevin Chow 3e60c0817d Merge pull request #339 from dzivkovi/docs/configuration-md
docs: add CONFIGURATION.md + README pointers
2026-05-17 01:01:37 -07:00
Trevin Chow d530c90239 fix(hooks): remove timeout field per PR description intent 2026-05-17 00:59:41 -07:00
Trevin Chow aef6f35460 docs(providers): note why OpenRouter slug includes -preview suffix 2026-05-17 00:59:31 -07:00
GAOJIAN-0106 77bd235f64 fix: correct invalid OPENROUTER_DEFAULT model ID
google/gemini-flash-2.0 is not a valid OpenRouter model ID (segments reversed).
Every rerank and FunJudge call fails with HTTP 400 when REASONING_PROVIDER=openrouter
and LAST30DAYS_RERANK_MODEL is not explicitly pinned, silently falling back to
local-score heuristics.

OpenRouter error body:
  {"message":"google/gemini-flash-2.0 is not a valid model ID","code":400}

Switching to google/gemini-3.1-flash-lite-preview, which matches the
GEMINI_FLASH_LITE constant already used by the native Gemini provider on
line 12 of the same file. This makes the Gemini and OpenRouter providers
consistent and avoids a future divergence.

Validated with /last30days 'Claude Opus 4.7' --quick:
- Rerank/FunJudge HTTP 400 errors: 1 per run -> 0
- 'Why: fallback-local-score' markers in output: every cluster -> 0
- LLM-generated 'Why:' reasoning lines: 0 -> 11
2026-05-17 00:59:11 -07:00
flyingnobita e1017e95c9 fix: make hooks.json polyglot for Gemini CLI and Claude Code compatibility 2026-05-17 00:58:46 -07:00
Trevin Chow 71b1e8a411 Merge pull request #302 from nidhi-singh02/fix/github-repo-canonicalization
fix: Canonicalize ambiguous GitHub repo resolution for product comparisons
2026-05-17 00:57:05 -07:00
Trevin Chow 618458eb7e Merge pull request #320 from kaushikgopal/feat/brave-cookie-extraction
Add Brave browser support for X/Twitter cookie extraction
2026-05-17 00:56:44 -07:00
Trevin Chow 8cccd3e982 Merge pull request #334 from iamitp/codex/last-run-config-state
Preserve clean mode for last run state
2026-05-17 00:56:28 -07:00
Trevin Chow 17fb17222b fix(docs): use 'defaults to' wording for LAST30DAYS_MEMORY_DIR refs
The version_consistency test (test_no_stray_hardcoded_memory_dir_paths)
flagged 4 lines where `~/Documents/Last30Days/` appeared without the
canonical "defaults to" phrasing or the ${LAST30DAYS_MEMORY_DIR:-...}
literal form. Tightened the wording in the CONFIGURATION.md table +
footer paragraph, and the README "Where research files are saved"
section, so each path mention is anchored at the env-var override
contract rather than as a bare hardcoded default.
2026-05-17 00:54:49 -07:00
Trevin Chow 8ccd778366 fix(canonicalization): predicate-based call lookup + skip double-canon on auto-resolve
Two findings from Greptile review on PR #302:

1. tests/test_cli_v3.py:302 - The test asserted run_mock.call_args_list[0]
   was the main runner's invocation, but fanout.run_competitor_fanout
   submits main + competitors to a ThreadPoolExecutor and iterates with
   as_completed. With zero-latency mocks, thread scheduling determines
   which pipeline.run call lands first, so the competitor's call could
   take index [0] and flake CI. Replace [0] indexing with a predicate
   match on the canonicalized github_repos kwargs.

2. skills/last30days/scripts/last30days.py:662 - When auto_resolve returns
   github_repos, it has already run canonicalize_github_repos(cap=5) and
   ranked by relevance. The downstream block then re-canonicalized with
   cap=None, which can re-sort by topic-slug match and clobber the
   auto_resolve relevance order. Guard the second canonicalization with
   a repos_from_auto_resolve flag so it only fires for user-supplied
   --github-repo input.
2026-05-17 00:51:21 -07:00
Trevin Chow a3f173dc8a docs(readme): drop community-video link from "Going deeper" callout
The 6-min architecture walkthrough video is the contributor's own
work — not affiliated with the project. Removing the README pointer
to avoid implying endorsement. CONFIGURATION.md prose stands.
2026-05-17 00:51:14 -07:00
Trevin Chow 5a3ac8ca37 docs(config): document briefing.py show [--date DATE] subcommand 2026-05-17 00:50:36 -07:00
Trevin Chow e8eb15102f docs(agents): steer agents on maintaining CONFIGURATION.md 2026-05-17 00:50:36 -07:00
Daniel Zivkovic fd6e70c539 docs: add CONFIGURATION.md + README pointers + community video link
Adds CONFIGURATION.md at repo root - a focused configuration reference
covering save paths, the per-source API-key matrix, reasoning and
web-search backend priority, the trend-monitoring stack (--store +
watchlist.py + briefing.py), and per-client patterns.

Surfaces two things that ship in the engine but were not documented
for users:

- The project-scoped .claude/last30days.env config file (currently only
  referenced in hooks/scripts/check-config.sh) which takes precedence
  over the global ~/.config/last30days/.env when present. Cleanest
  pattern for per-client setups - drop a file in the client folder, cd
  in, run normally.
- The existing trend-monitoring scripts (--store flag, watchlist.py,
  briefing.py) that the README did not surface for users.

Updates README with a brief "Configuration" section pointing to the new
file, plus a one-line "Going deeper" callout linking a 6-min community
architecture walkthrough on YouTube.

All CLI surface claims (watchlist subcommands, briefing modes, source
dedupe key, env file priority chain) fact-checked against the live
scripts/ source before commit.
2026-05-17 00:50:36 -07:00
nidhi-singh02 d0b990e211 Canonicalize GitHub repo resolution for ambiguous product repos 2026-05-17 00:50:05 -07:00
Trevin Chow 0f03a67166 Merge pull request #343 from Bortlesboat/codex/use-sandboxed-safari-cookie-path
fix: prefer sandboxed Safari cookie path
2026-05-17 00:47:32 -07:00
Trevin Chow 5a625fda9f Merge pull request #355 from dinakars777/test/cover-parallel-grounding-backend
test: cover parallel grounding backend
2026-05-17 00:47:20 -07:00
Trevin Chow b296a65515 fix(last-run): guard python3 absence + hoist datetime + use context manager 2026-05-17 00:44:30 -07:00
Trevin Chow f2737fc035 test(grounding): fix published_date → publish_date mock key mismatch 2026-05-17 00:42:07 -07:00
Trevin Chow 9ce7264d43 test(grounding): add serper>parallel priority + parallel empty-results coverage 2026-05-17 00:41:19 -07:00
Dinakar Sarbada f458e0f5af test: cover parallel grounding backend 2026-05-17 00:41:19 -07:00
Trevin Chow 8f565ee241 fix(chrome_cookies): sort Brave profiles by mtime, not alphabetically 2026-05-17 00:40:10 -07:00
Amit Patnaik dd7e6a1562 Preserve clean mode for last run state 2026-05-17 00:39:44 -07:00
KG 65313ce542 feat(cookies): add Brave browser cookie extraction for macOS
Brave uses identical v10 AES-128-CBC encryption to Chrome; only the
DB path (BraveSoftware/Brave-Browser) and Keychain service name
("Brave Safe Storage") differ. Refactored chrome_cookies.py to share
a single _extract_chromium_cookies_macos helper rather than duplicating
the decryption logic.

Profile discovery tries Default/ first, then scans numbered Profile N/
directories so non-default Brave profiles are covered.
2026-05-17 00:39:30 -07:00
Trevin Chow 5ab8c3ba76 Merge pull request #345 from dinakars777/docs/fix-stale-script-paths
docs: fix stale script paths
2026-05-17 00:38:17 -07:00
Trevin Chow 38bfb504e1 Merge pull request #349 from dinakars777/chore/sync-gemini-extension-version
chore: sync gemini extension version
2026-05-17 00:38:01 -07:00
Trevin Chow e8f23b4205 Merge pull request #340 from dzivkovi/fix/youtube-transcript-observability
fix(youtube): surface transcript-fetch ratio + add degraded nudge for stale yt-dlp
2026-05-17 00:31:37 -07:00
Trevin Chow b78ce34922 test(safari_cookies): add coverage for legacy fallback path 2026-05-17 00:31:09 -07:00
Bortlesboat 0656b868e7 fix safari cookie path resolution 2026-05-17 00:30:25 -07:00
Trevin Chow 16a4fa9c39 Merge pull request #341 from flyingice/main
fix(grounding): align Parallel AI search with current API schema
2026-05-17 00:27:50 -07:00
Trevin Chow 3b75ff1537 Merge pull request #354 from dinakars777/fix/parallel-web-backend-source
fix: route parallel web backend through grounding
2026-05-17 00:26:50 -07:00
Trevin Chow 321975e144 Merge pull request #356 from dinakars777/fix/allow-threads-pinterest-search
fix: honor explicit optional source requests
2026-05-17 00:26:25 -07:00
Trevin Chow 5f9c637bbe Merge pull request #336 from davemorin/fix/319-xquik-source-capabilities
fix(planner): register xquik in SOURCE_CAPABILITIES (#319)
2026-05-17 00:26:08 -07:00
Trevin Chow 19132b0b5e Merge pull request #338 from dzivkovi/fix/windows-save-path-footer
fix(render): use forward slashes in save-path footer for Windows
2026-05-17 00:25:45 -07:00
Trevin Chow 1e4150ad78 Merge pull request #347 from dinakars777/docs/update-how-search-key-files
docs: update search key file paths
2026-05-17 00:24:55 -07:00
Trevin Chow 6a15afd8e8 Update skills/last30days/scripts/last30days.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-05-17 00:23:53 -07:00
Trevin Chow 0ab7051bc5 fix(planner): also register xquik in QUICK_SOURCE_PRIORITY 2026-05-17 00:22:04 -07:00
Trevin Chow ec0b126af6 test(pipeline): relax grounding assertion to stable source key 2026-05-17 00:21:57 -07:00
Trevin Chow d9e8a046ef docs(how-search): swap score.py->relevance.py at line 145 2026-05-17 00:21:36 -07:00
Dave Morin 87bf3debcc fix(planner): register xquik in SOURCE_CAPABILITIES (#319)
Without this entry, the planner's _default_sources_for_intent() drops
xquik from the candidate pool for how_to / comparison / news intents
because SOURCE_CAPABILITIES.get("xquik", set()) returns the empty set.
Users with XQUIK_API_KEY set get zero Xquik results even though the
engine recognizes the key.

Mirrors the capabilities for "x" since both are X/Twitter-shaped
discussion + social sources.

Fixes #319
2026-05-17 00:21:36 -07:00
Dinakar Sarbada 2f4b023db8 docs: update search key file paths 2026-05-17 00:21:15 -07:00
Trevin Chow 261ea5895c refactor(pipeline): remove dead threads-explicit-request branch 2026-05-17 00:20:56 -07:00
Trevin Chow 2692e0f4a2 Merge pull request #351 from dinakars777/docs/fix-changelog-skill-link-note
docs: correct changelog skill link note
2026-05-17 00:20:46 -07:00
Trevin Chow 10f35f82fe fix(grounding): guard parallel excerpts against None + cap at 500 chars 2026-05-17 00:20:27 -07:00
Dinakar Sarbada 5b29b8f427 fix: honor explicit perplexity source requests 2026-05-17 00:20:25 -07:00
Dinakar Sarbada 6a5a122195 fix: honor explicit threads source requests 2026-05-17 00:20:25 -07:00
Dinakar Sarbada 7bda02169d fix: allow threads and pinterest search sources 2026-05-17 00:20:25 -07:00
Trevin Chow 6b40d2c46f Merge pull request #346 from dinakars777/docs/fix-bug-report-repro-command
docs: fix bug report repro command
2026-05-17 00:20:21 -07:00
Trevin Chow 4a99c4f557 Merge pull request #337 from UncleMike1988/fix/path-quoting-spaces
Fix path-quoting in SessionStart check-config hook (handles spaces in…
2026-05-17 00:20:09 -07:00
Dinakar Sarbada 5c802b0daa fix: route parallel web backend through grounding 2026-05-17 00:20:05 -07:00
Trevin Chow 0e353ae03f fix(render): apply as_posix to fallback branch + hoist shutil import 2026-05-17 00:19:49 -07:00
Daniel Zivkovic 0a5102e193 fix(youtube): surface transcript-fetch ratio in footer + add degraded nudge
When yt-dlp is installed but stale (or otherwise unable to fetch transcripts
for any returned videos), runs previously reported YouTube as fully
successful in two user-facing surfaces:

  1. Footer (render.py): showed "N videos | M views" with no indication
     that zero transcripts were captured. The "with transcripts" segment
     was conditionally suppressed when the count was zero - converting
     the canonical stale-binary failure mode into a silent absence at
     the very surface users read for "did this work?".

  2. Quality nudge (quality_nudge.py): classified YouTube as "active"
     based purely on yt-dlp installation + absence of a top-level error.
     Per-video transcript-fetch ratio was never inspected. A run that
     returned N videos with 0 transcripts (canonical stale-binary
     failure) was reported as fully active.

The engine itself logs the failure correctly at default stderr level
(`[YouTube] Got transcripts for 0/N videos (N failed)`), but that line
gets buried in 100+ lines of parallel-source progress output and is
contradicted by the success-shaped footer and nudge that follow.

This change makes both conclusion surfaces honest:

* render.py footer always renders "M/N with transcripts" so the ratio
  is visible regardless of value. Zero is no longer hidden. Format is
  M/N (not bare M) so the denominator is in the message and the user
  does not have to cross-reference the "videos" count.

* quality_nudge.py adds a third tier between "active" and "missing":
  "degraded". Triggered when yt-dlp is installed AND videos were
  returned AND transcript-fetch ratio is below threshold (default 50%,
  tunable via DEGRADED_TRANSCRIPT_THRESHOLD env var). Emits an
  actionable nudge: "YouTube returned N videos but only M transcripts
  captured. The most common cause is a stale yt-dlp binary - YouTube's
  caption format changes frequently and old binaries silently fail
  every transcript. Update via your package manager: scoop update
  yt-dlp (Windows), brew upgrade yt-dlp (macOS), or pip install -U
  yt-dlp."

* last30days.py populates youtube_videos_count and
  youtube_transcripts_count in the research_results dict it passes to
  compute_quality_score, enabling the new degraded check at the call
  site.

Threshold rationale: 50% accommodates a few legitimate
caption-disabled videos in a multi-video result, but a stale-binary
run that fails every transcript trips the nudge cleanly.

Score impact: degradation is informational, not score-affecting.
YouTube still counts as "active" in score_pct so users do not see
their score drop for a fixable client-side issue. The nudge directs
them to their own package manager.

Tests:

* tests/test_quality_nudge.py: 6 new TestYouTubeDegraded cases cover
  zero-transcripts-flags-degraded, partial-above-threshold-does-not-flag,
  zero-videos-does-not-flag (no false positives on absence),
  one-of-three-flags-degraded, threshold-tunable-via-config, and
  degraded-does-not-affect-score.

* tests/test_render_v3.py: 4 new YoutubeFooterTranscriptRatioTests
  cases cover zero-transcripts-with-videos-renders-zero-over-total
  (the regression repro), partial-renders-ratio, full-renders-ratio,
  and no-videos-suppresses-entire-segment.

All 29 new test cases verified GREEN with the fix and RED without it
(temp-reverted both files separately to confirm each test catches the
specific regression it asserts).

Integration validation: ran the engine against an intentionally stale
yt-dlp 2025.03.31 binary placed first on PATH. Pre-fix the footer
showed `YouTube: 3 videos | 386,815 views` (no transcript signal).
Post-fix the footer shows `YouTube: 3 videos | 386,815 views | 0/3
with transcripts` and stderr emits "Degraded: YouTube" plus the
actionable update-yt-dlp nudge.

Out of scope (deserves its own PR): exposing transcripts_captured in
the EVIDENCE FOR SYNTHESIS block so the synthesizing model can flag
degradation in prose. Larger schema-touching change.
2026-05-17 00:13:56 -07:00
Dinakar Sarbada cfde1dbbe4 docs: fix stale script paths 2026-05-17 00:08:57 -07:00
Dinakar Sarbada 608381a818 docs: fix bug report repro command 2026-05-17 00:08:17 -07:00
flyingice af4cf7c03d fix(grounding): align Parallel AI search with current API schema 2026-05-17 00:08:10 -07:00
Michael Turner 5b0308b9e4 Fix path-quoting in SessionStart check-config hook (handles spaces in CLAUDE_PLUGIN_ROOT)
If CLAUDE_PLUGIN_ROOT ever expands to a path containing whitespace
(e.g. ~/Library/Application Support/...), the unquoted ${CLAUDE_PLUGIN_ROOT}
in hooks/hooks.json word-splits and bash receives the path as multiple
arguments, failing with "No such file or directory" on the first split.

Quoting the expansion makes the invocation correct regardless of the
characters in the resolved path. Verified manually:
  unquoted + space  -> bash: /tmp/with: No such file or directory
  quoted   + space  -> bash: /tmp/with spaces/.../check-config.sh: No such file
  quoted   + real   -> /last30days: Ready - 7 sources active.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 00:08:01 -07:00
Daniel Zivkovic 5817ef8387 test(cli): regression test for Windows save-path display
Asserts compute_save_path_display() never returns a backslash when the
save_dir is under the user's home directory, regardless of host OS.

Reproduces the original bug on Windows (failed message before the fix:
  AssertionError: '\' unexpectedly found in
  '~/l30d_save_path__luu2g76\Documents\Last30Days\british-airways-middle-east-raw-v3.md'
)
and locks in the contract on POSIX hosts (passes trivially today; would
fail if anyone removes .as_posix() in the future).

Verified by temporarily reverting the fix and confirming RED, then
re-applying the fix and confirming GREEN. All 13 CliV3Tests pass.
2026-05-17 00:07:44 -07:00
Daniel Zivkovic a87c1ba058 fix(render): use forward slashes in save-path footer for Windows
The footer line `📎 Raw results saved to ~/Documents\Last30Days\…`
mangled the home-relative path on Windows because `f"~/{relative}"`
stringifies a `pathlib.Path` with the OS-native separator. The result
mixes a Unix tilde with backslashes, which neither File Explorer,
PowerShell, nor a `file://` URI can resolve.

`Path.as_posix()` always returns forward slashes, which is the
convention `~/`-prefixed paths require on every platform. macOS and
Linux output is unchanged because their separator is already `/`.

Repro on Windows:
  python3 last30days.py "anything" --emit=compact --save-dir="$HOME/Documents/Last30Days"
  # before: 📎 Raw results saved to ~/Documents\Last30Days\anything-raw.md
  # after:  📎 Raw results saved to ~/Documents/Last30Days/anything-raw.md
2026-05-17 00:07:44 -07:00
Trevin Chow 7214dd6051 Merge pull request #348 from dinakars777/docs/fix-readme-skill-link
docs: fix runtime skill spec link
2026-05-17 00:04:30 -07:00
Trevin Chow 6acf2fdbe2 Merge pull request #419 from mvanhorn/chore/remove-orphaned-spec-tasks
chore: remove orphaned SPEC.md and TASKS.md
2026-05-17 00:03:06 -07:00
Trevin Chow 07a3bdb3cf Merge pull request #364 from davemorin/fix/361-unsafe-eval-check-config
fix(hooks): replace unsafe eval with declare in check-config.sh
2026-05-17 00:02:44 -07:00
Dinakar Sarbada 87577ff126 test: guard gemini extension version 2026-05-17 00:00:59 -07:00
Dinakar Sarbada 400fc4cc00 docs: fix runtime skill spec link 2026-05-17 00:00:58 -07:00
Dinakar Sarbada e9ecce0b1c chore: sync gemini extension version 2026-05-17 00:00:51 -07:00
Dinakar Sarbada c9cf3ef92f docs: correct changelog skill link note 2026-05-16 23:59:30 -07:00
Trevin Chow 46cf2328aa fix(hooks): use printf -v for bash 3.2 compat (declare -g is 4.2+)
macOS ships /bin/bash 3.2 and the script uses #!/bin/bash with
set -euo pipefail, so declare -g would abort the SessionStart hook
with "invalid option" on every Mac. printf -v writes via assignment
semantics (global from inside a function on 3.2+) — same scope
outcome, broader compatibility.
2026-05-16 23:49:44 -07:00
Trevin Chow 7506cbd542 fix(hooks): scope ENV_* to global (declare -g) so caller sees values 2026-05-16 23:49:05 -07:00
Dave Morin a6bd481e61 fix(hooks): replace unsafe eval with declare in check-config.sh
The load_env_vars function used eval to assign .env values, which
executes command substitutions in backtick-containing comments.
Replace eval with declare and strip inline comments before assignment.

Fixes #361
2026-05-16 23:49:05 -07:00
51 changed files with 2482 additions and 203 deletions
+1 -1
View File
@@ -11,7 +11,7 @@
{
"name": "last30days",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"version": "3.2.4",
"version": "3.3.0",
"author": {
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "last30days",
"version": "3.2.4",
"version": "3.3.0",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"author": {
"name": "Matt Van Horn",
+1 -1
View File
@@ -16,7 +16,7 @@ body:
label: Steps to Reproduce
description: How can we reproduce this?
placeholder: |
1. Run `python3 scripts/last30days.py "topic" --emit compact`
1. Run `python3 skills/last30days/scripts/last30days.py "topic" --emit=compact`
2. ...
validations:
required: true
+4
View File
@@ -28,3 +28,7 @@ htmlcov/
# Internal planning docs (ce:plan output) — keep local, don't publish
docs/plans/
.context/
/work
/print
+16
View File
@@ -33,6 +33,22 @@ npx skills add . -g -y # one-time: symlink this repo into every detected harne
- Keep examples safe by redacting secrets and avoiding copy/pasteable live credentials in docs, fixtures, and test data.
- Do not weaken or disable the advisory security workflow (`.github/workflows/security.yml`) without explaining why in the PR description or review thread.
## Maintaining CONFIGURATION.md
`CONFIGURATION.md` is the user-facing configuration reference — save paths, per-source API keys, web-search backend priority, trend-monitoring stack, per-client install patterns. Distinct from `SKILL.md` (the canonical runtime spec).
Update `CONFIGURATION.md` when:
- adding a new env var (e.g. `LAST30DAYS_*`, `BSKY_*`, `*_API_KEY`)
- adding a new CLI flag that affects configuration (e.g. `--store`, `--web-backend`)
- adding a new per-client install pattern (Claude Code, Gemini, Codex, Cursor, Hermes…)
- adding a new optional source that requires its own credential
- changing the priority order of config layers (per-run flag > env > `.env` file > defaults)
Keep the existing structure organized by how often each layer is touched: per-run flags → env vars / `.env` → optional trend-monitoring stack → per-client patterns. Add new content into the right section rather than appending at the end.
When a new config concept lands in `SKILL.md` or `AGENTS.md`, mirror the user-facing knob in `CONFIGURATION.md` so non-agent readers can configure the skill without reverse-engineering it from the runtime spec.
## Beta channel
Experimental changes get tested on `mvanhorn/last30days-skill-private`, which installs as a parallel `/last30days-beta` slash command. Beta-only changes never ship to public without a review PR here. Workflow guide lives at `BETA.md` in the private repo. Plan that established this setup: `docs/plans/2026-04-17-005-feat-beta-skill-from-private-repo-plan.md`.
+138 -9
View File
@@ -7,23 +7,152 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [3.3.0] - 2026-05-17
A week-long shipping cycle: ~75 PRs merged plus 7 community fixes salvaged through PR triage. Big themes: install story modernized for the multi-harness world (Claude Code, Codex, Cursor, Gemini CLI, Copilot, Windsurf, and 50+ Agent Skills hosts), new emit and source modes, and a substantial reliability sweep across Reddit, X, Windows, YouTube, and the planner.
### Added
- `LAST30DAYS_YOUTUBE_SSH_HOST` env var: when set, yt-dlp YouTube search invocations are routed through `ssh <host>` for residential-IP egress. Bypasses YouTube's bot-wall on datacenter IPs (Hetzner/DigitalOcean/AWS) where `ytsearch:` returns 0 results regardless of cookies (the IP fingerprint is checked first). The named host must be configured in `~/.ssh/config` and have yt-dlp installed. Host value is validated against `^[a-zA-Z0-9._-]+$` to reject SSH option-injection (e.g. a leading `-` masquerading as a flag). The transcript path is unchanged (uses the existing HTTP fallback when SSH-routing is on, since the timedtext API isn't bot-walled).
**Emit modes and sources**
- `--emit=html` for shareable, print-friendly HTML research briefs ([#332](https://github.com/mvanhorn/last30days-skill/pull/332)).
- **Digg AI 1000 source**, auto-enabled when `digg-pp-cli` is on PATH ([#370](https://github.com/mvanhorn/last30days-skill/pull/370)). Surfaces curated story clusters from the AI 1000 leaderboard and pulls attributable X-post quotes into the brief.
**Configuration knobs**
- `EXCLUDE_SOURCES` env var — the inverse of `INCLUDE_SOURCES`, honored in source count and pipeline filter ([#399](https://github.com/mvanhorn/last30days-skill/pull/399)).
- `LAST30DAYS_YOUTUBE_SSH_HOST` — opt-in SSH routing for `yt-dlp` through a residential-IP host, for users on datacenter VPS hit by YouTube's bot-wall ([#376](https://github.com/mvanhorn/last30days-skill/pull/376)). Host validated against `^[a-zA-Z0-9._-]+$` to reject SSH option-injection. Transcript path unchanged (uses HTTP fallback).
- macOS Keychain as a credential source — reads from the system keychain when env vars and config files aren't set ([#407](https://github.com/mvanhorn/last30days-skill/pull/407)).
- Configuration enablement: env-var defaults and source-resilience patterns across the config layer ([#344](https://github.com/mvanhorn/last30days-skill/pull/344)).
**Pipeline and storage**
- Reddit URL auto-enrichment from web search via the public JSON API ([#366](https://github.com/mvanhorn/last30days-skill/pull/366)).
- Per-run finding sightings recorded in the SQLite store ([#373](https://github.com/mvanhorn/last30days-skill/pull/373)).
- Brave browser support for X/Twitter cookie extraction ([#320](https://github.com/mvanhorn/last30days-skill/pull/320)).
**Tests and CI**
- Full pytest suite restored to CI; 13 rotted tests repaired ([#416](https://github.com/mvanhorn/last30days-skill/pull/416)).
- `greptile.json` added with `triggerOnUpdates` + `statusCheck` ([#418](https://github.com/mvanhorn/last30days-skill/pull/418)).
- Advisory security workflow ([#368](https://github.com/mvanhorn/last30days-skill/pull/368)).
- Parallel grounding backend test coverage ([#355](https://github.com/mvanhorn/last30days-skill/pull/355)).
**Docs**
- New `CONFIGURATION.md` with README pointers ([#339](https://github.com/mvanhorn/last30days-skill/pull/339)).
- `docs/solutions/` learning capture for release-time consistency-test cascades ([#413](https://github.com/mvanhorn/last30days-skill/pull/413)) and the eval-not-in-CI design decision ([#417](https://github.com/mvanhorn/last30days-skill/pull/417)).
### Changed
- Replace the SKILL_ROOT resolver loops in Step 1 and comparison-mode with a single `SKILL_DIR` substitution pattern. The model templates the absolute path of the SKILL.md's own directory (which it always knows from the Read tool result); the bash block just validates that `scripts/last30days.py` lives there. Removes ~80 lines of bash across the two locations. Fixes a real bug: the previous resolver could pick a different install than the SKILL.md the model loaded from (spec-vs-engine divergence) and didn't enumerate harnesses like Hermes at all. The simplification works for any harness without enumeration because it just uses wherever SKILL.md was loaded from. STEP 0's marketplaces-stale-clone hop is unchanged.
- Rename "Digg AI 1000" to just "Digg" in user-facing output (footer line, source label, inline-quote suffix, why_relevant, container attribution). Internal references to the upstream Digg AI 1000 product remain in code comments and docstrings.
- Bump `POSTS_PER_CLUSTER` from 3 to 5 and the render-side display limit from 2 to 3 to match the per-source enrichment caps used by Reddit, HN, YouTube, TikTok, and GitHub. The previous 3/2 caps routinely truncated cluster context (e.g. dropped a Jason Calacanis quote tweet on a `cli-printing-press` run).
- Rewrite SKILL.md path resolution. STEP 0 narrows from a global canonical-path enforcement to a Claude-Code-marketplaces-only stale-clone guard. Step 1 SKILL_ROOT resolver walks a single precedence list (Claude plugin cache, then `~/.codex/skills/`, `~/.agents/skills/`, repo checkout, `./.skills/last30days` for `npx skills add`, CWD, Gemini). Adds SKILL.md frontmatter fallback to `render.py::_skill_version` so the badge no longer prints `v?` on installs that don't include `.claude-plugin/plugin.json`.
**Install story modernized**
- Switch SKILL.md's `--plan` and `--competitors-plan` invocation templates from inline single-quoted JSON to heredoc-written tmpfiles. Apostrophes in resolved context strings ("McDonald's", "people's choice", "developer's") previously closed the outer single-quote and broke shell parsing before the engine started — observed in a Codex run during PR #400 testing. The engine's `parse_plan()` / `parse_competitors_plan()` already supported file paths (via `os.path.isfile()` probe); only the template prose changed. Fixes [#403](https://github.com/mvanhorn/last30days-skill/issues/403).
- `npx skills add` is now the canonical install path for every harness ([#405](https://github.com/mvanhorn/last30days-skill/pull/405)). README and SKILL.md flipped to recommend `npx skills add . -g -y` over per-harness manual instructions. Surfaces Gemini CLI, Copilot, Windsurf, and 50+ other Agent Skills hosts that the install pattern reaches.
- README dropped the Gemini CLI native-extension install path (now covered by `npx skills add`).
- `hooks.json` made polyglot for Gemini CLI + Claude Code compatibility ([#318](https://github.com/mvanhorn/last30days-skill/pull/318)).
**Skill semantics and multi-harness reframe**
- `AGENTS.md` is now canonical; `CLAUDE.md` points at it ([#410](https://github.com/mvanhorn/last30days-skill/pull/410)). Reframes the project as a multi-harness Agent Skills package rather than a Claude-Code-specific tool.
- SKILL.md path resolution rewritten: STEP 0 narrows to a Claude-Code-marketplaces-only stale-clone guard; Step 1 walks a single `SKILL_DIR` substitution pattern ([#400](https://github.com/mvanhorn/last30days-skill/pull/400), [#409](https://github.com/mvanhorn/last30days-skill/pull/409)). Removes ~80 lines of bash and fixes a real spec-vs-engine divergence where the previous resolver could pick a different install than the SKILL.md the model loaded from.
- SKILL.md version regex consolidated into `lib/skill_meta.py` ([#412](https://github.com/mvanhorn/last30days-skill/pull/412)).
- `--plan` / `--competitors-plan` invocation templates switched from inline single-quoted JSON to heredoc-written tmpfiles ([#404](https://github.com/mvanhorn/last30days-skill/pull/404), fixes [#403](https://github.com/mvanhorn/last30days-skill/issues/403)). Apostrophes in resolved context strings ("McDonald's", "people's choice") no longer break shell parsing.
- `POSTS_PER_CLUSTER` raised 3→5 and render-side display limit 2→3 to match the per-source enrichment caps used by Reddit, HN, YouTube, TikTok, and GitHub. The previous caps routinely truncated cluster context.
- Digg AI 1000 renamed to "Digg" in user-facing output ([#372](https://github.com/mvanhorn/last30days-skill/pull/372)) — footer line, source label, inline-quote suffix, why_relevant, container attribution. Internal references retain the upstream product name.
- GitHub repo resolution canonicalized for ambiguous product comparisons ([#302](https://github.com/mvanhorn/last30days-skill/pull/302)).
**Dependencies and tooling**
- Dropped `requests` runtime dependency. All providers route through stdlib `urllib` via the `lib/http` wrapper ([#393](https://github.com/mvanhorn/last30days-skill/pull/393)).
- Migrated to `gemini-3.1-flash-lite` GA model ([#378](https://github.com/mvanhorn/last30days-skill/pull/378)).
- Aligned Codex/Claude plugin manifests + added Codex `AGENTS.md` ([#321](https://github.com/mvanhorn/last30days-skill/pull/321)).
- pytest dev dep bumped 9.0.2 → 9.0.3 ([#414](https://github.com/mvanhorn/last30days-skill/pull/414)).
### Removed
- **BREAKING for Codex native-plugin users:** `.codex-plugin/plugin.json` and the matching SKILL_ROOT resolver branch in SKILL.md Step 1. Codex users should install via `npx skills add mvanhorn/last30days-skill` or copy the skill to `~/.codex/skills/last30days/`.
- **`skills/last30days/scripts/sync.sh`.** The maintainer dev-deploy script is gone. Every job it did has a better replacement: `npx skills add . -g -y` symlinks the working tree into every detected harness's skill dir (better than sync.sh's copy model edits propagate live), `hermes skills install mvanhorn/last30days-skill --force` handles Hermes, `clawhub install last30days-official` handles OpenClaw, and the Claude marketplace cache target was a "test against the official install path" hack we shouldn't have been recommending in the first place. The `test_sync_cache_path_uses_skill_version` test was dropped along with it. CLAUDE.md, HERMES_SETUP.md, the PR template, and a render.py docstring were updated to drop references; CHANGELOG and historical docs (release notes, plan files) keep their existing mentions as accurate history.
- **BREAKING for Codex native-plugin users:** `.codex-plugin/plugin.json` and the matching SKILL_ROOT resolver branch in SKILL.md Step 1 ([#400](https://github.com/mvanhorn/last30days-skill/pull/400)). Codex users should install via `npx skills add mvanhorn/last30days-skill` or copy the skill to `~/.codex/skills/last30days/`.
- **`skills/last30days/scripts/sync.sh`** maintainer dev-deploy script ([#405](https://github.com/mvanhorn/last30days-skill/pull/405)). Replaced by `npx skills add . -g -y` (live-symlink into every detected harness's skill dir better than sync.sh's copy model since edits propagate live). Hermes uses `hermes skills install mvanhorn/last30days-skill --force`; OpenClaw uses `clawhub install last30days-official`.
- Orphaned `SPEC.md` and `TASKS.md` ([#419](https://github.com/mvanhorn/last30days-skill/pull/419)).
### Fixed
**Reddit**
- `lstrip("r/")` mangled subreddits starting with `r` (`r/robotics``obotics`, `r/ruby``uby`); replaced with `removeprefix("r/")` at 4 sites (Alex Key, salvaged from #288).
- Browser-like User-Agent + `Accept-Language`/`Accept-Encoding`/`Connection` headers + gzip decompression to fix `urllib` 403s on Reddit's public JSON endpoint (Franco Carballar, salvaged from #199).
- HTTP 402 re-raised across all three ScrapeCreators paths (`_global_search`, `_subreddit_search`, `fetch_post_comments`) so the OpenAI/public-JSON fallback chain triggers when credits are exhausted (Jonathan Oppenheim, salvaged from #170).
**Authentication and credentials**
- Restored multi-key rotation for `SCRAPECREATORS_API_KEY` accidentally dropped in v3.0.6 (Eric Oberhofer, salvaged from #287). Comma-separated keys round-robin via `random.choice` per run.
**Windows compatibility**
- `os.killpg` in `_cleanup_children()` guarded with `hasattr(os, "killpg")`, falls back to `os.kill(SIGTERM)` (gujishh, salvaged from #226).
- POSIX-style secret-permission warning skipped on Windows ([#357](https://github.com/mvanhorn/last30days-skill/pull/357)).
- Render uses forward slashes in save-path footer for Windows ([#338](https://github.com/mvanhorn/last30days-skill/pull/338)).
**xAI / X / xurl**
- `parse_x_response` now raises `http.HTTPError` on empty output, missing JSON, or decode failure — surfaces in `errors_by_source` instead of silently returning an empty result list (Kaustav Mishra, salvaged from #155).
- `xurl` treats `PermissionError` from PATH lookup as unavailable ([#322](https://github.com/mvanhorn/last30days-skill/pull/322)).
**YouTube**
- SC YouTube + multi-token HN searches unblocked ([#388](https://github.com/mvanhorn/last30days-skill/pull/388)).
- Transcript-fetch ratio surfaced + degraded-run nudge for stale `yt-dlp` ([#340](https://github.com/mvanhorn/last30days-skill/pull/340)).
**bird_x / HTTP**
- Subprocess retry on non-JSON stdout to handle X anti-bot HTML interstitials ([#383](https://github.com/mvanhorn/last30days-skill/pull/383)).
- HTTP retry budget expanded + exponential backoff on DNS resolution failure ([#382](https://github.com/mvanhorn/last30days-skill/pull/382)).
- Parallel AI search aligned with current API schema ([#341](https://github.com/mvanhorn/last30days-skill/pull/341)).
- Parallel web backend routed through grounding ([#354](https://github.com/mvanhorn/last30days-skill/pull/354)).
**Planner and sources**
- `xquik` registered in `SOURCE_CAPABILITIES` ([#336](https://github.com/mvanhorn/last30days-skill/pull/336), fixes [#319](https://github.com/mvanhorn/last30days-skill/issues/319)).
- Honor explicit optional source requests ([#356](https://github.com/mvanhorn/last30days-skill/pull/356)).
- ScrapeCreators source-gating aligned between code and docs ([#415](https://github.com/mvanhorn/last30days-skill/pull/415)).
- OpenClaw works without ScrapeCreators key ([#392](https://github.com/mvanhorn/last30days-skill/pull/392), by @thinkun).
**Render, version display, hosting paths**
- Hardcoded `v3.0.0` in render replaced with dynamic `_skill_version()` ([#365](https://github.com/mvanhorn/last30days-skill/pull/365)).
- Comparison HTML artifacts saved correctly ([#389](https://github.com/mvanhorn/last30days-skill/pull/389)).
- `OPENROUTER_DEFAULT` model ID corrected ([#323](https://github.com/mvanhorn/last30days-skill/pull/323)).
- OpenClaw poll-timing initialized once ([#358](https://github.com/mvanhorn/last30days-skill/pull/358)).
- Prefer sandboxed Safari cookie path ([#343](https://github.com/mvanhorn/last30days-skill/pull/343)).
- Preserve clean mode for last-run state ([#334](https://github.com/mvanhorn/last30days-skill/pull/334)).
- Replaced hardcoded `/Users/mvanhorn/...` paths in `test-v1-vs-v2.sh` with portable env-var overrides (Dave Morin, salvaged from #297).
**Hooks**
- `check-config.sh` path-quoting fix for paths with spaces ([#337](https://github.com/mvanhorn/last30days-skill/pull/337)).
- Replaced unsafe `eval` with `declare` in `check-config.sh` ([#364](https://github.com/mvanhorn/last30days-skill/pull/364)).
**Sync and version metadata**
- `sync.sh` pointed at this repo's plugin cache, not the private repo's ([#402](https://github.com/mvanhorn/last30days-skill/pull/402)).
- Sync cache target bumped to 3.2.1 to match SKILL.md ([#397](https://github.com/mvanhorn/last30days-skill/pull/397)).
- ScrapeCreators free-tier credit count corrected to 100 in docs ([#369](https://github.com/mvanhorn/last30days-skill/pull/369), fixes [#367](https://github.com/mvanhorn/last30days-skill/issues/367)).
- Gemini extension version synced ([#349](https://github.com/mvanhorn/last30days-skill/pull/349)).
- Various stale path/link fixes ([#345](https://github.com/mvanhorn/last30days-skill/pull/345), [#346](https://github.com/mvanhorn/last30days-skill/pull/346), [#347](https://github.com/mvanhorn/last30days-skill/pull/347), [#348](https://github.com/mvanhorn/last30days-skill/pull/348), [#351](https://github.com/mvanhorn/last30days-skill/pull/351)).
### Contributors
First-time contributors whose fixes shipped in this release (most via PR triage salvage — fix re-applied directly to main with co-author credit when path migration made the original branch un-rebaseable):
- Dave Morin — portable test-harness paths
- Alex Key — `removeprefix("r/")` for subreddit names
- Eric Oberhofer — multi-key rotation restored
- gujishh — Windows process cleanup
- Franco Carballar — Reddit browser-like headers
- Jonathan Oppenheim — Reddit 402 fallback chain
- Kaustav Mishra — xAI error surfacing
- [@thinkun](https://github.com/thinkun) ([#363](https://github.com/mvanhorn/last30days-skill/pull/363)) — OpenClaw ScrapeCreators-key-optional fix
Full PR list at [github.com/mvanhorn/last30days-skill/releases/tag/v3.3.0](https://github.com/mvanhorn/last30days-skill/releases/tag/v3.3.0).
## [3.2.0] - 2026-05-09
@@ -50,7 +179,7 @@ Consolidates the 3.0.10 to 3.0.14 dev cycle (commenter handles, `--competitors`,
### Fixed
- **Claude Code plugin manifest path-escape.** The `.claude-plugin/plugin.json` `skills` key was removed in commit `93fbed2` but never shipped in a tagged release. Installing via `/plugin install last30days-skill` could hit `/doctor`'s `Path escapes plugin directory: ./ (skills)` error. This release ships the fix. Closes [#306](https://github.com/mvanhorn/last30days-skill/issues/306).
- **Broken README link.** The README's "source of truth" link pointed at `skills/last30days/SKILL.md`, a path that does not exist. Fixed to point at root `SKILL.md`.
- **Broken README link.** The README's "source of truth" link pointed at root `SKILL.md`, which is no longer maintained after the plugin-layout restructure. Fixed to point at `skills/last30days/SKILL.md`.
### Dev cycle journal (3.0.10 - 3.0.14, not separately tagged)
+268
View File
@@ -0,0 +1,268 @@
# Configuration
Everything you can tune in `/last30days` without editing the engine source.
Three layers, in order of how often you'll touch them:
1. **Per-run flags** - what you pass on the command line.
2. **Environment variables and `.env`** - what's enabled across all runs.
3. **Optional trend-monitoring stack** - SQLite store, watchlist, briefings.
Per-client patterns and the experimental beta channel are at the bottom.
> Skip ahead: [Where output is saved](#where-output-is-saved) - [API keys](#api-keys-env) - [Reasoning provider](#reasoning-provider-priority) - [Web search backend](#web-search-backend-priority) - [Trend monitoring](#trend-monitoring-store--watchlist--briefings) - [Per-client patterns](#per-client-patterns) - [Beta channel](#beta-channel)
## Why this document exists
This is a focused **configuration reference** maintained alongside the engine. The runtime contract (the voice rules, the planner protocol, the LAWs the synthesizing model follows) lives in [`skills/last30days/SKILL.md`](skills/last30days/SKILL.md) - that file is authoritative when the two ever differ. This file's job is narrower: surface every knob a user or operator can turn, in one place, kept current with the code so client-facing setups stay reliable. New configuration knobs added to the engine should be reflected here in the same PR.
---
## Where output is saved
| Platform | Default path | Override |
|---|---|---|
| Linux / macOS | `LAST30DAYS_MEMORY_DIR` defaults to `~/Documents/Last30Days/` | set `LAST30DAYS_MEMORY_DIR=/path` |
| Windows | `LAST30DAYS_MEMORY_DIR` defaults to `C:\Users\<you>\Documents\Last30Days\` | set `LAST30DAYS_MEMORY_DIR=C:\path` |
Each run produces one file per topic, slug-named:
`<slug>-raw[-suffix].md`. Same topic + same suffix on the same day overwrites; same topic + same suffix on different days appends a date stamp.
**Per-run overrides:**
- `--save-dir <path>` - one-off output location.
- `--save-suffix <name>` - distinguish runs of the same topic (e.g. per client: `--save-suffix=acme`).
The footer line `📎 Raw results saved to ${LAST30DAYS_MEMORY_DIR:-$HOME/Documents/Last30Days}/<slug>-raw.md` is the canonical pointer; if it shows backslashes on Windows update past v3.1.1.
---
## API keys (`.env`)
The skill reads keys from a `.env` file. Two locations are supported, in priority order:
1. **`.claude/last30days.env`** in the current project directory (project-scoped) - takes precedence when present.
2. **`~/.config/last30days/.env`** at the user level (global default) - the fallback.
Override the global location with `LAST30DAYS_CONFIG_DIR=/path` (or `LAST30DAYS_CONFIG_DIR=""` for no-config mode). File permissions should be `600` on POSIX hosts - the engine warns on every run if they aren't.
The project-scoped file is the cleanest pattern for **per-client setups**: drop a `.claude/last30days.env` into each client folder (`SCRAPECREATORS_API_KEY`, `INCLUDE_SOURCES`, `LAST30DAYS_MEMORY_DIR`, `BSKY_HANDLE`, etc), `cd` into that folder, and the skill picks up that client's configuration automatically. No wrapper scripts needed for the common case.
**Source-by-source** - what each key unlocks:
| Source | Key(s) | Required for | Free tier |
|---|---|---|---|
| Reddit (public) | none | always on | yes |
| Hacker News | none | always on | yes |
| Polymarket | none | always on | yes |
| GitHub | `gh` CLI installed (uses your GitHub auth) | always on if `gh` present | yes |
| YouTube | `yt-dlp` CLI installed | always on if `yt-dlp` present | yes |
| X / Twitter | one of: `AUTH_TOKEN` + `CT0` (browser cookies, Bird CLI), `XAI_API_KEY`, `SCRAPECREATORS_API_KEY`, or `FROM_BROWSER` (cookie-jar auth) | X items in results | cookie-jar / Bird = free; xAI / ScrapeCreators = paid |
| TikTok | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `tiktok` | TikTok items | 10K free calls |
| Instagram | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `instagram` | Instagram Reels | 10K free calls; raise `LAST30DAYS_TRANSCRIPT_TIMEOUT` (default 30s) if SC is slow on your network |
| Threads | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `threads` | Threads items | 10K free calls |
| Pinterest | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `pinterest` | Pinterest items | 10K free calls |
| Bluesky | `BSKY_HANDLE` + `BSKY_APP_PASSWORD` | Bluesky items | yes (app password at bsky.app) |
| TruthSocial | `TRUTHSOCIAL_TOKEN` | TruthSocial items | yes |
| Web search | one of: `BRAVE_API_KEY`, `EXA_API_KEY`, `SERPER_API_KEY`, `PARALLEL_API_KEY` | `--auto-resolve` and Step 2 supplements | Brave has a free tier; native WebSearch on Claude Code / Codex / Gemini works as a fallback |
| Perplexity Deep Research | `OPENROUTER_API_KEY` | `--deep-research` flag (~$0.90/query) | no |
| Apify (alternate scraper) | `APIFY_API_TOKEN` | fallback for Reddit/TikTok/Instagram when ScrapeCreators is exhausted | yes (limited) |
**Example `.env` skeleton** (placeholders only - replace with your own values):
```bash
# Reasoning + planning (one provider; see priority below)
GOOGLE_API_KEY=<your-gemini-key>
# Web search backend (one is enough; Brave is the cheapest)
BRAVE_API_KEY=<your-brave-key>
# Optional sources
SCRAPECREATORS_API_KEY=<your-scrapecreators-key>
INCLUDE_SOURCES=tiktok,instagram
# X authentication (one option only)
XAI_API_KEY=<your-xai-key>
# OR cookie-jar (no key needed; logs in via your browser session)
# FROM_BROWSER=firefox
# Bluesky
BSKY_HANDLE=<your-handle>.bsky.social
BSKY_APP_PASSWORD=<your-app-password>
```
After editing: `chmod 600 ~/.config/last30days/.env` (or `chmod 600 .claude/last30days.env` if using the project-scoped variant).
**Troubleshooting:** if a source you expected to see isn't appearing in results, run `python3 scripts/last30days.py --diagnose`. It prints a per-source availability report (which keys were detected, which CLIs are installed, which backends are reachable) without running a full search.
### Bluesky app-password format and search host
`BSKY_APP_PASSWORD` should be a 19-char app password in `xxxx-xxxx-xxxx-xxxx` format (lowercase alphanumeric, three hyphens). Generate one at <https://bsky.app/settings/app-passwords>. The AT Protocol's `createSession` endpoint also accepts your main account login password, but that's bad hygiene — main passwords have no scope (an app password can be limited to non-DM access) and can't be revoked individually.
The skill defaults to `api.bsky.app` for `searchPosts`, which is the canonical authenticated AppView. The previous default `public.api.bsky.app` is the unauthenticated public mirror and is currently blocked by BunnyCDN for `searchPosts` regardless of auth header (verified 2026-05-04). If Bluesky migrates infrastructure again, override the host without a code change by setting `BSKY_SEARCH_HOST` in your `.env`:
```bash
BSKY_SEARCH_HOST=api.bsky.app # default — change only if Bluesky moves
```
---
## Reasoning provider priority
`/last30days` needs one reasoning model for planning + reranking when you don't pass `--plan` yourself. Auto-detect priority (set `LAST30DAYS_REASONING_PROVIDER=<name>` to pin one):
1. **Gemini** - `GOOGLE_API_KEY` / `GEMINI_API_KEY` / `GOOGLE_GENAI_API_KEY`
2. **OpenAI** - `OPENAI_API_KEY` (or Codex auth at `~/.codex/auth.json`)
3. **xAI** - `XAI_API_KEY`
4. **OpenRouter** - `OPENROUTER_API_KEY` (also unlocks `--deep-research`)
5. **Local / deterministic** - always available, lowest quality
When you invoke `/last30days` from Claude Code, Codex, or Gemini, the host model **is** the reasoning provider for plan + synthesis - you don't need any of the keys above unless you also run the script headlessly (cron, CI, watchlist).
---
## Web search backend priority
Used by `--auto-resolve` (when WebSearch isn't available from the host) and Step 2 supplements. Auto-detect priority (override per-run with `--web-backend=<name>`):
1. **Brave** - `BRAVE_API_KEY`
2. **Exa** - `EXA_API_KEY`
3. **Serper** - `SERPER_API_KEY`
4. **Parallel** - `PARALLEL_API_KEY`
5. **Host's native WebSearch** - Claude Code, Codex, Gemini all have one built in
Visible quality difference between hosts with vs without a configured backend. If your client setup produces thinner results than yours, this is usually why.
---
## Trend monitoring (`--store` + watchlist + briefings)
The default behavior - one slug-named file per topic, overwritten on rerun - is the snapshot mode. For continuous monitoring, the repo ships three components most users miss:
### `--store` flag
Adding `--store` to any run persists every finding to a SQLite database (default at `~/.local/share/last30days/research.db`). Findings dedupe on the `source_url` column (UNIQUE constraint), so the same URL across runs updates the existing row instead of creating a duplicate. The markdown file still saves; the SQLite is the time-series substrate.
**Always-on alternative:** set `LAST30DAYS_STORE=1` in your `.env` instead of remembering `--store` on every invocation. The flag still works as before; the env var is purely additive. Same hybrid pattern as `LAST30DAYS_DEBUG` — works whether shell-exported or in `.env`.
Relevant tables: `topics`, `research_runs`, `findings`, `settings`. Schema: [`scripts/store.py`](skills/last30days/scripts/store.py).
### `watchlist.py` - recurring topics
[`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) manages topics that should be researched on a schedule. Subcommands: `add`, `remove`, `list`, `run-one`, `run-all`, `config`. Built-in delivery to Slack incoming webhooks (`hooks.slack.com/...`) or any HTTPS endpoint, fired only when new findings appear.
Two-step flow (the watchlist holds the topic; an external scheduler invokes the run):
```bash
# 1. Add the topic to the watchlist
# Default schedule daily 8am; --weekly switches to Mondays 8am
python3 scripts/watchlist.py add "british airways middle east" --weekly
# 2. Configure delivery and budget (optional)
python3 scripts/watchlist.py config delivery "https://hooks.slack.com/services/..."
python3 scripts/watchlist.py config budget 5.00
# 3. Trigger via cron / Task Scheduler / GitHub Actions
python3 scripts/watchlist.py run-one "british airways middle east"
# or run every enabled topic, gated by daily_budget
python3 scripts/watchlist.py run-all
```
The schedule field stored on each topic is metadata - the actual cron / Task Scheduler invocation is your responsibility. Watchlist runs hardcode `--quick` and `--lookback-days 90` when spawning the underlying engine.
### `briefing.py` - daily / weekly digests
[`scripts/briefing.py`](skills/last30days/scripts/briefing.py) reads the SQLite store and emits structured data the agent then synthesizes into prose. Modes: `generate` (daily), `generate --weekly`, `show [--date DATE]` (display a saved briefing). Briefs save to `~/.local/share/last30days/briefs/`.
### Recommended cadence pattern
| Step | Cadence | Command |
|---|---|---|
| Baseline | one-time per topic | `/last30days "<topic>" --days=30 --store` |
| Add to watchlist | one-time per topic | `python3 scripts/watchlist.py add "<topic>" --weekly` |
| Recurring run | daily or weekly (external scheduler) | `python3 scripts/watchlist.py run-all` |
| Digest | weekly | `python3 scripts/briefing.py generate --weekly` |
---
## Per-client patterns
The skill is built to flex around different client environments. Four patterns that compose well:
### 1. Per-client `.claude/last30days.env` (preferred when you cd into client folders)
The simplest pattern when each client has its own working directory: drop a `.claude/last30days.env` into the client folder. The skill picks it up automatically (see [API keys](#api-keys-env) for the lookup priority). Typical contents:
```bash
LAST30DAYS_MEMORY_DIR=C:\Users\<you>\Clients\acme\Research\Last30Days
SCRAPECREATORS_API_KEY=<acme-scoped-key-or-shared>
INCLUDE_SOURCES=tiktok,instagram
BSKY_HANDLE=<acme-bluesky-handle>.bsky.social
```
`cd` into the client folder, run `/last30days <topic>` as normal, no flags or wrappers. Combine with `--save-suffix=<client-slug>` per run if you also need to differentiate filenames within that folder.
### 2. Per-client save dir + suffix wrapper
For workflows where you don't `cd` into a client folder (running from anywhere, scripted batches), a tiny shell function isolates each client's research without engine changes.
PowerShell example:
```powershell
function Run-L30D-Client {
param([string]$ClientSlug, [Parameter(ValueFromRemainingArguments=$true)]$Args)
$env:LAST30DAYS_MEMORY_DIR = "C:\Users\$env:USERNAME\Clients\$ClientSlug\Research\Last30Days"
/last30days @Args --save-suffix=$ClientSlug
}
# Usage: Run-L30D-Client acme "british airways middle east"
```
Bash example:
```bash
l30d-client() {
local client=$1; shift
LAST30DAYS_MEMORY_DIR="$HOME/Clients/$client/Research/Last30Days" \
/last30days "$@" --save-suffix="$client"
}
# Usage: l30d-client acme "british airways middle east"
```
### 3. Custom category-peer subreddits
[`scripts/lib/categories.py`](skills/last30days/scripts/lib/categories.py) holds a table of `(category_id, trigger_keywords, peer_subreddits)`. If a client lives in a vertical that isn't covered (legal-tech, real-estate-tech, B2B HR SaaS), add a row. Pure data, no logic.
Section 2a of `SKILL.md` documents the merging rule the skill applies when your topic matches a category.
### 4. Pre-built `--competitors-plan` JSON
For competitor-vs-comparisons that recur, a pre-written JSON skeleton per client industry saves real time:
```json
{
"Competitor B": {
"x_handle": "competitor_b_handle",
"subreddits": ["sub1", "sub2"],
"github_user": "competitor-b-org",
"context": "Founded 2019, focused on ..."
},
"Competitor C": { ... }
}
```
Pass as `--competitors-plan @client/competitors-plan.json` (or as a string). See `SKILL.md` section "If QUERY_TYPE = COMPARISON" for the full schema.
---
## Beta channel
Experimental customizations live on a private companion repo (`mvanhorn/last30days-skill-private`) installed as `/last30days-beta`. Never ship beta-only changes to the public marketplace without a review PR against the public repo. Workflow guide: `BETA.md` in the private repo.
This is the right home for client-specific changes you don't intend to upstream - custom category rows, internal subreddit lists, per-vertical plan templates.
---
## Cross-references
- The CLI flag surface: `python3 scripts/last30days.py --help`
- The skill contract (voice, LAWs, pre-flight protocol): [`skills/last30days/SKILL.md`](skills/last30days/SKILL.md)
- Engine spec (some sections stale; SKILL.md wins on conflicts): [`SPEC.md`](SPEC.md)
- Contributor guidance: [`CONTRIBUTORS.md`](CONTRIBUTORS.md)
+14 -1
View File
@@ -12,11 +12,12 @@
**An AI agent-led search engine scored by upvotes, likes, and real money - not editors.**
This README tracks the current v3 pipeline. The runtime skill spec lives in [SKILL.md](SKILL.md), which is the source of truth for the latest command and setup behavior.
This README tracks the current v3 pipeline. The runtime skill spec lives in [skills/last30days/SKILL.md](skills/last30days/SKILL.md), which is the source of truth for the latest command and setup behavior.
**Claude Code (recommended — auto-updates via marketplace):**
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days
```
**Codex, Cursor, Copilot, Gemini CLI, or any of 50+ [Agent Skills](https://agentskills.io) hosts:**
@@ -280,6 +281,18 @@ skills/last30days/scripts/setup-keychain.sh --delete XAI_API_KEY
Items are stored under service name `last30days-<KEY>` for the current user. On non-Darwin platforms the loader is a no-op, so there is no behaviour change for Linux/Windows users.
See [CONFIGURATION.md](CONFIGURATION.md) for the full per-source key matrix, reasoning provider priority, and web-search backend priority.
## Configuration
Two things you'll likely want to know on day one:
**Where research files are saved.** `LAST30DAYS_MEMORY_DIR` defaults to `~/Documents/Last30Days/` (Windows: `C:\Users\<you>\Documents\Last30Days\`). Override by setting that env var to any path in your shell, or `--save-dir <path>` per run. Use `--save-suffix=<name>` to keep multiple variations of the same topic separate (e.g. per client). Each run produces `<slug>-raw[-suffix].md`.
**Trend monitoring across runs.** The default mode produces a fresh markdown snapshot per run. To accumulate findings over time, add `--store` to persist into a SQLite database, then use [`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) for scheduled runs (with optional Slack / webhook delivery on new findings) and [`scripts/briefing.py`](skills/last30days/scripts/briefing.py) for daily / weekly digests. The full cadence pattern is in [CONFIGURATION.md](CONFIGURATION.md#trend-monitoring-store--watchlist--briefings).
Per-client wrapper scripts, custom category-peer subreddits, and the experimental beta channel for in-progress customizations are also documented in [CONFIGURATION.md](CONFIGURATION.md).
## How it works
1. **You type a topic.** Person, company, product, technology, "X vs Y." Anything.
+12 -11
View File
@@ -142,7 +142,7 @@ The repo vendors a search-only subset of Bird's Twitter GraphQL client and shell
| Likes/reposts | Real (X API) | Real (x_search tool) |
| Replies/quotes | Real | Real |
| Author handle | Real | Real |
| Relevance score | Default 0.7 (re-ranked by score.py) | AI-assessed 0.0-1.0 |
| Relevance score | Default 0.7 (re-ranked by relevance.py) | AI-assessed 0.0-1.0 |
### Depth settings
@@ -183,13 +183,14 @@ After both searches complete:
| File | Purpose |
|---|---|
| `scripts/last30days.py` | Main orchestrator, concurrent execution |
| `scripts/lib/openai_reddit.py` | Reddit search via OpenAI Responses API |
| `scripts/lib/reddit_enrich.py` | Fetch real engagement data from Reddit JSON API |
| `scripts/lib/xai_x.py` | X search via xAI API |
| `scripts/lib/bird_x.py` | X search via bundled Bird client (free) |
| `scripts/lib/models.py` | Auto-select best available model |
| `scripts/lib/env.py` | API key loading, source detection |
| `scripts/lib/http.py` | HTTP transport with retries |
| `scripts/lib/score.py` | Relevance scoring |
| `scripts/lib/dedupe.py` | URL-based deduplication |
| `skills/last30days/scripts/last30days.py` | Main CLI entry point |
| `skills/last30days/scripts/lib/pipeline.py` | Multi-source retrieval orchestration |
| `skills/last30days/scripts/lib/reddit_public.py` | Reddit public JSON search |
| `skills/last30days/scripts/lib/reddit_enrich.py` | Fetch real engagement data from Reddit JSON API |
| `skills/last30days/scripts/lib/xai_x.py` | X search via xAI API |
| `skills/last30days/scripts/lib/bird_x.py` | X search via bundled Bird client (free) |
| `skills/last30days/scripts/lib/providers.py` | Reasoning provider and model selection |
| `skills/last30days/scripts/lib/env.py` | API key loading, source detection |
| `skills/last30days/scripts/lib/http.py` | HTTP transport with retries |
| `skills/last30days/scripts/lib/relevance.py` | Query matching and relevance scoring |
| `skills/last30days/scripts/lib/dedupe.py` | URL-based deduplication |
+3 -3
View File
@@ -1,6 +1,6 @@
# Search Quality Eval
`scripts/evaluate_search_quality.py` is an optional local evaluation step for retrieval quality. It is not part of the user-facing runtime and does not need to run in CI by default.
`skills/last30days/scripts/evaluate_search_quality.py` is an optional local evaluation step for retrieval quality. It is not part of the user-facing runtime and does not need to run in CI by default.
What it does:
@@ -18,13 +18,13 @@ What it does:
Recommended usage:
```bash
uv run python scripts/evaluate_search_quality.py
uv run python skills/last30days/scripts/evaluate_search_quality.py
```
Useful flags:
```bash
uv run python scripts/evaluate_search_quality.py \
uv run python skills/last30days/scripts/evaluate_search_quality.py \
--baseline-rev origin/main \
--candidate-rev HEAD \
--no-default-topics \
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "last30days-skill",
"version": "3.0.5",
"version": "3.2.4",
"description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"settings": [
{
+1 -2
View File
@@ -6,8 +6,7 @@
"hooks": [
{
"type": "command",
"command": "bash ${CLAUDE_PLUGIN_ROOT}/hooks/scripts/check-config.sh",
"timeout": 5
"command": "bash \"${CLAUDE_PLUGIN_ROOT:-${extensionPath:-.}}/hooks/scripts/check-config.sh\""
}
]
}
+49 -1
View File
@@ -33,8 +33,13 @@ load_env_vars() {
[[ -z "$key" ]] && continue
key=$(echo "$key" | xargs)
value=$(echo "$value" | xargs | sed 's/^["'\''"]//;s/["'\''"]$//')
# Strip inline comments (# preceded by whitespace) to prevent
# command substitution in backtick-containing comments
value="${value%%[[:space:]]#*}"
if [[ -n "$key" && -n "$value" ]]; then
eval "ENV_${key}=\"${value}\""
# printf -v writes via assignment semantics (global from inside a
# function), works on macOS's /bin/bash 3.2 — `declare -g` is 4.2+.
printf -v "ENV_${key}" '%s' "$value"
fi
done < "$file"
fi
@@ -58,14 +63,53 @@ fi
# Check SETUP_COMPLETE (from file or env)
SETUP_COMPLETE="${ENV_SETUP_COMPLETE:-${SETUP_COMPLETE:-}}"
# Compute last-run summary line (if last-run.json exists)
if [[ "${LAST30DAYS_CONFIG_DIR+x}" == "x" ]]; then
if [[ -n "$LAST30DAYS_CONFIG_DIR" ]]; then
LAST_RUN_FILE="$LAST30DAYS_CONFIG_DIR/last-run.json"
else
LAST_RUN_FILE=""
fi
else
LAST_RUN_FILE="$HOME/.config/last30days/last-run.json"
fi
LAST_RUN_LINE=""
if [[ -n "$LAST_RUN_FILE" && -f "$LAST_RUN_FILE" ]] && command -v python3 &>/dev/null; then
LAST_RUN_LINE=$(LAST_RUN_FILE="$LAST_RUN_FILE" python3 - <<'PY' 2>/dev/null || true
import datetime
import json
import os
path = os.environ["LAST_RUN_FILE"]
try:
with open(path) as fh:
d = json.load(fh)
topic = (d.get("topic") or "?")[:60]
ts = d.get("timestamp", "")
dt = datetime.datetime.fromisoformat(ts.replace("Z", "+00:00"))
delta = (datetime.datetime.now(datetime.timezone.utc) - dt).total_seconds()
if delta < 60: ago = f"{int(delta)}s ago"
elif delta < 3600: ago = f"{int(delta//60)}m ago"
elif delta < 86400: ago = f"{int(delta//3600)}h ago"
else: ago = f"{int(delta//86400)}d ago"
total = d.get("total", 0)
print(f" Last run: \"{topic}\" · {ago} · {total} results")
except Exception:
pass
PY
)
fi
# If setup has never been run, show welcome message for new users
if [[ -z "$SETUP_COMPLETE" && -z "$CONFIG_FILE" && -z "${OPENAI_API_KEY:-}" && -z "${SCRAPECREATORS_API_KEY:-}" && -z "${AUTH_TOKEN:-}" && -z "${XAI_API_KEY:-}" ]]; then
cat <<'EOF'
/last30days: Ready to use. Run /last30days to get started — setup takes 30 seconds.
Research any topic across Reddit, HN, X, YouTube, Polymarket (last 30 days).
Reddit, Hacker News, and Polymarket work out of the box.
The setup wizard can unlock X/Twitter, YouTube, and more.
EOF
[[ -n "$LAST_RUN_LINE" ]] && echo "$LAST_RUN_LINE"
exit 0
fi
@@ -116,9 +160,13 @@ fi
if [[ -n "$HAS_SCRAPECREATORS" ]]; then
# Fully configured — compact ready message
echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
echo " Research any topic across social + market + web sources (last 30 days)."
[[ -n "$LAST_RUN_LINE" ]] && echo "$LAST_RUN_LINE"
else
# Setup done but missing ScrapeCreators — recommend it
echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
echo " Research any topic across social + market + web sources (last 30 days)."
[[ -n "$LAST_RUN_LINE" ]] && echo "$LAST_RUN_LINE"
echo " Tip: Add ScrapeCreators for Reddit comments + TikTok + Instagram."
echo " 100 free credits, no credit card — scrapecreators.com"
echo " last30days has no affiliation with any API provider."
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "last30days-skill"
version = "3.2.4"
version = "3.3.0"
description = "Multi-source last-30-days research skill"
readme = "README.md"
requires-python = ">=3.12"
+47 -43
View File
@@ -1,52 +1,64 @@
## v3.3.0 — install everywhere, ship the reliability sweep
The AI world reinvents itself every month. This skill keeps you current.
`/last30days` researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.
`/last30days` researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, Digg, and 5+ more sources from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.
## v3 is the intelligent search release
## What's new in v3.3.0
v3 is a ground-up engine rewrite by [@j-sperling](https://github.com/j-sperling). The old engine searched keywords. The new engine understands your topic first, then searches the right people and communities.
### Install everywhere with one command
Type "OpenClaw" and v3 resolves @steipete, r/openclaw, r/ClaudeCode, and the right YouTube channels and TikTok hashtags before a single API call fires. Type "Peter Steinberger" and it resolves his X handle and GitHub profile, switches to person mode, and shows what he shipped this month at 85% merge rate across 22 PRs. None of that was on Google.
`npx skills add mvanhorn/last30days-skill -g -y` is now the canonical install path for **every harness** — Claude Code, OpenAI Codex CLI, Cursor, Gemini CLI, GitHub Copilot, Windsurf, and 50+ other Agent Skills hosts. The skill auto-detects each harness's skills directory and symlinks in place, so edits propagate live. No more per-harness manual paths in the README.
## Headline features
### New emit mode: `--emit=html`
### Intelligent pre-research
Shareable, print-friendly HTML briefs. Drop the file in Slack, mail it to a stakeholder, or print it for the meeting. Same data as compact mode, structured for human reading.
The killer feature. A new Python pre-research brain resolves X handles, GitHub repos, subreddits, TikTok hashtags, and YouTube channels before searching. Bidirectional: person to company, product to founder, name to GitHub profile. The right subreddits, the right handles, the right hashtags, all resolved before a single API call.
### New source: Digg
### Best Takes
Digg surfaces curated story clusters from the AI 1000 leaderboard and pulls attributable X-post quotes directly into the brief. Auto-enabled when `digg-pp-cli` is on PATH. Footer line: `⛏️ Digg: N clusters │ K posts │ M authors`. No X auth required for the inline quotes.
A second LLM judge scores every result for humor, wit, and virality alongside relevance. Every brief now ends with a Best Takes section surfacing the cleverest one-liners and most viral quotes. The Reddit and X people are funny, and the old engine buried their best stuff.
### YouTube residential-IP routing (`LAST30DAYS_YOUTUBE_SSH_HOST`)
### Cross-source cluster merging
Running on a datacenter VPS (Hetzner, DigitalOcean, AWS, etc.)? YouTube's bot-wall fingerprints datacenter IP ranges before any cookie check. Set `LAST30DAYS_YOUTUBE_SSH_HOST=<ssh-alias>` and yt-dlp runs over SSH against a residential-IP host instead. One env var, no proxy service required.
When the same story hits Reddit, X, and YouTube, v3 merges them into one cluster instead of three duplicates. Entity-based overlap detection catches matches even when the titles use different words.
### macOS Keychain credential source
### Single-pass comparisons
When env vars and config files aren't set, the engine now reads credentials from the macOS Keychain. Stores secrets where macOS expects them; nothing on disk in plaintext.
"X vs Y" used to run three serial passes (12+ minutes). v3 runs one pass with entity-aware subqueries for both sides at once. Same depth, 3 minutes.
### `EXCLUDE_SOURCES` env var
### GitHub person-mode and project-mode
The inverse of `INCLUDE_SOURCES`. Useful for "everything except TikTok" or "everything except the slow ones."
When the topic is a person, the engine switches from keyword search to author-scoped queries. PR velocity, top repos by stars, release notes for what shipped this month, woven into the narrative alongside X posts and Reddit threads.
## Reliability sweep
When the topic is a project, it pulls live star counts, READMEs, releases, and top issues from the GitHub API. No stale blog posts.
This release closes a long tail of platform-specific issues that have been accumulating:
### ELI5 mode
- **Reddit**: subreddits starting with `r` no longer get mangled by `lstrip("r/")`. Browser-like headers + gzip handling fix urllib 403s on the public JSON endpoint. HTTP 402 now triggers the OpenAI/public-JSON fallback chain when ScrapeCreators credits are exhausted.
- **xAI**: empty or malformed responses now surface in `errors_by_source` instead of silently returning zero results.
- **Windows**: process cleanup no longer crashes on `os.killpg`. POSIX-style secret-permission warnings skipped. Save-path footer uses forward slashes.
- **Auth**: comma-separated `SCRAPECREATORS_API_KEY=key1,key2` rotation restored (accidentally dropped in v3.0.6).
- **YouTube + HN**: SC YouTube + multi-token HN searches unblocked. Transcript-fetch ratio surfaced.
- **HTTP**: retry budget expanded with exponential backoff on DNS failure. Parallel AI search aligned with current API schema.
- **OpenClaw**: now works without a ScrapeCreators key. Poll-timing initialized once.
Say "eli5 on" after any research run. The synthesis rewrites in plain language. No jargon. Same data, same sources, same citations, just clearer. Say "eli5 off" to go back.
## Multi-harness reframe
### 13+ sources
`AGENTS.md` is now the canonical project doc; `CLAUDE.md` points at it. The skill is positioned as a multi-harness Agent Skills package, not a Claude-Code-specific tool. SKILL.md's path resolution rewrote `SKILL_ROOT``SKILL_DIR`, removing ~80 lines of bash and fixing a real spec-vs-engine divergence bug.
v3 adds Threads, Pinterest, Perplexity, Bluesky, and Parallel AI grounding to the existing Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and Web lineup. Perplexity Deep Research (`--deep-research`) gives you 50+ citation reports for serious investigation.
## Breaking change
### Per-author cap and entity disambiguation
Max 3 items per author prevents single-voice dominance. Synthesis trusts resolved handles over fuzzy keyword matches.
**`.codex-plugin/plugin.json` removed.** Codex native-plugin users should install via `npx skills add mvanhorn/last30days-skill` or copy the skill to `~/.codex/skills/last30days/`. The `npx skills add` path now reaches every harness uniformly.
## Install
Claude Code:
Any harness (recommended):
```
npx skills add mvanhorn/last30days-skill -g -y
```
Claude Code marketplace:
```
/plugin marketplace add mvanhorn/last30days-skill
@@ -58,29 +70,21 @@ OpenClaw:
clawhub install last30days-official
```
OpenAI Codex CLI: install the repo as a local Codex marketplace/plugin. The plugin manifest lives at `.codex-plugin/plugin.json`, and the canonical skill payload is `skills/last30days/SKILL.md`.
Zero config. Reddit, Hacker News, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds.
## v3 Community
## Contributors
v3 was shaped by community contributors whose PRs and issues inspired core features. Their code wasn't merged directly (v3 was a ground-up rewrite), but their ideas drove what shipped.
First-time contributors whose fixes shipped in v3.3.0 (most via PR triage salvage — the fix re-applied directly to main with co-author credit when path migration made the original branch un-rebaseable):
Thanks to @uppinote20, @zerone0x, @thinkun, @thomasmktong, @fanispoulinakisai-boop, @pejmanjohn, @zl190, and @hnshah. See [CONTRIBUTORS.md](CONTRIBUTORS.md) for the full list.
- Dave Morin — portable test-harness paths
- Alex Key — `removeprefix("r/")` for subreddit names
- Eric Oberhofer — multi-key rotation restored
- gujishh — Windows process cleanup
- Franco Carballar — Reddit browser-like headers
- Jonathan Oppenheim — Reddit 402 fallback chain
- Kaustav Mishra — xAI error surfacing
- [@thinkun](https://github.com/thinkun) — OpenClaw ScrapeCreators-key-optional fix
Contributors who shaped the release itself:
- @Jah-yee (#153) surfaced the need for a real Codex CLI integration, which shipped in #219
- @Cody-Coyote (#204) reported the marketplace validation bug that needed fixing before v3 could ship cleanly
- @dannyshmueli pushed for v3 and Codex family support publicly on X
Full Added / Changed / Fixed detail lives in [CHANGELOG.md](CHANGELOG.md) under `[3.0.0]`.
## Earlier contributors
From the v1 and v2 lineage:
- [@galligan](https://github.com/galligan) for marketplace plugin inspiration
- [@hutchins](https://x.com/hutchins) for pushing the YouTube feature
Plus every contributor who shipped one of the ~75 PRs merged this cycle. See [CHANGELOG.md](CHANGELOG.md) under `[3.3.0]` for the full PR list and `git log v3.2.0..v3.3.0` for the complete commit graph.
30 days of research. 30 seconds of work. Thirteen sources. Zero stale prompts.
+6 -6
View File
@@ -1,6 +1,6 @@
---
name: last30days
version: "3.2.4"
version: "3.3.0"
description: "Research what people actually say about any topic in the last 30 days. Pulls posts and engagement from Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, and the web."
argument-hint: 'last30days nvidia earnings reaction | last30days AI video tools | last30days what users want in react'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
@@ -243,7 +243,7 @@ If your Bash call to `last30days.py` does NOT include the FULL pre-flight checkl
---
# last30days v3.2.4: Research Any Topic from the Last 30 Days
# last30days v3.3.0: Research Any Topic from the Last 30 Days
> **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`). X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section.
@@ -596,8 +596,8 @@ When the user asks "X vs Y" (or "X vs Y vs Z"), the engine fans out N full `pipe
# the Read tool result. Examples:
# Read ~/.claude/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.claude/skills/last30days
# Read ~/.codex/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.codex/skills/last30days
# Read ~/.claude/plugins/cache/last30days-skill/last30days/3.2.4/skills/last30days/SKILL.md
# → SKILL_DIR=$HOME/.claude/plugins/cache/last30days-skill/last30days/3.2.4/skills/last30days
# Read ~/.claude/plugins/cache/last30days-skill/last30days/3.3.0/skills/last30days/SKILL.md
# → SKILL_DIR=$HOME/.claude/plugins/cache/last30days-skill/last30days/3.3.0/skills/last30days
# scripts/last30days.py is always a direct child of SKILL_DIR (every install layout
# packages SKILL.md and scripts/ as siblings).
SKILL_DIR="<absolute path of the directory containing the SKILL.md you Read>"
@@ -914,8 +914,8 @@ Store your plan as `QUERY_PLAN_JSON` - you'll pass it to the script in the next
# the Read tool result. Examples:
# Read ~/.claude/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.claude/skills/last30days
# Read ~/.codex/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.codex/skills/last30days
# Read ~/.claude/plugins/cache/last30days-skill/last30days/3.2.4/skills/last30days/SKILL.md
# → SKILL_DIR=$HOME/.claude/plugins/cache/last30days-skill/last30days/3.2.4/skills/last30days
# Read ~/.claude/plugins/cache/last30days-skill/last30days/3.3.0/skills/last30days/SKILL.md
# → SKILL_DIR=$HOME/.claude/plugins/cache/last30days-skill/last30days/3.3.0/skills/last30days
# scripts/last30days.py is always a direct child of SKILL_DIR (every install layout
# packages SKILL.md and scripts/ as siblings).
SKILL_DIR="<absolute path of the directory containing the SKILL.md you Read>"
+83 -7
View File
@@ -6,6 +6,7 @@ from __future__ import annotations
import argparse
import atexit
import datetime
import json
import os
import re
@@ -62,7 +63,10 @@ def _cleanup_children() -> None:
pids = list(_child_pids)
for pid in pids:
try:
os.killpg(os.getpgid(pid), signal.SIGTERM)
if hasattr(os, "killpg"):
os.killpg(os.getpgid(pid), signal.SIGTERM)
else:
os.kill(pid, signal.SIGTERM)
except (ProcessLookupError, PermissionError, OSError):
continue
@@ -195,9 +199,9 @@ def compute_save_path_display(save_dir: str, topic: str, suffix: str, emit: str)
try:
home = _Path.home().resolve()
relative = raw.relative_to(home)
return f"~/{relative}"
return f"~/{relative.as_posix()}"
except ValueError:
return str(raw)
return raw.as_posix()
def read_synthesis_file(path: str) -> str:
@@ -387,7 +391,7 @@ def subrun_kwargs_for(
subreddits = _choose("subreddits", "subreddits")
if isinstance(subreddits, list):
subreddits = [s.strip().lstrip("r/") for s in subreddits if s.strip()] or None
subreddits = [s.strip().removeprefix("r/") for s in subreddits if s.strip()] or None
x_related = plan_entry.get("x_related")
if isinstance(x_related, list):
@@ -531,6 +535,24 @@ def _show_runtime_ui(
progress.show_promo(promo, diag=diag)
def _write_last_run(topic: str, report: "schema.Report") -> None:
try:
if env.CONFIG_DIR is None:
return
target = env.CONFIG_DIR
target.mkdir(parents=True, exist_ok=True)
counts = {source: len(items) for source, items in report.items_by_source.items()}
payload = {
"topic": topic,
"timestamp": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"sources": counts,
"total": sum(counts.values()),
}
(target / "last-run.json").write_text(json.dumps(payload, indent=2))
except Exception:
pass
def main() -> int:
parser = build_parser()
# Use parse_known_args so setup sub-flags (--device-auth, --github,
@@ -606,7 +628,7 @@ def main() -> int:
depth = "deep" if args.deep else "quick" if args.quick else "default"
try:
x_related = [h.strip() for h in args.x_related.split(",") if h.strip()] if args.x_related else None
subreddits = [s.strip().lstrip("r/") for s in args.subreddits.split(",") if s.strip()] if args.subreddits else None
subreddits = [s.strip().removeprefix("r/") for s in args.subreddits.split(",") if s.strip()] if args.subreddits else None
tiktok_hashtags = [h.strip().lstrip("#") for h in args.tiktok_hashtags.split(",") if h.strip()] if args.tiktok_hashtags else None
tiktok_creators = [c.strip().lstrip("@") for c in args.tiktok_creators.split(",") if c.strip()] if args.tiktok_creators else None
ig_creators = [c.strip().lstrip("@") for c in args.ig_creators.split(",") if c.strip()] if args.ig_creators else None
@@ -625,6 +647,7 @@ def main() -> int:
# Auto-resolve: use web search to discover subreddits/handles before planning.
# This is the engine-side equivalent of SKILL.md Steps 0.55/0.75 for platforms
# without WebSearch (OpenClaw, Codex, raw CLI).
repos_from_auto_resolve = False
if args.auto_resolve and not external_plan:
from lib import resolve
resolution = resolve.auto_resolve(topic, config)
@@ -639,6 +662,9 @@ def main() -> int:
sys.stderr.write(f"[AutoResolve] GitHub user: @{args.github_user}\n")
if resolution.get("github_repos") and not args.github_repo:
args.github_repo = ",".join(resolution["github_repos"])
# auto_resolve already canonicalized via canonicalize_github_repos(cap=5);
# mark so we don't re-canonicalize below and clobber its relevance order.
repos_from_auto_resolve = True
sys.stderr.write(f"[AutoResolve] GitHub repos: {args.github_repo}\n")
if resolution.get("context"):
# Inject context into external_plan metadata for the planner to use
@@ -651,6 +677,20 @@ def main() -> int:
github_user = args.github_user.lstrip("@").lower() if args.github_user else None
github_repos = [r.strip() for r in args.github_repo.split(",") if r.strip() and "/" in r.strip()] if args.github_repo else None
# Only canonicalize when repos came from a user-supplied --github-repo flag.
# When repos_from_auto_resolve is True, auto_resolve already ran
# canonicalize_github_repos(cap=5) and ranked by relevance; re-running here
# with cap=None can re-sort by topic-slug match and lose that ordering.
if github_repos and not repos_from_auto_resolve:
from lib import resolve as resolve_lib
original_github_repos = github_repos[:]
github_repos = resolve_lib.canonicalize_github_repos(topic, github_repos, cap=None)
if github_repos != original_github_repos:
sys.stderr.write(
"[GitHub] Canonicalized repos: "
f"{','.join(original_github_repos)} -> {','.join(github_repos)}\n"
)
# --deep-research: auto-enable perplexity source and set deep flag
if args.deep_research:
if not config.get("OPENROUTER_API_KEY"):
@@ -870,7 +910,18 @@ def main() -> int:
report, progress, diag,
suppress_web_promo=bool(external_plan or comp_plan),
)
if args.store:
_write_last_run(topic, report)
# LAST30DAYS_STORE env var = persistence default-on. Read both os.environ
# (for shell-exported users) and config (for users who set it in
# ~/.config/last30days/.env, which env.py loads but does not propagate
# to os.environ). Mirrors the LAST30DAYS_DEBUG / LAST30DAYS_SKIP_PREFLIGHT
# convention; env-var or config wins, with `--store` flag still working.
_store_env = (
os.environ.get("LAST30DAYS_STORE")
or config.get("LAST30DAYS_STORE")
or ""
).lower()
if args.store or _store_env in ("1", "true", "yes"):
counts = persist_report(report)
sys.stderr.write(
f"[last30days] Stored {counts['new']} new, {counts['updated']} updated findings\n"
@@ -880,7 +931,32 @@ def main() -> int:
# Show quality nudge if applicable
try:
from lib import quality_nudge
quality = quality_nudge.compute_quality_score(config, {})
# Populate transcript-fetch ratio so quality_nudge can detect the
# degraded-YouTube failure mode (videos returned but transcripts
# silently failed - typically a stale yt-dlp binary).
youtube_items = report.items_by_source.get("youtube") or []
instagram_items = report.items_by_source.get("instagram") or []
research_results = {
"youtube_videos_count": len(youtube_items),
"youtube_transcripts_count": sum(
1 for it in youtube_items
if (it.metadata.get("transcript_highlights") or it.metadata.get("transcript_snippet"))
),
"youtube_error": report.errors_by_source.get("youtube"),
"x_error": report.errors_by_source.get("x"),
# Captions-disabled videos can never produce a transcript regardless
# of yt-dlp version; subtract them from the degraded-ratio
# denominator so a single uploader-disabled video does not trip the
# "stale yt-dlp" nudge.
"youtube_captions_disabled_count": sum(
1 for it in youtube_items if it.metadata.get("captions_disabled")
),
# Track Instagram returned-zero-items so quality_nudge can detect
# the silent-failure case (SC configured but the v2 reels endpoint
# 500'd through both the original query and the hashtag retry).
"instagram_items_count": len(instagram_items),
}
quality = quality_nudge.compute_quality_score(config, research_results)
if quality.get("nudge_text"):
sys.stderr.write(f"\n{quality['nudge_text']}\n")
sys.stderr.flush()
+84 -4
View File
@@ -1,10 +1,19 @@
"""Bluesky search via AT Protocol (requires app password).
Uses bsky.social for auth and public.api.bsky.app for post search.
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars.
Uses bsky.social for auth and api.bsky.app for post search (the canonical
authenticated AppView). The previous default `public.api.bsky.app` is the
unauthenticated public mirror, which BunnyCDN now blocks for searchPosts
regardless of auth header (verified 2026-05-04). Override the search host
via BSKY_SEARCH_HOST env var if Bluesky migrates infrastructure again.
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars. App passwords are
19-char xxxx-xxxx-xxxx-xxxx; generate at bsky.app/settings/app-passwords.
The createSession endpoint accepts main-account passwords too, but they're
bad hygiene (no scope, can't revoke individually).
"""
import math
import os
import re
import sys
import time
@@ -14,7 +23,64 @@ from typing import Any, Dict, List, Optional
from . import http, log
BSKY_SESSION_URL = "https://bsky.social/xrpc/com.atproto.server.createSession"
BSKY_SEARCH_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts"
_DEFAULT_BSKY_SEARCH_HOST = "api.bsky.app"
def _resolve_search_url(config: Optional[Dict[str, Any]] = None) -> str:
"""Resolve the Bluesky search URL with BSKY_SEARCH_HOST override.
Default is api.bsky.app. Override via BSKY_SEARCH_HOST in shell env or
.env file. The project's env.py loads .env into config but not into
os.environ, so check both — same hybrid pattern as last30days.py for
LAST30DAYS_STORE.
Hardens user-supplied host values against three common mis-configurations:
whitespace (e.g. " api.bsky.app "), embedded path components (e.g.
"api.bsky.app/xrpc/proxy") that would double the /xrpc/ segment, and
embedded scheme prefixes (e.g. "https://api.bsky.app"). On any of these
we log a warning and fall back to the default rather than building an
invalid URL with an opaque downstream error.
"""
config = config or {}
raw = (
os.environ.get("BSKY_SEARCH_HOST")
or config.get("BSKY_SEARCH_HOST")
or _DEFAULT_BSKY_SEARCH_HOST
)
host = raw.strip().rstrip("/")
# Strip embedded scheme so users who paste full URLs do not break the f-string.
for prefix in ("https://", "http://"):
if host.lower().startswith(prefix):
host = host[len(prefix):]
break
if not host or "/" in host or " " in host:
# Embedded path or whitespace remains — don't trust it. Default + log.
if raw != _DEFAULT_BSKY_SEARCH_HOST:
_log(
f"BSKY_SEARCH_HOST={raw!r} is not a bare hostname; "
f"falling back to default {_DEFAULT_BSKY_SEARCH_HOST!r}"
)
host = _DEFAULT_BSKY_SEARCH_HOST
return f"https://{host}/xrpc/app.bsky.feed.searchPosts"
# App-password format: xxxx-xxxx-xxxx-xxxx (19 chars, lowercase alphanumeric
# with three hyphens at fixed positions).
_APP_PASSWORD_RE = re.compile(r"^[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}$")
def _validate_app_password_format(value) -> bool:
"""Return True if value matches Bluesky's 19-char app-password format.
False for non-strings (None, int, list) so callers passing config dict
values directly don't crash. Detect-but-not-gate: the createSession
endpoint also accepts main-account passwords, so failing this check is
a hygiene smell, not a hard error.
"""
if not isinstance(value, str):
return False
return bool(_APP_PASSWORD_RE.fullmatch(value))
DEPTH_CONFIG = {
"quick": 15,
@@ -144,6 +210,20 @@ def search_bluesky(
if not handle or not app_password:
return {"posts": [], "error": "Bluesky credentials not configured"}
# One-shot hygiene warning if BSKY_APP_PASSWORD is not in app-password
# form. createSession accepts main-account passwords too — but main
# passwords have no scope (full account access), can't be revoked
# individually, and rotating them breaks every service that holds them.
# We warn but do not gate, matching the project's detect-don't-block
# philosophy elsewhere.
if not _validate_app_password_format(app_password):
_log(
"BSKY_APP_PASSWORD does not look like an app password "
"(expected xxxx-xxxx-xxxx-xxxx, 19 chars). It may be a main "
"account password — those work but are bad hygiene. Generate "
"an app password at https://bsky.app/settings/app-passwords"
)
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
@@ -155,7 +235,7 @@ def search_bluesky(
"limit": str(min(count, 100)),
"sort": "top",
}
url = f"{BSKY_SEARCH_URL}?{urlencode(params)}"
url = f"{_resolve_search_url(config)}?{urlencode(params)}"
def _auth_and_search() -> tuple[Optional[Dict[str, Any]], Optional[str]]:
token = _create_session(handle, app_password)
+90 -33
View File
@@ -1,9 +1,12 @@
"""Chrome cookie extraction for macOS.
"""Chrome and Brave cookie extraction for macOS.
Extracts cookies from Chrome's encrypted SQLite database using only stdlib
modules and the system openssl CLI (ships with macOS). Zero pip dependencies.
Extracts cookies from Chromium-based browser SQLite databases using only
stdlib modules and the system openssl CLI (ships with macOS). Zero pip
dependencies.
Chrome on macOS uses v10 encryption (AES-128-CBC with Keychain-stored key).
Chromium on macOS uses v10 encryption (AES-128-CBC with Keychain-stored key).
Chrome and Brave share the same algorithm; only the DB path and Keychain
service name differ.
This is NOT affected by Windows App-Bound Encryption (v20).
"""
@@ -18,10 +21,11 @@ from typing import Optional
logger = logging.getLogger(__name__)
# Chrome cookie DB location on macOS
# Cookie DB locations on macOS
CHROME_COOKIES_DB = Path.home() / "Library" / "Application Support" / "Google" / "Chrome" / "Default" / "Cookies"
BRAVE_BASE_DIR = Path.home() / "Library" / "Application Support" / "BraveSoftware" / "Brave-Browser"
# Chrome v10 encryption constants
# Chromium v10 encryption constants (shared by Chrome and Brave)
CHROME_SALT = b"saltysalt"
CHROME_PBKDF2_ITERATIONS = 1003
CHROME_KEY_LENGTH = 16
@@ -29,8 +33,8 @@ CHROME_KEY_LENGTH = 16
CHROME_IV_HEX = "20" * 16
def _get_chrome_encryption_key() -> Optional[bytes]:
"""Retrieve Chrome's encryption passphrase from macOS Keychain.
def _get_chromium_encryption_key(service_name: str) -> Optional[bytes]:
"""Retrieve the encryption passphrase for a Chromium-based browser from macOS Keychain.
Calls `security find-generic-password` which may trigger a system dialog
on first access.
@@ -39,30 +43,34 @@ def _get_chrome_encryption_key() -> Optional[bytes]:
"""
try:
result = subprocess.run(
["security", "find-generic-password", "-w", "-s", "Chrome Safe Storage"],
["security", "find-generic-password", "-w", "-s", service_name],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode != 0:
logger.info("Chrome Keychain access denied or Chrome not installed: %s", result.stderr.strip())
logger.info("%s Keychain access denied or browser not installed: %s", service_name, result.stderr.strip())
return None
passphrase = result.stdout.strip()
if not passphrase:
logger.info("Chrome Keychain returned empty passphrase")
logger.info("%s Keychain returned empty passphrase", service_name)
return None
return passphrase.encode("utf-8")
except FileNotFoundError:
logger.info("'security' command not found — not on macOS?")
return None
except subprocess.TimeoutExpired:
logger.info("Chrome Keychain access timed out")
logger.info("%s Keychain access timed out", service_name)
return None
except Exception as e:
logger.info("Failed to get Chrome encryption key: %s", e)
logger.info("Failed to get %s encryption key: %s", service_name, e)
return None
def _get_chrome_encryption_key() -> Optional[bytes]:
return _get_chromium_encryption_key("Chrome Safe Storage")
def _derive_aes_key(passphrase: bytes) -> bytes:
"""Derive 16-byte AES key from Chrome's Keychain passphrase via PBKDF2."""
return hashlib.pbkdf2_hmac(
@@ -165,36 +173,42 @@ def _get_db_version(cursor: sqlite3.Cursor) -> int:
return 0
def extract_chrome_cookies_macos(domain: str, cookie_names: list[str]) -> Optional[dict[str, str]]:
"""Extract cookies from Chrome on macOS.
def _extract_chromium_cookies_macos(
db_path: Path,
keychain_service: str,
domain: str,
cookie_names: list[str],
) -> Optional[dict[str, str]]:
"""Extract cookies from any Chromium-based browser on macOS.
Copies the locked Cookies database to a temp file, reads specified cookies,
and decrypts v10-encrypted values using the Keychain-stored key.
Args:
domain: Cookie domain to match (e.g., ".twitter.com", ".x.com")
cookie_names: List of cookie names to extract
db_path: Path to the browser's Cookies SQLite file.
keychain_service: macOS Keychain service name (e.g. "Chrome Safe Storage").
domain: Cookie domain to match (e.g., ".twitter.com", ".x.com").
cookie_names: List of cookie names to extract.
Returns:
Dict mapping cookie name to decrypted value, or None on failure.
Only includes cookies that were successfully found and decrypted.
"""
if not CHROME_COOKIES_DB.exists():
logger.info("Chrome cookies database not found at %s", CHROME_COOKIES_DB)
if not db_path.exists():
logger.info("%s cookies database not found at %s", keychain_service, db_path)
return None
# Get encryption key from Keychain
passphrase = _get_chrome_encryption_key()
passphrase = _get_chromium_encryption_key(keychain_service)
aes_key = _derive_aes_key(passphrase) if passphrase else None
# Copy DB to temp file (Chrome locks the original)
# Copy DB to temp file (browser locks the original while running)
tmp_fd = None
tmp_path = None
try:
tmp_fd, tmp_path = tempfile.mkstemp(suffix=".sqlite")
shutil.copy2(str(CHROME_COOKIES_DB), tmp_path)
shutil.copy2(str(db_path), tmp_path)
except Exception as e:
logger.info("Failed to copy Chrome cookies database: %s", e)
logger.info("Failed to copy %s cookies database: %s", keychain_service, e)
if tmp_path:
try:
Path(tmp_path).unlink(missing_ok=True)
@@ -211,26 +225,22 @@ def extract_chrome_cookies_macos(domain: str, cookie_names: list[str]) -> Option
cursor = conn.cursor()
db_version = _get_db_version(cursor)
logger.debug("Chrome cookie DB version: %d", db_version)
logger.debug("%s cookie DB version: %d", keychain_service, db_version)
# Build query with placeholders for cookie names
placeholders = ",".join("?" for _ in cookie_names)
query = (
f"SELECT name, value, encrypted_value FROM cookies "
f"WHERE host_key LIKE ? AND name IN ({placeholders})"
)
# Use LIKE for domain matching (e.g., %.twitter.com matches .twitter.com)
params = [f"%{domain}"] + list(cookie_names)
cursor.execute(query, params)
results: dict[str, str] = {}
for name, value, encrypted_value in cursor.fetchall():
# Prefer unencrypted value if present
if value:
results[name] = value
continue
# Handle encrypted value
if encrypted_value and encrypted_value[:3] == b"v10":
if aes_key is None:
logger.debug("Skipping encrypted cookie %s — no Keychain access", name)
@@ -241,25 +251,72 @@ def extract_chrome_cookies_macos(domain: str, cookie_names: list[str]) -> Option
else:
logger.debug("Failed to decrypt cookie %s", name)
elif encrypted_value:
# Unknown encryption version
logger.debug("Unknown encryption for cookie %s (prefix: %r)", name, encrypted_value[:3])
conn.close()
if not results:
logger.info("No matching cookies found in Chrome for domain %s", domain)
logger.info("No matching cookies found in %s for domain %s", keychain_service, domain)
return None
return results
except sqlite3.Error as e:
logger.info("Failed to read Chrome cookies database: %s", e)
logger.info("Failed to read %s cookies database: %s", keychain_service, e)
return None
except Exception as e:
logger.info("Unexpected error reading Chrome cookies: %s", e)
logger.info("Unexpected error reading %s cookies: %s", keychain_service, e)
return None
finally:
try:
Path(tmp_path).unlink(missing_ok=True)
except Exception:
pass
def extract_chrome_cookies_macos(domain: str, cookie_names: list[str]) -> Optional[dict[str, str]]:
"""Extract cookies from Chrome on macOS."""
return _extract_chromium_cookies_macos(
CHROME_COOKIES_DB, "Chrome Safe Storage", domain, cookie_names
)
def _find_brave_cookies_db() -> Optional[Path]:
"""Find Brave's Cookies database on macOS.
Tries the Default profile first, then scans numbered Profile directories
by most-recently-modified. Brave creates extra profiles as "Profile 1",
"Profile 2", etc. alongside Default; the most recently used one is the
likeliest to hold current cookies. Lexicographic sort would visit
"Profile 10" before "Profile 2", which can return the wrong profile.
"""
default = BRAVE_BASE_DIR / "Default" / "Cookies"
if default.exists():
return default
try:
candidates = [
child for child in BRAVE_BASE_DIR.iterdir()
if child.is_dir() and child.name.startswith("Profile ")
]
for child in sorted(candidates, key=lambda p: p.stat().st_mtime, reverse=True):
candidate = child / "Cookies"
if candidate.exists():
return candidate
except OSError:
pass
return None
def extract_brave_cookies_macos(domain: str, cookie_names: list[str]) -> Optional[dict[str, str]]:
"""Extract cookies from Brave on macOS.
Brave uses the same v10 AES-128-CBC encryption as Chrome; only the DB
path and Keychain service name differ.
"""
db_path = _find_brave_cookies_db()
if db_path is None:
logger.info("Brave cookies database not found under %s", BRAVE_BASE_DIR)
return None
return _extract_chromium_cookies_macos(db_path, "Brave Safe Storage", domain, cookie_names)
@@ -1,6 +1,6 @@
"""Browser cookie extraction for last30days.
Extracts cookies from local browser databases (Firefox, Chrome, Safari)
Extracts cookies from local browser databases (Firefox, Chrome, Brave, Safari)
to enable zero-config authentication for services like X/Twitter.
Only uses Python stdlib no external dependencies.
@@ -255,6 +255,29 @@ def extract_chrome_cookies(
return None
def extract_brave_cookies(
domain: str, cookie_names: List[str]
) -> Optional[Dict[str, str]]:
"""Extract cookies from Brave for the given domain and cookie names.
macOS only Brave uses the same v10 AES-128-CBC encryption as Chrome,
with a different DB path and Keychain service name ("Brave Safe Storage").
Tries the Default profile first, then scans numbered Profile directories.
Returns:
Dict of {cookie_name: cookie_value} or None if extraction fails.
"""
if platform.system() != "Darwin":
logger.debug("Brave cookie extraction only supported on macOS")
return None
try:
from .chrome_cookies import extract_brave_cookies_macos
return extract_brave_cookies_macos(domain, cookie_names)
except Exception as exc:
logger.debug("Brave cookie extraction failed: %s", exc)
return None
def extract_safari_cookies(
domain: str, cookie_names: List[str]
) -> Optional[Dict[str, str]]:
@@ -282,9 +305,9 @@ def extract_cookies(
"""Extract cookies from the specified browser.
Args:
browser: One of 'firefox', 'chrome', 'safari', or 'auto'.
browser: One of 'firefox', 'chrome', 'brave', 'safari', or 'auto'.
'auto' tries browsers in platform-appropriate order:
- macOS: Chrome -> Firefox -> Safari
- macOS: Chrome -> Brave -> Firefox -> Safari
- Linux: Firefox only
domain: The cookie domain to match (e.g. ".x.com").
cookie_names: List of cookie names to extract.
@@ -333,7 +356,7 @@ def extract_cookies_with_source(
so callers can track the source.
Args:
browser: One of 'firefox', 'chrome', 'safari', or 'auto'.
browser: One of 'firefox', 'chrome', 'brave', 'safari', or 'auto'.
domain: The cookie domain to match (e.g. ".x.com").
cookie_names: List of cookie names to extract.
@@ -344,6 +367,7 @@ def extract_cookies_with_source(
extractors = {
"firefox": extract_firefox_cookies,
"chrome": extract_chrome_cookies,
"brave": extract_brave_cookies,
"safari": extract_safari_cookies,
}
@@ -360,7 +384,7 @@ def extract_cookies_with_source(
# Auto mode: try browsers in platform-appropriate order
system = platform.system()
if system == "Darwin":
order = ["chrome", "firefox", "safari"]
order = ["chrome", "brave", "firefox", "safari"]
elif system == "Linux":
order = ["firefox"]
else:
@@ -106,7 +106,7 @@ def _extract_subreddits(reddit_items: List[Dict[str, Any]]) -> List[str]:
for item in reddit_items:
# Primary subreddit
sub = item.get("subreddit", "").strip().lstrip("r/")
sub = item.get("subreddit", "").strip().removeprefix("r/")
if sub:
sub_counts[sub] += 1
+22
View File
@@ -314,6 +314,7 @@ def get_config() -> dict[str, Any]:
('LAST30DAYS_RERANK_MODEL', None),
('LAST30DAYS_X_MODEL', None),
('LAST30DAYS_X_BACKEND', None),
('LAST30DAYS_STORE', None),
('OPENAI_MODEL_PIN', None),
('XAI_MODEL_PIN', None),
('SCRAPECREATORS_API_KEY', None),
@@ -322,6 +323,7 @@ def get_config() -> dict[str, Any]:
('CT0', None),
('BSKY_HANDLE', None),
('BSKY_APP_PASSWORD', None),
('BSKY_SEARCH_HOST', None),
('TRUTHSOCIAL_TOKEN', None),
('BRAVE_API_KEY', None),
('EXA_API_KEY', None),
@@ -334,11 +336,31 @@ def get_config() -> dict[str, Any]:
('INCLUDE_SOURCES', ''),
('EXCLUDE_SOURCES', ''),
('LAST30DAYS_YOUTUBE_SSH_HOST', None),
('LAST30DAYS_TRANSCRIPT_TIMEOUT', None),
]
for key, default in keys:
config[key] = os.environ.get(key) or merged_env.get(key, default)
# Backward-compat: ScrapeCreators' own examples and tutorials use the
# SCRAPE_CREATORS_API_KEY spelling (with underscore between SCRAPE and
# CREATORS). Accept that form too so users who follow the vendor's docs
# don't silently end up with has_scrapecreators=False. Canonical name
# wins when both are set.
if not config.get('SCRAPECREATORS_API_KEY'):
legacy = os.environ.get('SCRAPE_CREATORS_API_KEY') or merged_env.get('SCRAPE_CREATORS_API_KEY')
if legacy:
config['SCRAPECREATORS_API_KEY'] = legacy
# Multi-key rotation: comma-separated SCRAPECREATORS_API_KEY round-robins
# via random.choice per run. Originally added in #268, accidentally dropped
# in v3.0.6, restored here.
sc_key_raw = config.get('SCRAPECREATORS_API_KEY') or ''
if ',' in sc_key_raw:
import random
sc_keys = [k.strip() for k in sc_key_raw.split(',') if k.strip()]
config['SCRAPECREATORS_API_KEY'] = random.choice(sc_keys) if sc_keys else ''
# Track which config source was used (highest-priority file source wins
# the label; keychain is only reported when nothing else is configured).
if project_env_path:
+6 -3
View File
@@ -140,7 +140,10 @@ def parallel_search(
data = http.request(
"POST", "https://api.parallel.ai/v1/search",
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
json_data={"query": query, "max_results": count},
json_data={
"search_queries": [query],
"advanced_settings": {"max_results": count},
},
timeout=15,
)
items = []
@@ -150,7 +153,7 @@ def parallel_search(
url = r.get("url", "")
if not url:
continue
raw_date = r.get("published_date") or ""
raw_date = r.get("publish_date") or ""
pub_date = _normalize_date(raw_date[:10]) if raw_date else None
if not _in_date_range(pub_date, date_range):
continue
@@ -159,7 +162,7 @@ def parallel_search(
"title": r.get("title", ""),
"url": url,
"source_domain": _domain(url),
"snippet": r.get("snippet", ""),
"snippet": ((r.get("excerpts") or [""])[0] or "")[:500],
"date": pub_date,
"relevance": 0.8,
"why_relevant": "Parallel AI web search",
+81 -4
View File
@@ -7,12 +7,14 @@ Requires SCRAPECREATORS_API_KEY in config. 100 free API calls, then PAYG.
API docs: https://scrapecreators.com/docs
"""
import os
import re
import sys
from datetime import datetime
from typing import Any, Dict, List, Optional, Set
from . import dates, http, log
from .relevance import token_overlap_relevance as _compute_relevance
SCRAPECREATORS_BASE = "https://api.scrapecreators.com"
@@ -26,7 +28,42 @@ DEPTH_CONFIG = {
# Max words to keep from each caption
CAPTION_MAX_WORDS = 500
from .relevance import token_overlap_relevance as _compute_relevance
# Default transcript fetch timeout (seconds). SC's
# /v2/instagram/media/transcript regularly takes >15s on real workloads,
# so the default is generous; override via LAST30DAYS_TRANSCRIPT_TIMEOUT.
DEFAULT_TRANSCRIPT_TIMEOUT = 30
def _resolve_transcript_timeout(
timeout: Optional[float] = None,
config: Optional[Dict[str, Any]] = None,
) -> float:
"""Resolve the IG transcript-fetch timeout.
Priority (highest wins):
1. Explicit ``timeout`` kwarg
2. ``LAST30DAYS_TRANSCRIPT_TIMEOUT`` in os.environ
3. ``LAST30DAYS_TRANSCRIPT_TIMEOUT`` in caller-supplied config dict
4. ``DEFAULT_TRANSCRIPT_TIMEOUT`` (30s)
Mirrors the ``os.environ.get(X) or config.get(X)`` pattern used for
LAST30DAYS_STORE in last30days.py so the env var works whether it's
shell-exported or set in ~/.config/last30days/.env.
"""
if timeout is not None:
try:
return float(timeout)
except (TypeError, ValueError):
pass
raw = os.environ.get("LAST30DAYS_TRANSCRIPT_TIMEOUT")
if not raw and config:
raw = config.get("LAST30DAYS_TRANSCRIPT_TIMEOUT")
if raw:
try:
return float(raw)
except (TypeError, ValueError):
pass
return float(DEFAULT_TRANSCRIPT_TIMEOUT)
def _extract_core_subject(topic: str) -> str:
@@ -44,6 +81,17 @@ def _extract_core_subject(topic: str) -> str:
return extract_core_subject(topic, noise=_INSTAGRAM_NOISE)
def _to_hashtag_form(query: str) -> str:
"""Collapse a multi-word query to hashtag form (no spaces, lowercase).
SC's /v2/instagram/reels/search wraps Google Search and is documented
to be flaky on multi-token queries. Single-token queries map to a
hashtag page lookup which is the stable path. Used as a 500-retry
fallback before the request bubbles up as a silent failure.
"""
return ''.join(query.split()).lower()
def _infer_query_intent(topic: str) -> str:
"""Tiny local intent classifier for Instagram query expansion."""
text = topic.lower().strip()
@@ -283,6 +331,26 @@ def search_instagram(
timeout=30,
retries=2,
)
except http.HTTPError as e:
# SC's v2 reels search wraps Google Search and 500s frequently on
# multi-token queries. Single tokens hit the stable hashtag-page
# path. Retry once with hashtag form before bubbling up.
if getattr(e, "status_code", None) == 500 and ' ' in core_topic:
_log(f"IG search 500 on '{core_topic}', retrying with hashtag form")
try:
data = http.get(
f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search",
params={"query": _to_hashtag_form(core_topic)},
headers=http.scrapecreators_headers(token),
timeout=30,
retries=2,
)
except Exception as retry_e:
_log(f"IG search retry failed: {retry_e}")
return {"items": [], "error": f"{type(retry_e).__name__}: {retry_e}"}
else:
_log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
except Exception as e:
_log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
@@ -317,6 +385,8 @@ def fetch_captions(
video_items: List[Dict[str, Any]],
token: str,
depth: str = "default",
timeout: Optional[float] = None,
config: Optional[Dict[str, Any]] = None,
) -> Dict[str, str]:
"""Fetch transcripts for top N Instagram reels via ScrapeCreators.
@@ -328,12 +398,19 @@ def fetch_captions(
video_items: Items from search_instagram()
token: ScrapeCreators API key
depth: Depth level for caption limit
timeout: Optional per-request transcript timeout in seconds. When
None, resolves from LAST30DAYS_TRANSCRIPT_TIMEOUT (env or
config), defaulting to DEFAULT_TRANSCRIPT_TIMEOUT (30s).
config: Optional config dict (from env.get_config()) used as a
fallback source for LAST30DAYS_TRANSCRIPT_TIMEOUT when the
value is not exported in os.environ.
Returns:
Dict mapping video_id -> caption text (truncated to 500 words)
"""
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
max_captions = config["max_captions"]
depth_cfg = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
max_captions = depth_cfg["max_captions"]
transcript_timeout = _resolve_transcript_timeout(timeout, config)
if not video_items or not token:
return {}
@@ -364,7 +441,7 @@ def fetch_captions(
f"{SCRAPECREATORS_BASE}/v2/instagram/media/transcript",
params={"url": url},
headers=http.scrapecreators_headers(token),
timeout=15,
timeout=transcript_timeout,
retries=1,
)
transcripts = data.get("transcripts") or []
@@ -251,6 +251,11 @@ def _normalize_youtube(
metadata: dict[str, Any] = {}
if highlights:
metadata["transcript_highlights"] = highlights
if item.get("captions_disabled"):
# Surfaced for quality_nudge: uploader disabled captions, so this
# video should be subtracted from the degraded-transcript-ratio
# denominator (it was never going to produce a transcript).
metadata["captions_disabled"] = True
metadata["top_comments"] = _remap_comments(
item.get("top_comments") or [],
score_keys=("score", "likes"),
+6 -2
View File
@@ -79,6 +79,8 @@ MOCK_AVAILABLE_SOURCES = [
"xiaohongshu",
"github",
"perplexity",
"threads",
"pinterest",
"xquik",
"digg",
]
@@ -118,7 +120,9 @@ def available_sources(config: dict[str, Any], requested_sources: list[str] | Non
available.append("grounding")
# Perplexity Sonar: opt-in additive source via INCLUDE_SOURCES=perplexity
include_sources = (config.get("INCLUDE_SOURCES") or "").lower().split(",")
if config.get("OPENROUTER_API_KEY") and "perplexity" in include_sources:
if config.get("OPENROUTER_API_KEY") and (
"perplexity" in include_sources or (requested_sources and "perplexity" in requested_sources)
):
available.append("perplexity")
if requested_sources and "xiaohongshu" in requested_sources and env.is_xiaohongshu_available(config):
available.append("xiaohongshu")
@@ -203,7 +207,7 @@ def run(
available = [source for source in available if source in requested_sources]
if web_backend == "none":
available = [s for s in available if s != "grounding"]
elif web_backend in ("brave", "exa", "serper") and "grounding" not in available:
elif web_backend in ("brave", "exa", "serper", "parallel") and "grounding" not in available:
available.append("grounding")
if not available:
raise RuntimeError("No sources are available for this run.")
+9 -8
View File
@@ -19,14 +19,14 @@ ALLOWED_INTENTS = {
}
ALLOWED_CLUSTER_MODES = {"none", "story", "workflow", "market", "debate"}
QUICK_SOURCE_PRIORITY = {
"factual": ["hackernews", "reddit", "x", "youtube"],
"product": ["youtube", "reddit", "x", "tiktok"],
"concept": ["hackernews", "reddit", "x", "youtube"],
"opinion": ["reddit", "x", "youtube", "hackernews"],
"how_to": ["youtube", "reddit", "x", "hackernews"],
"comparison": ["reddit", "x", "hackernews", "youtube"],
"breaking_news": ["x", "reddit", "hackernews", "youtube", "polymarket"],
"prediction": ["polymarket", "x", "hackernews", "reddit", "youtube"],
"factual": ["hackernews", "reddit", "x", "xquik", "youtube"],
"product": ["youtube", "reddit", "x", "xquik", "tiktok"],
"concept": ["hackernews", "reddit", "x", "xquik", "youtube"],
"opinion": ["reddit", "x", "xquik", "youtube", "hackernews"],
"how_to": ["youtube", "reddit", "x", "xquik", "hackernews"],
"comparison": ["reddit", "x", "xquik", "hackernews", "youtube"],
"breaking_news": ["x", "xquik", "reddit", "hackernews", "youtube", "polymarket"],
"prediction": ["polymarket", "x", "xquik", "hackernews", "reddit", "youtube"],
}
SOURCE_PRIORITY = {
"factual": ["hackernews", "reddit", "x", "youtube"],
@@ -60,6 +60,7 @@ INTENT_SOURCE_EXCLUSIONS: dict[str, set[str]] = {
SOURCE_CAPABILITIES = {
"reddit": {"discussion", "social"},
"x": {"discussion", "social"},
"xquik": {"discussion", "social"},
"youtube": {"video", "video_longform", "discussion"},
"tiktok": {"video", "video_shortform", "social"},
"instagram": {"video", "video_shortform", "social"},
+5 -1
View File
@@ -19,7 +19,11 @@ OPENAI_RESPONSES_URL = "https://api.openai.com/v1/responses"
CODEX_RESPONSES_URL = "https://chatgpt.com/backend-api/codex/responses"
XAI_RESPONSES_URL = "https://api.x.ai/v1/responses"
OPENROUTER_URL = "https://openrouter.ai/api/v1/chat/completions"
OPENROUTER_DEFAULT = "google/gemini-flash-2.0"
# OpenRouter routes the Gemini Flash Lite tier as the -preview slug; that is the
# stable form on that routing layer even though native Gemini's GEMINI_FLASH_LITE
# constant is suffix-free. If GEMINI_FLASH_LITE moves to a non-preview stable ID,
# double-check that OpenRouter's slug still maps to the same upstream model.
OPENROUTER_DEFAULT = "google/gemini-3.1-flash-lite-preview"
class ReasoningClient:
+150 -7
View File
@@ -45,26 +45,100 @@ def _is_youtube_active(config: dict, research_results: dict) -> bool:
return True
# Below this transcript-fetch ratio, YouTube is considered "degraded" rather
# than active. Picked at 50% so a single legitimate caption-disabled video in a
# multi-video result does not trip the nudge, but a stale-yt-dlp run that fails
# every transcript does. Tunable via DEGRADED_TRANSCRIPT_THRESHOLD env var if
# operators need to adjust without code changes.
DEFAULT_DEGRADED_TRANSCRIPT_THRESHOLD = 0.5
def _is_youtube_degraded(research_results: dict, threshold: float) -> bool:
"""YouTube is degraded when videos were returned but the transcript-fetch
ratio is below threshold. The canonical cause is a stale yt-dlp binary -
YouTube's caption format changes frequently and old binaries silently fail
every transcript while the search itself still succeeds.
Captions-disabled videos are subtracted from the denominator: an uploader
who turned off captions can never produce a transcript, so counting that
video toward "fetch failures" produces false positives. A single
captions-disabled video in a small result set was tripping the nudge.
"""
videos = int(research_results.get("youtube_videos_count") or 0)
transcripts = int(research_results.get("youtube_transcripts_count") or 0)
captions_disabled = int(research_results.get("youtube_captions_disabled_count") or 0)
if videos <= 0:
return False
eligible = videos - captions_disabled
if eligible <= 0:
# Every returned video had captions disabled - upstream content fact,
# not a yt-dlp problem. Don't flag.
return False
return (transcripts / eligible) < threshold
def _is_instagram_silent_failure(config: dict, research_results: dict) -> bool:
"""Instagram is silently failing when SC is configured but the source
returned zero items. The canonical cause is SC's v2 reels endpoint
500'ing on multi-token queries (it wraps Google Search and is documented
to be flaky there). Pre-fix the user got no signal at all - no Instagram
section in the brief, no error in the footer, just unexplained absence.
"""
if not config.get("SCRAPECREATORS_API_KEY"):
return False # not configured — not a silent failure
# Honor EXCLUDE_SOURCES: a user who set EXCLUDE_SOURCES=instagram
# intentionally turned the source off, so a zero-item count is
# expected, not a silent failure. Mirror the canonical parsing
# pattern from pipeline.available_sources().
excluded = {
s.strip().lower()
for s in (config.get("EXCLUDE_SOURCES") or "").split(",")
if s.strip()
}
# Symmetric case: INCLUDE_SOURCES is an opt-in allowlist. If it is
# non-empty and does not name instagram, the source was intentionally
# filtered out, so a zero-item count is expected — not a silent failure.
included = {
s.strip().lower()
for s in (config.get("INCLUDE_SOURCES") or "").split(",")
if s.strip()
}
if "instagram" in excluded or (included and "instagram" not in included):
return False
count = research_results.get("instagram_items_count")
if count is None:
return False # source not run this invocation
return int(count) == 0
def compute_quality_score(config: dict, research_results: dict) -> dict:
"""Compute research quality score based on 5 core sources.
Args:
config: Configuration dict from env.get_config()
research_results: Dict with keys like x_error, youtube_error,
reddit_error reflecting what happened this run.
reddit_error reflecting what happened this run. Optional keys
``youtube_videos_count`` and ``youtube_transcripts_count`` enable
degraded-YouTube detection (transcript-fetch ratio below threshold).
Optional key ``instagram_items_count`` enables silent-failure
detection for the bonus Instagram source.
Returns:
{
"score_pct": 40-100,
"core_active": ["hn", "polymarket", ...],
"core_missing": ["x", "youtube"],
"core_errored": [], # configured but errored
"nudge_text": "..." or None if 100%
"core_errored": [], # configured but errored at top level
"core_degraded": [], # configured and returned items but quality below threshold
"bonus_errored": [], # bonus sources (Instagram, etc.) configured but silent
"nudge_text": "..." or None if all sources healthy
}
"""
core_active: List[str] = []
core_missing: List[str] = []
core_errored: List[str] = []
core_degraded: List[str] = []
bonus_errored: List[str] = []
# HN, Polymarket, and Reddit are always active
core_active.append("hn")
@@ -84,6 +158,13 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
yt_active = _is_youtube_active(config, research_results)
if yt_active:
core_active.append("youtube")
# Active means yt-dlp is installed and search did not error at the top
# level. But search-success + transcript-failure is the canonical
# stale-binary failure mode that the footer used to hide. Flag as
# degraded so the user gets an actionable nudge to update the binary.
threshold = float(config.get("DEGRADED_TRANSCRIPT_THRESHOLD") or DEFAULT_DEGRADED_TRANSCRIPT_THRESHOLD)
if _is_youtube_degraded(research_results, threshold):
core_degraded.append("youtube")
else:
core_missing.append("youtube")
# Check if configured but errored (yt-dlp installed but failed this run)
@@ -95,28 +176,54 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
if has_ytdlp and research_results.get("youtube_error"):
core_errored.append("youtube")
# Bonus sources (Instagram, etc.): SC-key holders expect content from
# these but until now the pipeline fell silent on configured-but-zero.
if _is_instagram_silent_failure(config, research_results):
bonus_errored.append("instagram")
score_pct = int(len(core_active) / 5 * 100)
has_sc = bool(config.get("SCRAPECREATORS_API_KEY"))
active_sources = research_results.get("active_sources") or []
nudge_text = _build_nudge_text(core_missing, core_errored, has_sc=has_sc, active_sources=active_sources) if core_missing else None
nudge_text = _build_nudge_text(
core_missing,
core_errored,
core_degraded,
research_results,
has_sc=has_sc,
active_sources=active_sources,
bonus_errored=bonus_errored,
) if (core_missing or core_degraded or bonus_errored) else None
return {
"score_pct": score_pct,
"core_active": core_active,
"core_missing": core_missing,
"core_errored": core_errored,
"core_degraded": core_degraded,
"bonus_errored": bonus_errored,
"nudge_text": nudge_text,
}
def _build_nudge_text(core_missing: List[str], core_errored: List[str], has_sc: bool = False, active_sources: list = None) -> str:
"""Build human-readable nudge text describing what was missed.
def _build_nudge_text(
core_missing: List[str],
core_errored: List[str],
core_degraded: List[str] = None,
research_results: dict = None,
has_sc: bool = False,
active_sources: list = None,
bonus_errored: List[str] = None,
) -> str:
"""Build human-readable nudge text describing what was missed or degraded.
Prioritizes free suggestions. Optionally mentions bonus sources
(TikTok, Instagram, Threads, Pinterest) if ScrapeCreators key is configured.
"""
lines: List[str] = []
core_degraded = core_degraded or []
bonus_errored = bonus_errored or []
research_results = research_results or {}
# Describe what was missed
missed_parts: List[str] = []
@@ -129,7 +236,14 @@ def _build_nudge_text(core_missing: List[str], core_errored: List[str], has_sc:
active_count = 5 - len(core_missing)
lines.append(f"Research quality: {active_count}/5 core sources.")
lines.append(f"Missing: {', '.join(missed_parts)}.")
if missed_parts:
lines.append(f"Missing: {', '.join(missed_parts)}.")
if core_degraded:
degraded_labels = ", ".join(SOURCE_LABELS[s] for s in core_degraded)
lines.append(f"Degraded: {degraded_labels}.")
if bonus_errored:
bonus_labels = ", ".join(s.capitalize() for s in bonus_errored)
lines.append(f"Bonus source silent: {bonus_labels}.")
lines.append("")
# Free suggestions
@@ -159,6 +273,35 @@ def _build_nudge_text(core_missing: List[str], core_errored: List[str], has_sc:
"explanations on any topic. Install yt-dlp: brew install yt-dlp (free)"
)
if "youtube" in core_degraded:
videos = int(research_results.get("youtube_videos_count") or 0)
transcripts = int(research_results.get("youtube_transcripts_count") or 0)
captions_disabled = int(research_results.get("youtube_captions_disabled_count") or 0)
captions_note = ""
if captions_disabled > 0:
captions_note = (
f" ({captions_disabled} of those had captions disabled by the "
"uploader, which is a separate cause and not fixable on your end)"
)
free_suggestions.append(
f"YouTube returned {videos} videos but only {transcripts} transcripts "
f"captured{captions_note}. The most common remaining cause is a stale "
"yt-dlp binary - YouTube's caption format changes frequently and old "
"binaries silently fail every transcript. Update via your package "
"manager: scoop update yt-dlp (Windows), brew upgrade yt-dlp (macOS), "
"or pip install -U yt-dlp."
)
if "instagram" in bonus_errored:
free_suggestions.append(
"Instagram returned 0 reels despite SC being configured. SC's "
"v2 reels endpoint wraps Google Search and 500's frequently on "
"multi-token queries. The skill now retries with hashtag-form "
"automatically; if zero items still appear, the topic may have "
"no reel coverage on Instagram. Try a single-word topic like "
"the most distinctive noun in your query."
)
# Mention bonus opt-in sources when SC key is present
if has_sc:
bonus_hints = []
+11 -1
View File
@@ -334,7 +334,7 @@ def _global_search(
)
return data.get("posts", data.get("data", []))
except http.HTTPError as e:
if e.status_code in (401, 403):
if e.status_code in (401, 402, 403):
raise
_log(f"Global search error: {e}")
return []
@@ -376,6 +376,11 @@ def _subreddit_search(
retries=2,
)
return data.get("posts", data.get("data", []))
except http.HTTPError as e:
if e.status_code in (401, 402, 403):
raise
_log(f"Subreddit search error for r/{subreddit}: {e}")
return []
except Exception as e:
_log(f"Subreddit search error for r/{subreddit}: {e}")
return []
@@ -403,6 +408,11 @@ def fetch_post_comments(
retries=2,
)
return data.get("comments", data.get("data", []))
except http.HTTPError as e:
if e.status_code in (401, 402, 403):
raise
_log(f"Comment fetch error: {e}")
return []
except Exception as e:
_log(f"Comment fetch error: {e}")
return []
+14 -3
View File
@@ -11,6 +11,7 @@ Handles 429 rate limits with exponential backoff, HTML anti-bot responses,
network timeouts, and missing subreddits.
"""
import gzip
import json
import sys
import time
@@ -21,7 +22,11 @@ from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeou
from typing import Any, Dict, List, Optional
USER_AGENT = "last30days/3.0 (research tool)"
USER_AGENT = (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0.0.0 Safari/537.36"
)
# Depth-aware limits for thread counts
DEPTH_LIMITS = {
@@ -60,6 +65,9 @@ def _fetch_json(url: str, timeout: int = 15) -> Optional[Dict[str, Any]]:
headers = {
"User-Agent": USER_AGENT,
"Accept": "application/json",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate",
"Connection": "keep-alive",
}
req = urllib.request.Request(url, headers=headers)
@@ -71,7 +79,10 @@ def _fetch_json(url: str, timeout: int = 15) -> Optional[Dict[str, Any]]:
_log(f"Anti-bot HTML response (Content-Type: {content_type})")
return None
body = resp.read().decode("utf-8")
raw = resp.read()
if resp.headers.get("Content-Encoding", "").lower() == "gzip":
raw = gzip.decompress(raw)
body = raw.decode("utf-8")
return json.loads(body)
except urllib.error.HTTPError as e:
@@ -198,7 +209,7 @@ def search(
encoded_query = _url_encode(query)
if subreddit:
sub = subreddit.lstrip("r/").strip()
sub = subreddit.removeprefix("r/").strip()
url = (
f"https://www.reddit.com/r/{sub}/search.json"
f"?q={encoded_query}&restrict_sr=on&sort=relevance&t=month&limit={limit}&raw_json=1"
+5 -4
View File
@@ -1285,15 +1285,16 @@ def _build_source_footer_lines(report: schema.Report) -> list[str]:
if total > 0:
total_str = f"{total:,}" if total >= 1000 else str(total)
parts.append(f"{total_str} {word}")
# YouTube: append "N with transcripts" instead of a third likes-based column.
# Transcripts are a more meaningful research-depth signal than likes.
# YouTube: always append "M/N with transcripts" so a zero-transcript run
# (typically caused by a stale yt-dlp binary) is visible at the conclusion
# surface. Hiding zero converts a problem signal into an absence; the very
# case that needs to be loud is the one previously omitted from the footer.
if source_key == "youtube":
with_transcripts = sum(
1 for it in items
if (it.metadata.get("transcript_highlights") or it.metadata.get("transcript_snippet"))
)
if with_transcripts > 0:
parts.append(f"{with_transcripts} with transcripts")
parts.append(f"{with_transcripts}/{len(items)} with transcripts")
stats = "".join(parts)
out.append(_footer_line_for_source(emoji, label, len(items), item_word, stats))
+88 -1
View File
@@ -160,6 +160,93 @@ def _extract_github_repos(items: list[dict]) -> list[str]:
return repos[:5] # cap at 5 repos
_INTEGRATION_SUFFIX_KEYWORDS: dict[str, set[str]] = {
"-action": {"action", "actions", "workflow", "workflows"},
"-sdk": {"sdk", "client", "library"},
"-plugin": {"plugin", "plugins", "extension", "extensions"},
"-plugins": {"plugin", "plugins", "extension", "extensions"},
"-docs": {"docs", "documentation"},
"-examples": {"example", "examples", "sample", "samples"},
"-template": {"template", "templates", "starter", "boilerplate"},
}
def _topic_tokens(topic: str) -> set[str]:
return set(re.findall(r"[a-z0-9]+", (topic or "").lower()))
def _topic_entity_slugs(topic: str) -> list[str]:
entities = re.split(r"\b(?:vs|versus)\b", (topic or "").lower())
slugs: list[str] = []
for entity in entities:
tokens = re.findall(r"[a-z0-9]+", entity)
if tokens:
slugs.append("-".join(tokens))
return slugs
def _repo_slug(repo: str) -> str:
parts = repo.split("/", 1)
if len(parts) != 2:
return ""
return parts[1].lower()
def _canonicalize_integration_repo(topic: str, repo: str) -> str:
"""Map integration repos back to canonical product repos when intent allows.
Example:
anthropics/claude-code-action -> anthropics/claude-code
unless topic explicitly asks for "action"/"workflow".
"""
parts = repo.split("/", 1)
if len(parts) != 2:
return repo
owner, name = parts[0], parts[1]
lower_name = name.lower()
topic_words = _topic_tokens(topic)
for suffix, intent_words in _INTEGRATION_SUFFIX_KEYWORDS.items():
if not lower_name.endswith(suffix):
continue
if topic_words.intersection(intent_words):
return repo
base = name[: -len(suffix)]
if base:
return f"{owner}/{base}"
return repo
def canonicalize_github_repos(topic: str, repos: list[str], *, cap: int | None = 5) -> list[str]:
"""Normalize/priority-sort GitHub repos for the current topic.
- Rewrites common integration suffixes to canonical product repos when
topic intent does not mention those integrations.
- Promotes exact topic slug matches (e.g., `claude-code`) over partials.
"""
canonicalized: list[str] = []
seen: set[str] = set()
for repo in repos:
candidate = _canonicalize_integration_repo(topic, repo.strip())
if "/" not in candidate:
continue
key = candidate.lower()
if key in seen:
continue
seen.add(key)
canonicalized.append(candidate)
topic_slugs = set(_topic_entity_slugs(topic))
if topic_slugs:
exact = [r for r in canonicalized if _repo_slug(r) in topic_slugs]
prefixed = [r for r in canonicalized if any(_repo_slug(r).startswith(f"{slug}-") for slug in topic_slugs) and r not in exact]
rest = [r for r in canonicalized if r not in exact and r not in prefixed]
canonicalized = exact + prefixed + rest
if cap is not None:
return canonicalized[:cap]
return canonicalized
def _build_context_summary(items: list[dict]) -> str:
"""Build a 1-2 sentence current events summary from news search results."""
snippets: list[str] = []
@@ -240,7 +327,7 @@ def auto_resolve(topic: str, config: dict) -> dict:
subreddits = _extract_subreddits(results.get("subreddit", []))
x_handle = _extract_x_handle(results.get("x_handle", []))
github_user = _extract_github_user(results.get("github", []))
github_repos = _extract_github_repos(results.get("github", []))
github_repos = canonicalize_github_repos(topic, _extract_github_repos(results.get("github", [])))
context = _build_context_summary(results.get("news", []))
subreddits, category = _merge_category_peers(topic, subreddits)
@@ -107,7 +107,18 @@ def extract_safari_cookies_macos(
if sys.platform != "darwin":
return None
cookie_path = Path.home() / "Library" / "Cookies" / "Cookies.binarycookies"
cookie_paths = [
Path.home()
/ "Library"
/ "Containers"
/ "com.apple.Safari"
/ "Data"
/ "Library"
/ "Cookies"
/ "Cookies.binarycookies",
Path.home() / "Library" / "Cookies" / "Cookies.binarycookies",
]
cookie_path = next((path for path in cookie_paths if path.exists()), cookie_paths[0])
try:
raw = cookie_path.read_bytes()
+15 -7
View File
@@ -175,16 +175,24 @@ def parse_x_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
break
if not output_text:
return items
response_preview = str(response)[:200] if response else "(empty)"
raise http.HTTPError(
f"xAI API returned empty response (no output text found; response preview: {response_preview})"
)
# Extract JSON from the response
json_match = re.search(r'\{[\s\S]*"items"[\s\S]*\}', output_text)
if json_match:
try:
data = json.loads(json_match.group())
items = data.get("items", [])
except json.JSONDecodeError:
_log(f"Failed to parse xAI response JSON: {output_text[:200]}")
if not json_match:
raise http.HTTPError(
f"xAI API returned output without valid JSON items structure (output: {output_text[:200]})"
)
try:
data = json.loads(json_match.group())
items = data.get("items", [])
except json.JSONDecodeError:
raise http.HTTPError(
f"xAI API returned valid output but invalid JSON structure (output: {output_text[:200]})"
)
# Validate and clean items
clean_items = []
+41 -7
View File
@@ -385,7 +385,11 @@ def _clean_vtt(vtt_text: str) -> str:
_YT_USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
def _fetch_transcript_direct(
video_id: str,
timeout: int = 30,
status: Optional[Dict[str, Any]] = None,
) -> Optional[str]:
"""Fetch YouTube transcript via direct HTTP without yt-dlp.
Scrapes the watch page HTML for the captions track URL in
@@ -394,6 +398,9 @@ def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
Args:
video_id: YouTube video ID
timeout: HTTP request timeout in seconds
status: Optional dict mutated to record per-video signals. Sets
``status["no_caption_tracks"] = True`` when the player response
confirms the uploader has no caption tracks (vs. fetch failure).
Returns:
Raw VTT text, or None if captions are unavailable.
@@ -442,6 +449,8 @@ def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
if not caption_tracks:
_log(f"Direct transcript: no caption tracks for {video_id}")
if status is not None:
status["no_caption_tracks"] = True
return None
# Find English track (prefer exact 'en', then any en variant, then first track)
@@ -527,7 +536,11 @@ def _fetch_transcript_ytdlp(video_id: str, temp_dir: str) -> Optional[str]:
return None
def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
def fetch_transcript(
video_id: str,
temp_dir: str,
status: Optional[Dict[str, Any]] = None,
) -> Optional[str]:
"""Fetch auto-generated transcript for a YouTube video.
Uses yt-dlp when available (preferred, more robust). Falls back to
@@ -536,6 +549,10 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
Args:
video_id: YouTube video ID
temp_dir: Temporary directory for subtitle files
status: Optional dict mutated by the direct-HTTP path to record
per-video signals like ``no_caption_tracks``. Used to surface a
captions-disabled count so the quality nudge avoids false-positive
"stale yt-dlp" flags.
Returns:
Plaintext transcript string, or None if no captions available.
@@ -551,13 +568,13 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
raw_vtt = _fetch_transcript_ytdlp(video_id, temp_dir)
if not raw_vtt:
_log(f"yt-dlp transcript failed for {video_id}, trying direct HTTP fallback")
raw_vtt = _fetch_transcript_direct(video_id)
raw_vtt = _fetch_transcript_direct(video_id, status=status)
else:
if ssh_host:
_log("SSH-routing active, using direct HTTP transcript fetch")
else:
_log("yt-dlp not installed, using direct HTTP transcript fetch")
raw_vtt = _fetch_transcript_direct(video_id)
raw_vtt = _fetch_transcript_direct(video_id, status=status)
if not raw_vtt:
_log(f"No transcript available for {video_id} (no captions found)")
@@ -576,12 +593,16 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
def fetch_transcripts_parallel(
video_ids: List[str],
max_workers: int = 5,
out_captions_disabled: Optional[Set[str]] = None,
) -> Dict[str, Optional[str]]:
"""Fetch transcripts for multiple videos in parallel.
Args:
video_ids: List of YouTube video IDs
max_workers: Max parallel fetches
out_captions_disabled: Optional set mutated to record video_ids whose
uploader confirmed no caption tracks (vs. transient fetch failures).
Backward-compatible: callers that don't care can omit.
Returns:
Dict mapping video_id to transcript text (or None).
@@ -592,10 +613,11 @@ def fetch_transcripts_parallel(
_log(f"Fetching transcripts for {len(video_ids)} videos")
results = {}
statuses: Dict[str, Dict[str, Any]] = {vid: {} for vid in video_ids}
with tempfile.TemporaryDirectory() as temp_dir:
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {
executor.submit(fetch_transcript, vid, temp_dir): vid
executor.submit(fetch_transcript, vid, temp_dir, statuses[vid]): vid
for vid in video_ids
}
for future in as_completed(futures):
@@ -609,6 +631,11 @@ def fetch_transcripts_parallel(
_log(f"Unexpected transcript error for {vid}: {type(exc).__name__}: {exc}")
results[vid] = None
if out_captions_disabled is not None:
for vid, st in statuses.items():
if st.get("no_caption_tracks"):
out_captions_disabled.add(vid)
got = sum(1 for v in results.values() if v)
errors = sum(1 for v in results.values() if v is None)
_log(f"Got transcripts for {got}/{len(video_ids)} videos ({errors} failed)")
@@ -659,15 +686,21 @@ def search_and_transcribe(
# good chance of reaching the target number of successful transcripts.
transcript_limit = TRANSCRIPT_LIMITS.get(depth, TRANSCRIPT_LIMITS["default"])
transcripts: Dict[str, Optional[str]] = {}
captions_disabled_ids: Set[str] = set()
if transcript_limit > 0:
attempt_count = min(len(items), transcript_limit * 3)
candidate_ids = [item["video_id"] for item in items[:attempt_count]]
_log(f"Fetching transcripts for up to {attempt_count} videos (target: {transcript_limit}): {candidate_ids}")
transcripts = fetch_transcripts_parallel(candidate_ids)
transcripts = fetch_transcripts_parallel(
candidate_ids, out_captions_disabled=captions_disabled_ids,
)
else:
_log(f"Transcript limit is 0 for depth={depth}, skipping transcript fetch")
# Step 3: Attach transcripts and extract highlights
# Step 3: Attach transcripts and extract highlights. Mark captions_disabled
# so quality_nudge can subtract those videos from the degraded-ratio
# denominator (uploader-disabled captions can never produce a transcript;
# counting them was producing false-positive stale-yt-dlp nudges).
core_topic = _extract_core_subject(topic)
for item in items:
vid = item["video_id"]
@@ -676,6 +709,7 @@ def search_and_transcribe(
item["transcript_highlights"] = extract_transcript_highlights(
transcript or "", core_topic,
)
item["captions_disabled"] = vid in captions_disabled_ids
return {"items": items}
+3 -2
View File
@@ -6,7 +6,8 @@ set -euo pipefail
# using `claude --print` to capture real end-to-end output.
SKILL_DIR="$HOME/.claude/skills/last30days"
REPO_DIR="/Users/mvanhorn/last30days-skill"
REPO_DIR="${REPO_DIR:-$(cd "$(dirname "$0")/.." && pwd)}"
CLAUDE="${CLAUDE:-$(command -v claude || echo claude)}"
# Safety: always restore V2 SKILL.md on exit/crash
cleanup() {
@@ -101,7 +102,7 @@ run_version() {
# Run claude --print with the skill invocation
# No timeout — claude --print exits on its own; kill manually if stuck
if /Users/mvanhorn/.local/bin/claude --print \
if "$CLAUDE" --print \
"/last30days $query" \
> "$outfile" 2>"$errfile"; then
local end_time
+147
View File
@@ -1,5 +1,6 @@
"""Tests for bluesky module."""
import os
import sys
import unittest
from pathlib import Path
@@ -211,5 +212,151 @@ class TestSearchBlueskyAuth(unittest.TestCase):
self.assertEqual(mock_request.call_args_list[3].kwargs.get("headers", {}), {"Authorization": "Bearer tok-new"})
class TestSearchEndpointHostResolution(unittest.TestCase):
"""The default search host moved from `public.api.bsky.app` (the
unauthenticated public mirror, now BunnyCDN-blocked for searchPosts) to
`api.bsky.app` (the canonical authenticated AppView). BSKY_SEARCH_HOST
env var or config value can override the default if Bluesky migrates
infrastructure again. Same os.environ-or-config hybrid pattern as
LAST30DAYS_STORE.
"""
def setUp(self):
# Snapshot env so per-test overrides don't leak
self._saved_env = os.environ.pop("BSKY_SEARCH_HOST", None)
def tearDown(self):
if self._saved_env is not None:
os.environ["BSKY_SEARCH_HOST"] = self._saved_env
else:
os.environ.pop("BSKY_SEARCH_HOST", None)
def test_resolver_default_uses_canonical_appview(self):
# Regression guard against the public mirror reappearing as the default.
# Anchored at the resolver because that is the code path search_bluesky
# actually calls; a module-level constant would not catch a resolver
# regression.
self.assertIn("api.bsky.app", bluesky._resolve_search_url())
def test_resolver_default_does_not_use_public_mirror(self):
# Hard regression guard — the exact host that BunnyCDN was blocking.
# Asserted at the resolver level (the runtime path) so a default-host
# regression in _resolve_search_url is actually caught.
self.assertNotIn("public.api.bsky.app", bluesky._resolve_search_url())
def test_resolver_default_when_no_override(self):
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_env_var_override(self):
os.environ["BSKY_SEARCH_HOST"] = "staging.bsky.app"
self.assertEqual(
bluesky._resolve_search_url(),
"https://staging.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_config_dict_override(self):
# User has BSKY_SEARCH_HOST only in .env file (project loads .env into
# config, not os.environ). Resolver must read both.
url = bluesky._resolve_search_url({"BSKY_SEARCH_HOST": "pds.example.com"})
self.assertEqual(url, "https://pds.example.com/xrpc/app.bsky.feed.searchPosts")
def test_resolver_env_var_wins_over_config(self):
# When both are set, os.environ takes precedence (matches LAST30DAYS_STORE)
os.environ["BSKY_SEARCH_HOST"] = "shell-host.example"
url = bluesky._resolve_search_url({"BSKY_SEARCH_HOST": "config-host.example"})
self.assertIn("shell-host.example", url)
self.assertNotIn("config-host.example", url)
def test_resolver_output_does_not_use_public_mirror(self):
# Regression guard at the resolver level (not just the constant) —
# this is what runtime actually calls. The constant-level guard
# above doesn't catch a regression where the resolver reverts.
self.assertNotIn("public.api.bsky.app", bluesky._resolve_search_url())
def test_resolver_strips_surrounding_whitespace(self):
# Pre-fix: " api.bsky.app " produced "https:// api.bsky.app /xrpc/..."
# which urllib raises ValueError on with no hint the env var caused it.
os.environ["BSKY_SEARCH_HOST"] = " api.bsky.app "
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_rejects_embedded_path(self):
# "my-proxy.com/xrpc/prefix" would have doubled the /xrpc/ segment.
# We fall back to the default to avoid a guaranteed 404.
os.environ["BSKY_SEARCH_HOST"] = "my-proxy.example.com/xrpc/prefix"
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_strips_embedded_scheme(self):
# Users who paste a full URL get a sane outcome, not a malformed URL.
os.environ["BSKY_SEARCH_HOST"] = "https://api.bsky.app"
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_empty_string_falls_back_to_default(self):
os.environ["BSKY_SEARCH_HOST"] = ""
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
class TestAppPasswordFormat(unittest.TestCase):
"""Bluesky app passwords are 19-char xxxx-xxxx-xxxx-xxxx (lowercase
alphanumeric, three hyphens at fixed positions). Main-account passwords
are accepted by createSession but are bad hygiene. The validator detects
the format mismatch without gating any caller.
"""
def test_accepts_valid_app_password_form(self):
# Use a fake example — never a real password
self.assertTrue(bluesky._validate_app_password_format("wfwp-cq7o-5six-7wy5"))
def test_rejects_length_15_string(self):
# The exact failure mode that triggered the 2026-05-04 investigation:
# user stored their main login password (15 chars) in BSKY_APP_PASSWORD
self.assertFalse(bluesky._validate_app_password_format("mainpassword123"))
def test_rejects_16_char_no_hyphen_string(self):
# Hex-style API key shape — common confusion with other services
self.assertFalse(bluesky._validate_app_password_format("abcdef0123456789"))
def test_rejects_uppercase_letters(self):
# Bluesky app passwords are all-lowercase by spec
self.assertFalse(bluesky._validate_app_password_format("WFWP-cq7o-5six-7wy5"))
def test_rejects_underscore_separator(self):
# Wrong separator
self.assertFalse(bluesky._validate_app_password_format("wfwp_cq7o_5six_7wy5"))
def test_rejects_special_chars_in_groups(self):
# Special characters are not part of the alphanumeric class
self.assertFalse(bluesky._validate_app_password_format("wfwp-cq7o-5six-7wy@"))
def test_rejects_empty_string(self):
self.assertFalse(bluesky._validate_app_password_format(""))
def test_rejects_none(self):
# Callers may pass config.get('BSKY_APP_PASSWORD') which is None when unset
self.assertFalse(bluesky._validate_app_password_format(None))
def test_rejects_integer(self):
# Defensive: don't crash if a numeric value sneaks in
self.assertFalse(bluesky._validate_app_password_format(123456789012345))
def test_rejects_list(self):
# Defensive: don't crash on iterables
self.assertFalse(bluesky._validate_app_password_format(["wfwp", "cq7o", "5six", "7wy5"]))
if __name__ == "__main__":
unittest.main()
+2 -2
View File
@@ -286,7 +286,7 @@ class TestFullExtraction:
with mock.patch("scripts.lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)):
with mock.patch(
"scripts.lib.chrome_cookies._get_chrome_encryption_key",
"scripts.lib.chrome_cookies._get_chromium_encryption_key",
return_value=KNOWN_PASSPHRASE,
):
result = extract_chrome_cookies_macos(".x.com", ["auth_token", "ct0"])
@@ -319,7 +319,7 @@ class TestFullExtraction:
with mock.patch("scripts.lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)):
with mock.patch(
"scripts.lib.chrome_cookies._get_chrome_encryption_key",
"scripts.lib.chrome_cookies._get_chromium_encryption_key",
return_value=KNOWN_PASSPHRASE,
):
result = extract_chrome_cookies_macos(".x.com", ["auth_token"])
+90
View File
@@ -1,6 +1,7 @@
# ruff: noqa: E402
import json
import io
import shutil
import tempfile
import subprocess
import sys
@@ -71,6 +72,26 @@ class CliV3Tests(unittest.TestCase):
cli.parse_search_flag("web, reddit, hn, web"),
)
def test_parse_search_flag_accepts_optional_social_sources(self):
self.assertEqual(
["threads", "pinterest"],
cli.parse_search_flag("threads, pinterest"),
)
def test_explicit_threads_search_uses_scrapecreators_key_without_include_sources(self):
available = cli.pipeline.available_sources(
{"SCRAPECREATORS_API_KEY": "test-key", "INCLUDE_SOURCES": ""},
requested_sources=["threads"],
)
self.assertIn("threads", available)
def test_explicit_perplexity_search_uses_openrouter_key_without_include_sources(self):
available = cli.pipeline.available_sources(
{"OPENROUTER_API_KEY": "test-key", "INCLUDE_SOURCES": ""},
requested_sources=["perplexity"],
)
self.assertIn("perplexity", available)
def test_parse_search_flag_rejects_invalid_or_empty_inputs(self):
with self.assertRaises(SystemExit):
cli.parse_search_flag("unknown")
@@ -143,6 +164,30 @@ class CliV3Tests(unittest.TestCase):
_, kwargs = write_text.call_args
self.assertEqual("utf-8", kwargs.get("encoding"))
def test_compute_save_path_display_uses_posix_slashes_under_home(self):
# Regression: f"~/{relative}" stringified pathlib.Path with the
# OS-native separator, producing "~/Documents\\Last30Days\\..." on
# Windows that no shell or File Explorer could open. The fix is
# f"~/{relative.as_posix()}" which forces forward slashes regardless
# of host OS. On POSIX hosts this asserts the contract for
# cross-platform safety; on Windows hosts it would fail without the fix.
real_home = Path.home()
tmp_under_home = Path(tempfile.mkdtemp(prefix="l30d_save_path_", dir=str(real_home)))
try:
save_dir = tmp_under_home / "Documents" / "Last30Days"
save_dir.mkdir(parents=True, exist_ok=True)
display = cli.compute_save_path_display(
str(save_dir), "british airways middle east", "v3", "compact"
)
self.assertTrue(display.startswith("~/"), f"Expected '~/' prefix, got: {display}")
self.assertNotIn("\\", display, f"Backslash leaked into display: {display}")
self.assertTrue(
display.endswith("british-airways-middle-east-raw-v3.md"),
f"Expected slug+suffix at end, got: {display}",
)
finally:
shutil.rmtree(tmp_under_home, ignore_errors=True)
def test_persist_report_updates_run_status_on_success_and_failure(self):
report = self.make_report()
@@ -215,6 +260,51 @@ class CliV3Tests(unittest.TestCase):
fake_progress.show_promo.assert_called_once_with("both", diag=diag)
self.assertIn("# rendered", stdout.getvalue())
def test_main_canonicalizes_explicit_github_repo_flags(self):
report = self.make_report()
diag = {
"available_sources": ["grounding"],
"providers": {"google": True, "openai": False, "xai": False},
"x_backend": None,
"bird_installed": True,
"bird_authenticated": False,
"bird_username": None,
"native_web_backend": "brave",
}
with mock.patch.object(cli.env, "get_config", return_value={}), \
mock.patch.object(cli.pipeline, "diagnose", return_value=diag), \
mock.patch.object(cli.pipeline, "run", return_value=report) as run_mock, \
mock.patch.object(cli, "emit_output", return_value="# rendered"), \
mock.patch.object(sys, "argv", [
"last30days.py",
"claude",
"code",
"vs",
"codex",
"--github-repo",
"openai/codex,anthropics/claude-code-action",
]):
stdout = io.StringIO()
stderr = io.StringIO()
with redirect_stdout(stdout), redirect_stderr(stderr):
rc = cli.main()
self.assertEqual(0, rc)
# In vs-mode main + competitors run in parallel via ThreadPoolExecutor,
# so the order of pipeline.run invocations is non-deterministic. Find
# the main runner's call by predicate on the canonicalized github_repos
# rather than by index.
expected_repos = ["openai/codex", "anthropics/claude-code"]
main_call = next(
(c for c in run_mock.call_args_list if c.kwargs.get("github_repos") == expected_repos),
None,
)
self.assertIsNotNone(
main_call,
f"No pipeline.run call had github_repos={expected_repos}; "
f"saw {[c.kwargs.get('github_repos') for c in run_mock.call_args_list]}",
)
self.assertIn("[GitHub] Canonicalized repos:", stderr.getvalue())
if __name__ == "__main__":
unittest.main()
+66
View File
@@ -122,6 +122,54 @@ class ExaSearchTests(unittest.TestCase):
self.assertEqual(0, artifact["resultCount"])
class ParallelSearchTests(unittest.TestCase):
def test_parallel_search_filters_to_in_range_dated_items(self):
mock_response = {
"results": [
{
"title": "Parallel Result",
"url": "https://example.com/parallel",
"snippet": "A parallel snippet",
"publish_date": "2026-03-15T00:00:00Z",
},
{
"title": "Old Parallel Result",
"url": "https://example.com/old-parallel",
"snippet": "Should be filtered",
"publish_date": "2025-12-01T00:00:00Z",
},
{
"title": "Undated Parallel Result",
"url": "https://example.com/undated-parallel",
"snippet": "Should also be filtered",
},
]
}
with patch("lib.grounding.http.request", return_value=mock_response) as mock_req:
items, artifact = grounding.parallel_search(
"test", ("2026-02-25", "2026-03-27"), "fake-parallel-key"
)
self.assertEqual(1, len(items))
self.assertEqual("Parallel Result", items[0]["title"])
self.assertEqual("https://example.com/parallel", items[0]["url"])
self.assertEqual("2026-03-15", items[0]["date"])
self.assertTrue(items[0]["id"].startswith("WP"))
self.assertEqual("parallel", artifact["label"])
self.assertEqual(1, artifact["resultCount"])
self.assertEqual("POST", mock_req.call_args.args[0])
self.assertEqual("https://api.parallel.ai/v1/search", mock_req.call_args.args[1])
self.assertEqual(
"Bearer fake-parallel-key",
mock_req.call_args.kwargs["headers"]["Authorization"],
)
def test_parallel_search_returns_empty_for_no_results(self):
with patch("lib.grounding.http.request", return_value={"results": []}):
items, artifact = grounding.parallel_search("test", ("2026-02-25", "2026-03-27"), "key")
self.assertEqual([], items)
self.assertEqual(0, artifact["resultCount"])
class WebSearchDispatchTests(unittest.TestCase):
def test_auto_selects_brave_when_key_present(self):
config = {"BRAVE_API_KEY": "test-key"}
@@ -141,6 +189,12 @@ class WebSearchDispatchTests(unittest.TestCase):
grounding.web_search("test", ("2026-02-25", "2026-03-27"), config, backend="auto")
mock.assert_called_once()
def test_auto_selects_parallel_when_only_parallel_key(self):
config = {"PARALLEL_API_KEY": "test-key"}
with patch("lib.grounding.parallel_search", return_value=([], {})) as mock:
grounding.web_search("test", ("2026-02-25", "2026-03-27"), config, backend="auto")
mock.assert_called_once()
def test_auto_returns_empty_when_no_keys(self):
items, artifact = grounding.web_search("test", ("2026-02-25", "2026-03-27"), {}, backend="auto")
self.assertEqual([], items)
@@ -167,6 +221,14 @@ class WebSearchDispatchTests(unittest.TestCase):
mock_exa.assert_called_once()
mock_serper.assert_not_called()
def test_auto_prefers_serper_over_parallel(self):
config = {"SERPER_API_KEY": "serper-key", "PARALLEL_API_KEY": "parallel-key"}
with patch("lib.grounding.serper_search", return_value=([], {})) as mock_serper, \
patch("lib.grounding.parallel_search", return_value=([], {})) as mock_parallel:
grounding.web_search("test", ("2026-02-25", "2026-03-27"), config, backend="auto")
mock_serper.assert_called_once()
mock_parallel.assert_not_called()
def test_auto_prefers_brave_when_all_keys_present(self):
config = {"BRAVE_API_KEY": "brave-key", "EXA_API_KEY": "exa-key", "SERPER_API_KEY": "serper-key"}
with patch("lib.grounding.brave_search", return_value=([], {})) as mock_brave, \
@@ -185,6 +247,10 @@ class WebSearchDispatchTests(unittest.TestCase):
with self.assertRaises(RuntimeError):
grounding.web_search("test", ("2026-02-25", "2026-03-27"), {}, backend="brave")
def test_explicit_parallel_without_key_raises(self):
with self.assertRaises(RuntimeError):
grounding.web_search("test", ("2026-02-25", "2026-03-27"), {}, backend="parallel")
def test_unsupported_backend_raises(self):
with self.assertRaises(ValueError):
grounding.web_search("test", ("2026-02-25", "2026-03-27"), {}, backend="google")
+169
View File
@@ -1,8 +1,10 @@
"""Tests for instagram.py — ScrapeCreators Instagram search module."""
import os
import sys
import unittest
from pathlib import Path
from unittest.mock import MagicMock, patch
# Add lib to path
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts"))
@@ -85,5 +87,172 @@ class TestInstagramDepthConfig(unittest.TestCase):
)
class TestHashtagFormCollapse(unittest.TestCase):
"""Tests for _to_hashtag_form() — the multi-word retry workaround."""
def test_collapses_spaces(self):
self.assertEqual(instagram._to_hashtag_form("toronto real estate"), "torontorealestate")
def test_lowercases(self):
self.assertEqual(instagram._to_hashtag_form("Toronto REAL Estate"), "torontorealestate")
def test_idempotent_on_single_word(self):
self.assertEqual(instagram._to_hashtag_form("ozempic"), "ozempic")
def test_handles_extra_whitespace(self):
self.assertEqual(instagram._to_hashtag_form(" toronto real estate "), "torontorealestate")
class TestSearchRetryOn500(unittest.TestCase):
"""Tests for the multi-word -> hashtag retry on SC's flaky 500 path.
SC's /v2/instagram/reels/search wraps Google Search and is documented
to be unreliable on multi-token queries. The retry collapses to a
hashtag form which hits the stable hashtag-page lookup path.
"""
def test_multiword_500_triggers_retry_with_hashtag_form(self):
"""Multi-word query 500 -> retry with collapsed hashtag form."""
from lib import http as http_module
first_error = http_module.HTTPError("HTTP 500: Server Error", 500, "")
second_payload = {"reels": []}
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.side_effect = [first_error, second_payload]
instagram.search_instagram(
"toronto real estate", "2026-04-01", "2026-05-04",
depth="default", token="fake-token",
)
self.assertEqual(mock_http_get.call_count, 2)
# First call: original multi-word query
first_params = mock_http_get.call_args_list[0].kwargs["params"]
self.assertEqual(first_params["query"], "toronto real estate")
# Second call: collapsed hashtag form
second_params = mock_http_get.call_args_list[1].kwargs["params"]
self.assertEqual(second_params["query"], "torontorealestate")
def test_singleword_500_does_not_retry(self):
"""Single-word query 500 has no spaces to collapse - no retry."""
from lib import http as http_module
only_error = http_module.HTTPError("HTTP 500: Server Error", 500, "")
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.side_effect = only_error
result = instagram.search_instagram(
"ozempic", "2026-04-01", "2026-05-04",
depth="default", token="fake-token",
)
self.assertEqual(mock_http_get.call_count, 1)
self.assertIn("error", result)
self.assertEqual(result["items"], [])
def test_first_call_succeeds_no_retry(self):
"""200 on first call -> retry path is never entered."""
from lib import http as http_module
ok_payload = {"reels": []}
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.return_value = ok_payload
instagram.search_instagram(
"toronto real estate", "2026-04-01", "2026-05-04",
depth="default", token="fake-token",
)
self.assertEqual(mock_http_get.call_count, 1)
def test_no_token_short_circuits(self):
"""No SCRAPECREATORS_API_KEY -> error returned without HTTP call."""
from lib import http as http_module
with patch.object(http_module, "get") as mock_http_get:
result = instagram.search_instagram(
"toronto real estate", "2026-04-01", "2026-05-04",
depth="default", token=None,
)
mock_http_get.assert_not_called()
self.assertIn("error", result)
self.assertIn("SCRAPECREATORS_API_KEY", result["error"])
class TestTranscriptTimeoutConfig(unittest.TestCase):
"""Tests for LAST30DAYS_TRANSCRIPT_TIMEOUT configuration.
SC's /v2/instagram/media/transcript endpoint regularly takes >15s,
so the timeout must be configurable. Default is DEFAULT_TRANSCRIPT_TIMEOUT
(30s); the env var or per-call kwarg overrides it.
"""
def setUp(self):
# Snapshot any pre-existing env so we don't leak across tests
self._saved_env = os.environ.pop("LAST30DAYS_TRANSCRIPT_TIMEOUT", None)
def tearDown(self):
os.environ.pop("LAST30DAYS_TRANSCRIPT_TIMEOUT", None)
if self._saved_env is not None:
os.environ["LAST30DAYS_TRANSCRIPT_TIMEOUT"] = self._saved_env
def _ok_payload(self):
return {"transcripts": [{"text": "hello world"}]}
def _video_item(self, vid="abc123"):
return {
"video_id": vid,
"url": f"https://www.instagram.com/reel/{vid}/",
"text": "",
}
def test_default_timeout_is_30s_when_nothing_set(self):
"""No env var, no kwarg -> request uses 30s, not the legacy 15s."""
from lib import http as http_module
items = [self._video_item()]
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.return_value = self._ok_payload()
instagram.fetch_captions(items, token="fake-token")
kwargs = mock_http_get.call_args.kwargs
self.assertEqual(kwargs["timeout"], 30.0)
def test_env_var_override(self):
"""LAST30DAYS_TRANSCRIPT_TIMEOUT='60' -> request uses 60s."""
from lib import http as http_module
os.environ["LAST30DAYS_TRANSCRIPT_TIMEOUT"] = "60"
items = [self._video_item()]
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.return_value = self._ok_payload()
instagram.fetch_captions(items, token="fake-token")
kwargs = mock_http_get.call_args.kwargs
self.assertEqual(kwargs["timeout"], 60.0)
def test_explicit_timeout_kwarg_wins_over_env(self):
"""Explicit timeout= kwarg trumps the env var."""
from lib import http as http_module
os.environ["LAST30DAYS_TRANSCRIPT_TIMEOUT"] = "60"
items = [self._video_item()]
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.return_value = self._ok_payload()
instagram.fetch_captions(items, token="fake-token", timeout=10)
kwargs = mock_http_get.call_args.kwargs
self.assertEqual(kwargs["timeout"], 10.0)
def test_config_dict_fallback_when_env_unset(self):
"""config={'LAST30DAYS_TRANSCRIPT_TIMEOUT': '45'} -> request uses 45s."""
from lib import http as http_module
items = [self._video_item()]
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.return_value = self._ok_payload()
instagram.fetch_captions(
items,
token="fake-token",
config={"LAST30DAYS_TRANSCRIPT_TIMEOUT": "45"},
)
kwargs = mock_http_get.call_args.kwargs
self.assertEqual(kwargs["timeout"], 45.0)
def test_invalid_env_value_falls_back_to_default(self):
"""Garbage env var doesn't crash; falls back to 30s."""
from lib import http as http_module
os.environ["LAST30DAYS_TRANSCRIPT_TIMEOUT"] = "not-a-number"
items = [self._video_item()]
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.return_value = self._ok_payload()
instagram.fetch_captions(items, token="fake-token")
kwargs = mock_http_get.call_args.kwargs
self.assertEqual(kwargs["timeout"], 30.0)
if __name__ == "__main__":
unittest.main()
+84
View File
@@ -0,0 +1,84 @@
import json
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
LAST30DAYS_SCRIPT = REPO_ROOT / "skills" / "last30days" / "scripts" / "last30days.py"
def run_last30days(topic: str, env: dict[str, str]) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[sys.executable, str(LAST30DAYS_SCRIPT), topic, "--mock", "--emit=json"],
cwd=REPO_ROOT,
env=env,
capture_output=True,
text=True,
check=False,
)
class LastRunStateTests(unittest.TestCase):
def test_empty_config_override_disables_last_run_write(self):
with tempfile.TemporaryDirectory() as tmp:
home = Path(tmp) / "home"
env = os.environ.copy()
env["HOME"] = str(home)
env["LAST30DAYS_CONFIG_DIR"] = ""
result = run_last30days("synthetic eval query", env)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse((home / ".config" / "last30days" / "last-run.json").exists())
def test_custom_config_override_writes_last_run_to_custom_dir(self):
with tempfile.TemporaryDirectory() as tmp:
config_dir = Path(tmp) / "custom-config"
env = os.environ.copy()
env["HOME"] = str(Path(tmp) / "home")
env["LAST30DAYS_CONFIG_DIR"] = str(config_dir)
result = run_last30days("custom config query", env)
self.assertEqual(result.returncode, 0, result.stderr)
payload = json.loads((config_dir / "last-run.json").read_text())
self.assertEqual(payload["topic"], "custom config query")
self.assertGreaterEqual(payload["total"], 0)
def test_hook_reads_last_run_from_custom_config_dir(self):
with tempfile.TemporaryDirectory() as tmp:
config_dir = Path(tmp) / "custom-config"
config_dir.mkdir()
(config_dir / "last-run.json").write_text(
json.dumps(
{
"topic": "custom hook query",
"timestamp": "2026-04-30T00:00:00+00:00",
"sources": {"reddit": 2},
"total": 2,
}
)
)
env = os.environ.copy()
env["HOME"] = str(Path(tmp) / "home")
env["LAST30DAYS_CONFIG_DIR"] = str(config_dir)
result = subprocess.run(
["bash", "hooks/scripts/check-config.sh"],
cwd=REPO_ROOT,
env=env,
capture_output=True,
text=True,
check=False,
)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn('Last run: "custom hook query"', result.stdout)
if __name__ == "__main__":
unittest.main()
+30
View File
@@ -52,6 +52,36 @@ class PipelineV3Tests(unittest.TestCase):
# At least one per-subquery line.
self.assertIn("[Planner] sq1 label=", output)
def test_parallel_web_backend_enables_grounding_source(self):
plan = {
"intent": "news",
"freshness_mode": "balanced_recent",
"cluster_mode": "timeline",
"subqueries": [
{
"label": "primary",
"search_query": "test topic",
"ranking_query": "What happened with test topic?",
"sources": ["grounding"],
}
],
"source_weights": {"grounding": 1.0},
}
report = pipeline.run(
topic="test topic",
config={"LAST30DAYS_REASONING_PROVIDER": "auto"},
depth="quick",
requested_sources=["grounding"],
web_backend="parallel",
external_plan=plan,
)
# Anchor on the stable source key, not the exact wording of the
# grounding.py error message. Phrasing can shift (e.g., when the
# missing-key check moves or the message is reworded) without
# changing the contract that the grounding source registers an
# error when its required backend key is unset.
self.assertIn("grounding", report.errors_by_source)
class TestSourceFetchCap(unittest.TestCase):
"""X source fetch count must be capped by MAX_SOURCE_FETCHES."""
+1
View File
@@ -36,6 +36,7 @@ class TestPluginContract(unittest.TestCase):
self.assertEqual(version, _skill_version())
self.assertEqual(version, _json(ROOT / ".claude-plugin" / "plugin.json")["version"])
self.assertEqual(version, _json(ROOT / "gemini-extension.json")["version"])
marketplace = _json(ROOT / ".claude-plugin" / "marketplace.json")
plugins = marketplace.get("plugins") or []
+380 -2
View File
@@ -5,6 +5,11 @@ HN, Polymarket, Reddit (always active), X, YouTube.
ScrapeCreators adds TikTok + Instagram as bonus sources, not core.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
import pytest
from unittest.mock import patch
@@ -38,8 +43,8 @@ def _base_results(**overrides):
def _compute(config_overrides=None, result_overrides=None, ytdlp_installed=False):
"""Helper to call compute_quality_score with mocked yt-dlp check."""
from scripts.lib.quality_nudge import compute_quality_score
from scripts.lib import youtube_yt
from lib.quality_nudge import compute_quality_score
from lib import youtube_yt
config = _base_config(**(config_overrides or {}))
results = _base_results(**(result_overrides or {}))
@@ -199,3 +204,376 @@ class TestRedditNeverInCoreErrored:
# Reddit is always-active in core (public path), error doesn't demote it
assert "reddit" in q["core_active"]
assert q["score_pct"] == 100
class TestYouTubeDegraded:
"""YouTube is `degraded` when videos returned but transcripts below threshold.
Canonical failure mode: a stale yt-dlp binary still finds videos via search
but silently fails every transcript fetch because YouTube's caption format
has moved on. Pre-fix the user got no signal of this; the footer hid zero,
and quality_nudge only checked top-level errors.
"""
def test_zero_of_six_transcripts_flags_degraded(self):
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 6,
"youtube_transcripts_count": 0,
},
)
assert "youtube" in q["core_degraded"]
assert q["nudge_text"] is not None
# Counts surface in the message so the user sees the actual ratio
assert "6 videos" in q["nudge_text"]
assert "0 transcripts" in q["nudge_text"]
assert "stale yt-dlp" in q["nudge_text"].lower()
# Updates path mentions all three common package managers
assert "scoop" in q["nudge_text"].lower()
assert "brew" in q["nudge_text"].lower()
assert "pip install" in q["nudge_text"].lower()
def test_five_of_six_transcripts_does_not_flag_degraded(self):
# 83% transcript success - well above the 50% threshold
# X is also enabled so all 5 cores are active and no nudge should fire
q = _compute(
config_overrides={"AUTH_TOKEN": "tok123"},
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 6,
"youtube_transcripts_count": 5,
},
)
assert "youtube" not in q["core_degraded"]
assert q["nudge_text"] is None # All 5 core sources active, no degradation
def test_zero_videos_does_not_flag_degraded(self):
# No videos returned -> degraded check is meaningless and must not fire
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 0,
"youtube_transcripts_count": 0,
},
)
assert "youtube" not in q["core_degraded"]
def test_one_of_three_transcripts_flags_degraded(self):
# 33% - below 50% threshold; the canonical "yt-dlp partially working" case
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 3,
"youtube_transcripts_count": 1,
},
)
assert "youtube" in q["core_degraded"]
assert "Degraded: YouTube" in q["nudge_text"]
def test_threshold_tunable_via_config(self):
# Operator overrides threshold via env-style config to be more permissive
q = _compute(
config_overrides={"DEGRADED_TRANSCRIPT_THRESHOLD": "0.1"},
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 10,
"youtube_transcripts_count": 2, # 20%, below default 50% but above override 10%
},
)
assert "youtube" not in q["core_degraded"]
def test_degraded_does_not_affect_score(self):
# Degradation is informational, not score-affecting; YouTube still counts as active
q = _compute(
config_overrides={"AUTH_TOKEN": "tok123"},
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 6,
"youtube_transcripts_count": 0,
},
)
assert "youtube" in q["core_active"]
assert q["score_pct"] == 100 # Full active count regardless of degradation
# But nudge still fires
assert q["nudge_text"] is not None
assert "Degraded: YouTube" in q["nudge_text"]
class TestYouTubeCaptionsDisabledDoesNotFalseFlag:
"""Captions-disabled videos must not lower the transcript-fetch ratio.
A video where the uploader disabled captions can never produce a transcript,
no matter how fresh yt-dlp is. Counting it in the denominator of the
degraded-ratio check produces false positives - one captions-disabled video
in a small result set was triggering a "stale yt-dlp binary" nudge that was
wrong. Fix: subtract captions_disabled from the denominator.
"""
def test_zero_captions_disabled_preserves_existing_behavior(self):
# Pre-existing case: 0 of 6 transcripts is still degraded (no captions
# disabled to discount). Behavior is unchanged from TestYouTubeDegraded.
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 6,
"youtube_transcripts_count": 0,
"youtube_captions_disabled_count": 0,
},
)
assert "youtube" in q["core_degraded"]
def test_all_videos_captions_disabled_does_not_flag(self):
# Every returned video had captions disabled by the uploader.
# That's not a yt-dlp problem - it's an upstream content fact. Must not
# flag degraded.
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 3,
"youtube_transcripts_count": 0,
"youtube_captions_disabled_count": 3,
},
)
assert "youtube" not in q["core_degraded"]
def test_mixed_uses_corrected_denominator(self):
# 6 videos, 3 captions_disabled, 2 transcripts.
# Naive (buggy) ratio: 2/6 = 33% (would flag).
# Corrected ratio: 2/(6-3) = 67% (does NOT flag).
# This case demonstrates the fix changes the verdict.
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 6,
"youtube_transcripts_count": 2,
"youtube_captions_disabled_count": 3,
},
)
assert "youtube" not in q["core_degraded"]
def test_mixed_still_flags_when_truly_degraded(self):
# Even after discounting captions-disabled, the ratio is still bad.
# 8 videos, 1 captions_disabled, 1 transcript -> 1/(8-1) = 14% (flags).
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 8,
"youtube_transcripts_count": 1,
"youtube_captions_disabled_count": 1,
},
)
assert "youtube" in q["core_degraded"]
# Nudge should still mention the stale yt-dlp possibility but also
# acknowledge that captions-disabled is a separate cause.
assert q["nudge_text"] is not None
assert "captions disabled" in q["nudge_text"].lower()
def test_missing_count_defaults_to_zero(self):
# Older callers that don't pass the new key still work (default 0).
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 6,
"youtube_transcripts_count": 0,
# youtube_captions_disabled_count intentionally omitted
},
)
assert "youtube" in q["core_degraded"]
class TestInstagramSilentFailure:
"""Instagram is a `bonus` source via SC. Silent-failure detection: if SC
is configured but the source returned zero items, surface a nudge so the
user understands why the brief lacks an Instagram section.
Pre-fix the user got no signal - SC's /v2/instagram/reels/search 500s
frequently on multi-token queries and the pipeline silently returned
empty without any indication.
"""
def test_zero_items_with_sc_flags_bonus_errored(self):
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" in q["bonus_errored"]
assert q["nudge_text"] is not None
assert "Instagram" in q["nudge_text"]
def test_zero_items_without_sc_does_not_flag(self):
q = _compute(
config_overrides={"AUTH_TOKEN": "tok123"},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" not in q.get("bonus_errored", [])
def test_nonzero_items_does_not_flag(self):
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 5},
)
assert "instagram" not in q["bonus_errored"]
assert q["nudge_text"] is None
def test_missing_key_means_source_did_not_run(self):
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
},
ytdlp_installed=True,
)
assert "instagram" not in q["bonus_errored"]
assert q["nudge_text"] is None
def test_nudge_text_explains_workaround(self):
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert q["nudge_text"] is not None
text_lower = q["nudge_text"].lower()
assert "instagram" in text_lower
assert ("0 reels" in text_lower or "silent" in text_lower
or "hashtag" in text_lower)
def test_bonus_errored_does_not_affect_core_score(self):
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert q["score_pct"] == 100
assert "instagram" in q["bonus_errored"]
assert q["nudge_text"] is not None
assert "Bonus source silent" in q["nudge_text"]
def test_bonus_errored_field_always_present(self):
q = _compute()
assert q.get("bonus_errored") == []
def test_exclude_sources_instagram_suppresses_silent_failure(self):
"""User set EXCLUDE_SOURCES=instagram - the source intentionally did
not run, so the zero-count instagram_items_count written by
last30days.py is a non-event, not a silent failure. Pre-fix: the
nudge fired anyway because the gate only checked SC-key + count.
"""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"EXCLUDE_SOURCES": "instagram",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" not in q["bonus_errored"]
assert q["nudge_text"] is None
def test_exclude_sources_multi_value_with_instagram(self):
"""Canonical parsing pattern is comma-separated; case-insensitive."""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"EXCLUDE_SOURCES": "threads, Instagram , pinterest",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" not in q["bonus_errored"]
def test_exclude_sources_other_value_still_flags(self):
"""EXCLUDE_SOURCES that does not mention instagram must not suppress
the silent-failure nudge for instagram.
"""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"EXCLUDE_SOURCES": "threads",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" in q["bonus_errored"]
def test_include_sources_without_instagram_suppresses_silent_failure(self):
"""User set INCLUDE_SOURCES to an opt-in allowlist that omits
instagram the pipeline skips the source by allowlist filter, so
the zero-count instagram_items_count is intentional, not a silent
failure. Symmetric to the EXCLUDE_SOURCES=instagram guard.
"""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"INCLUDE_SOURCES": "reddit,hn,x,youtube",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" not in q["bonus_errored"]
assert q["nudge_text"] is None
def test_include_sources_multi_value_without_instagram(self):
"""Canonical parsing pattern is comma-separated; case-insensitive."""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"INCLUDE_SOURCES": " Reddit, HN , YouTube ",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" not in q["bonus_errored"]
def test_include_sources_with_instagram_still_flags(self):
"""INCLUDE_SOURCES that explicitly names instagram must not suppress
the silent-failure nudge the source was opted in, so a zero count
is a real silent failure.
"""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"INCLUDE_SOURCES": "reddit,instagram",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" in q["bonus_errored"]
def test_include_sources_empty_does_not_suppress(self):
"""Empty/unset INCLUDE_SOURCES means no allowlist filter, so the
silent-failure gate should still fire when instagram is zero.
"""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"INCLUDE_SOURCES": "",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" in q["bonus_errored"]
+1 -1
View File
@@ -313,7 +313,7 @@ class TestSearchRedditPublicHighLevel:
reddit_public.search("test")
req = mock_urlopen.call_args[0][0]
assert req.get_header("User-agent") == "last30days/3.0 (research tool)"
assert "Mozilla/5.0" in req.get_header("User-agent")
class TestMissingSubreddit:
+85
View File
@@ -552,5 +552,90 @@ class DegradedRunBannerTests(unittest.TestCase):
self.assertIn("--plan", text)
class YoutubeFooterTranscriptRatioTests(unittest.TestCase):
"""The YouTube footer line must surface the transcript-fetch ratio in all
cases where videos were returned. Pre-fix the segment was suppressed when
transcripts == 0, which converted the canonical stale-yt-dlp failure mode
into a silent absence at the footer (the very surface users read for
'did this work?'). Always-render the ratio so zero is loud.
"""
def _build_youtube_report(self, transcript_flags: list[bool]) -> schema.Report:
"""Build a Report with one YouTube item per entry in transcript_flags.
True means the item has transcript data; False means it does not.
"""
items = []
for idx, has_transcript in enumerate(transcript_flags):
metadata = {"views": 1000}
if has_transcript:
metadata["transcript_highlights"] = ["Some pre-extracted quote."]
items.append(schema.SourceItem(
item_id=f"yt{idx}",
source="youtube",
title=f"Video {idx}",
body=f"Description for video {idx}.",
url=f"https://youtube.com/watch?v=v{idx}",
container="some-channel",
published_at="2026-04-15",
date_confidence="high",
engagement={"views": 1000, "likes": 100},
metadata=metadata,
))
return schema.Report(
topic="test topic",
range_from="2026-04-01",
range_to="2026-05-01",
generated_at="2026-05-01T00:00:00+00:00",
provider_runtime=schema.ProviderRuntime(
reasoning_provider="gemini",
planner_model="gemini",
rerank_model="gemini",
),
query_plan=schema.QueryPlan(
intent="general",
freshness_mode="balanced_recent",
cluster_mode="none",
raw_topic="test topic",
subqueries=[schema.SubQuery(
label="primary", search_query="test topic",
ranking_query="What about test topic?", sources=["youtube"],
)],
source_weights={"youtube": 1.0},
),
clusters=[],
ranked_candidates=[],
items_by_source={"youtube": items},
errors_by_source={},
)
def test_zero_transcripts_with_videos_present_renders_zero_over_total(self):
# The canonical stale-yt-dlp case: 6 videos found, 0 transcripts captured.
# Pre-fix the footer hid this entirely; post-fix it must say "0/6 with transcripts".
report = self._build_youtube_report([False] * 6)
text = render.render_compact(report)
self.assertIn("0/6 with transcripts", text)
def test_partial_transcripts_renders_ratio(self):
# 5 of 6 transcripts captured - shows ratio so user knows one was missed.
report = self._build_youtube_report([True] * 5 + [False])
text = render.render_compact(report)
self.assertIn("5/6 with transcripts", text)
def test_full_transcripts_renders_ratio(self):
# All 3 transcripts captured - still shows ratio for consistency.
report = self._build_youtube_report([True] * 3)
text = render.render_compact(report)
self.assertIn("3/3 with transcripts", text)
def test_no_videos_no_transcript_segment(self):
# When YouTube has no items at all, the YouTube footer line is
# suppressed entirely (existing behavior) - the transcript segment
# should not appear without a parent line.
report = self._build_youtube_report([])
text = render.render_compact(report)
# No YouTube footer line at all - so no transcript segment either
self.assertNotIn("with transcripts", text)
if __name__ == "__main__":
unittest.main()
+18
View File
@@ -109,6 +109,24 @@ class TestBuildContextSummary(unittest.TestCase):
self.assertEqual(resolve._build_context_summary(items), "")
class TestCanonicalizeGithubRepos(unittest.TestCase):
def test_rewrites_integration_repo_to_canonical_product(self):
repos = ["openai/codex", "anthropics/claude-code-action"]
result = resolve.canonicalize_github_repos("claude code vs codex", repos, cap=None)
self.assertEqual(result, ["openai/codex", "anthropics/claude-code"])
def test_preserves_action_repo_when_topic_intends_action(self):
repos = ["anthropics/claude-code-action", "openai/codex"]
result = resolve.canonicalize_github_repos("claude code action setup", repos, cap=None)
self.assertIn("anthropics/claude-code-action", result)
self.assertNotIn("anthropics/claude-code", result)
def test_dedupes_case_insensitive_after_canonicalization(self):
repos = ["Anthropics/Claude-Code-Action", "anthropics/claude-code"]
result = resolve.canonicalize_github_repos("claude code", repos, cap=None)
self.assertEqual(result, ["Anthropics/Claude-Code"])
class TestAutoResolve(unittest.TestCase):
def test_no_backend_returns_empty(self):
result = resolve.auto_resolve("test topic", {})
+84 -14
View File
@@ -160,25 +160,95 @@ class TestErrorPaths:
result = extract_safari_cookies_macos("x.com", ["auth_token"])
assert result is None
def test_prefers_sandboxed_safari_cookie_path(
self, tmp_path: Path, x_cookies_file: bytes
):
sandbox_dir = (
tmp_path
/ "Library"
/ "Containers"
/ "com.apple.Safari"
/ "Data"
/ "Library"
/ "Cookies"
)
sandbox_dir.mkdir(parents=True)
(sandbox_dir / "Cookies.binarycookies").write_bytes(x_cookies_file)
legacy_dir = tmp_path / "Library" / "Cookies"
legacy_dir.mkdir(parents=True)
legacy_data = _build_binary_cookies_file(
[_build_page([_build_cookie_record(".x.com", "auth_token", "legacy")])]
)
(legacy_dir / "Cookies.binarycookies").write_bytes(legacy_data)
with patch(
"scripts.lib.safari_cookies.Path.home", return_value=tmp_path
), patch("scripts.lib.safari_cookies.sys") as mock_sys:
mock_sys.platform = "darwin"
mock_sys.stderr = sys.stderr
result = extract_safari_cookies_macos("x.com", ["auth_token", "ct0"])
assert result is not None
assert result["auth_token"] == "test_auth_abc123"
assert result["ct0"] == "test_ct0_xyz789"
def test_falls_back_to_legacy_safari_cookie_path(self, tmp_path: Path):
# Sandboxed path is intentionally NOT created — only the legacy path exists.
legacy_dir = tmp_path / "Library" / "Cookies"
legacy_dir.mkdir(parents=True)
legacy_data = _build_binary_cookies_file(
[_build_page([_build_cookie_record(".x.com", "auth_token", "legacy_auth")])]
)
(legacy_dir / "Cookies.binarycookies").write_bytes(legacy_data)
sandbox_path = (
tmp_path
/ "Library"
/ "Containers"
/ "com.apple.Safari"
/ "Data"
/ "Library"
/ "Cookies"
/ "Cookies.binarycookies"
)
assert not sandbox_path.exists()
with patch(
"scripts.lib.safari_cookies.Path.home", return_value=tmp_path
), patch("scripts.lib.safari_cookies.sys") as mock_sys:
mock_sys.platform = "darwin"
mock_sys.stderr = sys.stderr
result = extract_safari_cookies_macos("x.com", ["auth_token"])
assert result is not None
assert result["auth_token"] == "legacy_auth"
def test_permission_denied(self, tmp_path: Path, capsys):
cookie_dir = tmp_path / "Library" / "Cookies"
cookie_dir = (
tmp_path
/ "Library"
/ "Containers"
/ "com.apple.Safari"
/ "Data"
/ "Library"
/ "Cookies"
)
cookie_dir.mkdir(parents=True)
cookie_file = cookie_dir / "Cookies.binarycookies"
cookie_file.write_bytes(b"cook")
cookie_file.chmod(0o000)
try:
with patch(
"scripts.lib.safari_cookies.Path.home", return_value=tmp_path
), patch("scripts.lib.safari_cookies.sys") as mock_sys:
mock_sys.platform = "darwin"
mock_sys.stderr = sys.stderr
result = extract_safari_cookies_macos("x.com", ["auth_token"])
assert result is None
captured = capsys.readouterr()
assert "Full Disk Access" in captured.err
finally:
cookie_file.chmod(0o644)
with patch(
"scripts.lib.safari_cookies.Path.home", return_value=tmp_path
), patch("scripts.lib.safari_cookies.sys") as mock_sys, patch.object(
Path, "read_bytes", side_effect=PermissionError
):
mock_sys.platform = "darwin"
mock_sys.stderr = sys.stderr
result = extract_safari_cookies_macos("x.com", ["auth_token"])
assert result is None
captured = capsys.readouterr()
assert "Full Disk Access" in captured.err
def test_truncated_magic_only(self):
result = _parse_binary_cookies(b"cook", "x.com", ["auth_token"])
+2 -2
View File
@@ -252,7 +252,7 @@ class TestFetchTranscriptFallback(unittest.TestCase):
mock.patch.object(youtube_yt, "_fetch_transcript_direct", return_value=sample_vtt) as direct_mock:
result = youtube_yt.fetch_transcript("vid2", "/tmp/test")
yt_mock.assert_not_called()
direct_mock.assert_called_once_with("vid2")
direct_mock.assert_called_once_with("vid2", status=None)
self.assertIsNotNone(result)
self.assertIn("Direct transcript content", result)
@@ -363,7 +363,7 @@ class TestSearchAndTranscribe(unittest.TestCase):
]
# fetch_transcripts_parallel returns None for music videos, text for talks
def fake_parallel(video_ids, max_workers=5):
def fake_parallel(video_ids, max_workers=5, out_captions_disabled=None):
result = {}
for vid in video_ids:
if vid.startswith("talk"):