Compare commits

..

154 Commits

Author SHA1 Message Date
Trevin Chow daca71f89e chore(release): v3.3.0
Release / build-and-release (push) Has been cancelled
~75 PRs merged since v3.2.0 plus 7 community fixes salvaged via PR triage.

Highlights:
- Install everywhere: npx skills add is canonical for Claude Code, Codex,
  Cursor, Gemini CLI, Copilot, Windsurf, and 50+ Agent Skills hosts.
- New emit mode: --emit=html for shareable HTML briefs.
- New source: Digg (auto-enabled when digg-pp-cli on PATH).
- New env vars: EXCLUDE_SOURCES, LAST30DAYS_YOUTUBE_SSH_HOST.
- New credential source: macOS Keychain.
- Reliability sweep: Reddit (4xx + URL prefix + multi-key auth), xAI
  error surfacing, Windows compatibility, YouTube/HN unblock,
  HTTP retries, planner gating, render fixes.
- Multi-harness reframe: AGENTS.md becomes canonical, CLAUDE.md points
  at it. SKILL_ROOT → SKILL_DIR substitution.

Breaking:
- .codex-plugin/plugin.json removed. Codex installs via npx skills add.
2026-05-17 09:25:46 -07:00
Kaustav Mishra d51e91ea26 fix(xai): surface API errors instead of silently returning empty results
parse_x_response was returning an empty items list whenever xAI returned
a 200 OK with a malformed payload — empty output text, missing "items"
key, or invalid JSON. The pipeline saw "successful response with zero
items" and quietly handed the user a degraded report with no indication
the API had failed. Now raise http.HTTPError on each of those branches
so _retrieve_stream's caller catches it and surfaces the failure in
errors_by_source, giving the user a visible signal that X didn't work.

Closes #155.

Co-authored-by: Kaustav Mishra <km.git007@gmail.com>
2026-05-17 09:20:37 -07:00
Trevin Chow 170b570cbc fix(reddit): re-raise HTTP 402 so fallback chain triggers
The ScrapeCreators 402 (payment required / credits exhausted) status
was being swallowed by the broad except Exception handlers in
_global_search, _subreddit_search, and fetch_post_comments, returning
[] instead of propagating. That caused users with exhausted credits
to silently get zero Reddit results instead of falling through to
the OpenAI / public Reddit JSON fallback chain in _search_reddit_thread.
Add 402 to the existing 401/403 re-raise list across all three
ScrapeCreators call paths. Closes #170.

Co-authored-by: Jonathan Oppenheim <no-reply@postquantum.space>
2026-05-17 09:20:37 -07:00
Trevin Chow 4bae05e7fa fix(reddit): use browser-like headers to fix HTTP 403 from urllib
Reddit's public JSON endpoint returns 403 to requests carrying the
generic User-Agent and minimal header set urllib defaults to, while
matching curl requests succeed. Switch to a current-Chrome User-Agent
and add Accept-Language / Accept-Encoding / Connection headers so the
fingerprint matches a normal browser. Reddit now serves gzip when
Accept-Encoding includes it, so decompress the body before JSON parse.
Update the user-agent assertion in tests/test_reddit_public.py to match
the new browser-like string. Closes #199.

Co-authored-by: Franco Carballar <francocarballar@gmail.com>
2026-05-17 09:20:37 -07:00
Trevin Chow a4f1f94802 fix(env): restore multi-key rotation for SCRAPECREATORS_API_KEY
Originally added in #268 to spread load across free-tier accounts when
SCRAPECREATORS_API_KEY is set to a comma-separated list. The 7-line block
was inadvertently dropped during the v3.0.6 consolidation (d14814a) even
though the changelog still advertised the feature. Re-apply the same
random.choice rotation in get_config() so user-facing behavior matches
the documented contract. Closes #287.

Co-authored-by: Eric Oberhofer <eric@oberhofer.io>
2026-05-17 09:20:37 -07:00
Trevin Chow 16ce073d0c fix(cli): keep child cleanup working on Windows
_cleanup_children() called os.killpg unconditionally — Windows doesn't
have killpg as an attribute on os, so the call raised AttributeError
(not caught by the existing OSError-family handler) and aborted cleanup.
Guard with hasattr(os, "killpg") and fall back to os.kill(pid, SIGTERM)
on platforms without process-group APIs. Closes #226. Refs #110.

Co-authored-by: gujishh <baiaoshh@163.com>
2026-05-17 09:20:02 -07:00
Trevin Chow 5994b4f76a fix(reddit): use removeprefix("r/") for subreddit names, not lstrip("r/")
str.lstrip("r/") treats its argument as a character set, stripping
leading r and / repeatedly. Subreddits starting with 'r' (e.g. r/robotics,
r/ruby) were silently mangled to 'obotics' / 'uby'. Replace with
str.removeprefix("r/") at all four call sites. Python 3.9+ pattern is
safe here — project requires 3.12. Closes #288.

Co-authored-by: Alex Key <alexanderkey0508@gmail.com>
2026-05-17 09:20:02 -07:00
Trevin Chow bb5e6efbf9 fix(scripts): replace hardcoded developer paths in test-v1-vs-v2.sh
REPO_DIR now derives from the script's location (with env-var override)
and the Claude binary is looked up via PATH (with CLAUDE env-var override)
instead of hardcoded to /Users/mvanhorn/.local/bin/claude. Works on any
checkout. Closes #297.

Co-authored-by: Dave Morin <dave@morin.com>
2026-05-17 09:20:02 -07:00
Trevin Chow 76b8df40d3 Merge pull request #318 from flyingnobita/fix/gemini-claude-polyglot-hooks
fix: make hooks.json polyglot for Gemini CLI and Claude Code compatibility
2026-05-17 01:05:30 -07:00
Trevin Chow eb2d8b55e0 Merge pull request #344 from dzivkovi/feat/config-enablement
feat: configuration enablement — env-var defaults + source resilience
2026-05-17 01:05:08 -07:00
Trevin Chow 1a8ffd4847 fix(quality_nudge): also guard Instagram silent-failure on INCLUDE_SOURCES allowlist 2026-05-17 01:03:55 -07:00
Trevin Chow 1814bb1967 fix(quality_nudge,bluesky): gate Instagram nudge on EXCLUDE_SOURCES + anchor bluesky tests at resolver 2026-05-17 01:03:55 -07:00
Trevin Chow f236cff86a chore(pr-344): adapt to rebased base — fix test imports + memory-dir doc style
Rebased onto current main where:
- instagram.py uses unified http.get (not _requests fallback); tests now
  mock http.get and assert params/timeout kwargs.
- quality_nudge tests use lib.* import path with sys.path setup.
- README/CONFIGURATION.md memory-dir lines say "defaults to" so they pass
  test_no_stray_hardcoded_memory_dir_paths.
2026-05-17 01:03:55 -07:00
Daniel Zivkovic 44971a6aae feat: configuration enablement — env-var defaults + source resilience
Six small additive changes that make the skill correctly understand its
configured sources, plus tests + docs.

User-visible benefits

- LAST30DAYS_STORE=1 in .env turns persistence default-on without
  remembering --store on every invocation. Mirrors LAST30DAYS_DEBUG /
  LAST30DAYS_SKIP_PREFLIGHT convention.
- SCRAPE_CREATORS_API_KEY (with underscore) accepted as alias for the
  canonical name. Matches the spelling used in the vendor's own example
  code (Adrian Horning's repo); saves the next user the same diagnostic
  rabbit hole.
- Bluesky search now hits api.bsky.app (canonical AppView) instead of
  public.api.bsky.app (BunnyCDN-blocked public mirror as of 2026-05-04).
  BSKY_SEARCH_HOST env var lets users self-rescue future host migrations
  without a code release. Pre-fix: silent 0 Bluesky posts on every run.
- App-password format validator emits a one-shot stderr warning when
  BSKY_APP_PASSWORD doesn't match xxxx-xxxx-xxxx-xxxx form. Detect-don't-
  gate: createSession still accepts main passwords; the warning helps
  users identify a hygiene issue without breaking existing setups.
- Instagram retry on multi-token 500. SC's v2 reels endpoint wraps
  Google Search and 500's frequently on multi-word queries; a hashtag-
  form retry runs once before bubbling up. Documented vendor instability.
- LAST30DAYS_TRANSCRIPT_TIMEOUT env var (default 30s, was hardcoded 15s).
  SC's transcript endpoint regularly takes >15s; the old default was
  clipping legitimate responses.
- Silent-failure visibility: new bonus_errored field in the quality
  nudge fires when SC is configured but Instagram returned 0 items.
  Users see "Bonus source silent: Instagram" instead of unexplained
  absence.
- YouTube degraded-ratio false-positive fixed. Captions-disabled videos
  can never produce a transcript regardless of yt-dlp version; they're
  now subtracted from the denominator so a single uploader-disabled
  video doesn't false-trigger the "stale yt-dlp" nudge.
- urllib retry path: status_code attribute typo fix. The Instagram
  500-retry was dead code on the urllib branch (getattr(e, 'status', ...)
  while http.HTTPError exposes status_code).

Docs

- README.md: added /plugin install last30days step after marketplace add
  in three places (the install was previously omitted in the docs).
- CONFIGURATION.md: documented LAST30DAYS_STORE env var, added
  BSKY_SEARCH_HOST + app-password format section, mentioned
  LAST30DAYS_TRANSCRIPT_TIMEOUT in the Instagram source row.

Test plan

- 43 new unit tests across test_bluesky.py, test_instagram_sc.py,
  test_quality_nudge.py, test_youtube_yt.py
- 141 total tests passing in target suite
- Verified end-to-end: /last30days "Toronto resale condo market" with
  all 11+ sources active stored 35 new + 5 updated findings, all builder-
  PR-style accounts absent (organic agent voice in Instagram + TikTok
  results)

Backward compatibility

All changes are strictly additive. Optional kwargs default to None.
New env vars are opt-in. Existing CLI flags untouched. Existing callers
of public functions unaffected.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 01:03:15 -07:00
Daniel Zivkovic a8e462c978 chore(gitignore): ignore /work and /print (personal artifacts)
Personal directories used by the /note slash command (work/) and PDF
print exports (print/) - these are local research artifacts, not
shipping content. Lives on daniel/personal to keep upstream/main and
PR branches free of personal noise.
2026-05-17 01:02:47 -07:00
Trevin Chow d9a0ac31f2 Merge pull request #323 from GAOJIAN-0106/fix/openrouter-default-model-id
fix: correct invalid OPENROUTER_DEFAULT model ID
2026-05-17 01:01:54 -07:00
Trevin Chow 3e60c0817d Merge pull request #339 from dzivkovi/docs/configuration-md
docs: add CONFIGURATION.md + README pointers
2026-05-17 01:01:37 -07:00
Trevin Chow d530c90239 fix(hooks): remove timeout field per PR description intent 2026-05-17 00:59:41 -07:00
Trevin Chow aef6f35460 docs(providers): note why OpenRouter slug includes -preview suffix 2026-05-17 00:59:31 -07:00
GAOJIAN-0106 77bd235f64 fix: correct invalid OPENROUTER_DEFAULT model ID
google/gemini-flash-2.0 is not a valid OpenRouter model ID (segments reversed).
Every rerank and FunJudge call fails with HTTP 400 when REASONING_PROVIDER=openrouter
and LAST30DAYS_RERANK_MODEL is not explicitly pinned, silently falling back to
local-score heuristics.

OpenRouter error body:
  {"message":"google/gemini-flash-2.0 is not a valid model ID","code":400}

Switching to google/gemini-3.1-flash-lite-preview, which matches the
GEMINI_FLASH_LITE constant already used by the native Gemini provider on
line 12 of the same file. This makes the Gemini and OpenRouter providers
consistent and avoids a future divergence.

Validated with /last30days 'Claude Opus 4.7' --quick:
- Rerank/FunJudge HTTP 400 errors: 1 per run -> 0
- 'Why: fallback-local-score' markers in output: every cluster -> 0
- LLM-generated 'Why:' reasoning lines: 0 -> 11
2026-05-17 00:59:11 -07:00
flyingnobita e1017e95c9 fix: make hooks.json polyglot for Gemini CLI and Claude Code compatibility 2026-05-17 00:58:46 -07:00
Trevin Chow 71b1e8a411 Merge pull request #302 from nidhi-singh02/fix/github-repo-canonicalization
fix: Canonicalize ambiguous GitHub repo resolution for product comparisons
2026-05-17 00:57:05 -07:00
Trevin Chow 618458eb7e Merge pull request #320 from kaushikgopal/feat/brave-cookie-extraction
Add Brave browser support for X/Twitter cookie extraction
2026-05-17 00:56:44 -07:00
Trevin Chow 8cccd3e982 Merge pull request #334 from iamitp/codex/last-run-config-state
Preserve clean mode for last run state
2026-05-17 00:56:28 -07:00
Trevin Chow 17fb17222b fix(docs): use 'defaults to' wording for LAST30DAYS_MEMORY_DIR refs
The version_consistency test (test_no_stray_hardcoded_memory_dir_paths)
flagged 4 lines where `~/Documents/Last30Days/` appeared without the
canonical "defaults to" phrasing or the ${LAST30DAYS_MEMORY_DIR:-...}
literal form. Tightened the wording in the CONFIGURATION.md table +
footer paragraph, and the README "Where research files are saved"
section, so each path mention is anchored at the env-var override
contract rather than as a bare hardcoded default.
2026-05-17 00:54:49 -07:00
Trevin Chow 8ccd778366 fix(canonicalization): predicate-based call lookup + skip double-canon on auto-resolve
Two findings from Greptile review on PR #302:

1. tests/test_cli_v3.py:302 - The test asserted run_mock.call_args_list[0]
   was the main runner's invocation, but fanout.run_competitor_fanout
   submits main + competitors to a ThreadPoolExecutor and iterates with
   as_completed. With zero-latency mocks, thread scheduling determines
   which pipeline.run call lands first, so the competitor's call could
   take index [0] and flake CI. Replace [0] indexing with a predicate
   match on the canonicalized github_repos kwargs.

2. skills/last30days/scripts/last30days.py:662 - When auto_resolve returns
   github_repos, it has already run canonicalize_github_repos(cap=5) and
   ranked by relevance. The downstream block then re-canonicalized with
   cap=None, which can re-sort by topic-slug match and clobber the
   auto_resolve relevance order. Guard the second canonicalization with
   a repos_from_auto_resolve flag so it only fires for user-supplied
   --github-repo input.
2026-05-17 00:51:21 -07:00
Trevin Chow a3f173dc8a docs(readme): drop community-video link from "Going deeper" callout
The 6-min architecture walkthrough video is the contributor's own
work — not affiliated with the project. Removing the README pointer
to avoid implying endorsement. CONFIGURATION.md prose stands.
2026-05-17 00:51:14 -07:00
Trevin Chow 5a3ac8ca37 docs(config): document briefing.py show [--date DATE] subcommand 2026-05-17 00:50:36 -07:00
Trevin Chow e8eb15102f docs(agents): steer agents on maintaining CONFIGURATION.md 2026-05-17 00:50:36 -07:00
Daniel Zivkovic fd6e70c539 docs: add CONFIGURATION.md + README pointers + community video link
Adds CONFIGURATION.md at repo root - a focused configuration reference
covering save paths, the per-source API-key matrix, reasoning and
web-search backend priority, the trend-monitoring stack (--store +
watchlist.py + briefing.py), and per-client patterns.

Surfaces two things that ship in the engine but were not documented
for users:

- The project-scoped .claude/last30days.env config file (currently only
  referenced in hooks/scripts/check-config.sh) which takes precedence
  over the global ~/.config/last30days/.env when present. Cleanest
  pattern for per-client setups - drop a file in the client folder, cd
  in, run normally.
- The existing trend-monitoring scripts (--store flag, watchlist.py,
  briefing.py) that the README did not surface for users.

Updates README with a brief "Configuration" section pointing to the new
file, plus a one-line "Going deeper" callout linking a 6-min community
architecture walkthrough on YouTube.

All CLI surface claims (watchlist subcommands, briefing modes, source
dedupe key, env file priority chain) fact-checked against the live
scripts/ source before commit.
2026-05-17 00:50:36 -07:00
nidhi-singh02 d0b990e211 Canonicalize GitHub repo resolution for ambiguous product repos 2026-05-17 00:50:05 -07:00
Trevin Chow 0f03a67166 Merge pull request #343 from Bortlesboat/codex/use-sandboxed-safari-cookie-path
fix: prefer sandboxed Safari cookie path
2026-05-17 00:47:32 -07:00
Trevin Chow 5a625fda9f Merge pull request #355 from dinakars777/test/cover-parallel-grounding-backend
test: cover parallel grounding backend
2026-05-17 00:47:20 -07:00
Trevin Chow b296a65515 fix(last-run): guard python3 absence + hoist datetime + use context manager 2026-05-17 00:44:30 -07:00
Trevin Chow f2737fc035 test(grounding): fix published_date → publish_date mock key mismatch 2026-05-17 00:42:07 -07:00
Trevin Chow 9ce7264d43 test(grounding): add serper>parallel priority + parallel empty-results coverage 2026-05-17 00:41:19 -07:00
Dinakar Sarbada f458e0f5af test: cover parallel grounding backend 2026-05-17 00:41:19 -07:00
Trevin Chow 8f565ee241 fix(chrome_cookies): sort Brave profiles by mtime, not alphabetically 2026-05-17 00:40:10 -07:00
Amit Patnaik dd7e6a1562 Preserve clean mode for last run state 2026-05-17 00:39:44 -07:00
KG 65313ce542 feat(cookies): add Brave browser cookie extraction for macOS
Brave uses identical v10 AES-128-CBC encryption to Chrome; only the
DB path (BraveSoftware/Brave-Browser) and Keychain service name
("Brave Safe Storage") differ. Refactored chrome_cookies.py to share
a single _extract_chromium_cookies_macos helper rather than duplicating
the decryption logic.

Profile discovery tries Default/ first, then scans numbered Profile N/
directories so non-default Brave profiles are covered.
2026-05-17 00:39:30 -07:00
Trevin Chow 5ab8c3ba76 Merge pull request #345 from dinakars777/docs/fix-stale-script-paths
docs: fix stale script paths
2026-05-17 00:38:17 -07:00
Trevin Chow 38bfb504e1 Merge pull request #349 from dinakars777/chore/sync-gemini-extension-version
chore: sync gemini extension version
2026-05-17 00:38:01 -07:00
Trevin Chow e8f23b4205 Merge pull request #340 from dzivkovi/fix/youtube-transcript-observability
fix(youtube): surface transcript-fetch ratio + add degraded nudge for stale yt-dlp
2026-05-17 00:31:37 -07:00
Trevin Chow b78ce34922 test(safari_cookies): add coverage for legacy fallback path 2026-05-17 00:31:09 -07:00
Bortlesboat 0656b868e7 fix safari cookie path resolution 2026-05-17 00:30:25 -07:00
Trevin Chow 16a4fa9c39 Merge pull request #341 from flyingice/main
fix(grounding): align Parallel AI search with current API schema
2026-05-17 00:27:50 -07:00
Trevin Chow 3b75ff1537 Merge pull request #354 from dinakars777/fix/parallel-web-backend-source
fix: route parallel web backend through grounding
2026-05-17 00:26:50 -07:00
Trevin Chow 321975e144 Merge pull request #356 from dinakars777/fix/allow-threads-pinterest-search
fix: honor explicit optional source requests
2026-05-17 00:26:25 -07:00
Trevin Chow 5f9c637bbe Merge pull request #336 from davemorin/fix/319-xquik-source-capabilities
fix(planner): register xquik in SOURCE_CAPABILITIES (#319)
2026-05-17 00:26:08 -07:00
Trevin Chow 19132b0b5e Merge pull request #338 from dzivkovi/fix/windows-save-path-footer
fix(render): use forward slashes in save-path footer for Windows
2026-05-17 00:25:45 -07:00
Trevin Chow 1e4150ad78 Merge pull request #347 from dinakars777/docs/update-how-search-key-files
docs: update search key file paths
2026-05-17 00:24:55 -07:00
Trevin Chow 6a15afd8e8 Update skills/last30days/scripts/last30days.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-05-17 00:23:53 -07:00
Trevin Chow 0ab7051bc5 fix(planner): also register xquik in QUICK_SOURCE_PRIORITY 2026-05-17 00:22:04 -07:00
Trevin Chow ec0b126af6 test(pipeline): relax grounding assertion to stable source key 2026-05-17 00:21:57 -07:00
Trevin Chow d9e8a046ef docs(how-search): swap score.py->relevance.py at line 145 2026-05-17 00:21:36 -07:00
Dave Morin 87bf3debcc fix(planner): register xquik in SOURCE_CAPABILITIES (#319)
Without this entry, the planner's _default_sources_for_intent() drops
xquik from the candidate pool for how_to / comparison / news intents
because SOURCE_CAPABILITIES.get("xquik", set()) returns the empty set.
Users with XQUIK_API_KEY set get zero Xquik results even though the
engine recognizes the key.

Mirrors the capabilities for "x" since both are X/Twitter-shaped
discussion + social sources.

Fixes #319
2026-05-17 00:21:36 -07:00
Dinakar Sarbada 2f4b023db8 docs: update search key file paths 2026-05-17 00:21:15 -07:00
Trevin Chow 261ea5895c refactor(pipeline): remove dead threads-explicit-request branch 2026-05-17 00:20:56 -07:00
Trevin Chow 2692e0f4a2 Merge pull request #351 from dinakars777/docs/fix-changelog-skill-link-note
docs: correct changelog skill link note
2026-05-17 00:20:46 -07:00
Trevin Chow 10f35f82fe fix(grounding): guard parallel excerpts against None + cap at 500 chars 2026-05-17 00:20:27 -07:00
Dinakar Sarbada 5b29b8f427 fix: honor explicit perplexity source requests 2026-05-17 00:20:25 -07:00
Dinakar Sarbada 6a5a122195 fix: honor explicit threads source requests 2026-05-17 00:20:25 -07:00
Dinakar Sarbada 7bda02169d fix: allow threads and pinterest search sources 2026-05-17 00:20:25 -07:00
Trevin Chow 6b40d2c46f Merge pull request #346 from dinakars777/docs/fix-bug-report-repro-command
docs: fix bug report repro command
2026-05-17 00:20:21 -07:00
Trevin Chow 4a99c4f557 Merge pull request #337 from UncleMike1988/fix/path-quoting-spaces
Fix path-quoting in SessionStart check-config hook (handles spaces in…
2026-05-17 00:20:09 -07:00
Dinakar Sarbada 5c802b0daa fix: route parallel web backend through grounding 2026-05-17 00:20:05 -07:00
Trevin Chow 0e353ae03f fix(render): apply as_posix to fallback branch + hoist shutil import 2026-05-17 00:19:49 -07:00
Daniel Zivkovic 0a5102e193 fix(youtube): surface transcript-fetch ratio in footer + add degraded nudge
When yt-dlp is installed but stale (or otherwise unable to fetch transcripts
for any returned videos), runs previously reported YouTube as fully
successful in two user-facing surfaces:

  1. Footer (render.py): showed "N videos | M views" with no indication
     that zero transcripts were captured. The "with transcripts" segment
     was conditionally suppressed when the count was zero - converting
     the canonical stale-binary failure mode into a silent absence at
     the very surface users read for "did this work?".

  2. Quality nudge (quality_nudge.py): classified YouTube as "active"
     based purely on yt-dlp installation + absence of a top-level error.
     Per-video transcript-fetch ratio was never inspected. A run that
     returned N videos with 0 transcripts (canonical stale-binary
     failure) was reported as fully active.

The engine itself logs the failure correctly at default stderr level
(`[YouTube] Got transcripts for 0/N videos (N failed)`), but that line
gets buried in 100+ lines of parallel-source progress output and is
contradicted by the success-shaped footer and nudge that follow.

This change makes both conclusion surfaces honest:

* render.py footer always renders "M/N with transcripts" so the ratio
  is visible regardless of value. Zero is no longer hidden. Format is
  M/N (not bare M) so the denominator is in the message and the user
  does not have to cross-reference the "videos" count.

* quality_nudge.py adds a third tier between "active" and "missing":
  "degraded". Triggered when yt-dlp is installed AND videos were
  returned AND transcript-fetch ratio is below threshold (default 50%,
  tunable via DEGRADED_TRANSCRIPT_THRESHOLD env var). Emits an
  actionable nudge: "YouTube returned N videos but only M transcripts
  captured. The most common cause is a stale yt-dlp binary - YouTube's
  caption format changes frequently and old binaries silently fail
  every transcript. Update via your package manager: scoop update
  yt-dlp (Windows), brew upgrade yt-dlp (macOS), or pip install -U
  yt-dlp."

* last30days.py populates youtube_videos_count and
  youtube_transcripts_count in the research_results dict it passes to
  compute_quality_score, enabling the new degraded check at the call
  site.

Threshold rationale: 50% accommodates a few legitimate
caption-disabled videos in a multi-video result, but a stale-binary
run that fails every transcript trips the nudge cleanly.

Score impact: degradation is informational, not score-affecting.
YouTube still counts as "active" in score_pct so users do not see
their score drop for a fixable client-side issue. The nudge directs
them to their own package manager.

Tests:

* tests/test_quality_nudge.py: 6 new TestYouTubeDegraded cases cover
  zero-transcripts-flags-degraded, partial-above-threshold-does-not-flag,
  zero-videos-does-not-flag (no false positives on absence),
  one-of-three-flags-degraded, threshold-tunable-via-config, and
  degraded-does-not-affect-score.

* tests/test_render_v3.py: 4 new YoutubeFooterTranscriptRatioTests
  cases cover zero-transcripts-with-videos-renders-zero-over-total
  (the regression repro), partial-renders-ratio, full-renders-ratio,
  and no-videos-suppresses-entire-segment.

All 29 new test cases verified GREEN with the fix and RED without it
(temp-reverted both files separately to confirm each test catches the
specific regression it asserts).

Integration validation: ran the engine against an intentionally stale
yt-dlp 2025.03.31 binary placed first on PATH. Pre-fix the footer
showed `YouTube: 3 videos | 386,815 views` (no transcript signal).
Post-fix the footer shows `YouTube: 3 videos | 386,815 views | 0/3
with transcripts` and stderr emits "Degraded: YouTube" plus the
actionable update-yt-dlp nudge.

Out of scope (deserves its own PR): exposing transcripts_captured in
the EVIDENCE FOR SYNTHESIS block so the synthesizing model can flag
degradation in prose. Larger schema-touching change.
2026-05-17 00:13:56 -07:00
Dinakar Sarbada cfde1dbbe4 docs: fix stale script paths 2026-05-17 00:08:57 -07:00
Dinakar Sarbada 608381a818 docs: fix bug report repro command 2026-05-17 00:08:17 -07:00
flyingice af4cf7c03d fix(grounding): align Parallel AI search with current API schema 2026-05-17 00:08:10 -07:00
Michael Turner 5b0308b9e4 Fix path-quoting in SessionStart check-config hook (handles spaces in CLAUDE_PLUGIN_ROOT)
If CLAUDE_PLUGIN_ROOT ever expands to a path containing whitespace
(e.g. ~/Library/Application Support/...), the unquoted ${CLAUDE_PLUGIN_ROOT}
in hooks/hooks.json word-splits and bash receives the path as multiple
arguments, failing with "No such file or directory" on the first split.

Quoting the expansion makes the invocation correct regardless of the
characters in the resolved path. Verified manually:
  unquoted + space  -> bash: /tmp/with: No such file or directory
  quoted   + space  -> bash: /tmp/with spaces/.../check-config.sh: No such file
  quoted   + real   -> /last30days: Ready - 7 sources active.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 00:08:01 -07:00
Daniel Zivkovic 5817ef8387 test(cli): regression test for Windows save-path display
Asserts compute_save_path_display() never returns a backslash when the
save_dir is under the user's home directory, regardless of host OS.

Reproduces the original bug on Windows (failed message before the fix:
  AssertionError: '\' unexpectedly found in
  '~/l30d_save_path__luu2g76\Documents\Last30Days\british-airways-middle-east-raw-v3.md'
)
and locks in the contract on POSIX hosts (passes trivially today; would
fail if anyone removes .as_posix() in the future).

Verified by temporarily reverting the fix and confirming RED, then
re-applying the fix and confirming GREEN. All 13 CliV3Tests pass.
2026-05-17 00:07:44 -07:00
Daniel Zivkovic a87c1ba058 fix(render): use forward slashes in save-path footer for Windows
The footer line `📎 Raw results saved to ~/Documents\Last30Days\…`
mangled the home-relative path on Windows because `f"~/{relative}"`
stringifies a `pathlib.Path` with the OS-native separator. The result
mixes a Unix tilde with backslashes, which neither File Explorer,
PowerShell, nor a `file://` URI can resolve.

`Path.as_posix()` always returns forward slashes, which is the
convention `~/`-prefixed paths require on every platform. macOS and
Linux output is unchanged because their separator is already `/`.

Repro on Windows:
  python3 last30days.py "anything" --emit=compact --save-dir="$HOME/Documents/Last30Days"
  # before: 📎 Raw results saved to ~/Documents\Last30Days\anything-raw.md
  # after:  📎 Raw results saved to ~/Documents/Last30Days/anything-raw.md
2026-05-17 00:07:44 -07:00
Trevin Chow 7214dd6051 Merge pull request #348 from dinakars777/docs/fix-readme-skill-link
docs: fix runtime skill spec link
2026-05-17 00:04:30 -07:00
Trevin Chow 6acf2fdbe2 Merge pull request #419 from mvanhorn/chore/remove-orphaned-spec-tasks
chore: remove orphaned SPEC.md and TASKS.md
2026-05-17 00:03:06 -07:00
Trevin Chow 07a3bdb3cf Merge pull request #364 from davemorin/fix/361-unsafe-eval-check-config
fix(hooks): replace unsafe eval with declare in check-config.sh
2026-05-17 00:02:44 -07:00
Dinakar Sarbada 87577ff126 test: guard gemini extension version 2026-05-17 00:00:59 -07:00
Dinakar Sarbada 400fc4cc00 docs: fix runtime skill spec link 2026-05-17 00:00:58 -07:00
Dinakar Sarbada e9ecce0b1c chore: sync gemini extension version 2026-05-17 00:00:51 -07:00
Trevin Chow f3df47c381 chore: remove orphaned SPEC.md and TASKS.md
Both files lived at the repo root as pre-plugin-layout artifacts. On
current main neither is referenced from README, SKILL.md, AGENTS.md,
CHANGELOG, or docs/ — no inbound links to break by removing. Git history
preserves the content for anyone who needs to dig it up.

Closes #352, #353. The PRs by @dinakars777 correctly flagged the drift;
deletion is the cleaner resolution than annotating them as historical.
2026-05-17 00:00:47 -07:00
Dinakar Sarbada c9cf3ef92f docs: correct changelog skill link note 2026-05-16 23:59:30 -07:00
Trevin Chow 46cf2328aa fix(hooks): use printf -v for bash 3.2 compat (declare -g is 4.2+)
macOS ships /bin/bash 3.2 and the script uses #!/bin/bash with
set -euo pipefail, so declare -g would abort the SessionStart hook
with "invalid option" on every Mac. printf -v writes via assignment
semantics (global from inside a function on 3.2+) — same scope
outcome, broader compatibility.
2026-05-16 23:49:44 -07:00
Trevin Chow 7506cbd542 fix(hooks): scope ENV_* to global (declare -g) so caller sees values 2026-05-16 23:49:05 -07:00
Dave Morin a6bd481e61 fix(hooks): replace unsafe eval with declare in check-config.sh
The load_env_vars function used eval to assign .env values, which
executes command substitutions in backtick-containing comments.
Replace eval with declare and strip inline comments before assignment.

Fixes #361
2026-05-16 23:49:05 -07:00
Trevin Chow aba6172032 Merge pull request #366 from davemorin/feat/324-reddit-json-fallback
feat(web): auto-enrich Reddit URLs from web search via JSON API
2026-05-16 23:41:57 -07:00
Trevin Chow d07e4698e3 Merge pull request #358 from dinakars777/fix/openclaw-poll-clock-init
fix: initialize OpenClaw poll timing once
2026-05-16 23:41:42 -07:00
Trevin Chow 4c0282dd55 Merge pull request #365 from davemorin/fix/284-version-metadata-drift
fix(version): replace hardcoded v3.0.0 with dynamic _skill_version()
2026-05-16 23:40:53 -07:00
Trevin Chow 99909fca67 Merge pull request #368 from hnshah/ren/advisory-security-workflow-252
ci: add advisory security workflow
2026-05-16 23:40:28 -07:00
Trevin Chow 36c43d50b7 test: drop side_effect padding to match collapsed time.time() call 2026-05-16 23:39:14 -07:00
Dinakar Sarbada ecf68347db fix: initialize OpenClaw poll timing once 2026-05-16 23:38:30 -07:00
Trevin Chow e2d9d705f6 review: fix selftext key path + break on RedditRateLimitError 2026-05-16 23:37:00 -07:00
Hiten Shah 9c09a67ac2 ci: add advisory security workflow 2026-05-16 23:36:55 -07:00
Dave Morin 32da0bd6cb test: update version assertions for dynamic _skill_version()
Tests now check for version prefix without hardcoded version number,
matching the render.py change to use _skill_version() dynamically.
2026-05-16 23:35:26 -07:00
Dave Morin 863c3bc145 fix(version): replace hardcoded v3.0.0 with dynamic _skill_version()
render.py, ui.py, and last30days.py had hardcoded "v3.0.0" in titles
and headers while plugin.json was at 3.1.1. Use _skill_version()
(reads from plugin.json at runtime) so version strings stay in sync.

Fixes #284
2026-05-16 23:35:26 -07:00
Trevin Chow 9f39d10bc5 Merge pull request #373 from hnshah/ren/watchlist-sightings
feat(store): record per-run finding sightings
2026-05-16 23:24:09 -07:00
Trevin Chow 03043da407 Merge pull request #418 from tmchow/chore/greptile-config
chore: add greptile.json (triggerOnUpdates + statusCheck)
2026-05-16 23:23:48 -07:00
Trevin Chow 8bab997854 chore: add greptile.json to opt into update-triggered reviews + status check
Without this config, Greptile's documented default is `triggerOnUpdates: false`
(only the initial PR open triggers a review). Empirically Greptile has been
re-reviewing on force-push to this repo anyway, but documenting the intent
makes the behavior reliable across plan changes and any future config-source
shifts on Greptile's side.

`statusCheck: true` registers Greptile as a GitHub status check (not just a
PR comment). That gives maintainer-tooling a machine-readable heartbeat -
poll `GET /repos/.../commits/SHA/check-runs` and filter by app name to see
whether Greptile is `queued` / `in_progress` / `completed`. Without it the
only signal is "did a new Greptile comment appear" which is silently
ambiguous when Greptile re-reviews and finds nothing new.

If `statusCheck` is OSS-plan-restricted Greptile silently ignores the key,
which is fine - the rolling-summary comment with `Confidence Score: N/5`
remains the fallback signal.

Refs greptileai/skills `greploop` skill for the terminal-state pattern this
config enables.
2026-05-16 23:20:24 -07:00
Hiten Shah 375fd0bcc0 fix(store): enforce sighting finding id invariant 2026-05-16 22:57:32 -07:00
Hiten Shah 92d65723e4 fix(watchlist): refresh sighting retries 2026-05-16 22:57:04 -07:00
Hiten Shah f794f82af5 feat(store): record per-run finding sightings 2026-05-16 22:57:04 -07:00
Trevin Chow 791c0a57a0 review: gate web Reddit enrichment behind EXCLUDE_SOURCES
PR #366 routes Reddit URLs found in web-search results through the public
Reddit JSON API to recover thread body + top comments (the Claude Code
WebFetch tool blocks reddit.com directly). That bypass is sound and the
fixed problem is real - but the always-on shape ignores user intent on
source gating.

A user who sets EXCLUDE_SOURCES=reddit to suppress Reddit results would
still get Reddit content smuggled back in via web-search URLs that
happen to point at reddit.com threads. This contradicts the suppression
contract that EXCLUDE_SOURCES is supposed to provide (see
lib/pipeline.available_sources where the same env var gates the
top-level Reddit source).

Add a _reddit_excluded(config) check in web_search() that mirrors the
parsing pattern from lib/pipeline (comma-separated, case-insensitive,
whitespace-tolerant). When reddit is in EXCLUDE_SOURCES, skip the
enrichment pass entirely - the web results themselves still flow
through, but they're not augmented with Reddit body/comments.

Four new tests in test_grounding_v3.py cover:
- EXCLUDE_SOURCES=reddit skips enrichment
- case-insensitive parsing matches REDDIT/Reddit/whitespace-padded/csv
- Other sources in EXCLUDE_SOURCES don't trigger the gate
- Enrichment runs normally when reddit isn't excluded

19/19 grounding tests pass.
2026-05-16 22:52:06 -07:00
Dave Morin 211df0deaa feat(web): auto-enrich Reddit URLs from web search via JSON API
Web search backends (Brave, Exa, Serper) can return Reddit URLs as
results. Claude Code's WebFetch blocks reddit.com, so the model can't
retrieve full thread content. After web search, detect Reddit URLs
and fetch body text + top comments via reddit.com/.json endpoint
using the skill's own HTTP library.

Fixes #324
2026-05-16 22:50:39 -07:00
Trevin Chow d7b3995da1 Merge pull request #357 from dinakars777/fix/windows-env-permission-warning
fix: skip POSIX secret warning on Windows
2026-05-16 22:42:26 -07:00
Trevin Chow e217db77cc Merge pull request #369 from voidborne-d/fix/scrapecreators-100-credits
docs: correct ScrapeCreators free tier to 100 credits (closes #367)
2026-05-16 22:41:16 -07:00
Dinakar Sarbada 8ea207b348 fix: skip POSIX secret warning on Windows 2026-05-16 22:39:35 -07:00
voidborne-d b04212680d docs: correct ScrapeCreators free tier to 100 credits (closes #367)
The skill advertises ScrapeCreators as offering "10,000 free API calls" in
six places. The actual free tier on the ScrapeCreators pricing page is
"100 credits free · No credit card required · Credits never expire" — a
100x overstatement that surprises users on signup.

Reporter (#367) burned through their full free allocation on a single
/last30days run after taking the 10,000-call claim at face value. They
verified the actual tier directly against scrapecreators.com plus an
independent review at fahimai.com.

Sweep:
- hooks/scripts/check-config.sh:110  (SessionStart hook tip line)
- README.md:228                      (Sources × Cost table row)
- HERMES_SETUP.md:62                 (Optional: ScrapeCreators bullet)
- skills/last30days/scripts/lib/ui.py:199  (PROMO_SINGLE_KEY["reddit"])
- skills/last30days/SKILL.md:1648    ("PAYG after 10,000 free API calls")
- skills/last30days/SKILL.md:1661    ("10,000 free API calls, then PAYG")

Wording defaults to the provider's own framing — "100 free credits" — and
keeps PAYG language where it was already explicit, since the paid step is
the part users were actually getting blindsided by.

CI gates: tests/test_plugin_contract.py (4) + tests/test_version_consistency.py (4)
all pass.  shellcheck clean.  No tests pin the "10,000" string.
2026-05-16 22:33:41 -07:00
Trevin Chow 2c2cfb9e7e Merge pull request #417 from tmchow/docs/eval-not-in-ci-solution
docs: capture eval-not-in-CI design decision under docs/solutions/
2026-05-16 22:31:39 -07:00
Trevin Chow 3276496f49 Merge pull request #376 from shoobee/feat/yt-dlp-ssh-routing
feat(youtube): route yt-dlp through SSH host for residential IP egress
2026-05-16 22:31:18 -07:00
Trevin Chow 68ae74ff4f docs: capture eval-not-in-CI design decision under docs/solutions/
Closes #374 (adapted, not 1:1 merged).

@hnshah opened PR #374 proposing a docs/adr/ directory for architecture
decision records. The intent is right -- the "why is search-quality eval
manual?" reasoning drifts out of memory if it isn't written down -- but
the docs/adr/ convention doesn't fit alongside the existing
docs/solutions/ structure (compound-engineering ce-compound pattern with
frontmatter metadata, additive entries, no membership-contract test).

This commit adopts hnshah's ADR 002 content (search-quality eval is
manual by default) as a docs/solutions/architecture/ entry with the
canonical compound-style frontmatter (module, problem_type, applies_when,
related_components, tags). Drops the docs/adr/ directory pattern, the
README index, and the test_adr_docs.py contract test.

ADR 001 (multi-surface packaging) is intentionally not adopted here: it
referenced sync.sh as the deploy mechanism, but sync.sh was removed in
PR #405 in favor of `npx skills add . -g -y`. The multi-surface
packaging story is still real but has moved beyond what the original
ADR captured; a fresh "how we ship to multiple harnesses" entry would
make sense as a separate doc.

Co-authored-by: hnshah <hnshah@users.noreply.github.com>
2026-05-16 22:22:59 -07:00
Trevin Chow 27c90504c0 review: validate SSH host alias + rename LAST30DAYS_YT_SSH_HOST -> LAST30DAYS_YOUTUBE_SSH_HOST
Addresses two concerns surfaced during PR #376 review:

1. **SSH option-injection on the host value.** The original PR uses
   shlex.quote() on the remote command and added a `--` option terminator
   in front of the host, but neither one stops a hostile env var like
   `LAST30DAYS_YT_SSH_HOST=-oProxyCommand=...` from being read in the
   first place. Tighten `_ytdlp_ssh_host()` to validate the host against
   `^[a-zA-Z0-9._-]+$` (plain hostname/SSH-config-alias shape: letters,
   digits, dot, underscore, hyphen). Any value that doesn't match logs a
   warning to stderr and returns None, so the wrap function falls back to
   local execution. The `--` terminator stays as defense-in-depth for the
   case where a valid host happens to start with `-`, but the regex closes
   the door on the env var reaching ssh at all.

2. **Env var naming consistency.** Existing skill-internal config knobs
   spell out their domain: `LAST30DAYS_X_BACKEND`, `LAST30DAYS_X_MODEL`,
   `LAST30DAYS_PLANNER_MODEL`, `LAST30DAYS_RERANK_MODEL`, etc. The module
   is `youtube_yt.py`, the source key is `youtube`, the function family
   is `is_youtube_*()` — `YT` was the odd abbreviation out. Rename to
   `LAST30DAYS_YOUTUBE_SSH_HOST` so the variable matches the user mental
   model ("route YouTube fetches via residential IP") and the codebase's
   spelled-out convention.

Adds three new tests:
- test_host_alias_with_dash_prefix_is_rejected (validator rejects `-o...`)
- test_host_alias_with_shell_metacharacters_is_rejected (rejects spaces, ;, $, `, &)
- test_host_alias_validator_accepts_realistic_aliases (allows FQDNs, IPs, bare aliases)

The existing test_wrap_cmd_uses_option_terminator is rewritten to use a
valid host value (since an invalid one is now filtered upstream) and
continues to assert the `--` terminator placement as defense-in-depth.

44/44 youtube_yt tests pass (40 prior + 4 net new validator tests).
2026-05-16 22:19:35 -07:00
shoobee f4eb0af104 fix(youtube): address Greptile review feedback
Three changes from automated review on PR #376:

1. Add `--` option terminator before host in _wrap_ytdlp_cmd (P1 security)
   Prevents SSH option injection if LAST30DAYS_YT_SSH_HOST were ever set
   to a value starting with `-` (e.g. `-oProxyCommand=...`). Low
   exploitability since the env var is user-controlled config — but the
   fix is a single arg and turns a self-harm footgun into no footgun.

2. Hoist `import shlex` to module-level (P2 style)
   Pure stdlib import, no reason for the deferred form. Cleaner.

3. Cache _ytdlp_ssh_host() result in fetch_transcript (P2 style)
   Was being called 2-3x per video; the function is cheap (env lookup
   + strip) so this is purely about readability.

Adds test_wrap_cmd_uses_option_terminator covering the security fix
explicitly with a `-oFoo=bar` host value. Updates index assertions in
the two existing tests that check command shape (host is now at index
4, command string at 5, with `--` at 3).
2026-05-16 22:17:30 -07:00
shoobee 79b5d049ce feat(youtube): route yt-dlp through SSH host for residential IP egress
Adds LAST30DAYS_YT_SSH_HOST env var (or `~/.config/last30days/.env` key).
When set, yt-dlp YouTube search invocations are wrapped as
`ssh <host> "yt-dlp ..."` so they run on a residential-IP machine.

Motivation: when last30days runs on a datacenter VPS (Hetzner,
DigitalOcean, AWS, etc.), `ytsearch:` queries return 0 results because
YouTube's bot-wall fingerprints datacenter IP ranges before any cookie
check runs. Cookies alone don't fix this — the IP reputation is checked
first. Verified across yt-dlp stable 2026.03.17 and nightly builds.

The existing fallbacks (browser cookies, residential proxy services,
excluding YouTube) all have downsides: cookies expire, proxies cost
money, exclusion loses signal. Many users with a Mac mini, Pi, or
home server can host yt-dlp on their own residential IP — this just
needs an SSH alias and a one-line env var to wire it up.

Behaviour:
- Default (env var unset): identical to before, no shape change.
- Env var set: search command list is wrapped with `ssh -o BatchMode=yes
  <host> "<shell-quoted yt-dlp invocation>"`. is_ytdlp_installed()
  returns True without a local PATH check (the binary lives on the
  remote host).
- Transcript path: when SSH-routing is on, skips the yt-dlp transcript
  path (which writes a VTT file we couldn't easily read back over SSH)
  and uses the existing _fetch_transcript_direct HTTP fallback. The
  timedtext API isn't bot-walled, so this works fine on datacenter IPs.

Setup pitfall documented in the function docstring: on macOS hosts
with Homebrew, `eval "$(/opt/homebrew/bin/brew shellenv zsh)"` must
live in ~/.zshenv (not just ~/.zprofile) — non-login SSH shells don't
source .zprofile, so without this `ssh macmini "yt-dlp ..."` returns
"command not found" while interactive SSH works fine.

Tests: 10 new cases covering env var read, whitespace stripping,
empty-value handling, command wrapping passthrough/active modes,
shlex quoting, is_ytdlp_installed short-circuit, and end-to-end
search_youtube wrapping. Full test suite: 0 new failures (the 14
pre-existing failures in test_store, test_watchlist, test_setup_openclaw,
test_safari_cookies, test_version_consistency are unchanged on main).

Verified live: 0 results → 4 real hits for "claude code" search from a
Hetzner VPS routed through a Mac mini exit node on Tailscale.
2026-05-16 22:17:29 -07:00
Trevin Chow 0e2059661a Merge pull request #378 from j-sperling/chore/gemini-3.1-flash-lite-ga
chore: migrate to gemini-3.1-flash-lite GA model
2026-05-16 22:12:20 -07:00
Trevin Chow b1c5f8db82 Merge pull request #382 from lustrousgorilla/bugfix/reddit-gaierror-retry
fix(http): expand retry budget + exponential backoff on DNS resolution failure
2026-05-16 22:11:56 -07:00
Trevin Chow 89c5cb9d5d Merge pull request #416 from tmchow/worktree-inherited-discovering-pebble
fix(ci): run full pytest suite, repair 13 rotted tests
2026-05-16 22:11:34 -07:00
Trevin Chow 5d4f9ef2c5 fix(store): use UTC for all date arithmetic against SQLite columns
Greptile flagged that _cli_query's --since parsing uses datetime.now()
(local time) while first_seen is stored via SQLite's datetime('now') (UTC).
The same bug exists in three other call sites that compare against either
first_seen or run_date (both UTC):

- get_daily_cost: "today" defaults to local date, returns wrong day's cost
  near UTC midnight
- get_stats: "7 days ago" cutoff for runs_7d / successful_7d
- get_trending: "N days ago" cutoff for finding activity ranking
- _cli_query: "N days ago" cutoff for --since flag (Greptile's flag)

All four now use datetime.now(timezone.utc). Same root cause and same fix
as the test_get_new_findings_filters_by_date repair in the previous commit.
2026-05-16 21:40:25 -07:00
Jeffrey Sperling 01262f78c6 chore: import GEMINI_FLASH_LITE in evaluate_search_quality
Address Greptile review nit. DEFAULT_JUDGE_MODEL now reuses the
constant from lib/providers.py instead of duplicating the literal,
so a future identifier change only needs one edit.
2026-05-16 21:40:22 -07:00
Jeffrey Sperling 96a4a78faa chore: migrate to gemini-3.1-flash-lite GA model
The Gemini 3.1 Flash Lite preview model is being discontinued on
May 25, 2026. Per Google's GA announcement, the underlying model
architecture is identical and only the model identifier needs to
be updated from `gemini-3.1-flash-lite-preview` to
`gemini-3.1-flash-lite`.

Also relaxes the `_require_gemini_31_preview` guard to accept any
`gemini-3.1-*` identifier (renamed to `_require_gemini_31`), so the
GA name and the still-preview `gemini-3.1-pro-preview` both pass.
2026-05-16 21:40:22 -07:00
Trevin Chow eb2d7a0f37 fix(ci): run full pytest suite, repair 13 rotted tests
CI was running only test_plugin_contract.py and test_version_consistency.py
(2 of 84 test files), masking 13 rotted tests across 4 clusters. The suite is
fully offline-safe (1402 tests in ~7s without network), so the narrow scope
wasn't gating integration flakiness; it was just stale. validate.yml now runs
`uv run pytest` against the full suite.

Engine fix: store.findings_from_report is rerank-first. ranked_candidates is
the primary persistence path; hackernews/polymarket are unconditionally
supplemented from items_by_source because they rank poorly but matter for
watchlists. When ranked_candidates was empty (rerank failed or skipped),
reddit, x, and every other source were silently dropped. The supplement loop
now falls back to all sources only when ranked_candidates is empty; the normal
path is unchanged.

Test repairs:
- test_store.py (6) + test_watchlist_commands.py (2): cascade from the engine fix
- test_get_new_findings_filters_by_date (latent): local-time vs SQLite UTC
  flake — switched to datetime.now(timezone.utc)
- TestPollDeviceAuth (3): mock_time.time side_effect lists too short after
  impl added a last_reminder call — padded timeout test, pinned others to
  return_value=0 (loops terminate via urlopen, not the clock)
- test_bare_run_emits_web_promo: engine reads ~/.config/last30days/.env, so
  a contributor's saved EXA/PARALLEL key made grounding "available" and
  suppressed the web promo. Also missing X made the "x" promo preempt "web".
  Set LAST30DAYS_CONFIG_DIR="", subprocess cwd=tmpdir, XAI_API_KEY stub.
2026-05-16 21:34:22 -07:00
Trevin Chow 719cdef2fb fix(http): contain DNS retry-budget widening to DNS path only
PR #382 introduced an `effective_retries` widening on the first gaierror,
but the widening leaked: every non-DNS error path (HTTPError, non-DNS
URLError, OSError) was gated on `effective_retries - 1` and so inherited
the expanded bound. A caller passing `retries=2` who hit DNS-then-non-DNS
got 3 attempts instead of 2 — contrary to the PR description and the
fail-fast intent of small retry budgets.

Fix:
- Gate every non-DNS sleep/retry decision on the caller's original
  `retries`, not the widened `effective_retries`.
- Add an explicit `break` in each non-DNS branch when the original
  budget is exhausted, so the (possibly widened) outer loop bound
  can't pull us into an extra attempt.

Adds two regression tests covering the DNS-then-non-DNS-URLError and
DNS-then-OSError sequences flagged in Greptile review on PR #382.
2026-05-16 21:13:17 -07:00
Trevin Chow ac04b56acc Merge pull request #383 from lustrousgorilla/bugfix/bird-x-json-decode-retry
fix(bird_x): retry subprocess on non-JSON stdout (HTML anti-bot interstitial)
2026-05-16 21:08:35 -07:00
Trevin Chow 42bfc6c76c Merge pull request #415 from tmchow/fix/sc-source-gating-consistency
fix(sources): align SC source gating between code and docs
2026-05-16 21:08:10 -07:00
Gabriel Arrillaga 5a2fe5279b fix(http): expand retry budget + use exponential backoff on DNS failure
Transient DNS resolution failures (socket.gaierror, surfaced as
urllib.error.URLError with reason=gaierror) were retried with the
generic URLError handler — linear backoff (2s, 4s, 6s) and bounded by
the caller-passed `retries` parameter. For callers that pass small
retry values (e.g. lib/reddit.py::_subreddit_search uses retries=2), a
single first-attempt DNS hiccup followed by one quick retry on the
still-flaky resolver would exhaust the retry budget and wipe a whole
subreddit sweep — which the caller's broad `except Exception` then
silent-empties as `[]`.

Fix:
- Distinguish URLError-with-gaierror-reason from generic URLError via
  a new `_is_dns_failure()` helper.
- For DNS failures, use exponential backoff (1s, 2s, 4s, ...) instead
  of the linear default.
- For DNS failures, expand the effective retry budget to at least
  MIN_DNS_RETRIES (=3) on first occurrence, so callers that passed
  `retries=2` still get a meaningful retry budget for the transient
  case. Non-DNS URLErrors and HTTPErrors keep the caller's value.
- DNS attempts are counted separately (`dns_attempts`) so unrelated
  URLError or OSError failures within the same call don't accidentally
  expand the budget further.

Reported during a community-signal pass where the Reddit subreddit
sweep silently returned zero items after a first-round transient DNS
hiccup. The fix lives at the http layer (where the retry loop is)
rather than per-source so every caller benefits.

Tests:
- Verifies a caller-passed retries=2 still gets MIN_DNS_RETRIES=3
  attempts on gaierror.
- Verifies gaierror-then-success returns successfully on attempt 2.
- Verifies the exponential-backoff sleep pattern (1s, 2s) on the
  retry attempts before exhaustion.
- Verifies a non-DNS URLError (ConnectionRefusedError reason) does
  NOT expand the retry budget — only true DNS failures do.

All 12 http tests pass (8 baseline + 4 new). No regressions in the
broader test suite (1373 pass / 14 fail, vs 1369 pass / 14 fail on
main — the 14 failures are pre-existing and unrelated to this PR).
2026-05-16 20:54:14 -07:00
Gabriel Arrillaga a717dd2b2c fix(bird_x): retry subprocess on non-JSON stdout (HTML interstitial)
Twitter's edge intermittently serves an HTML anti-bot interstitial in
place of JSON when the bird-search subprocess hits a per-query rate
limit. Before this fix, that response made json.loads raise
JSONDecodeError and _run_bird_search() returned {"error": ..., "items":
[]} with the parsed exception message — silent-empty against an
orchestrator that has no way to distinguish "Twitter served HTML; retry
likely succeeds" from "no tweets matched the query."

Surfaced during a community-signal pass where a Karpathy-LLM-wiki
subquery returned zero X items, while a second identical run a few
seconds later returned full results.

Fix:
- Extract the subprocess invocation into _invoke_bird_subprocess() so
  the retry loop can call it multiple times cleanly. Returns
  (result, terminal_error) — terminal_error is non-None for
  unrecoverable cases (subprocess timeout, spawn failure) that should
  NOT be retried.
- In _run_bird_search(), wrap the json.loads parse in a retry loop
  bounded by MAX_JSON_DECODE_RETRIES (=2) with JSON_DECODE_RETRY_DELAY
  (=5s) between attempts.
- On non-JSON stdout, log a diagnostic that names the shape
  (`looks_html`, first-80-chars stdout preview, attempt counter) so
  silent-empty failures become legible in logs.
- On retry exhaustion, return an error dict whose message explicitly
  names "anti-bot interstitial" as the likely cause, distinguishing
  this failure from a genuine no-results case.

Subprocess timeout, spawn failure, and non-zero return-code paths are
unchanged — those are terminal and don't retry.

Tests:
- Verifies HTML-then-JSON returns success on attempt 2.
- Verifies all-HTML returns the diagnostic error dict mentioning the
  anti-bot interstitial cause.
- Verifies subprocess timeout is NOT retried.

All 12 bird_x tests pass (9 baseline + 3 new).
2026-05-16 20:52:56 -07:00
Trevin Chow 9f08bb68b5 fix(sources): align SC source gating between code and docs
Two related drifts surfaced while reviewing PR #399 (EXCLUDE_SOURCES) —
docs claimed several SC-backed sources required INCLUDE_SOURCES opt-in
that the code didn't actually enforce, and threads was inconsistently
gated relative to its same-key siblings.

This commit picks the "code as source of truth + EXCLUDE_SOURCES as
suppression knob" model and aligns docs to match. It also promotes
threads to the same auto-on tier as tiktok and instagram, since all
three share the SC key and per-call cost shape — there was no real
product reason for threads being opt-in while the other two weren't.

The resulting source-gating model is three-tier and intentional:

  • **Auto-on if backing infra present** (suppress via EXCLUDE_SOURCES):
    reddit, HN, polymarket, X, youtube, github, bluesky, truthsocial,
    grounding, **tiktok, instagram, threads**

  • **INCLUDE_SOURCES persistent opt-in** (cost/billing reasons):
    perplexity (different paid API — OpenRouter),
    tiktok_comments / youtube_comments (N× extra SC calls per video)

  • **--search per-query opt-in** (relevance reasons):
    pinterest (visual pins, narrow utility),
    xiaohongshu (Chinese-market specific)

Changes:

- env.py: `is_threads_available()` drops the INCLUDE_SOURCES check,
  now mirrors tiktok/instagram (SC key → True). Docstring updated.
- tests/test_env_v3.py: new `ThreadsAvailabilityTests` class locks in
  the new contract and includes a regression guard ("INCLUDE_SOURCES
  should not be needed").
- SKILL.md: lines 333-338 rewritten so the model's "Build
  ACTIVE_SOURCES_LIST" checklist reflects what the engine actually
  runs. Drops false INCLUDE_SOURCES requirement for
  tiktok/instagram/threads; corrects pinterest to mention --search;
  adds missing INCLUDE_SOURCES=perplexity requirement.
- README: same alignment for the user-facing "Everything else in v3"
  section.

Note on EXCLUDE_SOURCES references in the new docs: the suppression
flag is wired up in PR #399. SKILL.md and README mention EXCLUDE_SOURCES
as the opt-out path; that prose is forward-looking until #399 lands.
The behavior changes in this PR (threads auto-on) are self-contained
and don't require #399 to function — but for users who want to suppress
the newly-auto-on threads source, #399 needs to land first.
2026-05-16 20:17:31 -07:00
Trevin Chow 602de1ebda Merge pull request #388 from bradferguson/fix/sc-youtube-and-hn-tokenization
fix(sources): unblock SC YouTube + multi-token HN searches
2026-05-16 19:41:12 -07:00
Trevin Chow bf3a82a87e Merge pull request #389 from kuyua9/fix/save-comparison-html-kuyua9
fix: save comparison HTML artifacts
2026-05-16 19:40:59 -07:00
Trevin Chow c010feb8f8 Merge pull request #399 from spiky02plateau/feat/exclude-sources-banner-and-pipeline
feat: honor EXCLUDE_SOURCES env var in source count + pipeline filter
2026-05-16 19:39:51 -07:00
Brad Ferguson edea402b7c fix(sources): unblock SC YouTube + multi-token HN searches
Two related fixes that surface when running last30days with multi-keyword
themed queries (e.g. "claude, personal agents, agentic infra"). Both bugs
caused entire sources to silently return zero items.

YouTube (ScrapeCreators)
  SC's /v1/youtube/search rejects ?keyword= with HTTP 400:
    {"error":"missing_parameter","message":"You must provide a query"}
  The canonical SC parameter for that endpoint is `query`. Other SC
  endpoints we use (Reddit, TikTok, Instagram) happened to work because
  they use their own per-endpoint parameter names — YouTube was the lone
  outlier.

Hacker News (Algolia)
  Multi-keyword theme queries returned zero hits across every theme.
  Algolia treats query= as strict AND across tokens, so a 4-5 word query
  like "claude, personal agents, agentic infra" matches no stories.

  Three changes in hackernews.py:

  1. Hoist comma/hyphen flattening into _flatten_query_for_algolia() so
     search_hackernews and _title_matches_query normalize the query the
     same way — addresses Greptile P2 #2 about the two callsites needing
     to stay in sync.
  2. Pass `optionalWords` for all-but-the-first token so Algolia ranks
     by token-overlap instead of requiring every token.
  3. Relax _title_matches_query from all-words to any-word, *but match
     on word boundaries (\b<word>\b) rather than naive substring* —
     addresses Greptile P2 #1, which flagged that the previous any-word
     relaxation would let "ai" falsely match "email" or "rail".

  Token-overlap relevance scoring at parse time already demotes weak
  matches, so word-boundary any-word matching is safe.

Tests: added coverage for no-token-in-title rejection, word-boundary vs
substring, and hyphen/comma flattening alignment between the search
parameter and the post-filter.

Co-authored-by: Trevin Chow <trevin@trevinchow.com>
2026-05-16 19:37:16 -07:00
Trevin Chow 4d4ac97ffb refactor: hoist comparison-html gate into a single condition (Greptile DRY)
Greptile flagged that `entity_reports and args.emit == "html"` appeared in
two places — once when computing the footer display path, again when calling
save_output. The else-branches differ between the two callsites (the display
needs `report.topic` as fallback; the save call needs `None` so save_output
falls back to the report's own topic), so collapsing into one shared
expression would be wrong, but hoisting just the condition into a single
`is_comparison_html` bool eliminates the risk of drift while keeping the
two callsites' fallback semantics distinct.
2026-05-16 19:32:49 -07:00
kuyua9 cd34966b4f fix: save comparison HTML artifacts 2026-05-16 19:32:06 -07:00
Trevin Chow 85255be350 Merge pull request #414 from mvanhorn/dependabot/uv/pytest-9.0.3
chore(deps-dev): bump pytest from 9.0.2 to 9.0.3
2026-05-16 19:31:46 -07:00
Trevin Chow 1aa120a420 Merge pull request #407 from DamienStevens/feat/macos-keychain-source
feat(env): macOS Keychain credential source
2026-05-16 19:31:34 -07:00
Trevin Chow 306d8c2d73 fix(env): wire EXCLUDE_SOURCES through get_config + SKILL.md integration
The original PR added EXCLUDE_SOURCES filtering to pipeline.available_sources()
and to the check-config.sh banner, but env.py::get_config() builds its config
dict from a hardcoded keys list that didn't include EXCLUDE_SOURCES. The
result: setting EXCLUDE_SOURCES in the environment silently no-op'd through
the Python pipeline. Only the bash hook (which reads shell env directly)
worked. The PR's unit tests didn't catch this because they construct config
dicts directly, bypassing get_config().

Changes:
- Add ('EXCLUDE_SOURCES', '') to env.py's keys list so the env var actually
  propagates into config.
- Add an end-to-end regression test that goes through get_config() rather
  than constructing config dicts directly.
- Document EXCLUDE_SOURCES in SKILL.md's source-list checklist so the model
  invoking the skill knows to subtract excluded sources before displaying
  the active-sources line. (Per AGENTS.md: engine flags without SKILL.md
  prose are incomplete — the agent invoking the skill won't know the flag
  exists.)
2026-05-16 19:30:27 -07:00
Trevin Chow d0dcf751f1 fix(keychain): single source of truth for key list + robust USER fallback
Addresses Greptile review on PR #407:

- P1: setup-keychain.sh ALL_KEYS was missing GOOGLE_GENAI_API_KEY and
  XIAOHONGSHU_API_BASE relative to _load_keychain's inline list, so users
  manually storing those keys would not see them in --list and the
  interactive prompt would never offer to set them.

  Hoist the canonical key list into lib/env.py::KEYCHAIN_KEYS, have
  get_config() pass it through, and add a parity test that parses
  ALL_KEYS out of setup-keychain.sh and asserts equality. Drift is now
  caught at CI time instead of after a user reports a missing key.

- P2: os.environ.get("USER", "") silently returned "" under sudo, in
  Docker without --env USER, or in CI runners that strip USER. The
  resulting `security find-generic-password -a ""` call would never
  match items stored by setup-keychain.sh, so all lookups silently
  returned nothing. Fall back to pwd.getpwuid(os.getuid()).pw_name when
  USER is absent.

The P2 process-listing comment ("secret visible briefly via ps because
security has no stdin path for -w") has no clean fix — the README
already documents the manual `security add-generic-password` invocation
as an alternative for users with strict secret hygiene.
2026-05-16 19:25:05 -07:00
dependabot[bot] afd4b04d6d chore(deps-dev): bump pytest from 9.0.2 to 9.0.3
Bumps [pytest](https://github.com/pytest-dev/pytest) from 9.0.2 to 9.0.3.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.2...9.0.3)

---
updated-dependencies:
- dependency-name: pytest
  dependency-version: 9.0.3
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-17 02:22:02 +00:00
Trevin Chow 14d8f62e02 Merge pull request #413 from tmchow/docs/compound-release-cascade-pattern
docs: compound learning on release-time consistency-test cascade failures
2026-05-16 19:12:31 -07:00
Trevin Chow 0fd532d249 docs: compound learning on release-time consistency-test cascade failures
Documents the cascade pattern surfaced during this session's install-modernization
arc: a `test_sync_cache_path_uses_skill_version` test asserted that a hardcoded
version pin in `sync.sh` matched the version frontmatter in SKILL.md. When a
release bumped SKILL.md, every open PR's CI failed simultaneously on the
unrelated stale-pin assertion. Affected at least 5 PRs across the 2026-05-13
to 2026-05-15 window (#400, #390, #392, and two others) plus required hotfix
PR #397 to unblock the queue.

The permanent fix shipped in PR #405 (deleted sync.sh + the test). This doc
captures the design lesson so the pattern doesn't reappear: don't write
consistency tests that read two files and assert one matches a value derived
from the other. Either derive at runtime from a single source of truth, or
self-skip / merge-base-scope the test so deletion is a non-event.

Created via /ce-compound. Includes:

- docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md
  (the new learning — first entry under docs/solutions/)
- CONCEPTS.md (new — 4 entries: Skill, Engine, Harness, Beta channel,
  capturing project-specific vocabulary that surfaced across the session)
- AGENTS.md (added one-line Structure entries surfacing docs/solutions/ and
  CONCEPTS.md so fresh agents discover them)
- docs/plans/2026-04-22-{002,003,005,006}-*-plan.md (added deprecation banner
  to each, pointing readers at PR #405 and the new docs/solutions entry —
  these 4 historical plans still reference the deleted sync.sh inline)

Also: closed PR #379 (j-sperling's workaround for the same cascade,
superseded by PR #405's permanent fix).
2026-05-16 19:08:02 -07:00
Trevin Chow 8867a007ea Merge pull request #392 from Gujiassh/fix/openclaw-scrapecreators-optional-env
fix(openclaw): make ScrapeCreators key optional
2026-05-16 18:50:07 -07:00
gujishh 8af8f06b06 fix(openclaw): make ScrapeCreators key optional 2026-05-16 18:48:29 -07:00
Trevin Chow 01b5f3dc1e Merge pull request #363 from thinkun/pr/claim-contributor-entry
Claim contributor entry — @thinkun
2026-05-16 18:46:35 -07:00
Trevin Chow 2e39ee8ce4 Merge pull request #412 from tmchow/refactor/skill-meta-version-helper
refactor: consolidate SKILL.md version regex into lib/skill_meta.py
2026-05-16 18:42:49 -07:00
Trevin Chow 37033164da Merge pull request #410 from tmchow/docs/agents-orientation-multi-harness
docs: reframe as multi-harness Agent Skills package, flip CLAUDE.md ↔ AGENTS.md
2026-05-16 18:42:36 -07:00
Trevin Chow 9fe4b8f130 Update AGENTS.md
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-05-16 18:42:14 -07:00
Trevin Chow 73dc6b9996 refactor: consolidate SKILL.md version regex into lib/skill_meta.py
The same `^version:\s*"([^"]+)"\s*$` regex (or a slight variant) was
duplicated across three files: render.py inline, test_plugin_contract.py
local helper, test_version_consistency.py local helper. A future change
to the SKILL.md frontmatter version format would have needed to update
three places without any compile-time pressure to keep them in sync.

New skills/last30days/scripts/lib/skill_meta.py provides:
- `_VERSION_RE` private compiled pattern (accepts double-quoted,
  single-quoted, or unquoted YAML version scalars per the widening
  landed in 997708a)
- `read_skill_version(skill_md_path: Path) -> str | None` helper that
  catches OSError + UnicodeDecodeError and returns None on miss

Callers updated:
- render.py::_skill_version now calls skill_meta.read_skill_version
  inside the SKILL.md fallback loop, returning `read_skill_version(...) or "?"`.
  Semantically equivalent to the old break-after-first-SKILL.md logic.
- test_plugin_contract.py and test_version_consistency.py import the
  helper instead of defining the regex inline. Both files use the
  established sys.path.insert pattern.

Added tests/test_skill_meta.py with 6 direct unit tests covering the
helper's full contract: missing file, undecodable bytes, no-version-line,
and all three quoting styles (double, single, unquoted). Previously the
helper was only exercised transitively through render._skill_version().

Added test_skill_md_uses_double_quoted_version to
test_version_consistency.py — the old per-test regex incidentally
asserted "this repo's SKILL.md uses double-quotes" by being strict;
the shared helper accepts all three styles, so the assertion is now
explicit instead of implicit.

Code-reviewed by ce-code-review (8 reviewers); safe_auto fixes applied
inline (rename to _VERSION_RE, group or-chain instead of generator,
docstring tightened, dropped unnecessary `from __future__ import
annotations`, tightened signature to Path-only).

Conftest.py refactor for the sys.path.insert duplication across ~20 test
files filed as issue #411 — out of scope for this PR (touches many
files, separate concern).

Test results: 23 passed in the affected test set (16 prior + 6 new
test_skill_meta tests + 1 new double-quote assertion). Full suite shows
same 13 pre-existing failures as main; zero new failures.
2026-05-16 18:17:33 -07:00
Trevin Chow 1fd763e09f docs: flip CLAUDE.md ↔ AGENTS.md — AGENTS.md becomes canonical, CLAUDE.md points at it
Mirrors the multi-harness reframing of the project itself. CLAUDE.md is
Claude-Code-specific by name; AGENTS.md is the multi-harness convention
that Codex, Cursor, Gemini CLI, GitHub Copilot, and most other Agent
Skills hosts also read. The canonical content belongs in the file
multi-harness tooling expects.

git mv preserves history — the Orientation section and everything else
that was in CLAUDE.md is now tracked under AGENTS.md, with full blame
continuity. The new CLAUDE.md is a one-line `@AGENTS.md` reference so
Claude Code continues to load the content (it follows @ references).
2026-05-16 17:03:11 -07:00
Trevin Chow e0f6ef845a docs(claude.md): add Orientation section, reframe as multi-harness Agent Skills package
Closes the spirit of #335 (closed in favor of this fresh PR after the
sync.sh thread became obsolete via PR #405).

Two changes:

1. Top-of-file description reframed from "Claude Code skill" to
   "Agent Skills package... installable across Claude Code (most common
   host), Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ other
   Agent Skills hosts". The skill works across every major agent host
   after the install-modernization work in PR #400/#404/#405/#409.
   Calling it "Claude Code skill" undersells the surface and biases
   contributors toward Claude-Code-specific assumptions.

2. New ## Orientation section (4 bullets) framing the project for
   contributors who would otherwise read the python3 invocation in
   ## Commands and form a CLI-first mental model. Names the trap
   explicitly with one concrete invalid-syntax example
   (`/last30days OpenClaw --emit=html | pbcopy` — slash commands don't
   pass shell mechanics through). Bullets adapted from #335 with
   multi-harness framing replacing the Claude-Code-only framing.

No code changes. No SKILL.md changes. CLAUDE.md only. AGENTS.md
inherits via @CLAUDE.md.
2026-05-16 17:01:04 -07:00
Trevin Chow c918e18465 Merge pull request #409 from tmchow/refactor/skill-dir-relative-resolver
refactor(skill): replace SKILL_ROOT resolver with SKILL_DIR substitution
2026-05-16 16:55:32 -07:00
Trevin Chow 6fe0aca7ee refactor(skill): replace SKILL_ROOT resolver with SKILL_DIR substitution
The Step 1 and comparison-mode resolver loops walked a hardcoded list
of install paths trying to find scripts/last30days.py. Two problems:

1. The list was never exhaustive — it covered ~/.codex/skills, but not
   ~/.claude/skills, ~/.cursor/skills, ~/.gemini/skills, ~/.copilot/skills,
   ~/.hermes/skills/research, etc. PR #406 was about to fix that by
   enumerating more paths, but enumeration is the wrong shape.
2. The resolver could pick a different install than the SKILL.md the
   model loaded from. Spec-vs-engine divergence is subtle and confusing
   when it triggers.

The model already knows the SKILL.md path it loaded (from its Read tool
result). Templating that into the bash block is strictly better than
guessing across an enumerated list:

- Works for every harness without enumeration (Hermes, Cursor, anything
  new) because we just use wherever the harness loaded SKILL.md from
- Aligns spec with engine — the engine runs from the same install the
  spec was read from
- Deletes ~80 lines of bash across Step 1 + comparison-mode + the
  prose preamble describing the resolver

Mechanics:

- SKILL_DIR placeholder in both bash blocks — model substitutes the
  absolute path of the directory containing the SKILL.md it just Read
- One-line validation `[ ! -f "$SKILL_DIR/scripts/last30days.py" ]`
  catches bad templating with a clear error
- All references to $SKILL_ROOT replaced with $SKILL_DIR (badge
  VERSION lookup, prose description in the LAW-7 preamble area)
- STEP 0 unchanged — different concern (marketplaces stale-clone hop)

Version 3.2.3 -> 3.2.4 (behavior change: install paths the resolver
list never enumerated now work; install paths it did enumerate work
the same way they used to but via the SKILL_DIR template).
2026-05-16 16:26:45 -07:00
Damien Stevens 74a387b093 feat(env): macOS Keychain credential source
Adds the macOS Keychain as the lowest-priority credential source on Darwin.
Items stored as generic passwords with service name "last30days-<KEY>" for
the current user are picked up automatically by get_config() — file env
and process env still win on collision.

No new config knob: behavior is strictly additive. On non-Darwin (or when
the `security` binary is missing) the loader is a no-op, so Linux/Windows
behavior is unchanged.

  Priority (highest wins):
    1. Environment variables
    2. .claude/last30days.env (per-project)
    3. ~/.config/last30days/.env (global)
    4. macOS Keychain items prefixed last30days- (new)

Includes:
  - lib/env.py: KEYCHAIN_SERVICE_PREFIX constant, _load_keychain helper
    (platform-gated, shutil.which-gated, subprocess-error tolerant),
    wiring into get_config before get_openai_auth so OPENAI_API_KEY can
    come from Keychain too, _CONFIG_SOURCE reports "keychain" when no
    file source is present.
  - scripts/setup-keychain.sh: bash helper with interactive set,
    --list, --delete, --replace modes. Uses `security add-generic-password`.
  - tests/test_env_keychain.py: 12 tests covering platform gate,
    missing-binary gate, success path, whitespace stripping, subprocess
    errors swallowed, get_config precedence, and an OPENAI_AUTH wiring
    regression test.
  - tests/test_env_cookies.py: existing integration test mocks the new
    _load_keychain hook so it stays hermetic on Darwin developer
    machines that have real keychain entries.
  - README.md: new "macOS Keychain (optional)" subsection under
    "Bring your own keys" documenting setup-keychain.sh and the manual
    `security add-generic-password` invocation.

Tested on macOS with a populated keychain and against the existing pytest
suite — CI-tracked tests (test_plugin_contract.py, test_version_consistency.py)
plus all env-touching tests pass. Pre-existing unrelated failures in
test_store.py / test_watchlist_commands.py / test_setup_openclaw.py /
test_footer_nudge_suppression.py are untouched.
2026-05-16 09:01:28 -04:00
Tobi 095bcae915 fix(check-config): normalize EXCLUDE_SOURCES (lowercase + whitespace) before matching
The bash banner accounting used raw substring matching while
pipeline.py normalises EXCLUDE_SOURCES via .strip().lower(). With
EXCLUDE_SOURCES=TikTok,Instagram (or with surrounding spaces),
pipeline correctly excludes the sources but the banner did not
deduct them — count showed 1-2 higher than what the pipeline
actually runs. Normalisation now mirrors the Python side
(lowercase, collapse whitespace around commas, strip outer whitespace).

Reproducer (clean HOME with config EXCLUDE_SOURCES=TikTok,Instagram):
  before: /last30days: Ready — 7 sources active.
  after:  /last30days: Ready — 5 sources active.

Addresses Greptile review comment P1 on #399.
2026-05-16 08:05:34 +02:00
Tobi 4f6b86c456 feat: honor EXCLUDE_SOURCES env var in source count + pipeline filter
Adds a per-run denylist via the existing-but-unused EXCLUDE_SOURCES
config key. Two coupled changes:

1. pipeline.available_sources() filters out any source listed in
   config["EXCLUDE_SOURCES"] (comma-separated, case-insensitive,
   whitespace-tolerant) before returning.
2. hooks/scripts/check-config.sh "Ready — N sources active" banner
   subtracts excluded sources from the ScrapeCreators +3 (Reddit
   comments + TikTok + Instagram) so the count matches what the
   pipeline actually runs.

Use case: skip TikTok/Instagram on runs where you only want
text-substantive sources, without unsetting SCRAPECREATORS_API_KEY
(which would also kill Reddit comments). The existing INCLUDE_SOURCES
allowlist covers Perplexity opt-in but doesn't cover this denylist case
— tiktok and instagram are added unconditionally when
SCRAPECREATORS_API_KEY is set, with no opt-out short of removing the key.

Tests (tests/test_pipeline_v3.py::TestExcludeSources):
- excludes tiktok+instagram when listed
- no exclusion when env unset or empty string
- case-insensitive + whitespace-tolerant parsing
- works for any source (e.g. EXCLUDE_SOURCES=hackernews), not just SC-backed
2026-05-16 00:54:18 +02:00
Anurag Chakradhar ed455ca036 Claim contributor entry — @thinkun 2026-05-08 17:15:08 +10:00
93 changed files with 4992 additions and 601 deletions
+1 -1
View File
@@ -11,7 +11,7 @@
{
"name": "last30days",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"version": "3.2.3",
"version": "3.3.0",
"author": {
"name": "Matt Van Horn",
"url": "https://github.com/mvanhorn"
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "last30days",
"version": "3.2.3",
"version": "3.3.0",
"description": "Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources. AI agent scores by upvotes, likes, and real money - not editors.",
"author": {
"name": "Matt Van Horn",
+1 -1
View File
@@ -16,7 +16,7 @@ body:
label: Steps to Reproduce
description: How can we reproduce this?
placeholder: |
1. Run `python3 scripts/last30days.py "topic" --emit compact`
1. Run `python3 skills/last30days/scripts/last30days.py "topic" --emit=compact`
2. ...
validations:
required: true
+67
View File
@@ -0,0 +1,67 @@
name: Security
on:
pull_request:
push:
branches:
- main
workflow_dispatch:
permissions:
contents: read
jobs:
dependency-audit:
name: Dependency audit
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v5
- name: Set up Python
run: uv python install 3.12
- name: Export locked dependency set
run: |
uv export \
--locked \
--all-groups \
--no-hashes \
--format requirements.txt \
--output-file /tmp/last30days-requirements.txt
# Advisory-first: visibility before enforcement. This repo handles API keys,
# cookies, browser tokens, and local env files, so dependency CVEs should be
# visible in CI logs even before the project has a clean blocking baseline.
# Set continue-on-error: false once a clean baseline run is confirmed.
- name: Run pip-audit against locked dependencies
continue-on-error: true
run: uvx --python 3.12 pip-audit -r /tmp/last30days-requirements.txt --progress-spinner=off
secret-scan:
name: Secret scan
runs-on: ubuntu-latest
steps:
- name: Checkout full history for diff-aware scanning
uses: actions/checkout@v4
with:
fetch-depth: 0
# Advisory-first: this reports verified secrets in pull requests and pushes to
# main, but does not block merges until maintainers confirm a clean baseline.
# The TruffleHog action automatically scans the PR range for pull_request
# events and the pushed commit range for push events.
# Set continue-on-error: false once a clean baseline run is confirmed.
# Contributor policy: never commit real secrets in fixtures, tests, docs, or
# examples; use obvious dummy values and env-based auth patterns instead.
- name: Run TruffleHog OSS secret scan
if: github.event_name == 'pull_request' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
uses: trufflesecurity/trufflehog@v3.95.2
continue-on-error: true
with:
path: ./
version: v3.95.2
extra_args: --only-verified
+3 -3
View File
@@ -10,7 +10,7 @@ permissions:
contents: read
jobs:
plugin-contract:
tests:
runs-on: ubuntu-latest
steps:
- name: Checkout
@@ -22,5 +22,5 @@ jobs:
- name: Set up Python
run: uv python install 3.12
- name: Run plugin contract tests
run: uv run pytest tests/test_plugin_contract.py tests/test_version_consistency.py
- name: Run test suite
run: uv run pytest
+4
View File
@@ -28,3 +28,7 @@ htmlcov/
# Internal planning docs (ce:plan output) — keep local, don't publish
docs/plans/
.context/
/work
/print
+54 -1
View File
@@ -1 +1,54 @@
@CLAUDE.md
# last30days Skill
Agent Skills package for researching any topic across Reddit, X, YouTube, and web. Installable across Claude Code (most common host), Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ other [Agent Skills](https://agentskills.io) hosts. Python scripts with multi-source search aggregation.
## Structure
- `skills/last30days/SKILL.md` — canonical skill definition
- `skills/last30days/scripts/last30days.py` — main research engine
- `skills/last30days/scripts/lib/` — search, enrichment, rendering modules
- `skills/last30days/scripts/lib/vendor/bird-search/` — vendored X search client
- `docs/solutions/` — documented solutions to past problems (bugs, best practices, workflow patterns), organized by category with YAML frontmatter (`module`, `tags`, `problem_type`)
- `CONCEPTS.md` — shared domain vocabulary (Skill, Engine, Harness, Beta channel) — relevant when orienting to the codebase or discussing project terminology
## Orientation
- This is an Agent Skills package, not a CLI tool. The product is the slash-command-invoked skill (`/last30days <topic>` in most harnesses); `scripts/last30days.py` is implementation. Claude Code is the most common host but not the only one — features must work across every harness the skill installs into.
- Feature design starts from the slash-command UX. A new engine flag with no SKILL.md integration is incomplete — the model invoking the skill won't know the flag exists.
- README and PR examples show `/last30days <topic>` first. Direct CLI invocation (`python3 scripts/last30days.py ...`) is a fallback for scripting, cron, and dev-time engine testing; label it as such, never as the primary path.
- Slash commands don't pass shell mechanics through. `/last30days OpenClaw --emit=html | pbcopy` is invalid in any harness — either use the slash form (no flags or pipes; let the model translate user intent into engine flags) or use the direct CLI form (full `python3 ...` with explicit flags and a real shell).
## Commands
```bash
# Dev/fallback: direct engine invocation (scripting, cron, or engine testing only)
python3 skills/last30days/scripts/last30days.py "test query" --emit=compact
npx skills add . -g -y # one-time: symlink this repo into every detected harness's skill dir
## Rules
- `lib/__init__.py` must be bare package marker (comment only, NO eager imports)
- One-time setup: `npx skills add . -g -y` creates symlinks from each detected harness's skill dir to this repo. Edits in the working tree propagate live to every harness — no re-deploy step needed.
- Git remote: origin = public (`mvanhorn/last30days-skill`)
## Security hygiene
- Never commit real API keys, browser cookies, auth tokens, app passwords, access tokens, or `.env` contents.
- Use the env-based auth patterns in `skills/last30days/scripts/lib/env.py`; tests and fixtures must use obvious dummy values only.
- Keep examples safe by redacting secrets and avoiding copy/pasteable live credentials in docs, fixtures, and test data.
- Do not weaken or disable the advisory security workflow (`.github/workflows/security.yml`) without explaining why in the PR description or review thread.
## Maintaining CONFIGURATION.md
`CONFIGURATION.md` is the user-facing configuration reference — save paths, per-source API keys, web-search backend priority, trend-monitoring stack, per-client install patterns. Distinct from `SKILL.md` (the canonical runtime spec).
Update `CONFIGURATION.md` when:
- adding a new env var (e.g. `LAST30DAYS_*`, `BSKY_*`, `*_API_KEY`)
- adding a new CLI flag that affects configuration (e.g. `--store`, `--web-backend`)
- adding a new per-client install pattern (Claude Code, Gemini, Codex, Cursor, Hermes…)
- adding a new optional source that requires its own credential
- changing the priority order of config layers (per-run flag > env > `.env` file > defaults)
Keep the existing structure organized by how often each layer is touched: per-run flags → env vars / `.env` → optional trend-monitoring stack → per-client patterns. Add new content into the right section rather than appending at the end.
When a new config concept lands in `SKILL.md` or `AGENTS.md`, mirror the user-facing knob in `CONFIGURATION.md` so non-agent readers can configure the skill without reverse-engineering it from the runtime spec.
## Beta channel
Experimental changes get tested on `mvanhorn/last30days-skill-private`, which installs as a parallel `/last30days-beta` slash command. Beta-only changes never ship to public without a review PR here. Workflow guide lives at `BETA.md` in the private repo. Plan that established this setup: `docs/plans/2026-04-17-005-feat-beta-skill-from-private-repo-plan.md`.
+141 -7
View File
@@ -7,18 +7,152 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [3.3.0] - 2026-05-17
A week-long shipping cycle: ~75 PRs merged plus 7 community fixes salvaged through PR triage. Big themes: install story modernized for the multi-harness world (Claude Code, Codex, Cursor, Gemini CLI, Copilot, Windsurf, and 50+ Agent Skills hosts), new emit and source modes, and a substantial reliability sweep across Reddit, X, Windows, YouTube, and the planner.
### Added
**Emit modes and sources**
- `--emit=html` for shareable, print-friendly HTML research briefs ([#332](https://github.com/mvanhorn/last30days-skill/pull/332)).
- **Digg AI 1000 source**, auto-enabled when `digg-pp-cli` is on PATH ([#370](https://github.com/mvanhorn/last30days-skill/pull/370)). Surfaces curated story clusters from the AI 1000 leaderboard and pulls attributable X-post quotes into the brief.
**Configuration knobs**
- `EXCLUDE_SOURCES` env var — the inverse of `INCLUDE_SOURCES`, honored in source count and pipeline filter ([#399](https://github.com/mvanhorn/last30days-skill/pull/399)).
- `LAST30DAYS_YOUTUBE_SSH_HOST` — opt-in SSH routing for `yt-dlp` through a residential-IP host, for users on datacenter VPS hit by YouTube's bot-wall ([#376](https://github.com/mvanhorn/last30days-skill/pull/376)). Host validated against `^[a-zA-Z0-9._-]+$` to reject SSH option-injection. Transcript path unchanged (uses HTTP fallback).
- macOS Keychain as a credential source — reads from the system keychain when env vars and config files aren't set ([#407](https://github.com/mvanhorn/last30days-skill/pull/407)).
- Configuration enablement: env-var defaults and source-resilience patterns across the config layer ([#344](https://github.com/mvanhorn/last30days-skill/pull/344)).
**Pipeline and storage**
- Reddit URL auto-enrichment from web search via the public JSON API ([#366](https://github.com/mvanhorn/last30days-skill/pull/366)).
- Per-run finding sightings recorded in the SQLite store ([#373](https://github.com/mvanhorn/last30days-skill/pull/373)).
- Brave browser support for X/Twitter cookie extraction ([#320](https://github.com/mvanhorn/last30days-skill/pull/320)).
**Tests and CI**
- Full pytest suite restored to CI; 13 rotted tests repaired ([#416](https://github.com/mvanhorn/last30days-skill/pull/416)).
- `greptile.json` added with `triggerOnUpdates` + `statusCheck` ([#418](https://github.com/mvanhorn/last30days-skill/pull/418)).
- Advisory security workflow ([#368](https://github.com/mvanhorn/last30days-skill/pull/368)).
- Parallel grounding backend test coverage ([#355](https://github.com/mvanhorn/last30days-skill/pull/355)).
**Docs**
- New `CONFIGURATION.md` with README pointers ([#339](https://github.com/mvanhorn/last30days-skill/pull/339)).
- `docs/solutions/` learning capture for release-time consistency-test cascades ([#413](https://github.com/mvanhorn/last30days-skill/pull/413)) and the eval-not-in-CI design decision ([#417](https://github.com/mvanhorn/last30days-skill/pull/417)).
### Changed
- Rename "Digg AI 1000" to just "Digg" in user-facing output (footer line, source label, inline-quote suffix, why_relevant, container attribution). Internal references to the upstream Digg AI 1000 product remain in code comments and docstrings.
- Bump `POSTS_PER_CLUSTER` from 3 to 5 and the render-side display limit from 2 to 3 to match the per-source enrichment caps used by Reddit, HN, YouTube, TikTok, and GitHub. The previous 3/2 caps routinely truncated cluster context (e.g. dropped a Jason Calacanis quote tweet on a `cli-printing-press` run).
- Rewrite SKILL.md path resolution. STEP 0 narrows from a global canonical-path enforcement to a Claude-Code-marketplaces-only stale-clone guard. Step 1 SKILL_ROOT resolver walks a single precedence list (Claude plugin cache, then `~/.codex/skills/`, `~/.agents/skills/`, repo checkout, `./.skills/last30days` for `npx skills add`, CWD, Gemini). Adds SKILL.md frontmatter fallback to `render.py::_skill_version` so the badge no longer prints `v?` on installs that don't include `.claude-plugin/plugin.json`.
**Install story modernized**
- Switch SKILL.md's `--plan` and `--competitors-plan` invocation templates from inline single-quoted JSON to heredoc-written tmpfiles. Apostrophes in resolved context strings ("McDonald's", "people's choice", "developer's") previously closed the outer single-quote and broke shell parsing before the engine started — observed in a Codex run during PR #400 testing. The engine's `parse_plan()` / `parse_competitors_plan()` already supported file paths (via `os.path.isfile()` probe); only the template prose changed. Fixes [#403](https://github.com/mvanhorn/last30days-skill/issues/403).
- `npx skills add` is now the canonical install path for every harness ([#405](https://github.com/mvanhorn/last30days-skill/pull/405)). README and SKILL.md flipped to recommend `npx skills add . -g -y` over per-harness manual instructions. Surfaces Gemini CLI, Copilot, Windsurf, and 50+ other Agent Skills hosts that the install pattern reaches.
- README dropped the Gemini CLI native-extension install path (now covered by `npx skills add`).
- `hooks.json` made polyglot for Gemini CLI + Claude Code compatibility ([#318](https://github.com/mvanhorn/last30days-skill/pull/318)).
**Skill semantics and multi-harness reframe**
- `AGENTS.md` is now canonical; `CLAUDE.md` points at it ([#410](https://github.com/mvanhorn/last30days-skill/pull/410)). Reframes the project as a multi-harness Agent Skills package rather than a Claude-Code-specific tool.
- SKILL.md path resolution rewritten: STEP 0 narrows to a Claude-Code-marketplaces-only stale-clone guard; Step 1 walks a single `SKILL_DIR` substitution pattern ([#400](https://github.com/mvanhorn/last30days-skill/pull/400), [#409](https://github.com/mvanhorn/last30days-skill/pull/409)). Removes ~80 lines of bash and fixes a real spec-vs-engine divergence where the previous resolver could pick a different install than the SKILL.md the model loaded from.
- SKILL.md version regex consolidated into `lib/skill_meta.py` ([#412](https://github.com/mvanhorn/last30days-skill/pull/412)).
- `--plan` / `--competitors-plan` invocation templates switched from inline single-quoted JSON to heredoc-written tmpfiles ([#404](https://github.com/mvanhorn/last30days-skill/pull/404), fixes [#403](https://github.com/mvanhorn/last30days-skill/issues/403)). Apostrophes in resolved context strings ("McDonald's", "people's choice") no longer break shell parsing.
- `POSTS_PER_CLUSTER` raised 3→5 and render-side display limit 2→3 to match the per-source enrichment caps used by Reddit, HN, YouTube, TikTok, and GitHub. The previous caps routinely truncated cluster context.
- Digg AI 1000 renamed to "Digg" in user-facing output ([#372](https://github.com/mvanhorn/last30days-skill/pull/372)) — footer line, source label, inline-quote suffix, why_relevant, container attribution. Internal references retain the upstream product name.
- GitHub repo resolution canonicalized for ambiguous product comparisons ([#302](https://github.com/mvanhorn/last30days-skill/pull/302)).
**Dependencies and tooling**
- Dropped `requests` runtime dependency. All providers route through stdlib `urllib` via the `lib/http` wrapper ([#393](https://github.com/mvanhorn/last30days-skill/pull/393)).
- Migrated to `gemini-3.1-flash-lite` GA model ([#378](https://github.com/mvanhorn/last30days-skill/pull/378)).
- Aligned Codex/Claude plugin manifests + added Codex `AGENTS.md` ([#321](https://github.com/mvanhorn/last30days-skill/pull/321)).
- pytest dev dep bumped 9.0.2 → 9.0.3 ([#414](https://github.com/mvanhorn/last30days-skill/pull/414)).
### Removed
- **BREAKING for Codex native-plugin users:** `.codex-plugin/plugin.json` and the matching SKILL_ROOT resolver branch in SKILL.md Step 1. Codex users should install via `npx skills add mvanhorn/last30days-skill` or copy the skill to `~/.codex/skills/last30days/`.
- **`skills/last30days/scripts/sync.sh`.** The maintainer dev-deploy script is gone. Every job it did has a better replacement: `npx skills add . -g -y` symlinks the working tree into every detected harness's skill dir (better than sync.sh's copy model edits propagate live), `hermes skills install mvanhorn/last30days-skill --force` handles Hermes, `clawhub install last30days-official` handles OpenClaw, and the Claude marketplace cache target was a "test against the official install path" hack we shouldn't have been recommending in the first place. The `test_sync_cache_path_uses_skill_version` test was dropped along with it. CLAUDE.md, HERMES_SETUP.md, the PR template, and a render.py docstring were updated to drop references; CHANGELOG and historical docs (release notes, plan files) keep their existing mentions as accurate history.
- **BREAKING for Codex native-plugin users:** `.codex-plugin/plugin.json` and the matching SKILL_ROOT resolver branch in SKILL.md Step 1 ([#400](https://github.com/mvanhorn/last30days-skill/pull/400)). Codex users should install via `npx skills add mvanhorn/last30days-skill` or copy the skill to `~/.codex/skills/last30days/`.
- **`skills/last30days/scripts/sync.sh`** maintainer dev-deploy script ([#405](https://github.com/mvanhorn/last30days-skill/pull/405)). Replaced by `npx skills add . -g -y` (live-symlink into every detected harness's skill dir better than sync.sh's copy model since edits propagate live). Hermes uses `hermes skills install mvanhorn/last30days-skill --force`; OpenClaw uses `clawhub install last30days-official`.
- Orphaned `SPEC.md` and `TASKS.md` ([#419](https://github.com/mvanhorn/last30days-skill/pull/419)).
### Fixed
**Reddit**
- `lstrip("r/")` mangled subreddits starting with `r` (`r/robotics``obotics`, `r/ruby``uby`); replaced with `removeprefix("r/")` at 4 sites (Alex Key, salvaged from #288).
- Browser-like User-Agent + `Accept-Language`/`Accept-Encoding`/`Connection` headers + gzip decompression to fix `urllib` 403s on Reddit's public JSON endpoint (Franco Carballar, salvaged from #199).
- HTTP 402 re-raised across all three ScrapeCreators paths (`_global_search`, `_subreddit_search`, `fetch_post_comments`) so the OpenAI/public-JSON fallback chain triggers when credits are exhausted (Jonathan Oppenheim, salvaged from #170).
**Authentication and credentials**
- Restored multi-key rotation for `SCRAPECREATORS_API_KEY` accidentally dropped in v3.0.6 (Eric Oberhofer, salvaged from #287). Comma-separated keys round-robin via `random.choice` per run.
**Windows compatibility**
- `os.killpg` in `_cleanup_children()` guarded with `hasattr(os, "killpg")`, falls back to `os.kill(SIGTERM)` (gujishh, salvaged from #226).
- POSIX-style secret-permission warning skipped on Windows ([#357](https://github.com/mvanhorn/last30days-skill/pull/357)).
- Render uses forward slashes in save-path footer for Windows ([#338](https://github.com/mvanhorn/last30days-skill/pull/338)).
**xAI / X / xurl**
- `parse_x_response` now raises `http.HTTPError` on empty output, missing JSON, or decode failure — surfaces in `errors_by_source` instead of silently returning an empty result list (Kaustav Mishra, salvaged from #155).
- `xurl` treats `PermissionError` from PATH lookup as unavailable ([#322](https://github.com/mvanhorn/last30days-skill/pull/322)).
**YouTube**
- SC YouTube + multi-token HN searches unblocked ([#388](https://github.com/mvanhorn/last30days-skill/pull/388)).
- Transcript-fetch ratio surfaced + degraded-run nudge for stale `yt-dlp` ([#340](https://github.com/mvanhorn/last30days-skill/pull/340)).
**bird_x / HTTP**
- Subprocess retry on non-JSON stdout to handle X anti-bot HTML interstitials ([#383](https://github.com/mvanhorn/last30days-skill/pull/383)).
- HTTP retry budget expanded + exponential backoff on DNS resolution failure ([#382](https://github.com/mvanhorn/last30days-skill/pull/382)).
- Parallel AI search aligned with current API schema ([#341](https://github.com/mvanhorn/last30days-skill/pull/341)).
- Parallel web backend routed through grounding ([#354](https://github.com/mvanhorn/last30days-skill/pull/354)).
**Planner and sources**
- `xquik` registered in `SOURCE_CAPABILITIES` ([#336](https://github.com/mvanhorn/last30days-skill/pull/336), fixes [#319](https://github.com/mvanhorn/last30days-skill/issues/319)).
- Honor explicit optional source requests ([#356](https://github.com/mvanhorn/last30days-skill/pull/356)).
- ScrapeCreators source-gating aligned between code and docs ([#415](https://github.com/mvanhorn/last30days-skill/pull/415)).
- OpenClaw works without ScrapeCreators key ([#392](https://github.com/mvanhorn/last30days-skill/pull/392), by @thinkun).
**Render, version display, hosting paths**
- Hardcoded `v3.0.0` in render replaced with dynamic `_skill_version()` ([#365](https://github.com/mvanhorn/last30days-skill/pull/365)).
- Comparison HTML artifacts saved correctly ([#389](https://github.com/mvanhorn/last30days-skill/pull/389)).
- `OPENROUTER_DEFAULT` model ID corrected ([#323](https://github.com/mvanhorn/last30days-skill/pull/323)).
- OpenClaw poll-timing initialized once ([#358](https://github.com/mvanhorn/last30days-skill/pull/358)).
- Prefer sandboxed Safari cookie path ([#343](https://github.com/mvanhorn/last30days-skill/pull/343)).
- Preserve clean mode for last-run state ([#334](https://github.com/mvanhorn/last30days-skill/pull/334)).
- Replaced hardcoded `/Users/mvanhorn/...` paths in `test-v1-vs-v2.sh` with portable env-var overrides (Dave Morin, salvaged from #297).
**Hooks**
- `check-config.sh` path-quoting fix for paths with spaces ([#337](https://github.com/mvanhorn/last30days-skill/pull/337)).
- Replaced unsafe `eval` with `declare` in `check-config.sh` ([#364](https://github.com/mvanhorn/last30days-skill/pull/364)).
**Sync and version metadata**
- `sync.sh` pointed at this repo's plugin cache, not the private repo's ([#402](https://github.com/mvanhorn/last30days-skill/pull/402)).
- Sync cache target bumped to 3.2.1 to match SKILL.md ([#397](https://github.com/mvanhorn/last30days-skill/pull/397)).
- ScrapeCreators free-tier credit count corrected to 100 in docs ([#369](https://github.com/mvanhorn/last30days-skill/pull/369), fixes [#367](https://github.com/mvanhorn/last30days-skill/issues/367)).
- Gemini extension version synced ([#349](https://github.com/mvanhorn/last30days-skill/pull/349)).
- Various stale path/link fixes ([#345](https://github.com/mvanhorn/last30days-skill/pull/345), [#346](https://github.com/mvanhorn/last30days-skill/pull/346), [#347](https://github.com/mvanhorn/last30days-skill/pull/347), [#348](https://github.com/mvanhorn/last30days-skill/pull/348), [#351](https://github.com/mvanhorn/last30days-skill/pull/351)).
### Contributors
First-time contributors whose fixes shipped in this release (most via PR triage salvage — fix re-applied directly to main with co-author credit when path migration made the original branch un-rebaseable):
- Dave Morin — portable test-harness paths
- Alex Key — `removeprefix("r/")` for subreddit names
- Eric Oberhofer — multi-key rotation restored
- gujishh — Windows process cleanup
- Franco Carballar — Reddit browser-like headers
- Jonathan Oppenheim — Reddit 402 fallback chain
- Kaustav Mishra — xAI error surfacing
- [@thinkun](https://github.com/thinkun) ([#363](https://github.com/mvanhorn/last30days-skill/pull/363)) — OpenClaw ScrapeCreators-key-optional fix
Full PR list at [github.com/mvanhorn/last30days-skill/releases/tag/v3.3.0](https://github.com/mvanhorn/last30days-skill/releases/tag/v3.3.0).
## [3.2.0] - 2026-05-09
@@ -45,7 +179,7 @@ Consolidates the 3.0.10 to 3.0.14 dev cycle (commenter handles, `--competitors`,
### Fixed
- **Claude Code plugin manifest path-escape.** The `.claude-plugin/plugin.json` `skills` key was removed in commit `93fbed2` but never shipped in a tagged release. Installing via `/plugin install last30days-skill` could hit `/doctor`'s `Path escapes plugin directory: ./ (skills)` error. This release ships the fix. Closes [#306](https://github.com/mvanhorn/last30days-skill/issues/306).
- **Broken README link.** The README's "source of truth" link pointed at `skills/last30days/SKILL.md`, a path that does not exist. Fixed to point at root `SKILL.md`.
- **Broken README link.** The README's "source of truth" link pointed at root `SKILL.md`, which is no longer maintained after the plugin-layout restructure. Fixed to point at `skills/last30days/SKILL.md`.
### Dev cycle journal (3.0.10 - 3.0.14, not separately tagged)
+1 -25
View File
@@ -1,25 +1 @@
# last30days Skill
Claude Code skill for researching any topic across Reddit, X, YouTube, and web.
Python scripts with multi-source search aggregation.
## Structure
- `skills/last30days/SKILL.md` — canonical skill definition
- `skills/last30days/scripts/last30days.py` — main research engine
- `skills/last30days/scripts/lib/` — search, enrichment, rendering modules
- `skills/last30days/scripts/lib/vendor/bird-search/` — vendored X search client
## Commands
```bash
python3 skills/last30days/scripts/last30days.py "test query" --emit=compact
npx skills add . -g -y # one-time: symlink this repo into every detected harness's skill dir
```
## Rules
- `lib/__init__.py` must be bare package marker (comment only, NO eager imports)
- One-time setup: `npx skills add . -g -y` creates symlinks from each detected harness's skill dir to this repo. Edits in the working tree propagate live to every harness — no re-deploy step needed.
- Git remote: origin = public (`mvanhorn/last30days-skill`)
## Beta channel
Experimental changes get tested on `mvanhorn/last30days-skill-private`, which installs as a parallel `/last30days-beta` slash command. Beta-only changes never ship to public without a review PR here. Workflow guide lives at `BETA.md` in the private repo. Plan that established this setup: `docs/plans/2026-04-17-005-feat-beta-skill-from-private-repo-plan.md`.
@AGENTS.md
+23
View File
@@ -0,0 +1,23 @@
# Concepts
Shared vocabulary for `last30days-skill`. Terms here have a precise project-specific meaning — distinct enough from their general technical sense that a new contributor would need them defined to follow conversations, PR descriptions, or the SKILL.md contract.
## The package
### Skill
A self-contained agent-instructions package consisting of a `SKILL.md` prose contract plus a sibling `scripts/` directory containing the executable code the SKILL.md invokes. The package conforms to the [Agent Skills](https://agentskills.io) open format and installs across every major harness (Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, and 50+ others) via `npx skills add`, harness-native plugin installers, or per-harness skill directories. A Skill is the unit of distribution; the Skill is the product.
### Engine
The Python script (`scripts/last30days.py`) the Skill's SKILL.md invokes to do the actual research work. The Engine and SKILL.md have a contract: SKILL.md tells the model which flags to pass (`--plan`, `--competitors-plan`, `--x-handle`, `--subreddits`, `--emit=compact`, etc.), and the Engine produces a specific output shape (badge line, ranked evidence clusters, emoji-tree footer) that the model is contractually required to pass through. The Engine is implementation; the SKILL.md prose is the agent-facing surface.
### Harness
The agent runtime that loads Skills and invokes them on the user's behalf. Claude Code is the most common Harness for this Skill but not the only one — Codex, Cursor, GitHub Copilot, Gemini CLI, and the rest of the Agent Skills ecosystem also count. "Multi-harness" describes a Skill that works correctly across every Harness it installs into; features written without multi-harness awareness (e.g., engine flags with no SKILL.md integration, or paths hardcoded to one Harness's install layout) regress on Harnesses other than the one they were tested against.
## Distribution
### Beta channel
A parallel install of the Skill, sourced from the private `mvanhorn/last30days-skill-private` repo and installed as `/last30days-beta` rather than `/last30days`. The Beta channel exists so experimental changes can be tested by real users before they ship to the public `/last30days`. Promotion from Beta to public happens via a review PR against this (public) repo — Beta-only changes never ship to public without that PR. The Beta channel workflow guide lives in `BETA.md` in the private repo.
+268
View File
@@ -0,0 +1,268 @@
# Configuration
Everything you can tune in `/last30days` without editing the engine source.
Three layers, in order of how often you'll touch them:
1. **Per-run flags** - what you pass on the command line.
2. **Environment variables and `.env`** - what's enabled across all runs.
3. **Optional trend-monitoring stack** - SQLite store, watchlist, briefings.
Per-client patterns and the experimental beta channel are at the bottom.
> Skip ahead: [Where output is saved](#where-output-is-saved) - [API keys](#api-keys-env) - [Reasoning provider](#reasoning-provider-priority) - [Web search backend](#web-search-backend-priority) - [Trend monitoring](#trend-monitoring-store--watchlist--briefings) - [Per-client patterns](#per-client-patterns) - [Beta channel](#beta-channel)
## Why this document exists
This is a focused **configuration reference** maintained alongside the engine. The runtime contract (the voice rules, the planner protocol, the LAWs the synthesizing model follows) lives in [`skills/last30days/SKILL.md`](skills/last30days/SKILL.md) - that file is authoritative when the two ever differ. This file's job is narrower: surface every knob a user or operator can turn, in one place, kept current with the code so client-facing setups stay reliable. New configuration knobs added to the engine should be reflected here in the same PR.
---
## Where output is saved
| Platform | Default path | Override |
|---|---|---|
| Linux / macOS | `LAST30DAYS_MEMORY_DIR` defaults to `~/Documents/Last30Days/` | set `LAST30DAYS_MEMORY_DIR=/path` |
| Windows | `LAST30DAYS_MEMORY_DIR` defaults to `C:\Users\<you>\Documents\Last30Days\` | set `LAST30DAYS_MEMORY_DIR=C:\path` |
Each run produces one file per topic, slug-named:
`<slug>-raw[-suffix].md`. Same topic + same suffix on the same day overwrites; same topic + same suffix on different days appends a date stamp.
**Per-run overrides:**
- `--save-dir <path>` - one-off output location.
- `--save-suffix <name>` - distinguish runs of the same topic (e.g. per client: `--save-suffix=acme`).
The footer line `📎 Raw results saved to ${LAST30DAYS_MEMORY_DIR:-$HOME/Documents/Last30Days}/<slug>-raw.md` is the canonical pointer; if it shows backslashes on Windows update past v3.1.1.
---
## API keys (`.env`)
The skill reads keys from a `.env` file. Two locations are supported, in priority order:
1. **`.claude/last30days.env`** in the current project directory (project-scoped) - takes precedence when present.
2. **`~/.config/last30days/.env`** at the user level (global default) - the fallback.
Override the global location with `LAST30DAYS_CONFIG_DIR=/path` (or `LAST30DAYS_CONFIG_DIR=""` for no-config mode). File permissions should be `600` on POSIX hosts - the engine warns on every run if they aren't.
The project-scoped file is the cleanest pattern for **per-client setups**: drop a `.claude/last30days.env` into each client folder (`SCRAPECREATORS_API_KEY`, `INCLUDE_SOURCES`, `LAST30DAYS_MEMORY_DIR`, `BSKY_HANDLE`, etc), `cd` into that folder, and the skill picks up that client's configuration automatically. No wrapper scripts needed for the common case.
**Source-by-source** - what each key unlocks:
| Source | Key(s) | Required for | Free tier |
|---|---|---|---|
| Reddit (public) | none | always on | yes |
| Hacker News | none | always on | yes |
| Polymarket | none | always on | yes |
| GitHub | `gh` CLI installed (uses your GitHub auth) | always on if `gh` present | yes |
| YouTube | `yt-dlp` CLI installed | always on if `yt-dlp` present | yes |
| X / Twitter | one of: `AUTH_TOKEN` + `CT0` (browser cookies, Bird CLI), `XAI_API_KEY`, `SCRAPECREATORS_API_KEY`, or `FROM_BROWSER` (cookie-jar auth) | X items in results | cookie-jar / Bird = free; xAI / ScrapeCreators = paid |
| TikTok | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `tiktok` | TikTok items | 10K free calls |
| Instagram | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `instagram` | Instagram Reels | 10K free calls; raise `LAST30DAYS_TRANSCRIPT_TIMEOUT` (default 30s) if SC is slow on your network |
| Threads | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `threads` | Threads items | 10K free calls |
| Pinterest | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `pinterest` | Pinterest items | 10K free calls |
| Bluesky | `BSKY_HANDLE` + `BSKY_APP_PASSWORD` | Bluesky items | yes (app password at bsky.app) |
| TruthSocial | `TRUTHSOCIAL_TOKEN` | TruthSocial items | yes |
| Web search | one of: `BRAVE_API_KEY`, `EXA_API_KEY`, `SERPER_API_KEY`, `PARALLEL_API_KEY` | `--auto-resolve` and Step 2 supplements | Brave has a free tier; native WebSearch on Claude Code / Codex / Gemini works as a fallback |
| Perplexity Deep Research | `OPENROUTER_API_KEY` | `--deep-research` flag (~$0.90/query) | no |
| Apify (alternate scraper) | `APIFY_API_TOKEN` | fallback for Reddit/TikTok/Instagram when ScrapeCreators is exhausted | yes (limited) |
**Example `.env` skeleton** (placeholders only - replace with your own values):
```bash
# Reasoning + planning (one provider; see priority below)
GOOGLE_API_KEY=<your-gemini-key>
# Web search backend (one is enough; Brave is the cheapest)
BRAVE_API_KEY=<your-brave-key>
# Optional sources
SCRAPECREATORS_API_KEY=<your-scrapecreators-key>
INCLUDE_SOURCES=tiktok,instagram
# X authentication (one option only)
XAI_API_KEY=<your-xai-key>
# OR cookie-jar (no key needed; logs in via your browser session)
# FROM_BROWSER=firefox
# Bluesky
BSKY_HANDLE=<your-handle>.bsky.social
BSKY_APP_PASSWORD=<your-app-password>
```
After editing: `chmod 600 ~/.config/last30days/.env` (or `chmod 600 .claude/last30days.env` if using the project-scoped variant).
**Troubleshooting:** if a source you expected to see isn't appearing in results, run `python3 scripts/last30days.py --diagnose`. It prints a per-source availability report (which keys were detected, which CLIs are installed, which backends are reachable) without running a full search.
### Bluesky app-password format and search host
`BSKY_APP_PASSWORD` should be a 19-char app password in `xxxx-xxxx-xxxx-xxxx` format (lowercase alphanumeric, three hyphens). Generate one at <https://bsky.app/settings/app-passwords>. The AT Protocol's `createSession` endpoint also accepts your main account login password, but that's bad hygiene — main passwords have no scope (an app password can be limited to non-DM access) and can't be revoked individually.
The skill defaults to `api.bsky.app` for `searchPosts`, which is the canonical authenticated AppView. The previous default `public.api.bsky.app` is the unauthenticated public mirror and is currently blocked by BunnyCDN for `searchPosts` regardless of auth header (verified 2026-05-04). If Bluesky migrates infrastructure again, override the host without a code change by setting `BSKY_SEARCH_HOST` in your `.env`:
```bash
BSKY_SEARCH_HOST=api.bsky.app # default — change only if Bluesky moves
```
---
## Reasoning provider priority
`/last30days` needs one reasoning model for planning + reranking when you don't pass `--plan` yourself. Auto-detect priority (set `LAST30DAYS_REASONING_PROVIDER=<name>` to pin one):
1. **Gemini** - `GOOGLE_API_KEY` / `GEMINI_API_KEY` / `GOOGLE_GENAI_API_KEY`
2. **OpenAI** - `OPENAI_API_KEY` (or Codex auth at `~/.codex/auth.json`)
3. **xAI** - `XAI_API_KEY`
4. **OpenRouter** - `OPENROUTER_API_KEY` (also unlocks `--deep-research`)
5. **Local / deterministic** - always available, lowest quality
When you invoke `/last30days` from Claude Code, Codex, or Gemini, the host model **is** the reasoning provider for plan + synthesis - you don't need any of the keys above unless you also run the script headlessly (cron, CI, watchlist).
---
## Web search backend priority
Used by `--auto-resolve` (when WebSearch isn't available from the host) and Step 2 supplements. Auto-detect priority (override per-run with `--web-backend=<name>`):
1. **Brave** - `BRAVE_API_KEY`
2. **Exa** - `EXA_API_KEY`
3. **Serper** - `SERPER_API_KEY`
4. **Parallel** - `PARALLEL_API_KEY`
5. **Host's native WebSearch** - Claude Code, Codex, Gemini all have one built in
Visible quality difference between hosts with vs without a configured backend. If your client setup produces thinner results than yours, this is usually why.
---
## Trend monitoring (`--store` + watchlist + briefings)
The default behavior - one slug-named file per topic, overwritten on rerun - is the snapshot mode. For continuous monitoring, the repo ships three components most users miss:
### `--store` flag
Adding `--store` to any run persists every finding to a SQLite database (default at `~/.local/share/last30days/research.db`). Findings dedupe on the `source_url` column (UNIQUE constraint), so the same URL across runs updates the existing row instead of creating a duplicate. The markdown file still saves; the SQLite is the time-series substrate.
**Always-on alternative:** set `LAST30DAYS_STORE=1` in your `.env` instead of remembering `--store` on every invocation. The flag still works as before; the env var is purely additive. Same hybrid pattern as `LAST30DAYS_DEBUG` — works whether shell-exported or in `.env`.
Relevant tables: `topics`, `research_runs`, `findings`, `settings`. Schema: [`scripts/store.py`](skills/last30days/scripts/store.py).
### `watchlist.py` - recurring topics
[`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) manages topics that should be researched on a schedule. Subcommands: `add`, `remove`, `list`, `run-one`, `run-all`, `config`. Built-in delivery to Slack incoming webhooks (`hooks.slack.com/...`) or any HTTPS endpoint, fired only when new findings appear.
Two-step flow (the watchlist holds the topic; an external scheduler invokes the run):
```bash
# 1. Add the topic to the watchlist
# Default schedule daily 8am; --weekly switches to Mondays 8am
python3 scripts/watchlist.py add "british airways middle east" --weekly
# 2. Configure delivery and budget (optional)
python3 scripts/watchlist.py config delivery "https://hooks.slack.com/services/..."
python3 scripts/watchlist.py config budget 5.00
# 3. Trigger via cron / Task Scheduler / GitHub Actions
python3 scripts/watchlist.py run-one "british airways middle east"
# or run every enabled topic, gated by daily_budget
python3 scripts/watchlist.py run-all
```
The schedule field stored on each topic is metadata - the actual cron / Task Scheduler invocation is your responsibility. Watchlist runs hardcode `--quick` and `--lookback-days 90` when spawning the underlying engine.
### `briefing.py` - daily / weekly digests
[`scripts/briefing.py`](skills/last30days/scripts/briefing.py) reads the SQLite store and emits structured data the agent then synthesizes into prose. Modes: `generate` (daily), `generate --weekly`, `show [--date DATE]` (display a saved briefing). Briefs save to `~/.local/share/last30days/briefs/`.
### Recommended cadence pattern
| Step | Cadence | Command |
|---|---|---|
| Baseline | one-time per topic | `/last30days "<topic>" --days=30 --store` |
| Add to watchlist | one-time per topic | `python3 scripts/watchlist.py add "<topic>" --weekly` |
| Recurring run | daily or weekly (external scheduler) | `python3 scripts/watchlist.py run-all` |
| Digest | weekly | `python3 scripts/briefing.py generate --weekly` |
---
## Per-client patterns
The skill is built to flex around different client environments. Four patterns that compose well:
### 1. Per-client `.claude/last30days.env` (preferred when you cd into client folders)
The simplest pattern when each client has its own working directory: drop a `.claude/last30days.env` into the client folder. The skill picks it up automatically (see [API keys](#api-keys-env) for the lookup priority). Typical contents:
```bash
LAST30DAYS_MEMORY_DIR=C:\Users\<you>\Clients\acme\Research\Last30Days
SCRAPECREATORS_API_KEY=<acme-scoped-key-or-shared>
INCLUDE_SOURCES=tiktok,instagram
BSKY_HANDLE=<acme-bluesky-handle>.bsky.social
```
`cd` into the client folder, run `/last30days <topic>` as normal, no flags or wrappers. Combine with `--save-suffix=<client-slug>` per run if you also need to differentiate filenames within that folder.
### 2. Per-client save dir + suffix wrapper
For workflows where you don't `cd` into a client folder (running from anywhere, scripted batches), a tiny shell function isolates each client's research without engine changes.
PowerShell example:
```powershell
function Run-L30D-Client {
param([string]$ClientSlug, [Parameter(ValueFromRemainingArguments=$true)]$Args)
$env:LAST30DAYS_MEMORY_DIR = "C:\Users\$env:USERNAME\Clients\$ClientSlug\Research\Last30Days"
/last30days @Args --save-suffix=$ClientSlug
}
# Usage: Run-L30D-Client acme "british airways middle east"
```
Bash example:
```bash
l30d-client() {
local client=$1; shift
LAST30DAYS_MEMORY_DIR="$HOME/Clients/$client/Research/Last30Days" \
/last30days "$@" --save-suffix="$client"
}
# Usage: l30d-client acme "british airways middle east"
```
### 3. Custom category-peer subreddits
[`scripts/lib/categories.py`](skills/last30days/scripts/lib/categories.py) holds a table of `(category_id, trigger_keywords, peer_subreddits)`. If a client lives in a vertical that isn't covered (legal-tech, real-estate-tech, B2B HR SaaS), add a row. Pure data, no logic.
Section 2a of `SKILL.md` documents the merging rule the skill applies when your topic matches a category.
### 4. Pre-built `--competitors-plan` JSON
For competitor-vs-comparisons that recur, a pre-written JSON skeleton per client industry saves real time:
```json
{
"Competitor B": {
"x_handle": "competitor_b_handle",
"subreddits": ["sub1", "sub2"],
"github_user": "competitor-b-org",
"context": "Founded 2019, focused on ..."
},
"Competitor C": { ... }
}
```
Pass as `--competitors-plan @client/competitors-plan.json` (or as a string). See `SKILL.md` section "If QUERY_TYPE = COMPARISON" for the full schema.
---
## Beta channel
Experimental customizations live on a private companion repo (`mvanhorn/last30days-skill-private`) installed as `/last30days-beta`. Never ship beta-only changes to the public marketplace without a review PR against the public repo. Workflow guide: `BETA.md` in the private repo.
This is the right home for client-specific changes you don't intend to upstream - custom category rows, internal subreddit lists, per-vertical plan templates.
---
## Cross-references
- The CLI flag surface: `python3 scripts/last30days.py --help`
- The skill contract (voice, LAWs, pre-flight protocol): [`skills/last30days/SKILL.md`](skills/last30days/SKILL.md)
- Engine spec (some sections stale; SKILL.md wins on conflicts): [`SPEC.md`](SPEC.md)
- Contributor guidance: [`CONTRIBUTORS.md`](CONTRIBUTORS.md)
+1 -1
View File
@@ -23,7 +23,7 @@ v3 has full GitHub search: issues, PRs, person-mode profiles, project-mode repos
### @thinkun
[PR #116](https://github.com/mvanhorn/last30days-skill/pull/116) - Resilient Reddit, prevent enrichment timeout from discarding results
v3 has parallel enrichment with per-item timeouts. No results are ever dropped.
> _Add your bio, website, or anything you'd like here._
> Thinker, technologist, AI expert, music-tinkerer. Founder of [Thinkun](https://thinkun.com). [@thinkun on GitHub](https://github.com/thinkun) · [@unthink on X](https://x.com/unthink)
### @thomasmktong
[PR #124](https://github.com/mvanhorn/last30days-skill/pull/124) - Pure Python Reddit fallback
+1 -1
View File
@@ -51,7 +51,7 @@ On first run, the skill will guide you through setup:
2. **Optional: ScrapeCreators**
- Adds TikTok, Instagram, Reddit backup
- 10,000 free API calls
- 100 free credits (no expiration)
- Sign up at scrapecreators.com
3. **Optional: API Keys**
+37 -4
View File
@@ -12,11 +12,12 @@
**An AI agent-led search engine scored by upvotes, likes, and real money - not editors.**
This README tracks the current v3 pipeline. The runtime skill spec lives in [SKILL.md](SKILL.md), which is the source of truth for the latest command and setup behavior.
This README tracks the current v3 pipeline. The runtime skill spec lives in [skills/last30days/SKILL.md](skills/last30days/SKILL.md), which is the source of truth for the latest command and setup behavior.
**Claude Code (recommended — auto-updates via marketplace):**
```
/plugin marketplace add mvanhorn/last30days-skill
/plugin install last30days
```
**Codex, Cursor, Copilot, Gemini CLI, or any of 50+ [Agent Skills](https://agentskills.io) hosts:**
@@ -152,8 +153,10 @@ Say "eli5 on" after any research run. The synthesis rewrites in plain language.
- **Free Reddit comments.** Public JSON gives you threads + top comments with upvote counts. No API key, no ScrapeCreators. Just works.
- **YouTube transcripts that actually work.** Widened candidate pool 3x past music videos to reach talk/review content with captions.
- **Threads, Pinterest, YouTube + TikTok comments.** Opt-in sources via ScrapeCreators. Set `INCLUDE_SOURCES=tiktok,instagram` and add threads, pinterest, youtube_comments, tiktok_comments for more. `youtube_comments` and `tiktok_comments` surface top comments with vote counts the same way Reddit does.
- **Perplexity Sonar.** Grounded web search with citations via OpenRouter. Add `OPENROUTER_API_KEY` to unlock.
- **TikTok, Instagram, Threads.** All three activate automatically once `SCRAPECREATORS_API_KEY` is set — same key, same per-call cost. Suppress any of them with `EXCLUDE_SOURCES=tiktok,instagram,threads` (any comma-separated subset).
- **Pinterest.** Per-query opt-in (visual pins, narrow utility): the model passes `--search=pinterest` for the runs that need it. Requires `SCRAPECREATORS_API_KEY`.
- **YouTube + TikTok comments.** Persistent opt-in via `INCLUDE_SOURCES=youtube_comments,tiktok_comments` because each video pulls N extra ScrapeCreators calls on top of the base search. Surface top comments with vote counts the same way Reddit does.
- **Perplexity Sonar.** Grounded web search with citations via OpenRouter. Add `OPENROUTER_API_KEY` and `INCLUDE_SOURCES=perplexity` (it's a separate paid API — opt-in keeps you from being surprise-billed).
- **Polymarket noise filtering.** Common-word disambiguation prevents "Apple" from matching "Will Apple release a car?"
- **Resilient Reddit.** Timeout budgets and runtime fallback. One slow thread doesn't kill the whole run.
- **Fun judge v2.** Humor scoring baked into the narrative. Reddit's cleverest one-liners mixed into the synthesis where they fit, not dumped in a separate section.
@@ -256,10 +259,40 @@ These platforms don't have relationships with each other. X doesn't know what Re
| X / Twitter | Log into x.com in any browser | Free |
| YouTube | `brew install yt-dlp` | Free |
| Bluesky | App password from bsky.app | Free |
| TikTok + Instagram + Threads + Pinterest + YouTube comments | ScrapeCreators key | 10,000 free calls |
| TikTok + Instagram + Threads + Pinterest + YouTube comments | ScrapeCreators key | 100 free credits, then PAYG |
| Perplexity Sonar | OpenRouter key | Pay as you go |
| Web search | Brave Search key | 2,000 free queries/month |
### macOS Keychain (optional)
On macOS you can store keys in the system Keychain instead of a `.env` file. The skill picks them up automatically as the lowest-priority source — `.env` files and process environment still win on collision.
```bash
# Interactive setup — prompts for each known key, skip with empty input
skills/last30days/scripts/setup-keychain.sh
# Or store a single key by hand
security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."
# Inspect / clean up
skills/last30days/scripts/setup-keychain.sh --list
skills/last30days/scripts/setup-keychain.sh --delete XAI_API_KEY
```
Items are stored under service name `last30days-<KEY>` for the current user. On non-Darwin platforms the loader is a no-op, so there is no behaviour change for Linux/Windows users.
See [CONFIGURATION.md](CONFIGURATION.md) for the full per-source key matrix, reasoning provider priority, and web-search backend priority.
## Configuration
Two things you'll likely want to know on day one:
**Where research files are saved.** `LAST30DAYS_MEMORY_DIR` defaults to `~/Documents/Last30Days/` (Windows: `C:\Users\<you>\Documents\Last30Days\`). Override by setting that env var to any path in your shell, or `--save-dir <path>` per run. Use `--save-suffix=<name>` to keep multiple variations of the same topic separate (e.g. per client). Each run produces `<slug>-raw[-suffix].md`.
**Trend monitoring across runs.** The default mode produces a fresh markdown snapshot per run. To accumulate findings over time, add `--store` to persist into a SQLite database, then use [`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) for scheduled runs (with optional Slack / webhook delivery on new findings) and [`scripts/briefing.py`](skills/last30days/scripts/briefing.py) for daily / weekly digests. The full cadence pattern is in [CONFIGURATION.md](CONFIGURATION.md#trend-monitoring-store--watchlist--briefings).
Per-client wrapper scripts, custom category-peer subreddits, and the experimental beta channel for in-progress customizations are also documented in [CONFIGURATION.md](CONFIGURATION.md).
## How it works
1. **You type a topic.** Person, company, product, technology, "X vs Y." Anything.
-77
View File
@@ -1,77 +0,0 @@
# last30days Skill Specification
## Overview
`last30days` is a Claude Code skill that researches a given topic across Reddit and X (Twitter) using the OpenAI Responses API and xAI Responses API respectively. It enforces a strict 30-day recency window, popularity-aware ranking, and produces actionable outputs including best practices, a prompt pack, and a reusable context snippet. OpenAI auth can come from `OPENAI_API_KEY` or Codex login credentials.
The skill operates in three modes depending on available API keys: **reddit-only** (OpenAI key), **x-only** (xAI key), or **both** (full cross-validation). It uses automatic model selection to stay current with the latest models from both providers, with optional pinning for stability.
## Architecture
The orchestrator (`last30days.py`) coordinates discovery, enrichment, normalization, scoring, deduplication, and rendering. Each concern is isolated in `scripts/lib/`:
- **env.py**: Load API keys from `~/.config/last30days/.env` and Codex auth from `~/.codex/auth.json`
- **dates.py**: Date range calculation and confidence scoring
- **cache.py**: 24-hour TTL caching keyed by topic + date range
- **http.py**: stdlib-only HTTP client with retry logic
- **models.py**: Auto-selection of OpenAI/xAI models with 7-day caching
- **openai_reddit.py**: OpenAI Responses API + web_search for Reddit
- **xai_x.py**: xAI Responses API + x_search for X
- **reddit_enrich.py**: Fetch Reddit thread JSON for real engagement metrics
- **hackernews.py**: Hacker News search via Algolia API (free, no auth)
- **polymarket.py**: Polymarket prediction market search via Gamma API (free, no auth)
- **normalize.py**: Convert raw API responses to canonical schema
- **score.py**: Compute popularity-aware scores (relevance + recency + engagement)
- **dedupe.py**: Near-duplicate detection via text similarity
- **render.py**: Generate markdown and JSON outputs
- **schema.py**: Type definitions and validation
## Embedding in Other Skills
Other skills can import the research context in several ways:
### Inline Context Injection
```markdown
## Recent Research Context
!python3 ~/.claude/skills/last30days/scripts/last30days.py "your topic" --emit=context
```
### Read from File
```markdown
## Research Context
!cat ~/.local/share/last30days/out/last30days.context.md
```
### Get Path for Dynamic Loading
```bash
CONTEXT_PATH=$(python3 ~/.claude/skills/last30days/scripts/last30days.py "topic" --emit=path)
cat "$CONTEXT_PATH"
```
### JSON for Programmatic Use
```bash
python3 ~/.claude/skills/last30days/scripts/last30days.py "topic" --emit=json > research.json
```
## CLI Reference
```
python3 ~/.claude/skills/last30days/scripts/last30days.py <topic> [options]
Options:
--refresh Bypass cache and fetch fresh data
--mock Use fixtures instead of real API calls
--emit=MODE Output mode: compact|json|md|context|path (default: compact)
--sources=MODE Source selection: auto|reddit|x|both (default: auto)
```
## Output Files
All outputs are written to `~/.local/share/last30days/out/`:
- `report.md` - Human-readable full report
- `report.json` - Normalized data with scores
- `last30days.context.md` - Compact reusable snippet for other skills
- `raw_openai.json` - Raw OpenAI API response
- `raw_xai.json` - Raw xAI API response
- `raw_reddit_threads_enriched.json` - Enriched Reddit thread data
-47
View File
@@ -1,47 +0,0 @@
# last30days Implementation Tasks
## Setup & Configuration
- [x] Create directory structure
- [x] Write SPEC.md
- [x] Write TASKS.md
- [x] Write SKILL.md with proper frontmatter
## Core Library Modules
- [x] scripts/lib/env.py - Environment and API key loading
- [x] scripts/lib/dates.py - Date range and confidence utilities
- [x] scripts/lib/cache.py - TTL-based caching
- [x] scripts/lib/http.py - HTTP client with retry
- [x] scripts/lib/models.py - Auto model selection
- [x] scripts/lib/schema.py - Data structures
- [x] scripts/lib/openai_reddit.py - OpenAI Responses API
- [x] scripts/lib/xai_x.py - xAI Responses API
- [x] scripts/lib/reddit_enrich.py - Reddit thread JSON fetcher
- [x] scripts/lib/normalize.py - Schema normalization
- [x] scripts/lib/score.py - Popularity scoring
- [x] scripts/lib/dedupe.py - Near-duplicate detection
- [x] scripts/lib/render.py - Output rendering
## Main Script
- [x] scripts/last30days.py - CLI orchestrator
## Fixtures
- [x] fixtures/openai_sample.json
- [x] fixtures/xai_sample.json
- [x] fixtures/reddit_thread_sample.json
- [x] fixtures/models_openai_sample.json
- [x] fixtures/models_xai_sample.json
## Tests
- [x] tests/test_dates.py
- [x] tests/test_cache.py
- [x] tests/test_models.py
- [x] tests/test_score.py
- [x] tests/test_dedupe.py
- [x] tests/test_normalize.py
- [x] tests/test_render.py
## Validation
- [x] Run tests in mock mode
- [x] Demo --emit=compact
- [x] Demo --emit=context
- [x] Verify file tree
+12 -11
View File
@@ -142,7 +142,7 @@ The repo vendors a search-only subset of Bird's Twitter GraphQL client and shell
| Likes/reposts | Real (X API) | Real (x_search tool) |
| Replies/quotes | Real | Real |
| Author handle | Real | Real |
| Relevance score | Default 0.7 (re-ranked by score.py) | AI-assessed 0.0-1.0 |
| Relevance score | Default 0.7 (re-ranked by relevance.py) | AI-assessed 0.0-1.0 |
### Depth settings
@@ -183,13 +183,14 @@ After both searches complete:
| File | Purpose |
|---|---|
| `scripts/last30days.py` | Main orchestrator, concurrent execution |
| `scripts/lib/openai_reddit.py` | Reddit search via OpenAI Responses API |
| `scripts/lib/reddit_enrich.py` | Fetch real engagement data from Reddit JSON API |
| `scripts/lib/xai_x.py` | X search via xAI API |
| `scripts/lib/bird_x.py` | X search via bundled Bird client (free) |
| `scripts/lib/models.py` | Auto-select best available model |
| `scripts/lib/env.py` | API key loading, source detection |
| `scripts/lib/http.py` | HTTP transport with retries |
| `scripts/lib/score.py` | Relevance scoring |
| `scripts/lib/dedupe.py` | URL-based deduplication |
| `skills/last30days/scripts/last30days.py` | Main CLI entry point |
| `skills/last30days/scripts/lib/pipeline.py` | Multi-source retrieval orchestration |
| `skills/last30days/scripts/lib/reddit_public.py` | Reddit public JSON search |
| `skills/last30days/scripts/lib/reddit_enrich.py` | Fetch real engagement data from Reddit JSON API |
| `skills/last30days/scripts/lib/xai_x.py` | X search via xAI API |
| `skills/last30days/scripts/lib/bird_x.py` | X search via bundled Bird client (free) |
| `skills/last30days/scripts/lib/providers.py` | Reasoning provider and model selection |
| `skills/last30days/scripts/lib/env.py` | API key loading, source detection |
| `skills/last30days/scripts/lib/http.py` | HTTP transport with retries |
| `skills/last30days/scripts/lib/relevance.py` | Query matching and relevance scoring |
| `skills/last30days/scripts/lib/dedupe.py` | URL-based deduplication |
@@ -1,4 +1,7 @@
---
> **NOTE (added 2026-05-16):** This plan references `bash scripts/sync.sh`. That script was deleted in [PR #405](https://github.com/mvanhorn/last30days-skill/pull/405); the install workflow is now `npx skills add . -g -y` (symlinks the working tree across every detected harness). For context on why sync.sh went away, see [docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md](../solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md). The decisions captured in this plan remain accurate; only the deploy mechanism changed.
title: "feat: --competitors flag for auto-discovered comparison fan-out"
type: feat
status: active
@@ -1,4 +1,7 @@
---
> **NOTE (added 2026-05-16):** This plan references `bash scripts/sync.sh`. That script was deleted in [PR #405](https://github.com/mvanhorn/last30days-skill/pull/405); the install workflow is now `npx skills add . -g -y` (symlinks the working tree across every detected harness). For context on why sync.sh went away, see [docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md](../solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md). The decisions captured in this plan remain accurate; only the deploy mechanism changed.
title: "fix: per-entity resolution, default-2, and stale-path guard for --competitors"
type: fix
status: active
@@ -1,4 +1,7 @@
---
> **NOTE (added 2026-05-16):** This plan references `bash scripts/sync.sh`. That script was deleted in [PR #405](https://github.com/mvanhorn/last30days-skill/pull/405); the install workflow is now `npx skills add . -g -y` (symlinks the working tree across every detected harness). For context on why sync.sh went away, see [docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md](../solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md). The decisions captured in this plan remain accurate; only the deploy mechanism changed.
title: "feat: vs mode runs N full passes and --competitors is vs with auto-discovery"
type: feat
status: active
@@ -1,4 +1,7 @@
---
> **NOTE (added 2026-05-16):** This plan references `bash scripts/sync.sh`. That script was deleted in [PR #405](https://github.com/mvanhorn/last30days-skill/pull/405); the install workflow is now `npx skills add . -g -y` (symlinks the working tree across every detected harness). For context on why sync.sh went away, see [docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md](../solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md). The decisions captured in this plan remain accurate; only the deploy mechanism changed.
title: "fix: comparison title says (/Last30Days) instead of (Last 30 Days)"
type: fix
status: active
+3 -3
View File
@@ -1,6 +1,6 @@
# Search Quality Eval
`scripts/evaluate_search_quality.py` is an optional local evaluation step for retrieval quality. It is not part of the user-facing runtime and does not need to run in CI by default.
`skills/last30days/scripts/evaluate_search_quality.py` is an optional local evaluation step for retrieval quality. It is not part of the user-facing runtime and does not need to run in CI by default.
What it does:
@@ -18,13 +18,13 @@ What it does:
Recommended usage:
```bash
uv run python scripts/evaluate_search_quality.py
uv run python skills/last30days/scripts/evaluate_search_quality.py
```
Useful flags:
```bash
uv run python scripts/evaluate_search_quality.py \
uv run python skills/last30days/scripts/evaluate_search_quality.py \
--baseline-rev origin/main \
--candidate-rev HEAD \
--no-default-topics \
@@ -0,0 +1,82 @@
---
title: Search-quality eval is manual by default, not a CI gate on every PR
date: 2026-05-10
category: docs/solutions/architecture
module: skills/last30days/scripts/evaluate_search_quality.py
problem_type: design_decision
component: ci_policy
severity: low
applies_when:
- a contributor proposes wiring search-quality eval into PR CI
- a change affects retrieval, ranking, grounding, or synthesis quality and a reviewer asks "why aren't we testing this in CI?"
- someone is deciding whether a new evaluator-style script belongs in the default CI workflow
related_components:
- search_quality_evaluation
- ci_workflow
- llm_judging
tags:
- ci-policy
- eval
- design-decision
- cost-vs-signal
- non-determinism
- manual-gates
---
# Search-quality eval is manual by default, not a CI gate on every PR
## Context
`skills/last30days/scripts/evaluate_search_quality.py` compares a baseline revision against a candidate revision across a fixed pool of reviewer topics. It produces two flavors of metrics: deterministic overlap (Jaccard, retention) and LLM-judged quality scores. The natural impulse on seeing an evaluator script is to wire it into CI on every PR — "regression catcher, run it automatically." We deliberately don't.
Three properties of this particular evaluator make CI-on-every-PR the wrong default:
1. **Live API access.** The candidate revision typically needs the engine to actually run, which means real ScrapeCreators calls, real reddit fetches, real YouTube searches. CI runs would either need production credentials or a record/replay fixture set that drifts almost immediately as external APIs change shape.
2. **Cost and latency.** A full eval pass runs the pipeline N times across reviewer topics. Multiplied by every PR (including doc-only PRs), the spend is meaningful and the wall-clock pushes CI from ~30s to many minutes.
3. **Non-determinism in the judging path.** The LLM-judged metrics are valuable for review but depend on judge-model behavior on a given day. A flaky eval that fails 1 PR in 20 because the judge re-scored an item differently is a worse CI signal than no eval at all — it teaches contributors to retry rather than read the result.
The deterministic overlap metrics are useful regression signals but they are not the same as user-facing correctness. A change that improves overlap can degrade synthesis quality; a change that drops overlap can be a deliberate improvement. So even the deterministic side isn't safe to auto-fail on.
## Guidance
### 1. Keep search-quality eval available, just not automatic
The script stays runnable by maintainers and contributors. The pattern is:
```bash
LAST30DAYS_PYTHON=python3.13 \
python3 skills/last30days/scripts/evaluate_search_quality.py \
--baseline main --candidate HEAD
```
Reviewers can request a manual eval run when a PR is in the retrieval/ranking/synthesis path and the risk warrants it. Contributors can run it locally before submitting if they want signal upfront.
### 2. Standard PR CI gates remain deterministic and contract-shaped
`pytest` (offline-safe), plugin-contract checks, version-consistency contracts, ruff/lint. Anything that returns the same answer twice for the same input. Quality-of-output assessment lives outside that loop.
### 3. The middle ground is `workflow_dispatch`, not auto-PR-gating
If maintainers want a GitHub-triggered eval that doesn't make every PR pay the live-API cost, the right shape is a manually-dispatched workflow (or a label-triggered one) — not a `pull_request:` workflow that runs unconditionally. That keeps the cost knob in human hands.
### 4. Revisit if the eval can ever be made offline-deterministic
The blocker is the live-API + non-determinism combination. If a future iteration of the script can compute meaningful Jaccard/retention metrics against static fixtures (no live API calls, no LLM judging), the decision flips and it becomes a candidate for default CI. The decision below tracks that condition; revisit when it's met.
## What this means in practice
- Don't merge PRs that wire `evaluate_search_quality.py` into the default `validate.yml` workflow.
- Do merge PRs that add `workflow_dispatch` triggers or label-gated runs.
- When reviewing a retrieval/ranking change, request a manual eval if the diff suggests it could regress quality — don't expect CI to catch it.
## Links
- `skills/last30days/scripts/evaluate_search_quality.py` — the evaluator script
- `docs/search-quality-eval.md` — user-facing usage documentation
- `.github/workflows/validate.yml` — the default CI workflow (deterministic gates only)
---
*Adapted from a draft ADR proposed by @hnshah in [#374](https://github.com/mvanhorn/last30days-skill/pull/374), restructured into the `docs/solutions/` convention. The original ADR text correctly identified the constraint; this version adds the "why workflow_dispatch is the middle ground" framing and the revisit-condition.*
@@ -0,0 +1,219 @@
---
title: Release-time consistency tests cause cascade CI failures across all open PRs
date: 2026-05-16
category: docs/solutions/workflow-issues
module: ci-release-engineering
problem_type: workflow_issue
component: testing_framework
severity: high
applies_when:
- a test asserts consistency between two release-time artifacts (e.g., SKILL.md version and a hardcoded pin in a shell script)
- one artifact is updated as part of a version bump and the other requires a manual lockstep update
- multiple long-lived PRs are open simultaneously against the same base branch
symptoms:
- every open PR's CI fails after a version bump even though the PRs are unrelated to versioning
- the failing test references a stale hardcoded value that was not updated alongside the bumped version
- PR authors must rebase and manually fix an artifact they did not touch
root_cause: missing_workflow_step
resolution_type: code_fix
related_components:
- development_workflow
- documentation
tags:
- ci
- release-engineering
- consistency-test
- version-pin
- cascade-failure
- test-design
- workflow
---
# Release-time consistency tests cause cascade CI failures across all open PRs
## Context
A `tests/test_version_consistency.py::test_sync_cache_path_uses_skill_version` test was added to enforce that the version string embedded in `skills/last30days/scripts/sync.sh` (a hardcoded plugin-cache path segment) matched the version frontmatter in `skills/last30days/SKILL.md`. The intention was sound: the cache path had to stay in lockstep with the skill version or the sync would silently pull stale files.
The test worked as designed until a release shipped. At that point it turned into a cascade-failure machine:
1. A release PR bumps `SKILL.md` version (e.g., 3.2.0 → 3.2.1) **and** bumps the `sync.sh` pin. That PR's CI is green.
2. The release PR merges to `main`.
3. Every PR that was open at merge time was branched from pre-release `main`. Those PRs have `SKILL.md` 3.2.1 (inherited via merge-base with `main`) but their branch never touched `sync.sh`.
4. CI for those PRs runs the consistency test against the new `main``SKILL.md` says 3.2.1, `sync.sh` still says 3.2.0 — and fails.
5. All open PRs are now red simultaneously, with a failure that has nothing to do with their changes.
This affected at least five PRs during the 2026-05-13 to 2026-05-15 window: PR #400 (caught during rebase, required a manual pin bump), PRs #390 and #392 (OpenClaw `SCRAPECREATORS_API_KEY` fix, both stalled for the same stale-pin reason), and at least two others. A follow-up hotfix PR (#397`fix(sync): bump cache target to 3.2.1 to match SKILL.md`) was required just to unblock the queue.
The permanent fix was PR #405: delete `sync.sh` entirely (the install workflow made it redundant) and drop `test_sync_cache_path_uses_skill_version`. Once both were gone, no version-consistency cascade was possible.
## Guidance
### 1. Don't write consistency tests that read two files and assert one matches a substring derived from the other
This pattern looks safe but is not:
```python
def test_sync_cache_path_uses_skill_version(self) -> None:
sync_text = (SKILL_ROOT / "scripts" / "sync.sh").read_text(encoding="utf-8")
version = _skill_version() # reads SKILL.md
self.assertIn(
f'last30days-skill/last30days/{version}"',
sync_text, # asserts sync.sh contains that string
)
```
It encodes the assumption that both files are always updated together, in the same commit, on the same branch. That assumption breaks the moment two files have independent lifecycle owners — a versioned manifest and a deployment script are archetypal examples.
### 2. If the values genuinely need to stay in sync, derive one from the other at runtime
Remove the hardcoded pin from `sync.sh` and compute it:
```bash
# sync.sh — derive version from SKILL.md at runtime, no pin to maintain
SKILL_VERSION=$(grep -m1 '^version:' "$(dirname "$0")/../SKILL.md" \
| sed 's/version:[[:space:]]*"\([^"]*\)"/\1/')
CACHE_PATH="last30days-skill/last30days/${SKILL_VERSION}"
```
Now there is only one source of truth (`SKILL.md`). The test that asserted they matched becomes vacuous and should be deleted. If `SKILL.md` is wrong, the sync itself will fail loudly — which is better feedback than a CI gate on a different PR.
### 3. If two values must stay independent for legitimate reasons, update them together and make the test self-skip if either source is missing
If separate versioning is genuinely required (e.g., SKILL.md versions for harness consumers, sync.sh versions a private artifact store with its own cadence), update both in the same PR — never staggered — and write the test to self-skip rather than error when either file is absent:
```python
def test_sync_cache_path_uses_skill_version(self) -> None:
sync_sh = SKILL_ROOT / "scripts" / "sync.sh"
if not sync_sh.exists():
self.skipTest("sync.sh not present; skipping pin consistency check")
sync_text = sync_sh.read_text(encoding="utf-8")
version = _skill_version()
self.assertIn(
f'last30days-skill/last30days/{version}"',
sync_text,
)
```
Self-skipping means deleting the file is a non-event in CI — no cascading red, no hotfix PR to the queue.
### 4. Run consistency tests against the merge-base diff, not main
If you keep a two-file consistency test, scope it so it only fails when the PR itself modifies one of the two files but not the other. A GitHub Actions step can do this:
```yaml
- name: Check sync.sh version pin consistency
run: |
BASE=$(git merge-base HEAD origin/main)
SKILL_CHANGED=$(git diff --name-only "$BASE" HEAD | grep -c 'SKILL\.md' || true)
SYNC_CHANGED=$(git diff --name-only "$BASE" HEAD | grep -c 'sync\.sh' || true)
if [ "$SKILL_CHANGED" -gt 0 ] && [ "$SYNC_CHANGED" -eq 0 ]; then
echo "SKILL.md version bumped but sync.sh pin was not updated"
exit 1
fi
```
This only fires when your PR touched `SKILL.md` and left `sync.sh` alone — never because a release merged to `main` after you branched.
### 5. Ask whether you actually need this test
If the values are wrong, downstream tooling will fail loudly: the sync will fetch the wrong artifact, installs will break, or the harness will reject the version. A test that exists only to catch a human-bookkeeping error at release time adds cascade-fail risk without offering a meaningfully earlier signal. Weigh that cost before adding any two-file consistency gate.
## Why This Matters
The damage from a stale-pin consistency test is asymmetric. It:
- Fails on every open PR simultaneously the moment a release lands on `main` — not just the PR that forgot to update the pin.
- Produces a failure message that points at a line in a test file with no obvious relationship to the PR's actual changes.
- Requires either a hotfix PR (touching a file the failing PRs have no business touching) or a manual rebase of every affected branch.
- Blocks work that has already been reviewed and approved.
In this repo the effect was measurable: at least five PRs stalled across a two-day window, one hotfix PR was shipped just to unblock the queue, and multiple authors spent time debugging a failure completely unrelated to their changes.
The broader principle is that tests which gate on *bookkeeping consistency between files* impose their maintenance cost on every contributor, every time, even when those contributors did nothing wrong. That cost compounds with team size and release cadence.
## When to Apply
Apply this guidance whenever you find yourself:
- Writing a test that reads two files and asserts that a string in one matches a value derived from the other.
- Adding a CI step labeled "consistency check," "sync check," or "pin check" where the check compares a hardcoded value against a computed one from a separate file.
- Working in a repo where a versioned manifest (e.g., `SKILL.md`, `package.json`, `pyproject.toml`) and a deployment artifact (e.g., a shell script, a Dockerfile, a Helm values file) are both maintained by hand.
- Reviewing a PR that touches only one of two "paired" files and fails a consistency test for the other.
It does *not* apply to tests that read a single source of truth and validate its internal structure (e.g., asserting that `SKILL.md`'s frontmatter version is double-quoted, or that `package.json`'s `version` field is a valid semver string). Those tests have one file and one assertion; they cannot cascade across branches.
## Examples
### Before — the pattern that caused the cascade
Original `tests/test_version_consistency.py` (deleted in commit `9fb19ea`):
```python
import re
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SKILL_ROOT = ROOT / "skills" / "last30days"
def _skill_version() -> str:
text = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
match = re.search(r'^version:\s*"([^"]+)"\s*$', text, re.MULTILINE)
if not match:
raise AssertionError("SKILL.md version frontmatter not found")
return match.group(1)
class TestVersionConsistency(unittest.TestCase):
def test_sync_cache_path_uses_skill_version(self) -> None:
sync_text = (SKILL_ROOT / "scripts" / "sync.sh").read_text(encoding="utf-8")
version = _skill_version() # source 1: SKILL.md frontmatter
self.assertIn( # assertion: sync.sh must contain
f'last30days-skill/last30days/{version}"',
sync_text, # source 2: hardcoded string in sync.sh
)
```
`sync.sh` contained a line like:
```bash
PLUGIN_CACHE="$HOME/.cache/last30days-skill/last30days/3.2.0"
```
When SKILL.md bumped to `3.2.1` in a release PR, `sync.sh` was updated in the same PR and CI stayed green. But every PR branched before that release still had `sync.sh` at `3.2.0`. Their CI failed immediately, with an assertion error pointing at the test, not at the release PR.
### After — what we did: delete both
PR #405 deleted `sync.sh` (the install workflow replaced it) and dropped `test_sync_cache_path_uses_skill_version` in the same change. No consistency gate, no pin to maintain, no cascade possible.
### After — what we could have done instead: derive at runtime
If `sync.sh` had still been needed, the right fix would have been to remove the hardcoded version from the script and derive it from `SKILL.md`:
```bash
#!/usr/bin/env bash
# sync.sh — no hardcoded version; reads SKILL.md as single source of truth
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SKILL_VERSION=$(grep -m1 '^version:' "${SCRIPT_DIR}/../SKILL.md" \
| sed 's/version:[[:space:]]*"\([^"]*\)"/\1/')
if [ -z "$SKILL_VERSION" ]; then
echo "error: could not parse version from SKILL.md" >&2
exit 1
fi
PLUGIN_CACHE="$HOME/.cache/last30days-skill/last30days/${SKILL_VERSION}"
# ... rest of sync logic
```
With this in place, `test_sync_cache_path_uses_skill_version` has no reason to exist — there is nothing to assert. Delete it. If the version parsing breaks, `sync.sh` itself exits non-zero with a clear message.
## Related
- **PR #397** (merged) — `fix(sync): bump cache target to 3.2.1 to match SKILL.md`. The hotfix that unblocked the cascade temporarily by bumping the pin.
- **PR #400** (merged) — caught the same cascade during rebase; had to bump the pin to clear CI.
- **PR #390** (closed) and **PR #392** (rebased + merged) — OpenClaw `SCRAPECREATORS_API_KEY` fix; both blocked by the cascade until rebased onto post-#405 main.
- **PR #405** (merged) — the permanent fix: deleted `sync.sh` + `test_sync_cache_path_uses_skill_version` together.
- **PR #412** (merged) — adjacent work that consolidated SKILL.md version parsing into `lib/skill_meta.py`, reducing future drift risk by giving the version field one canonical reader.
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "last30days-skill",
"version": "3.0.5",
"version": "3.2.4",
"description": "Research a topic from the last 30 days across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web.",
"settings": [
{
+4
View File
@@ -0,0 +1,4 @@
{
"triggerOnUpdates": true,
"statusCheck": true
}
+1 -2
View File
@@ -6,8 +6,7 @@
"hooks": [
{
"type": "command",
"command": "bash ${CLAUDE_PLUGIN_ROOT}/hooks/scripts/check-config.sh",
"timeout": 5
"command": "bash \"${CLAUDE_PLUGIN_ROOT:-${extensionPath:-.}}/hooks/scripts/check-config.sh\""
}
]
}
+64 -3
View File
@@ -33,8 +33,13 @@ load_env_vars() {
[[ -z "$key" ]] && continue
key=$(echo "$key" | xargs)
value=$(echo "$value" | xargs | sed 's/^["'\''"]//;s/["'\''"]$//')
# Strip inline comments (# preceded by whitespace) to prevent
# command substitution in backtick-containing comments
value="${value%%[[:space:]]#*}"
if [[ -n "$key" && -n "$value" ]]; then
eval "ENV_${key}=\"${value}\""
# printf -v writes via assignment semantics (global from inside a
# function), works on macOS's /bin/bash 3.2 — `declare -g` is 4.2+.
printf -v "ENV_${key}" '%s' "$value"
fi
done < "$file"
fi
@@ -58,14 +63,53 @@ fi
# Check SETUP_COMPLETE (from file or env)
SETUP_COMPLETE="${ENV_SETUP_COMPLETE:-${SETUP_COMPLETE:-}}"
# Compute last-run summary line (if last-run.json exists)
if [[ "${LAST30DAYS_CONFIG_DIR+x}" == "x" ]]; then
if [[ -n "$LAST30DAYS_CONFIG_DIR" ]]; then
LAST_RUN_FILE="$LAST30DAYS_CONFIG_DIR/last-run.json"
else
LAST_RUN_FILE=""
fi
else
LAST_RUN_FILE="$HOME/.config/last30days/last-run.json"
fi
LAST_RUN_LINE=""
if [[ -n "$LAST_RUN_FILE" && -f "$LAST_RUN_FILE" ]] && command -v python3 &>/dev/null; then
LAST_RUN_LINE=$(LAST_RUN_FILE="$LAST_RUN_FILE" python3 - <<'PY' 2>/dev/null || true
import datetime
import json
import os
path = os.environ["LAST_RUN_FILE"]
try:
with open(path) as fh:
d = json.load(fh)
topic = (d.get("topic") or "?")[:60]
ts = d.get("timestamp", "")
dt = datetime.datetime.fromisoformat(ts.replace("Z", "+00:00"))
delta = (datetime.datetime.now(datetime.timezone.utc) - dt).total_seconds()
if delta < 60: ago = f"{int(delta)}s ago"
elif delta < 3600: ago = f"{int(delta//60)}m ago"
elif delta < 86400: ago = f"{int(delta//3600)}h ago"
else: ago = f"{int(delta//86400)}d ago"
total = d.get("total", 0)
print(f" Last run: \"{topic}\" · {ago} · {total} results")
except Exception:
pass
PY
)
fi
# If setup has never been run, show welcome message for new users
if [[ -z "$SETUP_COMPLETE" && -z "$CONFIG_FILE" && -z "${OPENAI_API_KEY:-}" && -z "${SCRAPECREATORS_API_KEY:-}" && -z "${AUTH_TOKEN:-}" && -z "${XAI_API_KEY:-}" ]]; then
cat <<'EOF'
/last30days: Ready to use. Run /last30days to get started — setup takes 30 seconds.
Research any topic across Reddit, HN, X, YouTube, Polymarket (last 30 days).
Reddit, Hacker News, and Polymarket work out of the box.
The setup wizard can unlock X/Twitter, YouTube, and more.
EOF
[[ -n "$LAST_RUN_LINE" ]] && echo "$LAST_RUN_LINE"
exit 0
fi
@@ -97,16 +141,33 @@ if [[ -n "$HAS_BSKY" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 1))
fi
if [[ -n "$HAS_SCRAPECREATORS" ]]; then
SOURCE_COUNT=$((SOURCE_COUNT + 3)) # Reddit comments + TikTok + Instagram
# Start with Reddit comments + TikTok + Instagram, subtract any in EXCLUDE_SOURCES.
# Normalise EXCLUDED (lowercase + collapse whitespace around commas + strip outer
# whitespace) so the matching mirrors pipeline.py's .strip().lower() parsing.
SC_ADD=3
EXCLUDED="${ENV_EXCLUDE_SOURCES:-${EXCLUDE_SOURCES:-}}"
EXCLUDED_NORM=$(printf '%s' "$EXCLUDED" | tr '[:upper:]' '[:lower:]' \
| sed -E 's/[[:space:]]*,[[:space:]]*/,/g; s/^[[:space:]]+//; s/[[:space:]]+$//')
if [[ ",$EXCLUDED_NORM," == *",tiktok,"* ]]; then
SC_ADD=$((SC_ADD - 1))
fi
if [[ ",$EXCLUDED_NORM," == *",instagram,"* ]]; then
SC_ADD=$((SC_ADD - 1))
fi
SOURCE_COUNT=$((SOURCE_COUNT + SC_ADD))
fi
if [[ -n "$HAS_SCRAPECREATORS" ]]; then
# Fully configured — compact ready message
echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
echo " Research any topic across social + market + web sources (last 30 days)."
[[ -n "$LAST_RUN_LINE" ]] && echo "$LAST_RUN_LINE"
else
# Setup done but missing ScrapeCreators — recommend it
echo "/last30days: Ready — ${SOURCE_COUNT} sources active."
echo " Research any topic across social + market + web sources (last 30 days)."
[[ -n "$LAST_RUN_LINE" ]] && echo "$LAST_RUN_LINE"
echo " Tip: Add ScrapeCreators for Reddit comments + TikTok + Instagram."
echo " 10,000 free API calls, no credit card — scrapecreators.com"
echo " 100 free credits, no credit card — scrapecreators.com"
echo " last30days has no affiliation with any API provider."
fi
+2 -2
View File
@@ -1,6 +1,6 @@
[project]
name = "last30days-skill"
version = "3.2.3"
version = "3.3.0"
description = "Multi-source last-30-days research skill"
readme = "README.md"
requires-python = ">=3.12"
@@ -8,7 +8,7 @@ dependencies = []
[dependency-groups]
dev = [
"pytest>=9,<10",
"pytest>=9.0.3,<10",
"pytest-cov>=7,<8",
]
+47 -43
View File
@@ -1,52 +1,64 @@
## v3.3.0 — install everywhere, ship the reliability sweep
The AI world reinvents itself every month. This skill keeps you current.
`/last30days` researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and 5+ more sources from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.
`/last30days` researches your topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, Digg, and 5+ more sources from the last 30 days, finds what the community is actually upvoting, sharing, betting on, and saying on camera, and writes you a grounded narrative with real citations.
## v3 is the intelligent search release
## What's new in v3.3.0
v3 is a ground-up engine rewrite by [@j-sperling](https://github.com/j-sperling). The old engine searched keywords. The new engine understands your topic first, then searches the right people and communities.
### Install everywhere with one command
Type "OpenClaw" and v3 resolves @steipete, r/openclaw, r/ClaudeCode, and the right YouTube channels and TikTok hashtags before a single API call fires. Type "Peter Steinberger" and it resolves his X handle and GitHub profile, switches to person mode, and shows what he shipped this month at 85% merge rate across 22 PRs. None of that was on Google.
`npx skills add mvanhorn/last30days-skill -g -y` is now the canonical install path for **every harness** — Claude Code, OpenAI Codex CLI, Cursor, Gemini CLI, GitHub Copilot, Windsurf, and 50+ other Agent Skills hosts. The skill auto-detects each harness's skills directory and symlinks in place, so edits propagate live. No more per-harness manual paths in the README.
## Headline features
### New emit mode: `--emit=html`
### Intelligent pre-research
Shareable, print-friendly HTML briefs. Drop the file in Slack, mail it to a stakeholder, or print it for the meeting. Same data as compact mode, structured for human reading.
The killer feature. A new Python pre-research brain resolves X handles, GitHub repos, subreddits, TikTok hashtags, and YouTube channels before searching. Bidirectional: person to company, product to founder, name to GitHub profile. The right subreddits, the right handles, the right hashtags, all resolved before a single API call.
### New source: Digg
### Best Takes
Digg surfaces curated story clusters from the AI 1000 leaderboard and pulls attributable X-post quotes directly into the brief. Auto-enabled when `digg-pp-cli` is on PATH. Footer line: `⛏️ Digg: N clusters │ K posts │ M authors`. No X auth required for the inline quotes.
A second LLM judge scores every result for humor, wit, and virality alongside relevance. Every brief now ends with a Best Takes section surfacing the cleverest one-liners and most viral quotes. The Reddit and X people are funny, and the old engine buried their best stuff.
### YouTube residential-IP routing (`LAST30DAYS_YOUTUBE_SSH_HOST`)
### Cross-source cluster merging
Running on a datacenter VPS (Hetzner, DigitalOcean, AWS, etc.)? YouTube's bot-wall fingerprints datacenter IP ranges before any cookie check. Set `LAST30DAYS_YOUTUBE_SSH_HOST=<ssh-alias>` and yt-dlp runs over SSH against a residential-IP host instead. One env var, no proxy service required.
When the same story hits Reddit, X, and YouTube, v3 merges them into one cluster instead of three duplicates. Entity-based overlap detection catches matches even when the titles use different words.
### macOS Keychain credential source
### Single-pass comparisons
When env vars and config files aren't set, the engine now reads credentials from the macOS Keychain. Stores secrets where macOS expects them; nothing on disk in plaintext.
"X vs Y" used to run three serial passes (12+ minutes). v3 runs one pass with entity-aware subqueries for both sides at once. Same depth, 3 minutes.
### `EXCLUDE_SOURCES` env var
### GitHub person-mode and project-mode
The inverse of `INCLUDE_SOURCES`. Useful for "everything except TikTok" or "everything except the slow ones."
When the topic is a person, the engine switches from keyword search to author-scoped queries. PR velocity, top repos by stars, release notes for what shipped this month, woven into the narrative alongside X posts and Reddit threads.
## Reliability sweep
When the topic is a project, it pulls live star counts, READMEs, releases, and top issues from the GitHub API. No stale blog posts.
This release closes a long tail of platform-specific issues that have been accumulating:
### ELI5 mode
- **Reddit**: subreddits starting with `r` no longer get mangled by `lstrip("r/")`. Browser-like headers + gzip handling fix urllib 403s on the public JSON endpoint. HTTP 402 now triggers the OpenAI/public-JSON fallback chain when ScrapeCreators credits are exhausted.
- **xAI**: empty or malformed responses now surface in `errors_by_source` instead of silently returning zero results.
- **Windows**: process cleanup no longer crashes on `os.killpg`. POSIX-style secret-permission warnings skipped. Save-path footer uses forward slashes.
- **Auth**: comma-separated `SCRAPECREATORS_API_KEY=key1,key2` rotation restored (accidentally dropped in v3.0.6).
- **YouTube + HN**: SC YouTube + multi-token HN searches unblocked. Transcript-fetch ratio surfaced.
- **HTTP**: retry budget expanded with exponential backoff on DNS failure. Parallel AI search aligned with current API schema.
- **OpenClaw**: now works without a ScrapeCreators key. Poll-timing initialized once.
Say "eli5 on" after any research run. The synthesis rewrites in plain language. No jargon. Same data, same sources, same citations, just clearer. Say "eli5 off" to go back.
## Multi-harness reframe
### 13+ sources
`AGENTS.md` is now the canonical project doc; `CLAUDE.md` points at it. The skill is positioned as a multi-harness Agent Skills package, not a Claude-Code-specific tool. SKILL.md's path resolution rewrote `SKILL_ROOT``SKILL_DIR`, removing ~80 lines of bash and fixing a real spec-vs-engine divergence bug.
v3 adds Threads, Pinterest, Perplexity, Bluesky, and Parallel AI grounding to the existing Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, GitHub, and Web lineup. Perplexity Deep Research (`--deep-research`) gives you 50+ citation reports for serious investigation.
## Breaking change
### Per-author cap and entity disambiguation
Max 3 items per author prevents single-voice dominance. Synthesis trusts resolved handles over fuzzy keyword matches.
**`.codex-plugin/plugin.json` removed.** Codex native-plugin users should install via `npx skills add mvanhorn/last30days-skill` or copy the skill to `~/.codex/skills/last30days/`. The `npx skills add` path now reaches every harness uniformly.
## Install
Claude Code:
Any harness (recommended):
```
npx skills add mvanhorn/last30days-skill -g -y
```
Claude Code marketplace:
```
/plugin marketplace add mvanhorn/last30days-skill
@@ -58,29 +70,21 @@ OpenClaw:
clawhub install last30days-official
```
OpenAI Codex CLI: install the repo as a local Codex marketplace/plugin. The plugin manifest lives at `.codex-plugin/plugin.json`, and the canonical skill payload is `skills/last30days/SKILL.md`.
Zero config. Reddit, Hacker News, Polymarket, and GitHub work immediately. Run it once and the setup wizard unlocks X, YouTube, TikTok, and more in 30 seconds.
## v3 Community
## Contributors
v3 was shaped by community contributors whose PRs and issues inspired core features. Their code wasn't merged directly (v3 was a ground-up rewrite), but their ideas drove what shipped.
First-time contributors whose fixes shipped in v3.3.0 (most via PR triage salvage — the fix re-applied directly to main with co-author credit when path migration made the original branch un-rebaseable):
Thanks to @uppinote20, @zerone0x, @thinkun, @thomasmktong, @fanispoulinakisai-boop, @pejmanjohn, @zl190, and @hnshah. See [CONTRIBUTORS.md](CONTRIBUTORS.md) for the full list.
- Dave Morin — portable test-harness paths
- Alex Key — `removeprefix("r/")` for subreddit names
- Eric Oberhofer — multi-key rotation restored
- gujishh — Windows process cleanup
- Franco Carballar — Reddit browser-like headers
- Jonathan Oppenheim — Reddit 402 fallback chain
- Kaustav Mishra — xAI error surfacing
- [@thinkun](https://github.com/thinkun) — OpenClaw ScrapeCreators-key-optional fix
Contributors who shaped the release itself:
- @Jah-yee (#153) surfaced the need for a real Codex CLI integration, which shipped in #219
- @Cody-Coyote (#204) reported the marketplace validation bug that needed fixing before v3 could ship cleanly
- @dannyshmueli pushed for v3 and Codex family support publicly on X
Full Added / Changed / Fixed detail lives in [CHANGELOG.md](CHANGELOG.md) under `[3.0.0]`.
## Earlier contributors
From the v1 and v2 lineage:
- [@galligan](https://github.com/galligan) for marketplace plugin inspiration
- [@hutchins](https://x.com/hutchins) for pushing the YouTube feature
Plus every contributor who shipped one of the ~75 PRs merged this cycle. See [CHANGELOG.md](CHANGELOG.md) under `[3.3.0]` for the full PR list and `git log v3.2.0..v3.3.0` for the complete commit graph.
30 days of research. 30 seconds of work. Thirteen sources. Zero stale prompts.
+42 -69
View File
@@ -1,6 +1,6 @@
---
name: last30days
version: "3.2.3"
version: "3.3.0"
description: "Research what people actually say about any topic in the last 30 days. Pulls posts and engagement from Reddit, X, YouTube, TikTok, Hacker News, Polymarket, GitHub, and the web."
argument-hint: 'last30days nvidia earnings reaction | last30days AI video tools | last30days what users want in react'
allowed-tools: Bash, Read, Write, AskUserQuestion, WebSearch
@@ -13,9 +13,9 @@ metadata:
openclaw:
emoji: "📰"
requires:
env:
- SCRAPECREATORS_API_KEY
env: []
optionalEnv:
- SCRAPECREATORS_API_KEY
- OPENAI_API_KEY
- XAI_API_KEY
- OPENROUTER_API_KEY
@@ -97,7 +97,7 @@ You are inside the `/last30days` SKILL. This is a specific research tool with a
**How v3.0.7 fixes it:** three structural anchors.
1. **The MANDATORY first-line badge** (`🌐 last30days v{VERSION} · synced {YYYY-MM-DD}`) at the top of every response is the LAW 2 / LAW 4 enforcement anchor. See "BADGE (MANDATORY, FIRST LINE OF OUTPUT)" in the synthesis section.
2. **The SKILL_ROOT resolver** in the engine Bash calls walks a precedence list of known install locations and picks the highest-versioned freshest copy, never `~/.openclaw/` or other stale copies.
2. **The SKILL_DIR substitution** in the engine Bash calls uses the directory of the SKILL.md the model just Read — no resolver list, no precedence walk. Whichever install the harness loaded SKILL.md from is the install whose engine runs. Aligns spec-with-code and works for any harness without enumerating its install path.
3. **This preface** tells you plainly: do NOT improvise. Follow SKILL.md top to bottom.
If you catch yourself about to write a `##` section header in a GENERAL-query body, a custom title line, a `Sources:` bullet list, a `for dir in ...` path-discovery loop, or a bare `python3 scripts/last30days.py "{TOPIC}"` engine call with no pre-flight flags — stop. Those are the exact failure modes the LAWs and this contract exist to prevent. The 10/10 beta validation from 2026-04-18 and the 0/8 public v3.0.6 regression from the same day had THE SAME MODEL and SIMILAR SKILL.md CONTENT; the delta is the three anchors this release restores. Read SKILL.md top to bottom before emitting your first response.
@@ -114,7 +114,7 @@ These anchors used to live at line 1094 of this file. Three independent Opus 4.7
🌐 last30days v{VERSION} · synced {YYYY-MM-DD}
```
Replace `{VERSION}` with the installed plugin version (`jq -r '.version' "$SKILL_ROOT/../../.claude-plugin/plugin.json" 2>/dev/null || awk '/^version:/{gsub(/"/,"",$2); print $2; exit}' "$SKILL_ROOT/SKILL.md"`) and `{YYYY-MM-DD}` with today's date. No other text on this line. One blank line after, then the synthesis begins.
Replace `{VERSION}` with the installed plugin version (`jq -r '.version' "$SKILL_DIR/../../.claude-plugin/plugin.json" 2>/dev/null || awk '/^version:/{gsub(/"/,"",$2); print $2; exit}' "$SKILL_DIR/SKILL.md"`) and `{YYYY-MM-DD}` with today's date. No other text on this line. One blank line after, then the synthesis begins.
**Why the badge is MANDATORY:** it is the structural anchor for the canonical output shape. Without it the model drifts into blog-post narrative format with `##` section headers and invented titles, violating LAW 2 and LAW 4. The 2026-04-18 public v3.0.6 0/8 regression produced outputs with section headers like "The headline", "Why he is everywhere", "1. gstack dominates", "The 'Homecoming' peak". Direct cause: this anchor was absent. Do NOT skip the badge. Do NOT describe it. Do NOT paraphrase it. Emit it verbatim as line 1.
@@ -243,7 +243,7 @@ If your Bash call to `last30days.py` does NOT include the FULL pre-flight checkl
---
# last30days v3.2.3: Research Any Topic from the Last 30 Days
# last30days v3.3.0: Research Any Topic from the Last 30 Days
> **Permissions overview:** Reads public web/platform data and optionally saves research briefings to `LAST30DAYS_MEMORY_DIR` (defaults to `~/Documents/Last30Days`). X/Twitter search uses optional user-provided tokens (AUTH_TOKEN/CT0 env vars). Bluesky search uses optional app password (BSKY_HANDLE/BSKY_APP_PASSWORD env vars - create at bsky.app/settings/app-passwords). All credential usage and data writes are documented in the [Security & Permissions](#security--permissions) section.
@@ -330,12 +330,11 @@ Common patterns:
- If digg-pp-cli is installed (check `which digg-pp-cli`): add Digg
- If AUTH_TOKEN/CT0 or XAI_API_KEY or FROM_BROWSER is set, or xurl CLI is installed and authenticated: add X
- If yt-dlp is installed (check `which yt-dlp`): add YouTube
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains tiktok: add TikTok
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains instagram: add Instagram
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains threads: add Threads
- If SCRAPECREATORS_API_KEY is set and INCLUDE_SOURCES contains pinterest: add Pinterest
- If SCRAPECREATORS_API_KEY is set: add TikTok, Instagram, Threads (suppress any of these via EXCLUDE_SOURCES)
- If SCRAPECREATORS_API_KEY is set and the user explicitly requested pinterest for this query (e.g. via `--search=pinterest`): add Pinterest
- If BSKY_HANDLE and BSKY_APP_PASSWORD are set: add Bluesky
- If OPENROUTER_API_KEY is set: add Perplexity
- If OPENROUTER_API_KEY is set and INCLUDE_SOURCES contains perplexity: add Perplexity
- If EXCLUDE_SOURCES is set (comma-separated, case-insensitive): drop any matching source from the list above before displaying
Then display (use "and more" if 5+ sources, otherwise list all with Oxford comma):
@@ -592,27 +591,21 @@ When the user asks "X vs Y" (or "X vs Y vs Z"), the engine fans out N full `pipe
**Invocation:**
```bash
# Comparison mode skips Step 1, so resolve SKILL_ROOT inline here (same precedence
# walk as Step 1 — keep the two in sync if you edit either).
SKILL_ROOT=""
CLAUDE_PLUGIN_ROOT="$(find "$HOME/.claude/plugins/cache/last30days-skill/last30days" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | sort -V | tail -1)"
if [ -n "$CLAUDE_PLUGIN_ROOT" ]; then
if [ -f "$CLAUDE_PLUGIN_ROOT/skills/last30days/scripts/last30days.py" ]; then
SKILL_ROOT="$CLAUDE_PLUGIN_ROOT/skills/last30days"
elif [ -f "$CLAUDE_PLUGIN_ROOT/scripts/last30days.py" ]; then
SKILL_ROOT="$CLAUDE_PLUGIN_ROOT"
fi
fi
if [ -z "$SKILL_ROOT" ] || [ ! -f "$SKILL_ROOT/scripts/last30days.py" ]; then
for dir in \
"$HOME/.codex/skills/last30days" \
"$HOME/.agents/skills/last30days" \
"./skills/last30days" \
"./.skills/last30days" \
"." \
"${GEMINI_EXTENSION_DIR:-}"; do
[ -n "$dir" ] && [ -f "$dir/scripts/last30days.py" ] && SKILL_ROOT="$dir" && break
done
# SKILL_DIR = absolute path of the directory containing THIS SKILL.md you just Read.
# Substitute the actual path below — your harness told you where this file lives via
# the Read tool result. Examples:
# Read ~/.claude/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.claude/skills/last30days
# Read ~/.codex/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.codex/skills/last30days
# Read ~/.claude/plugins/cache/last30days-skill/last30days/3.3.0/skills/last30days/SKILL.md
# SKILL_DIR=$HOME/.claude/plugins/cache/last30days-skill/last30days/3.3.0/skills/last30days
# scripts/last30days.py is always a direct child of SKILL_DIR (every install layout
# packages SKILL.md and scripts/ as siblings).
SKILL_DIR="<absolute path of the directory containing the SKILL.md you Read>"
if [ ! -f "$SKILL_DIR/scripts/last30days.py" ]; then
echo "ERROR: scripts/last30days.py not found under SKILL_DIR=$SKILL_DIR" >&2
echo "Re-check the directory of the SKILL.md you Read and substitute it as SKILL_DIR above." >&2
exit 1
fi
# Write the per-entity plan to a tmpfile and pass the path to the engine.
@@ -631,7 +624,7 @@ cat > "$COMPETITORS_PLAN_FILE" <<'PLAN_EOF'
}
PLAN_EOF
"${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" "{TOPIC_A} vs {TOPIC_B} vs {TOPIC_C}" \
"${LAST30DAYS_PYTHON}" "${SKILL_DIR}/scripts/last30days.py" "{TOPIC_A} vs {TOPIC_B} vs {TOPIC_C}" \
--emit=compact \
--save-dir="${LAST30DAYS_MEMORY_DIR}" \
--save-suffix=v3 \
@@ -916,44 +909,24 @@ Store your plan as `QUERY_PLAN_JSON` - you'll pass it to the script in the next
**IMPORTANT: Include `--x-handle={RESOLVED_HANDLE}` in the command. For comparison mode: Pass `--x-handle={TOPIC_A_HANDLE}` to the first pass, `--x-handle={TOPIC_B_HANDLE}` to the second pass, and both to the head-to-head pass. Also include `--subreddits={RESOLVED_SUBREDDITS}`, `--tiktok-hashtags={RESOLVED_HASHTAGS}`, `--tiktok-creators={RESOLVED_TIKTOK_CREATORS}`, and `--ig-creators={RESOLVED_IG_CREATORS}` from Step 0.55. Omit any flag where the value was not resolved (empty).**
```bash
# Resolve SKILL_ROOT by walking a precedence list of known install locations.
# Claude Code plugin cache wins when present (highest version dir picked on upgrade),
# then common per-harness skill dirs, then a repo checkout.
SKILL_ROOT=""
# SKILL_DIR = absolute path of the directory containing THIS SKILL.md you just Read.
# Substitute the actual path below — your harness told you where this file lives via
# the Read tool result. Examples:
# Read ~/.claude/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.claude/skills/last30days
# Read ~/.codex/skills/last30days/SKILL.md → SKILL_DIR=$HOME/.codex/skills/last30days
# Read ~/.claude/plugins/cache/last30days-skill/last30days/3.3.0/skills/last30days/SKILL.md
# → SKILL_DIR=$HOME/.claude/plugins/cache/last30days-skill/last30days/3.3.0/skills/last30days
# scripts/last30days.py is always a direct child of SKILL_DIR (every install layout
# packages SKILL.md and scripts/ as siblings).
SKILL_DIR="<absolute path of the directory containing the SKILL.md you Read>"
# 1. Claude Code plugin cache (versioned, sort -V picks freshest). Two cache layouts ship in the wild:
# nested ({cache}/{version}/skills/last30days/scripts/...) and flat ({cache}/{version}/scripts/...).
# `find` (not `ls + glob`) because zsh errors on globs that match nothing, leaking
# noisy "no matches found" stderr in Codex/zsh sessions even with 2>/dev/null.
CLAUDE_PLUGIN_ROOT="$(find "$HOME/.claude/plugins/cache/last30days-skill/last30days" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | sort -V | tail -1)"
if [ -n "$CLAUDE_PLUGIN_ROOT" ]; then
if [ -f "$CLAUDE_PLUGIN_ROOT/skills/last30days/scripts/last30days.py" ]; then
SKILL_ROOT="$CLAUDE_PLUGIN_ROOT/skills/last30days"
elif [ -f "$CLAUDE_PLUGIN_ROOT/scripts/last30days.py" ]; then
SKILL_ROOT="$CLAUDE_PLUGIN_ROOT"
fi
fi
# 2. Common per-harness skill dirs and repo checkout (npx skills, Codex, Agents, Gemini, etc).
if [ -z "$SKILL_ROOT" ] || [ ! -f "$SKILL_ROOT/scripts/last30days.py" ]; then
for dir in \
"$HOME/.codex/skills/last30days" \
"$HOME/.agents/skills/last30days" \
"./skills/last30days" \
"./.skills/last30days" \
"." \
"${GEMINI_EXTENSION_DIR:-}"; do
[ -n "$dir" ] && [ -f "$dir/scripts/last30days.py" ] && SKILL_ROOT="$dir" && break
done
fi
if [ -z "${SKILL_ROOT:-}" ] || [ ! -f "$SKILL_ROOT/scripts/last30days.py" ]; then
echo "ERROR: Could not find scripts/last30days.py in any known install location" >&2
echo "Searched: ~/.claude/plugins/cache/, ~/.codex/skills/, ~/.agents/skills/, ./skills/last30days, ./.skills/last30days, ." >&2
if [ ! -f "$SKILL_DIR/scripts/last30days.py" ]; then
echo "ERROR: scripts/last30days.py not found under SKILL_DIR=$SKILL_DIR" >&2
echo "Re-check the directory of the SKILL.md you Read and substitute it as SKILL_DIR above." >&2
exit 1
fi
"${LAST30DAYS_PYTHON}" "${SKILL_ROOT}/scripts/last30days.py" $ARGUMENTS --emit=compact --save-dir="${LAST30DAYS_MEMORY_DIR}" --save-suffix=v3
"${LAST30DAYS_PYTHON}" "${SKILL_DIR}/scripts/last30days.py" $ARGUMENTS --emit=compact --save-dir="${LAST30DAYS_MEMORY_DIR}" --save-suffix=v3
```
**If you ran Steps 0.55 and 0.75 (agent planning), pass the plan via a tmpfile and add the targeting flags:**
@@ -1715,7 +1688,7 @@ Want another prompt? Just tell me what you're creating next.
- Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)
- Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth)
- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (PAYG after 10,000 free API calls)
- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (PAYG after 100 free credits)
- Optionally sends search queries to Brave Search API, Parallel AI API, or OpenRouter API for web search
- Fetches public Reddit thread data from `reddit.com` for engagement metrics
- Stores research findings in local SQLite database (watchlist mode only)
@@ -1728,7 +1701,7 @@ Want another prompt? Just tell me what you're creating next.
- Does not log, cache, or write API keys to output files
- Does not send data to any endpoint not listed above
- Hacker News and Polymarket sources are always available (no API key, no binary dependency)
- TikTok and Instagram sources require SCRAPECREATORS_API_KEY (10,000 free API calls, then PAYG). Reddit uses ScrapeCreators only as a backup when public Reddit is unavailable.
- TikTok and Instagram sources require SCRAPECREATORS_API_KEY (100 free credits one-time, then PAYG). Reddit uses ScrapeCreators only as a backup when public Reddit is unavailable.
- Can be invoked autonomously by agents via the Skill tool (runs inline, not forked); pass `--agent` for non-interactive report output
**Bundled scripts:** `scripts/last30days.py` (main research engine), `scripts/lib/` (search, enrichment, rendering modules), `scripts/lib/vendor/bird-search/` (vendored X search client, MIT licensed)
@@ -20,6 +20,7 @@ sys.path.insert(0, str(Path(__file__).parent))
from lib import env as envlib
from lib import schema
from lib.providers import GEMINI_FLASH_LITE
SKILL_ROOT = Path(__file__).resolve().parents[1]
@@ -43,7 +44,7 @@ def _load_default_topics() -> list[tuple[str, str]]:
DEFAULT_TOPICS = _load_default_topics()
DEFAULT_SEARCH = ""
DEFAULT_JUDGE_MODEL = "gemini-3.1-flash-lite-preview"
DEFAULT_JUDGE_MODEL = GEMINI_FLASH_LITE
GEMINI_API_URL = "https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent?key={api_key}"
+109 -11
View File
@@ -1,11 +1,12 @@
#!/usr/bin/env python3
# ruff: noqa: E402
"""last30days v3.0.0 CLI."""
"""last30days CLI."""
from __future__ import annotations
import argparse
import atexit
import datetime
import json
import os
import re
@@ -62,7 +63,10 @@ def _cleanup_children() -> None:
pids = list(_child_pids)
for pid in pids:
try:
os.killpg(os.getpgid(pid), signal.SIGTERM)
if hasattr(os, "killpg"):
os.killpg(os.getpgid(pid), signal.SIGTERM)
else:
os.kill(pid, signal.SIGTERM)
except (ProcessLookupError, PermissionError, OSError):
continue
@@ -97,11 +101,13 @@ def save_output(
save_dir: str,
suffix: str = "",
synthesis_md: str | None = None,
topic_override: str | None = None,
rendered_content: str | None = None,
) -> Path:
from datetime import datetime
path = Path(save_dir).expanduser().resolve()
path.mkdir(parents=True, exist_ok=True)
slug = slugify(report.topic)
slug = slugify(topic_override or report.topic)
extension = "json" if emit == "json" else "html" if emit == "html" else "md"
raw_label = "raw-html" if emit == "html" else "raw"
suffix_part = f"-{suffix}" if suffix else ""
@@ -110,7 +116,9 @@ def save_output(
out_path = path / f"{slug}-{raw_label}{suffix_part}-{datetime.now().strftime('%Y-%m-%d')}.{extension}"
# Markdown saves keep the complete debug artifact. JSON and HTML preserve
# their requested wire format so file extensions match their content.
if emit in {"json", "html"}:
if rendered_content is not None:
content = rendered_content
elif emit in {"json", "html"}:
content = emit_output(report, emit, synthesis_md=synthesis_md)
else:
content = render.render_full(report)
@@ -171,6 +179,10 @@ def emit_comparison_output(
raise SystemExit(f"Unsupported emit mode: {emit}")
def comparison_topic(entity_reports: list[tuple[str, schema.Report]]) -> str:
return " vs ".join(label for label, _ in entity_reports)
def compute_save_path_display(save_dir: str, topic: str, suffix: str, emit: str) -> str:
"""Compute the user-friendly save path string that will be shown in the footer.
@@ -187,9 +199,9 @@ def compute_save_path_display(save_dir: str, topic: str, suffix: str, emit: str)
try:
home = _Path.home().resolve()
relative = raw.relative_to(home)
return f"~/{relative}"
return f"~/{relative.as_posix()}"
except ValueError:
return str(raw)
return raw.as_posix()
def read_synthesis_file(path: str) -> str:
@@ -379,7 +391,7 @@ def subrun_kwargs_for(
subreddits = _choose("subreddits", "subreddits")
if isinstance(subreddits, list):
subreddits = [s.strip().lstrip("r/") for s in subreddits if s.strip()] or None
subreddits = [s.strip().removeprefix("r/") for s in subreddits if s.strip()] or None
x_related = plan_entry.get("x_related")
if isinstance(x_related, list):
@@ -523,6 +535,24 @@ def _show_runtime_ui(
progress.show_promo(promo, diag=diag)
def _write_last_run(topic: str, report: "schema.Report") -> None:
try:
if env.CONFIG_DIR is None:
return
target = env.CONFIG_DIR
target.mkdir(parents=True, exist_ok=True)
counts = {source: len(items) for source, items in report.items_by_source.items()}
payload = {
"topic": topic,
"timestamp": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"sources": counts,
"total": sum(counts.values()),
}
(target / "last-run.json").write_text(json.dumps(payload, indent=2))
except Exception:
pass
def main() -> int:
parser = build_parser()
# Use parse_known_args so setup sub-flags (--device-auth, --github,
@@ -533,6 +563,13 @@ def main() -> int:
config = env.get_config()
# Surface SSH-routing config as an env var so library modules (e.g.
# youtube_yt) can read it without taking a config dependency. This
# routes yt-dlp through `ssh <host>` to bypass YouTube's bot-wall on
# datacenter IPs (see lib/youtube_yt.py for details).
if config.get("LAST30DAYS_YOUTUBE_SSH_HOST") and "LAST30DAYS_YOUTUBE_SSH_HOST" not in os.environ:
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = config["LAST30DAYS_YOUTUBE_SSH_HOST"]
# Handle setup subcommand
topic = " ".join(args.topic).strip()
if topic.lower() == "setup":
@@ -591,7 +628,7 @@ def main() -> int:
depth = "deep" if args.deep else "quick" if args.quick else "default"
try:
x_related = [h.strip() for h in args.x_related.split(",") if h.strip()] if args.x_related else None
subreddits = [s.strip().lstrip("r/") for s in args.subreddits.split(",") if s.strip()] if args.subreddits else None
subreddits = [s.strip().removeprefix("r/") for s in args.subreddits.split(",") if s.strip()] if args.subreddits else None
tiktok_hashtags = [h.strip().lstrip("#") for h in args.tiktok_hashtags.split(",") if h.strip()] if args.tiktok_hashtags else None
tiktok_creators = [c.strip().lstrip("@") for c in args.tiktok_creators.split(",") if c.strip()] if args.tiktok_creators else None
ig_creators = [c.strip().lstrip("@") for c in args.ig_creators.split(",") if c.strip()] if args.ig_creators else None
@@ -610,6 +647,7 @@ def main() -> int:
# Auto-resolve: use web search to discover subreddits/handles before planning.
# This is the engine-side equivalent of SKILL.md Steps 0.55/0.75 for platforms
# without WebSearch (OpenClaw, Codex, raw CLI).
repos_from_auto_resolve = False
if args.auto_resolve and not external_plan:
from lib import resolve
resolution = resolve.auto_resolve(topic, config)
@@ -624,6 +662,9 @@ def main() -> int:
sys.stderr.write(f"[AutoResolve] GitHub user: @{args.github_user}\n")
if resolution.get("github_repos") and not args.github_repo:
args.github_repo = ",".join(resolution["github_repos"])
# auto_resolve already canonicalized via canonicalize_github_repos(cap=5);
# mark so we don't re-canonicalize below and clobber its relevance order.
repos_from_auto_resolve = True
sys.stderr.write(f"[AutoResolve] GitHub repos: {args.github_repo}\n")
if resolution.get("context"):
# Inject context into external_plan metadata for the planner to use
@@ -636,6 +677,20 @@ def main() -> int:
github_user = args.github_user.lstrip("@").lower() if args.github_user else None
github_repos = [r.strip() for r in args.github_repo.split(",") if r.strip() and "/" in r.strip()] if args.github_repo else None
# Only canonicalize when repos came from a user-supplied --github-repo flag.
# When repos_from_auto_resolve is True, auto_resolve already ran
# canonicalize_github_repos(cap=5) and ranked by relevance; re-running here
# with cap=None can re-sort by topic-slug match and lose that ordering.
if github_repos and not repos_from_auto_resolve:
from lib import resolve as resolve_lib
original_github_repos = github_repos[:]
github_repos = resolve_lib.canonicalize_github_repos(topic, github_repos, cap=None)
if github_repos != original_github_repos:
sys.stderr.write(
"[GitHub] Canonicalized repos: "
f"{','.join(original_github_repos)} -> {','.join(github_repos)}\n"
)
# --deep-research: auto-enable perplexity source and set deep flag
if args.deep_research:
if not config.get("OPENROUTER_API_KEY"):
@@ -855,7 +910,18 @@ def main() -> int:
report, progress, diag,
suppress_web_promo=bool(external_plan or comp_plan),
)
if args.store:
_write_last_run(topic, report)
# LAST30DAYS_STORE env var = persistence default-on. Read both os.environ
# (for shell-exported users) and config (for users who set it in
# ~/.config/last30days/.env, which env.py loads but does not propagate
# to os.environ). Mirrors the LAST30DAYS_DEBUG / LAST30DAYS_SKIP_PREFLIGHT
# convention; env-var or config wins, with `--store` flag still working.
_store_env = (
os.environ.get("LAST30DAYS_STORE")
or config.get("LAST30DAYS_STORE")
or ""
).lower()
if args.store or _store_env in ("1", "true", "yes"):
counts = persist_report(report)
sys.stderr.write(
f"[last30days] Stored {counts['new']} new, {counts['updated']} updated findings\n"
@@ -865,7 +931,32 @@ def main() -> int:
# Show quality nudge if applicable
try:
from lib import quality_nudge
quality = quality_nudge.compute_quality_score(config, {})
# Populate transcript-fetch ratio so quality_nudge can detect the
# degraded-YouTube failure mode (videos returned but transcripts
# silently failed - typically a stale yt-dlp binary).
youtube_items = report.items_by_source.get("youtube") or []
instagram_items = report.items_by_source.get("instagram") or []
research_results = {
"youtube_videos_count": len(youtube_items),
"youtube_transcripts_count": sum(
1 for it in youtube_items
if (it.metadata.get("transcript_highlights") or it.metadata.get("transcript_snippet"))
),
"youtube_error": report.errors_by_source.get("youtube"),
"x_error": report.errors_by_source.get("x"),
# Captions-disabled videos can never produce a transcript regardless
# of yt-dlp version; subtract them from the degraded-ratio
# denominator so a single uploader-disabled video does not trip the
# "stale yt-dlp" nudge.
"youtube_captions_disabled_count": sum(
1 for it in youtube_items if it.metadata.get("captions_disabled")
),
# Track Instagram returned-zero-items so quality_nudge can detect
# the silent-failure case (SC configured but the v2 reels endpoint
# 500'd through both the original query and the hashtag retry).
"instagram_items_count": len(instagram_items),
}
quality = quality_nudge.compute_quality_score(config, research_results)
if quality.get("nudge_text"):
sys.stderr.write(f"\n{quality['nudge_text']}\n")
sys.stderr.flush()
@@ -873,10 +964,15 @@ def main() -> int:
pass
fun_level = config.get("FUN_LEVEL", "medium").lower()
# Comparison HTML is the one case where the saved file's title and content
# have to be overridden away from the leading entity's report. Compute the
# gate once so the footer-display and save-output paths can't disagree.
is_comparison_html = bool(entity_reports) and args.emit == "html"
footer_save_path = None
if args.save_dir:
save_topic_for_display = comparison_topic(entity_reports) if is_comparison_html else report.topic
footer_save_path = compute_save_path_display(
args.save_dir, report.topic, args.save_suffix or "", args.emit
args.save_dir, save_topic_for_display, args.save_suffix or "", args.emit
)
# Signal to render_compact whether pre-research flags were supplied.
@@ -917,6 +1013,8 @@ def main() -> int:
args.save_dir,
suffix=args.save_suffix or "",
synthesis_md=synthesis_md,
topic_override=comparison_topic(entity_reports) if is_comparison_html else None,
rendered_content=rendered if is_comparison_html else None,
)
sys.stderr.write(f"[last30days] Saved output to {save_path}\n")
# Competitor / vs-mode: also save a per-entity raw file for each peer.
+86 -24
View File
@@ -9,6 +9,7 @@ import json
import os
import shutil
import sys
import time
from pathlib import Path
from . import http, log, subproc
@@ -17,6 +18,11 @@ from typing import Any, Dict, List, Optional, Tuple
from .relevance import token_overlap_relevance as _compute_relevance
# How many times to retry the bird-search subprocess when stdout is non-JSON
# (typically an HTML anti-bot interstitial from Twitter's edge).
MAX_JSON_DECODE_RETRIES = 2
JSON_DECODE_RETRY_DELAY = 5.0 # seconds between retry attempts
def _first_of(*values):
"""Return first value that is not None."""
@@ -148,16 +154,14 @@ def get_bird_status() -> Dict[str, Any]:
}
def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
"""Run a search using the vendored bird-search.mjs module.
def _invoke_bird_subprocess(query: str, count: int, timeout: int):
"""Invoke the vendored bird-search.mjs subprocess once.
Args:
query: Full search query string (including since: filter)
count: Number of results to request
timeout: Timeout in seconds
Returns:
Raw Bird JSON response or error dict.
Returns (result, error_dict). If error_dict is non-None, treat it as the
final result and do not retry those errors are terminal (timeout,
spawn failure). If error_dict is None, the subprocess ran to completion
and `result` is the SubprocResult; the caller decides whether to retry
based on the result.stdout content.
"""
cmd = [
"node", str(_BIRD_SEARCH_MJS),
@@ -184,9 +188,9 @@ def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
on_pid=_register,
)
except subproc.SubprocTimeout:
return {"error": f"Search timed out after {timeout}s", "items": []}
return None, {"error": f"Search timed out after {timeout}s", "items": []}
except Exception as e:
return {"error": str(e), "items": []}
return None, {"error": str(e), "items": []}
finally:
if pid_holder:
try:
@@ -195,22 +199,80 @@ def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
except Exception:
pass
if result.returncode != 0:
error = result.stderr.strip() or "Bird search failed"
return {"error": error, "items": []}
return result, None
output = result.stdout.strip()
if not output:
return {"items": []}
try:
parsed = json.loads(output)
except json.JSONDecodeError as e:
return {"error": f"Invalid JSON response: {e}", "items": []}
def _run_bird_search(query: str, count: int, timeout: int) -> Dict[str, Any]:
"""Run a search using the vendored bird-search.mjs module.
if isinstance(parsed, list):
return {"items": parsed}
return parsed
Retries the subprocess on JSON-decode failure (typically a Twitter
anti-bot HTML interstitial in stdout) up to MAX_JSON_DECODE_RETRIES
times with JSON_DECODE_RETRY_DELAY seconds between attempts. Terminal
errors (subprocess timeout, non-zero return code) are returned
immediately without retry.
Args:
query: Full search query string (including since: filter)
count: Number of results to request
timeout: Timeout in seconds (per attempt)
Returns:
Raw Bird JSON response or error dict.
"""
last_decode_error: Optional[str] = None
for attempt in range(MAX_JSON_DECODE_RETRIES):
result, terminal_error = _invoke_bird_subprocess(query, count, timeout)
if terminal_error is not None:
return terminal_error
if result.returncode != 0:
error = result.stderr.strip() or "Bird search failed"
return {"error": error, "items": []}
output = result.stdout.strip()
if not output:
return {"items": []}
try:
parsed = json.loads(output)
except json.JSONDecodeError as e:
# Twitter's edge sometimes serves an HTML anti-bot interstitial
# in place of JSON. Tag the failure shape so it's distinguishable
# from "no results" in logs, then retry the subprocess.
looks_html = output.lstrip().lower().startswith(("<!doctype", "<html", "<"))
attempt_num = attempt + 1
log_msg = (
f"Bird search returned non-JSON stdout "
f"(looks_html={looks_html}, attempt {attempt_num}/{MAX_JSON_DECODE_RETRIES}, "
f"first 80 chars: {output[:80]!r})"
)
last_decode_error = str(e)
if attempt_num < MAX_JSON_DECODE_RETRIES:
log.source_log(
"X/bird",
f"{log_msg}; retrying in {JSON_DECODE_RETRY_DELAY:.0f}s",
)
time.sleep(JSON_DECODE_RETRY_DELAY)
continue
log.source_log("X/bird", log_msg)
return {
"error": (
f"Invalid JSON response after {MAX_JSON_DECODE_RETRIES} attempts "
f"(likely Twitter anti-bot interstitial): {e}"
),
"items": [],
}
if isinstance(parsed, list):
return {"items": parsed}
return parsed
# Defensive fallthrough — loop should always return above.
return {
"error": f"Bird search exhausted retries: {last_decode_error}",
"items": [],
}
def search_x(
+84 -4
View File
@@ -1,10 +1,19 @@
"""Bluesky search via AT Protocol (requires app password).
Uses bsky.social for auth and public.api.bsky.app for post search.
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars.
Uses bsky.social for auth and api.bsky.app for post search (the canonical
authenticated AppView). The previous default `public.api.bsky.app` is the
unauthenticated public mirror, which BunnyCDN now blocks for searchPosts
regardless of auth header (verified 2026-05-04). Override the search host
via BSKY_SEARCH_HOST env var if Bluesky migrates infrastructure again.
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars. App passwords are
19-char xxxx-xxxx-xxxx-xxxx; generate at bsky.app/settings/app-passwords.
The createSession endpoint accepts main-account passwords too, but they're
bad hygiene (no scope, can't revoke individually).
"""
import math
import os
import re
import sys
import time
@@ -14,7 +23,64 @@ from typing import Any, Dict, List, Optional
from . import http, log
BSKY_SESSION_URL = "https://bsky.social/xrpc/com.atproto.server.createSession"
BSKY_SEARCH_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts"
_DEFAULT_BSKY_SEARCH_HOST = "api.bsky.app"
def _resolve_search_url(config: Optional[Dict[str, Any]] = None) -> str:
"""Resolve the Bluesky search URL with BSKY_SEARCH_HOST override.
Default is api.bsky.app. Override via BSKY_SEARCH_HOST in shell env or
.env file. The project's env.py loads .env into config but not into
os.environ, so check both same hybrid pattern as last30days.py for
LAST30DAYS_STORE.
Hardens user-supplied host values against three common mis-configurations:
whitespace (e.g. " api.bsky.app "), embedded path components (e.g.
"api.bsky.app/xrpc/proxy") that would double the /xrpc/ segment, and
embedded scheme prefixes (e.g. "https://api.bsky.app"). On any of these
we log a warning and fall back to the default rather than building an
invalid URL with an opaque downstream error.
"""
config = config or {}
raw = (
os.environ.get("BSKY_SEARCH_HOST")
or config.get("BSKY_SEARCH_HOST")
or _DEFAULT_BSKY_SEARCH_HOST
)
host = raw.strip().rstrip("/")
# Strip embedded scheme so users who paste full URLs do not break the f-string.
for prefix in ("https://", "http://"):
if host.lower().startswith(prefix):
host = host[len(prefix):]
break
if not host or "/" in host or " " in host:
# Embedded path or whitespace remains — don't trust it. Default + log.
if raw != _DEFAULT_BSKY_SEARCH_HOST:
_log(
f"BSKY_SEARCH_HOST={raw!r} is not a bare hostname; "
f"falling back to default {_DEFAULT_BSKY_SEARCH_HOST!r}"
)
host = _DEFAULT_BSKY_SEARCH_HOST
return f"https://{host}/xrpc/app.bsky.feed.searchPosts"
# App-password format: xxxx-xxxx-xxxx-xxxx (19 chars, lowercase alphanumeric
# with three hyphens at fixed positions).
_APP_PASSWORD_RE = re.compile(r"^[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}$")
def _validate_app_password_format(value) -> bool:
"""Return True if value matches Bluesky's 19-char app-password format.
False for non-strings (None, int, list) so callers passing config dict
values directly don't crash. Detect-but-not-gate: the createSession
endpoint also accepts main-account passwords, so failing this check is
a hygiene smell, not a hard error.
"""
if not isinstance(value, str):
return False
return bool(_APP_PASSWORD_RE.fullmatch(value))
DEPTH_CONFIG = {
"quick": 15,
@@ -144,6 +210,20 @@ def search_bluesky(
if not handle or not app_password:
return {"posts": [], "error": "Bluesky credentials not configured"}
# One-shot hygiene warning if BSKY_APP_PASSWORD is not in app-password
# form. createSession accepts main-account passwords too — but main
# passwords have no scope (full account access), can't be revoked
# individually, and rotating them breaks every service that holds them.
# We warn but do not gate, matching the project's detect-don't-block
# philosophy elsewhere.
if not _validate_app_password_format(app_password):
_log(
"BSKY_APP_PASSWORD does not look like an app password "
"(expected xxxx-xxxx-xxxx-xxxx, 19 chars). It may be a main "
"account password — those work but are bad hygiene. Generate "
"an app password at https://bsky.app/settings/app-passwords"
)
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
@@ -155,7 +235,7 @@ def search_bluesky(
"limit": str(min(count, 100)),
"sort": "top",
}
url = f"{BSKY_SEARCH_URL}?{urlencode(params)}"
url = f"{_resolve_search_url(config)}?{urlencode(params)}"
def _auth_and_search() -> tuple[Optional[Dict[str, Any]], Optional[str]]:
token = _create_session(handle, app_password)
+90 -33
View File
@@ -1,9 +1,12 @@
"""Chrome cookie extraction for macOS.
"""Chrome and Brave cookie extraction for macOS.
Extracts cookies from Chrome's encrypted SQLite database using only stdlib
modules and the system openssl CLI (ships with macOS). Zero pip dependencies.
Extracts cookies from Chromium-based browser SQLite databases using only
stdlib modules and the system openssl CLI (ships with macOS). Zero pip
dependencies.
Chrome on macOS uses v10 encryption (AES-128-CBC with Keychain-stored key).
Chromium on macOS uses v10 encryption (AES-128-CBC with Keychain-stored key).
Chrome and Brave share the same algorithm; only the DB path and Keychain
service name differ.
This is NOT affected by Windows App-Bound Encryption (v20).
"""
@@ -18,10 +21,11 @@ from typing import Optional
logger = logging.getLogger(__name__)
# Chrome cookie DB location on macOS
# Cookie DB locations on macOS
CHROME_COOKIES_DB = Path.home() / "Library" / "Application Support" / "Google" / "Chrome" / "Default" / "Cookies"
BRAVE_BASE_DIR = Path.home() / "Library" / "Application Support" / "BraveSoftware" / "Brave-Browser"
# Chrome v10 encryption constants
# Chromium v10 encryption constants (shared by Chrome and Brave)
CHROME_SALT = b"saltysalt"
CHROME_PBKDF2_ITERATIONS = 1003
CHROME_KEY_LENGTH = 16
@@ -29,8 +33,8 @@ CHROME_KEY_LENGTH = 16
CHROME_IV_HEX = "20" * 16
def _get_chrome_encryption_key() -> Optional[bytes]:
"""Retrieve Chrome's encryption passphrase from macOS Keychain.
def _get_chromium_encryption_key(service_name: str) -> Optional[bytes]:
"""Retrieve the encryption passphrase for a Chromium-based browser from macOS Keychain.
Calls `security find-generic-password` which may trigger a system dialog
on first access.
@@ -39,30 +43,34 @@ def _get_chrome_encryption_key() -> Optional[bytes]:
"""
try:
result = subprocess.run(
["security", "find-generic-password", "-w", "-s", "Chrome Safe Storage"],
["security", "find-generic-password", "-w", "-s", service_name],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode != 0:
logger.info("Chrome Keychain access denied or Chrome not installed: %s", result.stderr.strip())
logger.info("%s Keychain access denied or browser not installed: %s", service_name, result.stderr.strip())
return None
passphrase = result.stdout.strip()
if not passphrase:
logger.info("Chrome Keychain returned empty passphrase")
logger.info("%s Keychain returned empty passphrase", service_name)
return None
return passphrase.encode("utf-8")
except FileNotFoundError:
logger.info("'security' command not found — not on macOS?")
return None
except subprocess.TimeoutExpired:
logger.info("Chrome Keychain access timed out")
logger.info("%s Keychain access timed out", service_name)
return None
except Exception as e:
logger.info("Failed to get Chrome encryption key: %s", e)
logger.info("Failed to get %s encryption key: %s", service_name, e)
return None
def _get_chrome_encryption_key() -> Optional[bytes]:
return _get_chromium_encryption_key("Chrome Safe Storage")
def _derive_aes_key(passphrase: bytes) -> bytes:
"""Derive 16-byte AES key from Chrome's Keychain passphrase via PBKDF2."""
return hashlib.pbkdf2_hmac(
@@ -165,36 +173,42 @@ def _get_db_version(cursor: sqlite3.Cursor) -> int:
return 0
def extract_chrome_cookies_macos(domain: str, cookie_names: list[str]) -> Optional[dict[str, str]]:
"""Extract cookies from Chrome on macOS.
def _extract_chromium_cookies_macos(
db_path: Path,
keychain_service: str,
domain: str,
cookie_names: list[str],
) -> Optional[dict[str, str]]:
"""Extract cookies from any Chromium-based browser on macOS.
Copies the locked Cookies database to a temp file, reads specified cookies,
and decrypts v10-encrypted values using the Keychain-stored key.
Args:
domain: Cookie domain to match (e.g., ".twitter.com", ".x.com")
cookie_names: List of cookie names to extract
db_path: Path to the browser's Cookies SQLite file.
keychain_service: macOS Keychain service name (e.g. "Chrome Safe Storage").
domain: Cookie domain to match (e.g., ".twitter.com", ".x.com").
cookie_names: List of cookie names to extract.
Returns:
Dict mapping cookie name to decrypted value, or None on failure.
Only includes cookies that were successfully found and decrypted.
"""
if not CHROME_COOKIES_DB.exists():
logger.info("Chrome cookies database not found at %s", CHROME_COOKIES_DB)
if not db_path.exists():
logger.info("%s cookies database not found at %s", keychain_service, db_path)
return None
# Get encryption key from Keychain
passphrase = _get_chrome_encryption_key()
passphrase = _get_chromium_encryption_key(keychain_service)
aes_key = _derive_aes_key(passphrase) if passphrase else None
# Copy DB to temp file (Chrome locks the original)
# Copy DB to temp file (browser locks the original while running)
tmp_fd = None
tmp_path = None
try:
tmp_fd, tmp_path = tempfile.mkstemp(suffix=".sqlite")
shutil.copy2(str(CHROME_COOKIES_DB), tmp_path)
shutil.copy2(str(db_path), tmp_path)
except Exception as e:
logger.info("Failed to copy Chrome cookies database: %s", e)
logger.info("Failed to copy %s cookies database: %s", keychain_service, e)
if tmp_path:
try:
Path(tmp_path).unlink(missing_ok=True)
@@ -211,26 +225,22 @@ def extract_chrome_cookies_macos(domain: str, cookie_names: list[str]) -> Option
cursor = conn.cursor()
db_version = _get_db_version(cursor)
logger.debug("Chrome cookie DB version: %d", db_version)
logger.debug("%s cookie DB version: %d", keychain_service, db_version)
# Build query with placeholders for cookie names
placeholders = ",".join("?" for _ in cookie_names)
query = (
f"SELECT name, value, encrypted_value FROM cookies "
f"WHERE host_key LIKE ? AND name IN ({placeholders})"
)
# Use LIKE for domain matching (e.g., %.twitter.com matches .twitter.com)
params = [f"%{domain}"] + list(cookie_names)
cursor.execute(query, params)
results: dict[str, str] = {}
for name, value, encrypted_value in cursor.fetchall():
# Prefer unencrypted value if present
if value:
results[name] = value
continue
# Handle encrypted value
if encrypted_value and encrypted_value[:3] == b"v10":
if aes_key is None:
logger.debug("Skipping encrypted cookie %s — no Keychain access", name)
@@ -241,25 +251,72 @@ def extract_chrome_cookies_macos(domain: str, cookie_names: list[str]) -> Option
else:
logger.debug("Failed to decrypt cookie %s", name)
elif encrypted_value:
# Unknown encryption version
logger.debug("Unknown encryption for cookie %s (prefix: %r)", name, encrypted_value[:3])
conn.close()
if not results:
logger.info("No matching cookies found in Chrome for domain %s", domain)
logger.info("No matching cookies found in %s for domain %s", keychain_service, domain)
return None
return results
except sqlite3.Error as e:
logger.info("Failed to read Chrome cookies database: %s", e)
logger.info("Failed to read %s cookies database: %s", keychain_service, e)
return None
except Exception as e:
logger.info("Unexpected error reading Chrome cookies: %s", e)
logger.info("Unexpected error reading %s cookies: %s", keychain_service, e)
return None
finally:
try:
Path(tmp_path).unlink(missing_ok=True)
except Exception:
pass
def extract_chrome_cookies_macos(domain: str, cookie_names: list[str]) -> Optional[dict[str, str]]:
"""Extract cookies from Chrome on macOS."""
return _extract_chromium_cookies_macos(
CHROME_COOKIES_DB, "Chrome Safe Storage", domain, cookie_names
)
def _find_brave_cookies_db() -> Optional[Path]:
"""Find Brave's Cookies database on macOS.
Tries the Default profile first, then scans numbered Profile directories
by most-recently-modified. Brave creates extra profiles as "Profile 1",
"Profile 2", etc. alongside Default; the most recently used one is the
likeliest to hold current cookies. Lexicographic sort would visit
"Profile 10" before "Profile 2", which can return the wrong profile.
"""
default = BRAVE_BASE_DIR / "Default" / "Cookies"
if default.exists():
return default
try:
candidates = [
child for child in BRAVE_BASE_DIR.iterdir()
if child.is_dir() and child.name.startswith("Profile ")
]
for child in sorted(candidates, key=lambda p: p.stat().st_mtime, reverse=True):
candidate = child / "Cookies"
if candidate.exists():
return candidate
except OSError:
pass
return None
def extract_brave_cookies_macos(domain: str, cookie_names: list[str]) -> Optional[dict[str, str]]:
"""Extract cookies from Brave on macOS.
Brave uses the same v10 AES-128-CBC encryption as Chrome; only the DB
path and Keychain service name differ.
"""
db_path = _find_brave_cookies_db()
if db_path is None:
logger.info("Brave cookies database not found under %s", BRAVE_BASE_DIR)
return None
return _extract_chromium_cookies_macos(db_path, "Brave Safe Storage", domain, cookie_names)
@@ -1,6 +1,6 @@
"""Browser cookie extraction for last30days.
Extracts cookies from local browser databases (Firefox, Chrome, Safari)
Extracts cookies from local browser databases (Firefox, Chrome, Brave, Safari)
to enable zero-config authentication for services like X/Twitter.
Only uses Python stdlib no external dependencies.
@@ -255,6 +255,29 @@ def extract_chrome_cookies(
return None
def extract_brave_cookies(
domain: str, cookie_names: List[str]
) -> Optional[Dict[str, str]]:
"""Extract cookies from Brave for the given domain and cookie names.
macOS only Brave uses the same v10 AES-128-CBC encryption as Chrome,
with a different DB path and Keychain service name ("Brave Safe Storage").
Tries the Default profile first, then scans numbered Profile directories.
Returns:
Dict of {cookie_name: cookie_value} or None if extraction fails.
"""
if platform.system() != "Darwin":
logger.debug("Brave cookie extraction only supported on macOS")
return None
try:
from .chrome_cookies import extract_brave_cookies_macos
return extract_brave_cookies_macos(domain, cookie_names)
except Exception as exc:
logger.debug("Brave cookie extraction failed: %s", exc)
return None
def extract_safari_cookies(
domain: str, cookie_names: List[str]
) -> Optional[Dict[str, str]]:
@@ -282,9 +305,9 @@ def extract_cookies(
"""Extract cookies from the specified browser.
Args:
browser: One of 'firefox', 'chrome', 'safari', or 'auto'.
browser: One of 'firefox', 'chrome', 'brave', 'safari', or 'auto'.
'auto' tries browsers in platform-appropriate order:
- macOS: Chrome -> Firefox -> Safari
- macOS: Chrome -> Brave -> Firefox -> Safari
- Linux: Firefox only
domain: The cookie domain to match (e.g. ".x.com").
cookie_names: List of cookie names to extract.
@@ -333,7 +356,7 @@ def extract_cookies_with_source(
so callers can track the source.
Args:
browser: One of 'firefox', 'chrome', 'safari', or 'auto'.
browser: One of 'firefox', 'chrome', 'brave', 'safari', or 'auto'.
domain: The cookie domain to match (e.g. ".x.com").
cookie_names: List of cookie names to extract.
@@ -344,6 +367,7 @@ def extract_cookies_with_source(
extractors = {
"firefox": extract_firefox_cookies,
"chrome": extract_chrome_cookies,
"brave": extract_brave_cookies,
"safari": extract_safari_cookies,
}
@@ -360,7 +384,7 @@ def extract_cookies_with_source(
# Auto mode: try browsers in platform-appropriate order
system = platform.system()
if system == "Darwin":
order = ["chrome", "firefox", "safari"]
order = ["chrome", "brave", "firefox", "safari"]
elif system == "Linux":
order = ["firefox"]
else:
@@ -106,7 +106,7 @@ def _extract_subreddits(reddit_items: List[Dict[str, Any]]) -> List[str]:
for item in reddit_items:
# Primary subreddit
sub = item.get("subreddit", "").strip().lstrip("r/")
sub = item.get("subreddit", "").strip().removeprefix("r/")
if sub:
sub_counts[sub] += 1
+101 -7
View File
@@ -29,6 +29,23 @@ else:
CODEX_AUTH_FILE = Path(os.environ.get("CODEX_AUTH_FILE", str(Path.home() / ".codex" / "auth.json")))
# macOS Keychain integration: items stored with this service prefix are picked
# up automatically on Darwin as the lowest-priority credential source.
# Example: `security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."`.
KEYCHAIN_SERVICE_PREFIX = "last30days-"
# Single source of truth for which credentials the Keychain loader looks up.
# The setup-keychain.sh helper mirrors this list and is held in sync via
# tests/test_env_keychain.py::test_keychain_keys_match_setup_script.
KEYCHAIN_KEYS = (
"OPENAI_API_KEY", "XAI_API_KEY", "GOOGLE_API_KEY", "GEMINI_API_KEY",
"GOOGLE_GENAI_API_KEY", "SCRAPECREATORS_API_KEY", "APIFY_API_TOKEN",
"AUTH_TOKEN", "CT0", "BSKY_HANDLE", "BSKY_APP_PASSWORD",
"TRUTHSOCIAL_TOKEN", "BRAVE_API_KEY", "EXA_API_KEY", "SERPER_API_KEY",
"OPENROUTER_API_KEY", "PARALLEL_API_KEY", "XQUIK_API_KEY",
"XIAOHONGSHU_API_BASE",
)
AuthSource = Literal["api_key", "codex", "none"]
AuthStatus = Literal["ok", "missing", "expired", "missing_account_id"]
@@ -53,6 +70,10 @@ class OpenAIAuth:
def _check_file_permissions(path: Path) -> None:
"""Warn to stderr if a secrets file has overly permissive permissions."""
if os.name == "nt":
# Windows reports synthesized POSIX mode bits that do not reflect NTFS ACLs.
return
try:
mode = path.stat().st_mode
# Check if group or other can read (bits 0o044)
@@ -91,6 +112,46 @@ def load_env_file(path: Path) -> dict[str, str]:
return env
def _load_keychain(keys: list[str]) -> dict[str, str]:
"""Load credentials from macOS Keychain (no-op on other platforms).
Each key is looked up as a generic password with service name
``f"{KEYCHAIN_SERVICE_PREFIX}{key}"`` for the current user. Missing items
and lookup failures are silent Keychain is the lowest-priority source
and is meant to be additive over `.env` files and process environment.
"""
import platform
if platform.system() != "Darwin":
return {}
import shutil
security = shutil.which("security")
if not security:
return {}
import subprocess
import pwd
# USER can be unset under sudo, in Docker without --env USER, or in some CI
# runners; fall back to the OS user record so lookups still match items
# stored by setup-keychain.sh (which uses $USER).
user = os.environ.get("USER") or pwd.getpwuid(os.getuid()).pw_name
env: dict[str, str] = {}
for key in keys:
try:
result = subprocess.run(
[security, "find-generic-password",
"-a", user,
"-s", f"{KEYCHAIN_SERVICE_PREFIX}{key}",
"-w"],
capture_output=True, text=True, timeout=5,
)
except (subprocess.TimeoutExpired, OSError):
continue
if result.returncode == 0 and result.stdout.strip():
env[key] = result.stdout.strip()
return env
def _decode_jwt_payload(token: str) -> dict[str, Any] | None:
"""Decode JWT payload without verification."""
try:
@@ -214,6 +275,7 @@ def get_config() -> dict[str, Any]:
1. Environment variables (os.environ)
2. .claude/last30days.env (per-project config)
3. ~/.config/last30days/.env (global config)
4. macOS Keychain items prefixed ``last30days-`` (Darwin only)
"""
# Load from global config file
file_env = load_env_file(CONFIG_FILE) if CONFIG_FILE else {}
@@ -222,9 +284,14 @@ def get_config() -> dict[str, Any]:
project_env_path = _find_project_env()
project_env = load_env_file(project_env_path) if project_env_path else {}
# Merge: project overrides global
# Merge file sources: project > global
merged_env = {**file_env, **project_env}
# Keychain is the lowest-priority source (Darwin only; no-op elsewhere).
# Loaded before openai_auth so OPENAI_API_KEY can come from Keychain too.
keychain_env = _load_keychain(list(KEYCHAIN_KEYS))
merged_env = {**keychain_env, **merged_env}
openai_auth = get_openai_auth(merged_env)
# Build config: Codex/OpenAI auth + process.env > project .env > global .env
@@ -247,6 +314,7 @@ def get_config() -> dict[str, Any]:
('LAST30DAYS_RERANK_MODEL', None),
('LAST30DAYS_X_MODEL', None),
('LAST30DAYS_X_BACKEND', None),
('LAST30DAYS_STORE', None),
('OPENAI_MODEL_PIN', None),
('XAI_MODEL_PIN', None),
('SCRAPECREATORS_API_KEY', None),
@@ -255,6 +323,7 @@ def get_config() -> dict[str, Any]:
('CT0', None),
('BSKY_HANDLE', None),
('BSKY_APP_PASSWORD', None),
('BSKY_SEARCH_HOST', None),
('TRUTHSOCIAL_TOKEN', None),
('BRAVE_API_KEY', None),
('EXA_API_KEY', None),
@@ -265,16 +334,41 @@ def get_config() -> dict[str, Any]:
('FROM_BROWSER', None),
('SETUP_COMPLETE', None),
('INCLUDE_SOURCES', ''),
('EXCLUDE_SOURCES', ''),
('LAST30DAYS_YOUTUBE_SSH_HOST', None),
('LAST30DAYS_TRANSCRIPT_TIMEOUT', None),
]
for key, default in keys:
config[key] = os.environ.get(key) or merged_env.get(key, default)
# Track which config source was used
# Backward-compat: ScrapeCreators' own examples and tutorials use the
# SCRAPE_CREATORS_API_KEY spelling (with underscore between SCRAPE and
# CREATORS). Accept that form too so users who follow the vendor's docs
# don't silently end up with has_scrapecreators=False. Canonical name
# wins when both are set.
if not config.get('SCRAPECREATORS_API_KEY'):
legacy = os.environ.get('SCRAPE_CREATORS_API_KEY') or merged_env.get('SCRAPE_CREATORS_API_KEY')
if legacy:
config['SCRAPECREATORS_API_KEY'] = legacy
# Multi-key rotation: comma-separated SCRAPECREATORS_API_KEY round-robins
# via random.choice per run. Originally added in #268, accidentally dropped
# in v3.0.6, restored here.
sc_key_raw = config.get('SCRAPECREATORS_API_KEY') or ''
if ',' in sc_key_raw:
import random
sc_keys = [k.strip() for k in sc_key_raw.split(',') if k.strip()]
config['SCRAPECREATORS_API_KEY'] = random.choice(sc_keys) if sc_keys else ''
# Track which config source was used (highest-priority file source wins
# the label; keychain is only reported when nothing else is configured).
if project_env_path:
config['_CONFIG_SOURCE'] = f'project:{project_env_path}'
elif CONFIG_FILE and CONFIG_FILE.exists():
config['_CONFIG_SOURCE'] = f'global:{CONFIG_FILE}'
elif keychain_env:
config['_CONFIG_SOURCE'] = 'keychain'
else:
config['_CONFIG_SOURCE'] = 'env_only'
@@ -517,12 +611,12 @@ def _parse_include_sources(config: dict[str, Any]) -> set[str]:
def is_threads_available(config: dict[str, Any]) -> bool:
"""Check if Threads source is available.
Requires SCRAPECREATORS_API_KEY AND 'threads' in INCLUDE_SOURCES.
Threads is an opt-in source - it is not activated by default.
Returns True when SCRAPECREATORS_API_KEY is set. Threads runs alongside
TikTok and Instagram as part of the SC family same key, same per-call
cost shape, so the same default-on rule applies. Suppress via
EXCLUDE_SOURCES=threads.
"""
if not config.get('SCRAPECREATORS_API_KEY'):
return False
return 'threads' in _parse_include_sources(config)
return bool(config.get('SCRAPECREATORS_API_KEY'))
def is_instagram_available(config: dict[str, Any]) -> bool:
+77 -12
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import sys
import urllib.parse
from datetime import datetime
from urllib.parse import urlparse
@@ -139,7 +140,10 @@ def parallel_search(
data = http.request(
"POST", "https://api.parallel.ai/v1/search",
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
json_data={"query": query, "max_results": count},
json_data={
"search_queries": [query],
"advanced_settings": {"max_results": count},
},
timeout=15,
)
items = []
@@ -149,7 +153,7 @@ def parallel_search(
url = r.get("url", "")
if not url:
continue
raw_date = r.get("published_date") or ""
raw_date = r.get("publish_date") or ""
pub_date = _normalize_date(raw_date[:10]) if raw_date else None
if not _in_date_range(pub_date, date_range):
continue
@@ -158,7 +162,7 @@ def parallel_search(
"title": r.get("title", ""),
"url": url,
"source_domain": _domain(url),
"snippet": r.get("snippet", ""),
"snippet": ((r.get("excerpts") or [""])[0] or "")[:500],
"date": pub_date,
"relevance": 0.8,
"why_relevant": "Parallel AI web search",
@@ -205,29 +209,90 @@ def web_search(
backend = "parallel"
else:
return [], {}
items: list[dict] = []
artifact: dict = {}
if backend == "brave":
key = config.get("BRAVE_API_KEY")
if not key:
raise RuntimeError("BRAVE_API_KEY is required when web_backend='brave'")
return brave_search(query, date_range, key)
if backend == "exa":
items, artifact = brave_search(query, date_range, key)
elif backend == "exa":
key = config.get("EXA_API_KEY")
if not key:
raise RuntimeError("EXA_API_KEY is required when web_backend='exa'")
return exa_search(query, date_range, key)
if backend == "serper":
items, artifact = exa_search(query, date_range, key)
elif backend == "serper":
key = config.get("SERPER_API_KEY")
if not key:
raise RuntimeError("SERPER_API_KEY is required when web_backend='serper'")
return serper_search(query, date_range, key)
if backend == "parallel":
items, artifact = serper_search(query, date_range, key)
elif backend == "parallel":
key = config.get("PARALLEL_API_KEY")
if not key:
raise RuntimeError("PARALLEL_API_KEY is required when web_backend='parallel'")
return parallel_search(query, date_range, key)
if backend != "none":
items, artifact = parallel_search(query, date_range, key)
elif backend != "none":
raise ValueError(f"Unsupported web backend: {backend!r}")
return [], {}
else:
return [], {}
if items and not _reddit_excluded(config):
items = _enrich_reddit_items(items)
return items, artifact
def _reddit_excluded(config: dict) -> bool:
"""Return True when EXCLUDE_SOURCES contains 'reddit'.
Respects the same suppression knob the pipeline uses for source gating,
so a user who set EXCLUDE_SOURCES=reddit doesn't get Reddit content
smuggled back in via web-search URLs.
"""
raw = (config.get("EXCLUDE_SOURCES") or "").split(",")
return any(s.strip().lower() == "reddit" for s in raw)
def _enrich_reddit_items(items: list[dict]) -> list[dict]:
"""Enrich web search results that are Reddit URLs with thread body and comments.
Claude Code's WebFetch blocks reddit.com, so the model can't retrieve
Reddit content from web search results. This fetches it via the public
JSON API (reddit.com/.../.json) which bypasses that restriction.
Callers should gate this with EXCLUDE_SOURCES=reddit handling (see
`_reddit_excluded`) so a user who explicitly excluded Reddit doesn't
get Reddit content via web-search URLs.
"""
from . import reddit_enrich
from .reddit_enrich import RedditRateLimitError
for item in items:
url = item.get("url", "")
if "reddit.com" not in url or "/comments/" not in url:
continue
try:
thread_data = reddit_enrich.fetch_thread_data(url, timeout=8)
if not thread_data:
continue
parsed = reddit_enrich.parse_thread_data(thread_data)
# selftext lives under parsed["submission"], not at the top level
selftext = (parsed.get("submission") or {}).get("selftext", "")
if selftext:
item["snippet"] = selftext[:2000]
comments = parsed.get("comments", [])
top = reddit_enrich.get_top_comments(comments)
if top:
item["top_comments"] = [
{"score": c.get("score", 0), "excerpt": (c.get("body") or "")[:200]}
for c in top[:5]
]
item["enriched_via"] = "reddit_json_api"
except RedditRateLimitError as exc:
# Stop iterating to avoid flooding more 429s
sys.stderr.write(f"[Web] Reddit rate-limited, halting enrichment: {exc}\n")
break
except Exception as exc:
sys.stderr.write(f"[Web] Reddit enrichment failed for {url}: {exc}\n")
return items
# ---------------------------------------------------------------------------
+54 -17
View File
@@ -88,17 +88,26 @@ def search_hackernews(
# Use extracted core subject instead of raw topic for cleaner Algolia matching
core = extract_core_subject(topic)
_log(f"Searching for '{core}' (raw: '{topic}', since {from_date}, count={count})")
# Hyphens and commas tokenize awkwardly in Algolia; flatten them so themed
# queries like "ts-bun-node" or "claude, personal agents" become plain words.
core_flat = _flatten_query_for_algolia(core)
_log(f"Searching for '{core_flat}' (raw: '{topic}', since {from_date}, count={count})")
# Use relevance-sorted search with minimum engagement filter.
# NOTE: restrictSearchableAttributes=title omitted intentionally — it would
# miss Ask HN/Show HN threads where the topic appears in the body.
params = {
"query": core,
"query": core_flat,
"tags": "story",
"numericFilters": f"created_at_i>{from_ts},created_at_i<{to_ts},points>2",
"hitsPerPage": str(count),
}
# Algolia defaults to AND across query tokens, so a 4-5 word theme query
# matches no stories. Mark all-but-the-first token as optional so Algolia
# ranks by how many tokens match instead of requiring every one.
tokens = core_flat.split()
if len(tokens) > 1:
params["optionalWords"] = " ".join(tokens[1:])
from urllib.parse import urlencode
url = f"{ALGOLIA_SEARCH_URL}?{urlencode(params)}"
@@ -117,28 +126,56 @@ def search_hackernews(
return response
def _title_matches_query(title: str, query: str, author: str = "") -> bool:
"""Check if the query term appears in the title content, not just an HN prefix or author.
_WORD_BOUNDARY_RE_CACHE: Dict[str, "re.Pattern[str]"] = {}
Returns True if the query (or any multi-word token) appears in the title
after stripping "Tell HN:", "Show HN:", "Ask HN:", "Launch HN:" prefixes
and ignoring the author name. Returns True when query is empty (no filter).
def _flatten_query_for_algolia(text: str) -> str:
"""Normalise query for Algolia + post-filter comparison.
Multi-keyword theme queries frequently contain commas (delimiters) or
hyphens (compound terms like ``ts-bun-node``); both tokenize awkwardly.
Flatten them to spaces and collapse runs of whitespace so the search
parameter and the post-filter operate on the same shape.
"""
return " ".join(text.replace(",", " ").replace("-", " ").split())
def _title_matches_query(title: str, query: str, author: str = "") -> bool:
"""Check if any query token appears as a whole word in the title.
Returns True when the query is empty (no filter), or when at least one
query token matches as a whole word in the title after stripping
"Tell HN:", "Show HN:", "Ask HN:", "Launch HN:" prefixes.
We previously required *every* token to appear (all-words), which killed
every Algolia hit on multi-keyword themes like "claude, personal agents,
agentic infra" because real HN titles never contain all five tokens
verbatim. Relaxing to any-word matches Algolia's `optionalWords` behaviour
in `search_hackernews`. Token-overlap relevance scoring at parse time
demotes hits where only one weak token matched, so the loosened gate
won't surface noise to the top of the ranking.
Word-boundary matching (rather than naive substring) prevents short
tokens like ``ai`` or ``ts`` from matching unrelated words like
``email`` or ``artists``.
"""
if not query:
return True
stripped = _HN_PREFIXES.sub("", title).strip()
# Also check that the match isn't solely in the author's username
check_text = stripped.lower()
query_lower = query.lower()
# Check each word of the query independently; all must appear somewhere
# in the stripped title (not just the prefix).
query_words = query_lower.split()
# Normalise the query the same way search_hackernews does so post-filter
# tokens line up with what Algolia actually saw.
query_words = [w for w in _flatten_query_for_algolia(query.lower()).split() if w]
if not query_words:
return True
for word in query_words:
if word in check_text:
continue
# Word not found in stripped title — reject
return False
return True
pattern = _WORD_BOUNDARY_RE_CACHE.get(word)
if pattern is None:
pattern = re.compile(rf"\b{re.escape(word)}\b")
_WORD_BOUNDARY_RE_CACHE[word] = pattern
if pattern.search(check_text):
return True
return False
def parse_hackernews_response(response: Dict[str, Any], query: str = "") -> List[Dict[str, Any]]:
+59 -2
View File
@@ -2,6 +2,7 @@
import json
import re
import socket
import sys
import time
import urllib.error
@@ -22,9 +23,19 @@ def log(msg: str):
MAX_RETRIES = 5
MAX_429_RETRIES = 2
RETRY_DELAY = 2.0
# DNS resolution failures (gaierror) are transient — typically resolved by a
# brief backoff and retry. Use a dedicated minimum attempt count + exponential
# delays (1s, 2s, 4s) so callers that pass a small `retries` value still get a
# meaningful chance to recover from a transient resolution failure.
MIN_DNS_RETRIES = 3
USER_AGENT = "last30days-skill/3.0 (Assistant Skill)"
def _is_dns_failure(err: urllib.error.URLError) -> bool:
"""Return True if a URLError was caused by DNS resolution (gaierror)."""
return isinstance(getattr(err, "reason", None), socket.gaierror)
class HTTPError(Exception):
"""HTTP request error with status code."""
def __init__(self, message: str, status_code: Optional[int] = None, body: Optional[str] = None):
@@ -85,7 +96,13 @@ def request(
last_error = None
rate_limit_count = 0
for attempt in range(retries):
# DNS failures get a dedicated minimum attempt count + exponential backoff.
# `effective_retries` is the actual loop bound; we expand it on the first
# gaierror if the caller passed a smaller `retries` value than MIN_DNS_RETRIES.
effective_retries = retries
dns_attempts = 0
attempt = 0
while attempt < effective_retries:
try:
with urllib.request.urlopen(req, timeout=timeout) as response:
body = response.read().decode('utf-8')
@@ -115,6 +132,8 @@ def request(
if rate_limit_count >= max_429_retries:
raise last_error
# HTTP errors respect the caller's original `retries`; only DNS
# failures get the widened `effective_retries` budget.
if attempt < retries - 1:
if e.code == 429:
# Respect Retry-After header, fall back to exponential backoff
@@ -130,11 +149,43 @@ def request(
else:
delay = RETRY_DELAY * (2 ** attempt)
time.sleep(delay)
else:
# Caller's original retry budget exhausted; an earlier DNS
# failure may have widened `effective_retries`, but that
# widening is DNS-only — don't grant extra HTTP attempts.
break
except urllib.error.URLError as e:
log(f"URL Error: {e.reason}")
last_error = HTTPError(f"URL Error: {e.reason}")
if attempt < retries - 1:
if _is_dns_failure(e):
# DNS resolution failures are transient; expand the retry budget
# to MIN_DNS_RETRIES if the caller passed fewer, and use
# exponential backoff (1s, 2s, 4s, ...) instead of the linear
# default. Counts DNS attempts separately so other URLError
# causes don't bypass the regular retry budget.
dns_attempts += 1
if effective_retries < MIN_DNS_RETRIES:
log(
f"DNS resolution failed; expanding retry budget from "
f"{effective_retries} to {MIN_DNS_RETRIES}"
)
effective_retries = MIN_DNS_RETRIES
if attempt < effective_retries - 1:
delay = 2 ** (dns_attempts - 1) # 1s, 2s, 4s, 8s, ...
log(
f"DNS resolution failure (attempt {dns_attempts}); "
f"retrying in {delay:.1f}s"
)
time.sleep(delay)
elif attempt < retries - 1:
# Non-DNS URLError (e.g. ConnectionRefused) respects the
# caller's original retry budget, not the DNS-widened bound.
time.sleep(RETRY_DELAY * (attempt + 1))
else:
# Caller's original retry budget exhausted; an earlier DNS
# failure widening `effective_retries` does not carry over
# to non-DNS error paths.
break
except json.JSONDecodeError as e:
log(f"JSON decode error: {e}")
last_error = HTTPError(f"Invalid JSON response: {e}")
@@ -144,7 +195,13 @@ def request(
log(f"Connection error: {type(e).__name__}: {e}")
last_error = HTTPError(f"Connection error: {type(e).__name__}: {e}")
if attempt < retries - 1:
# Socket errors respect the caller's original retry budget.
time.sleep(RETRY_DELAY * (attempt + 1))
else:
# Original budget exhausted; DNS widening doesn't apply here.
break
attempt += 1
if last_error:
raise last_error
+81 -4
View File
@@ -7,12 +7,14 @@ Requires SCRAPECREATORS_API_KEY in config. 100 free API calls, then PAYG.
API docs: https://scrapecreators.com/docs
"""
import os
import re
import sys
from datetime import datetime
from typing import Any, Dict, List, Optional, Set
from . import dates, http, log
from .relevance import token_overlap_relevance as _compute_relevance
SCRAPECREATORS_BASE = "https://api.scrapecreators.com"
@@ -26,7 +28,42 @@ DEPTH_CONFIG = {
# Max words to keep from each caption
CAPTION_MAX_WORDS = 500
from .relevance import token_overlap_relevance as _compute_relevance
# Default transcript fetch timeout (seconds). SC's
# /v2/instagram/media/transcript regularly takes >15s on real workloads,
# so the default is generous; override via LAST30DAYS_TRANSCRIPT_TIMEOUT.
DEFAULT_TRANSCRIPT_TIMEOUT = 30
def _resolve_transcript_timeout(
timeout: Optional[float] = None,
config: Optional[Dict[str, Any]] = None,
) -> float:
"""Resolve the IG transcript-fetch timeout.
Priority (highest wins):
1. Explicit ``timeout`` kwarg
2. ``LAST30DAYS_TRANSCRIPT_TIMEOUT`` in os.environ
3. ``LAST30DAYS_TRANSCRIPT_TIMEOUT`` in caller-supplied config dict
4. ``DEFAULT_TRANSCRIPT_TIMEOUT`` (30s)
Mirrors the ``os.environ.get(X) or config.get(X)`` pattern used for
LAST30DAYS_STORE in last30days.py so the env var works whether it's
shell-exported or set in ~/.config/last30days/.env.
"""
if timeout is not None:
try:
return float(timeout)
except (TypeError, ValueError):
pass
raw = os.environ.get("LAST30DAYS_TRANSCRIPT_TIMEOUT")
if not raw and config:
raw = config.get("LAST30DAYS_TRANSCRIPT_TIMEOUT")
if raw:
try:
return float(raw)
except (TypeError, ValueError):
pass
return float(DEFAULT_TRANSCRIPT_TIMEOUT)
def _extract_core_subject(topic: str) -> str:
@@ -44,6 +81,17 @@ def _extract_core_subject(topic: str) -> str:
return extract_core_subject(topic, noise=_INSTAGRAM_NOISE)
def _to_hashtag_form(query: str) -> str:
"""Collapse a multi-word query to hashtag form (no spaces, lowercase).
SC's /v2/instagram/reels/search wraps Google Search and is documented
to be flaky on multi-token queries. Single-token queries map to a
hashtag page lookup which is the stable path. Used as a 500-retry
fallback before the request bubbles up as a silent failure.
"""
return ''.join(query.split()).lower()
def _infer_query_intent(topic: str) -> str:
"""Tiny local intent classifier for Instagram query expansion."""
text = topic.lower().strip()
@@ -283,6 +331,26 @@ def search_instagram(
timeout=30,
retries=2,
)
except http.HTTPError as e:
# SC's v2 reels search wraps Google Search and 500s frequently on
# multi-token queries. Single tokens hit the stable hashtag-page
# path. Retry once with hashtag form before bubbling up.
if getattr(e, "status_code", None) == 500 and ' ' in core_topic:
_log(f"IG search 500 on '{core_topic}', retrying with hashtag form")
try:
data = http.get(
f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search",
params={"query": _to_hashtag_form(core_topic)},
headers=http.scrapecreators_headers(token),
timeout=30,
retries=2,
)
except Exception as retry_e:
_log(f"IG search retry failed: {retry_e}")
return {"items": [], "error": f"{type(retry_e).__name__}: {retry_e}"}
else:
_log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
except Exception as e:
_log(f"ScrapeCreators error: {e}")
return {"items": [], "error": f"{type(e).__name__}: {e}"}
@@ -317,6 +385,8 @@ def fetch_captions(
video_items: List[Dict[str, Any]],
token: str,
depth: str = "default",
timeout: Optional[float] = None,
config: Optional[Dict[str, Any]] = None,
) -> Dict[str, str]:
"""Fetch transcripts for top N Instagram reels via ScrapeCreators.
@@ -328,12 +398,19 @@ def fetch_captions(
video_items: Items from search_instagram()
token: ScrapeCreators API key
depth: Depth level for caption limit
timeout: Optional per-request transcript timeout in seconds. When
None, resolves from LAST30DAYS_TRANSCRIPT_TIMEOUT (env or
config), defaulting to DEFAULT_TRANSCRIPT_TIMEOUT (30s).
config: Optional config dict (from env.get_config()) used as a
fallback source for LAST30DAYS_TRANSCRIPT_TIMEOUT when the
value is not exported in os.environ.
Returns:
Dict mapping video_id -> caption text (truncated to 500 words)
"""
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
max_captions = config["max_captions"]
depth_cfg = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
max_captions = depth_cfg["max_captions"]
transcript_timeout = _resolve_transcript_timeout(timeout, config)
if not video_items or not token:
return {}
@@ -364,7 +441,7 @@ def fetch_captions(
f"{SCRAPECREATORS_BASE}/v2/instagram/media/transcript",
params={"url": url},
headers=http.scrapecreators_headers(token),
timeout=15,
timeout=transcript_timeout,
retries=1,
)
transcripts = data.get("transcripts") or []
@@ -251,6 +251,11 @@ def _normalize_youtube(
metadata: dict[str, Any] = {}
if highlights:
metadata["transcript_highlights"] = highlights
if item.get("captions_disabled"):
# Surfaced for quality_nudge: uploader disabled captions, so this
# video should be subtracted from the degraded-transcript-ratio
# denominator (it was never going to produce a transcript).
metadata["captions_disabled"] = True
metadata["top_comments"] = _remap_comments(
item.get("top_comments") or [],
score_keys=("score", "likes"),
+9 -2
View File
@@ -79,6 +79,8 @@ MOCK_AVAILABLE_SOURCES = [
"xiaohongshu",
"github",
"perplexity",
"threads",
"pinterest",
"xquik",
"digg",
]
@@ -118,7 +120,9 @@ def available_sources(config: dict[str, Any], requested_sources: list[str] | Non
available.append("grounding")
# Perplexity Sonar: opt-in additive source via INCLUDE_SOURCES=perplexity
include_sources = (config.get("INCLUDE_SOURCES") or "").lower().split(",")
if config.get("OPENROUTER_API_KEY") and "perplexity" in include_sources:
if config.get("OPENROUTER_API_KEY") and (
"perplexity" in include_sources or (requested_sources and "perplexity" in requested_sources)
):
available.append("perplexity")
if requested_sources and "xiaohongshu" in requested_sources and env.is_xiaohongshu_available(config):
available.append("xiaohongshu")
@@ -128,6 +132,9 @@ def available_sources(config: dict[str, Any], requested_sources: list[str] | Non
available.append("pinterest")
if env.is_xquik_available(config):
available.append("xquik")
exclude = {s.strip().lower() for s in (config.get("EXCLUDE_SOURCES") or "").split(",") if s.strip()}
if exclude:
available = [s for s in available if s not in exclude]
return available
@@ -200,7 +207,7 @@ def run(
available = [source for source in available if source in requested_sources]
if web_backend == "none":
available = [s for s in available if s != "grounding"]
elif web_backend in ("brave", "exa", "serper") and "grounding" not in available:
elif web_backend in ("brave", "exa", "serper", "parallel") and "grounding" not in available:
available.append("grounding")
if not available:
raise RuntimeError("No sources are available for this run.")
+9 -8
View File
@@ -19,14 +19,14 @@ ALLOWED_INTENTS = {
}
ALLOWED_CLUSTER_MODES = {"none", "story", "workflow", "market", "debate"}
QUICK_SOURCE_PRIORITY = {
"factual": ["hackernews", "reddit", "x", "youtube"],
"product": ["youtube", "reddit", "x", "tiktok"],
"concept": ["hackernews", "reddit", "x", "youtube"],
"opinion": ["reddit", "x", "youtube", "hackernews"],
"how_to": ["youtube", "reddit", "x", "hackernews"],
"comparison": ["reddit", "x", "hackernews", "youtube"],
"breaking_news": ["x", "reddit", "hackernews", "youtube", "polymarket"],
"prediction": ["polymarket", "x", "hackernews", "reddit", "youtube"],
"factual": ["hackernews", "reddit", "x", "xquik", "youtube"],
"product": ["youtube", "reddit", "x", "xquik", "tiktok"],
"concept": ["hackernews", "reddit", "x", "xquik", "youtube"],
"opinion": ["reddit", "x", "xquik", "youtube", "hackernews"],
"how_to": ["youtube", "reddit", "x", "xquik", "hackernews"],
"comparison": ["reddit", "x", "xquik", "hackernews", "youtube"],
"breaking_news": ["x", "xquik", "reddit", "hackernews", "youtube", "polymarket"],
"prediction": ["polymarket", "x", "xquik", "hackernews", "reddit", "youtube"],
}
SOURCE_PRIORITY = {
"factual": ["hackernews", "reddit", "x", "youtube"],
@@ -60,6 +60,7 @@ INTENT_SOURCE_EXCLUSIONS: dict[str, set[str]] = {
SOURCE_CAPABILITIES = {
"reddit": {"discussion", "social"},
"x": {"discussion", "social"},
"xquik": {"discussion", "social"},
"youtube": {"video", "video_longform", "discussion"},
"tiktok": {"video", "video_shortform", "social"},
"instagram": {"video", "video_shortform", "social"},
+11 -7
View File
@@ -9,7 +9,7 @@ from typing import Any
from . import env, http, schema
GEMINI_FLASH_LITE = "gemini-3.1-flash-lite-preview"
GEMINI_FLASH_LITE = "gemini-3.1-flash-lite"
GEMINI_PRO = "gemini-3.1-pro-preview"
OPENAI_DEFAULT = "gpt-5.4-nano"
XAI_DEFAULT = "grok-4-1-fast"
@@ -19,7 +19,11 @@ OPENAI_RESPONSES_URL = "https://api.openai.com/v1/responses"
CODEX_RESPONSES_URL = "https://chatgpt.com/backend-api/codex/responses"
XAI_RESPONSES_URL = "https://api.x.ai/v1/responses"
OPENROUTER_URL = "https://openrouter.ai/api/v1/chat/completions"
OPENROUTER_DEFAULT = "google/gemini-flash-2.0"
# OpenRouter routes the Gemini Flash Lite tier as the -preview slug; that is the
# stable form on that routing layer even though native Gemini's GEMINI_FLASH_LITE
# constant is suffix-free. If GEMINI_FLASH_LITE moves to a non-preview stable ID,
# double-check that OpenRouter's slug still maps to the same upstream model.
OPENROUTER_DEFAULT = "google/gemini-3.1-flash-lite-preview"
class ReasoningClient:
@@ -232,8 +236,8 @@ def _resolve_model_pins(config: dict[str, Any], depth: str, provider_name: str)
rerank_model = config.get("LAST30DAYS_RERANK_MODEL") or default_rerank
if provider_name == "gemini":
_require_gemini_31_preview(planner_model, role="planner")
_require_gemini_31_preview(rerank_model, role="rerank")
_require_gemini_31(planner_model, role="planner")
_require_gemini_31(rerank_model, role="rerank")
return planner_model, rerank_model
@@ -344,11 +348,11 @@ def _resolve_x_backend(config: dict[str, Any]) -> str | None:
return env.get_x_source(config)
def _require_gemini_31_preview(model: str, *, role: str) -> None:
if model.startswith("gemini-3.1-") and model.endswith("-preview"):
def _require_gemini_31(model: str, *, role: str) -> None:
if model.startswith("gemini-3.1-"):
return
raise RuntimeError(
f"{role} must use a Gemini 3.1 preview model. Got: {model}"
f"{role} must use a Gemini 3.1 model. Got: {model}"
)
+150 -7
View File
@@ -45,26 +45,100 @@ def _is_youtube_active(config: dict, research_results: dict) -> bool:
return True
# Below this transcript-fetch ratio, YouTube is considered "degraded" rather
# than active. Picked at 50% so a single legitimate caption-disabled video in a
# multi-video result does not trip the nudge, but a stale-yt-dlp run that fails
# every transcript does. Tunable via DEGRADED_TRANSCRIPT_THRESHOLD env var if
# operators need to adjust without code changes.
DEFAULT_DEGRADED_TRANSCRIPT_THRESHOLD = 0.5
def _is_youtube_degraded(research_results: dict, threshold: float) -> bool:
"""YouTube is degraded when videos were returned but the transcript-fetch
ratio is below threshold. The canonical cause is a stale yt-dlp binary -
YouTube's caption format changes frequently and old binaries silently fail
every transcript while the search itself still succeeds.
Captions-disabled videos are subtracted from the denominator: an uploader
who turned off captions can never produce a transcript, so counting that
video toward "fetch failures" produces false positives. A single
captions-disabled video in a small result set was tripping the nudge.
"""
videos = int(research_results.get("youtube_videos_count") or 0)
transcripts = int(research_results.get("youtube_transcripts_count") or 0)
captions_disabled = int(research_results.get("youtube_captions_disabled_count") or 0)
if videos <= 0:
return False
eligible = videos - captions_disabled
if eligible <= 0:
# Every returned video had captions disabled - upstream content fact,
# not a yt-dlp problem. Don't flag.
return False
return (transcripts / eligible) < threshold
def _is_instagram_silent_failure(config: dict, research_results: dict) -> bool:
"""Instagram is silently failing when SC is configured but the source
returned zero items. The canonical cause is SC's v2 reels endpoint
500'ing on multi-token queries (it wraps Google Search and is documented
to be flaky there). Pre-fix the user got no signal at all - no Instagram
section in the brief, no error in the footer, just unexplained absence.
"""
if not config.get("SCRAPECREATORS_API_KEY"):
return False # not configured — not a silent failure
# Honor EXCLUDE_SOURCES: a user who set EXCLUDE_SOURCES=instagram
# intentionally turned the source off, so a zero-item count is
# expected, not a silent failure. Mirror the canonical parsing
# pattern from pipeline.available_sources().
excluded = {
s.strip().lower()
for s in (config.get("EXCLUDE_SOURCES") or "").split(",")
if s.strip()
}
# Symmetric case: INCLUDE_SOURCES is an opt-in allowlist. If it is
# non-empty and does not name instagram, the source was intentionally
# filtered out, so a zero-item count is expected — not a silent failure.
included = {
s.strip().lower()
for s in (config.get("INCLUDE_SOURCES") or "").split(",")
if s.strip()
}
if "instagram" in excluded or (included and "instagram" not in included):
return False
count = research_results.get("instagram_items_count")
if count is None:
return False # source not run this invocation
return int(count) == 0
def compute_quality_score(config: dict, research_results: dict) -> dict:
"""Compute research quality score based on 5 core sources.
Args:
config: Configuration dict from env.get_config()
research_results: Dict with keys like x_error, youtube_error,
reddit_error reflecting what happened this run.
reddit_error reflecting what happened this run. Optional keys
``youtube_videos_count`` and ``youtube_transcripts_count`` enable
degraded-YouTube detection (transcript-fetch ratio below threshold).
Optional key ``instagram_items_count`` enables silent-failure
detection for the bonus Instagram source.
Returns:
{
"score_pct": 40-100,
"core_active": ["hn", "polymarket", ...],
"core_missing": ["x", "youtube"],
"core_errored": [], # configured but errored
"nudge_text": "..." or None if 100%
"core_errored": [], # configured but errored at top level
"core_degraded": [], # configured and returned items but quality below threshold
"bonus_errored": [], # bonus sources (Instagram, etc.) configured but silent
"nudge_text": "..." or None if all sources healthy
}
"""
core_active: List[str] = []
core_missing: List[str] = []
core_errored: List[str] = []
core_degraded: List[str] = []
bonus_errored: List[str] = []
# HN, Polymarket, and Reddit are always active
core_active.append("hn")
@@ -84,6 +158,13 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
yt_active = _is_youtube_active(config, research_results)
if yt_active:
core_active.append("youtube")
# Active means yt-dlp is installed and search did not error at the top
# level. But search-success + transcript-failure is the canonical
# stale-binary failure mode that the footer used to hide. Flag as
# degraded so the user gets an actionable nudge to update the binary.
threshold = float(config.get("DEGRADED_TRANSCRIPT_THRESHOLD") or DEFAULT_DEGRADED_TRANSCRIPT_THRESHOLD)
if _is_youtube_degraded(research_results, threshold):
core_degraded.append("youtube")
else:
core_missing.append("youtube")
# Check if configured but errored (yt-dlp installed but failed this run)
@@ -95,28 +176,54 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
if has_ytdlp and research_results.get("youtube_error"):
core_errored.append("youtube")
# Bonus sources (Instagram, etc.): SC-key holders expect content from
# these but until now the pipeline fell silent on configured-but-zero.
if _is_instagram_silent_failure(config, research_results):
bonus_errored.append("instagram")
score_pct = int(len(core_active) / 5 * 100)
has_sc = bool(config.get("SCRAPECREATORS_API_KEY"))
active_sources = research_results.get("active_sources") or []
nudge_text = _build_nudge_text(core_missing, core_errored, has_sc=has_sc, active_sources=active_sources) if core_missing else None
nudge_text = _build_nudge_text(
core_missing,
core_errored,
core_degraded,
research_results,
has_sc=has_sc,
active_sources=active_sources,
bonus_errored=bonus_errored,
) if (core_missing or core_degraded or bonus_errored) else None
return {
"score_pct": score_pct,
"core_active": core_active,
"core_missing": core_missing,
"core_errored": core_errored,
"core_degraded": core_degraded,
"bonus_errored": bonus_errored,
"nudge_text": nudge_text,
}
def _build_nudge_text(core_missing: List[str], core_errored: List[str], has_sc: bool = False, active_sources: list = None) -> str:
"""Build human-readable nudge text describing what was missed.
def _build_nudge_text(
core_missing: List[str],
core_errored: List[str],
core_degraded: List[str] = None,
research_results: dict = None,
has_sc: bool = False,
active_sources: list = None,
bonus_errored: List[str] = None,
) -> str:
"""Build human-readable nudge text describing what was missed or degraded.
Prioritizes free suggestions. Optionally mentions bonus sources
(TikTok, Instagram, Threads, Pinterest) if ScrapeCreators key is configured.
"""
lines: List[str] = []
core_degraded = core_degraded or []
bonus_errored = bonus_errored or []
research_results = research_results or {}
# Describe what was missed
missed_parts: List[str] = []
@@ -129,7 +236,14 @@ def _build_nudge_text(core_missing: List[str], core_errored: List[str], has_sc:
active_count = 5 - len(core_missing)
lines.append(f"Research quality: {active_count}/5 core sources.")
lines.append(f"Missing: {', '.join(missed_parts)}.")
if missed_parts:
lines.append(f"Missing: {', '.join(missed_parts)}.")
if core_degraded:
degraded_labels = ", ".join(SOURCE_LABELS[s] for s in core_degraded)
lines.append(f"Degraded: {degraded_labels}.")
if bonus_errored:
bonus_labels = ", ".join(s.capitalize() for s in bonus_errored)
lines.append(f"Bonus source silent: {bonus_labels}.")
lines.append("")
# Free suggestions
@@ -159,6 +273,35 @@ def _build_nudge_text(core_missing: List[str], core_errored: List[str], has_sc:
"explanations on any topic. Install yt-dlp: brew install yt-dlp (free)"
)
if "youtube" in core_degraded:
videos = int(research_results.get("youtube_videos_count") or 0)
transcripts = int(research_results.get("youtube_transcripts_count") or 0)
captions_disabled = int(research_results.get("youtube_captions_disabled_count") or 0)
captions_note = ""
if captions_disabled > 0:
captions_note = (
f" ({captions_disabled} of those had captions disabled by the "
"uploader, which is a separate cause and not fixable on your end)"
)
free_suggestions.append(
f"YouTube returned {videos} videos but only {transcripts} transcripts "
f"captured{captions_note}. The most common remaining cause is a stale "
"yt-dlp binary - YouTube's caption format changes frequently and old "
"binaries silently fail every transcript. Update via your package "
"manager: scoop update yt-dlp (Windows), brew upgrade yt-dlp (macOS), "
"or pip install -U yt-dlp."
)
if "instagram" in bonus_errored:
free_suggestions.append(
"Instagram returned 0 reels despite SC being configured. SC's "
"v2 reels endpoint wraps Google Search and 500's frequently on "
"multi-token queries. The skill now retries with hashtag-form "
"automatically; if zero items still appear, the topic may have "
"no reel coverage on Instagram. Try a single-word topic like "
"the most distinctive noun in your query."
)
# Mention bonus opt-in sources when SC key is present
if has_sc:
bonus_hints = []
+11 -1
View File
@@ -334,7 +334,7 @@ def _global_search(
)
return data.get("posts", data.get("data", []))
except http.HTTPError as e:
if e.status_code in (401, 403):
if e.status_code in (401, 402, 403):
raise
_log(f"Global search error: {e}")
return []
@@ -376,6 +376,11 @@ def _subreddit_search(
retries=2,
)
return data.get("posts", data.get("data", []))
except http.HTTPError as e:
if e.status_code in (401, 402, 403):
raise
_log(f"Subreddit search error for r/{subreddit}: {e}")
return []
except Exception as e:
_log(f"Subreddit search error for r/{subreddit}: {e}")
return []
@@ -403,6 +408,11 @@ def fetch_post_comments(
retries=2,
)
return data.get("comments", data.get("data", []))
except http.HTTPError as e:
if e.status_code in (401, 402, 403):
raise
_log(f"Comment fetch error: {e}")
return []
except Exception as e:
_log(f"Comment fetch error: {e}")
return []
+14 -3
View File
@@ -11,6 +11,7 @@ Handles 429 rate limits with exponential backoff, HTML anti-bot responses,
network timeouts, and missing subreddits.
"""
import gzip
import json
import sys
import time
@@ -21,7 +22,11 @@ from concurrent.futures import ThreadPoolExecutor, TimeoutError as FuturesTimeou
from typing import Any, Dict, List, Optional
USER_AGENT = "last30days/3.0 (research tool)"
USER_AGENT = (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0.0.0 Safari/537.36"
)
# Depth-aware limits for thread counts
DEPTH_LIMITS = {
@@ -60,6 +65,9 @@ def _fetch_json(url: str, timeout: int = 15) -> Optional[Dict[str, Any]]:
headers = {
"User-Agent": USER_AGENT,
"Accept": "application/json",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate",
"Connection": "keep-alive",
}
req = urllib.request.Request(url, headers=headers)
@@ -71,7 +79,10 @@ def _fetch_json(url: str, timeout: int = 15) -> Optional[Dict[str, Any]]:
_log(f"Anti-bot HTML response (Content-Type: {content_type})")
return None
body = resp.read().decode("utf-8")
raw = resp.read()
if resp.headers.get("Content-Encoding", "").lower() == "gzip":
raw = gzip.decompress(raw)
body = raw.decode("utf-8")
return json.loads(body)
except urllib.error.HTTPError as e:
@@ -198,7 +209,7 @@ def search(
encoded_query = _url_encode(query)
if subreddit:
sub = subreddit.lstrip("r/").strip()
sub = subreddit.removeprefix("r/").strip()
url = (
f"https://www.reddit.com/r/{sub}/search.json"
f"?q={encoded_query}&restrict_sr=on&sort=relevance&t=month&limit={limit}&raw_json=1"
+15 -25
View File
@@ -4,18 +4,11 @@ from __future__ import annotations
import json
import pathlib
import re
from collections import Counter
from datetime import date
from urllib.parse import urlparse
from . import dates, schema
_VERSION_RE = re.compile(
r'''^version:\s*(?:"([^"]+)"|'([^']+)'|(\S+))\s*$''',
re.MULTILINE,
)
from . import dates, schema, skill_meta
def _skill_version() -> str:
@@ -25,11 +18,12 @@ def _skill_version() -> str:
Hermes, etc.) do not always carry `.claude-plugin/plugin.json` that file ships with
plugin-cache installs but not with per-harness skill installs. SKILL.md frontmatter is
the fallback that keeps the badge from emitting v? on those installs. Returns "?" only
if both sources are missing.
if no usable version string is found from either source (missing files, corrupt JSON,
or SKILL.md without a version line).
A corrupt manifest at one ancestor does not shadow a valid manifest at a deeper one
(continue, not break). YAML frontmatter accepts double-quoted, single-quoted, or
unquoted version scalars.
(continue, not break). SKILL.md parsing accepts double-quoted, single-quoted, or
unquoted YAML version scalars (delegated to skill_meta.read_skill_version).
"""
here = pathlib.Path(__file__).resolve()
for parent in here.parents:
@@ -43,16 +37,11 @@ def _skill_version() -> str:
return version
# No usable manifest found at any ancestor — fall back to SKILL.md frontmatter.
# First SKILL.md found in the walk is THIS skill's; never traverse past it.
for parent in here.parents:
skill_md = parent / "SKILL.md"
if skill_md.is_file():
try:
match = _VERSION_RE.search(skill_md.read_text())
except (OSError, UnicodeDecodeError):
break
if match:
return next(g for g in match.groups() if g is not None)
break
return skill_meta.read_skill_version(skill_md) or "?"
return "?"
@@ -107,7 +96,7 @@ def render_compact(report: schema.Report, cluster_limit: int = 8, fun_level: str
non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
lines = [
*_render_badge(),
f"# last30days v3.0.0: {report.topic}",
f"# last30days v{_skill_version()}: {report.topic}",
"",
*_assistant_safety_lines(),
f"- Date range: {report.range_from} to {report.range_to}",
@@ -613,7 +602,7 @@ def render_comparison_multi(
lines: list[str] = [
*_render_badge(),
f"# last30days v3.0.0: {synthesized_topic}",
f"# last30days v{_skill_version()}: {synthesized_topic}",
"",
*_assistant_safety_lines(),
f"- Comparison mode: {len(entities)} entities ({', '.join(entities)})",
@@ -801,7 +790,7 @@ def render_full(report: schema.Report) -> str:
# Start with the same header as compact
non_empty = [s for s, items in sorted(report.items_by_source.items()) if items]
lines = [
f"# last30days v3.0.0: {report.topic}",
f"# last30days v{_skill_version()}: {report.topic}",
"",
*_assistant_safety_lines(),
f"- Date range: {report.range_from} to {report.range_to}",
@@ -1296,15 +1285,16 @@ def _build_source_footer_lines(report: schema.Report) -> list[str]:
if total > 0:
total_str = f"{total:,}" if total >= 1000 else str(total)
parts.append(f"{total_str} {word}")
# YouTube: append "N with transcripts" instead of a third likes-based column.
# Transcripts are a more meaningful research-depth signal than likes.
# YouTube: always append "M/N with transcripts" so a zero-transcript run
# (typically caused by a stale yt-dlp binary) is visible at the conclusion
# surface. Hiding zero converts a problem signal into an absence; the very
# case that needs to be loud is the one previously omitted from the footer.
if source_key == "youtube":
with_transcripts = sum(
1 for it in items
if (it.metadata.get("transcript_highlights") or it.metadata.get("transcript_snippet"))
)
if with_transcripts > 0:
parts.append(f"{with_transcripts} with transcripts")
parts.append(f"{with_transcripts}/{len(items)} with transcripts")
stats = "".join(parts)
out.append(_footer_line_for_source(emoji, label, len(items), item_word, stats))
+88 -1
View File
@@ -160,6 +160,93 @@ def _extract_github_repos(items: list[dict]) -> list[str]:
return repos[:5] # cap at 5 repos
_INTEGRATION_SUFFIX_KEYWORDS: dict[str, set[str]] = {
"-action": {"action", "actions", "workflow", "workflows"},
"-sdk": {"sdk", "client", "library"},
"-plugin": {"plugin", "plugins", "extension", "extensions"},
"-plugins": {"plugin", "plugins", "extension", "extensions"},
"-docs": {"docs", "documentation"},
"-examples": {"example", "examples", "sample", "samples"},
"-template": {"template", "templates", "starter", "boilerplate"},
}
def _topic_tokens(topic: str) -> set[str]:
return set(re.findall(r"[a-z0-9]+", (topic or "").lower()))
def _topic_entity_slugs(topic: str) -> list[str]:
entities = re.split(r"\b(?:vs|versus)\b", (topic or "").lower())
slugs: list[str] = []
for entity in entities:
tokens = re.findall(r"[a-z0-9]+", entity)
if tokens:
slugs.append("-".join(tokens))
return slugs
def _repo_slug(repo: str) -> str:
parts = repo.split("/", 1)
if len(parts) != 2:
return ""
return parts[1].lower()
def _canonicalize_integration_repo(topic: str, repo: str) -> str:
"""Map integration repos back to canonical product repos when intent allows.
Example:
anthropics/claude-code-action -> anthropics/claude-code
unless topic explicitly asks for "action"/"workflow".
"""
parts = repo.split("/", 1)
if len(parts) != 2:
return repo
owner, name = parts[0], parts[1]
lower_name = name.lower()
topic_words = _topic_tokens(topic)
for suffix, intent_words in _INTEGRATION_SUFFIX_KEYWORDS.items():
if not lower_name.endswith(suffix):
continue
if topic_words.intersection(intent_words):
return repo
base = name[: -len(suffix)]
if base:
return f"{owner}/{base}"
return repo
def canonicalize_github_repos(topic: str, repos: list[str], *, cap: int | None = 5) -> list[str]:
"""Normalize/priority-sort GitHub repos for the current topic.
- Rewrites common integration suffixes to canonical product repos when
topic intent does not mention those integrations.
- Promotes exact topic slug matches (e.g., `claude-code`) over partials.
"""
canonicalized: list[str] = []
seen: set[str] = set()
for repo in repos:
candidate = _canonicalize_integration_repo(topic, repo.strip())
if "/" not in candidate:
continue
key = candidate.lower()
if key in seen:
continue
seen.add(key)
canonicalized.append(candidate)
topic_slugs = set(_topic_entity_slugs(topic))
if topic_slugs:
exact = [r for r in canonicalized if _repo_slug(r) in topic_slugs]
prefixed = [r for r in canonicalized if any(_repo_slug(r).startswith(f"{slug}-") for slug in topic_slugs) and r not in exact]
rest = [r for r in canonicalized if r not in exact and r not in prefixed]
canonicalized = exact + prefixed + rest
if cap is not None:
return canonicalized[:cap]
return canonicalized
def _build_context_summary(items: list[dict]) -> str:
"""Build a 1-2 sentence current events summary from news search results."""
snippets: list[str] = []
@@ -240,7 +327,7 @@ def auto_resolve(topic: str, config: dict) -> dict:
subreddits = _extract_subreddits(results.get("subreddit", []))
x_handle = _extract_x_handle(results.get("x_handle", []))
github_user = _extract_github_user(results.get("github", []))
github_repos = _extract_github_repos(results.get("github", []))
github_repos = canonicalize_github_repos(topic, _extract_github_repos(results.get("github", [])))
context = _build_context_summary(results.get("news", []))
subreddits, category = _merge_category_peers(topic, subreddits)
@@ -107,7 +107,18 @@ def extract_safari_cookies_macos(
if sys.platform != "darwin":
return None
cookie_path = Path.home() / "Library" / "Cookies" / "Cookies.binarycookies"
cookie_paths = [
Path.home()
/ "Library"
/ "Containers"
/ "com.apple.Safari"
/ "Data"
/ "Library"
/ "Cookies"
/ "Cookies.binarycookies",
Path.home() / "Library" / "Cookies" / "Cookies.binarycookies",
]
cookie_path = next((path for path in cookie_paths if path.exists()), cookie_paths[0])
try:
raw = cookie_path.read_bytes()
@@ -335,8 +335,9 @@ def poll_device_auth(
"""
import sys
deadline = time.time() + timeout
last_reminder = time.time()
started_at = time.time()
deadline = started_at + timeout
last_reminder = started_at
reminder_count = 0
max_reminders = 4
reminder_interval = 30 # seconds between reminders
@@ -0,0 +1,33 @@
"""SKILL.md metadata helpers — single source of truth for parsing skill frontmatter.
Centralizes the version regex that previously lived in render.py and was
duplicated in tests/test_plugin_contract.py and tests/test_version_consistency.py.
"""
import re
from pathlib import Path
# Matches `version: "x.y.z"`, `version: 'x.y.z'`, or `version: x.y.z` in YAML
# frontmatter. Multiline so the pattern can be applied to a full SKILL.md text.
# Three alternation groups — exactly one captures per successful match.
_VERSION_RE = re.compile(
r'''^version:\s*(?:"([^"]+)"|'([^']+)'|(\S+))\s*$''',
re.MULTILINE,
)
def read_skill_version(skill_md_path: Path) -> str | None:
"""Return the version string from a SKILL.md's frontmatter, or None.
Returns None if the file can't be read (missing, permission, decode error)
or if no `version:` line is found. Accepts double-quoted, single-quoted,
or unquoted YAML version scalars.
"""
try:
text = skill_md_path.read_text()
except (OSError, UnicodeDecodeError):
return None
match = _VERSION_RE.search(text)
if not match:
return None
return match.group(1) or match.group(2) or match.group(3)
+7 -3
View File
@@ -6,6 +6,8 @@ import threading
import random
from typing import Optional
from .render import _skill_version
# Check if we're in a real terminal (not captured by Claude Code)
IS_TTY = sys.stderr.isatty()
@@ -198,7 +200,7 @@ Just start with "last30" and talk to me like normal.
# Shorter promo for single missing key
PROMO_SINGLE_KEY = {
"reddit": "\n💡 Unlock TikTok and Instagram with SCRAPECREATORS_API_KEY - 10,000 free calls, no CC - scrapecreators.com\n",
"reddit": "\n💡 Unlock TikTok and Instagram with SCRAPECREATORS_API_KEY - 100 free credits, no CC - scrapecreators.com\n",
"x": "\n💡 Unlock X: log into x.com in Firefox or Safari, then re-run. Or add AUTH_TOKEN/CT0 or XAI_API_KEY.\n",
"web": "\n💡 You can unlock native grounded web search with BRAVE_API_KEY or SERPER_API_KEY.\n",
}
@@ -509,7 +511,8 @@ def show_diagnostic_banner(diag: dict):
if IS_TTY:
lines.append(f"{Colors.DIM}┌─────────────────────────────────────────────────────┐{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.BOLD}/last30days v3.0.0 - Source Status{Colors.RESET} {Colors.DIM}{Colors.RESET}")
_header = f"/last30days v{_skill_version()} - Source Status"
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.BOLD}{_header}{Colors.RESET}{' ' * (52 - len(_header))}{Colors.DIM}{Colors.RESET}")
lines.append(f"{Colors.DIM}{Colors.RESET} {Colors.DIM}{Colors.RESET}")
# Reddit
@@ -556,7 +559,8 @@ def show_diagnostic_banner(diag: dict):
else:
# Plain text for non-TTY (Claude Code / Codex)
lines.append("┌─────────────────────────────────────────────────────┐")
lines.append("/last30days v3.0.0 - Source Status")
_header_plain = f"/last30days v{_skill_version()} - Source Status"
lines.append(f"{_header_plain}{' ' * (52 - len(_header_plain))}")
lines.append("│ │")
if has_reddit and has_scrapecreators:
+15 -7
View File
@@ -175,16 +175,24 @@ def parse_x_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
break
if not output_text:
return items
response_preview = str(response)[:200] if response else "(empty)"
raise http.HTTPError(
f"xAI API returned empty response (no output text found; response preview: {response_preview})"
)
# Extract JSON from the response
json_match = re.search(r'\{[\s\S]*"items"[\s\S]*\}', output_text)
if json_match:
try:
data = json.loads(json_match.group())
items = data.get("items", [])
except json.JSONDecodeError:
_log(f"Failed to parse xAI response JSON: {output_text[:200]}")
if not json_match:
raise http.HTTPError(
f"xAI API returned output without valid JSON items structure (output: {output_text[:200]})"
)
try:
data = json.loads(json_match.group())
items = data.get("items", [])
except json.JSONDecodeError:
raise http.HTTPError(
f"xAI API returned valid output but invalid JSON structure (output: {output_text[:200]})"
)
# Validate and clean items
clean_items = []
+126 -11
View File
@@ -8,7 +8,9 @@ Inspired by Peter Steinberger's toolchain approach (yt-dlp + summarize CLI).
import json
import math
import os
import re
import shlex
import shutil
import sys
import tempfile
@@ -96,10 +98,76 @@ def _log(msg: str):
def is_ytdlp_installed() -> bool:
"""Check if yt-dlp is available in PATH."""
"""Check if yt-dlp is available locally, or if SSH routing is configured.
When LAST30DAYS_YOUTUBE_SSH_HOST is set, returns True without a local check
yt-dlp lives on the remote host. Failures surface naturally on first use.
"""
if _ytdlp_ssh_host():
return True
return shutil.which("yt-dlp") is not None
# Host aliases must be plain hostnames / SSH config aliases — no flags, no
# shell metacharacters. Rejects any value that could be reinterpreted by ssh
# (or the surrounding shell) as something other than a destination.
_SSH_HOST_ALIAS_RE = re.compile(r"^[a-zA-Z0-9._-]+$")
def _ytdlp_ssh_host() -> Optional[str]:
"""Return SSH host alias if yt-dlp should be routed via SSH, else None.
Set LAST30DAYS_YOUTUBE_SSH_HOST=<ssh-alias> (e.g. 'macmini') in the environment
to route yt-dlp through SSH for residential IP egress. This bypasses
YouTube's bot-wall on datacenter IPs (Hetzner, DigitalOcean, AWS, etc.)
where ytsearch returns 0 results regardless of cookies.
The remote host must have yt-dlp installed and reachable via the named
SSH alias (configured in ~/.ssh/config). On macOS hosts with Homebrew,
add brew shellenv to ~/.zshenv (not just ~/.zprofile) so non-login SSH
shells find yt-dlp on PATH.
Validation: host value must match ``[A-Za-z0-9._-]+``. Anything starting
with ``-`` or containing shell/SSH metacharacters is rejected with a
stderr warning and treated as unset, so a misconfigured or attacker-
controlled value can't slip through as an SSH option flag or proxy command.
The ``--`` option terminator in ``_wrap_ytdlp_cmd`` is a second line of
defense; this regex closes the door on the env var ever reaching ssh
in the first place.
To use a value from ~/.config/last30days/.env, export it into the
environment before invoking the engine, e.g. in a wrapper:
set -a; source ~/.config/last30days/.env; set +a
python3 last30days.py "..."
"""
host = os.environ.get("LAST30DAYS_YOUTUBE_SSH_HOST", "").strip()
if not host:
return None
if not _SSH_HOST_ALIAS_RE.match(host):
sys.stderr.write(
f"[youtube_yt] WARNING: LAST30DAYS_YOUTUBE_SSH_HOST={host!r} "
"does not look like a plain hostname/alias; ignoring. "
"Expected pattern: letters, digits, dot, underscore, hyphen.\n"
)
return None
return host
def _wrap_ytdlp_cmd(cmd: List[str]) -> List[str]:
"""Wrap a yt-dlp command list with `ssh <host>` when SSH routing is set.
Args are shell-quoted to survive the remote shell. Uses BatchMode=yes so
a misconfigured key fails fast instead of hanging on a password prompt.
The `--` option terminator prevents an SSH option-injection if
LAST30DAYS_YOUTUBE_SSH_HOST were ever set to a value starting with `-`.
"""
host = _ytdlp_ssh_host()
if not host:
return cmd
remote_cmd = " ".join(shlex.quote(a) for a in cmd)
return ["ssh", "-o", "BatchMode=yes", "--", host, remote_cmd]
def _extract_core_subject(topic: str) -> str:
"""Extract core subject from verbose query for YouTube search.
@@ -223,6 +291,7 @@ def search_youtube(
"--no-warnings",
"--no-download",
]
cmd = _wrap_ytdlp_cmd(cmd)
try:
result = subproc.run_with_timeout(cmd, timeout=120)
@@ -316,7 +385,11 @@ def _clean_vtt(vtt_text: str) -> str:
_YT_USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
def _fetch_transcript_direct(
video_id: str,
timeout: int = 30,
status: Optional[Dict[str, Any]] = None,
) -> Optional[str]:
"""Fetch YouTube transcript via direct HTTP without yt-dlp.
Scrapes the watch page HTML for the captions track URL in
@@ -325,6 +398,9 @@ def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
Args:
video_id: YouTube video ID
timeout: HTTP request timeout in seconds
status: Optional dict mutated to record per-video signals. Sets
``status["no_caption_tracks"] = True`` when the player response
confirms the uploader has no caption tracks (vs. fetch failure).
Returns:
Raw VTT text, or None if captions are unavailable.
@@ -373,6 +449,8 @@ def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
if not caption_tracks:
_log(f"Direct transcript: no caption tracks for {video_id}")
if status is not None:
status["no_caption_tracks"] = True
return None
# Find English track (prefer exact 'en', then any en variant, then first track)
@@ -458,7 +536,11 @@ def _fetch_transcript_ytdlp(video_id: str, temp_dir: str) -> Optional[str]:
return None
def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
def fetch_transcript(
video_id: str,
temp_dir: str,
status: Optional[Dict[str, Any]] = None,
) -> Optional[str]:
"""Fetch auto-generated transcript for a YouTube video.
Uses yt-dlp when available (preferred, more robust). Falls back to
@@ -467,19 +549,32 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
Args:
video_id: YouTube video ID
temp_dir: Temporary directory for subtitle files
status: Optional dict mutated by the direct-HTTP path to record
per-video signals like ``no_caption_tracks``. Used to surface a
captions-disabled count so the quality nudge avoids false-positive
"stale yt-dlp" flags.
Returns:
Plaintext transcript string, or None if no captions available.
"""
raw_vtt = None
if is_ytdlp_installed():
# When SSH-routing is on, the yt-dlp transcript path would write a VTT
# file on the remote host that we can't easily read back. Skip it and
# use the HTTP transcript fallback (different YouTube endpoint, less
# bot-walled, works fine from datacenter IPs).
ssh_host = _ytdlp_ssh_host()
use_ytdlp = is_ytdlp_installed() and not ssh_host
if use_ytdlp:
raw_vtt = _fetch_transcript_ytdlp(video_id, temp_dir)
if not raw_vtt:
_log(f"yt-dlp transcript failed for {video_id}, trying direct HTTP fallback")
raw_vtt = _fetch_transcript_direct(video_id)
raw_vtt = _fetch_transcript_direct(video_id, status=status)
else:
_log("yt-dlp not installed, using direct HTTP transcript fetch")
raw_vtt = _fetch_transcript_direct(video_id)
if ssh_host:
_log("SSH-routing active, using direct HTTP transcript fetch")
else:
_log("yt-dlp not installed, using direct HTTP transcript fetch")
raw_vtt = _fetch_transcript_direct(video_id, status=status)
if not raw_vtt:
_log(f"No transcript available for {video_id} (no captions found)")
@@ -498,12 +593,16 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
def fetch_transcripts_parallel(
video_ids: List[str],
max_workers: int = 5,
out_captions_disabled: Optional[Set[str]] = None,
) -> Dict[str, Optional[str]]:
"""Fetch transcripts for multiple videos in parallel.
Args:
video_ids: List of YouTube video IDs
max_workers: Max parallel fetches
out_captions_disabled: Optional set mutated to record video_ids whose
uploader confirmed no caption tracks (vs. transient fetch failures).
Backward-compatible: callers that don't care can omit.
Returns:
Dict mapping video_id to transcript text (or None).
@@ -514,10 +613,11 @@ def fetch_transcripts_parallel(
_log(f"Fetching transcripts for {len(video_ids)} videos")
results = {}
statuses: Dict[str, Dict[str, Any]] = {vid: {} for vid in video_ids}
with tempfile.TemporaryDirectory() as temp_dir:
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {
executor.submit(fetch_transcript, vid, temp_dir): vid
executor.submit(fetch_transcript, vid, temp_dir, statuses[vid]): vid
for vid in video_ids
}
for future in as_completed(futures):
@@ -531,6 +631,11 @@ def fetch_transcripts_parallel(
_log(f"Unexpected transcript error for {vid}: {type(exc).__name__}: {exc}")
results[vid] = None
if out_captions_disabled is not None:
for vid, st in statuses.items():
if st.get("no_caption_tracks"):
out_captions_disabled.add(vid)
got = sum(1 for v in results.values() if v)
errors = sum(1 for v in results.values() if v is None)
_log(f"Got transcripts for {got}/{len(video_ids)} videos ({errors} failed)")
@@ -581,15 +686,21 @@ def search_and_transcribe(
# good chance of reaching the target number of successful transcripts.
transcript_limit = TRANSCRIPT_LIMITS.get(depth, TRANSCRIPT_LIMITS["default"])
transcripts: Dict[str, Optional[str]] = {}
captions_disabled_ids: Set[str] = set()
if transcript_limit > 0:
attempt_count = min(len(items), transcript_limit * 3)
candidate_ids = [item["video_id"] for item in items[:attempt_count]]
_log(f"Fetching transcripts for up to {attempt_count} videos (target: {transcript_limit}): {candidate_ids}")
transcripts = fetch_transcripts_parallel(candidate_ids)
transcripts = fetch_transcripts_parallel(
candidate_ids, out_captions_disabled=captions_disabled_ids,
)
else:
_log(f"Transcript limit is 0 for depth={depth}, skipping transcript fetch")
# Step 3: Attach transcripts and extract highlights
# Step 3: Attach transcripts and extract highlights. Mark captions_disabled
# so quality_nudge can subtract those videos from the degraded-ratio
# denominator (uploader-disabled captions can never produce a transcript;
# counting them was producing false-positive stale-yt-dlp nudges).
core_topic = _extract_core_subject(topic)
for item in items:
vid = item["video_id"]
@@ -598,6 +709,7 @@ def search_and_transcribe(
item["transcript_highlights"] = extract_transcript_highlights(
transcript or "", core_topic,
)
item["captions_disabled"] = vid in captions_disabled_ids
return {"items": items}
@@ -866,9 +978,12 @@ def _sc_youtube_search(keyword: str, token: str) -> List[Dict[str, Any]]:
List of raw video dicts from the API.
"""
try:
# SC's /v1/youtube/search rejects ?keyword= with HTTP 400; the canonical
# parameter for that endpoint is `query`. Other SC endpoints use their
# own per-endpoint param names so this was the lone outlier.
data = http.get(
f"{SCRAPECREATORS_YT_BASE}/search",
params={"keyword": keyword},
params={"query": keyword},
headers=http.scrapecreators_headers(token),
timeout=30,
retries=2,
+122
View File
@@ -0,0 +1,122 @@
#!/bin/bash
# Store last30days API keys in the macOS Keychain.
#
# Keys are stored as generic passwords with service name `last30days-<KEY>`
# for the current user. The lib/env.py loader picks them up automatically as
# the lowest-priority credential source on Darwin.
#
# Usage:
# ./setup-keychain.sh # interactive: prompts for each key
# ./setup-keychain.sh KEY [KEY..] # prompt only for the listed keys
# ./setup-keychain.sh --list # list which last30days-* items exist
# ./setup-keychain.sh --delete KEY # remove a stored key
#
# Existing values are shown as "(set)" and skipped unless --replace is passed.
# Skip any prompt with empty input.
set -euo pipefail
PREFIX="last30days-"
# Mirrors lib/env.py::KEYCHAIN_KEYS — kept in sync via
# tests/test_env_keychain.py::test_keychain_keys_match_setup_script.
ALL_KEYS=(
OPENAI_API_KEY
XAI_API_KEY
GOOGLE_API_KEY
GEMINI_API_KEY
GOOGLE_GENAI_API_KEY
SCRAPECREATORS_API_KEY
APIFY_API_TOKEN
AUTH_TOKEN
CT0
BSKY_HANDLE
BSKY_APP_PASSWORD
TRUTHSOCIAL_TOKEN
BRAVE_API_KEY
EXA_API_KEY
SERPER_API_KEY
OPENROUTER_API_KEY
PARALLEL_API_KEY
XQUIK_API_KEY
XIAOHONGSHU_API_BASE
)
if [[ "${OSTYPE:-}" != darwin* ]]; then
echo "setup-keychain.sh requires macOS (security command). Got: $OSTYPE" >&2
exit 1
fi
if ! command -v security >/dev/null 2>&1; then
echo "security command not found on PATH" >&2
exit 1
fi
REPLACE=0
ACTION="prompt"
TARGETS=()
while [[ $# -gt 0 ]]; do
case "$1" in
--list) ACTION="list"; shift ;;
--delete) ACTION="delete"; shift ;;
--replace) REPLACE=1; shift ;;
--help|-h) sed -n '2,/^$/p' "$0" | sed 's/^# //; s/^#//'; exit 0 ;;
-*) echo "unknown flag: $1" >&2; exit 2 ;;
*) TARGETS+=("$1"); shift ;;
esac
done
case "$ACTION" in
list)
echo "Stored last30days-* keychain items:"
for key in "${ALL_KEYS[@]}"; do
if security find-generic-password -a "$USER" -s "${PREFIX}${key}" -w >/dev/null 2>&1; then
echo " $key"
fi
done
exit 0
;;
delete)
if [[ ${#TARGETS[@]} -eq 0 ]]; then
echo "--delete needs at least one KEY name" >&2; exit 2
fi
for key in "${TARGETS[@]}"; do
if security delete-generic-password -a "$USER" -s "${PREFIX}${key}" >/dev/null 2>&1; then
echo "deleted: $key"
else
echo "not found: $key"
fi
done
exit 0
;;
esac
if [[ ${#TARGETS[@]} -eq 0 ]]; then
TARGETS=("${ALL_KEYS[@]}")
fi
added=0; skipped=0; replaced=0
for key in "${TARGETS[@]}"; do
existing="$(security find-generic-password -a "$USER" -s "${PREFIX}${key}" -w 2>/dev/null || true)"
if [[ -n "$existing" && "$REPLACE" -eq 0 ]]; then
printf " %-28s (set, skipping — use --replace to overwrite)\n" "$key"
skipped=$((skipped + 1))
continue
fi
printf " %-28s " "$key"
IFS= read -rs value
echo
if [[ -z "$value" ]]; then
skipped=$((skipped + 1))
continue
fi
security add-generic-password -U -a "$USER" -s "${PREFIX}${key}" -w "$value"
if [[ -n "$existing" ]]; then
replaced=$((replaced + 1))
else
added=$((added + 1))
fi
done
echo
echo "Done. added=$added replaced=$replaced skipped=$skipped"
echo "Verify with: $0 --list"
+125 -19
View File
@@ -14,7 +14,7 @@ import argparse
import json
import sqlite3
import sys
from datetime import datetime, timedelta
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any, Dict, List, Optional
@@ -159,7 +159,30 @@ _UPDATABLE_FINDING_COLUMNS = frozenset({
})
# Future migrations keyed by version number
MIGRATIONS: Dict[int, str] = {}
MIGRATIONS: Dict[int, str] = {
2: """
CREATE TABLE IF NOT EXISTS finding_sightings (
id INTEGER PRIMARY KEY,
finding_id INTEGER NOT NULL REFERENCES findings(id) ON DELETE CASCADE,
run_id INTEGER REFERENCES research_runs(id) ON DELETE CASCADE,
topic_id INTEGER REFERENCES topics(id) ON DELETE CASCADE,
source TEXT NOT NULL,
source_url TEXT NOT NULL,
source_title TEXT,
engagement_score REAL,
relevance_score REAL,
seen_at TEXT DEFAULT (datetime('now')),
UNIQUE(run_id, finding_id)
);
CREATE INDEX IF NOT EXISTS idx_finding_sightings_run
ON finding_sightings(run_id, topic_id);
CREATE INDEX IF NOT EXISTS idx_finding_sightings_topic_seen
ON finding_sightings(topic_id, seen_at);
CREATE INDEX IF NOT EXISTS idx_finding_sightings_url
ON finding_sightings(source_url);
""",
}
def _connect(db_path: Optional[Path] = None) -> sqlite3.Connection:
@@ -423,6 +446,7 @@ def store_findings(
new_count = len(insert_rows)
updated_count = len(update_rows)
_record_sightings(conn, run_id, topic_id, with_urls, existing_by_url)
conn.execute(
"UPDATE research_runs SET findings_new = ?, findings_updated = ? WHERE id = ?",
(new_count, updated_count, run_id),
@@ -434,6 +458,84 @@ def store_findings(
return {"new": new_count, "updated": updated_count}
def _record_sightings(
conn: sqlite3.Connection,
run_id: int,
topic_id: int,
findings_with_urls: List[tuple[str, Dict[str, Any]]],
existing_by_url: Optional[Dict[str, sqlite3.Row]] = None,
) -> None:
"""Record the findings observed during this run.
The aggregate findings table keeps one row per URL and updates that row on
re-sighting. This ledger preserves the run/topic membership needed for
watchlist deltas and dossiers.
"""
if not findings_with_urls:
return
by_url = {url: finding for url, finding in findings_with_urls}
rows_by_url = dict(existing_by_url or {})
missing_urls = [url for url in by_url if url not in rows_by_url]
if missing_urls:
placeholders = ",".join("?" for _ in missing_urls)
rows = conn.execute(
f"SELECT id, source_url FROM findings WHERE source_url IN ({placeholders})",
missing_urls,
).fetchall()
rows_by_url.update({row["source_url"]: row for row in rows})
sighting_rows = []
for url, finding in by_url.items():
row = rows_by_url.get(url)
if row is None:
continue
sighting_rows.append((
row["id"],
run_id,
topic_id,
finding.get("source", "unknown"),
url,
finding.get("source_title") or finding.get("title", ""),
finding.get("engagement_score", 0),
finding.get("relevance_score", 0),
))
if not sighting_rows:
return
conn.executemany(
"""INSERT INTO finding_sightings
(finding_id, run_id, topic_id, source, source_url, source_title,
engagement_score, relevance_score)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(run_id, finding_id) DO UPDATE SET
topic_id = excluded.topic_id,
source = excluded.source,
source_url = excluded.source_url,
source_title = excluded.source_title,
engagement_score = excluded.engagement_score,
relevance_score = excluded.relevance_score""",
sighting_rows,
)
def get_sightings_for_run(topic_id: int, run_id: int) -> List[Dict[str, Any]]:
"""Return findings observed for a topic during a specific run."""
conn = _connect()
try:
rows = conn.execute(
"""SELECT * FROM finding_sightings
WHERE topic_id = ? AND run_id = ?
ORDER BY id""",
(topic_id, run_id),
).fetchall()
return [dict(r) for r in rows]
finally:
conn.close()
def get_new_findings(
topic_id: int,
since: Optional[str] = None,
@@ -519,7 +621,7 @@ def get_daily_cost(date: Optional[str] = None) -> float:
conn = _connect()
try:
if not date:
date = datetime.now().strftime("%Y-%m-%d")
date = datetime.now(timezone.utc).strftime("%Y-%m-%d")
row = conn.execute(
"""SELECT COALESCE(SUM(token_cost), 0) as total
FROM research_runs
@@ -575,7 +677,7 @@ def get_stats() -> Dict[str, Any]:
topic_count = conn.execute("SELECT COUNT(*) FROM topics WHERE enabled = 1").fetchone()[0]
finding_count = conn.execute("SELECT COUNT(*) FROM findings").fetchone()[0]
week_ago = (datetime.now() - timedelta(days=7)).strftime("%Y-%m-%d")
week_ago = (datetime.now(timezone.utc) - timedelta(days=7)).strftime("%Y-%m-%d")
runs_7d = conn.execute(
"SELECT COUNT(*) FROM research_runs WHERE run_date >= ?", (week_ago,)
).fetchone()[0]
@@ -621,7 +723,7 @@ def get_trending(days: int = 7) -> List[Dict[str, Any]]:
"""Get topics ranked by recent finding activity."""
conn = _connect()
try:
since = (datetime.now() - timedelta(days=days)).strftime("%Y-%m-%d")
since = (datetime.now(timezone.utc) - timedelta(days=days)).strftime("%Y-%m-%d")
rows = conn.execute(
"""SELECT t.name, t.id,
COUNT(f.id) as new_findings,
@@ -673,27 +775,31 @@ def findings_from_report(
limit: Optional[int] = None,
) -> List[Dict[str, Any]]:
"""Convert report into persisted findings.
Uses ranked candidates (post-rerank) when available for quality scores and explanations.
Supplements with raw items from items_by_source for HN/PM that didn't rank highly
but are valuable for watchlist persistence.
but are valuable for watchlist persistence. When ranked_candidates is empty
(degraded path rerank failed or was skipped), falls back to supplementing
all sources from items_by_source so findings aren't silently dropped.
"""
findings = []
seen_urls = set()
# Phase 1: Process ranked candidates (high-quality data with explanations and corroboration)
for candidate in report.ranked_candidates:
finding = finding_from_candidate(candidate)
findings.append(finding)
findings.append(finding_from_candidate(candidate))
seen_urls.add(candidate.url)
# Phase 2: Add HN/PM items not already captured in ranked candidates
for source_name in ["hackernews", "polymarket"]:
supplement_sources = (
list(report.items_by_source)
if not report.ranked_candidates
else ["hackernews", "polymarket"]
)
for source_name in supplement_sources:
if source_name not in report.items_by_source:
continue
for item in report.items_by_source[source_name]:
if item.url in seen_urls:
continue # Already captured with rich data
continue
findings.append({
"source": source_name,
"source_url": item.url,
@@ -705,8 +811,7 @@ def findings_from_report(
"relevance_score": item.local_relevance or 0.5,
})
seen_urls.add(item.url)
# Apply global limit after collecting all findings (fix: was per-source, now global)
return findings[:limit] if limit is not None else findings
@@ -722,9 +827,10 @@ def _cli_query(args):
since = None
if args.since:
# Parse duration like "7d", "30d"
# Parse duration like "7d", "30d". Use UTC to match SQLite's
# datetime('now') which writes first_seen in UTC.
days = int(args.since.rstrip("d"))
since = (datetime.now() - timedelta(days=days)).strftime("%Y-%m-%d")
since = (datetime.now(timezone.utc) - timedelta(days=days)).strftime("%Y-%m-%d")
findings = get_new_findings(topic["id"], since)
print(json.dumps({"topic": topic["name"], "findings": findings, "count": len(findings)}, default=str))
+3 -2
View File
@@ -6,7 +6,8 @@ set -euo pipefail
# using `claude --print` to capture real end-to-end output.
SKILL_DIR="$HOME/.claude/skills/last30days"
REPO_DIR="/Users/mvanhorn/last30days-skill"
REPO_DIR="${REPO_DIR:-$(cd "$(dirname "$0")/.." && pwd)}"
CLAUDE="${CLAUDE:-$(command -v claude || echo claude)}"
# Safety: always restore V2 SKILL.md on exit/crash
cleanup() {
@@ -101,7 +102,7 @@ run_version() {
# Run claude --print with the skill invocation
# No timeout — claude --print exits on its own; kill manually if stuck
if /Users/mvanhorn/.local/bin/claude --print \
if "$CLAUDE" --print \
"/last30days $query" \
> "$outfile" 2>"$errfile"; then
local end_time
+74
View File
@@ -232,5 +232,79 @@ class TestVendoredBirdRuntime(unittest.TestCase):
self.assertEqual(5, items[0]["engagement"]["likes"])
class TestRunBirdSearchJsonDecodeRetry(unittest.TestCase):
"""When bird-search returns non-JSON stdout, retry the subprocess.
Twitter's edge sometimes serves an HTML anti-bot interstitial in place of
JSON. Before this fix, that response made json.loads raise JSONDecodeError
and the function returned {"items": []} with no diagnostic silent-empty
against an orchestrator that can't distinguish "Twitter blocked us" from
"no tweets matched the query."
"""
def _make_result(self, stdout: str, stderr: str = "", returncode: int = 0):
from lib.subproc import SubprocResult
return SubprocResult(returncode=returncode, stdout=stdout, stderr=stderr)
def test_retries_subprocess_on_html_interstitial_then_succeeds(self):
"""First subprocess attempt returns HTML; second returns JSON → success."""
from unittest import mock
from lib import bird_x
html_interstitial = "<!DOCTYPE html><html><body>Rate limited</body></html>"
json_success = '[{"id": "1", "text": "tweet"}]'
results = [
(self._make_result(stdout=html_interstitial), None),
(self._make_result(stdout=json_success), None),
]
with mock.patch.object(bird_x, "_invoke_bird_subprocess", side_effect=results), \
mock.patch.object(bird_x.time, "sleep") as mock_sleep:
response = bird_x._run_bird_search("test", count=10, timeout=30)
self.assertNotIn("error", response)
self.assertEqual(response["items"], [{"id": "1", "text": "tweet"}])
# Should have slept between the failed first attempt and the retry.
mock_sleep.assert_called_once_with(bird_x.JSON_DECODE_RETRY_DELAY)
def test_returns_error_after_all_retries_exhausted(self):
"""All attempts return HTML → error dict with diagnostic + items=[]."""
from unittest import mock
from lib import bird_x
html_interstitial = "<!DOCTYPE html><html>blocked</html>"
results = [
(self._make_result(stdout=html_interstitial), None),
(self._make_result(stdout=html_interstitial), None),
]
with mock.patch.object(bird_x, "_invoke_bird_subprocess", side_effect=results), \
mock.patch.object(bird_x.time, "sleep"):
response = bird_x._run_bird_search("test", count=10, timeout=30)
self.assertIn("error", response)
self.assertIn("Invalid JSON response", response["error"])
# Diagnostic message names the anti-bot interstitial so it's
# distinguishable from a genuine no-results case in logs.
self.assertIn("anti-bot interstitial", response["error"].lower())
self.assertEqual(response["items"], [])
def test_terminal_subprocess_error_is_not_retried(self):
"""Subprocess timeout / spawn failure → terminal error, no retry."""
from unittest import mock
from lib import bird_x
timeout_error = {"error": "Search timed out after 30s", "items": []}
results = [(None, timeout_error)]
with mock.patch.object(bird_x, "_invoke_bird_subprocess", side_effect=results), \
mock.patch.object(bird_x.time, "sleep") as mock_sleep:
response = bird_x._run_bird_search("test", count=10, timeout=30)
self.assertEqual(response, timeout_error)
mock_sleep.assert_not_called()
if __name__ == "__main__":
unittest.main()
+147
View File
@@ -1,5 +1,6 @@
"""Tests for bluesky module."""
import os
import sys
import unittest
from pathlib import Path
@@ -211,5 +212,151 @@ class TestSearchBlueskyAuth(unittest.TestCase):
self.assertEqual(mock_request.call_args_list[3].kwargs.get("headers", {}), {"Authorization": "Bearer tok-new"})
class TestSearchEndpointHostResolution(unittest.TestCase):
"""The default search host moved from `public.api.bsky.app` (the
unauthenticated public mirror, now BunnyCDN-blocked for searchPosts) to
`api.bsky.app` (the canonical authenticated AppView). BSKY_SEARCH_HOST
env var or config value can override the default if Bluesky migrates
infrastructure again. Same os.environ-or-config hybrid pattern as
LAST30DAYS_STORE.
"""
def setUp(self):
# Snapshot env so per-test overrides don't leak
self._saved_env = os.environ.pop("BSKY_SEARCH_HOST", None)
def tearDown(self):
if self._saved_env is not None:
os.environ["BSKY_SEARCH_HOST"] = self._saved_env
else:
os.environ.pop("BSKY_SEARCH_HOST", None)
def test_resolver_default_uses_canonical_appview(self):
# Regression guard against the public mirror reappearing as the default.
# Anchored at the resolver because that is the code path search_bluesky
# actually calls; a module-level constant would not catch a resolver
# regression.
self.assertIn("api.bsky.app", bluesky._resolve_search_url())
def test_resolver_default_does_not_use_public_mirror(self):
# Hard regression guard — the exact host that BunnyCDN was blocking.
# Asserted at the resolver level (the runtime path) so a default-host
# regression in _resolve_search_url is actually caught.
self.assertNotIn("public.api.bsky.app", bluesky._resolve_search_url())
def test_resolver_default_when_no_override(self):
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_env_var_override(self):
os.environ["BSKY_SEARCH_HOST"] = "staging.bsky.app"
self.assertEqual(
bluesky._resolve_search_url(),
"https://staging.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_config_dict_override(self):
# User has BSKY_SEARCH_HOST only in .env file (project loads .env into
# config, not os.environ). Resolver must read both.
url = bluesky._resolve_search_url({"BSKY_SEARCH_HOST": "pds.example.com"})
self.assertEqual(url, "https://pds.example.com/xrpc/app.bsky.feed.searchPosts")
def test_resolver_env_var_wins_over_config(self):
# When both are set, os.environ takes precedence (matches LAST30DAYS_STORE)
os.environ["BSKY_SEARCH_HOST"] = "shell-host.example"
url = bluesky._resolve_search_url({"BSKY_SEARCH_HOST": "config-host.example"})
self.assertIn("shell-host.example", url)
self.assertNotIn("config-host.example", url)
def test_resolver_output_does_not_use_public_mirror(self):
# Regression guard at the resolver level (not just the constant) —
# this is what runtime actually calls. The constant-level guard
# above doesn't catch a regression where the resolver reverts.
self.assertNotIn("public.api.bsky.app", bluesky._resolve_search_url())
def test_resolver_strips_surrounding_whitespace(self):
# Pre-fix: " api.bsky.app " produced "https:// api.bsky.app /xrpc/..."
# which urllib raises ValueError on with no hint the env var caused it.
os.environ["BSKY_SEARCH_HOST"] = " api.bsky.app "
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_rejects_embedded_path(self):
# "my-proxy.com/xrpc/prefix" would have doubled the /xrpc/ segment.
# We fall back to the default to avoid a guaranteed 404.
os.environ["BSKY_SEARCH_HOST"] = "my-proxy.example.com/xrpc/prefix"
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_strips_embedded_scheme(self):
# Users who paste a full URL get a sane outcome, not a malformed URL.
os.environ["BSKY_SEARCH_HOST"] = "https://api.bsky.app"
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_empty_string_falls_back_to_default(self):
os.environ["BSKY_SEARCH_HOST"] = ""
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
class TestAppPasswordFormat(unittest.TestCase):
"""Bluesky app passwords are 19-char xxxx-xxxx-xxxx-xxxx (lowercase
alphanumeric, three hyphens at fixed positions). Main-account passwords
are accepted by createSession but are bad hygiene. The validator detects
the format mismatch without gating any caller.
"""
def test_accepts_valid_app_password_form(self):
# Use a fake example — never a real password
self.assertTrue(bluesky._validate_app_password_format("wfwp-cq7o-5six-7wy5"))
def test_rejects_length_15_string(self):
# The exact failure mode that triggered the 2026-05-04 investigation:
# user stored their main login password (15 chars) in BSKY_APP_PASSWORD
self.assertFalse(bluesky._validate_app_password_format("mainpassword123"))
def test_rejects_16_char_no_hyphen_string(self):
# Hex-style API key shape — common confusion with other services
self.assertFalse(bluesky._validate_app_password_format("abcdef0123456789"))
def test_rejects_uppercase_letters(self):
# Bluesky app passwords are all-lowercase by spec
self.assertFalse(bluesky._validate_app_password_format("WFWP-cq7o-5six-7wy5"))
def test_rejects_underscore_separator(self):
# Wrong separator
self.assertFalse(bluesky._validate_app_password_format("wfwp_cq7o_5six_7wy5"))
def test_rejects_special_chars_in_groups(self):
# Special characters are not part of the alphanumeric class
self.assertFalse(bluesky._validate_app_password_format("wfwp-cq7o-5six-7wy@"))
def test_rejects_empty_string(self):
self.assertFalse(bluesky._validate_app_password_format(""))
def test_rejects_none(self):
# Callers may pass config.get('BSKY_APP_PASSWORD') which is None when unset
self.assertFalse(bluesky._validate_app_password_format(None))
def test_rejects_integer(self):
# Defensive: don't crash if a numeric value sneaks in
self.assertFalse(bluesky._validate_app_password_format(123456789012345))
def test_rejects_list(self):
# Defensive: don't crash on iterables
self.assertFalse(bluesky._validate_app_password_format(["wfwp", "cq7o", "5six", "7wy5"]))
if __name__ == "__main__":
unittest.main()
+2 -2
View File
@@ -286,7 +286,7 @@ class TestFullExtraction:
with mock.patch("scripts.lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)):
with mock.patch(
"scripts.lib.chrome_cookies._get_chrome_encryption_key",
"scripts.lib.chrome_cookies._get_chromium_encryption_key",
return_value=KNOWN_PASSPHRASE,
):
result = extract_chrome_cookies_macos(".x.com", ["auth_token", "ct0"])
@@ -319,7 +319,7 @@ class TestFullExtraction:
with mock.patch("scripts.lib.chrome_cookies.CHROME_COOKIES_DB", Path(db_path)):
with mock.patch(
"scripts.lib.chrome_cookies._get_chrome_encryption_key",
"scripts.lib.chrome_cookies._get_chromium_encryption_key",
return_value=KNOWN_PASSPHRASE,
):
result = extract_chrome_cookies_macos(".x.com", ["auth_token"])
+94 -4
View File
@@ -1,6 +1,7 @@
# ruff: noqa: E402
import json
import io
import shutil
import tempfile
import subprocess
import sys
@@ -27,8 +28,8 @@ class CliV3Tests(unittest.TestCase):
generated_at="2026-03-16T00:00:00+00:00",
provider_runtime=schema.ProviderRuntime(
reasoning_provider="gemini",
planner_model="gemini-3.1-flash-lite-preview",
rerank_model="gemini-3.1-flash-lite-preview",
planner_model="gemini-3.1-flash-lite",
rerank_model="gemini-3.1-flash-lite",
),
query_plan=schema.QueryPlan(
intent="comparison",
@@ -71,6 +72,26 @@ class CliV3Tests(unittest.TestCase):
cli.parse_search_flag("web, reddit, hn, web"),
)
def test_parse_search_flag_accepts_optional_social_sources(self):
self.assertEqual(
["threads", "pinterest"],
cli.parse_search_flag("threads, pinterest"),
)
def test_explicit_threads_search_uses_scrapecreators_key_without_include_sources(self):
available = cli.pipeline.available_sources(
{"SCRAPECREATORS_API_KEY": "test-key", "INCLUDE_SOURCES": ""},
requested_sources=["threads"],
)
self.assertIn("threads", available)
def test_explicit_perplexity_search_uses_openrouter_key_without_include_sources(self):
available = cli.pipeline.available_sources(
{"OPENROUTER_API_KEY": "test-key", "INCLUDE_SOURCES": ""},
requested_sources=["perplexity"],
)
self.assertIn("perplexity", available)
def test_parse_search_flag_rejects_invalid_or_empty_inputs(self):
with self.assertRaises(SystemExit):
cli.parse_search_flag("unknown")
@@ -114,13 +135,13 @@ class CliV3Tests(unittest.TestCase):
def test_slugify_and_emit_output_cover_supported_modes(self):
report = self.make_report()
self.assertEqual("openclaw-vs-nanoclaw", cli.slugify(report.topic))
self.assertEqual("last30days v3.0.0 CLI.", cli.__doc__)
self.assertEqual("last30days CLI.", cli.__doc__)
compact = cli.emit_output(report, "compact")
json_output = cli.emit_output(report, "json")
context = cli.emit_output(report, "context")
self.assertIn("# last30days v3.0.0", compact)
self.assertIn("# last30days v", compact)
self.assertIn('"topic": "OpenClaw vs NanoClaw"', json_output)
self.assertIsInstance(context, str)
@@ -143,6 +164,30 @@ class CliV3Tests(unittest.TestCase):
_, kwargs = write_text.call_args
self.assertEqual("utf-8", kwargs.get("encoding"))
def test_compute_save_path_display_uses_posix_slashes_under_home(self):
# Regression: f"~/{relative}" stringified pathlib.Path with the
# OS-native separator, producing "~/Documents\\Last30Days\\..." on
# Windows that no shell or File Explorer could open. The fix is
# f"~/{relative.as_posix()}" which forces forward slashes regardless
# of host OS. On POSIX hosts this asserts the contract for
# cross-platform safety; on Windows hosts it would fail without the fix.
real_home = Path.home()
tmp_under_home = Path(tempfile.mkdtemp(prefix="l30d_save_path_", dir=str(real_home)))
try:
save_dir = tmp_under_home / "Documents" / "Last30Days"
save_dir.mkdir(parents=True, exist_ok=True)
display = cli.compute_save_path_display(
str(save_dir), "british airways middle east", "v3", "compact"
)
self.assertTrue(display.startswith("~/"), f"Expected '~/' prefix, got: {display}")
self.assertNotIn("\\", display, f"Backslash leaked into display: {display}")
self.assertTrue(
display.endswith("british-airways-middle-east-raw-v3.md"),
f"Expected slug+suffix at end, got: {display}",
)
finally:
shutil.rmtree(tmp_under_home, ignore_errors=True)
def test_persist_report_updates_run_status_on_success_and_failure(self):
report = self.make_report()
@@ -215,6 +260,51 @@ class CliV3Tests(unittest.TestCase):
fake_progress.show_promo.assert_called_once_with("both", diag=diag)
self.assertIn("# rendered", stdout.getvalue())
def test_main_canonicalizes_explicit_github_repo_flags(self):
report = self.make_report()
diag = {
"available_sources": ["grounding"],
"providers": {"google": True, "openai": False, "xai": False},
"x_backend": None,
"bird_installed": True,
"bird_authenticated": False,
"bird_username": None,
"native_web_backend": "brave",
}
with mock.patch.object(cli.env, "get_config", return_value={}), \
mock.patch.object(cli.pipeline, "diagnose", return_value=diag), \
mock.patch.object(cli.pipeline, "run", return_value=report) as run_mock, \
mock.patch.object(cli, "emit_output", return_value="# rendered"), \
mock.patch.object(sys, "argv", [
"last30days.py",
"claude",
"code",
"vs",
"codex",
"--github-repo",
"openai/codex,anthropics/claude-code-action",
]):
stdout = io.StringIO()
stderr = io.StringIO()
with redirect_stdout(stdout), redirect_stderr(stderr):
rc = cli.main()
self.assertEqual(0, rc)
# In vs-mode main + competitors run in parallel via ThreadPoolExecutor,
# so the order of pipeline.run invocations is non-deterministic. Find
# the main runner's call by predicate on the canonicalized github_repos
# rather than by index.
expected_repos = ["openai/codex", "anthropics/claude-code"]
main_call = next(
(c for c in run_mock.call_args_list if c.kwargs.get("github_repos") == expected_repos),
None,
)
self.assertIsNotNone(
main_call,
f"No pipeline.run call had github_repos={expected_repos}; "
f"saw {[c.kwargs.get('github_repos') for c in run_mock.call_args_list]}",
)
self.assertIn("[GitHub] Canonicalized repos:", stderr.getvalue())
if __name__ == "__main__":
unittest.main()
+2 -1
View File
@@ -114,9 +114,10 @@ class TestGetConfigCookieIntegration:
@patch("lib.cookie_extract.extract_cookies")
@patch("lib.env._find_project_env", return_value=None)
@patch("lib.env.load_env_file", return_value={})
@patch("lib.env._load_keychain", return_value={})
@patch("lib.env.get_openai_auth")
def test_get_config_injects_cookies(
self, mock_openai, mock_load, mock_proj, mock_extract
self, mock_openai, mock_keychain, mock_load, mock_proj, mock_extract
):
from lib.env import get_config, OpenAIAuth
mock_openai.return_value = OpenAIAuth(
+182
View File
@@ -0,0 +1,182 @@
"""Tests for macOS Keychain credential source in lib/env.py.
Covers:
- non-Darwin returns {}
- missing `security` binary returns {}
- successful lookups return parsed key/value pairs
- subprocess timeout / OSError are swallowed
- get_config merges keychain at lowest priority and labels _CONFIG_SOURCE
"""
from __future__ import annotations
import re
import subprocess
import sys
from pathlib import Path
from unittest import mock
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
from lib import env # noqa: E402
SETUP_KEYCHAIN_SH = Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts" / "setup-keychain.sh"
# ---------------------------------------------------------------------------
# _load_keychain unit tests
# ---------------------------------------------------------------------------
def test_load_keychain_returns_empty_on_non_darwin():
with mock.patch("platform.system", return_value="Linux"):
assert env._load_keychain(["XAI_API_KEY"]) == {}
def test_load_keychain_returns_empty_when_security_missing():
with mock.patch("platform.system", return_value="Darwin"), \
mock.patch("shutil.which", return_value=None):
assert env._load_keychain(["XAI_API_KEY"]) == {}
def _run_result(returncode: int, stdout: str = "") -> subprocess.CompletedProcess:
return subprocess.CompletedProcess(args=[], returncode=returncode, stdout=stdout, stderr="")
def test_load_keychain_loads_present_keys_skips_missing():
def fake_run(cmd, **kwargs):
service = cmd[cmd.index("-s") + 1]
if service == "last30days-XAI_API_KEY":
return _run_result(0, "xai-abc\n")
if service == "last30days-BRAVE_API_KEY":
return _run_result(0, "brv-xyz\n")
return _run_result(44) # security's "not found" exit code
with mock.patch("platform.system", return_value="Darwin"), \
mock.patch("shutil.which", return_value="/usr/bin/security"), \
mock.patch("subprocess.run", side_effect=fake_run):
result = env._load_keychain(["XAI_API_KEY", "BRAVE_API_KEY", "OPENAI_API_KEY"])
assert result == {"XAI_API_KEY": "xai-abc", "BRAVE_API_KEY": "brv-xyz"}
def test_load_keychain_strips_whitespace_and_newlines():
with mock.patch("platform.system", return_value="Darwin"), \
mock.patch("shutil.which", return_value="/usr/bin/security"), \
mock.patch("subprocess.run", return_value=_run_result(0, " hello-key \n")):
result = env._load_keychain(["FOO"])
assert result == {"FOO": "hello-key"}
def test_load_keychain_swallows_subprocess_errors():
def fake_run(cmd, **kwargs):
raise subprocess.TimeoutExpired(cmd=cmd, timeout=5)
with mock.patch("platform.system", return_value="Darwin"), \
mock.patch("shutil.which", return_value="/usr/bin/security"), \
mock.patch("subprocess.run", side_effect=fake_run):
assert env._load_keychain(["XAI_API_KEY"]) == {}
def test_load_keychain_swallows_oserror():
with mock.patch("platform.system", return_value="Darwin"), \
mock.patch("shutil.which", return_value="/usr/bin/security"), \
mock.patch("subprocess.run", side_effect=OSError("boom")):
assert env._load_keychain(["XAI_API_KEY"]) == {}
def test_load_keychain_skips_empty_stdout():
with mock.patch("platform.system", return_value="Darwin"), \
mock.patch("shutil.which", return_value="/usr/bin/security"), \
mock.patch("subprocess.run", return_value=_run_result(0, "")):
assert env._load_keychain(["XAI_API_KEY"]) == {}
# ---------------------------------------------------------------------------
# get_config integration tests
# ---------------------------------------------------------------------------
@pytest.fixture
def clean_env(monkeypatch, tmp_path):
"""Hide every key get_config might touch and point CONFIG_FILE at a
non-existent path so no real user config bleeds in."""
for var in [
"OPENAI_API_KEY", "XAI_API_KEY", "BRAVE_API_KEY", "AUTH_TOKEN", "CT0",
"SCRAPECREATORS_API_KEY", "APIFY_API_TOKEN", "BSKY_HANDLE",
"BSKY_APP_PASSWORD", "TRUTHSOCIAL_TOKEN", "EXA_API_KEY",
"SERPER_API_KEY", "OPENROUTER_API_KEY", "PARALLEL_API_KEY",
"XQUIK_API_KEY", "GOOGLE_API_KEY", "GEMINI_API_KEY",
"GOOGLE_GENAI_API_KEY", "INCLUDE_SOURCES", "FROM_BROWSER",
]:
monkeypatch.delenv(var, raising=False)
monkeypatch.setattr(env, "CONFIG_FILE", tmp_path / "does-not-exist.env")
monkeypatch.chdir(tmp_path) # no project .env in this tree either
def test_get_config_reports_keychain_source(clean_env):
with mock.patch.object(env, "_load_keychain", return_value={"XAI_API_KEY": "xai-from-kc"}):
cfg = env.get_config()
assert cfg["_CONFIG_SOURCE"] == "keychain"
assert cfg["XAI_API_KEY"] == "xai-from-kc"
def test_get_config_env_var_overrides_keychain(clean_env, monkeypatch):
monkeypatch.setenv("XAI_API_KEY", "xai-from-env")
with mock.patch.object(env, "_load_keychain", return_value={"XAI_API_KEY": "xai-from-kc"}):
cfg = env.get_config()
assert cfg["XAI_API_KEY"] == "xai-from-env"
def test_get_config_reports_env_only_when_keychain_empty(clean_env):
with mock.patch.object(env, "_load_keychain", return_value={}):
cfg = env.get_config()
assert cfg["_CONFIG_SOURCE"] == "env_only"
def test_get_config_global_file_outranks_keychain(clean_env, tmp_path, monkeypatch):
cfg_file = tmp_path / "global.env"
cfg_file.write_text("XAI_API_KEY=xai-from-file\n")
monkeypatch.setattr(env, "CONFIG_FILE", cfg_file)
with mock.patch.object(env, "_load_keychain", return_value={"XAI_API_KEY": "xai-from-kc"}):
cfg = env.get_config()
assert cfg["XAI_API_KEY"] == "xai-from-file"
assert cfg["_CONFIG_SOURCE"].startswith("global:")
def test_get_config_openai_key_can_come_from_keychain(clean_env):
"""OPENAI_API_KEY must be visible to get_openai_auth via the keychain
merge wiring regression test."""
with mock.patch.object(env, "_load_keychain", return_value={"OPENAI_API_KEY": "sk-from-kc"}):
cfg = env.get_config()
assert cfg["OPENAI_API_KEY"] == "sk-from-kc"
assert cfg["OPENAI_AUTH_SOURCE"] == "api_key"
# ---------------------------------------------------------------------------
# Drift guard: lib/env.py KEYCHAIN_KEYS and setup-keychain.sh ALL_KEYS must
# stay in lockstep. A mismatch means users storing a key via the helper script
# wouldn't see it picked up by the loader, or vice versa.
# ---------------------------------------------------------------------------
def _parse_all_keys_from_shell(script: Path) -> list[str]:
text = script.read_text(encoding="utf-8")
match = re.search(r"ALL_KEYS=\(\s*(.*?)\s*\)", text, re.DOTALL)
if not match:
raise AssertionError(f"ALL_KEYS=( ... ) array not found in {script}")
body = match.group(1)
# Strip shell comments and split on whitespace
body = re.sub(r"#[^\n]*", "", body)
return [tok for tok in body.split() if tok]
def test_keychain_keys_match_setup_script():
shell_keys = _parse_all_keys_from_shell(SETUP_KEYCHAIN_SH)
python_keys = list(env.KEYCHAIN_KEYS)
assert shell_keys == python_keys, (
"lib/env.py::KEYCHAIN_KEYS and scripts/setup-keychain.sh::ALL_KEYS "
f"have drifted.\n python: {python_keys}\n shell: {shell_keys}"
)
+28
View File
@@ -41,6 +41,34 @@ class EnvV3Tests(unittest.TestCase):
with mock.patch.dict(os.environ, {}, clear=False):
self.assertIsNone(bird_x.is_bird_authenticated())
def test_file_permission_check_skips_windows_posix_mode_bits(self):
path = mock.Mock(spec=Path)
with mock.patch.object(env.os, "name", "nt"), mock.patch.object(env.sys.stderr, "write") as write:
env._check_file_permissions(path)
path.stat.assert_not_called()
write.assert_not_called()
class ThreadsAvailabilityTests(unittest.TestCase):
"""Threads is in the SC default-on family: same key, same per-call cost
shape as TikTok / Instagram, so the same default-on rule applies.
Suppression goes through EXCLUDE_SOURCES, not gated opt-in."""
def test_threads_available_with_sc_key_only(self):
self.assertTrue(env.is_threads_available({"SCRAPECREATORS_API_KEY": "k"}))
def test_threads_unavailable_without_sc_key(self):
self.assertFalse(env.is_threads_available({}))
self.assertFalse(env.is_threads_available({"INCLUDE_SOURCES": "threads"}))
def test_threads_does_not_require_include_sources(self):
"""Regression guard: INCLUDE_SOURCES should not be needed."""
self.assertTrue(env.is_threads_available({
"SCRAPECREATORS_API_KEY": "k",
"INCLUDE_SOURCES": "",
}))
if __name__ == "__main__":
unittest.main()
+2 -2
View File
@@ -125,7 +125,7 @@ class EvaluatorV3Tests(unittest.TestCase):
topic="test topic",
query_type="general",
items=[{"key": "a"}],
judge_model="gemini-3.1-flash-lite-preview",
judge_model="gemini-3.1-flash-lite",
gemini_api_key="key",
)
self.assertEqual({"a": 3}, cached)
@@ -136,7 +136,7 @@ class EvaluatorV3Tests(unittest.TestCase):
topic="test topic",
query_type="general",
items=[],
judge_model="gemini-3.1-flash-lite-preview",
judge_model="gemini-3.1-flash-lite",
gemini_api_key=None,
)
self.assertEqual({}, skipped)
+23 -4
View File
@@ -6,6 +6,7 @@ from __future__ import annotations
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
@@ -27,13 +28,31 @@ class FooterNudgeSuppressionTests(unittest.TestCase):
"--emit=md",
*argv,
]
env = {**os.environ, "LAST30DAYS_SKIP_PREFLIGHT": "1"}
env = {
**os.environ,
"LAST30DAYS_SKIP_PREFLIGHT": "1",
# Skip ~/.config/last30days/.env so a contributor's saved
# BRAVE/EXA/SERPER/PARALLEL key doesn't make grounding "available"
# and suppress the promo we're checking for.
"LAST30DAYS_CONFIG_DIR": "",
# Pin X as available so _missing_sources_for_promo selects "web"
# (otherwise the "x" promo wins and the BRAVE_API_KEY string never
# appears).
"XAI_API_KEY": "test-stub",
}
# Strip any grounded-web keys the host might have so the promo path
# triggers deterministically in mock + no-backend.
# triggers deterministically in mock + no-backend. Also strip X cookie
# credentials so XAI_API_KEY is the unambiguous X backend.
for key in ("BRAVE_API_KEY", "EXA_API_KEY", "SERPER_API_KEY",
"PARALLEL_API_KEY", "OPENROUTER_API_KEY"):
"PARALLEL_API_KEY", "OPENROUTER_API_KEY",
"AUTH_TOKEN", "CT0", "LAST30DAYS_X_BACKEND"):
env.pop(key, None)
return subprocess.run(cmd, capture_output=True, text=True, env=env)
# Run from a tmpdir so _find_project_env() can't walk up into any
# .claude/last30days.env above the repo on the contributor's machine.
with tempfile.TemporaryDirectory() as tmp:
return subprocess.run(
cmd, capture_output=True, text=True, env=env, cwd=tmp,
)
def test_bare_run_emits_web_promo(self):
result = self._run(topic="OpenAI")
+149
View File
@@ -122,6 +122,54 @@ class ExaSearchTests(unittest.TestCase):
self.assertEqual(0, artifact["resultCount"])
class ParallelSearchTests(unittest.TestCase):
def test_parallel_search_filters_to_in_range_dated_items(self):
mock_response = {
"results": [
{
"title": "Parallel Result",
"url": "https://example.com/parallel",
"snippet": "A parallel snippet",
"publish_date": "2026-03-15T00:00:00Z",
},
{
"title": "Old Parallel Result",
"url": "https://example.com/old-parallel",
"snippet": "Should be filtered",
"publish_date": "2025-12-01T00:00:00Z",
},
{
"title": "Undated Parallel Result",
"url": "https://example.com/undated-parallel",
"snippet": "Should also be filtered",
},
]
}
with patch("lib.grounding.http.request", return_value=mock_response) as mock_req:
items, artifact = grounding.parallel_search(
"test", ("2026-02-25", "2026-03-27"), "fake-parallel-key"
)
self.assertEqual(1, len(items))
self.assertEqual("Parallel Result", items[0]["title"])
self.assertEqual("https://example.com/parallel", items[0]["url"])
self.assertEqual("2026-03-15", items[0]["date"])
self.assertTrue(items[0]["id"].startswith("WP"))
self.assertEqual("parallel", artifact["label"])
self.assertEqual(1, artifact["resultCount"])
self.assertEqual("POST", mock_req.call_args.args[0])
self.assertEqual("https://api.parallel.ai/v1/search", mock_req.call_args.args[1])
self.assertEqual(
"Bearer fake-parallel-key",
mock_req.call_args.kwargs["headers"]["Authorization"],
)
def test_parallel_search_returns_empty_for_no_results(self):
with patch("lib.grounding.http.request", return_value={"results": []}):
items, artifact = grounding.parallel_search("test", ("2026-02-25", "2026-03-27"), "key")
self.assertEqual([], items)
self.assertEqual(0, artifact["resultCount"])
class WebSearchDispatchTests(unittest.TestCase):
def test_auto_selects_brave_when_key_present(self):
config = {"BRAVE_API_KEY": "test-key"}
@@ -141,6 +189,12 @@ class WebSearchDispatchTests(unittest.TestCase):
grounding.web_search("test", ("2026-02-25", "2026-03-27"), config, backend="auto")
mock.assert_called_once()
def test_auto_selects_parallel_when_only_parallel_key(self):
config = {"PARALLEL_API_KEY": "test-key"}
with patch("lib.grounding.parallel_search", return_value=([], {})) as mock:
grounding.web_search("test", ("2026-02-25", "2026-03-27"), config, backend="auto")
mock.assert_called_once()
def test_auto_returns_empty_when_no_keys(self):
items, artifact = grounding.web_search("test", ("2026-02-25", "2026-03-27"), {}, backend="auto")
self.assertEqual([], items)
@@ -167,6 +221,14 @@ class WebSearchDispatchTests(unittest.TestCase):
mock_exa.assert_called_once()
mock_serper.assert_not_called()
def test_auto_prefers_serper_over_parallel(self):
config = {"SERPER_API_KEY": "serper-key", "PARALLEL_API_KEY": "parallel-key"}
with patch("lib.grounding.serper_search", return_value=([], {})) as mock_serper, \
patch("lib.grounding.parallel_search", return_value=([], {})) as mock_parallel:
grounding.web_search("test", ("2026-02-25", "2026-03-27"), config, backend="auto")
mock_serper.assert_called_once()
mock_parallel.assert_not_called()
def test_auto_prefers_brave_when_all_keys_present(self):
config = {"BRAVE_API_KEY": "brave-key", "EXA_API_KEY": "exa-key", "SERPER_API_KEY": "serper-key"}
with patch("lib.grounding.brave_search", return_value=([], {})) as mock_brave, \
@@ -185,10 +247,97 @@ class WebSearchDispatchTests(unittest.TestCase):
with self.assertRaises(RuntimeError):
grounding.web_search("test", ("2026-02-25", "2026-03-27"), {}, backend="brave")
def test_explicit_parallel_without_key_raises(self):
with self.assertRaises(RuntimeError):
grounding.web_search("test", ("2026-02-25", "2026-03-27"), {}, backend="parallel")
def test_unsupported_backend_raises(self):
with self.assertRaises(ValueError):
grounding.web_search("test", ("2026-02-25", "2026-03-27"), {}, backend="google")
class RedditEnrichmentGateTests(unittest.TestCase):
"""EXCLUDE_SOURCES=reddit must suppress the web-search Reddit enrichment.
Otherwise a user who explicitly excluded Reddit would still get Reddit
content smuggled back in via web-search URLs that happen to point at
reddit.com threads.
"""
def test_reddit_excluded_via_exclude_sources_skips_enrichment(self):
config = {"BRAVE_API_KEY": "k", "EXCLUDE_SOURCES": "reddit"}
items = [{"url": "https://www.reddit.com/r/python/comments/abc/title/", "snippet": "original"}]
with patch("lib.grounding.brave_search", return_value=(items, {})), \
patch("lib.grounding._enrich_reddit_items") as enrich_mock:
grounding.web_search("test", ("2026-02-25", "2026-03-27"), config, backend="auto")
enrich_mock.assert_not_called()
def test_reddit_excluded_case_insensitive(self):
for value in ("REDDIT", "Reddit", " reddit ", "x,reddit,y"):
config = {"BRAVE_API_KEY": "k", "EXCLUDE_SOURCES": value}
self.assertTrue(
grounding._reddit_excluded(config),
msg=f"_reddit_excluded should be True for EXCLUDE_SOURCES={value!r}",
)
def test_reddit_not_excluded_when_other_sources_listed(self):
config = {"EXCLUDE_SOURCES": "tiktok,instagram"}
self.assertFalse(grounding._reddit_excluded(config))
def test_enrichment_runs_when_reddit_not_excluded(self):
config = {"BRAVE_API_KEY": "k"}
items = [{"url": "https://www.reddit.com/r/python/comments/abc/title/", "snippet": "original"}]
with patch("lib.grounding.brave_search", return_value=(items, {})), \
patch("lib.grounding._enrich_reddit_items", return_value=items) as enrich_mock:
grounding.web_search("test", ("2026-02-25", "2026-03-27"), config, backend="auto")
enrich_mock.assert_called_once()
class RedditEnrichItemsTests(unittest.TestCase):
"""Direct tests for `_enrich_reddit_items` covering the selftext key path
and the RedditRateLimitError early-exit behavior.
"""
def test_selftext_under_submission_populates_snippet(self):
from lib import reddit_enrich
item = {
"url": "https://www.reddit.com/r/python/comments/abc/title/",
"snippet": "original",
}
parsed = {
"submission": {"selftext": "thread body content"},
"comments": [],
}
with patch.object(reddit_enrich, "fetch_thread_data", return_value={"raw": True}), \
patch.object(reddit_enrich, "parse_thread_data", return_value=parsed):
result = grounding._enrich_reddit_items([item])
self.assertEqual("thread body content", result[0]["snippet"])
self.assertEqual("reddit_json_api", result[0]["enriched_via"])
def test_rate_limit_error_halts_iteration(self):
from lib import reddit_enrich
item1 = {"url": "https://www.reddit.com/r/python/comments/aaa/x/"}
item2 = {"url": "https://www.reddit.com/r/python/comments/bbb/y/"}
def fake_fetch(url, *args, **kwargs):
raise reddit_enrich.RedditRateLimitError(f"429 for {url}")
captured_stderr: list[str] = []
with patch.object(reddit_enrich, "fetch_thread_data", side_effect=fake_fetch) as fetch_mock, \
patch("lib.grounding.sys.stderr.write", side_effect=lambda s: captured_stderr.append(s)):
grounding._enrich_reddit_items([item1, item2])
# Only the first item should have triggered a fetch attempt
self.assertEqual(1, fetch_mock.call_count)
# A stderr message about the rate-limit halt should have been emitted
self.assertTrue(
any("rate-limited" in msg.lower() or "rate limited" in msg.lower() for msg in captured_stderr),
msg=f"Expected a rate-limit stderr message, got: {captured_stderr!r}",
)
if __name__ == "__main__":
unittest.main()
+36 -4
View File
@@ -160,12 +160,44 @@ def test_title_matches_query_empty_query():
def test_title_matches_query_partial_match():
"""Test that all query words must match."""
"""Any-word matching: at least one query token in title is enough.
Previously required *all* tokens, which killed every hit on multi-keyword
theme queries like 'claude, personal agents, agentic infra' since no real
HN title contains all 5 tokens verbatim. Token-overlap relevance at parse
time still demotes weak matches, so the loosened gate is safe.
"""
title = "New AI framework"
query = "AI blockchain"
# "blockchain" is not in title, so should fail
assert hackernews._title_matches_query(title, query) is False
# "AI" matches as a whole word, even though "blockchain" doesn't appear
assert hackernews._title_matches_query(title, query) is True
def test_title_matches_query_no_token_in_title():
"""If no query token appears in the title at all, reject."""
assert hackernews._title_matches_query("New rust compiler", "AI blockchain") is False
def test_title_matches_query_word_boundary_not_substring():
"""Short tokens must match on word boundaries, not as substrings.
Without word-boundary matching, 'ai' would falsely match 'email',
'rail', 'artists', etc.
"""
# 'ai' as a substring of 'email' must not match
assert hackernews._title_matches_query("New email service", "ai blockchain") is False
# 'ai' as a whole word does match
assert hackernews._title_matches_query("Cool AI tool launched", "ai blockchain") is True
def test_title_matches_query_flattens_hyphens_and_commas():
"""Query tokens split on hyphens/commas the same way search_hackernews
flattens them, so the post-filter stays aligned with what Algolia saw."""
# query 'ts-bun-node' flattens to ['ts', 'bun', 'node']; title contains 'bun'
assert hackernews._title_matches_query("Bun 1.2 released", "ts-bun-node") is True
# query 'rust, go, zig' flattens; title contains 'go'
assert hackernews._title_matches_query("Go 1.24 generics update", "rust, go, zig") is True
# === Tests for search_hackernews() ===
+20
View File
@@ -277,6 +277,26 @@ class HtmlCliIntegrationTests(unittest.TestCase):
path = cli.compute_save_path_display("/tmp", report.topic, "v3", "html")
self.assertTrue(path.endswith("/ai-agent-frameworks-raw-html-v3.html"))
def test_save_output_can_persist_comparison_html(self):
reports = [
("OpenClaw", _report("OpenClaw", ["Containers"])),
("Hermes", _report("Hermes", ["Memory"])),
]
rendered = cli.emit_comparison_output(reports, "html")
with tempfile.TemporaryDirectory() as tmpdir:
path = cli.save_output(
reports[0][1],
"html",
tmpdir,
topic_override=cli.comparison_topic(reports),
rendered_content=rendered,
)
self.assertEqual("openclaw-vs-hermes-raw-html.html", path.name)
saved = path.read_text(encoding="utf-8")
self.assertIn("last30days · OpenClaw vs Hermes", saved)
self.assertIn("comparing 2: OpenClaw, Hermes", saved)
self.assertNotIn("last30days · OpenClaw</title>", saved)
if __name__ == "__main__":
unittest.main()
+114
View File
@@ -104,3 +104,117 @@ class TestParamsEncoding(unittest.TestCase):
sent_url = self._sent_url(mock_urlopen)
self.assertIn("count=25", sent_url)
self.assertIn("raw=True", sent_url)
class TestDNSResolutionRetry(unittest.TestCase):
"""DNS resolution failures (gaierror) must retry with exponential backoff.
Caller-passed `retries` values smaller than MIN_DNS_RETRIES are expanded
on the first gaierror so a transient resolution failure doesn't wipe a
request just because the caller passed retries=2.
"""
@patch("lib.http.urllib.request.urlopen")
@patch("lib.http.time.sleep")
def test_gaierror_retries_up_to_min_dns_retries_even_when_caller_passes_fewer(
self, mock_sleep, mock_urlopen
):
"""Caller passed retries=2; gaierror should still get MIN_DNS_RETRIES attempts."""
import socket
err = urllib.error.URLError(socket.gaierror(-2, "Name or service not known"))
mock_urlopen.side_effect = err
with self.assertRaises(http.HTTPError):
http.request("GET", "http://nonexistent.example", retries=2)
# Caller passed retries=2, but the budget expanded to MIN_DNS_RETRIES=3.
self.assertEqual(mock_urlopen.call_count, http.MIN_DNS_RETRIES)
@patch("lib.http.urllib.request.urlopen")
@patch("lib.http.time.sleep")
def test_gaierror_succeeds_after_transient_failure(self, mock_sleep, mock_urlopen):
"""gaierror on attempt 1, then success — should NOT raise."""
import socket
success_response = MagicMock()
success_response.read.return_value = b'{"ok": true}'
success_response.status = 200
success_response.__enter__ = lambda self: self
success_response.__exit__ = lambda *args: None
err = urllib.error.URLError(socket.gaierror(-2, "Name or service not known"))
mock_urlopen.side_effect = [err, success_response]
result = http.request("GET", "http://flaky.example", retries=2)
self.assertEqual(result, {"ok": True})
self.assertEqual(mock_urlopen.call_count, 2)
@patch("lib.http.urllib.request.urlopen")
@patch("lib.http.time.sleep")
def test_gaierror_uses_exponential_backoff(self, mock_sleep, mock_urlopen):
"""Backoff delays for gaierror should be 1s, 2s, 4s — not the linear default."""
import socket
err = urllib.error.URLError(socket.gaierror(-2, "Name or service not known"))
mock_urlopen.side_effect = err
with self.assertRaises(http.HTTPError):
http.request("GET", "http://nonexistent.example", retries=3)
# Expected sleep calls: 1s (after attempt 1), 2s (after attempt 2).
# No sleep after the final attempt (the loop exits to raise).
sleep_delays = [call.args[0] for call in mock_sleep.call_args_list]
self.assertEqual(sleep_delays, [1, 2])
@patch("lib.http.urllib.request.urlopen")
@patch("lib.http.time.sleep")
def test_non_dns_urlerror_uses_linear_backoff_not_dns_branch(
self, mock_sleep, mock_urlopen
):
"""A URLError that's NOT a gaierror must NOT expand the retry budget."""
# ConnectionRefusedError-style URLError reason (not gaierror)
err = urllib.error.URLError(ConnectionRefusedError(111, "Connection refused"))
mock_urlopen.side_effect = err
with self.assertRaises(http.HTTPError):
http.request("GET", "http://refused.example", retries=2)
# Caller passed retries=2, and non-DNS URLError doesn't expand it.
self.assertEqual(mock_urlopen.call_count, 2)
@patch("lib.http.urllib.request.urlopen")
@patch("lib.http.time.sleep")
def test_dns_widening_does_not_leak_into_subsequent_non_dns_urlerror(
self, mock_sleep, mock_urlopen
):
"""Mixed sequence: DNS-then-non-DNS must respect caller's original retries.
Without the fix, the first gaierror widens effective_retries from 2 to
MIN_DNS_RETRIES=3, and a subsequent ConnectionRefused on attempt 1
slips into a third overall attempt exceeding what the caller asked
for. Each non-DNS error path must gate on the original `retries`.
"""
import socket
dns_err = urllib.error.URLError(socket.gaierror(-2, "Name or service not known"))
conn_err = urllib.error.URLError(ConnectionRefusedError(111, "Connection refused"))
mock_urlopen.side_effect = [dns_err, conn_err, conn_err] # 3rd would only fire if budget leaked
with self.assertRaises(http.HTTPError):
http.request("GET", "http://flaky.example", retries=2)
# Caller asked for at most 2 attempts. DNS widening must not give us a 3rd.
self.assertEqual(mock_urlopen.call_count, 2)
@patch("lib.http.urllib.request.urlopen")
@patch("lib.http.time.sleep")
def test_dns_widening_does_not_leak_into_subsequent_oserror(
self, mock_sleep, mock_urlopen
):
"""Mixed sequence: DNS-then-OSError must respect caller's original retries."""
import socket
dns_err = urllib.error.URLError(socket.gaierror(-2, "Name or service not known"))
mock_urlopen.side_effect = [dns_err, TimeoutError("timed out"), TimeoutError("timed out")]
with self.assertRaises(http.HTTPError):
http.request("GET", "http://flaky.example", retries=2)
self.assertEqual(mock_urlopen.call_count, 2)
+169
View File
@@ -1,8 +1,10 @@
"""Tests for instagram.py — ScrapeCreators Instagram search module."""
import os
import sys
import unittest
from pathlib import Path
from unittest.mock import MagicMock, patch
# Add lib to path
sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "last30days" / "scripts"))
@@ -85,5 +87,172 @@ class TestInstagramDepthConfig(unittest.TestCase):
)
class TestHashtagFormCollapse(unittest.TestCase):
"""Tests for _to_hashtag_form() — the multi-word retry workaround."""
def test_collapses_spaces(self):
self.assertEqual(instagram._to_hashtag_form("toronto real estate"), "torontorealestate")
def test_lowercases(self):
self.assertEqual(instagram._to_hashtag_form("Toronto REAL Estate"), "torontorealestate")
def test_idempotent_on_single_word(self):
self.assertEqual(instagram._to_hashtag_form("ozempic"), "ozempic")
def test_handles_extra_whitespace(self):
self.assertEqual(instagram._to_hashtag_form(" toronto real estate "), "torontorealestate")
class TestSearchRetryOn500(unittest.TestCase):
"""Tests for the multi-word -> hashtag retry on SC's flaky 500 path.
SC's /v2/instagram/reels/search wraps Google Search and is documented
to be unreliable on multi-token queries. The retry collapses to a
hashtag form which hits the stable hashtag-page lookup path.
"""
def test_multiword_500_triggers_retry_with_hashtag_form(self):
"""Multi-word query 500 -> retry with collapsed hashtag form."""
from lib import http as http_module
first_error = http_module.HTTPError("HTTP 500: Server Error", 500, "")
second_payload = {"reels": []}
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.side_effect = [first_error, second_payload]
instagram.search_instagram(
"toronto real estate", "2026-04-01", "2026-05-04",
depth="default", token="fake-token",
)
self.assertEqual(mock_http_get.call_count, 2)
# First call: original multi-word query
first_params = mock_http_get.call_args_list[0].kwargs["params"]
self.assertEqual(first_params["query"], "toronto real estate")
# Second call: collapsed hashtag form
second_params = mock_http_get.call_args_list[1].kwargs["params"]
self.assertEqual(second_params["query"], "torontorealestate")
def test_singleword_500_does_not_retry(self):
"""Single-word query 500 has no spaces to collapse - no retry."""
from lib import http as http_module
only_error = http_module.HTTPError("HTTP 500: Server Error", 500, "")
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.side_effect = only_error
result = instagram.search_instagram(
"ozempic", "2026-04-01", "2026-05-04",
depth="default", token="fake-token",
)
self.assertEqual(mock_http_get.call_count, 1)
self.assertIn("error", result)
self.assertEqual(result["items"], [])
def test_first_call_succeeds_no_retry(self):
"""200 on first call -> retry path is never entered."""
from lib import http as http_module
ok_payload = {"reels": []}
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.return_value = ok_payload
instagram.search_instagram(
"toronto real estate", "2026-04-01", "2026-05-04",
depth="default", token="fake-token",
)
self.assertEqual(mock_http_get.call_count, 1)
def test_no_token_short_circuits(self):
"""No SCRAPECREATORS_API_KEY -> error returned without HTTP call."""
from lib import http as http_module
with patch.object(http_module, "get") as mock_http_get:
result = instagram.search_instagram(
"toronto real estate", "2026-04-01", "2026-05-04",
depth="default", token=None,
)
mock_http_get.assert_not_called()
self.assertIn("error", result)
self.assertIn("SCRAPECREATORS_API_KEY", result["error"])
class TestTranscriptTimeoutConfig(unittest.TestCase):
"""Tests for LAST30DAYS_TRANSCRIPT_TIMEOUT configuration.
SC's /v2/instagram/media/transcript endpoint regularly takes >15s,
so the timeout must be configurable. Default is DEFAULT_TRANSCRIPT_TIMEOUT
(30s); the env var or per-call kwarg overrides it.
"""
def setUp(self):
# Snapshot any pre-existing env so we don't leak across tests
self._saved_env = os.environ.pop("LAST30DAYS_TRANSCRIPT_TIMEOUT", None)
def tearDown(self):
os.environ.pop("LAST30DAYS_TRANSCRIPT_TIMEOUT", None)
if self._saved_env is not None:
os.environ["LAST30DAYS_TRANSCRIPT_TIMEOUT"] = self._saved_env
def _ok_payload(self):
return {"transcripts": [{"text": "hello world"}]}
def _video_item(self, vid="abc123"):
return {
"video_id": vid,
"url": f"https://www.instagram.com/reel/{vid}/",
"text": "",
}
def test_default_timeout_is_30s_when_nothing_set(self):
"""No env var, no kwarg -> request uses 30s, not the legacy 15s."""
from lib import http as http_module
items = [self._video_item()]
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.return_value = self._ok_payload()
instagram.fetch_captions(items, token="fake-token")
kwargs = mock_http_get.call_args.kwargs
self.assertEqual(kwargs["timeout"], 30.0)
def test_env_var_override(self):
"""LAST30DAYS_TRANSCRIPT_TIMEOUT='60' -> request uses 60s."""
from lib import http as http_module
os.environ["LAST30DAYS_TRANSCRIPT_TIMEOUT"] = "60"
items = [self._video_item()]
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.return_value = self._ok_payload()
instagram.fetch_captions(items, token="fake-token")
kwargs = mock_http_get.call_args.kwargs
self.assertEqual(kwargs["timeout"], 60.0)
def test_explicit_timeout_kwarg_wins_over_env(self):
"""Explicit timeout= kwarg trumps the env var."""
from lib import http as http_module
os.environ["LAST30DAYS_TRANSCRIPT_TIMEOUT"] = "60"
items = [self._video_item()]
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.return_value = self._ok_payload()
instagram.fetch_captions(items, token="fake-token", timeout=10)
kwargs = mock_http_get.call_args.kwargs
self.assertEqual(kwargs["timeout"], 10.0)
def test_config_dict_fallback_when_env_unset(self):
"""config={'LAST30DAYS_TRANSCRIPT_TIMEOUT': '45'} -> request uses 45s."""
from lib import http as http_module
items = [self._video_item()]
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.return_value = self._ok_payload()
instagram.fetch_captions(
items,
token="fake-token",
config={"LAST30DAYS_TRANSCRIPT_TIMEOUT": "45"},
)
kwargs = mock_http_get.call_args.kwargs
self.assertEqual(kwargs["timeout"], 45.0)
def test_invalid_env_value_falls_back_to_default(self):
"""Garbage env var doesn't crash; falls back to 30s."""
from lib import http as http_module
os.environ["LAST30DAYS_TRANSCRIPT_TIMEOUT"] = "not-a-number"
items = [self._video_item()]
with patch.object(http_module, "get") as mock_http_get:
mock_http_get.return_value = self._ok_payload()
instagram.fetch_captions(items, token="fake-token")
kwargs = mock_http_get.call_args.kwargs
self.assertEqual(kwargs["timeout"], 30.0)
if __name__ == "__main__":
unittest.main()
+84
View File
@@ -0,0 +1,84 @@
import json
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[1]
LAST30DAYS_SCRIPT = REPO_ROOT / "skills" / "last30days" / "scripts" / "last30days.py"
def run_last30days(topic: str, env: dict[str, str]) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[sys.executable, str(LAST30DAYS_SCRIPT), topic, "--mock", "--emit=json"],
cwd=REPO_ROOT,
env=env,
capture_output=True,
text=True,
check=False,
)
class LastRunStateTests(unittest.TestCase):
def test_empty_config_override_disables_last_run_write(self):
with tempfile.TemporaryDirectory() as tmp:
home = Path(tmp) / "home"
env = os.environ.copy()
env["HOME"] = str(home)
env["LAST30DAYS_CONFIG_DIR"] = ""
result = run_last30days("synthetic eval query", env)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertFalse((home / ".config" / "last30days" / "last-run.json").exists())
def test_custom_config_override_writes_last_run_to_custom_dir(self):
with tempfile.TemporaryDirectory() as tmp:
config_dir = Path(tmp) / "custom-config"
env = os.environ.copy()
env["HOME"] = str(Path(tmp) / "home")
env["LAST30DAYS_CONFIG_DIR"] = str(config_dir)
result = run_last30days("custom config query", env)
self.assertEqual(result.returncode, 0, result.stderr)
payload = json.loads((config_dir / "last-run.json").read_text())
self.assertEqual(payload["topic"], "custom config query")
self.assertGreaterEqual(payload["total"], 0)
def test_hook_reads_last_run_from_custom_config_dir(self):
with tempfile.TemporaryDirectory() as tmp:
config_dir = Path(tmp) / "custom-config"
config_dir.mkdir()
(config_dir / "last-run.json").write_text(
json.dumps(
{
"topic": "custom hook query",
"timestamp": "2026-04-30T00:00:00+00:00",
"sources": {"reddit": 2},
"total": 2,
}
)
)
env = os.environ.copy()
env["HOME"] = str(Path(tmp) / "home")
env["LAST30DAYS_CONFIG_DIR"] = str(config_dir)
result = subprocess.run(
["bash", "hooks/scripts/check-config.sh"],
cwd=REPO_ROOT,
env=env,
capture_output=True,
text=True,
check=False,
)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn('Last run: "custom hook query"', result.stdout)
if __name__ == "__main__":
unittest.main()
+106
View File
@@ -52,6 +52,36 @@ class PipelineV3Tests(unittest.TestCase):
# At least one per-subquery line.
self.assertIn("[Planner] sq1 label=", output)
def test_parallel_web_backend_enables_grounding_source(self):
plan = {
"intent": "news",
"freshness_mode": "balanced_recent",
"cluster_mode": "timeline",
"subqueries": [
{
"label": "primary",
"search_query": "test topic",
"ranking_query": "What happened with test topic?",
"sources": ["grounding"],
}
],
"source_weights": {"grounding": 1.0},
}
report = pipeline.run(
topic="test topic",
config={"LAST30DAYS_REASONING_PROVIDER": "auto"},
depth="quick",
requested_sources=["grounding"],
web_backend="parallel",
external_plan=plan,
)
# Anchor on the stable source key, not the exact wording of the
# grounding.py error message. Phrasing can shift (e.g., when the
# missing-key check moves or the message is reworded) without
# changing the contract that the grounding source registers an
# error when its required backend key is unset.
self.assertIn("grounding", report.errors_by_source)
class TestSourceFetchCap(unittest.TestCase):
"""X source fetch count must be capped by MAX_SOURCE_FETCHES."""
@@ -904,5 +934,81 @@ class TestZeroKeyPipelineRun(unittest.TestCase):
self.assertEqual("fallback-local-score", candidate.explanation)
class TestExcludeSources(unittest.TestCase):
"""EXCLUDE_SOURCES env var filters sources out of available_sources().
The existing INCLUDE_SOURCES allowlist (used by Perplexity opt-in) does
not cover this case tiktok and instagram are added unconditionally
when SCRAPECREATORS_API_KEY is set, with no way to opt out short of
unsetting the key. EXCLUDE_SOURCES gives runs a per-invocation denylist.
"""
def test_excludes_tiktok_and_instagram(self):
config = {
"SCRAPECREATORS_API_KEY": "test-key",
"EXCLUDE_SOURCES": "tiktok,instagram",
}
sources = pipeline.available_sources(config)
self.assertNotIn("tiktok", sources)
self.assertNotIn("instagram", sources)
self.assertIn("reddit", sources)
self.assertIn("hackernews", sources)
def test_no_exclusion_when_unset(self):
config = {"SCRAPECREATORS_API_KEY": "test-key"}
sources = pipeline.available_sources(config)
self.assertIn("tiktok", sources)
self.assertIn("instagram", sources)
def test_empty_exclude_sources_is_noop(self):
config = {
"SCRAPECREATORS_API_KEY": "test-key",
"EXCLUDE_SOURCES": "",
}
sources = pipeline.available_sources(config)
self.assertIn("tiktok", sources)
self.assertIn("instagram", sources)
def test_whitespace_and_case_insensitive(self):
config = {
"SCRAPECREATORS_API_KEY": "test-key",
"EXCLUDE_SOURCES": " TikTok , INSTAGRAM ",
}
sources = pipeline.available_sources(config)
self.assertNotIn("tiktok", sources)
self.assertNotIn("instagram", sources)
def test_excludes_non_scrapecreators_source(self):
"""EXCLUDE_SOURCES applies to any source, not just SC-backed ones."""
config = {"EXCLUDE_SOURCES": "hackernews"}
sources = pipeline.available_sources(config)
self.assertNotIn("hackernews", sources)
self.assertIn("reddit", sources)
class TestExcludeSourcesEndToEnd(unittest.TestCase):
"""Wiring regression: EXCLUDE_SOURCES from the process environment must
reach available_sources() via env.get_config(). The unit tests above
construct config dicts directly; this one exercises the env-to-config
path so a missing entry in env.py's keys list is caught immediately."""
def test_exclude_sources_from_env_propagates_through_get_config(self):
import os
from unittest.mock import patch as _patch
from lib import env as env_mod
from importlib import reload
with _patch.dict(os.environ, {
"LAST30DAYS_CONFIG_DIR": "",
"EXCLUDE_SOURCES": "tiktok,instagram",
"SCRAPECREATORS_API_KEY": "fake",
}, clear=False):
reload(env_mod)
cfg = env_mod.get_config()
self.assertEqual(cfg.get("EXCLUDE_SOURCES"), "tiktok,instagram")
sources = pipeline.available_sources(cfg)
self.assertNotIn("tiktok", sources)
self.assertNotIn("instagram", sources)
if __name__ == "__main__":
unittest.main()
+8 -5
View File
@@ -1,5 +1,5 @@
import json
import re
import sys
import tomllib
import unittest
from pathlib import Path
@@ -8,17 +8,19 @@ from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SKILL_ROOT = ROOT / "skills" / "last30days"
sys.path.insert(0, str(SKILL_ROOT / "scripts"))
from lib.skill_meta import read_skill_version # noqa: E402
def _json(path: Path) -> dict:
return json.loads(path.read_text(encoding="utf-8"))
def _skill_version() -> str:
text = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
match = re.search(r'^version:\s*"([^"]+)"\s*$', text, re.MULTILINE)
if not match:
version = read_skill_version(SKILL_ROOT / "SKILL.md")
if not version:
raise AssertionError("SKILL.md version frontmatter not found")
return match.group(1)
return version
class TestPluginContract(unittest.TestCase):
@@ -34,6 +36,7 @@ class TestPluginContract(unittest.TestCase):
self.assertEqual(version, _skill_version())
self.assertEqual(version, _json(ROOT / ".claude-plugin" / "plugin.json")["version"])
self.assertEqual(version, _json(ROOT / "gemini-extension.json")["version"])
marketplace = _json(ROOT / ".claude-plugin" / "marketplace.json")
plugins = marketplace.get("plugins") or []
+380 -2
View File
@@ -5,6 +5,11 @@ HN, Polymarket, Reddit (always active), X, YouTube.
ScrapeCreators adds TikTok + Instagram as bonus sources, not core.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))
import pytest
from unittest.mock import patch
@@ -38,8 +43,8 @@ def _base_results(**overrides):
def _compute(config_overrides=None, result_overrides=None, ytdlp_installed=False):
"""Helper to call compute_quality_score with mocked yt-dlp check."""
from scripts.lib.quality_nudge import compute_quality_score
from scripts.lib import youtube_yt
from lib.quality_nudge import compute_quality_score
from lib import youtube_yt
config = _base_config(**(config_overrides or {}))
results = _base_results(**(result_overrides or {}))
@@ -199,3 +204,376 @@ class TestRedditNeverInCoreErrored:
# Reddit is always-active in core (public path), error doesn't demote it
assert "reddit" in q["core_active"]
assert q["score_pct"] == 100
class TestYouTubeDegraded:
"""YouTube is `degraded` when videos returned but transcripts below threshold.
Canonical failure mode: a stale yt-dlp binary still finds videos via search
but silently fails every transcript fetch because YouTube's caption format
has moved on. Pre-fix the user got no signal of this; the footer hid zero,
and quality_nudge only checked top-level errors.
"""
def test_zero_of_six_transcripts_flags_degraded(self):
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 6,
"youtube_transcripts_count": 0,
},
)
assert "youtube" in q["core_degraded"]
assert q["nudge_text"] is not None
# Counts surface in the message so the user sees the actual ratio
assert "6 videos" in q["nudge_text"]
assert "0 transcripts" in q["nudge_text"]
assert "stale yt-dlp" in q["nudge_text"].lower()
# Updates path mentions all three common package managers
assert "scoop" in q["nudge_text"].lower()
assert "brew" in q["nudge_text"].lower()
assert "pip install" in q["nudge_text"].lower()
def test_five_of_six_transcripts_does_not_flag_degraded(self):
# 83% transcript success - well above the 50% threshold
# X is also enabled so all 5 cores are active and no nudge should fire
q = _compute(
config_overrides={"AUTH_TOKEN": "tok123"},
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 6,
"youtube_transcripts_count": 5,
},
)
assert "youtube" not in q["core_degraded"]
assert q["nudge_text"] is None # All 5 core sources active, no degradation
def test_zero_videos_does_not_flag_degraded(self):
# No videos returned -> degraded check is meaningless and must not fire
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 0,
"youtube_transcripts_count": 0,
},
)
assert "youtube" not in q["core_degraded"]
def test_one_of_three_transcripts_flags_degraded(self):
# 33% - below 50% threshold; the canonical "yt-dlp partially working" case
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 3,
"youtube_transcripts_count": 1,
},
)
assert "youtube" in q["core_degraded"]
assert "Degraded: YouTube" in q["nudge_text"]
def test_threshold_tunable_via_config(self):
# Operator overrides threshold via env-style config to be more permissive
q = _compute(
config_overrides={"DEGRADED_TRANSCRIPT_THRESHOLD": "0.1"},
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 10,
"youtube_transcripts_count": 2, # 20%, below default 50% but above override 10%
},
)
assert "youtube" not in q["core_degraded"]
def test_degraded_does_not_affect_score(self):
# Degradation is informational, not score-affecting; YouTube still counts as active
q = _compute(
config_overrides={"AUTH_TOKEN": "tok123"},
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 6,
"youtube_transcripts_count": 0,
},
)
assert "youtube" in q["core_active"]
assert q["score_pct"] == 100 # Full active count regardless of degradation
# But nudge still fires
assert q["nudge_text"] is not None
assert "Degraded: YouTube" in q["nudge_text"]
class TestYouTubeCaptionsDisabledDoesNotFalseFlag:
"""Captions-disabled videos must not lower the transcript-fetch ratio.
A video where the uploader disabled captions can never produce a transcript,
no matter how fresh yt-dlp is. Counting it in the denominator of the
degraded-ratio check produces false positives - one captions-disabled video
in a small result set was triggering a "stale yt-dlp binary" nudge that was
wrong. Fix: subtract captions_disabled from the denominator.
"""
def test_zero_captions_disabled_preserves_existing_behavior(self):
# Pre-existing case: 0 of 6 transcripts is still degraded (no captions
# disabled to discount). Behavior is unchanged from TestYouTubeDegraded.
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 6,
"youtube_transcripts_count": 0,
"youtube_captions_disabled_count": 0,
},
)
assert "youtube" in q["core_degraded"]
def test_all_videos_captions_disabled_does_not_flag(self):
# Every returned video had captions disabled by the uploader.
# That's not a yt-dlp problem - it's an upstream content fact. Must not
# flag degraded.
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 3,
"youtube_transcripts_count": 0,
"youtube_captions_disabled_count": 3,
},
)
assert "youtube" not in q["core_degraded"]
def test_mixed_uses_corrected_denominator(self):
# 6 videos, 3 captions_disabled, 2 transcripts.
# Naive (buggy) ratio: 2/6 = 33% (would flag).
# Corrected ratio: 2/(6-3) = 67% (does NOT flag).
# This case demonstrates the fix changes the verdict.
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 6,
"youtube_transcripts_count": 2,
"youtube_captions_disabled_count": 3,
},
)
assert "youtube" not in q["core_degraded"]
def test_mixed_still_flags_when_truly_degraded(self):
# Even after discounting captions-disabled, the ratio is still bad.
# 8 videos, 1 captions_disabled, 1 transcript -> 1/(8-1) = 14% (flags).
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 8,
"youtube_transcripts_count": 1,
"youtube_captions_disabled_count": 1,
},
)
assert "youtube" in q["core_degraded"]
# Nudge should still mention the stale yt-dlp possibility but also
# acknowledge that captions-disabled is a separate cause.
assert q["nudge_text"] is not None
assert "captions disabled" in q["nudge_text"].lower()
def test_missing_count_defaults_to_zero(self):
# Older callers that don't pass the new key still work (default 0).
q = _compute(
ytdlp_installed=True,
result_overrides={
"youtube_videos_count": 6,
"youtube_transcripts_count": 0,
# youtube_captions_disabled_count intentionally omitted
},
)
assert "youtube" in q["core_degraded"]
class TestInstagramSilentFailure:
"""Instagram is a `bonus` source via SC. Silent-failure detection: if SC
is configured but the source returned zero items, surface a nudge so the
user understands why the brief lacks an Instagram section.
Pre-fix the user got no signal - SC's /v2/instagram/reels/search 500s
frequently on multi-token queries and the pipeline silently returned
empty without any indication.
"""
def test_zero_items_with_sc_flags_bonus_errored(self):
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" in q["bonus_errored"]
assert q["nudge_text"] is not None
assert "Instagram" in q["nudge_text"]
def test_zero_items_without_sc_does_not_flag(self):
q = _compute(
config_overrides={"AUTH_TOKEN": "tok123"},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" not in q.get("bonus_errored", [])
def test_nonzero_items_does_not_flag(self):
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 5},
)
assert "instagram" not in q["bonus_errored"]
assert q["nudge_text"] is None
def test_missing_key_means_source_did_not_run(self):
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
},
ytdlp_installed=True,
)
assert "instagram" not in q["bonus_errored"]
assert q["nudge_text"] is None
def test_nudge_text_explains_workaround(self):
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert q["nudge_text"] is not None
text_lower = q["nudge_text"].lower()
assert "instagram" in text_lower
assert ("0 reels" in text_lower or "silent" in text_lower
or "hashtag" in text_lower)
def test_bonus_errored_does_not_affect_core_score(self):
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert q["score_pct"] == 100
assert "instagram" in q["bonus_errored"]
assert q["nudge_text"] is not None
assert "Bonus source silent" in q["nudge_text"]
def test_bonus_errored_field_always_present(self):
q = _compute()
assert q.get("bonus_errored") == []
def test_exclude_sources_instagram_suppresses_silent_failure(self):
"""User set EXCLUDE_SOURCES=instagram - the source intentionally did
not run, so the zero-count instagram_items_count written by
last30days.py is a non-event, not a silent failure. Pre-fix: the
nudge fired anyway because the gate only checked SC-key + count.
"""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"EXCLUDE_SOURCES": "instagram",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" not in q["bonus_errored"]
assert q["nudge_text"] is None
def test_exclude_sources_multi_value_with_instagram(self):
"""Canonical parsing pattern is comma-separated; case-insensitive."""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"EXCLUDE_SOURCES": "threads, Instagram , pinterest",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" not in q["bonus_errored"]
def test_exclude_sources_other_value_still_flags(self):
"""EXCLUDE_SOURCES that does not mention instagram must not suppress
the silent-failure nudge for instagram.
"""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"EXCLUDE_SOURCES": "threads",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" in q["bonus_errored"]
def test_include_sources_without_instagram_suppresses_silent_failure(self):
"""User set INCLUDE_SOURCES to an opt-in allowlist that omits
instagram the pipeline skips the source by allowlist filter, so
the zero-count instagram_items_count is intentional, not a silent
failure. Symmetric to the EXCLUDE_SOURCES=instagram guard.
"""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"INCLUDE_SOURCES": "reddit,hn,x,youtube",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" not in q["bonus_errored"]
assert q["nudge_text"] is None
def test_include_sources_multi_value_without_instagram(self):
"""Canonical parsing pattern is comma-separated; case-insensitive."""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"INCLUDE_SOURCES": " Reddit, HN , YouTube ",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" not in q["bonus_errored"]
def test_include_sources_with_instagram_still_flags(self):
"""INCLUDE_SOURCES that explicitly names instagram must not suppress
the silent-failure nudge the source was opted in, so a zero count
is a real silent failure.
"""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"INCLUDE_SOURCES": "reddit,instagram",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" in q["bonus_errored"]
def test_include_sources_empty_does_not_suppress(self):
"""Empty/unset INCLUDE_SOURCES means no allowlist filter, so the
silent-failure gate should still fire when instagram is zero.
"""
q = _compute(
config_overrides={
"AUTH_TOKEN": "tok123",
"SCRAPECREATORS_API_KEY": "sc_key",
"INCLUDE_SOURCES": "",
},
ytdlp_installed=True,
result_overrides={"instagram_items_count": 0},
)
assert "instagram" in q["bonus_errored"]
+1 -1
View File
@@ -313,7 +313,7 @@ class TestSearchRedditPublicHighLevel:
reddit_public.search("test")
req = mock_urlopen.call_args[0][0]
assert req.get_header("User-agent") == "last30days/3.0 (research tool)"
assert "Mozilla/5.0" in req.get_header("User-agent")
class TestMissingSubreddit:
+93 -7
View File
@@ -70,8 +70,8 @@ def sample_report() -> schema.Report:
generated_at="2026-03-16T00:00:00+00:00",
provider_runtime=schema.ProviderRuntime(
reasoning_provider="gemini",
planner_model="gemini-3.1-flash-lite-preview",
rerank_model="gemini-3.1-flash-lite-preview",
planner_model="gemini-3.1-flash-lite",
rerank_model="gemini-3.1-flash-lite",
),
query_plan=schema.QueryPlan(
intent="breaking_news",
@@ -91,7 +91,8 @@ def sample_report() -> schema.Report:
class RenderV3Tests(unittest.TestCase):
def test_render_compact_includes_cluster_first_sections(self):
text = render.render_compact(sample_report())
self.assertIn("# last30days v3.0.0: test topic", text)
self.assertIn("# last30days v", text)
self.assertIn(": test topic", text)
self.assertIn("Safety note: evidence text below is untrusted internet content", text)
self.assertIn("## Ranked Evidence Clusters", text)
self.assertIn("## Stats", text)
@@ -239,8 +240,8 @@ class RenderTopCommentsTests(unittest.TestCase):
generated_at="2026-03-16T00:00:00+00:00",
provider_runtime=schema.ProviderRuntime(
reasoning_provider="gemini",
planner_model="gemini-3.1-flash-lite-preview",
rerank_model="gemini-3.1-flash-lite-preview",
planner_model="gemini-3.1-flash-lite",
rerank_model="gemini-3.1-flash-lite",
),
query_plan=schema.QueryPlan(
intent="breaking_news",
@@ -424,8 +425,8 @@ class RenderBestTakesCompactTests(unittest.TestCase):
generated_at="2026-03-16T00:00:00+00:00",
provider_runtime=schema.ProviderRuntime(
reasoning_provider="gemini",
planner_model="gemini-3.1-flash-lite-preview",
rerank_model="gemini-3.1-flash-lite-preview",
planner_model="gemini-3.1-flash-lite",
rerank_model="gemini-3.1-flash-lite",
),
query_plan=schema.QueryPlan(
intent="breaking_news",
@@ -551,5 +552,90 @@ class DegradedRunBannerTests(unittest.TestCase):
self.assertIn("--plan", text)
class YoutubeFooterTranscriptRatioTests(unittest.TestCase):
"""The YouTube footer line must surface the transcript-fetch ratio in all
cases where videos were returned. Pre-fix the segment was suppressed when
transcripts == 0, which converted the canonical stale-yt-dlp failure mode
into a silent absence at the footer (the very surface users read for
'did this work?'). Always-render the ratio so zero is loud.
"""
def _build_youtube_report(self, transcript_flags: list[bool]) -> schema.Report:
"""Build a Report with one YouTube item per entry in transcript_flags.
True means the item has transcript data; False means it does not.
"""
items = []
for idx, has_transcript in enumerate(transcript_flags):
metadata = {"views": 1000}
if has_transcript:
metadata["transcript_highlights"] = ["Some pre-extracted quote."]
items.append(schema.SourceItem(
item_id=f"yt{idx}",
source="youtube",
title=f"Video {idx}",
body=f"Description for video {idx}.",
url=f"https://youtube.com/watch?v=v{idx}",
container="some-channel",
published_at="2026-04-15",
date_confidence="high",
engagement={"views": 1000, "likes": 100},
metadata=metadata,
))
return schema.Report(
topic="test topic",
range_from="2026-04-01",
range_to="2026-05-01",
generated_at="2026-05-01T00:00:00+00:00",
provider_runtime=schema.ProviderRuntime(
reasoning_provider="gemini",
planner_model="gemini",
rerank_model="gemini",
),
query_plan=schema.QueryPlan(
intent="general",
freshness_mode="balanced_recent",
cluster_mode="none",
raw_topic="test topic",
subqueries=[schema.SubQuery(
label="primary", search_query="test topic",
ranking_query="What about test topic?", sources=["youtube"],
)],
source_weights={"youtube": 1.0},
),
clusters=[],
ranked_candidates=[],
items_by_source={"youtube": items},
errors_by_source={},
)
def test_zero_transcripts_with_videos_present_renders_zero_over_total(self):
# The canonical stale-yt-dlp case: 6 videos found, 0 transcripts captured.
# Pre-fix the footer hid this entirely; post-fix it must say "0/6 with transcripts".
report = self._build_youtube_report([False] * 6)
text = render.render_compact(report)
self.assertIn("0/6 with transcripts", text)
def test_partial_transcripts_renders_ratio(self):
# 5 of 6 transcripts captured - shows ratio so user knows one was missed.
report = self._build_youtube_report([True] * 5 + [False])
text = render.render_compact(report)
self.assertIn("5/6 with transcripts", text)
def test_full_transcripts_renders_ratio(self):
# All 3 transcripts captured - still shows ratio for consistency.
report = self._build_youtube_report([True] * 3)
text = render.render_compact(report)
self.assertIn("3/3 with transcripts", text)
def test_no_videos_no_transcript_segment(self):
# When YouTube has no items at all, the YouTube footer line is
# suppressed entirely (existing behavior) - the transcript segment
# should not appear without a parent line.
report = self._build_youtube_report([])
text = render.render_compact(report)
# No YouTube footer line at all - so no transcript segment either
self.assertNotIn("with transcripts", text)
if __name__ == "__main__":
unittest.main()
+2 -2
View File
@@ -172,10 +172,10 @@ class RerankV3Tests(unittest.TestCase):
plan=make_plan(),
candidates=[first, second],
provider=provider,
model="gemini-3.1-flash-lite-preview",
model="gemini-3.1-flash-lite",
shortlist_size=1,
)
self.assertEqual("gemini-3.1-flash-lite-preview", provider.model)
self.assertEqual("gemini-3.1-flash-lite", provider.model)
self.assertEqual(95.0, first.rerank_score)
self.assertEqual("high fit", first.explanation)
# Tail is scored via the fallback (may or may not carry the entity-miss
+18
View File
@@ -109,6 +109,24 @@ class TestBuildContextSummary(unittest.TestCase):
self.assertEqual(resolve._build_context_summary(items), "")
class TestCanonicalizeGithubRepos(unittest.TestCase):
def test_rewrites_integration_repo_to_canonical_product(self):
repos = ["openai/codex", "anthropics/claude-code-action"]
result = resolve.canonicalize_github_repos("claude code vs codex", repos, cap=None)
self.assertEqual(result, ["openai/codex", "anthropics/claude-code"])
def test_preserves_action_repo_when_topic_intends_action(self):
repos = ["anthropics/claude-code-action", "openai/codex"]
result = resolve.canonicalize_github_repos("claude code action setup", repos, cap=None)
self.assertIn("anthropics/claude-code-action", result)
self.assertNotIn("anthropics/claude-code", result)
def test_dedupes_case_insensitive_after_canonicalization(self):
repos = ["Anthropics/Claude-Code-Action", "anthropics/claude-code"]
result = resolve.canonicalize_github_repos("claude code", repos, cap=None)
self.assertEqual(result, ["Anthropics/Claude-Code"])
class TestAutoResolve(unittest.TestCase):
def test_no_backend_returns_empty(self):
result = resolve.auto_resolve("test topic", {})
+84 -14
View File
@@ -160,25 +160,95 @@ class TestErrorPaths:
result = extract_safari_cookies_macos("x.com", ["auth_token"])
assert result is None
def test_prefers_sandboxed_safari_cookie_path(
self, tmp_path: Path, x_cookies_file: bytes
):
sandbox_dir = (
tmp_path
/ "Library"
/ "Containers"
/ "com.apple.Safari"
/ "Data"
/ "Library"
/ "Cookies"
)
sandbox_dir.mkdir(parents=True)
(sandbox_dir / "Cookies.binarycookies").write_bytes(x_cookies_file)
legacy_dir = tmp_path / "Library" / "Cookies"
legacy_dir.mkdir(parents=True)
legacy_data = _build_binary_cookies_file(
[_build_page([_build_cookie_record(".x.com", "auth_token", "legacy")])]
)
(legacy_dir / "Cookies.binarycookies").write_bytes(legacy_data)
with patch(
"scripts.lib.safari_cookies.Path.home", return_value=tmp_path
), patch("scripts.lib.safari_cookies.sys") as mock_sys:
mock_sys.platform = "darwin"
mock_sys.stderr = sys.stderr
result = extract_safari_cookies_macos("x.com", ["auth_token", "ct0"])
assert result is not None
assert result["auth_token"] == "test_auth_abc123"
assert result["ct0"] == "test_ct0_xyz789"
def test_falls_back_to_legacy_safari_cookie_path(self, tmp_path: Path):
# Sandboxed path is intentionally NOT created — only the legacy path exists.
legacy_dir = tmp_path / "Library" / "Cookies"
legacy_dir.mkdir(parents=True)
legacy_data = _build_binary_cookies_file(
[_build_page([_build_cookie_record(".x.com", "auth_token", "legacy_auth")])]
)
(legacy_dir / "Cookies.binarycookies").write_bytes(legacy_data)
sandbox_path = (
tmp_path
/ "Library"
/ "Containers"
/ "com.apple.Safari"
/ "Data"
/ "Library"
/ "Cookies"
/ "Cookies.binarycookies"
)
assert not sandbox_path.exists()
with patch(
"scripts.lib.safari_cookies.Path.home", return_value=tmp_path
), patch("scripts.lib.safari_cookies.sys") as mock_sys:
mock_sys.platform = "darwin"
mock_sys.stderr = sys.stderr
result = extract_safari_cookies_macos("x.com", ["auth_token"])
assert result is not None
assert result["auth_token"] == "legacy_auth"
def test_permission_denied(self, tmp_path: Path, capsys):
cookie_dir = tmp_path / "Library" / "Cookies"
cookie_dir = (
tmp_path
/ "Library"
/ "Containers"
/ "com.apple.Safari"
/ "Data"
/ "Library"
/ "Cookies"
)
cookie_dir.mkdir(parents=True)
cookie_file = cookie_dir / "Cookies.binarycookies"
cookie_file.write_bytes(b"cook")
cookie_file.chmod(0o000)
try:
with patch(
"scripts.lib.safari_cookies.Path.home", return_value=tmp_path
), patch("scripts.lib.safari_cookies.sys") as mock_sys:
mock_sys.platform = "darwin"
mock_sys.stderr = sys.stderr
result = extract_safari_cookies_macos("x.com", ["auth_token"])
assert result is None
captured = capsys.readouterr()
assert "Full Disk Access" in captured.err
finally:
cookie_file.chmod(0o644)
with patch(
"scripts.lib.safari_cookies.Path.home", return_value=tmp_path
), patch("scripts.lib.safari_cookies.sys") as mock_sys, patch.object(
Path, "read_bytes", side_effect=PermissionError
):
mock_sys.platform = "darwin"
mock_sys.stderr = sys.stderr
result = extract_safari_cookies_macos("x.com", ["auth_token"])
assert result is None
captured = capsys.readouterr()
assert "Full Disk Access" in captured.err
def test_truncated_magic_only(self):
result = _parse_binary_cookies(b"cook", "x.com", ["auth_token"])
+2 -2
View File
@@ -16,8 +16,8 @@ class SchemaV3Tests(unittest.TestCase):
generated_at="2026-03-16T00:00:00+00:00",
provider_runtime=schema.ProviderRuntime(
reasoning_provider="gemini",
planner_model="gemini-3.1-flash-lite-preview",
rerank_model="gemini-3.1-flash-lite-preview",
planner_model="gemini-3.1-flash-lite",
rerank_model="gemini-3.1-flash-lite",
),
query_plan=schema.QueryPlan(
intent="breaking_news",
+53
View File
@@ -0,0 +1,53 @@
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
WORKFLOW = ROOT / ".github" / "workflows" / "security.yml"
# AGENTS.md is the canonical agent-guidance file; CLAUDE.md is a one-line
# pointer (`@AGENTS.md`) so anything Claude Code-shaped reads the same source.
AGENTS = ROOT / "AGENTS.md"
def _workflow_text() -> str:
return WORKFLOW.read_text(encoding="utf-8")
def test_security_workflow_exists() -> None:
assert WORKFLOW.is_file()
def test_security_workflow_runs_dependency_audit_advisory_first() -> None:
text = _workflow_text()
assert "dependency-audit:" in text
assert "pip-audit" in text
assert "continue-on-error: true" in text
assert "Set continue-on-error: false once a clean baseline run is confirmed" in text
def test_security_workflow_runs_secret_scan_for_pull_requests_and_main_pushes() -> None:
text = _workflow_text()
assert "secret-scan:" in text
assert "trufflesecurity/trufflehog" in text
assert "github.event_name == 'pull_request'" in text
assert "github.event_name == 'push'" in text
assert "--only-verified" in text
def test_security_workflow_documents_advisory_policy() -> None:
text = _workflow_text()
assert "advisory-first" in text.lower()
assert "does not block merges" in text.lower()
assert "fixtures" in text.lower()
assert "env-based auth" in text.lower()
def test_agent_guidance_mentions_secret_hygiene() -> None:
text = AGENTS.read_text(encoding="utf-8")
assert "Security hygiene" in text
assert "Never commit real API keys" in text
assert "skills/last30days/scripts/lib/env.py" in text
assert "fixtures" in text
+20 -3
View File
@@ -64,6 +64,19 @@ class TestRunOpenclawSetup:
assert result["keys"]["brave"] is True
assert result["keys"]["scrapecreators"] is False
def test_openclaw_metadata_keeps_scrapecreators_optional(self):
"""OpenClaw metadata should not hard-require the ScrapeCreators key."""
skill_md = Path(__file__).parent.parent / "skills" / "last30days" / "SKILL.md"
text = skill_md.read_text()
assert "SCRAPECREATORS_API_KEY" in text
expected = (
"requires:\n"
" env: []\n"
" optionalEnv:\n"
" - SCRAPECREATORS_API_KEY"
)
assert expected in text
@patch("shutil.which")
def test_x_method_xai(self, mock_which):
"""x_method is 'xai' when XAI_API_KEY is set."""
@@ -200,7 +213,9 @@ class TestPollDeviceAuth:
@patch("lib.setup_wizard.urlopen")
def test_timeout_returns_none(self, mock_urlopen, mock_time):
"""Returns None when timeout is exceeded."""
# Simulate time passing beyond deadline
# poll_device_auth captures started_at once, derives deadline + last_reminder
# from it, then checks time.time() in the while-loop. Two values: started_at,
# then a value past the deadline so the loop exits immediately.
mock_time.time = MagicMock(side_effect=[0, 301])
mock_time.sleep = MagicMock()
@@ -211,7 +226,9 @@ class TestPollDeviceAuth:
@patch("lib.setup_wizard.urlopen")
def test_expired_token_returns_none(self, mock_urlopen, mock_time):
"""Returns None on expired_token error."""
mock_time.time = MagicMock(side_effect=[0, 0])
# Loop terminates via urlopen response, not the clock — pin time to 0
# so the deadline check stays a non-event regardless of call count.
mock_time.time = MagicMock(return_value=0)
mock_time.sleep = MagicMock()
expired_resp = MagicMock()
@@ -230,7 +247,7 @@ class TestPollDeviceAuth:
"""HTTP 400 during polling continues (authorization pending)."""
from urllib.error import HTTPError
mock_time.time = MagicMock(side_effect=[0, 0, 0])
mock_time.time = MagicMock(return_value=0)
mock_time.sleep = MagicMock()
success_resp = MagicMock()
+61
View File
@@ -0,0 +1,61 @@
"""Direct unit tests for skill_meta.read_skill_version.
Covers the helper's own contract independent of render._skill_version which
exercises it transitively. Without these, regressions in error handling or
regex coverage inside the helper could pass CI because render.py's fallback
to "?" swallows the signal.
"""
import sys
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "skills" / "last30days" / "scripts"))
from lib.skill_meta import read_skill_version # noqa: E402
class ReadSkillVersionTests(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.tmp_path = Path(self._tmp.name)
def tearDown(self) -> None:
self._tmp.cleanup()
def _write_skill_md(self, body: str) -> Path:
path = self.tmp_path / "SKILL.md"
path.write_text(body)
return path
def test_double_quoted_version(self) -> None:
path = self._write_skill_md('---\nname: x\nversion: "9.9.9"\n---\n')
self.assertEqual("9.9.9", read_skill_version(path))
def test_single_quoted_version(self) -> None:
path = self._write_skill_md("---\nname: x\nversion: '8.8.8'\n---\n")
self.assertEqual("8.8.8", read_skill_version(path))
def test_unquoted_version(self) -> None:
path = self._write_skill_md("---\nname: x\nversion: 7.7.7\n---\n")
self.assertEqual("7.7.7", read_skill_version(path))
def test_missing_file_returns_none(self) -> None:
self.assertIsNone(read_skill_version(self.tmp_path / "does-not-exist.md"))
def test_no_version_line_returns_none(self) -> None:
path = self._write_skill_md("---\nname: x\n---\n# body without version\n")
self.assertIsNone(read_skill_version(path))
def test_undecodable_bytes_returns_none(self) -> None:
# Bytes 128-255 don't form valid UTF-8 sequences; read_text() raises
# UnicodeDecodeError which the helper must catch.
path = self.tmp_path / "SKILL.md"
path.write_bytes(bytes(range(128, 256)))
self.assertIsNone(read_skill_version(path))
if __name__ == "__main__":
unittest.main()
+194 -12
View File
@@ -3,7 +3,7 @@
import json
import sqlite3
import tempfile
from datetime import datetime, timedelta
from datetime import datetime, timedelta, timezone
from pathlib import Path
import pytest
@@ -59,7 +59,68 @@ def sample_report():
"source_weights": {},
},
"clusters": [],
"ranked_candidates": [],
"ranked_candidates": [
{
"candidate_id": "c-r1",
"item_id": "R1",
"source": "reddit",
"title": "Test Reddit Post",
"url": "https://reddit.com/r/test/1",
"snippet": "Reddit snippet",
"subquery_labels": ["primary"],
"native_ranks": {"reddit": 1},
"local_relevance": 0.8,
"freshness": 100,
"engagement": 50.0,
"source_quality": 0.8,
"rrf_score": 1.0,
"final_score": 0.8,
"explanation": "Reddit snippet",
"source_items": [
{
"item_id": "R1",
"source": "reddit",
"title": "Test Reddit Post",
"body": "Reddit discussion content",
"url": "https://reddit.com/r/test/1",
"author": "testuser",
"engagement_score": 50.0,
"local_relevance": 0.8,
"snippet": "Reddit snippet",
}
],
},
{
"candidate_id": "c-x1",
"item_id": "X1",
"source": "x",
"title": "Test X Post",
"url": "https://x.com/test/status/1",
"snippet": "X snippet",
"subquery_labels": ["primary"],
"native_ranks": {"x": 1},
"local_relevance": 0.85,
"freshness": 100,
"engagement": 75.0,
"source_quality": 0.8,
"rrf_score": 1.0,
"final_score": 0.85,
"explanation": "X snippet",
"source_items": [
{
"item_id": "X1",
"source": "x",
"title": "Test X Post",
"body": "X post content",
"url": "https://x.com/test/status/1",
"author": "xuser",
"engagement_score": 75.0,
"local_relevance": 0.85,
"snippet": "X snippet",
}
],
},
],
"items_by_source": {
"reddit": [
{
@@ -236,13 +297,13 @@ def test_findings_from_report_handles_missing_fields():
"clusters": [],
"ranked_candidates": [],
"items_by_source": {
"reddit": [
"hackernews": [
{
"item_id": "R1",
"source": "reddit",
"source": "hackernews",
"title": "Test",
"body": "Content",
"url": "https://reddit.com/1",
"url": "https://news.ycombinator.com/item?id=1",
"author": None, # Missing author
"engagement_score": None, # Missing engagement
"local_relevance": None, # Missing relevance
@@ -384,6 +445,127 @@ def test_store_findings_skips_items_without_url(temp_db):
assert counts["new"] == 1
def test_init_db_creates_finding_sightings_table(temp_db):
"""Test that the per-run sightings ledger is available on fresh databases."""
conn = sqlite3.connect(str(temp_db))
table = conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name='finding_sightings'"
).fetchone()
columns = {
row[1]: row[3]
for row in conn.execute("PRAGMA table_info(finding_sightings)").fetchall()
}
conn.close()
assert table is not None
assert columns["finding_id"] == 1
def test_store_findings_records_sightings_for_new_findings(temp_db):
"""Test that each stored finding is linked to the run that observed it."""
topic = store.add_topic("Test Topic")
run_id = store.record_run(topic["id"], source_mode="v3")
findings = [
{
"source": "reddit",
"source_url": "https://reddit.com/1",
"source_title": "Reddit 1",
"content": "Content 1",
"engagement_score": 10.0,
"relevance_score": 0.7,
},
{
"source": "x",
"source_url": "https://x.com/a/status/1",
"source_title": "X 1",
"content": "Content 2",
"engagement_score": 20.0,
"relevance_score": 0.8,
},
]
store.store_findings(run_id, topic["id"], findings)
sightings = store.get_sightings_for_run(topic["id"], run_id)
assert [s["source_url"] for s in sightings] == [
"https://reddit.com/1",
"https://x.com/a/status/1",
]
assert {s["source"] for s in sightings} == {"reddit", "x"}
def test_store_findings_records_sightings_for_resighted_findings(temp_db):
"""Test that a re-seen finding is recorded for each run that observes it."""
topic = store.add_topic("Test Topic")
first_run_id = store.record_run(topic["id"], source_mode="v3")
second_run_id = store.record_run(topic["id"], source_mode="v3")
finding = {
"source": "reddit",
"source_url": "https://reddit.com/1",
"source_title": "Reddit 1",
"content": "Content",
"engagement_score": 10.0,
"relevance_score": 0.7,
}
store.store_findings(first_run_id, topic["id"], [finding])
store.store_findings(second_run_id, topic["id"], [{**finding, "engagement_score": 15.0}])
first_sightings = store.get_sightings_for_run(topic["id"], first_run_id)
second_sightings = store.get_sightings_for_run(topic["id"], second_run_id)
assert len(first_sightings) == 1
assert len(second_sightings) == 1
assert first_sightings[0]["source_url"] == second_sightings[0]["source_url"]
assert second_sightings[0]["engagement_score"] == 15.0
def test_store_findings_sightings_are_idempotent_per_run(temp_db):
"""Test that storing the same finding twice for one run does not duplicate sightings."""
topic = store.add_topic("Test Topic")
run_id = store.record_run(topic["id"], source_mode="v3")
finding = {
"source": "reddit",
"source_url": "https://reddit.com/1",
"source_title": "Reddit 1",
"content": "Content",
"engagement_score": 10.0,
"relevance_score": 0.7,
}
store.store_findings(run_id, topic["id"], [finding])
store.store_findings(run_id, topic["id"], [finding])
sightings = store.get_sightings_for_run(topic["id"], run_id)
assert len(sightings) == 1
def test_store_findings_updates_existing_sighting_for_same_run(temp_db):
"""Test that retrying a run refreshes its sighting snapshot instead of freezing it."""
topic = store.add_topic("Test Topic")
run_id = store.record_run(topic["id"], source_mode="v3")
finding = {
"source": "reddit",
"source_url": "https://reddit.com/1",
"source_title": "Reddit 1",
"content": "Content",
"engagement_score": 10.0,
"relevance_score": 0.7,
}
store.store_findings(run_id, topic["id"], [finding])
store.store_findings(
run_id,
topic["id"],
[{**finding, "source_title": "Reddit 1 updated", "engagement_score": 15.0}],
)
sightings = store.get_sightings_for_run(topic["id"], run_id)
assert len(sightings) == 1
assert sightings[0]["source_title"] == "Reddit 1 updated"
assert sightings[0]["engagement_score"] == 15.0
def test_update_validates_allowed_columns(temp_db, sample_report):
"""Test update_run/update_finding accept valid keys and reject invalid keys."""
topic = store.add_topic("Test Topic")
@@ -503,16 +685,16 @@ def test_get_new_findings_filters_by_date(temp_db, sample_report):
findings = store.findings_from_report(sample_report)
store.store_findings(run_id, topic["id"], findings)
# Get findings since tomorrow (should be empty)
tomorrow = (datetime.now() + timedelta(days=1)).strftime("%Y-%m-%d")
# Use UTC because store writes first_seen via SQLite's datetime('now') (UTC).
# Local-time math here would flake near midnight UTC.
tomorrow = (datetime.now(timezone.utc) + timedelta(days=1)).strftime("%Y-%m-%d")
new_findings = store.get_new_findings(topic["id"], since=tomorrow)
assert len(new_findings) == 0
# Get findings since yesterday (should have all)
yesterday = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
yesterday = (datetime.now(timezone.utc) - timedelta(days=1)).strftime("%Y-%m-%d")
new_findings = store.get_new_findings(topic["id"], since=yesterday)
assert len(new_findings) == 4
+20 -4
View File
@@ -1,4 +1,5 @@
import re
import sys
import unittest
from pathlib import Path
@@ -6,16 +7,31 @@ from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SKILL_ROOT = ROOT / "skills" / "last30days"
sys.path.insert(0, str(SKILL_ROOT / "scripts"))
from lib.skill_meta import read_skill_version # noqa: E402
def _skill_version() -> str:
text = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
match = re.search(r'^version:\s*"([^"]+)"\s*$', text, re.MULTILINE)
if not match:
version = read_skill_version(SKILL_ROOT / "SKILL.md")
if not version:
raise AssertionError("SKILL.md version frontmatter not found")
return match.group(1)
return version
class TestVersionConsistency(unittest.TestCase):
def test_skill_md_uses_double_quoted_version(self) -> None:
# The shared VERSION_RE in skill_meta.py accepts double-quoted,
# single-quoted, and unquoted YAML version scalars. This repo's
# SKILL.md must use the double-quoted form so the badge string stays
# deterministic and contributors don't accidentally introduce a
# quoting style that's harder for downstream tooling to parse.
text = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
self.assertRegex(
text,
re.compile(r'^version:\s*"[^"]+"\s*$', re.MULTILINE),
msg="SKILL.md frontmatter version must use double-quoted form",
)
def test_root_skill_header_matches_frontmatter_version(self) -> None:
text = (SKILL_ROOT / "SKILL.md").read_text(encoding="utf-8")
version = _skill_version()
+64 -2
View File
@@ -251,7 +251,38 @@ def test_run_topic_success(mock_subprocess, temp_db):
"source_weights": {},
},
"clusters": [],
"ranked_candidates": [],
"ranked_candidates": [
{
"candidate_id": "c-r1",
"item_id": "R1",
"source": "reddit",
"title": "Test",
"url": "https://reddit.com/1",
"snippet": "Snippet",
"subquery_labels": ["primary"],
"native_ranks": {"reddit": 1},
"local_relevance": 0.8,
"freshness": 100,
"engagement": 50.0,
"source_quality": 0.8,
"rrf_score": 1.0,
"final_score": 0.8,
"explanation": "Snippet",
"source_items": [
{
"item_id": "R1",
"source": "reddit",
"title": "Test",
"body": "Content",
"url": "https://reddit.com/1",
"author": "user",
"engagement_score": 50.0,
"local_relevance": 0.8,
"snippet": "Snippet",
}
],
}
],
"items_by_source": {
"reddit": [
{
@@ -338,7 +369,38 @@ def test_run_topic_calls_delivery(mock_deliver, mock_subprocess, temp_db):
"source_weights": {},
},
"clusters": [],
"ranked_candidates": [],
"ranked_candidates": [
{
"candidate_id": "c-r1",
"item_id": "R1",
"source": "reddit",
"title": "Test",
"url": "https://reddit.com/1",
"snippet": "Snippet",
"subquery_labels": ["primary"],
"native_ranks": {"reddit": 1},
"local_relevance": 0.8,
"freshness": 100,
"engagement": 50.0,
"source_quality": 0.8,
"rrf_score": 1.0,
"final_score": 0.8,
"explanation": "Snippet",
"source_items": [
{
"item_id": "R1",
"source": "reddit",
"title": "Test",
"body": "Content",
"url": "https://reddit.com/1",
"author": "user",
"engagement_score": 50.0,
"local_relevance": 0.8,
"snippet": "Snippet",
}
],
}
],
"items_by_source": {
"reddit": [
{
+132 -2
View File
@@ -1,6 +1,7 @@
"""Tests for YouTube transcript highlights and yt-dlp safety flags."""
import json
import os
import sys
import tempfile
import unittest
@@ -251,7 +252,7 @@ class TestFetchTranscriptFallback(unittest.TestCase):
mock.patch.object(youtube_yt, "_fetch_transcript_direct", return_value=sample_vtt) as direct_mock:
result = youtube_yt.fetch_transcript("vid2", "/tmp/test")
yt_mock.assert_not_called()
direct_mock.assert_called_once_with("vid2")
direct_mock.assert_called_once_with("vid2", status=None)
self.assertIsNotNone(result)
self.assertIn("Direct transcript content", result)
@@ -362,7 +363,7 @@ class TestSearchAndTranscribe(unittest.TestCase):
]
# fetch_transcripts_parallel returns None for music videos, text for talks
def fake_parallel(video_ids, max_workers=5):
def fake_parallel(video_ids, max_workers=5, out_captions_disabled=None):
result = {}
for vid in video_ids:
if vid.startswith("talk"):
@@ -407,5 +408,134 @@ class TestSearchAndTranscribe(unittest.TestCase):
ft_mock.assert_not_called()
class TestYtdlpSSHRouting(unittest.TestCase):
"""LAST30DAYS_YOUTUBE_SSH_HOST routes yt-dlp invocations through SSH for residential IP."""
def setUp(self):
# Ensure clean env for each test
self._saved_env = os.environ.pop("LAST30DAYS_YOUTUBE_SSH_HOST", None)
def tearDown(self):
os.environ.pop("LAST30DAYS_YOUTUBE_SSH_HOST", None)
if self._saved_env is not None:
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = self._saved_env
def test_no_env_var_returns_none(self):
"""Without the env var set, _ytdlp_ssh_host returns None."""
self.assertIsNone(youtube_yt._ytdlp_ssh_host())
def test_env_var_returns_host(self):
"""With LAST30DAYS_YOUTUBE_SSH_HOST set, _ytdlp_ssh_host returns it."""
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = "macmini"
self.assertEqual(youtube_yt._ytdlp_ssh_host(), "macmini")
def test_env_var_whitespace_stripped(self):
"""Whitespace around the host alias is stripped."""
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = " macmini "
self.assertEqual(youtube_yt._ytdlp_ssh_host(), "macmini")
def test_empty_env_var_falls_back_to_none(self):
"""An empty env var is treated as unset."""
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = ""
self.assertIsNone(youtube_yt._ytdlp_ssh_host())
def test_wrap_cmd_passthrough_when_unset(self):
"""_wrap_ytdlp_cmd returns input unchanged when SSH routing is off."""
cmd = ["yt-dlp", "--ignore-config", "ytsearch5:test"]
self.assertEqual(youtube_yt._wrap_ytdlp_cmd(cmd), cmd)
def test_wrap_cmd_prepends_ssh_when_set(self):
"""_wrap_ytdlp_cmd prepends ssh <host> when SSH routing is on."""
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = "macmini"
cmd = ["yt-dlp", "--ignore-config", "ytsearch5:test"]
wrapped = youtube_yt._wrap_ytdlp_cmd(cmd)
self.assertEqual(wrapped[0], "ssh")
self.assertEqual(wrapped[1], "-o")
self.assertEqual(wrapped[2], "BatchMode=yes")
# `--` terminates SSH option parsing so a host starting with `-`
# (e.g. `-oProxyCommand=...`) cannot be reinterpreted as a flag.
self.assertEqual(wrapped[3], "--")
self.assertEqual(wrapped[4], "macmini")
# Final arg is the shell-quoted command string
self.assertIn("yt-dlp", wrapped[5])
self.assertIn("ytsearch5:test", wrapped[5])
def test_wrap_cmd_quotes_args_with_spaces(self):
"""Args containing spaces or special chars are shell-quoted."""
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = "macmini"
cmd = ["yt-dlp", "ytsearch5:hello world", "--dump-json"]
wrapped = youtube_yt._wrap_ytdlp_cmd(cmd)
# shlex.quote wraps the whole arg in single quotes when it contains spaces
self.assertIn("'ytsearch5:hello world'", wrapped[5])
def test_wrap_cmd_uses_option_terminator(self):
"""`--` is inserted before host as defense-in-depth even for valid hosts."""
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = "macmini"
cmd = ["yt-dlp", "--version"]
wrapped = youtube_yt._wrap_ytdlp_cmd(cmd)
dash_idx = wrapped.index("--")
self.assertEqual(wrapped[dash_idx + 1], "macmini")
def test_host_alias_with_dash_prefix_is_rejected(self):
"""A host value starting with `-` is rejected by the alias validator.
Without validation, ssh could parse `-oProxyCommand=...` as a flag
instead of a hostname. The `--` terminator in _wrap_ytdlp_cmd is
defense-in-depth; this regex on _ytdlp_ssh_host() rejects the value
before it ever reaches the ssh command line.
"""
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = "-oProxyCommand=evil"
self.assertIsNone(youtube_yt._ytdlp_ssh_host())
# And the wrap function falls back to the local-execution path.
cmd = ["yt-dlp", "--version"]
self.assertEqual(youtube_yt._wrap_ytdlp_cmd(cmd), cmd)
def test_host_alias_with_shell_metacharacters_is_rejected(self):
"""Host values containing spaces, semicolons, $, etc. are rejected."""
for bad in ("host;rm -rf /", "host name", "host$IFS", "host`whoami`", "host&cmd"):
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = bad
self.assertIsNone(
youtube_yt._ytdlp_ssh_host(),
msg=f"validator should reject {bad!r}",
)
def test_host_alias_validator_accepts_realistic_aliases(self):
"""Valid SSH config aliases are accepted: bare names, FQDNs, IPs."""
for good in ("macmini", "home-server", "pi5.local", "192.168.1.10", "homelab_box"):
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = good
self.assertEqual(youtube_yt._ytdlp_ssh_host(), good)
def test_is_ytdlp_installed_short_circuits_with_ssh(self):
"""is_ytdlp_installed returns True without local check when SSH routing is on."""
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = "macmini"
with mock.patch("lib.youtube_yt.shutil.which", return_value=None) as which_mock:
self.assertTrue(youtube_yt.is_ytdlp_installed())
which_mock.assert_not_called()
def test_is_ytdlp_installed_falls_through_without_ssh(self):
"""is_ytdlp_installed checks PATH normally when SSH routing is off."""
with mock.patch("lib.youtube_yt.shutil.which", return_value="/usr/bin/yt-dlp"):
self.assertTrue(youtube_yt.is_ytdlp_installed())
with mock.patch("lib.youtube_yt.shutil.which", return_value=None):
self.assertFalse(youtube_yt.is_ytdlp_installed())
def test_search_call_routes_through_ssh(self):
"""search_youtube wraps the yt-dlp invocation when SSH routing is on."""
os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = "macmini"
from lib.subproc import SubprocResult
fake_result = SubprocResult(returncode=0, stdout="", stderr="")
with mock.patch.object(youtube_yt.subproc, "run_with_timeout",
return_value=fake_result) as run_mock:
youtube_yt.search_youtube("test", "2026-02-01", "2026-03-01")
cmd = run_mock.call_args.args[0]
self.assertEqual(cmd[0], "ssh")
self.assertEqual(cmd[3], "--")
self.assertEqual(cmd[4], "macmini")
# The shell-quoted yt-dlp invocation lives at index 5
self.assertIn("yt-dlp", cmd[5])
self.assertIn("--ignore-config", cmd[5])
self.assertIn("--no-cookies-from-browser", cmd[5])
if __name__ == "__main__":
unittest.main()
Generated
+5 -5
View File
@@ -106,7 +106,7 @@ wheels = [
[[package]]
name = "last30days-skill"
version = "3.2.3"
version = "3.2.4"
source = { virtual = "." }
[package.dev-dependencies]
@@ -119,7 +119,7 @@ dev = [
[package.metadata.requires-dev]
dev = [
{ name = "pytest", specifier = ">=9,<10" },
{ name = "pytest", specifier = ">=9.0.3,<10" },
{ name = "pytest-cov", specifier = ">=7,<8" },
]
@@ -152,7 +152,7 @@ wheels = [
[[package]]
name = "pytest"
version = "9.0.2"
version = "9.0.3"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "colorama", marker = "sys_platform == 'win32'" },
@@ -161,9 +161,9 @@ dependencies = [
{ name = "pluggy" },
{ name = "pygments" },
]
sdist = { url = "https://files.pythonhosted.org/packages/d1/db/7ef3487e0fb0049ddb5ce41d3a49c235bf9ad299b6a25d5780a89f19230f/pytest-9.0.2.tar.gz", hash = "sha256:75186651a92bd89611d1d9fc20f0b4345fd827c41ccd5c299a868a05d70edf11", size = 1568901, upload-time = "2025-12-06T21:30:51.014Z" }
sdist = { url = "https://files.pythonhosted.org/packages/7d/0d/549bd94f1a0a402dc8cf64563a117c0f3765662e2e668477624baeec44d5/pytest-9.0.3.tar.gz", hash = "sha256:b86ada508af81d19edeb213c681b1d48246c1a91d304c6c81a427674c17eb91c", size = 1572165, upload-time = "2026-04-07T17:16:18.027Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/3b/ab/b3226f0bd7cdcf710fbede2b3548584366da3b19b5021e74f5bde2a8fa3f/pytest-9.0.2-py3-none-any.whl", hash = "sha256:711ffd45bf766d5264d487b917733b453d917afd2b0ad65223959f59089f875b", size = 374801, upload-time = "2025-12-06T21:30:49.154Z" },
{ url = "https://files.pythonhosted.org/packages/d4/24/a372aaf5c9b7208e7112038812994107bc65a84cd00e0354a88c2c77a617/pytest-9.0.3-py3-none-any.whl", hash = "sha256:2c5efc453d45394fdd706ade797c0a81091eccd1d6e4bccfcd476e2b8e0ab5d9", size = 375249, upload-time = "2026-04-07T17:16:16.13Z" },
]
[[package]]