Rebased onto current main where:
- instagram.py uses unified http.get (not _requests fallback); tests now
mock http.get and assert params/timeout kwargs.
- quality_nudge tests use lib.* import path with sys.path setup.
- README/CONFIGURATION.md memory-dir lines say "defaults to" so they pass
test_no_stray_hardcoded_memory_dir_paths.
Six small additive changes that make the skill correctly understand its
configured sources, plus tests + docs.
User-visible benefits
- LAST30DAYS_STORE=1 in .env turns persistence default-on without
remembering --store on every invocation. Mirrors LAST30DAYS_DEBUG /
LAST30DAYS_SKIP_PREFLIGHT convention.
- SCRAPE_CREATORS_API_KEY (with underscore) accepted as alias for the
canonical name. Matches the spelling used in the vendor's own example
code (Adrian Horning's repo); saves the next user the same diagnostic
rabbit hole.
- Bluesky search now hits api.bsky.app (canonical AppView) instead of
public.api.bsky.app (BunnyCDN-blocked public mirror as of 2026-05-04).
BSKY_SEARCH_HOST env var lets users self-rescue future host migrations
without a code release. Pre-fix: silent 0 Bluesky posts on every run.
- App-password format validator emits a one-shot stderr warning when
BSKY_APP_PASSWORD doesn't match xxxx-xxxx-xxxx-xxxx form. Detect-don't-
gate: createSession still accepts main passwords; the warning helps
users identify a hygiene issue without breaking existing setups.
- Instagram retry on multi-token 500. SC's v2 reels endpoint wraps
Google Search and 500's frequently on multi-word queries; a hashtag-
form retry runs once before bubbling up. Documented vendor instability.
- LAST30DAYS_TRANSCRIPT_TIMEOUT env var (default 30s, was hardcoded 15s).
SC's transcript endpoint regularly takes >15s; the old default was
clipping legitimate responses.
- Silent-failure visibility: new bonus_errored field in the quality
nudge fires when SC is configured but Instagram returned 0 items.
Users see "Bonus source silent: Instagram" instead of unexplained
absence.
- YouTube degraded-ratio false-positive fixed. Captions-disabled videos
can never produce a transcript regardless of yt-dlp version; they're
now subtracted from the denominator so a single uploader-disabled
video doesn't false-trigger the "stale yt-dlp" nudge.
- urllib retry path: status_code attribute typo fix. The Instagram
500-retry was dead code on the urllib branch (getattr(e, 'status', ...)
while http.HTTPError exposes status_code).
Docs
- README.md: added /plugin install last30days step after marketplace add
in three places (the install was previously omitted in the docs).
- CONFIGURATION.md: documented LAST30DAYS_STORE env var, added
BSKY_SEARCH_HOST + app-password format section, mentioned
LAST30DAYS_TRANSCRIPT_TIMEOUT in the Instagram source row.
Test plan
- 43 new unit tests across test_bluesky.py, test_instagram_sc.py,
test_quality_nudge.py, test_youtube_yt.py
- 141 total tests passing in target suite
- Verified end-to-end: /last30days "Toronto resale condo market" with
all 11+ sources active stored 35 new + 5 updated findings, all builder-
PR-style accounts absent (organic agent voice in Instagram + TikTok
results)
Backward compatibility
All changes are strictly additive. Optional kwargs default to None.
New env vars are opt-in. Existing CLI flags untouched. Existing callers
of public functions unaffected.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two findings from Greptile review on PR #302:
1. tests/test_cli_v3.py:302 - The test asserted run_mock.call_args_list[0]
was the main runner's invocation, but fanout.run_competitor_fanout
submits main + competitors to a ThreadPoolExecutor and iterates with
as_completed. With zero-latency mocks, thread scheduling determines
which pipeline.run call lands first, so the competitor's call could
take index [0] and flake CI. Replace [0] indexing with a predicate
match on the canonicalized github_repos kwargs.
2. skills/last30days/scripts/last30days.py:662 - When auto_resolve returns
github_repos, it has already run canonicalize_github_repos(cap=5) and
ranked by relevance. The downstream block then re-canonicalized with
cap=None, which can re-sort by topic-slug match and clobber the
auto_resolve relevance order. Guard the second canonicalization with
a repos_from_auto_resolve flag so it only fires for user-supplied
--github-repo input.
Brave uses identical v10 AES-128-CBC encryption to Chrome; only the
DB path (BraveSoftware/Brave-Browser) and Keychain service name
("Brave Safe Storage") differ. Refactored chrome_cookies.py to share
a single _extract_chromium_cookies_macos helper rather than duplicating
the decryption logic.
Profile discovery tries Default/ first, then scans numbered Profile N/
directories so non-default Brave profiles are covered.
When yt-dlp is installed but stale (or otherwise unable to fetch transcripts
for any returned videos), runs previously reported YouTube as fully
successful in two user-facing surfaces:
1. Footer (render.py): showed "N videos | M views" with no indication
that zero transcripts were captured. The "with transcripts" segment
was conditionally suppressed when the count was zero - converting
the canonical stale-binary failure mode into a silent absence at
the very surface users read for "did this work?".
2. Quality nudge (quality_nudge.py): classified YouTube as "active"
based purely on yt-dlp installation + absence of a top-level error.
Per-video transcript-fetch ratio was never inspected. A run that
returned N videos with 0 transcripts (canonical stale-binary
failure) was reported as fully active.
The engine itself logs the failure correctly at default stderr level
(`[YouTube] Got transcripts for 0/N videos (N failed)`), but that line
gets buried in 100+ lines of parallel-source progress output and is
contradicted by the success-shaped footer and nudge that follow.
This change makes both conclusion surfaces honest:
* render.py footer always renders "M/N with transcripts" so the ratio
is visible regardless of value. Zero is no longer hidden. Format is
M/N (not bare M) so the denominator is in the message and the user
does not have to cross-reference the "videos" count.
* quality_nudge.py adds a third tier between "active" and "missing":
"degraded". Triggered when yt-dlp is installed AND videos were
returned AND transcript-fetch ratio is below threshold (default 50%,
tunable via DEGRADED_TRANSCRIPT_THRESHOLD env var). Emits an
actionable nudge: "YouTube returned N videos but only M transcripts
captured. The most common cause is a stale yt-dlp binary - YouTube's
caption format changes frequently and old binaries silently fail
every transcript. Update via your package manager: scoop update
yt-dlp (Windows), brew upgrade yt-dlp (macOS), or pip install -U
yt-dlp."
* last30days.py populates youtube_videos_count and
youtube_transcripts_count in the research_results dict it passes to
compute_quality_score, enabling the new degraded check at the call
site.
Threshold rationale: 50% accommodates a few legitimate
caption-disabled videos in a multi-video result, but a stale-binary
run that fails every transcript trips the nudge cleanly.
Score impact: degradation is informational, not score-affecting.
YouTube still counts as "active" in score_pct so users do not see
their score drop for a fixable client-side issue. The nudge directs
them to their own package manager.
Tests:
* tests/test_quality_nudge.py: 6 new TestYouTubeDegraded cases cover
zero-transcripts-flags-degraded, partial-above-threshold-does-not-flag,
zero-videos-does-not-flag (no false positives on absence),
one-of-three-flags-degraded, threshold-tunable-via-config, and
degraded-does-not-affect-score.
* tests/test_render_v3.py: 4 new YoutubeFooterTranscriptRatioTests
cases cover zero-transcripts-with-videos-renders-zero-over-total
(the regression repro), partial-renders-ratio, full-renders-ratio,
and no-videos-suppresses-entire-segment.
All 29 new test cases verified GREEN with the fix and RED without it
(temp-reverted both files separately to confirm each test catches the
specific regression it asserts).
Integration validation: ran the engine against an intentionally stale
yt-dlp 2025.03.31 binary placed first on PATH. Pre-fix the footer
showed `YouTube: 3 videos | 386,815 views` (no transcript signal).
Post-fix the footer shows `YouTube: 3 videos | 386,815 views | 0/3
with transcripts` and stderr emits "Degraded: YouTube" plus the
actionable update-yt-dlp nudge.
Out of scope (deserves its own PR): exposing transcripts_captured in
the EVIDENCE FOR SYNTHESIS block so the synthesizing model can flag
degradation in prose. Larger schema-touching change.
Asserts compute_save_path_display() never returns a backslash when the
save_dir is under the user's home directory, regardless of host OS.
Reproduces the original bug on Windows (failed message before the fix:
AssertionError: '\' unexpectedly found in
'~/l30d_save_path__luu2g76\Documents\Last30Days\british-airways-middle-east-raw-v3.md'
)
and locks in the contract on POSIX hosts (passes trivially today; would
fail if anyone removes .as_posix() in the future).
Verified by temporarily reverting the fix and confirming RED, then
re-applying the fix and confirming GREEN. All 13 CliV3Tests pass.
PR #366 routes Reddit URLs found in web-search results through the public
Reddit JSON API to recover thread body + top comments (the Claude Code
WebFetch tool blocks reddit.com directly). That bypass is sound and the
fixed problem is real - but the always-on shape ignores user intent on
source gating.
A user who sets EXCLUDE_SOURCES=reddit to suppress Reddit results would
still get Reddit content smuggled back in via web-search URLs that
happen to point at reddit.com threads. This contradicts the suppression
contract that EXCLUDE_SOURCES is supposed to provide (see
lib/pipeline.available_sources where the same env var gates the
top-level Reddit source).
Add a _reddit_excluded(config) check in web_search() that mirrors the
parsing pattern from lib/pipeline (comma-separated, case-insensitive,
whitespace-tolerant). When reddit is in EXCLUDE_SOURCES, skip the
enrichment pass entirely - the web results themselves still flow
through, but they're not augmented with Reddit body/comments.
Four new tests in test_grounding_v3.py cover:
- EXCLUDE_SOURCES=reddit skips enrichment
- case-insensitive parsing matches REDDIT/Reddit/whitespace-padded/csv
- Other sources in EXCLUDE_SOURCES don't trigger the gate
- Enrichment runs normally when reddit isn't excluded
19/19 grounding tests pass.
Addresses two concerns surfaced during PR #376 review:
1. **SSH option-injection on the host value.** The original PR uses
shlex.quote() on the remote command and added a `--` option terminator
in front of the host, but neither one stops a hostile env var like
`LAST30DAYS_YT_SSH_HOST=-oProxyCommand=...` from being read in the
first place. Tighten `_ytdlp_ssh_host()` to validate the host against
`^[a-zA-Z0-9._-]+$` (plain hostname/SSH-config-alias shape: letters,
digits, dot, underscore, hyphen). Any value that doesn't match logs a
warning to stderr and returns None, so the wrap function falls back to
local execution. The `--` terminator stays as defense-in-depth for the
case where a valid host happens to start with `-`, but the regex closes
the door on the env var reaching ssh at all.
2. **Env var naming consistency.** Existing skill-internal config knobs
spell out their domain: `LAST30DAYS_X_BACKEND`, `LAST30DAYS_X_MODEL`,
`LAST30DAYS_PLANNER_MODEL`, `LAST30DAYS_RERANK_MODEL`, etc. The module
is `youtube_yt.py`, the source key is `youtube`, the function family
is `is_youtube_*()` — `YT` was the odd abbreviation out. Rename to
`LAST30DAYS_YOUTUBE_SSH_HOST` so the variable matches the user mental
model ("route YouTube fetches via residential IP") and the codebase's
spelled-out convention.
Adds three new tests:
- test_host_alias_with_dash_prefix_is_rejected (validator rejects `-o...`)
- test_host_alias_with_shell_metacharacters_is_rejected (rejects spaces, ;, $, `, &)
- test_host_alias_validator_accepts_realistic_aliases (allows FQDNs, IPs, bare aliases)
The existing test_wrap_cmd_uses_option_terminator is rewritten to use a
valid host value (since an invalid one is now filtered upstream) and
continues to assert the `--` terminator placement as defense-in-depth.
44/44 youtube_yt tests pass (40 prior + 4 net new validator tests).
Three changes from automated review on PR #376:
1. Add `--` option terminator before host in _wrap_ytdlp_cmd (P1 security)
Prevents SSH option injection if LAST30DAYS_YT_SSH_HOST were ever set
to a value starting with `-` (e.g. `-oProxyCommand=...`). Low
exploitability since the env var is user-controlled config — but the
fix is a single arg and turns a self-harm footgun into no footgun.
2. Hoist `import shlex` to module-level (P2 style)
Pure stdlib import, no reason for the deferred form. Cleaner.
3. Cache _ytdlp_ssh_host() result in fetch_transcript (P2 style)
Was being called 2-3x per video; the function is cheap (env lookup
+ strip) so this is purely about readability.
Adds test_wrap_cmd_uses_option_terminator covering the security fix
explicitly with a `-oFoo=bar` host value. Updates index assertions in
the two existing tests that check command shape (host is now at index
4, command string at 5, with `--` at 3).
Adds LAST30DAYS_YT_SSH_HOST env var (or `~/.config/last30days/.env` key).
When set, yt-dlp YouTube search invocations are wrapped as
`ssh <host> "yt-dlp ..."` so they run on a residential-IP machine.
Motivation: when last30days runs on a datacenter VPS (Hetzner,
DigitalOcean, AWS, etc.), `ytsearch:` queries return 0 results because
YouTube's bot-wall fingerprints datacenter IP ranges before any cookie
check runs. Cookies alone don't fix this — the IP reputation is checked
first. Verified across yt-dlp stable 2026.03.17 and nightly builds.
The existing fallbacks (browser cookies, residential proxy services,
excluding YouTube) all have downsides: cookies expire, proxies cost
money, exclusion loses signal. Many users with a Mac mini, Pi, or
home server can host yt-dlp on their own residential IP — this just
needs an SSH alias and a one-line env var to wire it up.
Behaviour:
- Default (env var unset): identical to before, no shape change.
- Env var set: search command list is wrapped with `ssh -o BatchMode=yes
<host> "<shell-quoted yt-dlp invocation>"`. is_ytdlp_installed()
returns True without a local PATH check (the binary lives on the
remote host).
- Transcript path: when SSH-routing is on, skips the yt-dlp transcript
path (which writes a VTT file we couldn't easily read back over SSH)
and uses the existing _fetch_transcript_direct HTTP fallback. The
timedtext API isn't bot-walled, so this works fine on datacenter IPs.
Setup pitfall documented in the function docstring: on macOS hosts
with Homebrew, `eval "$(/opt/homebrew/bin/brew shellenv zsh)"` must
live in ~/.zshenv (not just ~/.zprofile) — non-login SSH shells don't
source .zprofile, so without this `ssh macmini "yt-dlp ..."` returns
"command not found" while interactive SSH works fine.
Tests: 10 new cases covering env var read, whitespace stripping,
empty-value handling, command wrapping passthrough/active modes,
shlex quoting, is_ytdlp_installed short-circuit, and end-to-end
search_youtube wrapping. Full test suite: 0 new failures (the 14
pre-existing failures in test_store, test_watchlist, test_setup_openclaw,
test_safari_cookies, test_version_consistency are unchanged on main).
Verified live: 0 results → 4 real hits for "claude code" search from a
Hetzner VPS routed through a Mac mini exit node on Tailscale.
Trevin Chow