When yt-dlp is installed but stale (or otherwise unable to fetch transcripts
for any returned videos), runs previously reported YouTube as fully
successful in two user-facing surfaces:
1. Footer (render.py): showed "N videos | M views" with no indication
that zero transcripts were captured. The "with transcripts" segment
was conditionally suppressed when the count was zero - converting
the canonical stale-binary failure mode into a silent absence at
the very surface users read for "did this work?".
2. Quality nudge (quality_nudge.py): classified YouTube as "active"
based purely on yt-dlp installation + absence of a top-level error.
Per-video transcript-fetch ratio was never inspected. A run that
returned N videos with 0 transcripts (canonical stale-binary
failure) was reported as fully active.
The engine itself logs the failure correctly at default stderr level
(`[YouTube] Got transcripts for 0/N videos (N failed)`), but that line
gets buried in 100+ lines of parallel-source progress output and is
contradicted by the success-shaped footer and nudge that follow.
This change makes both conclusion surfaces honest:
* render.py footer always renders "M/N with transcripts" so the ratio
is visible regardless of value. Zero is no longer hidden. Format is
M/N (not bare M) so the denominator is in the message and the user
does not have to cross-reference the "videos" count.
* quality_nudge.py adds a third tier between "active" and "missing":
"degraded". Triggered when yt-dlp is installed AND videos were
returned AND transcript-fetch ratio is below threshold (default 50%,
tunable via DEGRADED_TRANSCRIPT_THRESHOLD env var). Emits an
actionable nudge: "YouTube returned N videos but only M transcripts
captured. The most common cause is a stale yt-dlp binary - YouTube's
caption format changes frequently and old binaries silently fail
every transcript. Update via your package manager: scoop update
yt-dlp (Windows), brew upgrade yt-dlp (macOS), or pip install -U
yt-dlp."
* last30days.py populates youtube_videos_count and
youtube_transcripts_count in the research_results dict it passes to
compute_quality_score, enabling the new degraded check at the call
site.
Threshold rationale: 50% accommodates a few legitimate
caption-disabled videos in a multi-video result, but a stale-binary
run that fails every transcript trips the nudge cleanly.
Score impact: degradation is informational, not score-affecting.
YouTube still counts as "active" in score_pct so users do not see
their score drop for a fixable client-side issue. The nudge directs
them to their own package manager.
Tests:
* tests/test_quality_nudge.py: 6 new TestYouTubeDegraded cases cover
zero-transcripts-flags-degraded, partial-above-threshold-does-not-flag,
zero-videos-does-not-flag (no false positives on absence),
one-of-three-flags-degraded, threshold-tunable-via-config, and
degraded-does-not-affect-score.
* tests/test_render_v3.py: 4 new YoutubeFooterTranscriptRatioTests
cases cover zero-transcripts-with-videos-renders-zero-over-total
(the regression repro), partial-renders-ratio, full-renders-ratio,
and no-videos-suppresses-entire-segment.
All 29 new test cases verified GREEN with the fix and RED without it
(temp-reverted both files separately to confirm each test catches the
specific regression it asserts).
Integration validation: ran the engine against an intentionally stale
yt-dlp 2025.03.31 binary placed first on PATH. Pre-fix the footer
showed `YouTube: 3 videos | 386,815 views` (no transcript signal).
Post-fix the footer shows `YouTube: 3 videos | 386,815 views | 0/3
with transcripts` and stderr emits "Degraded: YouTube" plus the
actionable update-yt-dlp nudge.
Out of scope (deserves its own PR): exposing transcripts_captured in
the EVIDENCE FOR SYNTHESIS block so the synthesizing model can flag
degradation in prose. Larger schema-touching change.
render.py, ui.py, and last30days.py had hardcoded "v3.0.0" in titles
and headers while plugin.json was at 3.1.1. Use _skill_version()
(reads from plugin.json at runtime) so version strings stay in sync.
Fixes#284
Addresses two concerns surfaced during PR #376 review:
1. **SSH option-injection on the host value.** The original PR uses
shlex.quote() on the remote command and added a `--` option terminator
in front of the host, but neither one stops a hostile env var like
`LAST30DAYS_YT_SSH_HOST=-oProxyCommand=...` from being read in the
first place. Tighten `_ytdlp_ssh_host()` to validate the host against
`^[a-zA-Z0-9._-]+$` (plain hostname/SSH-config-alias shape: letters,
digits, dot, underscore, hyphen). Any value that doesn't match logs a
warning to stderr and returns None, so the wrap function falls back to
local execution. The `--` terminator stays as defense-in-depth for the
case where a valid host happens to start with `-`, but the regex closes
the door on the env var reaching ssh at all.
2. **Env var naming consistency.** Existing skill-internal config knobs
spell out their domain: `LAST30DAYS_X_BACKEND`, `LAST30DAYS_X_MODEL`,
`LAST30DAYS_PLANNER_MODEL`, `LAST30DAYS_RERANK_MODEL`, etc. The module
is `youtube_yt.py`, the source key is `youtube`, the function family
is `is_youtube_*()` — `YT` was the odd abbreviation out. Rename to
`LAST30DAYS_YOUTUBE_SSH_HOST` so the variable matches the user mental
model ("route YouTube fetches via residential IP") and the codebase's
spelled-out convention.
Adds three new tests:
- test_host_alias_with_dash_prefix_is_rejected (validator rejects `-o...`)
- test_host_alias_with_shell_metacharacters_is_rejected (rejects spaces, ;, $, `, &)
- test_host_alias_validator_accepts_realistic_aliases (allows FQDNs, IPs, bare aliases)
The existing test_wrap_cmd_uses_option_terminator is rewritten to use a
valid host value (since an invalid one is now filtered upstream) and
continues to assert the `--` terminator placement as defense-in-depth.
44/44 youtube_yt tests pass (40 prior + 4 net new validator tests).
Adds LAST30DAYS_YT_SSH_HOST env var (or `~/.config/last30days/.env` key).
When set, yt-dlp YouTube search invocations are wrapped as
`ssh <host> "yt-dlp ..."` so they run on a residential-IP machine.
Motivation: when last30days runs on a datacenter VPS (Hetzner,
DigitalOcean, AWS, etc.), `ytsearch:` queries return 0 results because
YouTube's bot-wall fingerprints datacenter IP ranges before any cookie
check runs. Cookies alone don't fix this — the IP reputation is checked
first. Verified across yt-dlp stable 2026.03.17 and nightly builds.
The existing fallbacks (browser cookies, residential proxy services,
excluding YouTube) all have downsides: cookies expire, proxies cost
money, exclusion loses signal. Many users with a Mac mini, Pi, or
home server can host yt-dlp on their own residential IP — this just
needs an SSH alias and a one-line env var to wire it up.
Behaviour:
- Default (env var unset): identical to before, no shape change.
- Env var set: search command list is wrapped with `ssh -o BatchMode=yes
<host> "<shell-quoted yt-dlp invocation>"`. is_ytdlp_installed()
returns True without a local PATH check (the binary lives on the
remote host).
- Transcript path: when SSH-routing is on, skips the yt-dlp transcript
path (which writes a VTT file we couldn't easily read back over SSH)
and uses the existing _fetch_transcript_direct HTTP fallback. The
timedtext API isn't bot-walled, so this works fine on datacenter IPs.
Setup pitfall documented in the function docstring: on macOS hosts
with Homebrew, `eval "$(/opt/homebrew/bin/brew shellenv zsh)"` must
live in ~/.zshenv (not just ~/.zprofile) — non-login SSH shells don't
source .zprofile, so without this `ssh macmini "yt-dlp ..."` returns
"command not found" while interactive SSH works fine.
Tests: 10 new cases covering env var read, whitespace stripping,
empty-value handling, command wrapping passthrough/active modes,
shlex quoting, is_ytdlp_installed short-circuit, and end-to-end
search_youtube wrapping. Full test suite: 0 new failures (the 14
pre-existing failures in test_store, test_watchlist, test_setup_openclaw,
test_safari_cookies, test_version_consistency are unchanged on main).
Verified live: 0 results → 4 real hits for "claude code" search from a
Hetzner VPS routed through a Mac mini exit node on Tailscale.
Greptile flagged that `entity_reports and args.emit == "html"` appeared in
two places — once when computing the footer display path, again when calling
save_output. The else-branches differ between the two callsites (the display
needs `report.topic` as fallback; the save call needs `None` so save_output
falls back to the report's own topic), so collapsing into one shared
expression would be wrong, but hoisting just the condition into a single
`is_comparison_html` bool eliminates the risk of drift while keeping the
two callsites' fallback semantics distinct.
Adds a one-command shareable HTML mode to /last30days. The skill detects
HTML intent (explicit --emit=html / --emit:html / --html flag in
$ARGUMENTS, or natural-language asks like "give me a shareable brief",
"for Slack", "export as HTML"), runs the normal research + chat synthesis
flow, then saves a self-contained HTML file to
~/Documents/Last30Days/{topic}-brief.html. The synthesis appears in chat
as usual; the HTML is an additional artifact for sharing.
User experience:
/last30days OpenClaw --emit=html
/last30days OpenClaw, give me an HTML brief for Slack
Synthesis prints to chat. Last line of the response: "📎 Shareable brief
saved to ~/Documents/Last30Days/openclaw-brief.html". Open it, drag it
into a message, browser-print to PDF, email it.
Architecture:
- SKILL.md gets a small detection block (triggers + early exit +
MUST/MUST NOT rules + rationale) that points to a reference file.
- references/save-html-brief.md owns the implementation: capture the
synthesis verbatim into a temp file via heredoc, invoke the engine
with --emit=html --synthesis-file, save to disk, append the
confirmation line to chat.
- lib/render.py exposes render_for_html(report, synthesis_md=None) and
render_for_html_comparison(...) -- clean markdown for HTML
conversion. Omits debug file header, model-facing safety note, and
data quality warnings (those stay in engine stderr; recipients can't
act on them in a shared artifact).
- lib/html_render.py is a new module: ~200-line CSS template (dark
mode default, prefers-color-scheme switch, print stylesheet, mobile
breakpoint), stdlib-regex markdown-to-HTML converter, marker-based
META + engine-footer wrapping, PROSE_LABELS registry promoting plain
-text labels to <h2>, colophon builder.
- last30days.py adds --emit=html argparse choice and --synthesis-file
PATH flag (engine still callable directly without the skill in the
loop).
Design:
- Voice-led research brief, not corporate report. Inter + JetBrains
Mono via Google Fonts with full system fallbacks (no FOIT, works
offline). Brand purple #a855f7 (#7c3aed in light mode). Type ramp:
body 17px/400/muted, bold lead-in 17px/600/fg, h2 + .prose-label
20px/600/fg, monospace badge/meta/footer/colophon at 13-13.5px.
- 720px max-width, generous whitespace, no card layouts or shadows.
- Print stylesheet: light theme, A4 margins, [href]::after URL
footnotes, page-break-inside:avoid on the engine footer.
Templated (locked) shell:
- HTML5 boilerplate, Google Fonts <link> with preconnect, all CSS
inline.
- .badge / .meta / .engine-footer / .colophon containers.
Flexible (role-based):
- <h2> rendering covers BOTH plain ## headers (comparison mode per
LAW 4 exception) AND promoted prose labels via PROSE_LABELS
registry. Adding a new SKILL.md prose label is a one-line tuple
addition; no CSS or template changes.
- Marker-based engine boundaries (<!-- META: ... -->,
<!-- PASS-THROUGH FOOTER -->) survive the markdown converter and
get promoted post-conversion. Robust to engine output format
changes.
- Generic markdown-to-HTML for body content; future SKILL.md additions
(new sections, tables, blockquotes) render correctly without code
changes.
Tests: 30 new tests in tests/test_html_render.py covering snapshots
(rich/thin/comparison), CLI parsing, --synthesis-file end-to-end, prose
label promotion, warning exclusion from artifact, parseability via
html.parser, no-script self-containment.
No SKILL.md voice contract changes, no LAWs 1-8 changes, no new pip
dependencies, no JavaScript anywhere.
Daniel Zivkovic