feat: configuration enablement — env-var defaults + source resilience

Six small additive changes that make the skill correctly understand its configured sources, plus tests + docs. User-visible benefits - LAST30DAYS_STORE=1 in .env turns persistence default-on without remembering --store on every invocation. Mirrors LAST30DAYS_DEBUG / LAST30DAYS_SKIP_PREFLIGHT convention. - SCRAPE_CREATORS_API_KEY (with underscore) accepted as alias for the canonical name. Matches the spelling used in the vendor's own example code (Adrian Horning's repo); saves the next user the same diagnostic rabbit hole. - Bluesky search now hits api.bsky.app (canonical AppView) instead of public.api.bsky.app (BunnyCDN-blocked public mirror as of 2026-05-04). BSKY_SEARCH_HOST env var lets users self-rescue future host migrations without a code release. Pre-fix: silent 0 Bluesky posts on every run. - App-password format validator emits a one-shot stderr warning when BSKY_APP_PASSWORD doesn't match xxxx-xxxx-xxxx-xxxx form. Detect-don't- gate: createSession still accepts main passwords; the warning helps users identify a hygiene issue without breaking existing setups. - Instagram retry on multi-token 500. SC's v2 reels endpoint wraps Google Search and 500's frequently on multi-word queries; a hashtag- form retry runs once before bubbling up. Documented vendor instability. - LAST30DAYS_TRANSCRIPT_TIMEOUT env var (default 30s, was hardcoded 15s). SC's transcript endpoint regularly takes >15s; the old default was clipping legitimate responses. - Silent-failure visibility: new bonus_errored field in the quality nudge fires when SC is configured but Instagram returned 0 items. Users see "Bonus source silent: Instagram" instead of unexplained absence. - YouTube degraded-ratio false-positive fixed. Captions-disabled videos can never produce a transcript regardless of yt-dlp version; they're now subtracted from the denominator so a single uploader-disabled video doesn't false-trigger the "stale yt-dlp" nudge. - urllib retry path: status_code attribute typo fix. The Instagram 500-retry was dead code on the urllib branch (getattr(e, 'status', ...) while http.HTTPError exposes status_code). Docs - README.md: added /plugin install last30days step after marketplace add in three places (the install was previously omitted in the docs). - CONFIGURATION.md: documented LAST30DAYS_STORE env var, added BSKY_SEARCH_HOST + app-password format section, mentioned LAST30DAYS_TRANSCRIPT_TIMEOUT in the Instagram source row. Test plan - 43 new unit tests across test_bluesky.py, test_instagram_sc.py, test_quality_nudge.py, test_youtube_yt.py - 141 total tests passing in target suite - Verified end-to-end: /last30days "Toronto resale condo market" with all 11+ sources active stored 35 new + 5 updated findings, all builder- PR-style accounts absent (organic agent voice in Instagram + TikTok results) Backward compatibility All changes are strictly additive. Optional kwargs default to None. New env vars are opt-in. Existing CLI flags untouched. Existing callers of public functions unaffected. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 16:53:10 -04:00
parent a8e462c978
commit 44971a6aae
14 changed files with 865 additions and 27 deletions
@@ -908,7 +908,17 @@ def main() -> int:
        suppress_web_promo=bool(external_plan or comp_plan),
    )
    _write_last_run(topic, report)
-    if args.store:
+    # LAST30DAYS_STORE env var = persistence default-on. Read both os.environ
+    # (for shell-exported users) and config (for users who set it in
+    # ~/.config/last30days/.env, which env.py loads but does not propagate
+    # to os.environ). Mirrors the LAST30DAYS_DEBUG / LAST30DAYS_SKIP_PREFLIGHT
+    # convention; env-var or config wins, with `--store` flag still working.
+    _store_env = (
+        os.environ.get("LAST30DAYS_STORE")
+        or config.get("LAST30DAYS_STORE")
+        or ""
+    ).lower()
+    if args.store or _store_env in ("1", "true", "yes"):
        counts = persist_report(report)
        sys.stderr.write(
            f"[last30days] Stored {counts['new']} new, {counts['updated']} updated findings\n"
@@ -922,6 +932,7 @@ def main() -> int:
        # degraded-YouTube failure mode (videos returned but transcripts
        # silently failed - typically a stale yt-dlp binary).
        youtube_items = report.items_by_source.get("youtube") or []
+        instagram_items = report.items_by_source.get("instagram") or []
        research_results = {
            "youtube_videos_count": len(youtube_items),
            "youtube_transcripts_count": sum(
@@ -930,6 +941,17 @@ def main() -> int:
            ),
            "youtube_error": report.errors_by_source.get("youtube"),
            "x_error": report.errors_by_source.get("x"),
+            # Captions-disabled videos can never produce a transcript regardless
+            # of yt-dlp version; subtract them from the degraded-ratio
+            # denominator so a single uploader-disabled video does not trip the
+            # "stale yt-dlp" nudge.
+            "youtube_captions_disabled_count": sum(
+                1 for it in youtube_items if it.metadata.get("captions_disabled")
+            ),
+            # Track Instagram returned-zero-items so quality_nudge can detect
+            # the silent-failure case (SC configured but the v2 reels endpoint
+            # 500'd through both the original query and the hashtag retry).
+            "instagram_items_count": len(instagram_items),
        }
        quality = quality_nudge.compute_quality_score(config, research_results)
        if quality.get("nudge_text"):
@@ -1,10 +1,19 @@
 """Bluesky search via AT Protocol (requires app password).

-Uses bsky.social for auth and public.api.bsky.app for post search.
-Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars.
+Uses bsky.social for auth and api.bsky.app for post search (the canonical
+authenticated AppView). The previous default `public.api.bsky.app` is the
+unauthenticated public mirror, which BunnyCDN now blocks for searchPosts
+regardless of auth header (verified 2026-05-04). Override the search host
+via BSKY_SEARCH_HOST env var if Bluesky migrates infrastructure again.
+
+Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars. App passwords are
+19-char xxxx-xxxx-xxxx-xxxx; generate at bsky.app/settings/app-passwords.
+The createSession endpoint accepts main-account passwords too, but they're
+bad hygiene (no scope, can't revoke individually).
 """

 import math
+import os
 import re
 import sys
 import time
@@ -14,7 +23,65 @@ from typing import Any, Dict, List, Optional
 from . import http, log

 BSKY_SESSION_URL = "https://bsky.social/xrpc/com.atproto.server.createSession"
-BSKY_SEARCH_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts"
+_DEFAULT_BSKY_SEARCH_HOST = "api.bsky.app"
+BSKY_SEARCH_URL = f"https://{_DEFAULT_BSKY_SEARCH_HOST}/xrpc/app.bsky.feed.searchPosts"
+
+
+def _resolve_search_url(config: Optional[Dict[str, Any]] = None) -> str:
+    """Resolve the Bluesky search URL with BSKY_SEARCH_HOST override.
+
+    Default is api.bsky.app. Override via BSKY_SEARCH_HOST in shell env or
+    .env file. The project's env.py loads .env into config but not into
+    os.environ, so check both — same hybrid pattern as last30days.py for
+    LAST30DAYS_STORE.
+
+    Hardens user-supplied host values against three common mis-configurations:
+    whitespace (e.g. " api.bsky.app "), embedded path components (e.g.
+    "api.bsky.app/xrpc/proxy") that would double the /xrpc/ segment, and
+    embedded scheme prefixes (e.g. "https://api.bsky.app"). On any of these
+    we log a warning and fall back to the default rather than building an
+    invalid URL with an opaque downstream error.
+    """
+    config = config or {}
+    raw = (
+        os.environ.get("BSKY_SEARCH_HOST")
+        or config.get("BSKY_SEARCH_HOST")
+        or _DEFAULT_BSKY_SEARCH_HOST
+    )
+    host = raw.strip().rstrip("/")
+    # Strip embedded scheme so users who paste full URLs do not break the f-string.
+    for prefix in ("https://", "http://"):
+        if host.lower().startswith(prefix):
+            host = host[len(prefix):]
+            break
+    if not host or "/" in host or " " in host:
+        # Embedded path or whitespace remains — don't trust it. Default + log.
+        if raw != _DEFAULT_BSKY_SEARCH_HOST:
+            _log(
+                f"BSKY_SEARCH_HOST={raw!r} is not a bare hostname; "
+                f"falling back to default {_DEFAULT_BSKY_SEARCH_HOST!r}"
+            )
+        host = _DEFAULT_BSKY_SEARCH_HOST
+    return f"https://{host}/xrpc/app.bsky.feed.searchPosts"
+
+
+# App-password format: xxxx-xxxx-xxxx-xxxx (19 chars, lowercase alphanumeric
+# with three hyphens at fixed positions).
+_APP_PASSWORD_RE = re.compile(r"^[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}$")
+
+
+def _validate_app_password_format(value) -> bool:
+    """Return True if value matches Bluesky's 19-char app-password format.
+
+    False for non-strings (None, int, list) so callers passing config dict
+    values directly don't crash. Detect-but-not-gate: the createSession
+    endpoint also accepts main-account passwords, so failing this check is
+    a hygiene smell, not a hard error.
+    """
+    if not isinstance(value, str):
+        return False
+    return bool(_APP_PASSWORD_RE.fullmatch(value))
+

 DEPTH_CONFIG = {
    "quick": 15,
@@ -144,6 +211,20 @@ def search_bluesky(
    if not handle or not app_password:
        return {"posts": [], "error": "Bluesky credentials not configured"}

+    # One-shot hygiene warning if BSKY_APP_PASSWORD is not in app-password
+    # form. createSession accepts main-account passwords too — but main
+    # passwords have no scope (full account access), can't be revoked
+    # individually, and rotating them breaks every service that holds them.
+    # We warn but do not gate, matching the project's detect-don't-block
+    # philosophy elsewhere.
+    if not _validate_app_password_format(app_password):
+        _log(
+            "BSKY_APP_PASSWORD does not look like an app password "
+            "(expected xxxx-xxxx-xxxx-xxxx, 19 chars). It may be a main "
+            "account password — those work but are bad hygiene. Generate "
+            "an app password at https://bsky.app/settings/app-passwords"
+        )
+
    count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
    core_topic = _extract_core_subject(topic)

@@ -155,7 +236,7 @@ def search_bluesky(
        "limit": str(min(count, 100)),
        "sort": "top",
    }
-    url = f"{BSKY_SEARCH_URL}?{urlencode(params)}"
+    url = f"{_resolve_search_url(config)}?{urlencode(params)}"

    def _auth_and_search() -> tuple[Optional[Dict[str, Any]], Optional[str]]:
        token = _create_session(handle, app_password)
@@ -314,6 +314,7 @@ def get_config() -> dict[str, Any]:
        ('LAST30DAYS_RERANK_MODEL', None),
        ('LAST30DAYS_X_MODEL', None),
        ('LAST30DAYS_X_BACKEND', None),
+        ('LAST30DAYS_STORE', None),
        ('OPENAI_MODEL_PIN', None),
        ('XAI_MODEL_PIN', None),
        ('SCRAPECREATORS_API_KEY', None),
@@ -322,6 +323,7 @@ def get_config() -> dict[str, Any]:
        ('CT0', None),
        ('BSKY_HANDLE', None),
        ('BSKY_APP_PASSWORD', None),
+        ('BSKY_SEARCH_HOST', None),
        ('TRUTHSOCIAL_TOKEN', None),
        ('BRAVE_API_KEY', None),
        ('EXA_API_KEY', None),
@@ -334,11 +336,22 @@ def get_config() -> dict[str, Any]:
        ('INCLUDE_SOURCES', ''),
        ('EXCLUDE_SOURCES', ''),
        ('LAST30DAYS_YOUTUBE_SSH_HOST', None),
+        ('LAST30DAYS_TRANSCRIPT_TIMEOUT', None),
    ]

    for key, default in keys:
        config[key] = os.environ.get(key) or merged_env.get(key, default)

+    # Backward-compat: ScrapeCreators' own examples and tutorials use the
+    # SCRAPE_CREATORS_API_KEY spelling (with underscore between SCRAPE and
+    # CREATORS). Accept that form too so users who follow the vendor's docs
+    # don't silently end up with has_scrapecreators=False. Canonical name
+    # wins when both are set.
+    if not config.get('SCRAPECREATORS_API_KEY'):
+        legacy = os.environ.get('SCRAPE_CREATORS_API_KEY') or merged_env.get('SCRAPE_CREATORS_API_KEY')
+        if legacy:
+            config['SCRAPECREATORS_API_KEY'] = legacy
+
    # Track which config source was used (highest-priority file source wins
    # the label; keychain is only reported when nothing else is configured).
    if project_env_path:
@@ -7,12 +7,14 @@ Requires SCRAPECREATORS_API_KEY in config. 100 free API calls, then PAYG.
 API docs: https://scrapecreators.com/docs
 """

+import os
 import re
 import sys
 from datetime import datetime
 from typing import Any, Dict, List, Optional, Set

 from . import dates, http, log
+from .relevance import token_overlap_relevance as _compute_relevance

 SCRAPECREATORS_BASE = "https://api.scrapecreators.com"

@@ -26,7 +28,42 @@ DEPTH_CONFIG = {
 # Max words to keep from each caption
 CAPTION_MAX_WORDS = 500

-from .relevance import token_overlap_relevance as _compute_relevance
+# Default transcript fetch timeout (seconds). SC's
+# /v2/instagram/media/transcript regularly takes >15s on real workloads,
+# so the default is generous; override via LAST30DAYS_TRANSCRIPT_TIMEOUT.
+DEFAULT_TRANSCRIPT_TIMEOUT = 30
+
+
+def _resolve_transcript_timeout(
+    timeout: Optional[float] = None,
+    config: Optional[Dict[str, Any]] = None,
+) -> float:
+    """Resolve the IG transcript-fetch timeout.
+
+    Priority (highest wins):
+      1. Explicit ``timeout`` kwarg
+      2. ``LAST30DAYS_TRANSCRIPT_TIMEOUT`` in os.environ
+      3. ``LAST30DAYS_TRANSCRIPT_TIMEOUT`` in caller-supplied config dict
+      4. ``DEFAULT_TRANSCRIPT_TIMEOUT`` (30s)
+
+    Mirrors the ``os.environ.get(X) or config.get(X)`` pattern used for
+    LAST30DAYS_STORE in last30days.py so the env var works whether it's
+    shell-exported or set in ~/.config/last30days/.env.
+    """
+    if timeout is not None:
+        try:
+            return float(timeout)
+        except (TypeError, ValueError):
+            pass
+    raw = os.environ.get("LAST30DAYS_TRANSCRIPT_TIMEOUT")
+    if not raw and config:
+        raw = config.get("LAST30DAYS_TRANSCRIPT_TIMEOUT")
+    if raw:
+        try:
+            return float(raw)
+        except (TypeError, ValueError):
+            pass
+    return float(DEFAULT_TRANSCRIPT_TIMEOUT)


 def _extract_core_subject(topic: str) -> str:
@@ -44,6 +81,17 @@ def _extract_core_subject(topic: str) -> str:
    return extract_core_subject(topic, noise=_INSTAGRAM_NOISE)


+def _to_hashtag_form(query: str) -> str:
+    """Collapse a multi-word query to hashtag form (no spaces, lowercase).
+
+    SC's /v2/instagram/reels/search wraps Google Search and is documented
+    to be flaky on multi-token queries. Single-token queries map to a
+    hashtag page lookup which is the stable path. Used as a 500-retry
+    fallback before the request bubbles up as a silent failure.
+    """
+    return ''.join(query.split()).lower()
+
+
 def _infer_query_intent(topic: str) -> str:
    """Tiny local intent classifier for Instagram query expansion."""
    text = topic.lower().strip()
@@ -283,6 +331,26 @@ def search_instagram(
            timeout=30,
            retries=2,
        )
+    except http.HTTPError as e:
+        # SC's v2 reels search wraps Google Search and 500s frequently on
+        # multi-token queries. Single tokens hit the stable hashtag-page
+        # path. Retry once with hashtag form before bubbling up.
+        if getattr(e, "status_code", None) == 500 and ' ' in core_topic:
+            _log(f"IG search 500 on '{core_topic}', retrying with hashtag form")
+            try:
+                data = http.get(
+                    f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search",
+                    params={"query": _to_hashtag_form(core_topic)},
+                    headers=http.scrapecreators_headers(token),
+                    timeout=30,
+                    retries=2,
+                )
+            except Exception as retry_e:
+                _log(f"IG search retry failed: {retry_e}")
+                return {"items": [], "error": f"{type(retry_e).__name__}: {retry_e}"}
+        else:
+            _log(f"ScrapeCreators error: {e}")
+            return {"items": [], "error": f"{type(e).__name__}: {e}"}
    except Exception as e:
        _log(f"ScrapeCreators error: {e}")
        return {"items": [], "error": f"{type(e).__name__}: {e}"}
@@ -317,6 +385,8 @@ def fetch_captions(
    video_items: List[Dict[str, Any]],
    token: str,
    depth: str = "default",
+    timeout: Optional[float] = None,
+    config: Optional[Dict[str, Any]] = None,
 ) -> Dict[str, str]:
    """Fetch transcripts for top N Instagram reels via ScrapeCreators.

@@ -328,12 +398,19 @@ def fetch_captions(
        video_items: Items from search_instagram()
        token: ScrapeCreators API key
        depth: Depth level for caption limit
+        timeout: Optional per-request transcript timeout in seconds. When
+            None, resolves from LAST30DAYS_TRANSCRIPT_TIMEOUT (env or
+            config), defaulting to DEFAULT_TRANSCRIPT_TIMEOUT (30s).
+        config: Optional config dict (from env.get_config()) used as a
+            fallback source for LAST30DAYS_TRANSCRIPT_TIMEOUT when the
+            value is not exported in os.environ.

    Returns:
        Dict mapping video_id -> caption text (truncated to 500 words)
    """
-    config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
-    max_captions = config["max_captions"]
+    depth_cfg = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
+    max_captions = depth_cfg["max_captions"]
+    transcript_timeout = _resolve_transcript_timeout(timeout, config)

    if not video_items or not token:
        return {}
@@ -364,7 +441,7 @@ def fetch_captions(
                f"{SCRAPECREATORS_BASE}/v2/instagram/media/transcript",
                params={"url": url},
                headers=http.scrapecreators_headers(token),
-                timeout=15,
+                timeout=transcript_timeout,
                retries=1,
            )
            transcripts = data.get("transcripts") or []
@@ -251,6 +251,11 @@ def _normalize_youtube(
    metadata: dict[str, Any] = {}
    if highlights:
        metadata["transcript_highlights"] = highlights
+    if item.get("captions_disabled"):
+        # Surfaced for quality_nudge: uploader disabled captions, so this
+        # video should be subtracted from the degraded-transcript-ratio
+        # denominator (it was never going to produce a transcript).
+        metadata["captions_disabled"] = True
    metadata["top_comments"] = _remap_comments(
        item.get("top_comments") or [],
        score_keys=("score", "likes"),
@@ -58,12 +58,38 @@ def _is_youtube_degraded(research_results: dict, threshold: float) -> bool:
    ratio is below threshold. The canonical cause is a stale yt-dlp binary -
    YouTube's caption format changes frequently and old binaries silently fail
    every transcript while the search itself still succeeds.
+
+    Captions-disabled videos are subtracted from the denominator: an uploader
+    who turned off captions can never produce a transcript, so counting that
+    video toward "fetch failures" produces false positives. A single
+    captions-disabled video in a small result set was tripping the nudge.
    """
    videos = int(research_results.get("youtube_videos_count") or 0)
    transcripts = int(research_results.get("youtube_transcripts_count") or 0)
+    captions_disabled = int(research_results.get("youtube_captions_disabled_count") or 0)
    if videos <= 0:
        return False
-    return (transcripts / videos) < threshold
+    eligible = videos - captions_disabled
+    if eligible <= 0:
+        # Every returned video had captions disabled - upstream content fact,
+        # not a yt-dlp problem. Don't flag.
+        return False
+    return (transcripts / eligible) < threshold
+
+
+def _is_instagram_silent_failure(config: dict, research_results: dict) -> bool:
+    """Instagram is silently failing when SC is configured but the source
+    returned zero items. The canonical cause is SC's v2 reels endpoint
+    500'ing on multi-token queries (it wraps Google Search and is documented
+    to be flaky there). Pre-fix the user got no signal at all - no Instagram
+    section in the brief, no error in the footer, just unexplained absence.
+    """
+    if not config.get("SCRAPECREATORS_API_KEY"):
+        return False  # not configured — not a silent failure
+    count = research_results.get("instagram_items_count")
+    if count is None:
+        return False  # source not run this invocation
+    return int(count) == 0


 def compute_quality_score(config: dict, research_results: dict) -> dict:
@@ -75,6 +101,8 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
            reddit_error reflecting what happened this run. Optional keys
            ``youtube_videos_count`` and ``youtube_transcripts_count`` enable
            degraded-YouTube detection (transcript-fetch ratio below threshold).
+            Optional key ``instagram_items_count`` enables silent-failure
+            detection for the bonus Instagram source.

    Returns:
        {
@@ -83,13 +111,15 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
            "core_missing": ["x", "youtube"],
            "core_errored": [],          # configured but errored at top level
            "core_degraded": [],         # configured and returned items but quality below threshold
-            "nudge_text": "..." or None if 100%
+            "bonus_errored": [],         # bonus sources (Instagram, etc.) configured but silent
+            "nudge_text": "..." or None if all sources healthy
        }
    """
    core_active: List[str] = []
    core_missing: List[str] = []
    core_errored: List[str] = []
    core_degraded: List[str] = []
+    bonus_errored: List[str] = []

    # HN, Polymarket, and Reddit are always active
    core_active.append("hn")
@@ -127,6 +157,11 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
        if has_ytdlp and research_results.get("youtube_error"):
            core_errored.append("youtube")

+    # Bonus sources (Instagram, etc.): SC-key holders expect content from
+    # these but until now the pipeline fell silent on configured-but-zero.
+    if _is_instagram_silent_failure(config, research_results):
+        bonus_errored.append("instagram")
+
    score_pct = int(len(core_active) / 5 * 100)

    has_sc = bool(config.get("SCRAPECREATORS_API_KEY"))
@@ -138,7 +173,8 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
        research_results,
        has_sc=has_sc,
        active_sources=active_sources,
-    ) if (core_missing or core_degraded) else None
+        bonus_errored=bonus_errored,
+    ) if (core_missing or core_degraded or bonus_errored) else None

    return {
        "score_pct": score_pct,
@@ -146,6 +182,7 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
        "core_missing": core_missing,
        "core_errored": core_errored,
        "core_degraded": core_degraded,
+        "bonus_errored": bonus_errored,
        "nudge_text": nudge_text,
    }

@@ -157,6 +194,7 @@ def _build_nudge_text(
    research_results: dict = None,
    has_sc: bool = False,
    active_sources: list = None,
+    bonus_errored: List[str] = None,
 ) -> str:
    """Build human-readable nudge text describing what was missed or degraded.

@@ -165,6 +203,7 @@ def _build_nudge_text(
    """
    lines: List[str] = []
    core_degraded = core_degraded or []
+    bonus_errored = bonus_errored or []
    research_results = research_results or {}

    # Describe what was missed
@@ -183,6 +222,9 @@ def _build_nudge_text(
    if core_degraded:
        degraded_labels = ", ".join(SOURCE_LABELS[s] for s in core_degraded)
        lines.append(f"Degraded: {degraded_labels}.")
+    if bonus_errored:
+        bonus_labels = ", ".join(s.capitalize() for s in bonus_errored)
+        lines.append(f"Bonus source silent: {bonus_labels}.")
    lines.append("")

    # Free suggestions
@@ -215,12 +257,30 @@ def _build_nudge_text(
    if "youtube" in core_degraded:
        videos = int(research_results.get("youtube_videos_count") or 0)
        transcripts = int(research_results.get("youtube_transcripts_count") or 0)
+        captions_disabled = int(research_results.get("youtube_captions_disabled_count") or 0)
+        captions_note = ""
+        if captions_disabled > 0:
+            captions_note = (
+                f" ({captions_disabled} of those had captions disabled by the "
+                "uploader, which is a separate cause and not fixable on your end)"
+            )
        free_suggestions.append(
            f"YouTube returned {videos} videos but only {transcripts} transcripts "
-            "captured. The most common cause is a stale yt-dlp binary - YouTube's "
-            "caption format changes frequently and old binaries silently fail every "
-            "transcript. Update via your package manager: scoop update yt-dlp "
-            "(Windows), brew upgrade yt-dlp (macOS), or pip install -U yt-dlp."
+            f"captured{captions_note}. The most common remaining cause is a stale "
+            "yt-dlp binary - YouTube's caption format changes frequently and old "
+            "binaries silently fail every transcript. Update via your package "
+            "manager: scoop update yt-dlp (Windows), brew upgrade yt-dlp (macOS), "
+            "or pip install -U yt-dlp."
+        )
+
+    if "instagram" in bonus_errored:
+        free_suggestions.append(
+            "Instagram returned 0 reels despite SC being configured. SC's "
+            "v2 reels endpoint wraps Google Search and 500's frequently on "
+            "multi-token queries. The skill now retries with hashtag-form "
+            "automatically; if zero items still appear, the topic may have "
+            "no reel coverage on Instagram. Try a single-word topic like "
+            "the most distinctive noun in your query."
        )

    # Mention bonus opt-in sources when SC key is present
@@ -385,7 +385,11 @@ def _clean_vtt(vtt_text: str) -> str:
 _YT_USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"


-def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
+def _fetch_transcript_direct(
+    video_id: str,
+    timeout: int = 30,
+    status: Optional[Dict[str, Any]] = None,
+) -> Optional[str]:
    """Fetch YouTube transcript via direct HTTP without yt-dlp.

    Scrapes the watch page HTML for the captions track URL in
@@ -394,6 +398,9 @@ def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
    Args:
        video_id: YouTube video ID
        timeout: HTTP request timeout in seconds
+        status: Optional dict mutated to record per-video signals. Sets
+            ``status["no_caption_tracks"] = True`` when the player response
+            confirms the uploader has no caption tracks (vs. fetch failure).

    Returns:
        Raw VTT text, or None if captions are unavailable.
@@ -442,6 +449,8 @@ def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:

    if not caption_tracks:
        _log(f"Direct transcript: no caption tracks for {video_id}")
+        if status is not None:
+            status["no_caption_tracks"] = True
        return None

    # Find English track (prefer exact 'en', then any en variant, then first track)
@@ -527,7 +536,11 @@ def _fetch_transcript_ytdlp(video_id: str, temp_dir: str) -> Optional[str]:
        return None


-def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
+def fetch_transcript(
+    video_id: str,
+    temp_dir: str,
+    status: Optional[Dict[str, Any]] = None,
+) -> Optional[str]:
    """Fetch auto-generated transcript for a YouTube video.

    Uses yt-dlp when available (preferred, more robust). Falls back to
@@ -536,6 +549,10 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
    Args:
        video_id: YouTube video ID
        temp_dir: Temporary directory for subtitle files
+        status: Optional dict mutated by the direct-HTTP path to record
+            per-video signals like ``no_caption_tracks``. Used to surface a
+            captions-disabled count so the quality nudge avoids false-positive
+            "stale yt-dlp" flags.

    Returns:
        Plaintext transcript string, or None if no captions available.
@@ -551,13 +568,13 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
        raw_vtt = _fetch_transcript_ytdlp(video_id, temp_dir)
        if not raw_vtt:
            _log(f"yt-dlp transcript failed for {video_id}, trying direct HTTP fallback")
-            raw_vtt = _fetch_transcript_direct(video_id)
+            raw_vtt = _fetch_transcript_direct(video_id, status=status)
    else:
        if ssh_host:
            _log("SSH-routing active, using direct HTTP transcript fetch")
        else:
            _log("yt-dlp not installed, using direct HTTP transcript fetch")
-        raw_vtt = _fetch_transcript_direct(video_id)
+        raw_vtt = _fetch_transcript_direct(video_id, status=status)

    if not raw_vtt:
        _log(f"No transcript available for {video_id} (no captions found)")
@@ -576,12 +593,16 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
 def fetch_transcripts_parallel(
    video_ids: List[str],
    max_workers: int = 5,
+    out_captions_disabled: Optional[Set[str]] = None,
 ) -> Dict[str, Optional[str]]:
    """Fetch transcripts for multiple videos in parallel.

    Args:
        video_ids: List of YouTube video IDs
        max_workers: Max parallel fetches
+        out_captions_disabled: Optional set mutated to record video_ids whose
+            uploader confirmed no caption tracks (vs. transient fetch failures).
+            Backward-compatible: callers that don't care can omit.

    Returns:
        Dict mapping video_id to transcript text (or None).
@@ -592,10 +613,11 @@ def fetch_transcripts_parallel(
    _log(f"Fetching transcripts for {len(video_ids)} videos")

    results = {}
+    statuses: Dict[str, Dict[str, Any]] = {vid: {} for vid in video_ids}
    with tempfile.TemporaryDirectory() as temp_dir:
        with ThreadPoolExecutor(max_workers=max_workers) as executor:
            futures = {
-                executor.submit(fetch_transcript, vid, temp_dir): vid
+                executor.submit(fetch_transcript, vid, temp_dir, statuses[vid]): vid
                for vid in video_ids
            }
            for future in as_completed(futures):
@@ -609,6 +631,11 @@ def fetch_transcripts_parallel(
                    _log(f"Unexpected transcript error for {vid}: {type(exc).__name__}: {exc}")
                    results[vid] = None

+    if out_captions_disabled is not None:
+        for vid, st in statuses.items():
+            if st.get("no_caption_tracks"):
+                out_captions_disabled.add(vid)
+
    got = sum(1 for v in results.values() if v)
    errors = sum(1 for v in results.values() if v is None)
    _log(f"Got transcripts for {got}/{len(video_ids)} videos ({errors} failed)")
@@ -659,15 +686,21 @@ def search_and_transcribe(
    # good chance of reaching the target number of successful transcripts.
    transcript_limit = TRANSCRIPT_LIMITS.get(depth, TRANSCRIPT_LIMITS["default"])
    transcripts: Dict[str, Optional[str]] = {}
+    captions_disabled_ids: Set[str] = set()
    if transcript_limit > 0:
        attempt_count = min(len(items), transcript_limit * 3)
        candidate_ids = [item["video_id"] for item in items[:attempt_count]]
        _log(f"Fetching transcripts for up to {attempt_count} videos (target: {transcript_limit}): {candidate_ids}")
-        transcripts = fetch_transcripts_parallel(candidate_ids)
+        transcripts = fetch_transcripts_parallel(
+            candidate_ids, out_captions_disabled=captions_disabled_ids,
+        )
    else:
        _log(f"Transcript limit is 0 for depth={depth}, skipping transcript fetch")

-    # Step 3: Attach transcripts and extract highlights
+    # Step 3: Attach transcripts and extract highlights. Mark captions_disabled
+    # so quality_nudge can subtract those videos from the degraded-ratio
+    # denominator (uploader-disabled captions can never produce a transcript;
+    # counting them was producing false-positive stale-yt-dlp nudges).
    core_topic = _extract_core_subject(topic)
    for item in items:
        vid = item["video_id"]
@@ -676,6 +709,7 @@ def search_and_transcribe(
        item["transcript_highlights"] = extract_transcript_highlights(
            transcript or "", core_topic,
        )
+        item["captions_disabled"] = vid in captions_disabled_ids

    return {"items": items}