feat: configuration enablement — env-var defaults + source resilience
Six small additive changes that make the skill correctly understand its configured sources, plus tests + docs. User-visible benefits - LAST30DAYS_STORE=1 in .env turns persistence default-on without remembering --store on every invocation. Mirrors LAST30DAYS_DEBUG / LAST30DAYS_SKIP_PREFLIGHT convention. - SCRAPE_CREATORS_API_KEY (with underscore) accepted as alias for the canonical name. Matches the spelling used in the vendor's own example code (Adrian Horning's repo); saves the next user the same diagnostic rabbit hole. - Bluesky search now hits api.bsky.app (canonical AppView) instead of public.api.bsky.app (BunnyCDN-blocked public mirror as of 2026-05-04). BSKY_SEARCH_HOST env var lets users self-rescue future host migrations without a code release. Pre-fix: silent 0 Bluesky posts on every run. - App-password format validator emits a one-shot stderr warning when BSKY_APP_PASSWORD doesn't match xxxx-xxxx-xxxx-xxxx form. Detect-don't- gate: createSession still accepts main passwords; the warning helps users identify a hygiene issue without breaking existing setups. - Instagram retry on multi-token 500. SC's v2 reels endpoint wraps Google Search and 500's frequently on multi-word queries; a hashtag- form retry runs once before bubbling up. Documented vendor instability. - LAST30DAYS_TRANSCRIPT_TIMEOUT env var (default 30s, was hardcoded 15s). SC's transcript endpoint regularly takes >15s; the old default was clipping legitimate responses. - Silent-failure visibility: new bonus_errored field in the quality nudge fires when SC is configured but Instagram returned 0 items. Users see "Bonus source silent: Instagram" instead of unexplained absence. - YouTube degraded-ratio false-positive fixed. Captions-disabled videos can never produce a transcript regardless of yt-dlp version; they're now subtracted from the denominator so a single uploader-disabled video doesn't false-trigger the "stale yt-dlp" nudge. - urllib retry path: status_code attribute typo fix. The Instagram 500-retry was dead code on the urllib branch (getattr(e, 'status', ...) while http.HTTPError exposes status_code). Docs - README.md: added /plugin install last30days step after marketplace add in three places (the install was previously omitted in the docs). - CONFIGURATION.md: documented LAST30DAYS_STORE env var, added BSKY_SEARCH_HOST + app-password format section, mentioned LAST30DAYS_TRANSCRIPT_TIMEOUT in the Instagram source row. Test plan - 43 new unit tests across test_bluesky.py, test_instagram_sc.py, test_quality_nudge.py, test_youtube_yt.py - 141 total tests passing in target suite - Verified end-to-end: /last30days "Toronto resale condo market" with all 11+ sources active stored 35 new + 5 updated findings, all builder- PR-style accounts absent (organic agent voice in Instagram + TikTok results) Backward compatibility All changes are strictly additive. Optional kwargs default to None. New env vars are opt-in. Existing CLI flags untouched. Existing callers of public functions unaffected. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Daniel Zivkovic
committed by
Trevin Chow
Trevin Chow
parent
a8e462c978
commit
44971a6aae
@@ -1,10 +1,19 @@
|
||||
"""Bluesky search via AT Protocol (requires app password).
|
||||
|
||||
Uses bsky.social for auth and public.api.bsky.app for post search.
|
||||
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars.
|
||||
Uses bsky.social for auth and api.bsky.app for post search (the canonical
|
||||
authenticated AppView). The previous default `public.api.bsky.app` is the
|
||||
unauthenticated public mirror, which BunnyCDN now blocks for searchPosts
|
||||
regardless of auth header (verified 2026-05-04). Override the search host
|
||||
via BSKY_SEARCH_HOST env var if Bluesky migrates infrastructure again.
|
||||
|
||||
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars. App passwords are
|
||||
19-char xxxx-xxxx-xxxx-xxxx; generate at bsky.app/settings/app-passwords.
|
||||
The createSession endpoint accepts main-account passwords too, but they're
|
||||
bad hygiene (no scope, can't revoke individually).
|
||||
"""
|
||||
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
@@ -14,7 +23,65 @@ from typing import Any, Dict, List, Optional
|
||||
from . import http, log
|
||||
|
||||
BSKY_SESSION_URL = "https://bsky.social/xrpc/com.atproto.server.createSession"
|
||||
BSKY_SEARCH_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts"
|
||||
_DEFAULT_BSKY_SEARCH_HOST = "api.bsky.app"
|
||||
BSKY_SEARCH_URL = f"https://{_DEFAULT_BSKY_SEARCH_HOST}/xrpc/app.bsky.feed.searchPosts"
|
||||
|
||||
|
||||
def _resolve_search_url(config: Optional[Dict[str, Any]] = None) -> str:
|
||||
"""Resolve the Bluesky search URL with BSKY_SEARCH_HOST override.
|
||||
|
||||
Default is api.bsky.app. Override via BSKY_SEARCH_HOST in shell env or
|
||||
.env file. The project's env.py loads .env into config but not into
|
||||
os.environ, so check both — same hybrid pattern as last30days.py for
|
||||
LAST30DAYS_STORE.
|
||||
|
||||
Hardens user-supplied host values against three common mis-configurations:
|
||||
whitespace (e.g. " api.bsky.app "), embedded path components (e.g.
|
||||
"api.bsky.app/xrpc/proxy") that would double the /xrpc/ segment, and
|
||||
embedded scheme prefixes (e.g. "https://api.bsky.app"). On any of these
|
||||
we log a warning and fall back to the default rather than building an
|
||||
invalid URL with an opaque downstream error.
|
||||
"""
|
||||
config = config or {}
|
||||
raw = (
|
||||
os.environ.get("BSKY_SEARCH_HOST")
|
||||
or config.get("BSKY_SEARCH_HOST")
|
||||
or _DEFAULT_BSKY_SEARCH_HOST
|
||||
)
|
||||
host = raw.strip().rstrip("/")
|
||||
# Strip embedded scheme so users who paste full URLs do not break the f-string.
|
||||
for prefix in ("https://", "http://"):
|
||||
if host.lower().startswith(prefix):
|
||||
host = host[len(prefix):]
|
||||
break
|
||||
if not host or "/" in host or " " in host:
|
||||
# Embedded path or whitespace remains — don't trust it. Default + log.
|
||||
if raw != _DEFAULT_BSKY_SEARCH_HOST:
|
||||
_log(
|
||||
f"BSKY_SEARCH_HOST={raw!r} is not a bare hostname; "
|
||||
f"falling back to default {_DEFAULT_BSKY_SEARCH_HOST!r}"
|
||||
)
|
||||
host = _DEFAULT_BSKY_SEARCH_HOST
|
||||
return f"https://{host}/xrpc/app.bsky.feed.searchPosts"
|
||||
|
||||
|
||||
# App-password format: xxxx-xxxx-xxxx-xxxx (19 chars, lowercase alphanumeric
|
||||
# with three hyphens at fixed positions).
|
||||
_APP_PASSWORD_RE = re.compile(r"^[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}$")
|
||||
|
||||
|
||||
def _validate_app_password_format(value) -> bool:
|
||||
"""Return True if value matches Bluesky's 19-char app-password format.
|
||||
|
||||
False for non-strings (None, int, list) so callers passing config dict
|
||||
values directly don't crash. Detect-but-not-gate: the createSession
|
||||
endpoint also accepts main-account passwords, so failing this check is
|
||||
a hygiene smell, not a hard error.
|
||||
"""
|
||||
if not isinstance(value, str):
|
||||
return False
|
||||
return bool(_APP_PASSWORD_RE.fullmatch(value))
|
||||
|
||||
|
||||
DEPTH_CONFIG = {
|
||||
"quick": 15,
|
||||
@@ -144,6 +211,20 @@ def search_bluesky(
|
||||
if not handle or not app_password:
|
||||
return {"posts": [], "error": "Bluesky credentials not configured"}
|
||||
|
||||
# One-shot hygiene warning if BSKY_APP_PASSWORD is not in app-password
|
||||
# form. createSession accepts main-account passwords too — but main
|
||||
# passwords have no scope (full account access), can't be revoked
|
||||
# individually, and rotating them breaks every service that holds them.
|
||||
# We warn but do not gate, matching the project's detect-don't-block
|
||||
# philosophy elsewhere.
|
||||
if not _validate_app_password_format(app_password):
|
||||
_log(
|
||||
"BSKY_APP_PASSWORD does not look like an app password "
|
||||
"(expected xxxx-xxxx-xxxx-xxxx, 19 chars). It may be a main "
|
||||
"account password — those work but are bad hygiene. Generate "
|
||||
"an app password at https://bsky.app/settings/app-passwords"
|
||||
)
|
||||
|
||||
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
|
||||
core_topic = _extract_core_subject(topic)
|
||||
|
||||
@@ -155,7 +236,7 @@ def search_bluesky(
|
||||
"limit": str(min(count, 100)),
|
||||
"sort": "top",
|
||||
}
|
||||
url = f"{BSKY_SEARCH_URL}?{urlencode(params)}"
|
||||
url = f"{_resolve_search_url(config)}?{urlencode(params)}"
|
||||
|
||||
def _auth_and_search() -> tuple[Optional[Dict[str, Any]], Optional[str]]:
|
||||
token = _create_session(handle, app_password)
|
||||
|
||||
@@ -314,6 +314,7 @@ def get_config() -> dict[str, Any]:
|
||||
('LAST30DAYS_RERANK_MODEL', None),
|
||||
('LAST30DAYS_X_MODEL', None),
|
||||
('LAST30DAYS_X_BACKEND', None),
|
||||
('LAST30DAYS_STORE', None),
|
||||
('OPENAI_MODEL_PIN', None),
|
||||
('XAI_MODEL_PIN', None),
|
||||
('SCRAPECREATORS_API_KEY', None),
|
||||
@@ -322,6 +323,7 @@ def get_config() -> dict[str, Any]:
|
||||
('CT0', None),
|
||||
('BSKY_HANDLE', None),
|
||||
('BSKY_APP_PASSWORD', None),
|
||||
('BSKY_SEARCH_HOST', None),
|
||||
('TRUTHSOCIAL_TOKEN', None),
|
||||
('BRAVE_API_KEY', None),
|
||||
('EXA_API_KEY', None),
|
||||
@@ -334,11 +336,22 @@ def get_config() -> dict[str, Any]:
|
||||
('INCLUDE_SOURCES', ''),
|
||||
('EXCLUDE_SOURCES', ''),
|
||||
('LAST30DAYS_YOUTUBE_SSH_HOST', None),
|
||||
('LAST30DAYS_TRANSCRIPT_TIMEOUT', None),
|
||||
]
|
||||
|
||||
for key, default in keys:
|
||||
config[key] = os.environ.get(key) or merged_env.get(key, default)
|
||||
|
||||
# Backward-compat: ScrapeCreators' own examples and tutorials use the
|
||||
# SCRAPE_CREATORS_API_KEY spelling (with underscore between SCRAPE and
|
||||
# CREATORS). Accept that form too so users who follow the vendor's docs
|
||||
# don't silently end up with has_scrapecreators=False. Canonical name
|
||||
# wins when both are set.
|
||||
if not config.get('SCRAPECREATORS_API_KEY'):
|
||||
legacy = os.environ.get('SCRAPE_CREATORS_API_KEY') or merged_env.get('SCRAPE_CREATORS_API_KEY')
|
||||
if legacy:
|
||||
config['SCRAPECREATORS_API_KEY'] = legacy
|
||||
|
||||
# Track which config source was used (highest-priority file source wins
|
||||
# the label; keychain is only reported when nothing else is configured).
|
||||
if project_env_path:
|
||||
|
||||
@@ -7,12 +7,14 @@ Requires SCRAPECREATORS_API_KEY in config. 100 free API calls, then PAYG.
|
||||
API docs: https://scrapecreators.com/docs
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional, Set
|
||||
|
||||
from . import dates, http, log
|
||||
from .relevance import token_overlap_relevance as _compute_relevance
|
||||
|
||||
SCRAPECREATORS_BASE = "https://api.scrapecreators.com"
|
||||
|
||||
@@ -26,7 +28,42 @@ DEPTH_CONFIG = {
|
||||
# Max words to keep from each caption
|
||||
CAPTION_MAX_WORDS = 500
|
||||
|
||||
from .relevance import token_overlap_relevance as _compute_relevance
|
||||
# Default transcript fetch timeout (seconds). SC's
|
||||
# /v2/instagram/media/transcript regularly takes >15s on real workloads,
|
||||
# so the default is generous; override via LAST30DAYS_TRANSCRIPT_TIMEOUT.
|
||||
DEFAULT_TRANSCRIPT_TIMEOUT = 30
|
||||
|
||||
|
||||
def _resolve_transcript_timeout(
|
||||
timeout: Optional[float] = None,
|
||||
config: Optional[Dict[str, Any]] = None,
|
||||
) -> float:
|
||||
"""Resolve the IG transcript-fetch timeout.
|
||||
|
||||
Priority (highest wins):
|
||||
1. Explicit ``timeout`` kwarg
|
||||
2. ``LAST30DAYS_TRANSCRIPT_TIMEOUT`` in os.environ
|
||||
3. ``LAST30DAYS_TRANSCRIPT_TIMEOUT`` in caller-supplied config dict
|
||||
4. ``DEFAULT_TRANSCRIPT_TIMEOUT`` (30s)
|
||||
|
||||
Mirrors the ``os.environ.get(X) or config.get(X)`` pattern used for
|
||||
LAST30DAYS_STORE in last30days.py so the env var works whether it's
|
||||
shell-exported or set in ~/.config/last30days/.env.
|
||||
"""
|
||||
if timeout is not None:
|
||||
try:
|
||||
return float(timeout)
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
raw = os.environ.get("LAST30DAYS_TRANSCRIPT_TIMEOUT")
|
||||
if not raw and config:
|
||||
raw = config.get("LAST30DAYS_TRANSCRIPT_TIMEOUT")
|
||||
if raw:
|
||||
try:
|
||||
return float(raw)
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return float(DEFAULT_TRANSCRIPT_TIMEOUT)
|
||||
|
||||
|
||||
def _extract_core_subject(topic: str) -> str:
|
||||
@@ -44,6 +81,17 @@ def _extract_core_subject(topic: str) -> str:
|
||||
return extract_core_subject(topic, noise=_INSTAGRAM_NOISE)
|
||||
|
||||
|
||||
def _to_hashtag_form(query: str) -> str:
|
||||
"""Collapse a multi-word query to hashtag form (no spaces, lowercase).
|
||||
|
||||
SC's /v2/instagram/reels/search wraps Google Search and is documented
|
||||
to be flaky on multi-token queries. Single-token queries map to a
|
||||
hashtag page lookup which is the stable path. Used as a 500-retry
|
||||
fallback before the request bubbles up as a silent failure.
|
||||
"""
|
||||
return ''.join(query.split()).lower()
|
||||
|
||||
|
||||
def _infer_query_intent(topic: str) -> str:
|
||||
"""Tiny local intent classifier for Instagram query expansion."""
|
||||
text = topic.lower().strip()
|
||||
@@ -283,6 +331,26 @@ def search_instagram(
|
||||
timeout=30,
|
||||
retries=2,
|
||||
)
|
||||
except http.HTTPError as e:
|
||||
# SC's v2 reels search wraps Google Search and 500s frequently on
|
||||
# multi-token queries. Single tokens hit the stable hashtag-page
|
||||
# path. Retry once with hashtag form before bubbling up.
|
||||
if getattr(e, "status_code", None) == 500 and ' ' in core_topic:
|
||||
_log(f"IG search 500 on '{core_topic}', retrying with hashtag form")
|
||||
try:
|
||||
data = http.get(
|
||||
f"{SCRAPECREATORS_BASE}/v2/instagram/reels/search",
|
||||
params={"query": _to_hashtag_form(core_topic)},
|
||||
headers=http.scrapecreators_headers(token),
|
||||
timeout=30,
|
||||
retries=2,
|
||||
)
|
||||
except Exception as retry_e:
|
||||
_log(f"IG search retry failed: {retry_e}")
|
||||
return {"items": [], "error": f"{type(retry_e).__name__}: {retry_e}"}
|
||||
else:
|
||||
_log(f"ScrapeCreators error: {e}")
|
||||
return {"items": [], "error": f"{type(e).__name__}: {e}"}
|
||||
except Exception as e:
|
||||
_log(f"ScrapeCreators error: {e}")
|
||||
return {"items": [], "error": f"{type(e).__name__}: {e}"}
|
||||
@@ -317,6 +385,8 @@ def fetch_captions(
|
||||
video_items: List[Dict[str, Any]],
|
||||
token: str,
|
||||
depth: str = "default",
|
||||
timeout: Optional[float] = None,
|
||||
config: Optional[Dict[str, Any]] = None,
|
||||
) -> Dict[str, str]:
|
||||
"""Fetch transcripts for top N Instagram reels via ScrapeCreators.
|
||||
|
||||
@@ -328,12 +398,19 @@ def fetch_captions(
|
||||
video_items: Items from search_instagram()
|
||||
token: ScrapeCreators API key
|
||||
depth: Depth level for caption limit
|
||||
timeout: Optional per-request transcript timeout in seconds. When
|
||||
None, resolves from LAST30DAYS_TRANSCRIPT_TIMEOUT (env or
|
||||
config), defaulting to DEFAULT_TRANSCRIPT_TIMEOUT (30s).
|
||||
config: Optional config dict (from env.get_config()) used as a
|
||||
fallback source for LAST30DAYS_TRANSCRIPT_TIMEOUT when the
|
||||
value is not exported in os.environ.
|
||||
|
||||
Returns:
|
||||
Dict mapping video_id -> caption text (truncated to 500 words)
|
||||
"""
|
||||
config = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
|
||||
max_captions = config["max_captions"]
|
||||
depth_cfg = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
|
||||
max_captions = depth_cfg["max_captions"]
|
||||
transcript_timeout = _resolve_transcript_timeout(timeout, config)
|
||||
|
||||
if not video_items or not token:
|
||||
return {}
|
||||
@@ -364,7 +441,7 @@ def fetch_captions(
|
||||
f"{SCRAPECREATORS_BASE}/v2/instagram/media/transcript",
|
||||
params={"url": url},
|
||||
headers=http.scrapecreators_headers(token),
|
||||
timeout=15,
|
||||
timeout=transcript_timeout,
|
||||
retries=1,
|
||||
)
|
||||
transcripts = data.get("transcripts") or []
|
||||
|
||||
@@ -251,6 +251,11 @@ def _normalize_youtube(
|
||||
metadata: dict[str, Any] = {}
|
||||
if highlights:
|
||||
metadata["transcript_highlights"] = highlights
|
||||
if item.get("captions_disabled"):
|
||||
# Surfaced for quality_nudge: uploader disabled captions, so this
|
||||
# video should be subtracted from the degraded-transcript-ratio
|
||||
# denominator (it was never going to produce a transcript).
|
||||
metadata["captions_disabled"] = True
|
||||
metadata["top_comments"] = _remap_comments(
|
||||
item.get("top_comments") or [],
|
||||
score_keys=("score", "likes"),
|
||||
|
||||
@@ -58,12 +58,38 @@ def _is_youtube_degraded(research_results: dict, threshold: float) -> bool:
|
||||
ratio is below threshold. The canonical cause is a stale yt-dlp binary -
|
||||
YouTube's caption format changes frequently and old binaries silently fail
|
||||
every transcript while the search itself still succeeds.
|
||||
|
||||
Captions-disabled videos are subtracted from the denominator: an uploader
|
||||
who turned off captions can never produce a transcript, so counting that
|
||||
video toward "fetch failures" produces false positives. A single
|
||||
captions-disabled video in a small result set was tripping the nudge.
|
||||
"""
|
||||
videos = int(research_results.get("youtube_videos_count") or 0)
|
||||
transcripts = int(research_results.get("youtube_transcripts_count") or 0)
|
||||
captions_disabled = int(research_results.get("youtube_captions_disabled_count") or 0)
|
||||
if videos <= 0:
|
||||
return False
|
||||
return (transcripts / videos) < threshold
|
||||
eligible = videos - captions_disabled
|
||||
if eligible <= 0:
|
||||
# Every returned video had captions disabled - upstream content fact,
|
||||
# not a yt-dlp problem. Don't flag.
|
||||
return False
|
||||
return (transcripts / eligible) < threshold
|
||||
|
||||
|
||||
def _is_instagram_silent_failure(config: dict, research_results: dict) -> bool:
|
||||
"""Instagram is silently failing when SC is configured but the source
|
||||
returned zero items. The canonical cause is SC's v2 reels endpoint
|
||||
500'ing on multi-token queries (it wraps Google Search and is documented
|
||||
to be flaky there). Pre-fix the user got no signal at all - no Instagram
|
||||
section in the brief, no error in the footer, just unexplained absence.
|
||||
"""
|
||||
if not config.get("SCRAPECREATORS_API_KEY"):
|
||||
return False # not configured — not a silent failure
|
||||
count = research_results.get("instagram_items_count")
|
||||
if count is None:
|
||||
return False # source not run this invocation
|
||||
return int(count) == 0
|
||||
|
||||
|
||||
def compute_quality_score(config: dict, research_results: dict) -> dict:
|
||||
@@ -75,6 +101,8 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
|
||||
reddit_error reflecting what happened this run. Optional keys
|
||||
``youtube_videos_count`` and ``youtube_transcripts_count`` enable
|
||||
degraded-YouTube detection (transcript-fetch ratio below threshold).
|
||||
Optional key ``instagram_items_count`` enables silent-failure
|
||||
detection for the bonus Instagram source.
|
||||
|
||||
Returns:
|
||||
{
|
||||
@@ -83,13 +111,15 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
|
||||
"core_missing": ["x", "youtube"],
|
||||
"core_errored": [], # configured but errored at top level
|
||||
"core_degraded": [], # configured and returned items but quality below threshold
|
||||
"nudge_text": "..." or None if 100%
|
||||
"bonus_errored": [], # bonus sources (Instagram, etc.) configured but silent
|
||||
"nudge_text": "..." or None if all sources healthy
|
||||
}
|
||||
"""
|
||||
core_active: List[str] = []
|
||||
core_missing: List[str] = []
|
||||
core_errored: List[str] = []
|
||||
core_degraded: List[str] = []
|
||||
bonus_errored: List[str] = []
|
||||
|
||||
# HN, Polymarket, and Reddit are always active
|
||||
core_active.append("hn")
|
||||
@@ -127,6 +157,11 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
|
||||
if has_ytdlp and research_results.get("youtube_error"):
|
||||
core_errored.append("youtube")
|
||||
|
||||
# Bonus sources (Instagram, etc.): SC-key holders expect content from
|
||||
# these but until now the pipeline fell silent on configured-but-zero.
|
||||
if _is_instagram_silent_failure(config, research_results):
|
||||
bonus_errored.append("instagram")
|
||||
|
||||
score_pct = int(len(core_active) / 5 * 100)
|
||||
|
||||
has_sc = bool(config.get("SCRAPECREATORS_API_KEY"))
|
||||
@@ -138,7 +173,8 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
|
||||
research_results,
|
||||
has_sc=has_sc,
|
||||
active_sources=active_sources,
|
||||
) if (core_missing or core_degraded) else None
|
||||
bonus_errored=bonus_errored,
|
||||
) if (core_missing or core_degraded or bonus_errored) else None
|
||||
|
||||
return {
|
||||
"score_pct": score_pct,
|
||||
@@ -146,6 +182,7 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
|
||||
"core_missing": core_missing,
|
||||
"core_errored": core_errored,
|
||||
"core_degraded": core_degraded,
|
||||
"bonus_errored": bonus_errored,
|
||||
"nudge_text": nudge_text,
|
||||
}
|
||||
|
||||
@@ -157,6 +194,7 @@ def _build_nudge_text(
|
||||
research_results: dict = None,
|
||||
has_sc: bool = False,
|
||||
active_sources: list = None,
|
||||
bonus_errored: List[str] = None,
|
||||
) -> str:
|
||||
"""Build human-readable nudge text describing what was missed or degraded.
|
||||
|
||||
@@ -165,6 +203,7 @@ def _build_nudge_text(
|
||||
"""
|
||||
lines: List[str] = []
|
||||
core_degraded = core_degraded or []
|
||||
bonus_errored = bonus_errored or []
|
||||
research_results = research_results or {}
|
||||
|
||||
# Describe what was missed
|
||||
@@ -183,6 +222,9 @@ def _build_nudge_text(
|
||||
if core_degraded:
|
||||
degraded_labels = ", ".join(SOURCE_LABELS[s] for s in core_degraded)
|
||||
lines.append(f"Degraded: {degraded_labels}.")
|
||||
if bonus_errored:
|
||||
bonus_labels = ", ".join(s.capitalize() for s in bonus_errored)
|
||||
lines.append(f"Bonus source silent: {bonus_labels}.")
|
||||
lines.append("")
|
||||
|
||||
# Free suggestions
|
||||
@@ -215,12 +257,30 @@ def _build_nudge_text(
|
||||
if "youtube" in core_degraded:
|
||||
videos = int(research_results.get("youtube_videos_count") or 0)
|
||||
transcripts = int(research_results.get("youtube_transcripts_count") or 0)
|
||||
captions_disabled = int(research_results.get("youtube_captions_disabled_count") or 0)
|
||||
captions_note = ""
|
||||
if captions_disabled > 0:
|
||||
captions_note = (
|
||||
f" ({captions_disabled} of those had captions disabled by the "
|
||||
"uploader, which is a separate cause and not fixable on your end)"
|
||||
)
|
||||
free_suggestions.append(
|
||||
f"YouTube returned {videos} videos but only {transcripts} transcripts "
|
||||
"captured. The most common cause is a stale yt-dlp binary - YouTube's "
|
||||
"caption format changes frequently and old binaries silently fail every "
|
||||
"transcript. Update via your package manager: scoop update yt-dlp "
|
||||
"(Windows), brew upgrade yt-dlp (macOS), or pip install -U yt-dlp."
|
||||
f"captured{captions_note}. The most common remaining cause is a stale "
|
||||
"yt-dlp binary - YouTube's caption format changes frequently and old "
|
||||
"binaries silently fail every transcript. Update via your package "
|
||||
"manager: scoop update yt-dlp (Windows), brew upgrade yt-dlp (macOS), "
|
||||
"or pip install -U yt-dlp."
|
||||
)
|
||||
|
||||
if "instagram" in bonus_errored:
|
||||
free_suggestions.append(
|
||||
"Instagram returned 0 reels despite SC being configured. SC's "
|
||||
"v2 reels endpoint wraps Google Search and 500's frequently on "
|
||||
"multi-token queries. The skill now retries with hashtag-form "
|
||||
"automatically; if zero items still appear, the topic may have "
|
||||
"no reel coverage on Instagram. Try a single-word topic like "
|
||||
"the most distinctive noun in your query."
|
||||
)
|
||||
|
||||
# Mention bonus opt-in sources when SC key is present
|
||||
|
||||
@@ -385,7 +385,11 @@ def _clean_vtt(vtt_text: str) -> str:
|
||||
_YT_USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
|
||||
|
||||
|
||||
def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
|
||||
def _fetch_transcript_direct(
|
||||
video_id: str,
|
||||
timeout: int = 30,
|
||||
status: Optional[Dict[str, Any]] = None,
|
||||
) -> Optional[str]:
|
||||
"""Fetch YouTube transcript via direct HTTP without yt-dlp.
|
||||
|
||||
Scrapes the watch page HTML for the captions track URL in
|
||||
@@ -394,6 +398,9 @@ def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
|
||||
Args:
|
||||
video_id: YouTube video ID
|
||||
timeout: HTTP request timeout in seconds
|
||||
status: Optional dict mutated to record per-video signals. Sets
|
||||
``status["no_caption_tracks"] = True`` when the player response
|
||||
confirms the uploader has no caption tracks (vs. fetch failure).
|
||||
|
||||
Returns:
|
||||
Raw VTT text, or None if captions are unavailable.
|
||||
@@ -442,6 +449,8 @@ def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
|
||||
|
||||
if not caption_tracks:
|
||||
_log(f"Direct transcript: no caption tracks for {video_id}")
|
||||
if status is not None:
|
||||
status["no_caption_tracks"] = True
|
||||
return None
|
||||
|
||||
# Find English track (prefer exact 'en', then any en variant, then first track)
|
||||
@@ -527,7 +536,11 @@ def _fetch_transcript_ytdlp(video_id: str, temp_dir: str) -> Optional[str]:
|
||||
return None
|
||||
|
||||
|
||||
def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
|
||||
def fetch_transcript(
|
||||
video_id: str,
|
||||
temp_dir: str,
|
||||
status: Optional[Dict[str, Any]] = None,
|
||||
) -> Optional[str]:
|
||||
"""Fetch auto-generated transcript for a YouTube video.
|
||||
|
||||
Uses yt-dlp when available (preferred, more robust). Falls back to
|
||||
@@ -536,6 +549,10 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
|
||||
Args:
|
||||
video_id: YouTube video ID
|
||||
temp_dir: Temporary directory for subtitle files
|
||||
status: Optional dict mutated by the direct-HTTP path to record
|
||||
per-video signals like ``no_caption_tracks``. Used to surface a
|
||||
captions-disabled count so the quality nudge avoids false-positive
|
||||
"stale yt-dlp" flags.
|
||||
|
||||
Returns:
|
||||
Plaintext transcript string, or None if no captions available.
|
||||
@@ -551,13 +568,13 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
|
||||
raw_vtt = _fetch_transcript_ytdlp(video_id, temp_dir)
|
||||
if not raw_vtt:
|
||||
_log(f"yt-dlp transcript failed for {video_id}, trying direct HTTP fallback")
|
||||
raw_vtt = _fetch_transcript_direct(video_id)
|
||||
raw_vtt = _fetch_transcript_direct(video_id, status=status)
|
||||
else:
|
||||
if ssh_host:
|
||||
_log("SSH-routing active, using direct HTTP transcript fetch")
|
||||
else:
|
||||
_log("yt-dlp not installed, using direct HTTP transcript fetch")
|
||||
raw_vtt = _fetch_transcript_direct(video_id)
|
||||
raw_vtt = _fetch_transcript_direct(video_id, status=status)
|
||||
|
||||
if not raw_vtt:
|
||||
_log(f"No transcript available for {video_id} (no captions found)")
|
||||
@@ -576,12 +593,16 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
|
||||
def fetch_transcripts_parallel(
|
||||
video_ids: List[str],
|
||||
max_workers: int = 5,
|
||||
out_captions_disabled: Optional[Set[str]] = None,
|
||||
) -> Dict[str, Optional[str]]:
|
||||
"""Fetch transcripts for multiple videos in parallel.
|
||||
|
||||
Args:
|
||||
video_ids: List of YouTube video IDs
|
||||
max_workers: Max parallel fetches
|
||||
out_captions_disabled: Optional set mutated to record video_ids whose
|
||||
uploader confirmed no caption tracks (vs. transient fetch failures).
|
||||
Backward-compatible: callers that don't care can omit.
|
||||
|
||||
Returns:
|
||||
Dict mapping video_id to transcript text (or None).
|
||||
@@ -592,10 +613,11 @@ def fetch_transcripts_parallel(
|
||||
_log(f"Fetching transcripts for {len(video_ids)} videos")
|
||||
|
||||
results = {}
|
||||
statuses: Dict[str, Dict[str, Any]] = {vid: {} for vid in video_ids}
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
futures = {
|
||||
executor.submit(fetch_transcript, vid, temp_dir): vid
|
||||
executor.submit(fetch_transcript, vid, temp_dir, statuses[vid]): vid
|
||||
for vid in video_ids
|
||||
}
|
||||
for future in as_completed(futures):
|
||||
@@ -609,6 +631,11 @@ def fetch_transcripts_parallel(
|
||||
_log(f"Unexpected transcript error for {vid}: {type(exc).__name__}: {exc}")
|
||||
results[vid] = None
|
||||
|
||||
if out_captions_disabled is not None:
|
||||
for vid, st in statuses.items():
|
||||
if st.get("no_caption_tracks"):
|
||||
out_captions_disabled.add(vid)
|
||||
|
||||
got = sum(1 for v in results.values() if v)
|
||||
errors = sum(1 for v in results.values() if v is None)
|
||||
_log(f"Got transcripts for {got}/{len(video_ids)} videos ({errors} failed)")
|
||||
@@ -659,15 +686,21 @@ def search_and_transcribe(
|
||||
# good chance of reaching the target number of successful transcripts.
|
||||
transcript_limit = TRANSCRIPT_LIMITS.get(depth, TRANSCRIPT_LIMITS["default"])
|
||||
transcripts: Dict[str, Optional[str]] = {}
|
||||
captions_disabled_ids: Set[str] = set()
|
||||
if transcript_limit > 0:
|
||||
attempt_count = min(len(items), transcript_limit * 3)
|
||||
candidate_ids = [item["video_id"] for item in items[:attempt_count]]
|
||||
_log(f"Fetching transcripts for up to {attempt_count} videos (target: {transcript_limit}): {candidate_ids}")
|
||||
transcripts = fetch_transcripts_parallel(candidate_ids)
|
||||
transcripts = fetch_transcripts_parallel(
|
||||
candidate_ids, out_captions_disabled=captions_disabled_ids,
|
||||
)
|
||||
else:
|
||||
_log(f"Transcript limit is 0 for depth={depth}, skipping transcript fetch")
|
||||
|
||||
# Step 3: Attach transcripts and extract highlights
|
||||
# Step 3: Attach transcripts and extract highlights. Mark captions_disabled
|
||||
# so quality_nudge can subtract those videos from the degraded-ratio
|
||||
# denominator (uploader-disabled captions can never produce a transcript;
|
||||
# counting them was producing false-positive stale-yt-dlp nudges).
|
||||
core_topic = _extract_core_subject(topic)
|
||||
for item in items:
|
||||
vid = item["video_id"]
|
||||
@@ -676,6 +709,7 @@ def search_and_transcribe(
|
||||
item["transcript_highlights"] = extract_transcript_highlights(
|
||||
transcript or "", core_topic,
|
||||
)
|
||||
item["captions_disabled"] = vid in captions_disabled_ids
|
||||
|
||||
return {"items": items}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user