sawz/last30days-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: configuration enablement — env-var defaults + source resilience

Browse Source 
Six small additive changes that make the skill correctly understand its
configured sources, plus tests + docs.

User-visible benefits

- LAST30DAYS_STORE=1 in .env turns persistence default-on without
  remembering --store on every invocation. Mirrors LAST30DAYS_DEBUG /
  LAST30DAYS_SKIP_PREFLIGHT convention.
- SCRAPE_CREATORS_API_KEY (with underscore) accepted as alias for the
  canonical name. Matches the spelling used in the vendor's own example
  code (Adrian Horning's repo); saves the next user the same diagnostic
  rabbit hole.
- Bluesky search now hits api.bsky.app (canonical AppView) instead of
  public.api.bsky.app (BunnyCDN-blocked public mirror as of 2026-05-04).
  BSKY_SEARCH_HOST env var lets users self-rescue future host migrations
  without a code release. Pre-fix: silent 0 Bluesky posts on every run.
- App-password format validator emits a one-shot stderr warning when
  BSKY_APP_PASSWORD doesn't match xxxx-xxxx-xxxx-xxxx form. Detect-don't-
  gate: createSession still accepts main passwords; the warning helps
  users identify a hygiene issue without breaking existing setups.
- Instagram retry on multi-token 500. SC's v2 reels endpoint wraps
  Google Search and 500's frequently on multi-word queries; a hashtag-
  form retry runs once before bubbling up. Documented vendor instability.
- LAST30DAYS_TRANSCRIPT_TIMEOUT env var (default 30s, was hardcoded 15s).
  SC's transcript endpoint regularly takes >15s; the old default was
  clipping legitimate responses.
- Silent-failure visibility: new bonus_errored field in the quality
  nudge fires when SC is configured but Instagram returned 0 items.
  Users see "Bonus source silent: Instagram" instead of unexplained
  absence.
- YouTube degraded-ratio false-positive fixed. Captions-disabled videos
  can never produce a transcript regardless of yt-dlp version; they're
  now subtracted from the denominator so a single uploader-disabled
  video doesn't false-trigger the "stale yt-dlp" nudge.
- urllib retry path: status_code attribute typo fix. The Instagram
  500-retry was dead code on the urllib branch (getattr(e, 'status', ...)
  while http.HTTPError exposes status_code).

Docs

- README.md: added /plugin install last30days step after marketplace add
  in three places (the install was previously omitted in the docs).
- CONFIGURATION.md: documented LAST30DAYS_STORE env var, added
  BSKY_SEARCH_HOST + app-password format section, mentioned
  LAST30DAYS_TRANSCRIPT_TIMEOUT in the Instagram source row.

Test plan

- 43 new unit tests across test_bluesky.py, test_instagram_sc.py,
  test_quality_nudge.py, test_youtube_yt.py
- 141 total tests passing in target suite
- Verified end-to-end: /last30days "Toronto resale condo market" with
  all 11+ sources active stored 35 new + 5 updated findings, all builder-
  PR-style accounts absent (organic agent voice in Instagram + TikTok
  results)

Backward compatibility

All changes are strictly additive. Optional kwargs default to None.
New env vars are opt-in. Existing CLI flags untouched. Existing callers
of public functions unaffected.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Daniel Zivkovic
2026-05-05 16:53:10 -04:00
committed by Trevin Chow
parent a8e462c978
commit 44971a6aae
14 changed files with 865 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/last30days/scripts/lib/youtube_yt.py
+41 -7
View File
@@ -385,7 +385,11 @@ def _clean_vtt(vtt_text: str) -> str:
_YT_USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
def _fetch_transcript_direct(
video_id: str,
timeout: int = 30,
status: Optional[Dict[str, Any]] = None,
) -> Optional[str]:
"""Fetch YouTube transcript via direct HTTP without yt-dlp.
Scrapes the watch page HTML for the captions track URL in
@@ -394,6 +398,9 @@ def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
Args:
video_id: YouTube video ID
timeout: HTTP request timeout in seconds
status: Optional dict mutated to record per-video signals. Sets
``status["no_caption_tracks"] = True`` when the player response
confirms the uploader has no caption tracks (vs. fetch failure).
Returns:
Raw VTT text, or None if captions are unavailable.
@@ -442,6 +449,8 @@ def _fetch_transcript_direct(video_id: str, timeout: int = 30) -> Optional[str]:
if not caption_tracks:
_log(f"Direct transcript: no caption tracks for {video_id}")
if status is not None:
status["no_caption_tracks"] = True
return None
# Find English track (prefer exact 'en', then any en variant, then first track)
@@ -527,7 +536,11 @@ def _fetch_transcript_ytdlp(video_id: str, temp_dir: str) -> Optional[str]:
return None
def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
def fetch_transcript(
video_id: str,
temp_dir: str,
status: Optional[Dict[str, Any]] = None,
) -> Optional[str]:
"""Fetch auto-generated transcript for a YouTube video.
Uses yt-dlp when available (preferred, more robust). Falls back to
@@ -536,6 +549,10 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
Args:
video_id: YouTube video ID
temp_dir: Temporary directory for subtitle files
status: Optional dict mutated by the direct-HTTP path to record
per-video signals like ``no_caption_tracks``. Used to surface a
captions-disabled count so the quality nudge avoids false-positive
"stale yt-dlp" flags.
Returns:
Plaintext transcript string, or None if no captions available.
@@ -551,13 +568,13 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
raw_vtt = _fetch_transcript_ytdlp(video_id, temp_dir)
if not raw_vtt:
_log(f"yt-dlp transcript failed for {video_id}, trying direct HTTP fallback")
raw_vtt = _fetch_transcript_direct(video_id)
raw_vtt = _fetch_transcript_direct(video_id, status=status)
else:
if ssh_host:
_log("SSH-routing active, using direct HTTP transcript fetch")
else:
_log("yt-dlp not installed, using direct HTTP transcript fetch")
raw_vtt = _fetch_transcript_direct(video_id)
raw_vtt = _fetch_transcript_direct(video_id, status=status)
if not raw_vtt:
_log(f"No transcript available for {video_id} (no captions found)")
@@ -576,12 +593,16 @@ def fetch_transcript(video_id: str, temp_dir: str) -> Optional[str]:
def fetch_transcripts_parallel(
video_ids: List[str],
max_workers: int = 5,
out_captions_disabled: Optional[Set[str]] = None,
) -> Dict[str, Optional[str]]:
"""Fetch transcripts for multiple videos in parallel.
Args:
video_ids: List of YouTube video IDs
max_workers: Max parallel fetches
out_captions_disabled: Optional set mutated to record video_ids whose
uploader confirmed no caption tracks (vs. transient fetch failures).
Backward-compatible: callers that don't care can omit.
Returns:
Dict mapping video_id to transcript text (or None).
@@ -592,10 +613,11 @@ def fetch_transcripts_parallel(
_log(f"Fetching transcripts for {len(video_ids)} videos")
results = {}
statuses: Dict[str, Dict[str, Any]] = {vid: {} for vid in video_ids}
with tempfile.TemporaryDirectory() as temp_dir:
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {
executor.submit(fetch_transcript, vid, temp_dir): vid
executor.submit(fetch_transcript, vid, temp_dir, statuses[vid]): vid
for vid in video_ids
}
for future in as_completed(futures):
@@ -609,6 +631,11 @@ def fetch_transcripts_parallel(
_log(f"Unexpected transcript error for {vid}: {type(exc).__name__}: {exc}")
results[vid] = None
if out_captions_disabled is not None:
for vid, st in statuses.items():
if st.get("no_caption_tracks"):
out_captions_disabled.add(vid)
got = sum(1 for v in results.values() if v)
errors = sum(1 for v in results.values() if v is None)
_log(f"Got transcripts for {got}/{len(video_ids)} videos ({errors} failed)")
@@ -659,15 +686,21 @@ def search_and_transcribe(
# good chance of reaching the target number of successful transcripts.
transcript_limit = TRANSCRIPT_LIMITS.get(depth, TRANSCRIPT_LIMITS["default"])
transcripts: Dict[str, Optional[str]] = {}
captions_disabled_ids: Set[str] = set()
if transcript_limit > 0:
attempt_count = min(len(items), transcript_limit * 3)
candidate_ids = [item["video_id"] for item in items[:attempt_count]]
_log(f"Fetching transcripts for up to {attempt_count} videos (target: {transcript_limit}): {candidate_ids}")
transcripts = fetch_transcripts_parallel(candidate_ids)
transcripts = fetch_transcripts_parallel(
candidate_ids, out_captions_disabled=captions_disabled_ids,
)
else:
_log(f"Transcript limit is 0 for depth={depth}, skipping transcript fetch")
# Step 3: Attach transcripts and extract highlights
# Step 3: Attach transcripts and extract highlights. Mark captions_disabled
# so quality_nudge can subtract those videos from the degraded-ratio
# denominator (uploader-disabled captions can never produce a transcript;
# counting them was producing false-positive stale-yt-dlp nudges).
core_topic = _extract_core_subject(topic)
for item in items:
vid = item["video_id"]
@@ -676,6 +709,7 @@ def search_and_transcribe(
item["transcript_highlights"] = extract_transcript_highlights(
transcript or "", core_topic,
)
item["captions_disabled"] = vid in captions_disabled_ids
return {"items": items}