sawz/last30days-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: configuration enablement — env-var defaults + source resilience

Browse Source 
Six small additive changes that make the skill correctly understand its
configured sources, plus tests + docs.

User-visible benefits

- LAST30DAYS_STORE=1 in .env turns persistence default-on without
  remembering --store on every invocation. Mirrors LAST30DAYS_DEBUG /
  LAST30DAYS_SKIP_PREFLIGHT convention.
- SCRAPE_CREATORS_API_KEY (with underscore) accepted as alias for the
  canonical name. Matches the spelling used in the vendor's own example
  code (Adrian Horning's repo); saves the next user the same diagnostic
  rabbit hole.
- Bluesky search now hits api.bsky.app (canonical AppView) instead of
  public.api.bsky.app (BunnyCDN-blocked public mirror as of 2026-05-04).
  BSKY_SEARCH_HOST env var lets users self-rescue future host migrations
  without a code release. Pre-fix: silent 0 Bluesky posts on every run.
- App-password format validator emits a one-shot stderr warning when
  BSKY_APP_PASSWORD doesn't match xxxx-xxxx-xxxx-xxxx form. Detect-don't-
  gate: createSession still accepts main passwords; the warning helps
  users identify a hygiene issue without breaking existing setups.
- Instagram retry on multi-token 500. SC's v2 reels endpoint wraps
  Google Search and 500's frequently on multi-word queries; a hashtag-
  form retry runs once before bubbling up. Documented vendor instability.
- LAST30DAYS_TRANSCRIPT_TIMEOUT env var (default 30s, was hardcoded 15s).
  SC's transcript endpoint regularly takes >15s; the old default was
  clipping legitimate responses.
- Silent-failure visibility: new bonus_errored field in the quality
  nudge fires when SC is configured but Instagram returned 0 items.
  Users see "Bonus source silent: Instagram" instead of unexplained
  absence.
- YouTube degraded-ratio false-positive fixed. Captions-disabled videos
  can never produce a transcript regardless of yt-dlp version; they're
  now subtracted from the denominator so a single uploader-disabled
  video doesn't false-trigger the "stale yt-dlp" nudge.
- urllib retry path: status_code attribute typo fix. The Instagram
  500-retry was dead code on the urllib branch (getattr(e, 'status', ...)
  while http.HTTPError exposes status_code).

Docs

- README.md: added /plugin install last30days step after marketplace add
  in three places (the install was previously omitted in the docs).
- CONFIGURATION.md: documented LAST30DAYS_STORE env var, added
  BSKY_SEARCH_HOST + app-password format section, mentioned
  LAST30DAYS_TRANSCRIPT_TIMEOUT in the Instagram source row.

Test plan

- 43 new unit tests across test_bluesky.py, test_instagram_sc.py,
  test_quality_nudge.py, test_youtube_yt.py
- 141 total tests passing in target suite
- Verified end-to-end: /last30days "Toronto resale condo market" with
  all 11+ sources active stored 35 new + 5 updated findings, all builder-
  PR-style accounts absent (organic agent voice in Instagram + TikTok
  results)

Backward compatibility

All changes are strictly additive. Optional kwargs default to None.
New env vars are opt-in. Existing CLI flags untouched. Existing callers
of public functions unaffected.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Daniel Zivkovic
2026-05-05 16:53:10 -04:00
committed by Trevin Chow
parent a8e462c978
commit 44971a6aae
14 changed files with 865 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/last30days/scripts/lib/quality_nudge.py
+67 -7
View File
@@ -58,12 +58,38 @@ def _is_youtube_degraded(research_results: dict, threshold: float) -> bool:
ratio is below threshold. The canonical cause is a stale yt-dlp binary -
YouTube's caption format changes frequently and old binaries silently fail
every transcript while the search itself still succeeds.
Captions-disabled videos are subtracted from the denominator: an uploader
who turned off captions can never produce a transcript, so counting that
video toward "fetch failures" produces false positives. A single
captions-disabled video in a small result set was tripping the nudge.
"""
videos = int(research_results.get("youtube_videos_count") or 0)
transcripts = int(research_results.get("youtube_transcripts_count") or 0)
captions_disabled = int(research_results.get("youtube_captions_disabled_count") or 0)
if videos <= 0:
return False
return (transcripts / videos) < threshold
eligible = videos - captions_disabled
if eligible <= 0:
# Every returned video had captions disabled - upstream content fact,
# not a yt-dlp problem. Don't flag.
return False
return (transcripts / eligible) < threshold
def _is_instagram_silent_failure(config: dict, research_results: dict) -> bool:
"""Instagram is silently failing when SC is configured but the source
returned zero items. The canonical cause is SC's v2 reels endpoint
500'ing on multi-token queries (it wraps Google Search and is documented
to be flaky there). Pre-fix the user got no signal at all - no Instagram
section in the brief, no error in the footer, just unexplained absence.
"""
if not config.get("SCRAPECREATORS_API_KEY"):
return False # not configured — not a silent failure
count = research_results.get("instagram_items_count")
if count is None:
return False # source not run this invocation
return int(count) == 0
def compute_quality_score(config: dict, research_results: dict) -> dict:
@@ -75,6 +101,8 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
reddit_error reflecting what happened this run. Optional keys
``youtube_videos_count`` and ``youtube_transcripts_count`` enable
degraded-YouTube detection (transcript-fetch ratio below threshold).
Optional key ``instagram_items_count`` enables silent-failure
detection for the bonus Instagram source.
Returns:
{
@@ -83,13 +111,15 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
"core_missing": ["x", "youtube"],
"core_errored": [], # configured but errored at top level
"core_degraded": [], # configured and returned items but quality below threshold
"nudge_text": "..." or None if 100%
"bonus_errored": [], # bonus sources (Instagram, etc.) configured but silent
"nudge_text": "..." or None if all sources healthy
}
"""
core_active: List[str] = []
core_missing: List[str] = []
core_errored: List[str] = []
core_degraded: List[str] = []
bonus_errored: List[str] = []
# HN, Polymarket, and Reddit are always active
core_active.append("hn")
@@ -127,6 +157,11 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
if has_ytdlp and research_results.get("youtube_error"):
core_errored.append("youtube")
# Bonus sources (Instagram, etc.): SC-key holders expect content from
# these but until now the pipeline fell silent on configured-but-zero.
if _is_instagram_silent_failure(config, research_results):
bonus_errored.append("instagram")
score_pct = int(len(core_active) / 5 * 100)
has_sc = bool(config.get("SCRAPECREATORS_API_KEY"))
@@ -138,7 +173,8 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
research_results,
has_sc=has_sc,
active_sources=active_sources,
) if (core_missing or core_degraded) else None
bonus_errored=bonus_errored,
) if (core_missing or core_degraded or bonus_errored) else None
return {
"score_pct": score_pct,
@@ -146,6 +182,7 @@ def compute_quality_score(config: dict, research_results: dict) -> dict:
"core_missing": core_missing,
"core_errored": core_errored,
"core_degraded": core_degraded,
"bonus_errored": bonus_errored,
"nudge_text": nudge_text,
}
@@ -157,6 +194,7 @@ def _build_nudge_text(
research_results: dict = None,
has_sc: bool = False,
active_sources: list = None,
bonus_errored: List[str] = None,
) -> str:
"""Build human-readable nudge text describing what was missed or degraded.
@@ -165,6 +203,7 @@ def _build_nudge_text(
"""
lines: List[str] = []
core_degraded = core_degraded or []
bonus_errored = bonus_errored or []
research_results = research_results or {}
# Describe what was missed
@@ -183,6 +222,9 @@ def _build_nudge_text(
if core_degraded:
degraded_labels = ", ".join(SOURCE_LABELS[s] for s in core_degraded)
lines.append(f"Degraded: {degraded_labels}.")
if bonus_errored:
bonus_labels = ", ".join(s.capitalize() for s in bonus_errored)
lines.append(f"Bonus source silent: {bonus_labels}.")
lines.append("")
# Free suggestions
@@ -215,12 +257,30 @@ def _build_nudge_text(
if "youtube" in core_degraded:
videos = int(research_results.get("youtube_videos_count") or 0)
transcripts = int(research_results.get("youtube_transcripts_count") or 0)
captions_disabled = int(research_results.get("youtube_captions_disabled_count") or 0)
captions_note = ""
if captions_disabled > 0:
captions_note = (
f" ({captions_disabled} of those had captions disabled by the "
"uploader, which is a separate cause and not fixable on your end)"
)
free_suggestions.append(
f"YouTube returned {videos} videos but only {transcripts} transcripts "
"captured. The most common cause is a stale yt-dlp binary - YouTube's "
"caption format changes frequently and old binaries silently fail every "
"transcript. Update via your package manager: scoop update yt-dlp "
"(Windows), brew upgrade yt-dlp (macOS), or pip install -U yt-dlp."
f"captured{captions_note}. The most common remaining cause is a stale "
"yt-dlp binary - YouTube's caption format changes frequently and old "
"binaries silently fail every transcript. Update via your package "
"manager: scoop update yt-dlp (Windows), brew upgrade yt-dlp (macOS), "
"or pip install -U yt-dlp."
)
if "instagram" in bonus_errored:
free_suggestions.append(
"Instagram returned 0 reels despite SC being configured. SC's "
"v2 reels endpoint wraps Google Search and 500's frequently on "
"multi-token queries. The skill now retries with hashtag-form "
"automatically; if zero items still appear, the topic may have "
"no reel coverage on Instagram. Try a single-word topic like "
"the most distinctive noun in your query."
)
# Mention bonus opt-in sources when SC key is present