feat: configuration enablement — env-var defaults + source resilience

Six small additive changes that make the skill correctly understand its
configured sources, plus tests + docs.

User-visible benefits

- LAST30DAYS_STORE=1 in .env turns persistence default-on without
  remembering --store on every invocation. Mirrors LAST30DAYS_DEBUG /
  LAST30DAYS_SKIP_PREFLIGHT convention.
- SCRAPE_CREATORS_API_KEY (with underscore) accepted as alias for the
  canonical name. Matches the spelling used in the vendor's own example
  code (Adrian Horning's repo); saves the next user the same diagnostic
  rabbit hole.
- Bluesky search now hits api.bsky.app (canonical AppView) instead of
  public.api.bsky.app (BunnyCDN-blocked public mirror as of 2026-05-04).
  BSKY_SEARCH_HOST env var lets users self-rescue future host migrations
  without a code release. Pre-fix: silent 0 Bluesky posts on every run.
- App-password format validator emits a one-shot stderr warning when
  BSKY_APP_PASSWORD doesn't match xxxx-xxxx-xxxx-xxxx form. Detect-don't-
  gate: createSession still accepts main passwords; the warning helps
  users identify a hygiene issue without breaking existing setups.
- Instagram retry on multi-token 500. SC's v2 reels endpoint wraps
  Google Search and 500's frequently on multi-word queries; a hashtag-
  form retry runs once before bubbling up. Documented vendor instability.
- LAST30DAYS_TRANSCRIPT_TIMEOUT env var (default 30s, was hardcoded 15s).
  SC's transcript endpoint regularly takes >15s; the old default was
  clipping legitimate responses.
- Silent-failure visibility: new bonus_errored field in the quality
  nudge fires when SC is configured but Instagram returned 0 items.
  Users see "Bonus source silent: Instagram" instead of unexplained
  absence.
- YouTube degraded-ratio false-positive fixed. Captions-disabled videos
  can never produce a transcript regardless of yt-dlp version; they're
  now subtracted from the denominator so a single uploader-disabled
  video doesn't false-trigger the "stale yt-dlp" nudge.
- urllib retry path: status_code attribute typo fix. The Instagram
  500-retry was dead code on the urllib branch (getattr(e, 'status', ...)
  while http.HTTPError exposes status_code).

Docs

- README.md: added /plugin install last30days step after marketplace add
  in three places (the install was previously omitted in the docs).
- CONFIGURATION.md: documented LAST30DAYS_STORE env var, added
  BSKY_SEARCH_HOST + app-password format section, mentioned
  LAST30DAYS_TRANSCRIPT_TIMEOUT in the Instagram source row.

Test plan

- 43 new unit tests across test_bluesky.py, test_instagram_sc.py,
  test_quality_nudge.py, test_youtube_yt.py
- 141 total tests passing in target suite
- Verified end-to-end: /last30days "Toronto resale condo market" with
  all 11+ sources active stored 35 new + 5 updated findings, all builder-
  PR-style accounts absent (organic agent voice in Instagram + TikTok
  results)

Backward compatibility

All changes are strictly additive. Optional kwargs default to None.
New env vars are opt-in. Existing CLI flags untouched. Existing callers
of public functions unaffected.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Daniel Zivkovic
2026-05-05 16:53:10 -04:00
committed by Trevin Chow
parent a8e462c978
commit 44971a6aae
14 changed files with 865 additions and 27 deletions
+85 -4
View File
@@ -1,10 +1,19 @@
"""Bluesky search via AT Protocol (requires app password).
Uses bsky.social for auth and public.api.bsky.app for post search.
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars.
Uses bsky.social for auth and api.bsky.app for post search (the canonical
authenticated AppView). The previous default `public.api.bsky.app` is the
unauthenticated public mirror, which BunnyCDN now blocks for searchPosts
regardless of auth header (verified 2026-05-04). Override the search host
via BSKY_SEARCH_HOST env var if Bluesky migrates infrastructure again.
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars. App passwords are
19-char xxxx-xxxx-xxxx-xxxx; generate at bsky.app/settings/app-passwords.
The createSession endpoint accepts main-account passwords too, but they're
bad hygiene (no scope, can't revoke individually).
"""
import math
import os
import re
import sys
import time
@@ -14,7 +23,65 @@ from typing import Any, Dict, List, Optional
from . import http, log
BSKY_SESSION_URL = "https://bsky.social/xrpc/com.atproto.server.createSession"
BSKY_SEARCH_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts"
_DEFAULT_BSKY_SEARCH_HOST = "api.bsky.app"
BSKY_SEARCH_URL = f"https://{_DEFAULT_BSKY_SEARCH_HOST}/xrpc/app.bsky.feed.searchPosts"
def _resolve_search_url(config: Optional[Dict[str, Any]] = None) -> str:
"""Resolve the Bluesky search URL with BSKY_SEARCH_HOST override.
Default is api.bsky.app. Override via BSKY_SEARCH_HOST in shell env or
.env file. The project's env.py loads .env into config but not into
os.environ, so check both — same hybrid pattern as last30days.py for
LAST30DAYS_STORE.
Hardens user-supplied host values against three common mis-configurations:
whitespace (e.g. " api.bsky.app "), embedded path components (e.g.
"api.bsky.app/xrpc/proxy") that would double the /xrpc/ segment, and
embedded scheme prefixes (e.g. "https://api.bsky.app"). On any of these
we log a warning and fall back to the default rather than building an
invalid URL with an opaque downstream error.
"""
config = config or {}
raw = (
os.environ.get("BSKY_SEARCH_HOST")
or config.get("BSKY_SEARCH_HOST")
or _DEFAULT_BSKY_SEARCH_HOST
)
host = raw.strip().rstrip("/")
# Strip embedded scheme so users who paste full URLs do not break the f-string.
for prefix in ("https://", "http://"):
if host.lower().startswith(prefix):
host = host[len(prefix):]
break
if not host or "/" in host or " " in host:
# Embedded path or whitespace remains — don't trust it. Default + log.
if raw != _DEFAULT_BSKY_SEARCH_HOST:
_log(
f"BSKY_SEARCH_HOST={raw!r} is not a bare hostname; "
f"falling back to default {_DEFAULT_BSKY_SEARCH_HOST!r}"
)
host = _DEFAULT_BSKY_SEARCH_HOST
return f"https://{host}/xrpc/app.bsky.feed.searchPosts"
# App-password format: xxxx-xxxx-xxxx-xxxx (19 chars, lowercase alphanumeric
# with three hyphens at fixed positions).
_APP_PASSWORD_RE = re.compile(r"^[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}$")
def _validate_app_password_format(value) -> bool:
"""Return True if value matches Bluesky's 19-char app-password format.
False for non-strings (None, int, list) so callers passing config dict
values directly don't crash. Detect-but-not-gate: the createSession
endpoint also accepts main-account passwords, so failing this check is
a hygiene smell, not a hard error.
"""
if not isinstance(value, str):
return False
return bool(_APP_PASSWORD_RE.fullmatch(value))
DEPTH_CONFIG = {
"quick": 15,
@@ -144,6 +211,20 @@ def search_bluesky(
if not handle or not app_password:
return {"posts": [], "error": "Bluesky credentials not configured"}
# One-shot hygiene warning if BSKY_APP_PASSWORD is not in app-password
# form. createSession accepts main-account passwords too — but main
# passwords have no scope (full account access), can't be revoked
# individually, and rotating them breaks every service that holds them.
# We warn but do not gate, matching the project's detect-don't-block
# philosophy elsewhere.
if not _validate_app_password_format(app_password):
_log(
"BSKY_APP_PASSWORD does not look like an app password "
"(expected xxxx-xxxx-xxxx-xxxx, 19 chars). It may be a main "
"account password — those work but are bad hygiene. Generate "
"an app password at https://bsky.app/settings/app-passwords"
)
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
core_topic = _extract_core_subject(topic)
@@ -155,7 +236,7 @@ def search_bluesky(
"limit": str(min(count, 100)),
"sort": "top",
}
url = f"{BSKY_SEARCH_URL}?{urlencode(params)}"
url = f"{_resolve_search_url(config)}?{urlencode(params)}"
def _auth_and_search() -> tuple[Optional[Dict[str, Any]], Optional[str]]:
token = _create_session(handle, app_password)