feat: configuration enablement — env-var defaults + source resilience
Six small additive changes that make the skill correctly understand its configured sources, plus tests + docs. User-visible benefits - LAST30DAYS_STORE=1 in .env turns persistence default-on without remembering --store on every invocation. Mirrors LAST30DAYS_DEBUG / LAST30DAYS_SKIP_PREFLIGHT convention. - SCRAPE_CREATORS_API_KEY (with underscore) accepted as alias for the canonical name. Matches the spelling used in the vendor's own example code (Adrian Horning's repo); saves the next user the same diagnostic rabbit hole. - Bluesky search now hits api.bsky.app (canonical AppView) instead of public.api.bsky.app (BunnyCDN-blocked public mirror as of 2026-05-04). BSKY_SEARCH_HOST env var lets users self-rescue future host migrations without a code release. Pre-fix: silent 0 Bluesky posts on every run. - App-password format validator emits a one-shot stderr warning when BSKY_APP_PASSWORD doesn't match xxxx-xxxx-xxxx-xxxx form. Detect-don't- gate: createSession still accepts main passwords; the warning helps users identify a hygiene issue without breaking existing setups. - Instagram retry on multi-token 500. SC's v2 reels endpoint wraps Google Search and 500's frequently on multi-word queries; a hashtag- form retry runs once before bubbling up. Documented vendor instability. - LAST30DAYS_TRANSCRIPT_TIMEOUT env var (default 30s, was hardcoded 15s). SC's transcript endpoint regularly takes >15s; the old default was clipping legitimate responses. - Silent-failure visibility: new bonus_errored field in the quality nudge fires when SC is configured but Instagram returned 0 items. Users see "Bonus source silent: Instagram" instead of unexplained absence. - YouTube degraded-ratio false-positive fixed. Captions-disabled videos can never produce a transcript regardless of yt-dlp version; they're now subtracted from the denominator so a single uploader-disabled video doesn't false-trigger the "stale yt-dlp" nudge. - urllib retry path: status_code attribute typo fix. The Instagram 500-retry was dead code on the urllib branch (getattr(e, 'status', ...) while http.HTTPError exposes status_code). Docs - README.md: added /plugin install last30days step after marketplace add in three places (the install was previously omitted in the docs). - CONFIGURATION.md: documented LAST30DAYS_STORE env var, added BSKY_SEARCH_HOST + app-password format section, mentioned LAST30DAYS_TRANSCRIPT_TIMEOUT in the Instagram source row. Test plan - 43 new unit tests across test_bluesky.py, test_instagram_sc.py, test_quality_nudge.py, test_youtube_yt.py - 141 total tests passing in target suite - Verified end-to-end: /last30days "Toronto resale condo market" with all 11+ sources active stored 35 new + 5 updated findings, all builder- PR-style accounts absent (organic agent voice in Instagram + TikTok results) Backward compatibility All changes are strictly additive. Optional kwargs default to None. New env vars are opt-in. Existing CLI flags untouched. Existing callers of public functions unaffected. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
committed by
Trevin Chow
parent
a8e462c978
commit
44971a6aae
@@ -1,10 +1,19 @@
|
||||
"""Bluesky search via AT Protocol (requires app password).
|
||||
|
||||
Uses bsky.social for auth and public.api.bsky.app for post search.
|
||||
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars.
|
||||
Uses bsky.social for auth and api.bsky.app for post search (the canonical
|
||||
authenticated AppView). The previous default `public.api.bsky.app` is the
|
||||
unauthenticated public mirror, which BunnyCDN now blocks for searchPosts
|
||||
regardless of auth header (verified 2026-05-04). Override the search host
|
||||
via BSKY_SEARCH_HOST env var if Bluesky migrates infrastructure again.
|
||||
|
||||
Requires BSKY_HANDLE and BSKY_APP_PASSWORD env vars. App passwords are
|
||||
19-char xxxx-xxxx-xxxx-xxxx; generate at bsky.app/settings/app-passwords.
|
||||
The createSession endpoint accepts main-account passwords too, but they're
|
||||
bad hygiene (no scope, can't revoke individually).
|
||||
"""
|
||||
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
@@ -14,7 +23,65 @@ from typing import Any, Dict, List, Optional
|
||||
from . import http, log
|
||||
|
||||
BSKY_SESSION_URL = "https://bsky.social/xrpc/com.atproto.server.createSession"
|
||||
BSKY_SEARCH_URL = "https://public.api.bsky.app/xrpc/app.bsky.feed.searchPosts"
|
||||
_DEFAULT_BSKY_SEARCH_HOST = "api.bsky.app"
|
||||
BSKY_SEARCH_URL = f"https://{_DEFAULT_BSKY_SEARCH_HOST}/xrpc/app.bsky.feed.searchPosts"
|
||||
|
||||
|
||||
def _resolve_search_url(config: Optional[Dict[str, Any]] = None) -> str:
|
||||
"""Resolve the Bluesky search URL with BSKY_SEARCH_HOST override.
|
||||
|
||||
Default is api.bsky.app. Override via BSKY_SEARCH_HOST in shell env or
|
||||
.env file. The project's env.py loads .env into config but not into
|
||||
os.environ, so check both — same hybrid pattern as last30days.py for
|
||||
LAST30DAYS_STORE.
|
||||
|
||||
Hardens user-supplied host values against three common mis-configurations:
|
||||
whitespace (e.g. " api.bsky.app "), embedded path components (e.g.
|
||||
"api.bsky.app/xrpc/proxy") that would double the /xrpc/ segment, and
|
||||
embedded scheme prefixes (e.g. "https://api.bsky.app"). On any of these
|
||||
we log a warning and fall back to the default rather than building an
|
||||
invalid URL with an opaque downstream error.
|
||||
"""
|
||||
config = config or {}
|
||||
raw = (
|
||||
os.environ.get("BSKY_SEARCH_HOST")
|
||||
or config.get("BSKY_SEARCH_HOST")
|
||||
or _DEFAULT_BSKY_SEARCH_HOST
|
||||
)
|
||||
host = raw.strip().rstrip("/")
|
||||
# Strip embedded scheme so users who paste full URLs do not break the f-string.
|
||||
for prefix in ("https://", "http://"):
|
||||
if host.lower().startswith(prefix):
|
||||
host = host[len(prefix):]
|
||||
break
|
||||
if not host or "/" in host or " " in host:
|
||||
# Embedded path or whitespace remains — don't trust it. Default + log.
|
||||
if raw != _DEFAULT_BSKY_SEARCH_HOST:
|
||||
_log(
|
||||
f"BSKY_SEARCH_HOST={raw!r} is not a bare hostname; "
|
||||
f"falling back to default {_DEFAULT_BSKY_SEARCH_HOST!r}"
|
||||
)
|
||||
host = _DEFAULT_BSKY_SEARCH_HOST
|
||||
return f"https://{host}/xrpc/app.bsky.feed.searchPosts"
|
||||
|
||||
|
||||
# App-password format: xxxx-xxxx-xxxx-xxxx (19 chars, lowercase alphanumeric
|
||||
# with three hyphens at fixed positions).
|
||||
_APP_PASSWORD_RE = re.compile(r"^[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}$")
|
||||
|
||||
|
||||
def _validate_app_password_format(value) -> bool:
|
||||
"""Return True if value matches Bluesky's 19-char app-password format.
|
||||
|
||||
False for non-strings (None, int, list) so callers passing config dict
|
||||
values directly don't crash. Detect-but-not-gate: the createSession
|
||||
endpoint also accepts main-account passwords, so failing this check is
|
||||
a hygiene smell, not a hard error.
|
||||
"""
|
||||
if not isinstance(value, str):
|
||||
return False
|
||||
return bool(_APP_PASSWORD_RE.fullmatch(value))
|
||||
|
||||
|
||||
DEPTH_CONFIG = {
|
||||
"quick": 15,
|
||||
@@ -144,6 +211,20 @@ def search_bluesky(
|
||||
if not handle or not app_password:
|
||||
return {"posts": [], "error": "Bluesky credentials not configured"}
|
||||
|
||||
# One-shot hygiene warning if BSKY_APP_PASSWORD is not in app-password
|
||||
# form. createSession accepts main-account passwords too — but main
|
||||
# passwords have no scope (full account access), can't be revoked
|
||||
# individually, and rotating them breaks every service that holds them.
|
||||
# We warn but do not gate, matching the project's detect-don't-block
|
||||
# philosophy elsewhere.
|
||||
if not _validate_app_password_format(app_password):
|
||||
_log(
|
||||
"BSKY_APP_PASSWORD does not look like an app password "
|
||||
"(expected xxxx-xxxx-xxxx-xxxx, 19 chars). It may be a main "
|
||||
"account password — those work but are bad hygiene. Generate "
|
||||
"an app password at https://bsky.app/settings/app-passwords"
|
||||
)
|
||||
|
||||
count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])
|
||||
core_topic = _extract_core_subject(topic)
|
||||
|
||||
@@ -155,7 +236,7 @@ def search_bluesky(
|
||||
"limit": str(min(count, 100)),
|
||||
"sort": "top",
|
||||
}
|
||||
url = f"{BSKY_SEARCH_URL}?{urlencode(params)}"
|
||||
url = f"{_resolve_search_url(config)}?{urlencode(params)}"
|
||||
|
||||
def _auth_and_search() -> tuple[Optional[Dict[str, Any]], Optional[str]]:
|
||||
token = _create_session(handle, app_password)
|
||||
|
||||
Reference in New Issue
Block a user