feat: configuration enablement — env-var defaults + source resilience

Six small additive changes that make the skill correctly understand its
configured sources, plus tests + docs.

User-visible benefits

- LAST30DAYS_STORE=1 in .env turns persistence default-on without
  remembering --store on every invocation. Mirrors LAST30DAYS_DEBUG /
  LAST30DAYS_SKIP_PREFLIGHT convention.
- SCRAPE_CREATORS_API_KEY (with underscore) accepted as alias for the
  canonical name. Matches the spelling used in the vendor's own example
  code (Adrian Horning's repo); saves the next user the same diagnostic
  rabbit hole.
- Bluesky search now hits api.bsky.app (canonical AppView) instead of
  public.api.bsky.app (BunnyCDN-blocked public mirror as of 2026-05-04).
  BSKY_SEARCH_HOST env var lets users self-rescue future host migrations
  without a code release. Pre-fix: silent 0 Bluesky posts on every run.
- App-password format validator emits a one-shot stderr warning when
  BSKY_APP_PASSWORD doesn't match xxxx-xxxx-xxxx-xxxx form. Detect-don't-
  gate: createSession still accepts main passwords; the warning helps
  users identify a hygiene issue without breaking existing setups.
- Instagram retry on multi-token 500. SC's v2 reels endpoint wraps
  Google Search and 500's frequently on multi-word queries; a hashtag-
  form retry runs once before bubbling up. Documented vendor instability.
- LAST30DAYS_TRANSCRIPT_TIMEOUT env var (default 30s, was hardcoded 15s).
  SC's transcript endpoint regularly takes >15s; the old default was
  clipping legitimate responses.
- Silent-failure visibility: new bonus_errored field in the quality
  nudge fires when SC is configured but Instagram returned 0 items.
  Users see "Bonus source silent: Instagram" instead of unexplained
  absence.
- YouTube degraded-ratio false-positive fixed. Captions-disabled videos
  can never produce a transcript regardless of yt-dlp version; they're
  now subtracted from the denominator so a single uploader-disabled
  video doesn't false-trigger the "stale yt-dlp" nudge.
- urllib retry path: status_code attribute typo fix. The Instagram
  500-retry was dead code on the urllib branch (getattr(e, 'status', ...)
  while http.HTTPError exposes status_code).

Docs

- README.md: added /plugin install last30days step after marketplace add
  in three places (the install was previously omitted in the docs).
- CONFIGURATION.md: documented LAST30DAYS_STORE env var, added
  BSKY_SEARCH_HOST + app-password format section, mentioned
  LAST30DAYS_TRANSCRIPT_TIMEOUT in the Instagram source row.

Test plan

- 43 new unit tests across test_bluesky.py, test_instagram_sc.py,
  test_quality_nudge.py, test_youtube_yt.py
- 141 total tests passing in target suite
- Verified end-to-end: /last30days "Toronto resale condo market" with
  all 11+ sources active stored 35 new + 5 updated findings, all builder-
  PR-style accounts absent (organic agent voice in Instagram + TikTok
  results)

Backward compatibility

All changes are strictly additive. Optional kwargs default to None.
New env vars are opt-in. Existing CLI flags untouched. Existing callers
of public functions unaffected.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Daniel Zivkovic
2026-05-05 16:53:10 -04:00
committed by Trevin Chow
parent a8e462c978
commit 44971a6aae
14 changed files with 865 additions and 27 deletions
+142
View File
@@ -1,5 +1,6 @@
"""Tests for bluesky module."""
import os
import sys
import unittest
from pathlib import Path
@@ -211,5 +212,146 @@ class TestSearchBlueskyAuth(unittest.TestCase):
self.assertEqual(mock_request.call_args_list[3].kwargs.get("headers", {}), {"Authorization": "Bearer tok-new"})
class TestSearchEndpointHostResolution(unittest.TestCase):
"""The default search host moved from `public.api.bsky.app` (the
unauthenticated public mirror, now BunnyCDN-blocked for searchPosts) to
`api.bsky.app` (the canonical authenticated AppView). BSKY_SEARCH_HOST
env var or config value can override the default if Bluesky migrates
infrastructure again. Same os.environ-or-config hybrid pattern as
LAST30DAYS_STORE.
"""
def setUp(self):
# Snapshot env so per-test overrides don't leak
self._saved_env = os.environ.pop("BSKY_SEARCH_HOST", None)
def tearDown(self):
if self._saved_env is not None:
os.environ["BSKY_SEARCH_HOST"] = self._saved_env
else:
os.environ.pop("BSKY_SEARCH_HOST", None)
def test_module_constant_uses_canonical_appview(self):
# Regression guard against the public mirror reappearing as the default
self.assertIn("api.bsky.app", bluesky.BSKY_SEARCH_URL)
def test_module_constant_does_not_use_public_mirror(self):
# Hard regression guard — the exact host that BunnyCDN was blocking
self.assertNotIn("public.api.bsky.app", bluesky.BSKY_SEARCH_URL)
def test_resolver_default_when_no_override(self):
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_env_var_override(self):
os.environ["BSKY_SEARCH_HOST"] = "staging.bsky.app"
self.assertEqual(
bluesky._resolve_search_url(),
"https://staging.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_config_dict_override(self):
# User has BSKY_SEARCH_HOST only in .env file (project loads .env into
# config, not os.environ). Resolver must read both.
url = bluesky._resolve_search_url({"BSKY_SEARCH_HOST": "pds.example.com"})
self.assertEqual(url, "https://pds.example.com/xrpc/app.bsky.feed.searchPosts")
def test_resolver_env_var_wins_over_config(self):
# When both are set, os.environ takes precedence (matches LAST30DAYS_STORE)
os.environ["BSKY_SEARCH_HOST"] = "shell-host.example"
url = bluesky._resolve_search_url({"BSKY_SEARCH_HOST": "config-host.example"})
self.assertIn("shell-host.example", url)
self.assertNotIn("config-host.example", url)
def test_resolver_output_does_not_use_public_mirror(self):
# Regression guard at the resolver level (not just the constant) —
# this is what runtime actually calls. The constant-level guard
# above doesn't catch a regression where the resolver reverts.
self.assertNotIn("public.api.bsky.app", bluesky._resolve_search_url())
def test_resolver_strips_surrounding_whitespace(self):
# Pre-fix: " api.bsky.app " produced "https:// api.bsky.app /xrpc/..."
# which urllib raises ValueError on with no hint the env var caused it.
os.environ["BSKY_SEARCH_HOST"] = " api.bsky.app "
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_rejects_embedded_path(self):
# "my-proxy.com/xrpc/prefix" would have doubled the /xrpc/ segment.
# We fall back to the default to avoid a guaranteed 404.
os.environ["BSKY_SEARCH_HOST"] = "my-proxy.example.com/xrpc/prefix"
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_strips_embedded_scheme(self):
# Users who paste a full URL get a sane outcome, not a malformed URL.
os.environ["BSKY_SEARCH_HOST"] = "https://api.bsky.app"
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
def test_resolver_empty_string_falls_back_to_default(self):
os.environ["BSKY_SEARCH_HOST"] = ""
self.assertEqual(
bluesky._resolve_search_url(),
"https://api.bsky.app/xrpc/app.bsky.feed.searchPosts",
)
class TestAppPasswordFormat(unittest.TestCase):
"""Bluesky app passwords are 19-char xxxx-xxxx-xxxx-xxxx (lowercase
alphanumeric, three hyphens at fixed positions). Main-account passwords
are accepted by createSession but are bad hygiene. The validator detects
the format mismatch without gating any caller.
"""
def test_accepts_valid_app_password_form(self):
# Use a fake example — never a real password
self.assertTrue(bluesky._validate_app_password_format("wfwp-cq7o-5six-7wy5"))
def test_rejects_length_15_string(self):
# The exact failure mode that triggered the 2026-05-04 investigation:
# user stored their main login password (15 chars) in BSKY_APP_PASSWORD
self.assertFalse(bluesky._validate_app_password_format("mainpassword123"))
def test_rejects_16_char_no_hyphen_string(self):
# Hex-style API key shape — common confusion with other services
self.assertFalse(bluesky._validate_app_password_format("abcdef0123456789"))
def test_rejects_uppercase_letters(self):
# Bluesky app passwords are all-lowercase by spec
self.assertFalse(bluesky._validate_app_password_format("WFWP-cq7o-5six-7wy5"))
def test_rejects_underscore_separator(self):
# Wrong separator
self.assertFalse(bluesky._validate_app_password_format("wfwp_cq7o_5six_7wy5"))
def test_rejects_special_chars_in_groups(self):
# Special characters are not part of the alphanumeric class
self.assertFalse(bluesky._validate_app_password_format("wfwp-cq7o-5six-7wy@"))
def test_rejects_empty_string(self):
self.assertFalse(bluesky._validate_app_password_format(""))
def test_rejects_none(self):
# Callers may pass config.get('BSKY_APP_PASSWORD') which is None when unset
self.assertFalse(bluesky._validate_app_password_format(None))
def test_rejects_integer(self):
# Defensive: don't crash if a numeric value sneaks in
self.assertFalse(bluesky._validate_app_password_format(123456789012345))
def test_rejects_list(self):
# Defensive: don't crash on iterables
self.assertFalse(bluesky._validate_app_password_format(["wfwp", "cq7o", "5six", "7wy5"]))
if __name__ == "__main__":
unittest.main()