44971a6aae5ca831daf6d478080649c2503043ce
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Daniel Zivkovic
|
44971a6aae |
feat: configuration enablement — env-var defaults + source resilience
Six small additive changes that make the skill correctly understand its configured sources, plus tests + docs. User-visible benefits - LAST30DAYS_STORE=1 in .env turns persistence default-on without remembering --store on every invocation. Mirrors LAST30DAYS_DEBUG / LAST30DAYS_SKIP_PREFLIGHT convention. - SCRAPE_CREATORS_API_KEY (with underscore) accepted as alias for the canonical name. Matches the spelling used in the vendor's own example code (Adrian Horning's repo); saves the next user the same diagnostic rabbit hole. - Bluesky search now hits api.bsky.app (canonical AppView) instead of public.api.bsky.app (BunnyCDN-blocked public mirror as of 2026-05-04). BSKY_SEARCH_HOST env var lets users self-rescue future host migrations without a code release. Pre-fix: silent 0 Bluesky posts on every run. - App-password format validator emits a one-shot stderr warning when BSKY_APP_PASSWORD doesn't match xxxx-xxxx-xxxx-xxxx form. Detect-don't- gate: createSession still accepts main passwords; the warning helps users identify a hygiene issue without breaking existing setups. - Instagram retry on multi-token 500. SC's v2 reels endpoint wraps Google Search and 500's frequently on multi-word queries; a hashtag- form retry runs once before bubbling up. Documented vendor instability. - LAST30DAYS_TRANSCRIPT_TIMEOUT env var (default 30s, was hardcoded 15s). SC's transcript endpoint regularly takes >15s; the old default was clipping legitimate responses. - Silent-failure visibility: new bonus_errored field in the quality nudge fires when SC is configured but Instagram returned 0 items. Users see "Bonus source silent: Instagram" instead of unexplained absence. - YouTube degraded-ratio false-positive fixed. Captions-disabled videos can never produce a transcript regardless of yt-dlp version; they're now subtracted from the denominator so a single uploader-disabled video doesn't false-trigger the "stale yt-dlp" nudge. - urllib retry path: status_code attribute typo fix. The Instagram 500-retry was dead code on the urllib branch (getattr(e, 'status', ...) while http.HTTPError exposes status_code). Docs - README.md: added /plugin install last30days step after marketplace add in three places (the install was previously omitted in the docs). - CONFIGURATION.md: documented LAST30DAYS_STORE env var, added BSKY_SEARCH_HOST + app-password format section, mentioned LAST30DAYS_TRANSCRIPT_TIMEOUT in the Instagram source row. Test plan - 43 new unit tests across test_bluesky.py, test_instagram_sc.py, test_quality_nudge.py, test_youtube_yt.py - 141 total tests passing in target suite - Verified end-to-end: /last30days "Toronto resale condo market" with all 11+ sources active stored 35 new + 5 updated findings, all builder- PR-style accounts absent (organic agent voice in Instagram + TikTok results) Backward compatibility All changes are strictly additive. Optional kwargs default to None. New env vars are opt-in. Existing CLI flags untouched. Existing callers of public functions unaffected. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Daniel Zivkovic
|
0a5102e193 |
fix(youtube): surface transcript-fetch ratio in footer + add degraded nudge
When yt-dlp is installed but stale (or otherwise unable to fetch transcripts
for any returned videos), runs previously reported YouTube as fully
successful in two user-facing surfaces:
1. Footer (render.py): showed "N videos | M views" with no indication
that zero transcripts were captured. The "with transcripts" segment
was conditionally suppressed when the count was zero - converting
the canonical stale-binary failure mode into a silent absence at
the very surface users read for "did this work?".
2. Quality nudge (quality_nudge.py): classified YouTube as "active"
based purely on yt-dlp installation + absence of a top-level error.
Per-video transcript-fetch ratio was never inspected. A run that
returned N videos with 0 transcripts (canonical stale-binary
failure) was reported as fully active.
The engine itself logs the failure correctly at default stderr level
(`[YouTube] Got transcripts for 0/N videos (N failed)`), but that line
gets buried in 100+ lines of parallel-source progress output and is
contradicted by the success-shaped footer and nudge that follow.
This change makes both conclusion surfaces honest:
* render.py footer always renders "M/N with transcripts" so the ratio
is visible regardless of value. Zero is no longer hidden. Format is
M/N (not bare M) so the denominator is in the message and the user
does not have to cross-reference the "videos" count.
* quality_nudge.py adds a third tier between "active" and "missing":
"degraded". Triggered when yt-dlp is installed AND videos were
returned AND transcript-fetch ratio is below threshold (default 50%,
tunable via DEGRADED_TRANSCRIPT_THRESHOLD env var). Emits an
actionable nudge: "YouTube returned N videos but only M transcripts
captured. The most common cause is a stale yt-dlp binary - YouTube's
caption format changes frequently and old binaries silently fail
every transcript. Update via your package manager: scoop update
yt-dlp (Windows), brew upgrade yt-dlp (macOS), or pip install -U
yt-dlp."
* last30days.py populates youtube_videos_count and
youtube_transcripts_count in the research_results dict it passes to
compute_quality_score, enabling the new degraded check at the call
site.
Threshold rationale: 50% accommodates a few legitimate
caption-disabled videos in a multi-video result, but a stale-binary
run that fails every transcript trips the nudge cleanly.
Score impact: degradation is informational, not score-affecting.
YouTube still counts as "active" in score_pct so users do not see
their score drop for a fixable client-side issue. The nudge directs
them to their own package manager.
Tests:
* tests/test_quality_nudge.py: 6 new TestYouTubeDegraded cases cover
zero-transcripts-flags-degraded, partial-above-threshold-does-not-flag,
zero-videos-does-not-flag (no false positives on absence),
one-of-three-flags-degraded, threshold-tunable-via-config, and
degraded-does-not-affect-score.
* tests/test_render_v3.py: 4 new YoutubeFooterTranscriptRatioTests
cases cover zero-transcripts-with-videos-renders-zero-over-total
(the regression repro), partial-renders-ratio, full-renders-ratio,
and no-videos-suppresses-entire-segment.
All 29 new test cases verified GREEN with the fix and RED without it
(temp-reverted both files separately to confirm each test catches the
specific regression it asserts).
Integration validation: ran the engine against an intentionally stale
yt-dlp 2025.03.31 binary placed first on PATH. Pre-fix the footer
showed `YouTube: 3 videos | 386,815 views` (no transcript signal).
Post-fix the footer shows `YouTube: 3 videos | 386,815 views | 0/3
with transcripts` and stderr emits "Degraded: YouTube" plus the
actionable update-yt-dlp nudge.
Out of scope (deserves its own PR): exposing transcripts_captured in
the EVIDENCE FOR SYNTHESIS block so the synthesizing model can flag
degradation in prose. Larger schema-touching change.
|
||
|
Matt Van Horn
|
0a9ff16dfc |
feat: v3.0.0 - intelligent search, GitHub person/project mode, ELI5, 13+ sources
v3 rewrites the search engine from the ground up: - Intelligent pre-research: resolves X handles, GitHub repos, subreddits, TikTok hashtags, and YouTube channels before searching - GitHub person-mode: PR velocity, top repos by stars, release notes - GitHub project-mode: live star counts, README, releases, top issues - ELI5 mode: plain language synthesis, no jargon - 13+ sources: Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket, GitHub, Threads, Pinterest, Perplexity, Bluesky, Web - Free Reddit comments via public JSON (no API key needed) - Fun judge v2: humor scoring baked into narrative - Cookie consent before browser scanning - 10,000 free ScrapeCreators calls - 1,012 tests Thank you to the community contributors whose issues and PRs shaped v3: @uppinote20 (#143), @zerone0x (#134, #136), @thinkun (#116), @thomasmktong (#124), @fanispoulinakisai-boop (#100), @pejmanjohn (#78), @zl190 (#115), @hnshah (#84, #85, #86) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Matt Van Horn
|
775596ce21 |
feat: v2.9.6 — free-first NUX, cookie extraction, quality scoring
Setup wizard with consent-first cookie extraction (Chrome/Firefox/Safari), yt-dlp auto-install, ScrapeCreators push, quality scoring (5 core sources), status banner redesign, honest Reddit labeling, inline YouTube transcripts, Exa free web search, Reddit public fallback, and post-research quality nudge. Co-authored-by: Matt Van Horn <mvanhorn@MacBook-Pro.local> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |