Consolidates seven beta-validated plans into the public release. Validated
on nine+ topics across GENERAL, COMPARISON, RECOMMENDATIONS, and
demographic-shopping classes before ship.
Plans bundled in this release:
- 003 Engine-emitted Pre-Research Status warning + Polymarket summarization
+ VOICE CONTRACT LAW 1-5 + Step 0.55 MANDATORY
- 004 WebSearch deferred-tool loading (ToolSearch STEP 0) + LAW 5 universal
+ top-of-file imperative
- 005 Supplement floor (2-3 minimum) separate from Step 0.55 pre-research
- 006 Step 2.5 MANDATORY raw-file append with canonical format example +
count-equality self-check
- 007 Restored April 9 canonical comparison template with Quick Verdict,
per-entity Strengths/Weaknesses, 9-axis Head-to-Head, Bottom Line,
emerging stack + LAW 2/4 COMPARISON exceptions
- 008 Person-topic GitHub handle resolution MANDATORY + LAW 1 reinforcement
at Step 2 tail and Step 2.5 entry + RECOMMENDATIONS signal-weighted
ranking rewrite + Polymarket post-merge topic filter (engine change,
filter_items_against_topic helper + vs/versus in _NOISE_WORDS)
- 009 Unified pre-flight CHECKLIST + VOICE CONTRACT formatting-authority
preface + Step 0.45 Query Quality Pre-Flight (4 keyword-trap classes) +
post-synthesis Sources-block self-check
Beta validation topics (2026-04-18): Kanye West, Matt Van Horn, CLI vs MCP,
OpenClaw vs Paperclip vs Hermes, Paperclip vs Hermes vs Open Claw, Garry
Tan, Israel vs Lebanon, Best programming language for AI agents, Peter
Steinberger post plan 009, Birthday gift for 42 year old man (Class 1
pre-flight fired correctly), Vincent Koc (passed).
No breaking changes. No new CLI flags. No new public API. Plugin name
(last30days) and marketplace name (last30days-skill) unchanged.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape
_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.
Shared _remap_comments helper will be reused for TikTok in a later commit.
* feat(tiktok): fetch top comments via ScrapeCreators when opted in
Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.
Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.
* feat(normalize): pass TikTok top_comments through with digg_count->score mapping
Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.
* feat(signals): add YouTube + TikTok top-comment score to engagement formula
Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).
Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.
* feat(render): source-aware thresholds and vote labels for top comments
10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.
* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in
Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
The dedup hot path recomputed normalize_text() 4 times per comparison
and recomputed item_text() on every inner-loop iteration. Pre-computing
n-gram sets and token sets into a _PreparedText cache cuts dedup time
by 6x (2.16s to 0.39s on 300 unique items).
Bird handle searches spawned one Node process per handle sequentially.
Now uses ThreadPoolExecutor so N handles run concurrently. Same pattern
applied to YouTube comment enrichment (was serial, Reddit was already
parallel) and the retry-thin-sources phase in the pipeline.
Clustering now pre-computes candidate text and uses prepared_similarity
for the O(n^2) grouping and MMR representative selection loops.
Minor: _is_wsl() cached with lru_cache, Bundle.add_items() uses
extend() instead of list concatenation.
End-to-end: 5.2s -> 3.7s (29% faster) on a typical 4-source query.
Add Xquik (xquik.com) as a new X/Twitter search source that uses a REST
API with full engagement metrics (likes, retweets, replies, quotes,
views, bookmarks). Uses stdlib urllib only -- no new dependencies.
- scripts/lib/xquik.py: source module with search, parse, query expansion
- tests/test_xquik.py: 32 unit tests covering all functions
- env.py: XQUIK_API_KEY config and availability check
- pipeline.py: source registration and retrieve dispatch
- normalize.py: reuses _normalize_x (same item format as Bird)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Matt Van Horn