feat: surface YouTube + TikTok top comments alongside Reddit (#260)
* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape
_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.
Shared _remap_comments helper will be reused for TikTok in a later commit.
* feat(tiktok): fetch top comments via ScrapeCreators when opted in
Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.
Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.
* feat(normalize): pass TikTok top_comments through with digg_count->score mapping
Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.
* feat(signals): add YouTube + TikTok top-comment score to engagement formula
Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).
Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.
* feat(render): source-aware thresholds and vote labels for top comments
10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.
* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in
Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
This commit is contained in:
+30
-4
@@ -82,12 +82,11 @@ def _top_comment_score(item: schema.SourceItem) -> float:
|
||||
|
||||
|
||||
# Per-source engagement weights: list of (field_name, weight) tuples.
|
||||
# Reddit uses a custom function because upvote_ratio and top_comment_score
|
||||
# are not simple log1p fields.
|
||||
# Reddit, YouTube, and TikTok use custom functions because they include
|
||||
# a dedicated 10% top-comment-score slot (see _reddit_engagement,
|
||||
# _youtube_engagement, _tiktok_engagement).
|
||||
ENGAGEMENT_WEIGHTS: dict[str, list[tuple[str, float]]] = {
|
||||
"x": [("likes", 0.55), ("reposts", 0.25), ("replies", 0.15), ("quotes", 0.05)],
|
||||
"youtube": [("views", 0.50), ("likes", 0.35), ("comments", 0.15)],
|
||||
"tiktok": [("views", 0.50), ("likes", 0.30), ("comments", 0.20)],
|
||||
"instagram": [("views", 0.50), ("likes", 0.30), ("comments", 0.20)],
|
||||
"hackernews": [("points", 0.55), ("comments", 0.45)],
|
||||
"bluesky": [("likes", 0.40), ("reposts", 0.30), ("replies", 0.20), ("quotes", 0.10)],
|
||||
@@ -113,6 +112,29 @@ def _reddit_engagement(item: schema.SourceItem) -> float | None:
|
||||
return (0.50 * score) + (0.35 * comments) + (0.05 * (ratio * 10.0)) + (0.10 * top_comment)
|
||||
|
||||
|
||||
def _youtube_engagement(item: schema.SourceItem) -> float | None:
|
||||
views = log1p_safe(item.engagement.get("views"))
|
||||
likes = log1p_safe(item.engagement.get("likes"))
|
||||
comments = log1p_safe(item.engagement.get("comments"))
|
||||
top_comment = _top_comment_score(item)
|
||||
if not any([views, likes, comments, top_comment]):
|
||||
return None
|
||||
# Mirrors Reddit: carve out 10% for top-comment signal, keep view-weight
|
||||
# dominant. Without comments, the pre-change weights (0.50/0.35/0.15)
|
||||
# still govern relative ordering.
|
||||
return (0.45 * views) + (0.32 * likes) + (0.13 * comments) + (0.10 * top_comment)
|
||||
|
||||
|
||||
def _tiktok_engagement(item: schema.SourceItem) -> float | None:
|
||||
views = log1p_safe(item.engagement.get("views"))
|
||||
likes = log1p_safe(item.engagement.get("likes"))
|
||||
comments = log1p_safe(item.engagement.get("comments"))
|
||||
top_comment = _top_comment_score(item)
|
||||
if not any([views, likes, comments, top_comment]):
|
||||
return None
|
||||
return (0.45 * views) + (0.27 * likes) + (0.18 * comments) + (0.10 * top_comment)
|
||||
|
||||
|
||||
def _generic_engagement(item: schema.SourceItem) -> float | None:
|
||||
if not item.engagement:
|
||||
return None
|
||||
@@ -125,6 +147,10 @@ def _generic_engagement(item: schema.SourceItem) -> float | None:
|
||||
def engagement_raw(item: schema.SourceItem) -> float | None:
|
||||
if item.source == "reddit":
|
||||
return _reddit_engagement(item)
|
||||
if item.source == "youtube":
|
||||
return _youtube_engagement(item)
|
||||
if item.source == "tiktok":
|
||||
return _tiktok_engagement(item)
|
||||
weights = ENGAGEMENT_WEIGHTS.get(item.source)
|
||||
if weights:
|
||||
return _weighted_engagement(item, weights)
|
||||
|
||||
Reference in New Issue
Block a user