feat: surface YouTube + TikTok top comments alongside Reddit (#260)

* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape

_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.

Shared _remap_comments helper will be reused for TikTok in a later commit.

* feat(tiktok): fetch top comments via ScrapeCreators when opted in

Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.

Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.

* feat(normalize): pass TikTok top_comments through with digg_count->score mapping

Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.

* feat(signals): add YouTube + TikTok top-comment score to engagement formula

Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).

Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.

* feat(render): source-aware thresholds and vote labels for top comments

10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.

* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in

Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
This commit is contained in:
Matt Van Horn
2026-04-15 08:26:06 -04:00
committed by GitHub
parent 242e38ef56
commit 082efe03e3
12 changed files with 678 additions and 29 deletions
+28
View File
@@ -242,6 +242,34 @@ class RenderTopCommentsTests(unittest.TestCase):
self.assertNotIn("Comment (", text)
self.assertNotIn("upvotes)", text)
def test_youtube_comments_use_likes_label_and_50_threshold(self):
comments = [
{"score": 120, "excerpt": "legit fire tutorial", "author": "alice"},
{"score": 60, "excerpt": "saved me hours", "author": "bob"},
{"score": 10, "excerpt": "below threshold", "author": "carol"},
]
report = self._make_report_with_comments(source="youtube", top_comments=comments)
text = render.render_compact(report)
self.assertIn("Comment (120 likes): legit fire tutorial", text)
self.assertIn("Comment (60 likes): saved me hours", text)
self.assertNotIn("Comment (10 likes)", text)
# Render must not silently label YT as upvotes.
self.assertNotIn("Comment (120 upvotes)", text)
def test_tiktok_comments_use_likes_label_and_500_threshold(self):
comments = [
{"score": 2000, "excerpt": "this aged well", "author": "a"},
{"score": 600, "excerpt": "so real", "author": "b"},
{"score": 400, "excerpt": "below tt threshold", "author": "c"},
{"score": 50, "excerpt": "way below", "author": "d"},
]
report = self._make_report_with_comments(source="tiktok", top_comments=comments)
text = render.render_compact(report)
self.assertIn("Comment (2000 likes): this aged well", text)
self.assertIn("Comment (600 likes): so real", text)
self.assertNotIn("Comment (400 likes)", text)
self.assertNotIn("Comment (50 likes)", text)
class RenderBestTakesCompactTests(unittest.TestCase):
"""Tests for Best Takes section in compact output and fun tags on candidates."""