feat: surface YouTube + TikTok top comments alongside Reddit (#260)
* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape
_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.
Shared _remap_comments helper will be reused for TikTok in a later commit.
* feat(tiktok): fetch top comments via ScrapeCreators when opted in
Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.
Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.
* feat(normalize): pass TikTok top_comments through with digg_count->score mapping
Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.
* feat(signals): add YouTube + TikTok top-comment score to engagement formula
Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).
Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.
* feat(render): source-aware thresholds and vote labels for top comments
10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.
* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in
Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
This commit is contained in:
@@ -242,6 +242,34 @@ class RenderTopCommentsTests(unittest.TestCase):
|
||||
self.assertNotIn("Comment (", text)
|
||||
self.assertNotIn("upvotes)", text)
|
||||
|
||||
def test_youtube_comments_use_likes_label_and_50_threshold(self):
|
||||
comments = [
|
||||
{"score": 120, "excerpt": "legit fire tutorial", "author": "alice"},
|
||||
{"score": 60, "excerpt": "saved me hours", "author": "bob"},
|
||||
{"score": 10, "excerpt": "below threshold", "author": "carol"},
|
||||
]
|
||||
report = self._make_report_with_comments(source="youtube", top_comments=comments)
|
||||
text = render.render_compact(report)
|
||||
self.assertIn("Comment (120 likes): legit fire tutorial", text)
|
||||
self.assertIn("Comment (60 likes): saved me hours", text)
|
||||
self.assertNotIn("Comment (10 likes)", text)
|
||||
# Render must not silently label YT as upvotes.
|
||||
self.assertNotIn("Comment (120 upvotes)", text)
|
||||
|
||||
def test_tiktok_comments_use_likes_label_and_500_threshold(self):
|
||||
comments = [
|
||||
{"score": 2000, "excerpt": "this aged well", "author": "a"},
|
||||
{"score": 600, "excerpt": "so real", "author": "b"},
|
||||
{"score": 400, "excerpt": "below tt threshold", "author": "c"},
|
||||
{"score": 50, "excerpt": "way below", "author": "d"},
|
||||
]
|
||||
report = self._make_report_with_comments(source="tiktok", top_comments=comments)
|
||||
text = render.render_compact(report)
|
||||
self.assertIn("Comment (2000 likes): this aged well", text)
|
||||
self.assertIn("Comment (600 likes): so real", text)
|
||||
self.assertNotIn("Comment (400 likes)", text)
|
||||
self.assertNotIn("Comment (50 likes)", text)
|
||||
|
||||
|
||||
class RenderBestTakesCompactTests(unittest.TestCase):
|
||||
"""Tests for Best Takes section in compact output and fun tags on candidates."""
|
||||
|
||||
Reference in New Issue
Block a user