sawz/last30days-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: surface YouTube + TikTok top comments alongside Reddit (#260)

Browse Source 
* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape

_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.

Shared _remap_comments helper will be reused for TikTok in a later commit.

* feat(tiktok): fetch top comments via ScrapeCreators when opted in

Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.

Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.

* feat(normalize): pass TikTok top_comments through with digg_count->score mapping

Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.

* feat(signals): add YouTube + TikTok top-comment score to engagement formula

Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).

Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.

* feat(render): source-aware thresholds and vote labels for top comments

10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.

* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in

Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
This commit is contained in:
Matt Van Horn
2026-04-15 08:26:06 -04:00
committed by GitHub
parent 242e38ef56
commit 082efe03e3
12 changed files with 678 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_normalize_v3.py
+159
View File
@@ -49,6 +49,165 @@ class NormalizeV3Tests(unittest.TestCase):
)
self.assertEqual([], normalized)
def test_youtube_top_comments_passthrough_with_field_mapping(self):
"""YT comments from enrich_with_comments use likes/text; normalize must
carry them into metadata as the Reddit-compatible {score, excerpt} shape."""
items = [
{
"video_id": "vid-1",
"title": "How to deploy",
"url": "https://youtube.com/watch?v=vid-1",
"channel_name": "Example",
"date": "2026-03-01",
"engagement": {"views": 10000, "likes": 500, "comments": 30},
"top_comments": [
{"author": "Alice", "text": "Best tutorial ever", "likes": 120, "date": "2026-03-02"},
{"author": "Bob", "text": "Helped me ship", "likes": 45, "date": "2026-03-03"},
{"author": "Carol", "text": "Solid walkthrough", "likes": 7, "date": "2026-03-04"},
],
}
]
normalized = normalize.normalize_source_items(
"youtube", items, "2026-02-15", "2026-03-17",
)
self.assertEqual(1, len(normalized))
top = normalized[0].metadata.get("top_comments")
self.assertIsNotNone(top)
self.assertEqual(3, len(top))
# First comment: likes->score, text->excerpt
self.assertEqual(120, top[0]["score"])
self.assertEqual("Best tutorial ever", top[0]["excerpt"])
self.assertEqual("Alice", top[0]["author"])
self.assertEqual("2026-03-02", top[0]["date"])
# Preserves ordering from input (already sorted desc upstream)
self.assertEqual(45, top[1]["score"])
self.assertEqual(7, top[2]["score"])
def test_youtube_top_comments_empty_list_passes_through_cleanly(self):
items = [
{
"video_id": "vid-2",
"title": "Short clip",
"url": "https://youtube.com/watch?v=vid-2",
"channel_name": "Example",
"date": "2026-03-01",
"engagement": {"views": 50, "likes": 2},
"top_comments": [],
}
]
normalized = normalize.normalize_source_items(
"youtube", items, "2026-02-15", "2026-03-17",
)
self.assertEqual(1, len(normalized))
# Empty list is fine; metadata may have empty top_comments or omit it.
top = normalized[0].metadata.get("top_comments", [])
self.assertEqual([], top)
def test_youtube_without_top_comments_key_does_not_crash(self):
items = [
{
"video_id": "vid-3",
"title": "No comments fetched",
"url": "https://youtube.com/watch?v=vid-3",
"channel_name": "Example",
"date": "2026-03-01",
"engagement": {"views": 100, "likes": 5},
}
]
normalized = normalize.normalize_source_items(
"youtube", items, "2026-02-15", "2026-03-17",
)
self.assertEqual(1, len(normalized))
self.assertEqual([], normalized[0].metadata.get("top_comments", []))
def test_youtube_top_comments_feed_top_comment_score_signal(self):
"""Integration: after normalize, signals._top_comment_score should
return log1p(first comment score) for YT, proving the full chain."""
from lib import signals
import math
items = [
{
"video_id": "vid-4",
"title": "Viral comment thread",
"url": "https://youtube.com/watch?v=vid-4",
"channel_name": "Example",
"date": "2026-03-01",
"engagement": {"views": 1000, "likes": 50, "comments": 10},
"top_comments": [
{"author": "A", "text": "Legendary", "likes": 9999, "date": "2026-03-02"},
],
}
]
normalized = normalize.normalize_source_items(
"youtube", items, "2026-02-15", "2026-03-17",
)
self.assertAlmostEqual(math.log1p(9999), signals._top_comment_score(normalized[0]), places=4)
def test_tiktok_top_comments_passthrough_with_digg_count_mapping(self):
"""TikTok comments from enrich_with_comments use digg_count/text;
normalize must map to the shared {score, excerpt} shape."""
items = [
{
"id": "tt-1",
"text": "POV: shipping on Friday",
"url": "https://www.tiktok.com/@u/video/tt-1",
"author_name": "u",
"date": "2026-03-01",
"engagement": {"views": 50000, "likes": 2000, "comments": 300},
"top_comments": [
{"author": "Alice", "text": "dead", "digg_count": 1200, "date": "2026-03-02"},
{"author": "Bob", "text": "so real", "digg_count": 400, "date": "2026-03-03"},
],
}
]
normalized = normalize.normalize_source_items(
"tiktok", items, "2026-02-15", "2026-03-17",
)
self.assertEqual(1, len(normalized))
top = normalized[0].metadata.get("top_comments")
self.assertEqual(2, len(top))
self.assertEqual(1200, top[0]["score"])
self.assertEqual("dead", top[0]["excerpt"])
self.assertEqual("Alice", top[0]["author"])
self.assertEqual(400, top[1]["score"])
def test_tiktok_without_top_comments_does_not_crash(self):
items = [
{
"id": "tt-2",
"text": "plain clip",
"url": "https://www.tiktok.com/@u/video/tt-2",
"author_name": "u",
"date": "2026-03-01",
"engagement": {"views": 1000, "likes": 20},
}
]
normalized = normalize.normalize_source_items(
"tiktok", items, "2026-02-15", "2026-03-17",
)
self.assertEqual([], normalized[0].metadata.get("top_comments", []))
def test_tiktok_top_comments_feed_top_comment_score_signal(self):
from lib import signals
import math
items = [
{
"id": "tt-3",
"text": "viral",
"url": "https://www.tiktok.com/@u/video/tt-3",
"author_name": "u",
"date": "2026-03-01",
"engagement": {"views": 100000, "likes": 5000, "comments": 500},
"top_comments": [
{"author": "A", "text": "this aged well", "digg_count": 50000, "date": "2026-03-02"},
],
}
]
normalized = normalize.normalize_source_items(
"tiktok", items, "2026-02-15", "2026-03-17",
)
self.assertAlmostEqual(math.log1p(50000), signals._top_comment_score(normalized[0]), places=4)
def test_grounding_requires_a_usable_date(self):
items = [
{
tests/test_render_v3.py
+28
View File
@@ -242,6 +242,34 @@ class RenderTopCommentsTests(unittest.TestCase):
self.assertNotIn("Comment (", text)
self.assertNotIn("upvotes)", text)
def test_youtube_comments_use_likes_label_and_50_threshold(self):
comments = [
{"score": 120, "excerpt": "legit fire tutorial", "author": "alice"},
{"score": 60, "excerpt": "saved me hours", "author": "bob"},
{"score": 10, "excerpt": "below threshold", "author": "carol"},
]
report = self._make_report_with_comments(source="youtube", top_comments=comments)
text = render.render_compact(report)
self.assertIn("Comment (120 likes): legit fire tutorial", text)
self.assertIn("Comment (60 likes): saved me hours", text)
self.assertNotIn("Comment (10 likes)", text)
# Render must not silently label YT as upvotes.
self.assertNotIn("Comment (120 upvotes)", text)
def test_tiktok_comments_use_likes_label_and_500_threshold(self):
comments = [
{"score": 2000, "excerpt": "this aged well", "author": "a"},
{"score": 600, "excerpt": "so real", "author": "b"},
{"score": 400, "excerpt": "below tt threshold", "author": "c"},
{"score": 50, "excerpt": "way below", "author": "d"},
]
report = self._make_report_with_comments(source="tiktok", top_comments=comments)
text = render.render_compact(report)
self.assertIn("Comment (2000 likes): this aged well", text)
self.assertIn("Comment (600 likes): so real", text)
self.assertNotIn("Comment (400 likes)", text)
self.assertNotIn("Comment (50 likes)", text)
class RenderBestTakesCompactTests(unittest.TestCase):
"""Tests for Best Takes section in compact output and fun tags on candidates."""
tests/test_signals_v3.py
+102 -9
View File
@@ -28,6 +28,98 @@ class SignalsV3Tests(unittest.TestCase):
)
self.assertAlmostEqual(expected, signals.engagement_raw(item))
def test_youtube_engagement_adds_top_comment_slot(self):
with_comment = schema.SourceItem(
item_id="yt1",
source="youtube",
title="Title",
body="Body",
url="https://youtube.com/watch?v=a",
engagement={"views": 10000, "likes": 500, "comments": 30},
metadata={"top_comments": [{"score": 500}]},
)
without = schema.SourceItem(
item_id="yt2",
source="youtube",
title="Title",
body="Body",
url="https://youtube.com/watch?v=b",
engagement={"views": 10000, "likes": 500, "comments": 30},
metadata={"top_comments": []},
)
with_score = signals.engagement_raw(with_comment)
without_score = signals.engagement_raw(without)
self.assertIsNotNone(with_score)
self.assertIsNotNone(without_score)
self.assertGreater(with_score, without_score)
expected = (
0.45 * math.log1p(10000)
+ 0.32 * math.log1p(500)
+ 0.13 * math.log1p(30)
+ 0.10 * math.log1p(500)
)
self.assertAlmostEqual(expected, with_score, places=6)
def test_youtube_engagement_empty_returns_none(self):
item = schema.SourceItem(
item_id="yt-empty",
source="youtube",
title="Title",
body="Body",
url="https://youtube.com/watch?v=e",
engagement={},
metadata={"top_comments": []},
)
self.assertIsNone(signals.engagement_raw(item))
def test_tiktok_engagement_adds_top_comment_slot(self):
item = schema.SourceItem(
item_id="tt1",
source="tiktok",
title="Title",
body="Body",
url="https://tiktok.com/@u/video/1",
engagement={"views": 100000, "likes": 5000, "comments": 500},
metadata={"top_comments": [{"score": 1200}]},
)
expected = (
0.45 * math.log1p(100000)
+ 0.27 * math.log1p(5000)
+ 0.18 * math.log1p(500)
+ 0.10 * math.log1p(1200)
)
self.assertAlmostEqual(expected, signals.engagement_raw(item), places=6)
def test_youtube_ranking_promotes_viral_comment_thread(self):
"""A moderately-viewed YouTube video with a 10k-like comment should
outrank a slightly-higher-viewed video with no high-signal comments."""
viral_comment = schema.SourceItem(
item_id="yt-with-viral-comment",
source="youtube",
title="Deploy to Fly.io",
body="Deploy to Fly.io walkthrough",
url="https://youtube.com/watch?v=x",
published_at="2026-03-15",
engagement={"views": 5000, "likes": 200, "comments": 50},
metadata={"top_comments": [{"score": 10000}]},
)
higher_views = schema.SourceItem(
item_id="yt-higher-views-no-comment",
source="youtube",
title="Deploy to Fly.io",
body="Deploy to Fly.io walkthrough",
url="https://youtube.com/watch?v=y",
published_at="2026-03-15",
engagement={"views": 8000, "likes": 300, "comments": 60},
metadata={"top_comments": []},
)
ranked = signals.annotate_stream(
[higher_views, viral_comment],
ranking_query="How do I deploy on Fly.io?",
freshness_mode="balanced_recent",
)
self.assertEqual("yt-with-viral-comment", ranked[0].item_id)
def test_polymarket_engagement_uses_market_fields(self):
item = schema.SourceItem(
item_id="pm1",
@@ -221,7 +313,8 @@ class SignalsV3Tests(unittest.TestCase):
self.assertAlmostEqual(expected, result)
def test_youtube_engagement_dominant_weight(self):
"""YouTube: views at 0.50 should dominate over comments at 0.15."""
"""YouTube: views at 0.45 should dominate. With no top-comment data,
the remaining 0.90 of weight is split views/likes/comments 0.45/0.32/0.13."""
item = schema.SourceItem(
item_id="yt1", source="youtube", title="T", body="B",
url="https://example.com",
@@ -230,9 +323,9 @@ class SignalsV3Tests(unittest.TestCase):
result = signals.engagement_raw(item)
self.assertIsNotNone(result)
expected = (
0.50 * math.log1p(10000)
+ 0.35 * math.log1p(500)
+ 0.15 * math.log1p(80)
0.45 * math.log1p(10000)
+ 0.32 * math.log1p(500)
+ 0.13 * math.log1p(80)
)
self.assertAlmostEqual(expected, result)
@@ -252,7 +345,7 @@ class SignalsV3Tests(unittest.TestCase):
)
result = signals.engagement_raw(item)
self.assertIsNotNone(result)
expected = 0.50 * math.log1p(5000)
expected = 0.45 * math.log1p(5000)
self.assertAlmostEqual(expected, result)
def test_tiktok_engagement_dominant_weight(self):
@@ -264,9 +357,9 @@ class SignalsV3Tests(unittest.TestCase):
result = signals.engagement_raw(item)
self.assertIsNotNone(result)
expected = (
0.50 * math.log1p(50000)
+ 0.30 * math.log1p(3000)
+ 0.20 * math.log1p(200)
0.45 * math.log1p(50000)
+ 0.27 * math.log1p(3000)
+ 0.18 * math.log1p(200)
)
self.assertAlmostEqual(expected, result)
@@ -286,7 +379,7 @@ class SignalsV3Tests(unittest.TestCase):
)
result = signals.engagement_raw(item)
self.assertIsNotNone(result)
expected = 0.30 * math.log1p(1000)
expected = 0.27 * math.log1p(1000)
self.assertAlmostEqual(expected, result)
def test_instagram_engagement_dominant_weight(self):
tests/test_tiktok.py
+107
View File
@@ -105,5 +105,112 @@ class TestExpandTikTokQueries(unittest.TestCase):
self.assertEqual(len(queries), 1)
class TestTikTokCommentsGate(unittest.TestCase):
def test_gate_requires_key_and_token(self):
from lib import env
self.assertFalse(env.is_tiktok_comments_available({}))
self.assertFalse(env.is_tiktok_comments_available(
{"SCRAPECREATORS_API_KEY": "k"}
))
self.assertFalse(env.is_tiktok_comments_available(
{"INCLUDE_SOURCES": "tiktok_comments"}
))
self.assertTrue(env.is_tiktok_comments_available(
{"SCRAPECREATORS_API_KEY": "k", "INCLUDE_SOURCES": "tiktok,tiktok_comments"}
))
def test_gate_case_matches_youtube_pattern(self):
from lib import env
# Matches the existing youtube_comments behaviour — plain substring match via _parse_include_sources.
self.assertTrue(env.is_tiktok_comments_available(
{"SCRAPECREATORS_API_KEY": "k", "INCLUDE_SOURCES": "TIKTOK,TIKTOK_COMMENTS"}
))
class TestTikTokEnrichWithComments(unittest.TestCase):
def test_empty_items_returns_empty(self):
from lib import tiktok
self.assertEqual([], tiktok.enrich_with_comments([], token="k"))
def test_missing_token_is_noop(self):
from lib import tiktok
items = [{"video_id": "1", "url": "https://www.tiktok.com/@u/video/1", "engagement": {"views": 100}}]
result = tiktok.enrich_with_comments(items, token="")
self.assertNotIn("top_comments", result[0])
def test_fetch_post_comments_parses_sc_response(self):
from unittest.mock import patch
from lib import tiktok
fake_sc_response = {
"comments": [
{"text": "loved it", "user": {"nickname": "Alice"},
"digg_count": 420, "create_time": 1709251200},
{"text": "meh", "user": {"nickname": "Bob"},
"digg_count": 3, "create_time": 1709251300},
{"text": "", "user": {"nickname": "Skip"},
"digg_count": 999, "create_time": 1709251400},
],
"total": 3,
}
class FakeResp:
def raise_for_status(self):
pass
def json(self):
return fake_sc_response
with patch.object(tiktok, "_requests") as mock_req:
mock_req.get.return_value = FakeResp()
out = tiktok._fetch_post_comments(
"https://www.tiktok.com/@u/video/1",
token="k",
max_comments=5,
)
# Empty-text comment dropped; rest sorted desc by digg_count.
self.assertEqual(2, len(out))
self.assertEqual("loved it", out[0]["text"])
self.assertEqual(420, out[0]["digg_count"])
self.assertEqual("Alice", out[0]["author"])
self.assertEqual("2024-03-01", out[0]["date"])
self.assertEqual(3, out[1]["digg_count"])
def test_fetch_post_comments_swallows_http_error(self):
from unittest.mock import patch
from lib import tiktok
with patch.object(tiktok, "_requests") as mock_req:
mock_req.get.side_effect = Exception("429 rate limit")
out = tiktok._fetch_post_comments(
"https://www.tiktok.com/@u/video/1",
token="k",
max_comments=5,
)
self.assertEqual([], out)
def test_enrich_attaches_top_comments_to_top_ranked_items(self):
from unittest.mock import patch
from lib import tiktok
items = [
{"video_id": "low", "url": "https://www.tiktok.com/@u/video/low",
"engagement": {"views": 10, "likes": 1, "comments": 0}},
{"video_id": "high", "url": "https://www.tiktok.com/@u/video/high",
"engagement": {"views": 10000, "likes": 500, "comments": 30}},
{"video_id": "mid", "url": "https://www.tiktok.com/@u/video/mid",
"engagement": {"views": 1000, "likes": 50, "comments": 5}},
]
with patch.object(tiktok, "_fetch_post_comments") as mock_fetch:
mock_fetch.return_value = [
{"author": "A", "text": "fire", "digg_count": 100, "date": "2024-03-01"}
]
tiktok.enrich_with_comments(items, token="k", max_posts=2)
# High and mid get comments; low does not.
by_id = {i["video_id"]: i for i in items}
self.assertIn("top_comments", by_id["high"])
self.assertIn("top_comments", by_id["mid"])
self.assertNotIn("top_comments", by_id["low"])
if __name__ == "__main__":
unittest.main()