feat: surface YouTube + TikTok top comments alongside Reddit (#260)

* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape _normalize_youtube silently dropped top_comments after enrich_with_comments populated them, so the downstream signals/render/entity layers never saw YouTube comments. Map likes->score and text->excerpt so the existing Reddit-compatible readers Just Work. Shared _remap_comments helper will be reused for TikTok in a later commit. * feat(tiktok): fetch top comments via ScrapeCreators when opted in Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES), tiktok.enrich_with_comments ranks posts and fetches via GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname come across verbatim. Pipeline calls the enricher right after TikTok search when the gate is open. Comment-fetch errors never crash the pipeline — the enricher returns an empty list on 4xx/5xx. * feat(normalize): pass TikTok top_comments through with digg_count->score mapping Instagram uses the same shortform normalizer and has no comment fetcher today, so the key is harmlessly absent there — no Instagram regression. * feat(signals): add YouTube + TikTok top-comment score to engagement formula Mirrors Reddit's 10% top-comment slot. Without top_comments present, the formula reduces to views-dominant weighting; with a high-signal comment, the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92 on the engagement score). Updated the existing dominant-weight and missing-fields tests to the new weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate. * feat(render): source-aware thresholds and vote labels for top comments 10 upvotes on Reddit signals community interest; 10 likes on a viral TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50, tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for YT/TT). First-pass numbers — tune after live observation. * docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in Synthesis instructions previously called out Reddit top comments only. Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote labels (upvotes vs likes), and explicitly frame YT transcript highlights and comments as complementary signals. README and setup-wizard copy document the new tiktok_comments INCLUDE_SOURCES token. --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-15 08:26:06 -04:00
parent 242e38ef56
commit 082efe03e3
12 changed files with 678 additions and 29 deletions
@@ -128,7 +128,7 @@ Say "eli5 on" after any research run. The synthesis rewrites in plain language.

 - **Free Reddit comments.** Public JSON gives you threads + top comments with upvote counts. No API key, no ScrapeCreators. Just works.
 - **YouTube transcripts that actually work.** Widened candidate pool 3x past music videos to reach talk/review content with captions.
- **Threads, Pinterest, YouTube comments.** Opt-in sources via ScrapeCreators. Set `INCLUDE_SOURCES=tiktok,instagram` and add threads, pinterest, youtube_comments for more.
+- **Threads, Pinterest, YouTube + TikTok comments.** Opt-in sources via ScrapeCreators. Set `INCLUDE_SOURCES=tiktok,instagram` and add threads, pinterest, youtube_comments, tiktok_comments for more. `youtube_comments` and `tiktok_comments` surface top comments with vote counts the same way Reddit does.
 - **Perplexity Sonar.** Grounded web search with citations via OpenRouter. Add `OPENROUTER_API_KEY` to unlock.
 - **Polymarket noise filtering.** Common-word disambiguation prevents "Apple" from matching "Will Apple release a car?"
 - **Resilient Reddit.** Timeout budgets and runtime fallback. One slow thread doesn't kill the whole run.
@@ -203,8 +203,8 @@ Your ScrapeCreators key powers TikTok, Instagram, Threads, Pinterest, and YouTub
 **Call AskUserQuestion:**
 Question: "Which ScrapeCreators sources do you want on?"
 Options:
- "TikTok + Instagram (recommended)" - append `INCLUDE_SOURCES=tiktok,instagram` to ~/.config/last30days/.env. Confirm: "TikTok and Instagram are on, plus Reddit backup if public Reddit has issues. You can add threads, pinterest, youtube_comments to INCLUDE_SOURCES anytime."
- "Everything - TikTok, Instagram, Threads, Pinterest, YouTube comments" - append `INCLUDE_SOURCES=tiktok,instagram,threads,pinterest,youtube_comments` to ~/.config/last30days/.env. Confirm: "All ScrapeCreators sources are on."
+- "TikTok + Instagram (recommended)" - append `INCLUDE_SOURCES=tiktok,instagram` to ~/.config/last30days/.env. Confirm: "TikTok and Instagram are on, plus Reddit backup if public Reddit has issues. You can add threads, pinterest, youtube_comments, tiktok_comments to INCLUDE_SOURCES anytime."
+- "Everything - TikTok, Instagram, Threads, Pinterest, YouTube + TikTok comments" - append `INCLUDE_SOURCES=tiktok,instagram,threads,pinterest,youtube_comments,tiktok_comments` to ~/.config/last30days/.env. Confirm: "All ScrapeCreators sources are on."
 - "Just the basics - let's run our first search" - don't write the flag. Confirm: "Got it. ScrapeCreators will serve as Reddit backup. You can add sources to INCLUDE_SOURCES in your .env anytime."

 **After TikTok/Instagram opt-in (or SC skip), show the first research topic modal:**
@@ -244,7 +244,7 @@ YouTube (free, open source):

 Bonus: TikTok, Instagram, Threads, Pinterest, YouTube comments (ScrapeCreators):
 - `SCRAPECREATORS_API_KEY=xxx` - 10,000 free calls at scrapecreators.com.
- After adding your key, set `INCLUDE_SOURCES=tiktok,instagram` to turn on the most popular ones. Add threads, pinterest, youtube_comments for more.
+- After adding your key, set `INCLUDE_SOURCES=tiktok,instagram` to turn on the most popular ones. Add threads, pinterest, youtube_comments, tiktok_comments for more.

 GitHub Issues/PRs (free, no key needed):
 - If you have the `gh` CLI installed (`brew install gh`), GitHub search is automatic. No API key required.
@@ -290,7 +290,7 @@ Create `~/.config/last30days/.env` if it doesn't exist (check first!), pre-popul

 # ScrapeCreators (10,000 free calls - scrapecreators.com):
 # SCRAPECREATORS_API_KEY=    # Unlocks: TikTok, Instagram, Reddit backup (if public Reddit gets rate-limited)
-#                             # Optional: add threads, pinterest, youtube_comments for more
+#                             # Optional: add threads, pinterest, youtube_comments, tiktok_comments for more
 # INCLUDE_SOURCES=tiktok,instagram

 # YouTube: install yt-dlp (brew install yt-dlp) - no key needed
@@ -776,7 +776,7 @@ The script will automatically:

 **Read the ENTIRE output.** It contains EIGHT data sections in this order: Reddit items, X items, YouTube items, TikTok items, Instagram Reels items, Hacker News items, Polymarket items, and WebSearch items. If you miss sections, you will produce incomplete stats.

-**YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, **transcript highlights** (pre-extracted quotable excerpts from the video), and an optional full transcript in a collapsible section. **Quote the highlights directly in your synthesis** - they are the YouTube equivalent of Reddit top comments. Attribute quotes to the channel name. Count them and include them in your synthesis and stats block.
+**YouTube items in the output look like:** `**{video_id}** (score:N) {channel_name} [N views, N likes]` followed by a title, URL, **transcript highlights** (pre-extracted quotable excerpts from the video), and an optional full transcript in a collapsible section. **Quote the highlights directly in your synthesis.** When YouTube items also include top comments (enabled via `youtube_comments`), quote those too with their like counts — they capture how viewers reacted to the video. Transcript highlights and top comments are complementary signals; use both when present. Attribute transcript quotes to the channel name, comment quotes to the commenter. Count them and include them in your synthesis and stats block.

 **TikTok items in the output look like:** `**{TK_id}** (score:N) @{creator} [N views, N likes]` followed by a caption, URL, hashtags, and optional caption snippet. Count them and include them in your synthesis and stats block.

@@ -880,8 +880,8 @@ The Judge Agent must:
 2. Weight YouTube sources HIGH (they have views, likes, and transcript content)
 3. Weight TikTok sources HIGH (they have views, likes, and caption content — viral signal)
 4. Weight WebSearch sources LOWER (no engagement data)
-5. **For Reddit: Pay special attention to top comments** — they often contain the wittiest, most insightful, or funniest take. Quote them directly.
-6. **For YouTube: Quote transcript highlights directly.** Attribute to the channel name.
+5. **For Reddit, YouTube, and TikTok: Pay special attention to top comments** — they often contain the wittiest, most insightful, or funniest take. Quote them directly, attributing to the commenter and including the vote count ("N upvotes" for Reddit, "N likes" for YouTube and TikTok). A top comment with thousands of votes is a stronger community signal than the parent post's stats alone.
+6. **For YouTube: Quote transcript highlights AND top comments.** Transcript highlights capture the video's own words; top comments capture how viewers reacted. Both add value — use them together. Attribute transcript quotes to the channel name.
 7. Identify patterns that appear across ALL sources (strongest signals)
 8. Note any contradictions between sources
 9. **Multi-source clusters (items from 3+ platforms) are the strongest signals.** Lead with these.
@@ -1067,7 +1067,7 @@ CITATION RULE: Cite sources sparingly to prove research is real.

 CITATION PRIORITY (most to least preferred):
 1. @handles from X — "per @handle" (these prove the tool's unique value)
-2. r/subreddits from Reddit — "per r/subreddit" (when citing Reddit, prefer quoting top comments over just the thread title)
+2. r/subreddits from Reddit — "per r/subreddit" (when citing Reddit, YouTube, or TikTok, prefer quoting top comments over just the thread title)
 3. YouTube channels — "per [channel name] on YouTube" (transcript-backed insights)
 4. TikTok creators — "per @creator on TikTok" (viral/trending signal)
 5. Instagram creators — "per @creator on Instagram" (influencer/creator signal)
@@ -441,6 +441,18 @@ def is_youtube_comments_available(config: dict[str, Any]) -> bool:
    return 'youtube_comments' in include


+def is_tiktok_comments_available(config: dict[str, Any]) -> bool:
+    """Check if TikTok comment enrichment is available.
+
+    Requires SCRAPECREATORS_API_KEY AND tiktok_comments in INCLUDE_SOURCES.
+    Mirrors the youtube_comments opt-in pattern.
+    """
+    if not config.get('SCRAPECREATORS_API_KEY'):
+        return False
+    include = _parse_include_sources(config)
+    return 'tiktok_comments' in include
+
+
 def is_youtube_sc_available(config: dict[str, Any]) -> bool:
    """Check if ScrapeCreators YouTube search fallback is available.

@@ -69,6 +69,47 @@ def normalize_source_items(
    return filtered


+def _remap_comments(
+    raw: list[Any],
+    score_keys: tuple[str, ...],
+    excerpt_keys: tuple[str, ...],
+) -> list[dict[str, Any]]:
+    """Normalize comments from any source into the shared Reddit-compatible shape.
+
+    Downstream code (signals._top_comment_score, render._top_comments_list,
+    entity_extract, rerank) all expect `score` and `excerpt`. This helper maps
+    per-source field names (YT: likes/text, TikTok: digg_count/text) onto that
+    shape while preserving author/date/url passthrough.
+    """
+    out: list[dict[str, Any]] = []
+    for raw_c in raw:
+        if not isinstance(raw_c, dict):
+            continue
+        score = _first_present(raw_c, score_keys, default=0)
+        excerpt = _first_present(raw_c, excerpt_keys, default="")
+        try:
+            score_int = int(score or 0)
+        except (TypeError, ValueError):
+            score_int = 0
+        entry: dict[str, Any] = {
+            "score": score_int,
+            "excerpt": str(excerpt or "")[:400],
+            "author": str(raw_c.get("author") or ""),
+            "date": str(raw_c.get("date") or ""),
+        }
+        if raw_c.get("url"):
+            entry["url"] = str(raw_c["url"])
+        out.append(entry)
+    return out
+
+
+def _first_present(d: dict[str, Any], keys: tuple[str, ...], default: Any) -> Any:
+    for key in keys:
+        if key in d and d[key] not in (None, ""):
+            return d[key]
+    return default
+
+
 def _domain_from_url(url: str) -> str | None:
    if not url:
        return None
@@ -200,6 +241,11 @@ def _normalize_youtube(
    metadata: dict[str, Any] = {}
    if highlights:
        metadata["transcript_highlights"] = highlights
+    metadata["top_comments"] = _remap_comments(
+        item.get("top_comments") or [],
+        score_keys=("score", "likes"),
+        excerpt_keys=("excerpt", "text"),
+    )
    return _source_item(
        item_id=str(item.get("video_id") or item.get("id") or f"YT{index + 1}"),
        source=source,
@@ -242,7 +288,16 @@ def _normalize_shortform_video(
        relevance_hint=item.get("relevance", 0.5),
        why_relevant=str(item.get("why_relevant") or ""),
        snippet=caption,
-        metadata={"hashtags": item.get("hashtags") or []},
+        metadata={
+            "hashtags": item.get("hashtags") or [],
+            "top_comments": _remap_comments(
+                item.get("top_comments") or [],
+                # TikTok uses digg_count as the vote field; Instagram has no
+                # comment fetcher today so the key is harmlessly absent.
+                score_keys=("score", "digg_count", "likes"),
+                excerpt_keys=("excerpt", "text"),
+            ),
+        },
    )


@@ -887,7 +887,11 @@ def _retrieve_stream(
            hashtags=tiktok_hashtags,
            creators=tiktok_creators,
        )
-        return tiktok.parse_tiktok_response(result), {}
+        items = tiktok.parse_tiktok_response(result)
+        if items and env.is_tiktok_comments_available(config):
+            sc_token = config.get("SCRAPECREATORS_API_KEY", "")
+            tiktok.enrich_with_comments(items, token=sc_token)
+        return items, {}
    if source == "instagram":
        # Use raw_topic so expand_instagram_queries() generates diverse variants
        # from the original user topic, not the planner's narrowed search_query.
@@ -152,13 +152,14 @@ def render_full(report: schema.Report) -> str:
                lines.append(f"  *{item.container}*")
            if item.snippet:
                lines.append(f"  {item.snippet[:500]}")
-            # Top comments for Reddit
+            # Top comments for Reddit, YouTube, TikTok, HackerNews.
            top_comments = item.metadata.get("top_comments", [])
            if top_comments and isinstance(top_comments[0], dict):
+                vote_label = _vote_label_for(item.source)
                for tc in top_comments[:3]:
                    excerpt = tc.get("excerpt", tc.get("text", ""))[:200]
                    tc_score = tc.get("score", "")
-                    lines.append(f"  Top comment ({tc_score} upvotes): {excerpt}")
+                    lines.append(f"  Top comment ({tc_score} {vote_label}): {excerpt}")
            # Comment insights for Reddit
            insights = item.metadata.get("comment_insights", [])
            if insights:
@@ -276,7 +277,8 @@ def _render_candidate(candidate: schema.Candidate, prefix: str) -> list[str]:
    for tc in _top_comments_list(primary):
        excerpt = tc.get("excerpt") or tc.get("text") or ""
        score = tc.get("score", "")
-        lines.append(f"   - Comment ({score} upvotes): {_truncate(excerpt.strip(), 240)}")
+        vote_label = _vote_label_for(primary.source) if primary else "upvotes"
+        lines.append(f"   - Comment ({score} {vote_label}): {_truncate(excerpt.strip(), 240)}")
    insight = _comment_insight(primary)
    if insight:
        lines.append(f"   - Insight: {_truncate(insight, 220)}")
@@ -582,13 +584,42 @@ def _format_explanation(candidate: schema.Candidate) -> str | None:
    return candidate.explanation


-def _top_comments_list(item: schema.SourceItem | None, limit: int = 3, min_score: int = 10) -> list[dict]:
-    """Return up to `limit` top comments with score >= min_score."""
+# Per-source minimum vote counts for showing a top comment in compact emit.
+# Reddit upvotes, YouTube likes, and TikTok likes are not comparable units —
+# 10 upvotes on Reddit signals genuine community interest, 10 likes on a
+# viral TikTok is noise. First-pass values; tune after live observation.
+_TOP_COMMENT_MIN_SCORE: dict[str, int] = {
+    "reddit": 10,
+    "youtube": 50,
+    "tiktok": 500,
+    "hackernews": 5,
+}
+_TOP_COMMENT_VOTE_LABEL: dict[str, str] = {
+    "reddit": "upvotes",
+    "hackernews": "points",
+    "youtube": "likes",
+    "tiktok": "likes",
+}
+
+
+def _vote_label_for(source: str) -> str:
+    return _TOP_COMMENT_VOTE_LABEL.get(source, "votes")
+
+
+def _top_comments_list(item: schema.SourceItem | None, limit: int = 3, min_score: int | None = None) -> list[dict]:
+    """Return up to `limit` top comments with score at or above the source's minimum.
+
+    If `min_score` is passed explicitly it overrides the per-source default;
+    otherwise the source-keyed map is consulted, with an effective default of 0
+    (always show) for unknown sources so new sources don't get silently hidden.
+    """
    if not item:
        return []
    comments = item.metadata.get("top_comments") or []
    if not comments or not isinstance(comments[0], dict):
        return []
+    if min_score is None:
+        min_score = _TOP_COMMENT_MIN_SCORE.get(item.source, 0)
    return [c for c in comments if (c.get("score") or 0) >= min_score][:limit]


@@ -82,12 +82,11 @@ def _top_comment_score(item: schema.SourceItem) -> float:


 # Per-source engagement weights: list of (field_name, weight) tuples.
-# Reddit uses a custom function because upvote_ratio and top_comment_score
-# are not simple log1p fields.
+# Reddit, YouTube, and TikTok use custom functions because they include
+# a dedicated 10% top-comment-score slot (see _reddit_engagement,
+# _youtube_engagement, _tiktok_engagement).
 ENGAGEMENT_WEIGHTS: dict[str, list[tuple[str, float]]] = {
    "x":            [("likes", 0.55), ("reposts", 0.25), ("replies", 0.15), ("quotes", 0.05)],
-    "youtube":      [("views", 0.50), ("likes", 0.35), ("comments", 0.15)],
-    "tiktok":       [("views", 0.50), ("likes", 0.30), ("comments", 0.20)],
    "instagram":    [("views", 0.50), ("likes", 0.30), ("comments", 0.20)],
    "hackernews":   [("points", 0.55), ("comments", 0.45)],
    "bluesky":      [("likes", 0.40), ("reposts", 0.30), ("replies", 0.20), ("quotes", 0.10)],
@@ -113,6 +112,29 @@ def _reddit_engagement(item: schema.SourceItem) -> float | None:
    return (0.50 * score) + (0.35 * comments) + (0.05 * (ratio * 10.0)) + (0.10 * top_comment)


+def _youtube_engagement(item: schema.SourceItem) -> float | None:
+    views = log1p_safe(item.engagement.get("views"))
+    likes = log1p_safe(item.engagement.get("likes"))
+    comments = log1p_safe(item.engagement.get("comments"))
+    top_comment = _top_comment_score(item)
+    if not any([views, likes, comments, top_comment]):
+        return None
+    # Mirrors Reddit: carve out 10% for top-comment signal, keep view-weight
+    # dominant. Without comments, the pre-change weights (0.50/0.35/0.15)
+    # still govern relative ordering.
+    return (0.45 * views) + (0.32 * likes) + (0.13 * comments) + (0.10 * top_comment)
+
+
+def _tiktok_engagement(item: schema.SourceItem) -> float | None:
+    views = log1p_safe(item.engagement.get("views"))
+    likes = log1p_safe(item.engagement.get("likes"))
+    comments = log1p_safe(item.engagement.get("comments"))
+    top_comment = _top_comment_score(item)
+    if not any([views, likes, comments, top_comment]):
+        return None
+    return (0.45 * views) + (0.27 * likes) + (0.18 * comments) + (0.10 * top_comment)
+
+
 def _generic_engagement(item: schema.SourceItem) -> float | None:
    if not item.engagement:
        return None
@@ -125,6 +147,10 @@ def _generic_engagement(item: schema.SourceItem) -> float | None:
 def engagement_raw(item: schema.SourceItem) -> float | None:
    if item.source == "reddit":
        return _reddit_engagement(item)
+    if item.source == "youtube":
+        return _youtube_engagement(item)
+    if item.source == "tiktok":
+        return _tiktok_engagement(item)
    weights = ENGAGEMENT_WEIGHTS.get(item.source)
    if weights:
        return _weighted_engagement(item, weights)
@@ -539,3 +539,137 @@ def parse_tiktok_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
        List of item dicts ready for normalization.
    """
    return response.get("items", [])
+
+
+def _tiktok_total_engagement(item: Dict[str, Any]) -> int:
+    """Total engagement for ranking which posts deserve comment enrichment."""
+    eng = item.get("engagement", {})
+    return (eng.get("views", 0) or 0) + (eng.get("likes", 0) or 0) + (eng.get("comments", 0) or 0)
+
+
+def enrich_with_comments(
+    items: List[Dict[str, Any]],
+    token: str,
+    max_posts: int = 3,
+    max_comments: int = 5,
+) -> List[Dict[str, Any]]:
+    """Enrich top TikTok posts with comment data from ScrapeCreators.
+
+    For the top N posts by engagement, fetches comments via the SC API
+    and attaches them as a ``top_comments`` field on each item. Mirrors
+    youtube_yt.enrich_with_comments.
+
+    Args:
+        items: TikTok items from search_tiktok()
+        token: ScrapeCreators API key
+        max_posts: How many posts to enrich with comments
+        max_comments: Max comments to keep per post
+
+    Returns:
+        Items list (mutated in place) with top_comments added to enriched items.
+    """
+    if not items or not token or max_posts <= 0:
+        return items
+
+    ranked = sorted(items, key=_tiktok_total_engagement, reverse=True)
+    top_items = ranked[:max_posts]
+    _log(f"Enriching comments for {len(top_items)} TikTok posts")
+
+    from concurrent.futures import ThreadPoolExecutor, as_completed
+
+    def _enrich_one(item: dict) -> bool:
+        post_url = item.get("url", "")
+        if not post_url:
+            return False
+        try:
+            comments = _fetch_post_comments(post_url, token, max_comments)
+            if comments:
+                item["top_comments"] = comments
+                return True
+        except Exception as exc:
+            _log(f"Comment enrichment failed for {post_url}: {exc}")
+        return False
+
+    enriched_count = 0
+    with ThreadPoolExecutor(max_workers=min(4, len(top_items))) as executor:
+        futures = {executor.submit(_enrich_one, item): item for item in top_items}
+        for future in as_completed(futures):
+            if future.result():
+                enriched_count += 1
+
+    _log(f"Enriched {enriched_count}/{len(top_items)} posts with comments")
+    return items
+
+
+def _fetch_post_comments(
+    post_url: str,
+    token: str,
+    max_comments: int = 5,
+) -> List[Dict[str, Any]]:
+    """Fetch comments for a single TikTok post via ScrapeCreators.
+
+    SC endpoint: GET /v1/tiktok/video/comments?url=<video_url>
+    Response shape: { comments: [{text, user.nickname, digg_count, create_time, ...}], cursor, total }
+
+    Args:
+        post_url: Canonical TikTok post URL (share_url form works)
+        token: ScrapeCreators API key
+        max_comments: Maximum comments to return
+
+    Returns:
+        List of comment dicts with author, text, digg_count (likes), date.
+        Empty list on any error — comment failures never crash the pipeline.
+    """
+    if not _requests:
+        try:
+            from urllib.parse import urlencode
+            params = urlencode({"url": post_url, "trim": "true"})
+            url = f"{SCRAPECREATORS_BASE}/video/comments?{params}"
+            headers = http.scrapecreators_headers(token)
+            headers["User-Agent"] = http.USER_AGENT
+            data = http.get(url, headers=headers, timeout=30, retries=2)
+        except Exception as exc:
+            _log(f"Comment fetch error (urllib) for {post_url}: {exc}")
+            return []
+    else:
+        try:
+            resp = _requests.get(
+                f"{SCRAPECREATORS_BASE}/video/comments",
+                params={"url": post_url, "trim": "true"},
+                headers=http.scrapecreators_headers(token),
+                timeout=30,
+            )
+            resp.raise_for_status()
+            data = resp.json()
+        except Exception as exc:
+            _log(f"Comment fetch error for {post_url}: {exc}")
+            return []
+
+    raw_comments = data.get("comments") or data.get("data") or []
+    # Sort by digg_count desc so normalize sees the highest-signal first.
+    raw_comments = sorted(
+        raw_comments,
+        key=lambda c: c.get("digg_count", 0) or 0,
+        reverse=True,
+    )
+    out: List[Dict[str, Any]] = []
+    for c in raw_comments[:max_comments]:
+        text = c.get("text") or ""
+        if not text:
+            continue
+        user = c.get("user") if isinstance(c.get("user"), dict) else {}
+        author = user.get("nickname") or user.get("unique_id") or ""
+        create_time = c.get("create_time")
+        date_str = ""
+        if create_time:
+            try:
+                date_str = dates.timestamp_to_date(int(create_time)) or ""
+            except (ValueError, TypeError):
+                date_str = ""
+        out.append({
+            "author": author,
+            "text": text[:400],
+            "digg_count": c.get("digg_count", 0) or 0,
+            "date": date_str,
+        })
+    return out
@@ -49,6 +49,165 @@ class NormalizeV3Tests(unittest.TestCase):
        )
        self.assertEqual([], normalized)

+    def test_youtube_top_comments_passthrough_with_field_mapping(self):
+        """YT comments from enrich_with_comments use likes/text; normalize must
+        carry them into metadata as the Reddit-compatible {score, excerpt} shape."""
+        items = [
+            {
+                "video_id": "vid-1",
+                "title": "How to deploy",
+                "url": "https://youtube.com/watch?v=vid-1",
+                "channel_name": "Example",
+                "date": "2026-03-01",
+                "engagement": {"views": 10000, "likes": 500, "comments": 30},
+                "top_comments": [
+                    {"author": "Alice", "text": "Best tutorial ever", "likes": 120, "date": "2026-03-02"},
+                    {"author": "Bob", "text": "Helped me ship", "likes": 45, "date": "2026-03-03"},
+                    {"author": "Carol", "text": "Solid walkthrough", "likes": 7, "date": "2026-03-04"},
+                ],
+            }
+        ]
+        normalized = normalize.normalize_source_items(
+            "youtube", items, "2026-02-15", "2026-03-17",
+        )
+        self.assertEqual(1, len(normalized))
+        top = normalized[0].metadata.get("top_comments")
+        self.assertIsNotNone(top)
+        self.assertEqual(3, len(top))
+        # First comment: likes->score, text->excerpt
+        self.assertEqual(120, top[0]["score"])
+        self.assertEqual("Best tutorial ever", top[0]["excerpt"])
+        self.assertEqual("Alice", top[0]["author"])
+        self.assertEqual("2026-03-02", top[0]["date"])
+        # Preserves ordering from input (already sorted desc upstream)
+        self.assertEqual(45, top[1]["score"])
+        self.assertEqual(7, top[2]["score"])
+
+    def test_youtube_top_comments_empty_list_passes_through_cleanly(self):
+        items = [
+            {
+                "video_id": "vid-2",
+                "title": "Short clip",
+                "url": "https://youtube.com/watch?v=vid-2",
+                "channel_name": "Example",
+                "date": "2026-03-01",
+                "engagement": {"views": 50, "likes": 2},
+                "top_comments": [],
+            }
+        ]
+        normalized = normalize.normalize_source_items(
+            "youtube", items, "2026-02-15", "2026-03-17",
+        )
+        self.assertEqual(1, len(normalized))
+        # Empty list is fine; metadata may have empty top_comments or omit it.
+        top = normalized[0].metadata.get("top_comments", [])
+        self.assertEqual([], top)
+
+    def test_youtube_without_top_comments_key_does_not_crash(self):
+        items = [
+            {
+                "video_id": "vid-3",
+                "title": "No comments fetched",
+                "url": "https://youtube.com/watch?v=vid-3",
+                "channel_name": "Example",
+                "date": "2026-03-01",
+                "engagement": {"views": 100, "likes": 5},
+            }
+        ]
+        normalized = normalize.normalize_source_items(
+            "youtube", items, "2026-02-15", "2026-03-17",
+        )
+        self.assertEqual(1, len(normalized))
+        self.assertEqual([], normalized[0].metadata.get("top_comments", []))
+
+    def test_youtube_top_comments_feed_top_comment_score_signal(self):
+        """Integration: after normalize, signals._top_comment_score should
+        return log1p(first comment score) for YT, proving the full chain."""
+        from lib import signals
+        import math
+        items = [
+            {
+                "video_id": "vid-4",
+                "title": "Viral comment thread",
+                "url": "https://youtube.com/watch?v=vid-4",
+                "channel_name": "Example",
+                "date": "2026-03-01",
+                "engagement": {"views": 1000, "likes": 50, "comments": 10},
+                "top_comments": [
+                    {"author": "A", "text": "Legendary", "likes": 9999, "date": "2026-03-02"},
+                ],
+            }
+        ]
+        normalized = normalize.normalize_source_items(
+            "youtube", items, "2026-02-15", "2026-03-17",
+        )
+        self.assertAlmostEqual(math.log1p(9999), signals._top_comment_score(normalized[0]), places=4)
+
+    def test_tiktok_top_comments_passthrough_with_digg_count_mapping(self):
+        """TikTok comments from enrich_with_comments use digg_count/text;
+        normalize must map to the shared {score, excerpt} shape."""
+        items = [
+            {
+                "id": "tt-1",
+                "text": "POV: shipping on Friday",
+                "url": "https://www.tiktok.com/@u/video/tt-1",
+                "author_name": "u",
+                "date": "2026-03-01",
+                "engagement": {"views": 50000, "likes": 2000, "comments": 300},
+                "top_comments": [
+                    {"author": "Alice", "text": "dead", "digg_count": 1200, "date": "2026-03-02"},
+                    {"author": "Bob", "text": "so real", "digg_count": 400, "date": "2026-03-03"},
+                ],
+            }
+        ]
+        normalized = normalize.normalize_source_items(
+            "tiktok", items, "2026-02-15", "2026-03-17",
+        )
+        self.assertEqual(1, len(normalized))
+        top = normalized[0].metadata.get("top_comments")
+        self.assertEqual(2, len(top))
+        self.assertEqual(1200, top[0]["score"])
+        self.assertEqual("dead", top[0]["excerpt"])
+        self.assertEqual("Alice", top[0]["author"])
+        self.assertEqual(400, top[1]["score"])
+
+    def test_tiktok_without_top_comments_does_not_crash(self):
+        items = [
+            {
+                "id": "tt-2",
+                "text": "plain clip",
+                "url": "https://www.tiktok.com/@u/video/tt-2",
+                "author_name": "u",
+                "date": "2026-03-01",
+                "engagement": {"views": 1000, "likes": 20},
+            }
+        ]
+        normalized = normalize.normalize_source_items(
+            "tiktok", items, "2026-02-15", "2026-03-17",
+        )
+        self.assertEqual([], normalized[0].metadata.get("top_comments", []))
+
+    def test_tiktok_top_comments_feed_top_comment_score_signal(self):
+        from lib import signals
+        import math
+        items = [
+            {
+                "id": "tt-3",
+                "text": "viral",
+                "url": "https://www.tiktok.com/@u/video/tt-3",
+                "author_name": "u",
+                "date": "2026-03-01",
+                "engagement": {"views": 100000, "likes": 5000, "comments": 500},
+                "top_comments": [
+                    {"author": "A", "text": "this aged well", "digg_count": 50000, "date": "2026-03-02"},
+                ],
+            }
+        ]
+        normalized = normalize.normalize_source_items(
+            "tiktok", items, "2026-02-15", "2026-03-17",
+        )
+        self.assertAlmostEqual(math.log1p(50000), signals._top_comment_score(normalized[0]), places=4)
+
    def test_grounding_requires_a_usable_date(self):
        items = [
            {
@@ -242,6 +242,34 @@ class RenderTopCommentsTests(unittest.TestCase):
        self.assertNotIn("Comment (", text)
        self.assertNotIn("upvotes)", text)

+    def test_youtube_comments_use_likes_label_and_50_threshold(self):
+        comments = [
+            {"score": 120, "excerpt": "legit fire tutorial", "author": "alice"},
+            {"score": 60, "excerpt": "saved me hours", "author": "bob"},
+            {"score": 10, "excerpt": "below threshold", "author": "carol"},
+        ]
+        report = self._make_report_with_comments(source="youtube", top_comments=comments)
+        text = render.render_compact(report)
+        self.assertIn("Comment (120 likes): legit fire tutorial", text)
+        self.assertIn("Comment (60 likes): saved me hours", text)
+        self.assertNotIn("Comment (10 likes)", text)
+        # Render must not silently label YT as upvotes.
+        self.assertNotIn("Comment (120 upvotes)", text)
+
+    def test_tiktok_comments_use_likes_label_and_500_threshold(self):
+        comments = [
+            {"score": 2000, "excerpt": "this aged well", "author": "a"},
+            {"score": 600, "excerpt": "so real", "author": "b"},
+            {"score": 400, "excerpt": "below tt threshold", "author": "c"},
+            {"score": 50, "excerpt": "way below", "author": "d"},
+        ]
+        report = self._make_report_with_comments(source="tiktok", top_comments=comments)
+        text = render.render_compact(report)
+        self.assertIn("Comment (2000 likes): this aged well", text)
+        self.assertIn("Comment (600 likes): so real", text)
+        self.assertNotIn("Comment (400 likes)", text)
+        self.assertNotIn("Comment (50 likes)", text)
+

 class RenderBestTakesCompactTests(unittest.TestCase):
    """Tests for Best Takes section in compact output and fun tags on candidates."""
@@ -28,6 +28,98 @@ class SignalsV3Tests(unittest.TestCase):
        )
        self.assertAlmostEqual(expected, signals.engagement_raw(item))

+    def test_youtube_engagement_adds_top_comment_slot(self):
+        with_comment = schema.SourceItem(
+            item_id="yt1",
+            source="youtube",
+            title="Title",
+            body="Body",
+            url="https://youtube.com/watch?v=a",
+            engagement={"views": 10000, "likes": 500, "comments": 30},
+            metadata={"top_comments": [{"score": 500}]},
+        )
+        without = schema.SourceItem(
+            item_id="yt2",
+            source="youtube",
+            title="Title",
+            body="Body",
+            url="https://youtube.com/watch?v=b",
+            engagement={"views": 10000, "likes": 500, "comments": 30},
+            metadata={"top_comments": []},
+        )
+        with_score = signals.engagement_raw(with_comment)
+        without_score = signals.engagement_raw(without)
+        self.assertIsNotNone(with_score)
+        self.assertIsNotNone(without_score)
+        self.assertGreater(with_score, without_score)
+        expected = (
+            0.45 * math.log1p(10000)
+            + 0.32 * math.log1p(500)
+            + 0.13 * math.log1p(30)
+            + 0.10 * math.log1p(500)
+        )
+        self.assertAlmostEqual(expected, with_score, places=6)
+
+    def test_youtube_engagement_empty_returns_none(self):
+        item = schema.SourceItem(
+            item_id="yt-empty",
+            source="youtube",
+            title="Title",
+            body="Body",
+            url="https://youtube.com/watch?v=e",
+            engagement={},
+            metadata={"top_comments": []},
+        )
+        self.assertIsNone(signals.engagement_raw(item))
+
+    def test_tiktok_engagement_adds_top_comment_slot(self):
+        item = schema.SourceItem(
+            item_id="tt1",
+            source="tiktok",
+            title="Title",
+            body="Body",
+            url="https://tiktok.com/@u/video/1",
+            engagement={"views": 100000, "likes": 5000, "comments": 500},
+            metadata={"top_comments": [{"score": 1200}]},
+        )
+        expected = (
+            0.45 * math.log1p(100000)
+            + 0.27 * math.log1p(5000)
+            + 0.18 * math.log1p(500)
+            + 0.10 * math.log1p(1200)
+        )
+        self.assertAlmostEqual(expected, signals.engagement_raw(item), places=6)
+
+    def test_youtube_ranking_promotes_viral_comment_thread(self):
+        """A moderately-viewed YouTube video with a 10k-like comment should
+        outrank a slightly-higher-viewed video with no high-signal comments."""
+        viral_comment = schema.SourceItem(
+            item_id="yt-with-viral-comment",
+            source="youtube",
+            title="Deploy to Fly.io",
+            body="Deploy to Fly.io walkthrough",
+            url="https://youtube.com/watch?v=x",
+            published_at="2026-03-15",
+            engagement={"views": 5000, "likes": 200, "comments": 50},
+            metadata={"top_comments": [{"score": 10000}]},
+        )
+        higher_views = schema.SourceItem(
+            item_id="yt-higher-views-no-comment",
+            source="youtube",
+            title="Deploy to Fly.io",
+            body="Deploy to Fly.io walkthrough",
+            url="https://youtube.com/watch?v=y",
+            published_at="2026-03-15",
+            engagement={"views": 8000, "likes": 300, "comments": 60},
+            metadata={"top_comments": []},
+        )
+        ranked = signals.annotate_stream(
+            [higher_views, viral_comment],
+            ranking_query="How do I deploy on Fly.io?",
+            freshness_mode="balanced_recent",
+        )
+        self.assertEqual("yt-with-viral-comment", ranked[0].item_id)
+
    def test_polymarket_engagement_uses_market_fields(self):
        item = schema.SourceItem(
            item_id="pm1",
@@ -221,7 +313,8 @@ class SignalsV3Tests(unittest.TestCase):
        self.assertAlmostEqual(expected, result)

    def test_youtube_engagement_dominant_weight(self):
-        """YouTube: views at 0.50 should dominate over comments at 0.15."""
+        """YouTube: views at 0.45 should dominate. With no top-comment data,
+        the remaining 0.90 of weight is split views/likes/comments 0.45/0.32/0.13."""
        item = schema.SourceItem(
            item_id="yt1", source="youtube", title="T", body="B",
            url="https://example.com",
@@ -230,9 +323,9 @@ class SignalsV3Tests(unittest.TestCase):
        result = signals.engagement_raw(item)
        self.assertIsNotNone(result)
        expected = (
-            0.50 * math.log1p(10000)
-            + 0.35 * math.log1p(500)
-            + 0.15 * math.log1p(80)
+            0.45 * math.log1p(10000)
+            + 0.32 * math.log1p(500)
+            + 0.13 * math.log1p(80)
        )
        self.assertAlmostEqual(expected, result)

@@ -252,7 +345,7 @@ class SignalsV3Tests(unittest.TestCase):
        )
        result = signals.engagement_raw(item)
        self.assertIsNotNone(result)
-        expected = 0.50 * math.log1p(5000)
+        expected = 0.45 * math.log1p(5000)
        self.assertAlmostEqual(expected, result)

    def test_tiktok_engagement_dominant_weight(self):
@@ -264,9 +357,9 @@ class SignalsV3Tests(unittest.TestCase):
        result = signals.engagement_raw(item)
        self.assertIsNotNone(result)
        expected = (
-            0.50 * math.log1p(50000)
-            + 0.30 * math.log1p(3000)
-            + 0.20 * math.log1p(200)
+            0.45 * math.log1p(50000)
+            + 0.27 * math.log1p(3000)
+            + 0.18 * math.log1p(200)
        )
        self.assertAlmostEqual(expected, result)

@@ -286,7 +379,7 @@ class SignalsV3Tests(unittest.TestCase):
        )
        result = signals.engagement_raw(item)
        self.assertIsNotNone(result)
-        expected = 0.30 * math.log1p(1000)
+        expected = 0.27 * math.log1p(1000)
        self.assertAlmostEqual(expected, result)

    def test_instagram_engagement_dominant_weight(self):
@@ -105,5 +105,112 @@ class TestExpandTikTokQueries(unittest.TestCase):
        self.assertEqual(len(queries), 1)


+class TestTikTokCommentsGate(unittest.TestCase):
+    def test_gate_requires_key_and_token(self):
+        from lib import env
+        self.assertFalse(env.is_tiktok_comments_available({}))
+        self.assertFalse(env.is_tiktok_comments_available(
+            {"SCRAPECREATORS_API_KEY": "k"}
+        ))
+        self.assertFalse(env.is_tiktok_comments_available(
+            {"INCLUDE_SOURCES": "tiktok_comments"}
+        ))
+        self.assertTrue(env.is_tiktok_comments_available(
+            {"SCRAPECREATORS_API_KEY": "k", "INCLUDE_SOURCES": "tiktok,tiktok_comments"}
+        ))
+
+    def test_gate_case_matches_youtube_pattern(self):
+        from lib import env
+        # Matches the existing youtube_comments behaviour — plain substring match via _parse_include_sources.
+        self.assertTrue(env.is_tiktok_comments_available(
+            {"SCRAPECREATORS_API_KEY": "k", "INCLUDE_SOURCES": "TIKTOK,TIKTOK_COMMENTS"}
+        ))
+
+
+class TestTikTokEnrichWithComments(unittest.TestCase):
+    def test_empty_items_returns_empty(self):
+        from lib import tiktok
+        self.assertEqual([], tiktok.enrich_with_comments([], token="k"))
+
+    def test_missing_token_is_noop(self):
+        from lib import tiktok
+        items = [{"video_id": "1", "url": "https://www.tiktok.com/@u/video/1", "engagement": {"views": 100}}]
+        result = tiktok.enrich_with_comments(items, token="")
+        self.assertNotIn("top_comments", result[0])
+
+    def test_fetch_post_comments_parses_sc_response(self):
+        from unittest.mock import patch
+        from lib import tiktok
+
+        fake_sc_response = {
+            "comments": [
+                {"text": "loved it", "user": {"nickname": "Alice"},
+                 "digg_count": 420, "create_time": 1709251200},
+                {"text": "meh", "user": {"nickname": "Bob"},
+                 "digg_count": 3, "create_time": 1709251300},
+                {"text": "", "user": {"nickname": "Skip"},
+                 "digg_count": 999, "create_time": 1709251400},
+            ],
+            "total": 3,
+        }
+
+        class FakeResp:
+            def raise_for_status(self):
+                pass
+            def json(self):
+                return fake_sc_response
+
+        with patch.object(tiktok, "_requests") as mock_req:
+            mock_req.get.return_value = FakeResp()
+            out = tiktok._fetch_post_comments(
+                "https://www.tiktok.com/@u/video/1",
+                token="k",
+                max_comments=5,
+            )
+        # Empty-text comment dropped; rest sorted desc by digg_count.
+        self.assertEqual(2, len(out))
+        self.assertEqual("loved it", out[0]["text"])
+        self.assertEqual(420, out[0]["digg_count"])
+        self.assertEqual("Alice", out[0]["author"])
+        self.assertEqual("2024-03-01", out[0]["date"])
+        self.assertEqual(3, out[1]["digg_count"])
+
+    def test_fetch_post_comments_swallows_http_error(self):
+        from unittest.mock import patch
+        from lib import tiktok
+
+        with patch.object(tiktok, "_requests") as mock_req:
+            mock_req.get.side_effect = Exception("429 rate limit")
+            out = tiktok._fetch_post_comments(
+                "https://www.tiktok.com/@u/video/1",
+                token="k",
+                max_comments=5,
+            )
+        self.assertEqual([], out)
+
+    def test_enrich_attaches_top_comments_to_top_ranked_items(self):
+        from unittest.mock import patch
+        from lib import tiktok
+
+        items = [
+            {"video_id": "low", "url": "https://www.tiktok.com/@u/video/low",
+             "engagement": {"views": 10, "likes": 1, "comments": 0}},
+            {"video_id": "high", "url": "https://www.tiktok.com/@u/video/high",
+             "engagement": {"views": 10000, "likes": 500, "comments": 30}},
+            {"video_id": "mid", "url": "https://www.tiktok.com/@u/video/mid",
+             "engagement": {"views": 1000, "likes": 50, "comments": 5}},
+        ]
+        with patch.object(tiktok, "_fetch_post_comments") as mock_fetch:
+            mock_fetch.return_value = [
+                {"author": "A", "text": "fire", "digg_count": 100, "date": "2024-03-01"}
+            ]
+            tiktok.enrich_with_comments(items, token="k", max_posts=2)
+        # High and mid get comments; low does not.
+        by_id = {i["video_id"]: i for i in items}
+        self.assertIn("top_comments", by_id["high"])
+        self.assertIn("top_comments", by_id["mid"])
+        self.assertNotIn("top_comments", by_id["low"])
+
+
 if __name__ == "__main__":
    unittest.main()