feat: surface YouTube + TikTok top comments alongside Reddit (#260)

* feat(normalize): pass YouTube top_comments through with Reddit-compatible shape

_normalize_youtube silently dropped top_comments after enrich_with_comments
populated them, so the downstream signals/render/entity layers never saw
YouTube comments. Map likes->score and text->excerpt so the existing
Reddit-compatible readers Just Work.

Shared _remap_comments helper will be reused for TikTok in a later commit.

* feat(tiktok): fetch top comments via ScrapeCreators when opted in

Mirrors the youtube_comments pattern: new env.is_tiktok_comments_available
gate (requires SCRAPECREATORS_API_KEY + tiktok_comments in INCLUDE_SOURCES),
tiktok.enrich_with_comments ranks posts and fetches via
GET /v1/tiktok/video/comments. Vote field is digg_count; text and user.nickname
come across verbatim. Pipeline calls the enricher right after TikTok search
when the gate is open.

Comment-fetch errors never crash the pipeline — the enricher returns an
empty list on 4xx/5xx.

* feat(normalize): pass TikTok top_comments through with digg_count->score mapping

Instagram uses the same shortform normalizer and has no comment fetcher
today, so the key is harmlessly absent there — no Instagram regression.

* feat(signals): add YouTube + TikTok top-comment score to engagement formula

Mirrors Reddit's 10% top-comment slot. Without top_comments present, the
formula reduces to views-dominant weighting; with a high-signal comment,
the item gets a meaningful bump (log1p(10k) ~ 9.2, weighted 0.10 = ~0.92
on the engagement score).

Updated the existing dominant-weight and missing-fields tests to the new
weights (0.45/0.32/0.13 for YT, 0.45/0.27/0.18 for TT). Views still dominate.

* feat(render): source-aware thresholds and vote labels for top comments

10 upvotes on Reddit signals community interest; 10 likes on a viral
TikTok is noise. Introduce per-source minimums (reddit 10, youtube 50,
tiktok 500) and native vote labels ('upvotes' for Reddit, 'likes' for
YT/TT). First-pass numbers — tune after live observation.

* docs: generalize top-comment quoting to YouTube + TikTok, add tiktok_comments opt-in

Synthesis instructions previously called out Reddit top comments only.
Now cover Reddit/YouTube/TikTok uniformly with source-appropriate vote
labels (upvotes vs likes), and explicitly frame YT transcript highlights
and comments as complementary signals. README and setup-wizard copy
document the new tiktok_comments INCLUDE_SOURCES token.

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>

scripts/lib/tiktok.py

@@ -539,3 +539,137 @@ def parse_tiktok_response(response: Dict[str, Any]) -> List[Dict[str, Any]]:
         List of item dicts ready for normalization.
     """
     return response.get("items", [])
+
+
+def _tiktok_total_engagement(item: Dict[str, Any]) -> int:
+    """Total engagement for ranking which posts deserve comment enrichment."""
+    eng = item.get("engagement", {})
+    return (eng.get("views", 0) or 0) + (eng.get("likes", 0) or 0) + (eng.get("comments", 0) or 0)
+
+
+def enrich_with_comments(
+    items: List[Dict[str, Any]],
+    token: str,
+    max_posts: int = 3,
+    max_comments: int = 5,
+) -> List[Dict[str, Any]]:
+    """Enrich top TikTok posts with comment data from ScrapeCreators.
+
+    For the top N posts by engagement, fetches comments via the SC API
+    and attaches them as a ``top_comments`` field on each item. Mirrors
+    youtube_yt.enrich_with_comments.
+
+    Args:
+        items: TikTok items from search_tiktok()
+        token: ScrapeCreators API key
+        max_posts: How many posts to enrich with comments
+        max_comments: Max comments to keep per post
+
+    Returns:
+        Items list (mutated in place) with top_comments added to enriched items.
+    """
+    if not items or not token or max_posts <= 0:
+        return items
+
+    ranked = sorted(items, key=_tiktok_total_engagement, reverse=True)
+    top_items = ranked[:max_posts]
+    _log(f"Enriching comments for {len(top_items)} TikTok posts")
+
+    from concurrent.futures import ThreadPoolExecutor, as_completed
+
+    def _enrich_one(item: dict) -> bool:
+        post_url = item.get("url", "")
+        if not post_url:
+            return False
+        try:
+            comments = _fetch_post_comments(post_url, token, max_comments)
+            if comments:
+                item["top_comments"] = comments
+                return True
+        except Exception as exc:
+            _log(f"Comment enrichment failed for {post_url}: {exc}")
+        return False
+
+    enriched_count = 0
+    with ThreadPoolExecutor(max_workers=min(4, len(top_items))) as executor:
+        futures = {executor.submit(_enrich_one, item): item for item in top_items}
+        for future in as_completed(futures):
+            if future.result():
+                enriched_count += 1
+
+    _log(f"Enriched {enriched_count}/{len(top_items)} posts with comments")
+    return items
+
+
+def _fetch_post_comments(
+    post_url: str,
+    token: str,
+    max_comments: int = 5,
+) -> List[Dict[str, Any]]:
+    """Fetch comments for a single TikTok post via ScrapeCreators.
+
+    SC endpoint: GET /v1/tiktok/video/comments?url=<video_url>
+    Response shape: { comments: [{text, user.nickname, digg_count, create_time, ...}], cursor, total }
+
+    Args:
+        post_url: Canonical TikTok post URL (share_url form works)
+        token: ScrapeCreators API key
+        max_comments: Maximum comments to return
+
+    Returns:
+        List of comment dicts with author, text, digg_count (likes), date.
+        Empty list on any error — comment failures never crash the pipeline.
+    """
+    if not _requests:
+        try:
+            from urllib.parse import urlencode
+            params = urlencode({"url": post_url, "trim": "true"})
+            url = f"{SCRAPECREATORS_BASE}/video/comments?{params}"
+            headers = http.scrapecreators_headers(token)
+            headers["User-Agent"] = http.USER_AGENT
+            data = http.get(url, headers=headers, timeout=30, retries=2)
+        except Exception as exc:
+            _log(f"Comment fetch error (urllib) for {post_url}: {exc}")
+            return []
+    else:
+        try:
+            resp = _requests.get(
+                f"{SCRAPECREATORS_BASE}/video/comments",
+                params={"url": post_url, "trim": "true"},
+                headers=http.scrapecreators_headers(token),
+                timeout=30,
+            )
+            resp.raise_for_status()
+            data = resp.json()
+        except Exception as exc:
+            _log(f"Comment fetch error for {post_url}: {exc}")
+            return []
+
+    raw_comments = data.get("comments") or data.get("data") or []
+    # Sort by digg_count desc so normalize sees the highest-signal first.
+    raw_comments = sorted(
+        raw_comments,
+        key=lambda c: c.get("digg_count", 0) or 0,
+        reverse=True,
+    )
+    out: List[Dict[str, Any]] = []
+    for c in raw_comments[:max_comments]:
+        text = c.get("text") or ""
+        if not text:
+            continue
+        user = c.get("user") if isinstance(c.get("user"), dict) else {}
+        author = user.get("nickname") or user.get("unique_id") or ""
+        create_time = c.get("create_time")
+        date_str = ""
+        if create_time:
+            try:
+                date_str = dates.timestamp_to_date(int(create_time)) or ""
+            except (ValueError, TypeError):
+                date_str = ""
+        out.append({
+            "author": author,
+            "text": text[:400],
+            "digg_count": c.get("digg_count", 0) or 0,
+            "date": date_str,
+        })
+    return out