Matt Van Horn
db75f9e341
Add Instagram Reels as the 8th research source via ScrapeCreators API. One API key (SCRAPECREATORS_API_KEY) now covers both TikTok and Instagram. - Add scripts/lib/instagram.py: keyword search, transcript extraction, relevance scoring, engagement metrics (views, likes, comments) - Add InstagramItem to schema, normalization, scoring, dedup, rendering - Add Instagram to orchestrator pipeline, watchlist, and UI spinners - Update SKILL.md: stats template, citation priority, item format, URL-to-name extraction rules, anti-Sources instruction - Update README and CHANGELOG for v2.8 - Fix: Instagram/TikTok not running in --search= web-only path - Fix: web stats line showing full URLs instead of domain names - Replace APIFY_API_TOKEN with SCRAPECREATORS_API_KEY throughout Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
157 lines
7.3 KiB
Markdown
157 lines
7.3 KiB
Markdown
# refactor: Replace Apify with pay-as-you-go TikTok API
|
|
|
|
**Type:** refactor
|
|
**Date:** 2026-03-03
|
|
**Status:** Draft
|
|
|
|
## Problem
|
|
|
|
Apify requires a monthly subscription even for low-volume usage. We need a true pay-as-you-go TikTok data API that returns structured video data (views, likes, comments, shares, author, hashtags, captions) from keyword search.
|
|
|
|
## Current Architecture
|
|
|
|
Our TikTok integration makes **exactly 2 API calls per /last30days invocation**:
|
|
|
|
1. **Search call** — keyword search, returns 10-40 videos with engagement metrics
|
|
2. **Caption enrichment call** — fetches spoken-word subtitles for top 3-8 videos
|
|
|
|
We consume these fields from the response:
|
|
- `id`, `text` (description), `webVideoUrl`, `authorMeta.name`
|
|
- `playCount`, `diggCount`, `commentCount`, `shareCount`
|
|
- `hashtags[].name`, `videoMeta.duration`
|
|
- `createTimeISO` or `createTime` (date)
|
|
- `subtitleText` / `subtitles` (captions, secondary call)
|
|
|
|
Files involved:
|
|
- `scripts/lib/tiktok.py` — search, parse, relevance scoring, caption fetching
|
|
- `scripts/lib/apify_client_wrapper.py` — shared Apify client (also designed for future FB/IG)
|
|
- `scripts/lib/schema.py` — `TikTokItem` dataclass
|
|
- `scripts/lib/normalize.py` — `normalize_tiktok()`
|
|
- `scripts/last30days.py` — orchestrator (calls `tiktok.search_and_enrich()`)
|
|
|
|
## Why Most of Those Alternatives Won't Work
|
|
|
|
The services in the user's table (Spider, Zyte, Scrappey, Serper.dev) are **general web scrapers** — they return raw HTML, not structured TikTok data. We'd have to build our own HTML parser, handle anti-bot protections, and reverse-engineer TikTok's response format. That's a completely different (and fragile) approach.
|
|
|
|
What we need is a **TikTok-specific API** that returns structured JSON with engagement metrics from keyword search.
|
|
|
|
## Alternatives Evaluated
|
|
|
|
### RECOMMENDED: ScrapeCreators — Best True Pay-As-You-Go
|
|
|
|
| Attribute | Details |
|
|
|-----------|---------|
|
|
| **TikTok structured data** | Yes — 19 dedicated endpoints including keyword search |
|
|
| **Pricing model** | True PAYG — buy credits, credits never expire |
|
|
| **Cost at our volume** | ~$0.60/month ($10 buys 5,000 credits, lasts 16-33 months) |
|
|
| **Free tier** | 100-10,000 free credits on signup (no credit card) |
|
|
| **Python SDK** | No dedicated SDK — simple REST API (`requests.get()`) |
|
|
| **Search endpoint** | "Search by Keyword" and "Top Search" |
|
|
| **Risk** | Newer service, limited track record |
|
|
|
|
**Why it's the best fit:** $10 literally lasts over a year at our volume. No subscription, no expiring credits. The lack of a Python SDK is irrelevant — it's a single `requests.get()` call.
|
|
|
|
### Runner-up: EnsembleData — Best SDK, But Subscription
|
|
|
|
| Attribute | Details |
|
|
|-----------|---------|
|
|
| **TikTok structured data** | Full — 15+ endpoints, all engagement metrics |
|
|
| **Pricing model** | Monthly subscription ($100/mo after 7-day trial) |
|
|
| **Cost at our volume** | $0 during trial (50 units/day), $100/mo after |
|
|
| **Free tier** | 50 units/day for 7 days, no CC required |
|
|
| **Python SDK** | Yes — `pip install ensembledata` |
|
|
| **Search endpoint** | "Keyword Search" returns ~20 posts/call (1 unit) |
|
|
| **Risk** | $100/mo is overkill for 5-10 searches/day |
|
|
|
|
**Verdict:** Best data quality and SDK, but $100/mo is absurd for our ~150-300 requests/month. Same subscription problem as Apify.
|
|
|
|
### Backup: tikwm.com — Free but Risky
|
|
|
|
| Attribute | Details |
|
|
|-----------|---------|
|
|
| **TikTok structured data** | Likely yes (needs live testing) |
|
|
| **Pricing model** | Completely free, no API key required |
|
|
| **Cost at our volume** | $0 |
|
|
| **Free tier** | 5,000 requests/day |
|
|
| **Python SDK** | Community wrappers (damirTAG/TikTok-Module, kittenbark/tikwm) |
|
|
| **Search endpoint** | `https://www.tikwm.com/api/feed/search?keywords=TERM&count=20` |
|
|
| **Risk** | Unaffiliated third-party, could disappear anytime, no SLA |
|
|
|
|
**Verdict:** Great for development/testing. Too risky as sole production backend. Could be a zero-cost fallback.
|
|
|
|
### Not Recommended
|
|
|
|
| Service | Why Not |
|
|
|---------|---------|
|
|
| **TikAPI** | $50-189/mo subscription — same problem as Apify |
|
|
| **Bright Data** | $499/mo minimum — enterprise pricing |
|
|
| **davidteather/TikTok-Api** | Video search is broken, requires Playwright, fragile |
|
|
| **Spider, Zyte, Scrappey** | Generic scrapers — return raw HTML, no TikTok structure |
|
|
| **Piloterr** | No TikTok endpoints, subscription only |
|
|
|
|
## Recommended Approach
|
|
|
|
### Option A: ScrapeCreators as primary (Recommended)
|
|
|
|
- [ ] Sign up for ScrapeCreators, get free credits
|
|
- [ ] Test the "Search by Keyword" endpoint to verify it returns all required fields
|
|
- [ ] Refactor `tiktok.py` to use ScrapeCreators REST API instead of Apify actor
|
|
- [ ] Keep `apify_client_wrapper.py` for future FB/IG (or refactor to generic wrapper)
|
|
- [ ] Update `.env` config: `SCRAPECREATORS_API_KEY` replaces `APIFY_API_TOKEN` for TikTok
|
|
- [ ] Update README, SKILL.md installation instructions
|
|
- [ ] Buy $10 credits after confirming it works
|
|
|
|
### Option B: tikwm.com as primary (Zero cost, higher risk)
|
|
|
|
- [ ] Test tikwm.com search endpoint to verify response schema
|
|
- [ ] If it returns engagement metrics, implement as primary backend
|
|
- [ ] Add ScrapeCreators as paid fallback when tikwm fails
|
|
- [ ] No API key required — simplest user setup
|
|
|
|
### Option C: Keep Apify, document the subscription requirement
|
|
|
|
- [ ] Update README to clarify Apify requires a paid plan
|
|
- [ ] Add note about the free $5/mo credits tier (if it still works without subscription)
|
|
- [ ] Ship as-is with clear billing expectations
|
|
|
|
## Implementation Plan (Option A)
|
|
|
|
### Phase 1: Validate ScrapeCreators API
|
|
|
|
- [ ] Sign up and get API key
|
|
- [ ] Test keyword search endpoint: `GET /tiktok/search?keyword={topic}&count=20`
|
|
- [ ] Verify response contains: video ID, play count, likes, comments, shares, author, hashtags, date, description
|
|
- [ ] Test caption/subtitle availability (or confirm description text is sufficient)
|
|
|
|
### Phase 2: Swap the Backend
|
|
|
|
- [ ] Create `scripts/lib/scrapecreators_client.py` (simple REST wrapper, ~40 lines)
|
|
- [ ] Refactor `tiktok.py:search_tiktok()` to call ScrapeCreators instead of Apify
|
|
- [ ] Map ScrapeCreators response fields to our existing item dict format
|
|
- [ ] Refactor `tiktok.py:fetch_captions()` — check if ScrapeCreators provides subtitles, else use description text only
|
|
- [ ] Update `env.py` to read `SCRAPECREATORS_API_KEY` (keep `APIFY_API_TOKEN` for backward compat)
|
|
- [ ] Update `scripts/lib/ui.py` spinner messages if needed
|
|
|
|
### Phase 3: Update Docs & Ship
|
|
|
|
- [ ] Update README.md installation section (new API key)
|
|
- [ ] Update SKILL.md
|
|
- [ ] Update `~/.config/last30days/.env` locally
|
|
- [ ] Run full test suite
|
|
- [ ] Commit and push
|
|
|
|
## Acceptance Criteria
|
|
|
|
- [ ] TikTok search returns structured data with views, likes, comments, shares
|
|
- [ ] No subscription required — pay-as-you-go only
|
|
- [ ] Cost < $1/month at normal usage (5-10 searches/day)
|
|
- [ ] Existing test suite passes with new backend
|
|
- [ ] Graceful degradation when API key is missing (same behavior as today)
|
|
|
|
## Open Questions
|
|
|
|
1. Does ScrapeCreators' keyword search support date filtering, or do we filter post-API like we do with Apify?
|
|
2. Does ScrapeCreators return subtitle/caption data, or just video descriptions?
|
|
3. What's the response time? Apify actors took 30-120 seconds. REST APIs should be faster.
|
|
4. Should we keep Apify as a fallback backend (user configures one or the other)?
|