sawz/last30days-skill
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: v2.8 — Instagram Reels source + TikTok ScrapeCreators migration

Browse Source 
Add Instagram Reels as the 8th research source via ScrapeCreators API.
One API key (SCRAPECREATORS_API_KEY) now covers both TikTok and Instagram.

- Add scripts/lib/instagram.py: keyword search, transcript extraction,
  relevance scoring, engagement metrics (views, likes, comments)
- Add InstagramItem to schema, normalization, scoring, dedup, rendering
- Add Instagram to orchestrator pipeline, watchlist, and UI spinners
- Update SKILL.md: stats template, citation priority, item format,
  URL-to-name extraction rules, anti-Sources instruction
- Update README and CHANGELOG for v2.8
- Fix: Instagram/TikTok not running in --search= web-only path
- Fix: web stats line showing full URLs instead of domain names
- Replace APIFY_API_TOKEN with SCRAPECREATORS_API_KEY throughout

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Matt Van Horn
2026-03-04 06:57:52 -08:00
parent 740dcc5789
commit db75f9e341
17 changed files with 1609 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-03-03-refactor-tiktok-apify-alternative-plan.md
+156
View File
@@ -0,0 +1,156 @@
# refactor: Replace Apify with pay-as-you-go TikTok API
**Type:** refactor
**Date:** 2026-03-03
**Status:** Draft
## Problem
Apify requires a monthly subscription even for low-volume usage. We need a true pay-as-you-go TikTok data API that returns structured video data (views, likes, comments, shares, author, hashtags, captions) from keyword search.
## Current Architecture
Our TikTok integration makes **exactly 2 API calls per /last30days invocation**:
1. **Search call** — keyword search, returns 10-40 videos with engagement metrics
2. **Caption enrichment call** — fetches spoken-word subtitles for top 3-8 videos
We consume these fields from the response:
- `id`, `text` (description), `webVideoUrl`, `authorMeta.name`
- `playCount`, `diggCount`, `commentCount`, `shareCount`
- `hashtags[].name`, `videoMeta.duration`
- `createTimeISO` or `createTime` (date)
- `subtitleText` / `subtitles` (captions, secondary call)
Files involved:
- `scripts/lib/tiktok.py` — search, parse, relevance scoring, caption fetching
- `scripts/lib/apify_client_wrapper.py` — shared Apify client (also designed for future FB/IG)
- `scripts/lib/schema.py` — `TikTokItem` dataclass
- `scripts/lib/normalize.py` — `normalize_tiktok()`
- `scripts/last30days.py` — orchestrator (calls `tiktok.search_and_enrich()`)
## Why Most of Those Alternatives Won't Work
The services in the user's table (Spider, Zyte, Scrappey, Serper.dev) are **general web scrapers** — they return raw HTML, not structured TikTok data. We'd have to build our own HTML parser, handle anti-bot protections, and reverse-engineer TikTok's response format. That's a completely different (and fragile) approach.
What we need is a **TikTok-specific API** that returns structured JSON with engagement metrics from keyword search.
## Alternatives Evaluated
### RECOMMENDED: ScrapeCreators — Best True Pay-As-You-Go
| Attribute | Details |
|-----------|---------|
| **TikTok structured data** | Yes — 19 dedicated endpoints including keyword search |
| **Pricing model** | True PAYG — buy credits, credits never expire |
| **Cost at our volume** | ~$0.60/month ($10 buys 5,000 credits, lasts 16-33 months) |
| **Free tier** | 100-10,000 free credits on signup (no credit card) |
| **Python SDK** | No dedicated SDK — simple REST API (`requests.get()`) |
| **Search endpoint** | "Search by Keyword" and "Top Search" |
| **Risk** | Newer service, limited track record |
**Why it's the best fit:** $10 literally lasts over a year at our volume. No subscription, no expiring credits. The lack of a Python SDK is irrelevant — it's a single `requests.get()` call.
### Runner-up: EnsembleData — Best SDK, But Subscription
| Attribute | Details |
|-----------|---------|
| **TikTok structured data** | Full — 15+ endpoints, all engagement metrics |
| **Pricing model** | Monthly subscription ($100/mo after 7-day trial) |
| **Cost at our volume** | $0 during trial (50 units/day), $100/mo after |
| **Free tier** | 50 units/day for 7 days, no CC required |
| **Python SDK** | Yes — `pip install ensembledata` |
| **Search endpoint** | "Keyword Search" returns ~20 posts/call (1 unit) |
| **Risk** | $100/mo is overkill for 5-10 searches/day |
**Verdict:** Best data quality and SDK, but $100/mo is absurd for our ~150-300 requests/month. Same subscription problem as Apify.
### Backup: tikwm.com — Free but Risky
| Attribute | Details |
|-----------|---------|
| **TikTok structured data** | Likely yes (needs live testing) |
| **Pricing model** | Completely free, no API key required |
| **Cost at our volume** | $0 |
| **Free tier** | 5,000 requests/day |
| **Python SDK** | Community wrappers (damirTAG/TikTok-Module, kittenbark/tikwm) |
| **Search endpoint** | `https://www.tikwm.com/api/feed/search?keywords=TERM&count=20` |
| **Risk** | Unaffiliated third-party, could disappear anytime, no SLA |
**Verdict:** Great for development/testing. Too risky as sole production backend. Could be a zero-cost fallback.
### Not Recommended
| Service | Why Not |
|---------|---------|
| **TikAPI** | $50-189/mo subscription — same problem as Apify |
| **Bright Data** | $499/mo minimum — enterprise pricing |
| **davidteather/TikTok-Api** | Video search is broken, requires Playwright, fragile |
| **Spider, Zyte, Scrappey** | Generic scrapers — return raw HTML, no TikTok structure |
| **Piloterr** | No TikTok endpoints, subscription only |
## Recommended Approach
### Option A: ScrapeCreators as primary (Recommended)
- [ ] Sign up for ScrapeCreators, get free credits
- [ ] Test the "Search by Keyword" endpoint to verify it returns all required fields
- [ ] Refactor `tiktok.py` to use ScrapeCreators REST API instead of Apify actor
- [ ] Keep `apify_client_wrapper.py` for future FB/IG (or refactor to generic wrapper)
- [ ] Update `.env` config: `SCRAPECREATORS_API_KEY` replaces `APIFY_API_TOKEN` for TikTok
- [ ] Update README, SKILL.md installation instructions
- [ ] Buy $10 credits after confirming it works
### Option B: tikwm.com as primary (Zero cost, higher risk)
- [ ] Test tikwm.com search endpoint to verify response schema
- [ ] If it returns engagement metrics, implement as primary backend
- [ ] Add ScrapeCreators as paid fallback when tikwm fails
- [ ] No API key required — simplest user setup
### Option C: Keep Apify, document the subscription requirement
- [ ] Update README to clarify Apify requires a paid plan
- [ ] Add note about the free $5/mo credits tier (if it still works without subscription)
- [ ] Ship as-is with clear billing expectations
## Implementation Plan (Option A)
### Phase 1: Validate ScrapeCreators API
- [ ] Sign up and get API key
- [ ] Test keyword search endpoint: `GET /tiktok/search?keyword={topic}&count=20`
- [ ] Verify response contains: video ID, play count, likes, comments, shares, author, hashtags, date, description
- [ ] Test caption/subtitle availability (or confirm description text is sufficient)
### Phase 2: Swap the Backend
- [ ] Create `scripts/lib/scrapecreators_client.py` (simple REST wrapper, ~40 lines)
- [ ] Refactor `tiktok.py:search_tiktok()` to call ScrapeCreators instead of Apify
- [ ] Map ScrapeCreators response fields to our existing item dict format
- [ ] Refactor `tiktok.py:fetch_captions()` — check if ScrapeCreators provides subtitles, else use description text only
- [ ] Update `env.py` to read `SCRAPECREATORS_API_KEY` (keep `APIFY_API_TOKEN` for backward compat)
- [ ] Update `scripts/lib/ui.py` spinner messages if needed
### Phase 3: Update Docs & Ship
- [ ] Update README.md installation section (new API key)
- [ ] Update SKILL.md
- [ ] Update `~/.config/last30days/.env` locally
- [ ] Run full test suite
- [ ] Commit and push
## Acceptance Criteria
- [ ] TikTok search returns structured data with views, likes, comments, shares
- [ ] No subscription required — pay-as-you-go only
- [ ] Cost < $1/month at normal usage (5-10 searches/day)
- [ ] Existing test suite passes with new backend
- [ ] Graceful degradation when API key is missing (same behavior as today)
## Open Questions
1. Does ScrapeCreators' keyword search support date filtering, or do we filter post-API like we do with Apify?
2. Does ScrapeCreators return subtitle/caption data, or just video descriptions?
3. What's the response time? Apify actors took 30-120 seconds. REST APIs should be faster.
4. Should we keep Apify as a fallback backend (user configures one or the other)?
docs/plans/2026-03-04-feat-instagram-facebook-source-plan.md
+210
View File
@@ -0,0 +1,210 @@
# feat: Add Instagram and Facebook Sources via ScrapeCreators API
**Date:** 2026-03-04
**Type:** Enhancement
**Priority:** Instagram first, Facebook conditional ("if it's good")
## Summary
Add Instagram Reels and Facebook as new research sources in last30days, using the same ScrapeCreators REST API already powering TikTok. Instagram is the primary target; Facebook is a follow-on if the pattern works well.
Both sources share the existing `SCRAPECREATORS_API_KEY` — no new API keys needed.
## Approach
Replicate the TikTok integration pattern exactly. Each source follows the same 8-step pipeline:
```
tiktok.py pattern → instagram.py (new) → facebook.py (new, conditional)
```
## ScrapeCreators API Endpoints
### Instagram
| Endpoint | Path | Params | Credits | Notes |
|----------|------|--------|---------|-------|
| **Search Reels** | `GET /v1/instagram/reels/search` | `keyword`, pagination | 1 per 10 reels, max 60/req | Keyword search via Google (IG search requires login). V2 also available. |
| **Transcript** | `GET /v2/instagram/media/transcript` | `url` | 1 | Returns `{transcripts: [{id, shortcode, text}]}`. Videos <2min only. |
| **Comments** | `GET /v2/instagram/post/comments` | `url`, `cursor` | 1 | Returns `{comments: [{id, text, created_at, user}]}`. 100-300 per call. |
| **User Reels** | `GET /v1/instagram/user/reels` | `handle` or `user_id`, `max_id`, `trim` | 1 | All reels from a profile. Response: `{items: [...], paging_info}` |
**Primary search strategy:** `/v1/instagram/reels/search` with keyword param for topic search. This is the analog to TikTok's `/search/keyword`.
**Response fields per reel item:**
- `pk` / `code` (shortcode) — reel ID
- `taken_at` — unix timestamp
- `play_count` / `ig_play_count` — views
- `like_count` — likes
- `comment_count` — comments
- `video_duration` — seconds
- `has_audio` — boolean
- `user` object — username, full_name, is_verified, profile_pic_url
- `caption` object — text content
- Media URLs for thumbnails and video versions
### Facebook
| Endpoint | Path | Params | Credits | Notes |
|----------|------|--------|---------|-------|
| **Profile Posts** | `GET /v1/facebook/profile/posts` | `url` or `pageId`, `cursor` | 1 | Returns 3 posts at a time with engagement |
| **Profile Reels** | `GET /v1/facebook/profile/reels` | similar | 1 | 10 reels at a time |
| **Post** | `GET /v1/facebook/post` | `url` | 1 | Single post/reel by URL |
| **Transcript** | `GET /v1/facebook/post/transcript` | `url` | 1 | Video transcript, <2min |
| **Comments** | `GET /v1/facebook/post/comments` | `url`, `feedback_id` | 1 | Post/reel comments |
**Facebook limitation:** No keyword search endpoint. Only profile-based scraping (3 posts at a time). This makes Facebook significantly less useful for topic-based research vs. Instagram's keyword search.
**Response fields per post:**
- `id` — post ID
- `text` — post content
- `url` / `permalink` — post URL
- `author` — `{name, short_name, id}`
- `reactionCount` — total reactions
- `commentCount` — comments
- `videoViewCount` — video views (if applicable)
- `publishTime` — unix timestamp
- `topComments` — array of `{id, text, publishTime, author}`
## Implementation Plan
### Phase 1: Instagram Source (Primary)
#### 1.1 Create `scripts/lib/instagram.py`
- [x] Copy structure from `scripts/lib/tiktok.py`
- [x] Change `SCRAPECREATORS_BASE` to `"https://api.scrapecreators.com"`
- [x] Implement `search_instagram()` → calls `/v1/instagram/reels/search`
- Params: `keyword=core_topic`
- Parse response `items` array
- Extract: `pk`/`code` as video_id, `taken_at` as date, `play_count`/`like_count`/`comment_count` as engagement, `user.username` as author, `caption.text` as text
- Build URL: `https://www.instagram.com/reel/{code}`
- Reuse `_extract_core_subject()`, `_compute_relevance()`, `_tokenize()` from tiktok.py (or factor into shared util)
- Apply date range filter, sort by views descending
- [x] Implement `fetch_captions()` → calls `/v2/instagram/media/transcript`
- For top N items (per depth config), fetch transcript
- Response: `{transcripts: [{id, shortcode, text}]}`
- Fallback to caption text if transcript unavailable
- Truncate to 500 words
- [x] Implement `search_and_enrich()` → combines search + captions
- [x] Implement `parse_instagram_response()` → returns `response.get("items", [])`
- [x] Reuse shared helpers: `_sc_headers()`, `_log()`, `_clean_webvtt()`, `DEPTH_CONFIG`, `STOPWORDS`, `SYNONYMS`
**Key difference from TikTok:** Instagram response uses `play_count`/`like_count`/`comment_count` directly (no `statistics` wrapper), `user.username` (not `author.unique_id`), `caption.text` (not `desc`), `taken_at` (not `create_time`), `code` shortcode for URL construction.
#### 1.2 Add `InstagramItem` to `scripts/lib/schema.py`
- [x] Add dataclass mirroring `TikTokItem` structure:
```python
@dataclass
class InstagramItem:
id: str # "IG1", "IG2", ...
text: str # caption text
url: str # https://www.instagram.com/reel/{code}
author_name: str # Instagram handle
date: Optional[str] # YYYY-MM-DD from taken_at
date_confidence: str # "high"
engagement: Optional[Engagement] # views, likes, num_comments
caption_snippet: str # transcript or caption text
hashtags: List[str] # extracted from caption
relevance: float
why_relevant: str
subs: SubScores
score: int
cross_refs: List[str]
```
#### 1.3 Add normalization to `scripts/lib/normalize.py`
- [x] Add `normalize_instagram_items()` function
- Assign IDs as `IG1`, `IG2`, ...
- Map engagement: `views=play_count`, `likes=like_count`, `num_comments=comment_count`
- Set `date_confidence="high"` (unix timestamp)
#### 1.4 Add scoring to `scripts/lib/score.py`
- [x] Add `compute_instagram_engagement_raw()` — same formula as TikTok:
`0.50*log1p(views) + 0.30*log1p(likes) + 0.20*log1p(comments)`
Views dominate on Instagram Reels just like TikTok.
- [x] Add `score_instagram_items()` — same weights: 45% relevance, 25% recency, 30% engagement
#### 1.5 Add dedup to `scripts/lib/dedupe.py`
- [x] Add `dedupe_instagram()` — same as `dedupe_tiktok()`, calls `dedupe_items()` with 0.7 threshold
- [x] Update `get_item_text()` to handle `InstagramItem`
- [x] Update `_get_cross_source_text()` for cross-source linking
- [x] Add `IG` prefix to cross-ref detection in `cross_source_link()`
#### 1.6 Add rendering to `scripts/lib/render.py`
- [x] Add Instagram section in `render_compact()` — same pattern as TikTok block (lines 251-285)
- Show: score, @author, date, views/likes, caption snippet, hashtags, why_relevant
- [x] Update data freshness check to include `instagram_recent`
- [x] Update stats footer to include Instagram count
- [x] Add `'IG'` to cross-ref source name mapping
#### 1.7 Add `Report.instagram` field to `scripts/lib/schema.py`
- [x] Add `instagram: List[InstagramItem]` and `instagram_error: str` to `Report` dataclass
#### 1.8 Integrate into `scripts/last30days.py` orchestrator
- [x] Add `"instagram"` to `VALID_SEARCH_SOURCES`
- [x] Add `import` for `instagram` module in `scripts/lib/`
- [x] Add `is_instagram_available()` check in `env.py` — reuse `SCRAPECREATORS_API_KEY` (same key as TikTok)
- [x] Add `get_instagram_token()` in `env.py` — same as `get_tiktok_token()`, returns `SCRAPECREATORS_API_KEY`
- [x] Add `_search_instagram()` helper in orchestrator (mirrors `_search_tiktok()`)
- [x] Add Instagram to the thread pool executor block
- [x] Add Instagram timeout config (same as TikTok: 90/120/150s for quick/default/deep)
- [x] Wire through pipeline: normalize → filter → score → sort → dedupe → cross-link → report
- [x] Add Instagram to `progress.show_complete()` and UI spinner
#### 1.9 Add to watchlist extraction in `scripts/watchlist.py`
- [x] Add Instagram findings loop in `_run_topic()` (mirrors TikTok block at lines 204-213)
#### 1.10 Update README.md
- [x] Add Instagram to the sources list
- [x] Note that `SCRAPECREATORS_API_KEY` covers both TikTok and Instagram
### Phase 2: Facebook Source (Conditional)
**Recommendation: SKIP Facebook for now.** Here's why:
1. **No keyword search endpoint** — Facebook only offers profile-based scraping (`/profile/posts` returns 3 posts at a time). Can't search by topic.
2. **Low relevance for topic research** — Without keyword search, we'd need to know specific Facebook pages to scrape, which defeats the purpose of automated topic discovery.
3. **Poor ROI** — 3 posts per API call is very limited compared to Instagram's 60 reels per search.
4. **Same API key** — If Facebook search is added later, it's trivial to add since it shares `SCRAPECREATORS_API_KEY`.
If the user still wants Facebook, the implementation would follow the same pattern but would need a different discovery strategy (e.g., hardcoded page list per topic, or using the Ad Library search for commercial topics).
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `scripts/lib/instagram.py` | **CREATE** | Instagram search + transcript via ScrapeCreators |
| `scripts/lib/schema.py` | MODIFY | Add `InstagramItem` dataclass, add `instagram` to `Report` |
| `scripts/lib/normalize.py` | MODIFY | Add `normalize_instagram_items()` |
| `scripts/lib/score.py` | MODIFY | Add `compute_instagram_engagement_raw()`, `score_instagram_items()` |
| `scripts/lib/dedupe.py` | MODIFY | Add `dedupe_instagram()`, update text extractors |
| `scripts/lib/render.py` | MODIFY | Add Instagram render section, update stats |
| `scripts/lib/env.py` | MODIFY | Add `is_instagram_available()`, `get_instagram_token()` |
| `scripts/last30days.py` | MODIFY | Add Instagram to orchestrator pipeline |
| `scripts/watchlist.py` | MODIFY | Add Instagram findings extraction |
| `README.md` | MODIFY | Add Instagram to sources list |
## Shared Code Opportunity
`_extract_core_subject()`, `_compute_relevance()`, `_tokenize()`, `STOPWORDS`, `SYNONYMS`, and `DEPTH_CONFIG` are duplicated between `tiktok.py` and the new `instagram.py`. Two options:
1. **Copy-paste** (simpler, matches current pattern) — each source module is self-contained
2. **Extract to shared module** (cleaner) — move to `scripts/lib/search_utils.py`
**Recommendation:** Copy-paste for now to match existing pattern. Refactor later if a third ScrapeCreators source is added.
## Testing Strategy
- [x] Run `python3 scripts/lib/instagram.py` with test keyword (if standalone test added)
- [x] Run `python3 scripts/last30days.py "instagram reels trends" --search=instagram` to test isolated
- [x] Run full multi-source: `python3 scripts/last30days.py "AI tools" --search=reddit,instagram`
- [x] Verify JSON output: `--emit=json` includes `instagram` key
- [x] Verify watchlist extraction works with Instagram findings
- [x] Check credit usage is reasonable (1 credit per 10 reels search + 1 per transcript)
## Credits Budget
Per research run with Instagram at `default` depth:
- Search: 1 credit (per 10 reels, returns up to 20) ≈ 2 credits
- Transcripts: 5 credits (max_captions=5 at default depth)
- **Total: ~7 credits per topic** (vs TikTok ~6 credits)
docs/plans/2026-03-04-feat-v2.8-tiktok-instagram-release-plan.md
+147
View File
@@ -0,0 +1,147 @@
---
title: "feat: v2.8 Release — Instagram Reels + TikTok ScrapeCreators Migration"
type: enhancement
status: pending
date: 2026-03-04
---
# feat: v2.8 Release — Instagram Reels + TikTok ScrapeCreators Migration
## Summary
Ship everything from the last sprint as one combined GitHub release: TikTok's migration from Apify to ScrapeCreators (already committed) + Instagram Reels as the 8th source (uncommitted) + SKILL.md URL regression fixes. Version bump to v2.8.
## What's Shipping
### 1. Instagram Reels — 8th source (NEW)
- Search Instagram Reels by keyword via ScrapeCreators `/v1/instagram/reels/search`
- Spoken-word transcript extraction via `/v2/instagram/media/transcript`
- Full pipeline: search → normalize → score → dedupe → cross-link → render
- Shares `SCRAPECREATORS_API_KEY` with TikTok (no new API key needed)
- ~7 credits per topic at default depth
### 2. TikTok — Apify → ScrapeCreators migration (ALREADY COMMITTED)
- Replaced Apify dependency with ScrapeCreators API
- Same functionality, different backend
- No more `APIFY_API_TOKEN` — uses `SCRAPECREATORS_API_KEY`
### 3. SKILL.md quality fixes
- Instagram added to stats template, citation priority, data sections, footer
- URL regression fix: explicit URL-to-name extraction rules, stronger anti-Sources instruction
- Security section updated: Apify → ScrapeCreators
## Release Checklist
### Pre-commit: Update docs
- [ ] **README.md** — Update for v2.8:
- [ ] Change title from "v2.7" to "v2.8"
- [ ] Add "New in v2.8" banner: Instagram Reels + ScrapeCreators migration
- [ ] Update installation section: `APIFY_API_TOKEN` → `SCRAPECREATORS_API_KEY`
- [ ] Add Instagram to the "How it works" flow description (line 132)
- [ ] Update TikTok section: replace Apify references with ScrapeCreators
- [ ] Add Instagram section (after TikTok section, ~line 963)
- [ ] Update API endpoints table: `api.apify.com` → `api.scrapecreators.com`, add Instagram endpoints
- [ ] Update closing tagline to include Instagram
- [ ] Remove "The shared Apify client wrapper is designed for future Facebook and Instagram sources" (line 963) — Instagram is here now
- [ ] **CHANGELOG.md** — Add v2.8.0 entry:
```
## [2.8.0] - 2026-03-04
### Highlights
Instagram Reels as the 8th signal source, TikTok migrated from Apify to ScrapeCreators API, and SKILL.md quality improvements.
### Added
- Instagram Reels as 8th research source via ScrapeCreators API — keyword search, engagement metrics (views, likes, comments), spoken-word transcript extraction
- Instagram items in SKILL.md stats template, citation priority, and output footer
- URL-to-name extraction examples in SKILL.md for cleaner web source display
### Changed
- TikTok backend migrated from Apify to ScrapeCreators API (same key covers TikTok + Instagram)
- `APIFY_API_TOKEN` replaced by `SCRAPECREATORS_API_KEY` in config
- SKILL.md version bumped to v2.8
- WebSearch citation instruction strengthened to prevent trailing Sources: blocks
### Fixed
- Web stats line showing full URLs instead of plain domain names (regression from v2.7)
- Trailing "Sources:" block appearing after invitation (WebSearch tool mandate conflict)
- Instagram/TikTok not running in web-only mode when `--search=instagram` used without Reddit/X
```
- [ ] **SKILL.md** frontmatter — Bump version from "2.7" to "2.8"
- [ ] **SKILL.md** description — Add Instagram to the description field
### Commit & tag
- [ ] Stage all changes: modified files + 4 new files (`scripts/lib/instagram.py`, 3 plan docs)
- [ ] Commit with message:
```
feat: v2.8 — Instagram Reels source + TikTok ScrapeCreators migration
- Add Instagram Reels as 8th research source via ScrapeCreators API
- Migrate TikTok from Apify to ScrapeCreators (same API key)
- Add SCRAPECREATORS_API_KEY config (replaces APIFY_API_TOKEN)
- Fix web stats URL regression and trailing Sources: block
- Fix Instagram/TikTok not running in --search=instagram web-only path
- Update SKILL.md with Instagram stats, citations, URL formatting rules
```
- [ ] Create tag: `git tag -a v2.8.0 -m "v2.8.0: Instagram Reels + ScrapeCreators"`
- [ ] Push: `git push origin main --tags`
### Post-push: GitHub release
- [ ] Create GitHub release via `gh release create v2.8.0`:
```
## What's New in v2.8
**Instagram Reels** is now the 8th signal source. Search any topic and get trending Instagram Reels with views, likes, and spoken-word transcripts — scored and ranked alongside Reddit, X, YouTube, TikTok, HN, Polymarket, and the web.
**TikTok migrated to ScrapeCreators API.** Same functionality, new backend. Replace `APIFY_API_TOKEN` with `SCRAPECREATORS_API_KEY` in your config. One key now covers both TikTok and Instagram.
### Setup
Sign up at [scrapecreators.com](https://scrapecreators.com) (100 free credits, then PAYG) and add your key:
```bash
echo 'SCRAPECREATORS_API_KEY=your_key' >> ~/.config/last30days/.env
```
### Breaking Change
- `APIFY_API_TOKEN` is no longer used. Replace with `SCRAPECREATORS_API_KEY`.
### Bug Fixes
- Fixed web source URLs leaking into stats display
- Fixed Instagram/TikTok not running when used with `--search=` flag
```
### Post-release: Sync
- [ ] Run `bash scripts/sync.sh` to deploy to all 4 skill destinations
- [ ] Verify Instagram works in a fresh `/last30days` session
## Files Modified (this release)
| File | Status | Description |
|------|--------|-------------|
| `scripts/lib/instagram.py` | NEW | Instagram search + transcript via ScrapeCreators |
| `scripts/lib/schema.py` | MODIFIED | InstagramItem dataclass, Report.instagram field |
| `scripts/lib/normalize.py` | MODIFIED | normalize_instagram_items() |
| `scripts/lib/score.py` | MODIFIED | score_instagram_items(), engagement formula |
| `scripts/lib/dedupe.py` | MODIFIED | dedupe_instagram(), cross-source linking |
| `scripts/lib/render.py` | MODIFIED | Instagram render section, stats |
| `scripts/lib/env.py` | MODIFIED | is_instagram_available(), get_instagram_token() |
| `scripts/lib/ui.py` | MODIFIED | Instagram spinner messages |
| `scripts/last30days.py` | MODIFIED | Instagram in orchestrator pipeline |
| `scripts/watchlist.py` | MODIFIED | Instagram findings extraction |
| `SKILL.md` | MODIFIED | Instagram in stats/citations/footer, URL fixes |
| `README.md` | TO UPDATE | Instagram section, ScrapeCreators migration |
| `CHANGELOG.md` | TO UPDATE | v2.8.0 entry |
| `docs/plans/*.md` | NEW (3) | Plan documents for this work |
docs/plans/2026-03-04-fix-web-sources-url-regression-plan.md
+135
View File
@@ -0,0 +1,135 @@
---
title: "fix: Web sources showing full URLs instead of plain domain names"
type: fix
status: pending
date: 2026-03-04
---
# fix: Web Sources Showing Full URLs Instead of Plain Domain Names
## Problem
Two regressions in the `/last30days` skill output:
### Regression 1: Full URLs on the Web stats line
The `🌐 Web:` stats line is showing full URLs instead of plain source names:
**BAD (current — "Instagram Trends" run):**
```
├─ 🌐 Web: 10+ pages — https://later.com/blog/instagram-reels-trends/,
https://socialbee.com/blog/instagram-trends/,
https://buffer.com/resources/instagram-algorithms/,
https://metricool.com/instagram-trends/,
https://napoleoncat.com/blog/instagram-reels-trends/
```
**GOOD (expected):**
```
├─ 🌐 Web: 10+ pages — Later, SocialBee, Buffer, Metricool, NapoleonCat
```
### Regression 2: Trailing Sources: block with full URLs
A separate `Sources:` section appears at the bottom of the response with full URLs:
```
Sources:
- https://www.heyorca.com/blog/instagram-social-news
- https://socialbee.com/blog/instagram-updates/
- https://buffer.com/resources/instagram-algorithms/
- https://www.cnn.com/2026/02/22/tech/social-media-addiction-trial-tobacco-moment
```
This was fixed in commit `82efa61` (2026-03-02) but is regressing intermittently.
## Root Cause
The SKILL.md instructions at lines 219-221, 404, and 409 already say the right thing:
- Line 404: `├─ 🌐 Web: {N} pages — Source Name, Source Name, Source Name`
- Line 409: `"plain names, no URLs — URLs wrap badly in terminals"`
- Lines 219-221: "DO NOT output a separate Sources: block"
But the model ignores these because:
1. **The template `Source Name` is too abstract.** The model sees WebSearch results with full URLs and doesn't know how to extract a human-friendly name from `https://later.com/blog/instagram-reels-trends/`. It needs explicit examples showing the transformation.
2. **The WebSearch system mandate still wins.** The WebSearch tool's built-in instruction (`"you MUST include a Sources: section"`) outcompetes the skill's instruction. The current countermeasure (line 409) works sometimes but not reliably — it needs to be stronger and repeated.
3. **No explicit extraction rule.** The model needs a concrete rule for turning URLs into names: strip protocol, strip path, strip `www.`, capitalize.
## Proposed Solution
**SKILL.md edits only. No Python changes.**
### Fix 1: Add explicit URL-to-name examples in the stats template (line 404 area)
After the stats template block, add concrete examples showing the transformation:
```
**🌐 Web: line formatting:**
- Extract the SITE NAME from each URL — strip protocol, path, and "www."
- Use the publication's proper name when recognizable
- Examples:
- https://later.com/blog/instagram-reels-trends/ → "Later"
- https://socialbee.com/blog/instagram-trends/ → "SocialBee"
- https://buffer.com/resources/instagram-algorithms/ → "Buffer"
- https://www.cnn.com/2026/02/22/tech/... → "CNN"
- https://medium.com/the-ai-studio/... → "Medium"
- https://radicaldatascience.wordpress.com/... → "Radical Data Science"
- NEVER paste the URL itself. ONLY the site name as plain text.
- Separate names with commas: "Later, SocialBee, Buffer, CNN, Medium"
```
### Fix 2: Strengthen the anti-Sources instruction (line 409 area)
Replace the current single-paragraph note with a louder, more explicit instruction:
```
**⚠️ WebSearch citation requirement — ALREADY SATISFIED above.**
The WebSearch tool mandates source citation. That requirement is FULLY satisfied
by the source names on the 🌐 Web: line above. Do NOT append a separate
"Sources:" section at the end of your response. Do NOT list URLs anywhere in
your output. The 🌐 Web: line IS your citation. You're done.
```
### Fix 3: Add a negative example in the URL FORMATTING section (line 356 area)
Extend the existing BAD/GOOD examples to cover the stats line specifically:
```
URL FORMATTING: NEVER paste raw URLs anywhere in the output.
- BAD: "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/"
- GOOD: "per Rolling Stone"
- BAD stats line: "🌐 Web: 10 pages — https://later.com/blog/..., https://buffer.com/..."
- GOOD stats line: "🌐 Web: 10 pages — Later, Buffer, CNN, SocialBee"
```
### Fix 4: Update Security section (line 588, 600)
While we're in SKILL.md, update the stale Apify references to ScrapeCreators:
- Line 588: Change Apify reference to ScrapeCreators for TikTok
- Line 600: Update TikTok requirement note
- Add Instagram source mention
## Files to Modify
| File | Action | Description |
|------|--------|-------------|
| `SKILL.md` | MODIFY | Strengthen URL formatting rules, add examples, fix Apify refs |
## Implementation Steps
- [x] Add URL-to-name extraction examples after stats template (after line 407)
- [x] Strengthen anti-Sources instruction (replace line 409)
- [x] Add BAD/GOOD stats line example to URL FORMATTING section (around line 356)
- [x] Update Security section: Apify → ScrapeCreators, add Instagram (lines 588, 600)
- [x] Run `bash scripts/sync.sh` to deploy to all destinations
- [ ] Test with `/last30days Instagram Trends` — confirm plain names, no trailing Sources:
## Acceptance Criteria
- [ ] `🌐 Web:` line shows plain names only (e.g., "Later, SocialBee, Buffer")
- [ ] No `Sources:` section appears at the bottom of the response
- [ ] No raw URLs appear anywhere in the output (synthesis, stats, or footer)
- [ ] Security section reflects current source stack (ScrapeCreators, not Apify)