last30days-skill/docs/plans/2026-03-04-fix-web-sources-url-regression-plan.md

title, type, status, date

title	type	status	date
fix: Web sources showing full URLs instead of plain domain names	fix	pending	2026-03-04

fix: Web Sources Showing Full URLs Instead of Plain Domain Names

Problem

Two regressions in the /last30days skill output:

Regression 1: Full URLs on the Web stats line

The 🌐 Web: stats line is showing full URLs instead of plain source names:

BAD (current — "Instagram Trends" run):

├─ 🌐 Web: 10+ pages — https://later.com/blog/instagram-reels-trends/,
https://socialbee.com/blog/instagram-trends/,
https://buffer.com/resources/instagram-algorithms/,
https://metricool.com/instagram-trends/,
https://napoleoncat.com/blog/instagram-reels-trends/

GOOD (expected):

├─ 🌐 Web: 10+ pages — Later, SocialBee, Buffer, Metricool, NapoleonCat

Regression 2: Trailing Sources: block with full URLs

A separate Sources: section appears at the bottom of the response with full URLs:

Sources:
- https://www.heyorca.com/blog/instagram-social-news
- https://socialbee.com/blog/instagram-updates/
- https://buffer.com/resources/instagram-algorithms/
- https://www.cnn.com/2026/02/22/tech/social-media-addiction-trial-tobacco-moment

This was fixed in commit 82efa61 (2026-03-02) but is regressing intermittently.

Root Cause

The SKILL.md instructions at lines 219-221, 404, and 409 already say the right thing:

• Line 404: ├─ 🌐 Web: {N} pages — Source Name, Source Name, Source Name
• Line 409: "plain names, no URLs — URLs wrap badly in terminals"
• Lines 219-221: "DO NOT output a separate Sources: block"

But the model ignores these because:

1. The template Source Name is too abstract. The model sees WebSearch results with full URLs and doesn't know how to extract a human-friendly name from https://later.com/blog/instagram-reels-trends/. It needs explicit examples showing the transformation.

2. The WebSearch system mandate still wins. The WebSearch tool's built-in instruction ("you MUST include a Sources: section") outcompetes the skill's instruction. The current countermeasure (line 409) works sometimes but not reliably — it needs to be stronger and repeated.

3. No explicit extraction rule. The model needs a concrete rule for turning URLs into names: strip protocol, strip path, strip www., capitalize.

Proposed Solution

SKILL.md edits only. No Python changes.

Fix 1: Add explicit URL-to-name examples in the stats template (line 404 area)

After the stats template block, add concrete examples showing the transformation:

**🌐 Web: line formatting:**
- Extract the SITE NAME from each URL — strip protocol, path, and "www."
- Use the publication's proper name when recognizable
- Examples:
  - https://later.com/blog/instagram-reels-trends/ → "Later"
  - https://socialbee.com/blog/instagram-trends/ → "SocialBee"
  - https://buffer.com/resources/instagram-algorithms/ → "Buffer"
  - https://www.cnn.com/2026/02/22/tech/... → "CNN"
  - https://medium.com/the-ai-studio/... → "Medium"
  - https://radicaldatascience.wordpress.com/... → "Radical Data Science"
- NEVER paste the URL itself. ONLY the site name as plain text.
- Separate names with commas: "Later, SocialBee, Buffer, CNN, Medium"

Fix 2: Strengthen the anti-Sources instruction (line 409 area)

Replace the current single-paragraph note with a louder, more explicit instruction:

**⚠️ WebSearch citation requirement — ALREADY SATISFIED above.**
The WebSearch tool mandates source citation. That requirement is FULLY satisfied
by the source names on the 🌐 Web: line above. Do NOT append a separate
"Sources:" section at the end of your response. Do NOT list URLs anywhere in
your output. The 🌐 Web: line IS your citation. You're done.

Fix 3: Add a negative example in the URL FORMATTING section (line 356 area)

Extend the existing BAD/GOOD examples to cover the stats line specifically:

URL FORMATTING: NEVER paste raw URLs anywhere in the output.
- BAD: "per https://www.rollingstone.com/music/music-news/kanye-west-bully-1235506094/"
- GOOD: "per Rolling Stone"
- BAD stats line: "🌐 Web: 10 pages — https://later.com/blog/..., https://buffer.com/..."
- GOOD stats line: "🌐 Web: 10 pages — Later, Buffer, CNN, SocialBee"

Fix 4: Update Security section (line 588, 600)

While we're in SKILL.md, update the stale Apify references to ScrapeCreators:

• Line 588: Change Apify reference to ScrapeCreators for TikTok
• Line 600: Update TikTok requirement note
• Add Instagram source mention

Files to Modify

File	Action	Description
SKILL.md	MODIFY	Strengthen URL formatting rules, add examples, fix Apify refs

Implementation Steps

• Add URL-to-name extraction examples after stats template (after line 407)
• Strengthen anti-Sources instruction (replace line 409)
• Add BAD/GOOD stats line example to URL FORMATTING section (around line 356)
• Update Security section: Apify → ScrapeCreators, add Instagram (lines 588, 600)
• Run bash scripts/sync.sh to deploy to all destinations
• Test with /last30days Instagram Trends — confirm plain names, no trailing Sources:

Acceptance Criteria

• 🌐 Web: line shows plain names only (e.g., "Later, SocialBee, Buffer")
• No Sources: section appears at the bottom of the response
• No raw URLs appear anywhere in the output (synthesis, stats, or footer)
• Security section reflects current source stack (ScrapeCreators, not Apify)