From deb9f33437939709d946529b203371f208ecb338 Mon Sep 17 00:00:00 2001 From: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Date: Tue, 14 Apr 2026 12:18:19 -0400 Subject: [PATCH] chore: remove legacy plans/ directory (superseded by docs/plans/) Both plans describe work that was already shipped: - feat-add-websearch-source.md - websearch is in the v3 pipeline (scripts/lib/perplexity.py etc) - fix-strict-date-filtering.md - date filtering is enforced in scripts/lib/dates.py New planning goes in docs/plans/ following the ce:plan convention. --- ...01-fix-skill-upload-200-file-limit-plan.md | 445 ++++++++++++++++++ plans/feat-add-websearch-source.md | 395 ---------------- plans/fix-strict-date-filtering.md | 328 ------------- 3 files changed, 445 insertions(+), 723 deletions(-) create mode 100644 docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md delete mode 100644 plans/feat-add-websearch-source.md delete mode 100644 plans/fix-strict-date-filtering.md diff --git a/docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md b/docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md new file mode 100644 index 0000000..b60458b --- /dev/null +++ b/docs/plans/2026-04-14-001-fix-skill-upload-200-file-limit-plan.md @@ -0,0 +1,445 @@ +--- +title: Fix skill upload 200-file limit + packaging hygiene (public repo) +type: fix +status: active +date: 2026-04-14 +deepened: 2026-04-14 +--- + +# Fix skill upload 200-file limit + packaging hygiene (public repo) + +## Overview + +claude.ai's "Upload skill" UI rejects zips with more than 200 files. Zipping the public `mvanhorn/last30days-skill` repo produces 406 files, so the upload fails outright (evidence: Trevin's 2026-04-14 report). Root cause is an accidentally committed npm package under `vendor/` (215 files of dead weight from PR #48) plus the absence of a user-facing packaging path that matches Anthropic's canonical `.skill` format. + +Goal: let any user produce a compliant `last30days.skill` file in one command, matching Anthropic's skill-creator packaging contract, while also removing genuine dead weight from the repo (unused vendor, legacy plans). + +## Problem Frame + +- Trevin tried to upload the public repo as a Claude Skill and hit the 200-file limit +- 215 of 406 files are `vendor/package/` - an extracted `steipete-bird-0.8.0.tgz` that no code imports +- The real runtime X client lives at `scripts/lib/vendor/bird-search/` (15 files, referenced by `scripts/lib/bird_x.py:5` and `tests/test_bird_x.py:133`) +- `.clawhubignore` is ClawHub-specific and does not affect a hand-rolled zip upload +- Users have no documented path to produce a compliant upload zip +- Legacy top-level `plans/` folder holds pre-`docs/plans/` planning artifacts (confirmed waste by Matt, 2026-04-14 chat) + +## Requirements Trace + +- R1. After this plan lands, the produced upload zip is =200 files +- R2. The X/bird-search runtime still works - no regression in `tests/test_bird_x.py` +- R3. A contributor following README instructions can produce a Claude-Skill-upload-compatible `.skill` file in one command +- R4. Re-introduction of a root `vendor/` directory is prevented via `.gitignore` +- R5. No runtime behavior changes for existing skill consumers (Claude Code plugin, ClawHub, Gemini) +- R6. Produced zip matches Anthropic's canonical skill-folder layout: top-level directory named exactly `last30days` containing `SKILL.md` at its root, with YAML frontmatter `name: last30days` +- R7. Root `SKILL.md` frontmatter passes Anthropic's documented limits: `name` =64 chars (currently 10), `description` =200 chars (currently 228, needs trimming) +- R8. Produced zip contains exactly one `SKILL.md` (at `last30days/SKILL.md`) - no conflicting second skill spec, no symlinks that the uploader may reject or break +- R9. No runtime import reaches an excluded path (proven by import-graph audit, not just asserted) + +## Scope Boundaries + +Non-goals: +- Not touching the private repo or ClawHub publish flow (those have their own strip script) +- Not resolving the adjacent open issues (#239 plugin loader path-escape, #236 OpenClaw paths, #231 security scan, #190 version drift, #184 Gemini install) - each deserves its own plan +- Not redesigning the skill into self-contained subfolders or splitting scripts into a separate package +- Not adding CI enforcement of the 200-file cap (possible follow-up) + +## Context and Research + +### Anthropic's canonical skill-upload contract + +Sourced from Anthropic's skill-creator repo (`anthropics/skills/skills/skill-creator/scripts/package_skill.py`) and help-center docs: + +1. **Output format:** a `.skill` file, which is a standard zip with the `.skill` extension. +2. **Top-level entry in the zip must be a single directory** whose name matches `name:` in the skill's YAML frontmatter. Anthropic's packager uses `arcname = file_path.relative_to(skill_path.parent)`, so the zip always contains `/...`. +3. **That directory must contain `SKILL.md`** at its root (the packager explicitly validates this). +4. **Required YAML frontmatter:** `name` (=64 chars, lowercase + hyphens) and `description` (=200 chars). Our root SKILL.md already satisfies both. +5. **Canonical exclusions** applied by Anthropic's packager: + - Directories: `__pycache__`, `node_modules` + - Root-only: `evals/` + - File globs: `*.pyc` + - Files: `.DS_Store` +6. **Empirical limit:** the upload UI rejects =200 files (screenshot 2026-04-14). Not documented, but confirmed. +7. **Per-file size cap** is not publicly documented; general claude.ai uploads cap at 30MB per file. Conservative target: keep any single file under 10MB. + +### Relevant code and patterns in this repo + +- `SKILL.md` (root, 1382 lines, 80KB) - `name: last30days`, `user-invocable: true`. This is the skill. +- `skills/last30days/SKILL.md` (230 lines) - `name: last30days-v3-spec`, `user-invocable: false`. Internal architecture spec, separate skill name - not the upload target. +- `vendor/package/` - accidental commit from PR #48, 215 files, zero importers. +- `vendor/steipete-bird-0.8.0.tgz` - source tarball, also unused at runtime. +- `scripts/lib/vendor/bird-search/` - the ACTUAL vendored bird-search client (15 files). Keep. +- `plans/` (top-level, 2 files: `feat-add-websearch-source.md`, `fix-strict-date-filtering.md`) - legacy, pre-`docs/plans/` convention. Matt confirmed delete. +- `scripts/sync.sh` - deploys skill to `~/.claude`, `~/.agents`, `~/.codex`. Reference for runtime-required files. +- `.clawhubignore` - existing exclude list for the ClawHub path. Not used here, but good cross-reference for what is runtime-irrelevant. +- `.gitignore` - current dev excludes (`.venv/`, `__pycache__/`, `.DS_Store`, etc). + +### Institutional learnings + +- Private repo has `scripts/clawhub-publish.sh` + `scripts/strip_for_openclaw.py` that build a staging dir with only OpenClaw-safe files. Not needed for this public-path upload; `git archive` with `--prefix` is sufficient and dependency-free. +- PR #48 introduced `vendor/package/` unintentionally. No code imports from it. + +### File count math (verified via dry run) + +| Strategy | File count | Under cap? | +|---|---|---| +| Current repo, zip as-is | 406 | No | +| After `vendor/` deleted | 191 | Yes (thin margin) | +| After `vendor/` + `plans/` deleted, no further excludes | 189 | Yes | +| With full planned excludes (Anthropic canonical + tests/docs/fixtures/assets/dev manifests/nested skill dirs) | 81 | Comfortable headroom | + +Dry run run on 2026-04-14 against the current working tree. Simulated the proposed `.gitattributes` with a `find` filter matching the intended exclude list. Result: 81 files, 868KB uncompressed. Actual `git archive` output may differ slightly (by 1-2 files) but will land well under 200. + +### Runtime import audit (proves core experience unchanged) + +Grepped all `import`/`from` statements in `scripts/**/*.py`. Non-stdlib imports resolve to only: +- `lib.*` (internal package at `scripts/lib/`) +- `store` (internal module at `scripts/store.py`) +- `scripts.*` (internal) + +No runtime import reaches `tests/`, `docs/`, `fixtures/`, `vendor/` (root), `plans/`, `assets/`, `.agents/`, `.codex-plugin/`, `.hermes-plugin/`, or any other excluded path. The shipped `.skill` file contains everything the runtime needs and nothing it does not. + +### Symlink and multi-SKILL.md audit + +The repo contains one symlink: `skills/last30days-nux/SKILL.md -> ../../SKILL.md`. Three SKILL.md files in total: +- `SKILL.md` (root, `name: last30days`, `user-invocable: true`) - the actual skill +- `skills/last30days/SKILL.md` (`name: last30days-v3-spec`, `user-invocable: false`) - internal architecture doc +- `skills/last30days-nux/SKILL.md` (symlink to root) - nux variant reference + +Shipping all three inside one zip creates two rejection risks: +1. Uploader sees multiple `SKILL.md` with conflicting `name:` values and refuses or misbinds +2. `git archive` stores the symlink as a symlink entry; the uploader may reject symlinked entries on principle + +Both risks disappear by excluding `skills/` entirely from the zip. The two internal skill definitions are not needed for claude.ai skill execution - they serve the repo as documentation / Claude Code plugin layout, not the direct upload path. + +### Sources consulted + +- Anthropic skill help center article (general upload guidance, no file-count number documented) +- [anthropics/skills README](https://github.com/anthropics/skills/blob/main/README.md) - YAML frontmatter requirements +- [anthropics/skills package_skill.py](https://github.com/anthropics/skills/blob/main/skills/skill-creator/scripts/package_skill.py) - canonical exclusions and arcname shape +- Trevin's 2026-04-14 chat screenshot (empirical 200-file cap) +- Adjacent issues #239, #236, #190 for context on current packaging mess + +## Key Technical Decisions + +- **Delete `vendor/` outright** rather than gitignore-and-leave. Pure dead weight. Rationale: the real vendored client is at `scripts/lib/vendor/bird-search/`, root `vendor/` has zero importers; keeping it invites re-upload. +- **Delete top-level `plans/`** (Matt confirmed). Rationale: superseded by `docs/plans/`. Moving content into `docs/plans/` if any is still relevant; otherwise just delete. +- **Produce a `.skill` file (not a plain `.zip`)** via `git archive --format=zip --prefix=last30days/ -o dist/last30days.skill HEAD`. Rationale: matches Anthropic's canonical contract - zip extension is cosmetic, but the `.skill` affordance is what the upload UI expects. +- **Use `git archive` + `.gitattributes export-ignore`** rather than a Python packager. Rationale: no Python dependency at build time, honors git's declarative exclude model, reusable by anyone running `git archive` directly. +- **Mirror Anthropic's canonical exclusions in `.gitattributes`** (`__pycache__`, `node_modules`, `*.pyc`, `.DS_Store`, `evals/`) alongside our repo-specific excludes. Rationale: future-proof if a contributor adds node deps; keeps us aligned with the Anthropic baseline. +- **Exclude `skills/` from the upload zip** (covers `skills/last30days/SKILL.md` and `skills/last30days-nux/SKILL.md`). Rationale: shipping multiple SKILL.md files with different `name:` values is a likely uploader-rejection cause, and the symlink at `skills/last30days-nux/SKILL.md` is an independent rejection risk. Repo contents stay intact - Claude Code plugin and GitHub viewers still see the directory. +- **Keep `.clawhubignore` as-is** - it serves the ClawHub publish path separately. Do not merge the two lists; different consumers, different exclusions. +- **Prevent regression with a `/vendor/` entry in `.gitignore`** (leading slash, so `scripts/lib/vendor/` is unaffected). +- **Do not address #239 `"skills": ["./"]` path-escape here.** That is a plugin.json change, not a zip-packaging change. Separate plan. + +## Open Questions + +### Resolved during planning + +- Is root `vendor/` used? No. Grep for `vendor/package`, `vendor/steipete`, `from vendor` returns zero hits outside `scripts/lib/vendor/`. +- Is `scripts/lib/vendor/bird-search/` safe? Yes. Referenced by `scripts/lib/bird_x.py:5` and `tests/test_bird_x.py:133`. +- What name does the top-level zip directory need? `last30days` - matches `name: last30days` in the root `SKILL.md` frontmatter. +- Does `skills/last30days/SKILL.md` conflict? No. It declares a different skill name (`last30days-v3-spec`) and is `user-invocable: false`. Not the upload target, and safe to ship inside the zip. +- Is there a documented file-count cap? No. 200 is empirical from the UI error screenshot. +- Should we gate this on a version bump? Yes, 3.0.0 - 3.0.1. Same API, same runtime, smaller and uploadable package. + +### Deferred to implementation + +- Exact `.gitattributes` export-ignore entries may need one tuning pass if `git archive` surfaces a file we forgot. Verification step catches it. +- Whether to delete `SKILL-original.md` from the repo entirely or just export-ignore. Leaning export-ignore to preserve git history context. +- Whether any content in `plans/*.md` is still live reference material. If so, move to `docs/plans/` under new naming convention; if not, delete outright. + +## Implementation Units + +- [ ] **Unit 1: Remove accidental `vendor/` commit** + +**Goal:** Delete the root `vendor/` directory and the stray `.tgz`, both unused at runtime. + +**Requirements:** R1, R2, R5 + +**Dependencies:** None + +**Files:** +- Delete: `vendor/` (entire tree, 215 files) +- Delete: `vendor/steipete-bird-0.8.0.tgz` +- Modify: `.gitignore` (add `/vendor/` to prevent regression - leading slash to avoid matching `scripts/lib/vendor/`) + +**Approach:** +- Single commit: `chore: remove unused root vendor/ directory (215 files from PR #48)` +- Verify `scripts/lib/vendor/bird-search/` is untouched +- Verify no `from vendor` or `vendor/package` references appear in the diff + +**Patterns to follow:** +- Commit message style matches recent history + +**Test scenarios:** +- Happy path: `find . -type f -not -path './.git/*' | wc -l` returns =200 after commit +- Integration: `python -m pytest tests/test_bird_x.py -q` passes - confirms the real vendored client still resolves +- Integration: `bash scripts/sync.sh` completes without error + +**Verification:** +- Zero files remain under `vendor/` on `main` +- `tests/test_bird_x.py` still passes +- `.gitignore` now contains `/vendor/` + +- [ ] **Unit 2: Remove legacy top-level `plans/` directory** + +**Goal:** Delete the pre-`docs/plans/` folder (Matt confirmed waste). + +**Requirements:** R1, R5 + +**Dependencies:** None (independent of Unit 1) + +**Files:** +- Delete: `plans/feat-add-websearch-source.md` +- Delete: `plans/fix-strict-date-filtering.md` +- Delete: `plans/` (now empty) + +**Approach:** +- Skim both files first. If either still reflects real upcoming work, port it to `docs/plans/YYYY-MM-DD-NNN--*-plan.md` before deletion. If not, delete. +- Commit: `chore: remove legacy plans/ directory (superseded by docs/plans/)` + +**Test scenarios:** +- Test expectation: none - pure housekeeping, no code paths affected + +**Verification:** +- `plans/` does not exist on `main` +- Nothing in the repo references `plans/feat-add-websearch-source.md` or `plans/fix-strict-date-filtering.md` (grep to confirm) + +- [ ] **Unit 3: Declare zip-time excludes via `.gitattributes`** + +**Goal:** Use `export-ignore` so `git archive` produces a skill-shaped zip without hand-filtering. + +**Requirements:** R1, R3, R6 + +**Dependencies:** Unit 1, Unit 2 + +**Files:** +- Create: `.gitattributes` + +**Approach:** +- Anthropic canonical exclusions (match `package_skill.py`): + - `__pycache__/` export-ignore + - `node_modules/` export-ignore + - `*.pyc` export-ignore + - `.DS_Store` export-ignore + - `evals/` export-ignore +- Repo-specific exclusions (dev/docs/build artifacts not needed at runtime): + - `tests/` (64 files) + - `docs/` (17 files including `docs/test-results/`) + - `fixtures/` (7 files) + - `assets/` (5 files, 14MB of README media) + - `SKILL-original.md` (historical) + - `SPEC.md`, `TASKS.md`, `test-run.log`, `CONTRIBUTORS.md`, `HERMES_SETUP.md`, `release-notes.md`, `CHANGELOG.md` + - `uv.lock` + - `.agents/`, `.codex-plugin/`, `.hermes-plugin/`, `.claude-plugin/` (platform adapters - skill-upload path is platform-agnostic) + - `.clawhubignore`, `.gitignore`, `.gitattributes` + - `skills/` (avoid second SKILL.md with conflicting `name:`; also drops the symlink at `skills/last30days-nux/SKILL.md`) +- Keep in archive: `scripts/` (runtime), root `SKILL.md`, `README.md`, `LICENSE`, `pyproject.toml`, `CLAUDE.md`, `gemini-extension.json`, `agents/`, `hooks/` + +**Technical design:** *(directional guidance, not implementation spec)* + +```gitattributes +# Anthropic canonical skill-packaging excludes +__pycache__/ export-ignore +node_modules/ export-ignore +*.pyc export-ignore +.DS_Store export-ignore +evals/ export-ignore + +# Repo-specific: tests + docs + media (not runtime) +tests/ export-ignore +docs/ export-ignore +fixtures/ export-ignore +assets/ export-ignore + +# Repo-specific: historical + dev manifests +SKILL-original.md export-ignore +SPEC.md export-ignore +... +``` + +**Patterns to follow:** +- `.gitattributes` export-ignore syntax per [git docs](https://git-scm.com/docs/gitattributes#_creating_an_archive) + +**Test scenarios:** +- Happy path: `git archive --format=zip HEAD | zipinfo -1 - | wc -l` returns =200 +- Happy path: zip contains `SKILL.md`, `scripts/last30days.py`, `scripts/lib/bird_x.py`, `scripts/lib/vendor/bird-search/lib/cookies.js` +- Happy path: zip contains exactly one `SKILL.md` entry at the top level (not multiple, not a symlink) +- Edge case: zip does NOT contain `tests/`, `docs/`, `assets/*.jpeg`, `*.mp3`, `skills/` +- Edge case: no symlink entries in the zip (`unzip -l` lines starting with `l`) +- Edge case: zip size stays under ~2MB (if over 5MB an unintended large file slipped through) + +**Verification:** +- Running `git archive --format=zip --output=/tmp/test.zip HEAD && unzip -l /tmp/test.zip | tail -1` reports =200 files and a sane byte count + +- [ ] **Unit 4: Add `scripts/build-skill.sh` user-facing builder** + +**Goal:** One-command path to produce a Claude-upload-compatible `.skill` file. + +**Requirements:** R3, R6 + +**Dependencies:** Unit 3 + +**Files:** +- Create: `scripts/build-skill.sh` +- Modify: `.gitignore` (add `/dist/` for build artifact) + +**Approach:** +- Bash, executable, `set -euo pipefail` +- `git archive --format=zip --prefix=last30days/ --output=dist/last30days.skill HEAD` +- The `--prefix=last30days/` nests everything under `last30days/` inside the zip, matching Anthropic's arcname contract +- Refuse to build if working tree is dirty (`git diff --quiet && git diff --cached --quiet`) +- Print file count, archive size, and path to paste into the upload UI +- Fail with a clear error if count exceeds 200 (defensive check) + +**Technical design:** *(directional guidance, not implementation spec)* + +```bash +#!/usr/bin/env bash +# build-skill.sh - package repo as a Claude-upload-ready .skill file +# Usage: bash scripts/build-skill.sh +set -euo pipefail + +if ! git diff --quiet || ! git diff --cached --quiet; then + echo "error: working tree is dirty - commit or stash first" >&2; exit 1 +fi + +mkdir -p dist +out="dist/last30days.skill" +git archive --format=zip --prefix=last30days/ --output="$out" HEAD + +count=$(unzip -l "$out" | tail -1 | awk '{print $2}') +[ "$count" -le 200 ] || { echo "error: $count files in zip, cap is 200" >&2; exit 1; } +echo "built $out ($count files, $(du -h "$out" | cut -f1))" +``` + +**Patterns to follow:** +- Style of `scripts/sync.sh` (bash, top-of-file comment, `set -euo pipefail`) + +**Test scenarios:** +- Happy path: clean tree, `bash scripts/build-skill.sh` produces `dist/last30days.skill` with =200 files and the top-level entry is `last30days/` +- Happy path: `unzip -p dist/last30days.skill last30days/SKILL.md | head -2` shows `---` (frontmatter start) confirming SKILL.md is at the right location +- Edge case: dirty working tree - script exits non-zero with clear error +- Edge case: idempotent - running twice overwrites cleanly +- Error path: if a future change inflates file count past 200, the defensive `[ "$count" -le 200 ]` check fails and the script refuses to produce a broken output + +**Verification:** +- `bash scripts/build-skill.sh && unzip -l dist/last30days.skill | grep "^ 0 .* last30days/$"` confirms the prefix directory exists +- `unzip -l dist/last30days.skill | grep "last30days/SKILL.md"` confirms SKILL.md is at the expected path +- `unzip -l dist/last30days.skill | grep -c "SKILL.md"` returns exactly 1 +- `unzip -l dist/last30days.skill | awk '{print $NF}' | grep -v "^$" | sort -u | grep "skills/" || true` returns nothing (confirms internal skill dirs excluded) +- Gate: a contributor must run `bash scripts/build-skill.sh` on their branch and attach the produced file to their PR before merging any change that touches `.gitattributes` or exclude-sensitive paths + +- [ ] **Unit 5: Document the upload path in README** + +**Goal:** Users know how to produce an upload `.skill` without reading the source. + +**Requirements:** R3 + +**Dependencies:** Unit 4 + +**Files:** +- Modify: `README.md` (add a short "Upload as a Claude Skill" subsection under the existing install section) + +**Approach:** +- One paragraph plus a single command block: `bash scripts/build-skill.sh` +- Mention the 200-file cap as context so future changes do not bust it +- Point users at the claude.ai skill upload UI (note: link only if a stable URL exists at implementation time, otherwise describe the UI path) + +**Test scenarios:** +- Test expectation: none - pure documentation change + +**Verification:** +- `grep -n "build-skill" README.md` returns a hit +- Instructions match actual script behavior + +- [ ] **Unit 6: Trim SKILL.md description to =200 chars** + +**Goal:** Make root `SKILL.md` frontmatter pass Anthropic's documented `description` limit. + +**Requirements:** R7 + +**Dependencies:** None (independent of other units) + +**Files:** +- Modify: `SKILL.md` (frontmatter `description:` field only) + +**Approach:** +- Current description is 228 chars. Cut 28+ chars without losing signal. +- Suggested rewrite (196 chars): `"Multi-query social search with planned queries. Research any topic across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web. Gemini/OpenAI fallback when needed."` +- Confirm the trimmed version still surfaces for the right prompts (smoke test: run `python scripts/last30days.py "test" --emit=compact` and confirm behavior unchanged; description is metadata, not runtime input) +- Update `skills/last30days/SKILL.md` description too if it exceeds 200 chars (check during implementation) + +**Test scenarios:** +- Happy path: `python3 -c "import re; d=open('SKILL.md').read(); m=re.search(r'^description:\s*\"(.+?)\"', d, re.M); assert len(m.group(1)) <= 200, len(m.group(1))"` passes + +**Verification:** +- Description field is =200 chars in root SKILL.md +- Skill still triggers on relevant prompts (manual smoke check) + +- [ ] **Unit 7: Version bump and changelog** + +**Goal:** Ship as 3.0.1 so consumers see the packaging fix. + +**Requirements:** R5 + +**Dependencies:** Units 1-6 + +**Files:** +- Modify: `.claude-plugin/plugin.json` (3.0.0 - 3.0.1) +- Modify: `SKILL.md` frontmatter version +- Modify: `skills/last30days/SKILL.md` frontmatter version +- Modify: `gemini-extension.json` version (note: #190 flags this as stale at 2.9.5; bumping here partially addresses that but full resolution is out of scope) +- Modify: `CHANGELOG.md` +- Modify: `release-notes.md` + +**Approach:** +- Atomic version bump across all manifests +- Changelog entry: "Packaging: `scripts/build-skill.sh` produces a compliant `.skill` file; removed unused root `vendor/` (215 files) and legacy `plans/`; repo file count fits under claude.ai's 200-file upload cap" + +**Test scenarios:** +- Happy path: `grep -rn "3.0.1" SKILL.md skills/last30days/SKILL.md .claude-plugin/plugin.json gemini-extension.json` returns four consistent hits +- Integration: `bash scripts/sync.sh` completes cleanly + +**Verification:** +- All four version declarations read `3.0.1` +- CHANGELOG and release-notes have dated entries + +## System-Wide Impact + +- **Interaction graph:** Skill-runtime import graph is unchanged. Removed code (root `vendor/`, `plans/`) has zero importers. +- **Error propagation:** `build-skill.sh` is a new surface; failure mode is non-zero exit with clear stderr. No runtime error paths touched. +- **State lifecycle risks:** None. `dist/` is gitignored build output. +- **API surface parity:** No change to any user-facing API, CLI flag, config key, or SKILL.md contract. +- **Integration coverage:** `tests/test_bird_x.py` exercises the real vendored client - if it regressed, the test fails. Run it after Unit 1. +- **Unchanged invariants:** `scripts/lib/vendor/bird-search/` stays. `scripts/sync.sh` deploy behavior unchanged. ClawHub publish flow (private repo) untouched. Claude Code plugin install via GitHub URL still works. + +## Risks and Dependencies + +| Risk | Mitigation | +|------|------------| +| Deleting `vendor/` silently breaks something we missed | Run `pytest tests/test_bird_x.py` and `bash scripts/sync.sh` after the delete; grep for `vendor/package` before merging | +| claude.ai rejects the `.skill` file for a reason other than file count (e.g., frontmatter character, hidden file) | Test-upload the produced artifact against claude.ai once before merging; iterate on `.gitattributes` if needed | +| `.gitattributes` over-excludes and breaks the runtime skill | Unit 3 verification step explicitly checks runtime paths are present in the produced archive | +| A future PR re-vendors something at `/vendor/` and busts the 200 cap again | `/vendor/` in `.gitignore` plus the defensive `=200` check in `build-skill.sh` catches it | +| Version bump collides with in-flight PRs that also bump versions | Coordinate with #229, #217 which touched version strings; check before merging | +| `skills/last30days/SKILL.md` (internal spec) being shipped inside the zip confuses the claude.ai uploader | Resolved by excluding `skills/` from the zip (Unit 3). Internal spec remains in the repo for plugin consumers | +| `skills/last30days-nux/SKILL.md` is a symlink to `../../SKILL.md`; claude.ai may reject zips with symlink entries | Resolved by excluding `skills/` from the zip (Unit 3). Symlink never enters the archive | + +## Documentation and Operational Notes + +- Update README only (Unit 5). No runbook, no migration, no flag. +- No deployment step - plugin consumers get the packaging fix automatically on next update. +- Release notes flag: manual uploaders should re-zip via `scripts/build-skill.sh`. +- Opportunistic future work (out of scope here): CI check that fails PRs that push the zip over 200 files. + +## Sources and References + +- Trevin's 2026-04-14 chat screenshot: "Zip contains too many files (maximum 200)" +- [anthropics/skills README](https://github.com/anthropics/skills/blob/main/README.md) - YAML frontmatter requirements +- [anthropics/skills package_skill.py](https://github.com/anthropics/skills/blob/main/skills/skill-creator/scripts/package_skill.py) - canonical exclusions, arcname convention, validation gates +- [claude.ai skill help center](https://support.claude.com/en/articles/12512180-use-skills-in-claude) - upload failure modes (zip size, folder-name mismatch, missing SKILL.md) +- PR #48 (2026-02) - the merge that introduced `vendor/package/` +- Open issues adjacent but out of scope: #239, #236, #231, #190, #184 +- Related code: `scripts/lib/bird_x.py:5`, `tests/test_bird_x.py:133`, `.clawhubignore`, `scripts/sync.sh`, root `SKILL.md` frontmatter +- Private-repo reference pattern: `scripts/clawhub-publish.sh` + `scripts/strip_for_openclaw.py` - not copied here; `git archive` is simpler for the public path diff --git a/plans/feat-add-websearch-source.md b/plans/feat-add-websearch-source.md deleted file mode 100644 index d9cc103..0000000 --- a/plans/feat-add-websearch-source.md +++ /dev/null @@ -1,395 +0,0 @@ -# feat: Add WebSearch as Third Source (Zero-Config Fallback) - -## Overview - -Add Claude's built-in WebSearch tool as a third research source for `/last30days`. This enables the skill to work **out of the box with zero API keys** while preserving the primacy of Reddit/X as the "voice of real humans with popularity signals." - -**Key principle**: WebSearch is supplementary, not primary. Real human voices on Reddit/X with engagement metrics (upvotes, likes, comments) are more valuable than general web content. - -## Problem Statement - -Currently `/last30days` requires at least one API key (OpenAI or xAI) to function. Users without API keys get an error. Additionally, web search could fill gaps where Reddit/X coverage is thin. - -**User requirements**: -- Work out of the box (no API key needed) -- Must NOT overpower Reddit/X results -- Needs proper weighting -- Validate with before/after testing - -## Proposed Solution - -### Weighting Strategy: "Engagement-Adjusted Scoring" - -**Current formula** (same for Reddit/X): -``` -score = 0.45*relevance + 0.25*recency + 0.30*engagement - penalties -``` - -**Problem**: WebSearch has NO engagement metrics. Giving it `DEFAULT_ENGAGEMENT=35` with `-10 penalty` = 25 base, which still competes unfairly. - -**Solution**: Source-specific scoring with **engagement substitution**: - -| Source | Relevance | Recency | Engagement | Source Penalty | -|--------|-----------|---------|------------|----------------| -| Reddit | 45% | 25% | 30% (real metrics) | 0 | -| X | 45% | 25% | 30% (real metrics) | 0 | -| WebSearch | 55% | 35% | 0% (no data) | -15 points | - -**Rationale**: -- WebSearch items compete on relevance + recency only (reweighted to 100%) -- `-15 point source penalty` ensures WebSearch ranks below comparable Reddit/X items -- High-quality WebSearch can still surface (score 60-70) but won't dominate (Reddit/X score 70-85) - -### Mode Behavior - -| API Keys Available | Default Behavior | `--include-web` | -|--------------------|------------------|-----------------| -| None | **WebSearch only** | n/a | -| OpenAI only | Reddit only | Reddit + WebSearch | -| xAI only | X only | X + WebSearch | -| Both | Reddit + X | Reddit + X + WebSearch | - -**CLI flag**: `--include-web` (default: false when other sources available) - -## Technical Approach - -### Architecture - -``` -┌─────────────────────────────────────────────────────────────────┐ -│ last30days.py orchestrator │ -├─────────────────────────────────────────────────────────────────┤ -│ run_research() │ -│ ├── if sources includes "reddit": openai_reddit.search_reddit()│ -│ ├── if sources includes "x": xai_x.search_x() │ -│ └── if sources includes "web": websearch.search_web() ← NEW │ -└─────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ Processing Pipeline │ -├─────────────────────────────────────────────────────────────────┤ -│ normalize_websearch_items() → WebSearchItem schema ← NEW │ -│ score_websearch_items() → engagement-free scoring ← NEW │ -│ dedupe_websearch() → deduplication ← NEW │ -│ render_websearch_section() → output formatting ← NEW │ -└─────────────────────────────────────────────────────────────────┘ -``` - -### Implementation Phases - -#### Phase 1: Schema & Core Infrastructure - -**Files to create/modify:** - -```python -# scripts/lib/websearch.py (NEW) -"""Claude WebSearch API client for general web discovery.""" - -WEBSEARCH_PROMPT = """Search the web for content about: {topic} - -CRITICAL: Only include results from the last 30 days (after {from_date}). - -Find {min_items}-{max_items} high-quality, relevant web pages. Prefer: -- Blog posts, tutorials, documentation -- News articles, announcements -- Authoritative sources (official docs, reputable publications) - -AVOID: -- Reddit (covered separately) -- X/Twitter (covered separately) -- YouTube without transcripts -- Forum threads without clear answers - -Return ONLY valid JSON: -{{ - "items": [ - {{ - "title": "Page title", - "url": "https://...", - "source_domain": "example.com", - "snippet": "Brief excerpt (100-200 chars)", - "date": "YYYY-MM-DD or null", - "why_relevant": "Brief explanation", - "relevance": 0.85 - }} - ] -}} -""" - -def search_web(topic: str, from_date: str, to_date: str, depth: str = "default") -> dict: - """Search web using Claude's built-in WebSearch tool. - - NOTE: This runs INSIDE Claude Code, so we use the WebSearch tool directly. - No API key needed - uses Claude's session. - """ - # Implementation uses Claude's web_search_20250305 tool - pass - -def parse_websearch_response(response: dict) -> list[dict]: - """Parse WebSearch results into normalized format.""" - pass -``` - -```python -# scripts/lib/schema.py - ADD WebSearchItem - -@dataclass -class WebSearchItem: - """Normalized web search item.""" - id: str - title: str - url: str - source_domain: str # e.g., "medium.com", "github.com" - snippet: str - date: Optional[str] = None - date_confidence: str = "low" - relevance: float = 0.5 - why_relevant: str = "" - subs: SubScores = field(default_factory=SubScores) - score: int = 0 - - def to_dict(self) -> Dict[str, Any]: - return { - 'id': self.id, - 'title': self.title, - 'url': self.url, - 'source_domain': self.source_domain, - 'snippet': self.snippet, - 'date': self.date, - 'date_confidence': self.date_confidence, - 'relevance': self.relevance, - 'why_relevant': self.why_relevant, - 'subs': self.subs.to_dict(), - 'score': self.score, - } -``` - -#### Phase 2: Scoring System Updates - -```python -# scripts/lib/score.py - ADD websearch scoring - -# New constants -WEBSEARCH_SOURCE_PENALTY = 15 # Points deducted for lacking engagement - -# Reweighted for no engagement -WEBSEARCH_WEIGHT_RELEVANCE = 0.55 -WEBSEARCH_WEIGHT_RECENCY = 0.45 - -def score_websearch_items(items: List[schema.WebSearchItem]) -> List[schema.WebSearchItem]: - """Score WebSearch items WITHOUT engagement metrics. - - Uses reweighted formula: 55% relevance + 45% recency - 15pt source penalty - """ - for item in items: - rel_score = int(item.relevance * 100) - rec_score = dates.recency_score(item.date) - - item.subs = schema.SubScores( - relevance=rel_score, - recency=rec_score, - engagement=0, # Explicitly zero - no engagement data - ) - - overall = ( - WEBSEARCH_WEIGHT_RELEVANCE * rel_score + - WEBSEARCH_WEIGHT_RECENCY * rec_score - ) - - # Apply source penalty (WebSearch < Reddit/X) - overall -= WEBSEARCH_SOURCE_PENALTY - - # Apply date confidence penalty (same as other sources) - if item.date_confidence == "low": - overall -= 10 - elif item.date_confidence == "med": - overall -= 5 - - item.score = max(0, min(100, int(overall))) - - return items -``` - -#### Phase 3: Orchestrator Integration - -```python -# scripts/last30days.py - UPDATE run_research() - -def run_research(...) -> tuple: - """Run the research pipeline. - - Returns: (reddit_items, x_items, web_items, raw_openai, raw_xai, - raw_websearch, reddit_error, x_error, web_error) - """ - # ... existing Reddit/X code ... - - # WebSearch (new) - web_items = [] - raw_websearch = None - web_error = None - - if sources in ("all", "web", "reddit-web", "x-web"): - if progress: - progress.start_web() - - try: - raw_websearch = websearch.search_web(topic, from_date, to_date, depth) - web_items = websearch.parse_websearch_response(raw_websearch) - except Exception as e: - web_error = f"{type(e).__name__}: {e}" - - if progress: - progress.end_web(len(web_items)) - - return (reddit_items, x_items, web_items, raw_openai, raw_xai, - raw_websearch, reddit_error, x_error, web_error) -``` - -#### Phase 4: CLI & Environment Updates - -```python -# scripts/last30days.py - ADD CLI flag - -parser.add_argument( - "--include-web", - action="store_true", - help="Include general web search alongside Reddit/X (lower weighted)", -) - -# scripts/lib/env.py - UPDATE get_available_sources() - -def get_available_sources(config: dict) -> str: - """Determine available sources. WebSearch always available (no API key).""" - has_openai = bool(config.get('OPENAI_API_KEY')) - has_xai = bool(config.get('XAI_API_KEY')) - - if has_openai and has_xai: - return 'both' # WebSearch available but not default - elif has_openai: - return 'reddit' - elif has_xai: - return 'x' - else: - return 'web' # Fallback: WebSearch only (no keys needed) -``` - -## Acceptance Criteria - -### Functional Requirements - -- [x] Skill works with zero API keys (WebSearch-only mode) -- [x] `--include-web` flag adds WebSearch to Reddit/X searches -- [x] WebSearch items have lower average scores than Reddit/X items with similar relevance -- [x] WebSearch results exclude Reddit/X URLs (handled separately) -- [x] Date filtering uses natural language ("last 30 days") in prompt -- [x] Output clearly labels source type: `[WEB]`, `[Reddit]`, `[X]` - -### Non-Functional Requirements - -- [x] WebSearch adds <10s latency to total research time (0s - deferred to Claude) -- [x] Graceful degradation if WebSearch fails -- [ ] Cache includes WebSearch results appropriately - -### Quality Gates - -- [x] Before/after testing shows WebSearch doesn't dominate rankings (via -15pt penalty) -- [x] Test: 10 Reddit + 10 X + 10 WebSearch → WebSearch avg score 15-20pts lower (scoring formula verified) -- [x] Test: WebSearch-only mode produces useful results for common topics - -## Testing Plan - -### Before/After Comparison Script - -```python -# tests/test_websearch_weighting.py - -""" -Test harness to validate WebSearch doesn't overpower Reddit/X. - -Run same queries with: -1. Reddit + X only (baseline) -2. Reddit + X + WebSearch (comparison) - -Verify: WebSearch items rank lower on average. -""" - -TEST_QUERIES = [ - "best practices for react server components", - "AI coding assistants comparison", - "typescript 5.5 new features", -] - -def test_websearch_weighting(): - for query in TEST_QUERIES: - # Run without WebSearch - baseline = run_research(query, sources="both") - baseline_scores = [item.score for item in baseline.reddit + baseline.x] - - # Run with WebSearch - with_web = run_research(query, sources="both", include_web=True) - web_scores = [item.score for item in with_web.web] - reddit_x_scores = [item.score for item in with_web.reddit + with_web.x] - - # Assertions - avg_reddit_x = sum(reddit_x_scores) / len(reddit_x_scores) - avg_web = sum(web_scores) / len(web_scores) if web_scores else 0 - - assert avg_web < avg_reddit_x - 10, \ - f"WebSearch avg ({avg_web}) too close to Reddit/X avg ({avg_reddit_x})" - - # Check top 5 aren't all WebSearch - top_5 = sorted(with_web.reddit + with_web.x + with_web.web, - key=lambda x: -x.score)[:5] - web_in_top_5 = sum(1 for item in top_5 if isinstance(item, WebSearchItem)) - assert web_in_top_5 <= 2, f"Too many WebSearch items in top 5: {web_in_top_5}" -``` - -### Manual Test Scenarios - -| Scenario | Expected Outcome | -|----------|------------------| -| No API keys, run `/last30days AI tools` | WebSearch-only results, useful output | -| Both keys + `--include-web`, run `/last30days react` | Mix of all 3 sources, Reddit/X dominate top 10 | -| Niche topic (no Reddit/X coverage) | WebSearch fills gap, becomes primary | -| Popular topic (lots of Reddit/X) | WebSearch present but lower-ranked | - -## Dependencies & Prerequisites - -- Claude Code's WebSearch tool (`web_search_20250305`) - already available -- No new API keys required -- Existing test infrastructure in `tests/` - -## Risk Analysis & Mitigation - -| Risk | Likelihood | Impact | Mitigation | -|------|------------|--------|------------| -| WebSearch returns stale content | Medium | Medium | Enforce date in prompt, apply low-confidence penalty | -| WebSearch dominates rankings | Low | High | Source penalty (-15pts), testing validates | -| WebSearch adds spam/low-quality | Medium | Medium | Exclude social media domains, domain filtering | -| Date parsing unreliable | High | Medium | Accept "low" confidence as normal for WebSearch | - -## Future Considerations - -1. **Domain authority scoring**: Could proxy engagement with domain reputation -2. **User-configurable weights**: Let users adjust WebSearch penalty -3. **Domain whitelist/blacklist**: Filter WebSearch to trusted sources -4. **Parallel execution**: Run all 3 sources concurrently for speed - -## References - -### Internal References -- Scoring algorithm: `scripts/lib/score.py:8-15` -- Source detection: `scripts/lib/env.py:57-72` -- Schema patterns: `scripts/lib/schema.py:76-138` -- Orchestrator: `scripts/last30days.py:54-164` - -### External References -- Claude WebSearch docs: https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool -- WebSearch pricing: $10/1K searches + token costs -- Date filtering limitation: No explicit date params, use natural language - -### Research Findings -- Reddit upvotes are ~12% of ranking value in SEO (strong signal) -- E-E-A-T framework: Engagement metrics = trust signal -- MSA2C2 approach: Dynamic weight learning for multi-source aggregation diff --git a/plans/fix-strict-date-filtering.md b/plans/fix-strict-date-filtering.md deleted file mode 100644 index 2c0cd85..0000000 --- a/plans/fix-strict-date-filtering.md +++ /dev/null @@ -1,328 +0,0 @@ -# fix: Enforce Strict 30-Day Date Filtering - -## Overview - -The `/last30days` skill is returning content older than 30 days, violating its core promise. Analysis shows: -- **Reddit**: Only 40% of results within 30 days (9/15 were older, some from 2022!) -- **X**: 100% within 30 days (working correctly) -- **WebSearch**: 90% had unknown dates (can't verify freshness) - -## Problem Statement - -The skill's name is "last30days" - users expect ONLY content from the last 30 days. Currently: - -1. **Reddit search prompt** says "prefer recent threads, but include older relevant ones if recent ones are scarce" - this is too permissive -2. **X search prompt** explicitly includes `from_date` and `to_date` - this is why it works -3. **WebSearch** returns pages without publication dates - we can't verify they're recent -4. **Scoring penalties** (-10 for low date confidence) don't prevent old content from appearing - -## Proposed Solution - -### Strategy: "Hard Filter, Not Soft Penalty" - -Instead of penalizing old content, **exclude it entirely**. If it's not from the last 30 days, it shouldn't appear. - -| Source | Current Behavior | New Behavior | -|--------|------------------|--------------| -| Reddit | Weak "prefer recent" | Explicit date range + hard filter | -| X | Explicit date range (working) | No change needed | -| WebSearch | No date awareness | Require recent markers OR exclude | - -## Technical Approach - -### Phase 1: Fix Reddit Date Filtering - -**File: `scripts/lib/openai_reddit.py`** - -Current prompt (line 33): -``` -Find {min_items}-{max_items} relevant Reddit discussion threads. -Prefer recent threads, but include older relevant ones if recent ones are scarce. -``` - -New prompt: -``` -Find {min_items}-{max_items} relevant Reddit discussion threads from {from_date} to {to_date}. - -CRITICAL: Only include threads posted within the last 30 days (after {from_date}). -Do NOT include threads older than {from_date}, even if they seem relevant. -If you cannot find enough recent threads, return fewer results rather than older ones. -``` - -**Changes needed:** -1. Add `from_date` and `to_date` parameters to `search_reddit()` function -2. Inject dates into `REDDIT_SEARCH_PROMPT` like X does -3. Update caller in `last30days.py` to pass dates - -### Phase 2: Add Hard Date Filtering (Post-Processing) - -**File: `scripts/lib/normalize.py`** - -Add a filter step that DROPS items with dates before `from_date`: - -```python -def filter_by_date_range( - items: List[Union[RedditItem, XItem, WebSearchItem]], - from_date: str, - to_date: str, - require_date: bool = False, -) -> List: - """Hard filter: Remove items outside the date range. - - Args: - items: List of items to filter - from_date: Start date (YYYY-MM-DD) - to_date: End date (YYYY-MM-DD) - require_date: If True, also remove items with no date - - Returns: - Filtered list with only items in range - """ - result = [] - for item in items: - if item.date is None: - if not require_date: - result.append(item) # Keep unknown dates (with penalty) - continue - - # Hard filter: if date is before from_date, exclude - if item.date < from_date: - continue # DROP - too old - - if item.date > to_date: - continue # DROP - future date (likely parsing error) - - result.append(item) - - return result -``` - -### Phase 3: WebSearch Date Intelligence - -WebSearch CAN find recent content - Medium posts have dates, GitHub has commit timestamps, news sites have publication dates. We should **extract and prioritize** these signals. - -**Strategy: "Date Detective"** - -1. **Extract dates from URLs**: Many sites embed dates in URLs - - Medium: `medium.com/@author/title-abc123` (no date) vs news sites - - GitHub: Look for commit dates, release dates in snippets - - News: `/2026/01/24/article-title` - - Blogs: `/blog/2026/01/title` - -2. **Extract dates from snippets**: Look for date markers - - "January 24, 2026", "Jan 2026", "yesterday", "this week" - - "Published:", "Posted:", "Updated:" - - Relative markers: "2 days ago", "last week" - -3. **Prioritize results with verifiable dates**: - - Results with recent dates (within 30 days): Full score - - Results with old dates: EXCLUDE - - Results with no date signals: Heavy penalty (-20) but keep as supplementary - -**File: `scripts/lib/websearch.py`** - -Add date extraction functions: - -```python -import re -from datetime import datetime, timedelta - -# Patterns for date extraction -URL_DATE_PATTERNS = [ - r'/(\d{4})/(\d{2})/(\d{2})/', # /2026/01/24/ - r'/(\d{4})-(\d{2})-(\d{2})/', # /2026-01-24/ - r'/(\d{4})(\d{2})(\d{2})/', # /20260124/ -] - -SNIPPET_DATE_PATTERNS = [ - r'(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]* (\d{1,2}),? (\d{4})', - r'(\d{1,2}) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]* (\d{4})', - r'(\d{4})-(\d{2})-(\d{2})', - r'Published:?\s*(\d{4}-\d{2}-\d{2})', - r'(\d{1,2}) (days?|hours?|minutes?) ago', # Relative dates -] - -def extract_date_from_url(url: str) -> Optional[str]: - """Try to extract a date from URL path.""" - for pattern in URL_DATE_PATTERNS: - match = re.search(pattern, url) - if match: - # Parse and return YYYY-MM-DD format - ... - return None - -def extract_date_from_snippet(snippet: str) -> Optional[str]: - """Try to extract a date from text snippet.""" - for pattern in SNIPPET_DATE_PATTERNS: - match = re.search(pattern, snippet, re.IGNORECASE) - if match: - # Parse and return YYYY-MM-DD format - ... - return None - -def extract_date_signals(url: str, snippet: str, title: str) -> tuple[Optional[str], str]: - """Extract date from any available signal. - - Returns: (date_string, confidence) - - date from URL: 'high' confidence - - date from snippet: 'med' confidence - - no date found: None, 'low' confidence - """ - # Try URL first (most reliable) - url_date = extract_date_from_url(url) - if url_date: - return url_date, 'high' - - # Try snippet - snippet_date = extract_date_from_snippet(snippet) - if snippet_date: - return snippet_date, 'med' - - # Try title - title_date = extract_date_from_snippet(title) - if title_date: - return title_date, 'med' - - return None, 'low' -``` - -**Update WebSearch parsing to use date extraction:** - -```python -def parse_websearch_results(results, topic, from_date, to_date): - items = [] - for result in results: - url = result.get('url', '') - snippet = result.get('snippet', '') - title = result.get('title', '') - - # Extract date signals - extracted_date, confidence = extract_date_signals(url, snippet, title) - - # Hard filter: if we found a date and it's too old, skip - if extracted_date and extracted_date < from_date: - continue # DROP - verified old content - - item = { - 'date': extracted_date, - 'date_confidence': confidence, - ... - } - items.append(item) - - return items -``` - -**File: `scripts/lib/score.py`** - -Update WebSearch scoring to reward date-verified results: - -```python -# WebSearch date confidence adjustments -WEBSEARCH_NO_DATE_PENALTY = 20 # Heavy penalty for no date (was 10) -WEBSEARCH_VERIFIED_BONUS = 10 # Bonus for URL-verified recent date - -def score_websearch_items(items): - for item in items: - ... - # Date confidence adjustments - if item.date_confidence == 'high': - overall += WEBSEARCH_VERIFIED_BONUS # Reward verified dates - elif item.date_confidence == 'low': - overall -= WEBSEARCH_NO_DATE_PENALTY # Heavy penalty for unknown - ... -``` - -**Result**: WebSearch results with verifiable recent dates rank well. Results with no dates are heavily penalized but still appear as supplementary context. Old verified content is excluded entirely. - -### Phase 4: Update Statistics Display - -Only count Reddit and X in "from the last 30 days" claim. WebSearch should be clearly labeled as supplementary. - -## Acceptance Criteria - -### Functional Requirements - -- [x] Reddit search prompt includes explicit `from_date` and `to_date` -- [x] Items with dates before `from_date` are EXCLUDED, not just penalized -- [x] X search continues working (no regression) -- [x] WebSearch extracts dates from URLs (e.g., `/2026/01/24/`) -- [x] WebSearch extracts dates from snippets (e.g., "January 24, 2026") -- [x] WebSearch with verified recent dates gets +10 bonus -- [x] WebSearch with no date signals gets -20 penalty (but still appears) -- [x] WebSearch with verified OLD dates is EXCLUDED - -### Non-Functional Requirements - -- [ ] No increase in API latency -- [ ] Graceful handling when few recent results exist (return fewer, not older) -- [ ] Clear user messaging when results are limited due to strict filtering - -### Quality Gates - -- [ ] Test: Reddit search returns 0% results older than 30 days -- [ ] Test: X search continues to return 100% recent results -- [ ] Test: WebSearch is clearly differentiated in output -- [ ] Test: Edge case - topic with no recent content shows helpful message - -## Implementation Order - -1. **Phase 1**: Fix Reddit prompt (highest impact, simple change) -2. **Phase 2**: Add hard date filter in normalize.py (safety net) -3. **Phase 3**: Add WebSearch date extraction (URL + snippet parsing) -4. **Phase 4**: Update WebSearch scoring (bonus for verified, heavy penalty for unknown) -5. **Phase 5**: Update output display to show date confidence - -## Testing Plan - -### Before/After Test - -Run same query before and after fix: -``` -/last30days remotion launch videos -``` - -**Expected Before:** -- Reddit: 40% within 30 days - -**Expected After:** -- Reddit: 100% within 30 days (or fewer results if not enough recent content) - -### Edge Case Tests - -| Scenario | Expected Behavior | -|----------|-------------------| -| Topic with no recent content | Return 0 results + helpful message | -| Topic with 5 recent results | Return 5 results (not pad with old ones) | -| Mixed old/new results | Only return new ones | - -### WebSearch Date Extraction Tests - -| URL/Snippet | Expected Date | Confidence | -|-------------|---------------|------------| -| `medium.com/blog/2026/01/15/title` | 2026-01-15 | high | -| `github.com/repo` + "Released Jan 20, 2026" | 2026-01-20 | med | -| `docs.example.com/guide` (no date signals) | None | low | -| `news.site.com/2024/05/old-article` | 2024-05-XX | EXCLUDE (too old) | -| Snippet: "Updated 3 days ago" | calculated | med | - -## Risk Analysis - -| Risk | Likelihood | Impact | Mitigation | -|------|------------|--------|------------| -| Fewer results for niche topics | High | Medium | Explain why in output | -| User confusion about reduced results | Medium | Low | Clear messaging | -| Date parsing errors exclude valid content | Low | Medium | Keep items with unknown dates, just label clearly | - -## References - -### Internal References -- Reddit search: `scripts/lib/openai_reddit.py:25-63` -- X search (working example): `scripts/lib/xai_x.py:26-55` -- Date confidence: `scripts/lib/dates.py:62-90` -- Scoring penalties: `scripts/lib/score.py:149-153` -- Normalization: `scripts/lib/normalize.py:49,99` - -### External References -- OpenAI Responses API lacks native date filtering -- Must rely on prompt engineering + post-processing