Commit Graph

535 Commits

Author SHA1 Message Date
Trevin Chow 1e4150ad78 Merge pull request #347 from dinakars777/docs/update-how-search-key-files
docs: update search key file paths
2026-05-17 00:24:55 -07:00
Trevin Chow d9e8a046ef docs(how-search): swap score.py->relevance.py at line 145 2026-05-17 00:21:36 -07:00
Dinakar Sarbada 2f4b023db8 docs: update search key file paths 2026-05-17 00:21:15 -07:00
Trevin Chow 2692e0f4a2 Merge pull request #351 from dinakars777/docs/fix-changelog-skill-link-note
docs: correct changelog skill link note
2026-05-17 00:20:46 -07:00
Trevin Chow 6b40d2c46f Merge pull request #346 from dinakars777/docs/fix-bug-report-repro-command
docs: fix bug report repro command
2026-05-17 00:20:21 -07:00
Trevin Chow 4a99c4f557 Merge pull request #337 from UncleMike1988/fix/path-quoting-spaces
Fix path-quoting in SessionStart check-config hook (handles spaces in…
2026-05-17 00:20:09 -07:00
Dinakar Sarbada 608381a818 docs: fix bug report repro command 2026-05-17 00:08:17 -07:00
Michael Turner 5b0308b9e4 Fix path-quoting in SessionStart check-config hook (handles spaces in CLAUDE_PLUGIN_ROOT)
If CLAUDE_PLUGIN_ROOT ever expands to a path containing whitespace
(e.g. ~/Library/Application Support/...), the unquoted ${CLAUDE_PLUGIN_ROOT}
in hooks/hooks.json word-splits and bash receives the path as multiple
arguments, failing with "No such file or directory" on the first split.

Quoting the expansion makes the invocation correct regardless of the
characters in the resolved path. Verified manually:
  unquoted + space  -> bash: /tmp/with: No such file or directory
  quoted   + space  -> bash: /tmp/with spaces/.../check-config.sh: No such file
  quoted   + real   -> /last30days: Ready - 7 sources active.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 00:08:01 -07:00
Trevin Chow 7214dd6051 Merge pull request #348 from dinakars777/docs/fix-readme-skill-link
docs: fix runtime skill spec link
2026-05-17 00:04:30 -07:00
Trevin Chow 6acf2fdbe2 Merge pull request #419 from mvanhorn/chore/remove-orphaned-spec-tasks
chore: remove orphaned SPEC.md and TASKS.md
2026-05-17 00:03:06 -07:00
Trevin Chow 07a3bdb3cf Merge pull request #364 from davemorin/fix/361-unsafe-eval-check-config
fix(hooks): replace unsafe eval with declare in check-config.sh
2026-05-17 00:02:44 -07:00
Dinakar Sarbada 400fc4cc00 docs: fix runtime skill spec link 2026-05-17 00:00:58 -07:00
Trevin Chow f3df47c381 chore: remove orphaned SPEC.md and TASKS.md
Both files lived at the repo root as pre-plugin-layout artifacts. On
current main neither is referenced from README, SKILL.md, AGENTS.md,
CHANGELOG, or docs/ — no inbound links to break by removing. Git history
preserves the content for anyone who needs to dig it up.

Closes #352, #353. The PRs by @dinakars777 correctly flagged the drift;
deletion is the cleaner resolution than annotating them as historical.
2026-05-17 00:00:47 -07:00
Dinakar Sarbada c9cf3ef92f docs: correct changelog skill link note 2026-05-16 23:59:30 -07:00
Trevin Chow 46cf2328aa fix(hooks): use printf -v for bash 3.2 compat (declare -g is 4.2+)
macOS ships /bin/bash 3.2 and the script uses #!/bin/bash with
set -euo pipefail, so declare -g would abort the SessionStart hook
with "invalid option" on every Mac. printf -v writes via assignment
semantics (global from inside a function on 3.2+) — same scope
outcome, broader compatibility.
2026-05-16 23:49:44 -07:00
Trevin Chow 7506cbd542 fix(hooks): scope ENV_* to global (declare -g) so caller sees values 2026-05-16 23:49:05 -07:00
Dave Morin a6bd481e61 fix(hooks): replace unsafe eval with declare in check-config.sh
The load_env_vars function used eval to assign .env values, which
executes command substitutions in backtick-containing comments.
Replace eval with declare and strip inline comments before assignment.

Fixes #361
2026-05-16 23:49:05 -07:00
Trevin Chow aba6172032 Merge pull request #366 from davemorin/feat/324-reddit-json-fallback
feat(web): auto-enrich Reddit URLs from web search via JSON API
2026-05-16 23:41:57 -07:00
Trevin Chow d07e4698e3 Merge pull request #358 from dinakars777/fix/openclaw-poll-clock-init
fix: initialize OpenClaw poll timing once
2026-05-16 23:41:42 -07:00
Trevin Chow 4c0282dd55 Merge pull request #365 from davemorin/fix/284-version-metadata-drift
fix(version): replace hardcoded v3.0.0 with dynamic _skill_version()
2026-05-16 23:40:53 -07:00
Trevin Chow 99909fca67 Merge pull request #368 from hnshah/ren/advisory-security-workflow-252
ci: add advisory security workflow
2026-05-16 23:40:28 -07:00
Trevin Chow 36c43d50b7 test: drop side_effect padding to match collapsed time.time() call 2026-05-16 23:39:14 -07:00
Dinakar Sarbada ecf68347db fix: initialize OpenClaw poll timing once 2026-05-16 23:38:30 -07:00
Trevin Chow e2d9d705f6 review: fix selftext key path + break on RedditRateLimitError 2026-05-16 23:37:00 -07:00
Hiten Shah 9c09a67ac2 ci: add advisory security workflow 2026-05-16 23:36:55 -07:00
Dave Morin 32da0bd6cb test: update version assertions for dynamic _skill_version()
Tests now check for version prefix without hardcoded version number,
matching the render.py change to use _skill_version() dynamically.
2026-05-16 23:35:26 -07:00
Dave Morin 863c3bc145 fix(version): replace hardcoded v3.0.0 with dynamic _skill_version()
render.py, ui.py, and last30days.py had hardcoded "v3.0.0" in titles
and headers while plugin.json was at 3.1.1. Use _skill_version()
(reads from plugin.json at runtime) so version strings stay in sync.

Fixes #284
2026-05-16 23:35:26 -07:00
Trevin Chow 9f39d10bc5 Merge pull request #373 from hnshah/ren/watchlist-sightings
feat(store): record per-run finding sightings
2026-05-16 23:24:09 -07:00
Trevin Chow 03043da407 Merge pull request #418 from tmchow/chore/greptile-config
chore: add greptile.json (triggerOnUpdates + statusCheck)
2026-05-16 23:23:48 -07:00
Trevin Chow 8bab997854 chore: add greptile.json to opt into update-triggered reviews + status check
Without this config, Greptile's documented default is `triggerOnUpdates: false`
(only the initial PR open triggers a review). Empirically Greptile has been
re-reviewing on force-push to this repo anyway, but documenting the intent
makes the behavior reliable across plan changes and any future config-source
shifts on Greptile's side.

`statusCheck: true` registers Greptile as a GitHub status check (not just a
PR comment). That gives maintainer-tooling a machine-readable heartbeat -
poll `GET /repos/.../commits/SHA/check-runs` and filter by app name to see
whether Greptile is `queued` / `in_progress` / `completed`. Without it the
only signal is "did a new Greptile comment appear" which is silently
ambiguous when Greptile re-reviews and finds nothing new.

If `statusCheck` is OSS-plan-restricted Greptile silently ignores the key,
which is fine - the rolling-summary comment with `Confidence Score: N/5`
remains the fallback signal.

Refs greptileai/skills `greploop` skill for the terminal-state pattern this
config enables.
2026-05-16 23:20:24 -07:00
Hiten Shah 375fd0bcc0 fix(store): enforce sighting finding id invariant 2026-05-16 22:57:32 -07:00
Hiten Shah 92d65723e4 fix(watchlist): refresh sighting retries 2026-05-16 22:57:04 -07:00
Hiten Shah f794f82af5 feat(store): record per-run finding sightings 2026-05-16 22:57:04 -07:00
Trevin Chow 791c0a57a0 review: gate web Reddit enrichment behind EXCLUDE_SOURCES
PR #366 routes Reddit URLs found in web-search results through the public
Reddit JSON API to recover thread body + top comments (the Claude Code
WebFetch tool blocks reddit.com directly). That bypass is sound and the
fixed problem is real - but the always-on shape ignores user intent on
source gating.

A user who sets EXCLUDE_SOURCES=reddit to suppress Reddit results would
still get Reddit content smuggled back in via web-search URLs that
happen to point at reddit.com threads. This contradicts the suppression
contract that EXCLUDE_SOURCES is supposed to provide (see
lib/pipeline.available_sources where the same env var gates the
top-level Reddit source).

Add a _reddit_excluded(config) check in web_search() that mirrors the
parsing pattern from lib/pipeline (comma-separated, case-insensitive,
whitespace-tolerant). When reddit is in EXCLUDE_SOURCES, skip the
enrichment pass entirely - the web results themselves still flow
through, but they're not augmented with Reddit body/comments.

Four new tests in test_grounding_v3.py cover:
- EXCLUDE_SOURCES=reddit skips enrichment
- case-insensitive parsing matches REDDIT/Reddit/whitespace-padded/csv
- Other sources in EXCLUDE_SOURCES don't trigger the gate
- Enrichment runs normally when reddit isn't excluded

19/19 grounding tests pass.
2026-05-16 22:52:06 -07:00
Dave Morin 211df0deaa feat(web): auto-enrich Reddit URLs from web search via JSON API
Web search backends (Brave, Exa, Serper) can return Reddit URLs as
results. Claude Code's WebFetch blocks reddit.com, so the model can't
retrieve full thread content. After web search, detect Reddit URLs
and fetch body text + top comments via reddit.com/.json endpoint
using the skill's own HTTP library.

Fixes #324
2026-05-16 22:50:39 -07:00
Trevin Chow d7b3995da1 Merge pull request #357 from dinakars777/fix/windows-env-permission-warning
fix: skip POSIX secret warning on Windows
2026-05-16 22:42:26 -07:00
Trevin Chow e217db77cc Merge pull request #369 from voidborne-d/fix/scrapecreators-100-credits
docs: correct ScrapeCreators free tier to 100 credits (closes #367)
2026-05-16 22:41:16 -07:00
Dinakar Sarbada 8ea207b348 fix: skip POSIX secret warning on Windows 2026-05-16 22:39:35 -07:00
voidborne-d b04212680d docs: correct ScrapeCreators free tier to 100 credits (closes #367)
The skill advertises ScrapeCreators as offering "10,000 free API calls" in
six places. The actual free tier on the ScrapeCreators pricing page is
"100 credits free · No credit card required · Credits never expire" — a
100x overstatement that surprises users on signup.

Reporter (#367) burned through their full free allocation on a single
/last30days run after taking the 10,000-call claim at face value. They
verified the actual tier directly against scrapecreators.com plus an
independent review at fahimai.com.

Sweep:
- hooks/scripts/check-config.sh:110  (SessionStart hook tip line)
- README.md:228                      (Sources × Cost table row)
- HERMES_SETUP.md:62                 (Optional: ScrapeCreators bullet)
- skills/last30days/scripts/lib/ui.py:199  (PROMO_SINGLE_KEY["reddit"])
- skills/last30days/SKILL.md:1648    ("PAYG after 10,000 free API calls")
- skills/last30days/SKILL.md:1661    ("10,000 free API calls, then PAYG")

Wording defaults to the provider's own framing — "100 free credits" — and
keeps PAYG language where it was already explicit, since the paid step is
the part users were actually getting blindsided by.

CI gates: tests/test_plugin_contract.py (4) + tests/test_version_consistency.py (4)
all pass.  shellcheck clean.  No tests pin the "10,000" string.
2026-05-16 22:33:41 -07:00
Trevin Chow 2c2cfb9e7e Merge pull request #417 from tmchow/docs/eval-not-in-ci-solution
docs: capture eval-not-in-CI design decision under docs/solutions/
2026-05-16 22:31:39 -07:00
Trevin Chow 3276496f49 Merge pull request #376 from shoobee/feat/yt-dlp-ssh-routing
feat(youtube): route yt-dlp through SSH host for residential IP egress
2026-05-16 22:31:18 -07:00
Trevin Chow 68ae74ff4f docs: capture eval-not-in-CI design decision under docs/solutions/
Closes #374 (adapted, not 1:1 merged).

@hnshah opened PR #374 proposing a docs/adr/ directory for architecture
decision records. The intent is right -- the "why is search-quality eval
manual?" reasoning drifts out of memory if it isn't written down -- but
the docs/adr/ convention doesn't fit alongside the existing
docs/solutions/ structure (compound-engineering ce-compound pattern with
frontmatter metadata, additive entries, no membership-contract test).

This commit adopts hnshah's ADR 002 content (search-quality eval is
manual by default) as a docs/solutions/architecture/ entry with the
canonical compound-style frontmatter (module, problem_type, applies_when,
related_components, tags). Drops the docs/adr/ directory pattern, the
README index, and the test_adr_docs.py contract test.

ADR 001 (multi-surface packaging) is intentionally not adopted here: it
referenced sync.sh as the deploy mechanism, but sync.sh was removed in
PR #405 in favor of `npx skills add . -g -y`. The multi-surface
packaging story is still real but has moved beyond what the original
ADR captured; a fresh "how we ship to multiple harnesses" entry would
make sense as a separate doc.

Co-authored-by: hnshah <hnshah@users.noreply.github.com>
2026-05-16 22:22:59 -07:00
Trevin Chow 27c90504c0 review: validate SSH host alias + rename LAST30DAYS_YT_SSH_HOST -> LAST30DAYS_YOUTUBE_SSH_HOST
Addresses two concerns surfaced during PR #376 review:

1. **SSH option-injection on the host value.** The original PR uses
   shlex.quote() on the remote command and added a `--` option terminator
   in front of the host, but neither one stops a hostile env var like
   `LAST30DAYS_YT_SSH_HOST=-oProxyCommand=...` from being read in the
   first place. Tighten `_ytdlp_ssh_host()` to validate the host against
   `^[a-zA-Z0-9._-]+$` (plain hostname/SSH-config-alias shape: letters,
   digits, dot, underscore, hyphen). Any value that doesn't match logs a
   warning to stderr and returns None, so the wrap function falls back to
   local execution. The `--` terminator stays as defense-in-depth for the
   case where a valid host happens to start with `-`, but the regex closes
   the door on the env var reaching ssh at all.

2. **Env var naming consistency.** Existing skill-internal config knobs
   spell out their domain: `LAST30DAYS_X_BACKEND`, `LAST30DAYS_X_MODEL`,
   `LAST30DAYS_PLANNER_MODEL`, `LAST30DAYS_RERANK_MODEL`, etc. The module
   is `youtube_yt.py`, the source key is `youtube`, the function family
   is `is_youtube_*()` — `YT` was the odd abbreviation out. Rename to
   `LAST30DAYS_YOUTUBE_SSH_HOST` so the variable matches the user mental
   model ("route YouTube fetches via residential IP") and the codebase's
   spelled-out convention.

Adds three new tests:
- test_host_alias_with_dash_prefix_is_rejected (validator rejects `-o...`)
- test_host_alias_with_shell_metacharacters_is_rejected (rejects spaces, ;, $, `, &)
- test_host_alias_validator_accepts_realistic_aliases (allows FQDNs, IPs, bare aliases)

The existing test_wrap_cmd_uses_option_terminator is rewritten to use a
valid host value (since an invalid one is now filtered upstream) and
continues to assert the `--` terminator placement as defense-in-depth.

44/44 youtube_yt tests pass (40 prior + 4 net new validator tests).
2026-05-16 22:19:35 -07:00
shoobee f4eb0af104 fix(youtube): address Greptile review feedback
Three changes from automated review on PR #376:

1. Add `--` option terminator before host in _wrap_ytdlp_cmd (P1 security)
   Prevents SSH option injection if LAST30DAYS_YT_SSH_HOST were ever set
   to a value starting with `-` (e.g. `-oProxyCommand=...`). Low
   exploitability since the env var is user-controlled config — but the
   fix is a single arg and turns a self-harm footgun into no footgun.

2. Hoist `import shlex` to module-level (P2 style)
   Pure stdlib import, no reason for the deferred form. Cleaner.

3. Cache _ytdlp_ssh_host() result in fetch_transcript (P2 style)
   Was being called 2-3x per video; the function is cheap (env lookup
   + strip) so this is purely about readability.

Adds test_wrap_cmd_uses_option_terminator covering the security fix
explicitly with a `-oFoo=bar` host value. Updates index assertions in
the two existing tests that check command shape (host is now at index
4, command string at 5, with `--` at 3).
2026-05-16 22:17:30 -07:00
shoobee 79b5d049ce feat(youtube): route yt-dlp through SSH host for residential IP egress
Adds LAST30DAYS_YT_SSH_HOST env var (or `~/.config/last30days/.env` key).
When set, yt-dlp YouTube search invocations are wrapped as
`ssh <host> "yt-dlp ..."` so they run on a residential-IP machine.

Motivation: when last30days runs on a datacenter VPS (Hetzner,
DigitalOcean, AWS, etc.), `ytsearch:` queries return 0 results because
YouTube's bot-wall fingerprints datacenter IP ranges before any cookie
check runs. Cookies alone don't fix this — the IP reputation is checked
first. Verified across yt-dlp stable 2026.03.17 and nightly builds.

The existing fallbacks (browser cookies, residential proxy services,
excluding YouTube) all have downsides: cookies expire, proxies cost
money, exclusion loses signal. Many users with a Mac mini, Pi, or
home server can host yt-dlp on their own residential IP — this just
needs an SSH alias and a one-line env var to wire it up.

Behaviour:
- Default (env var unset): identical to before, no shape change.
- Env var set: search command list is wrapped with `ssh -o BatchMode=yes
  <host> "<shell-quoted yt-dlp invocation>"`. is_ytdlp_installed()
  returns True without a local PATH check (the binary lives on the
  remote host).
- Transcript path: when SSH-routing is on, skips the yt-dlp transcript
  path (which writes a VTT file we couldn't easily read back over SSH)
  and uses the existing _fetch_transcript_direct HTTP fallback. The
  timedtext API isn't bot-walled, so this works fine on datacenter IPs.

Setup pitfall documented in the function docstring: on macOS hosts
with Homebrew, `eval "$(/opt/homebrew/bin/brew shellenv zsh)"` must
live in ~/.zshenv (not just ~/.zprofile) — non-login SSH shells don't
source .zprofile, so without this `ssh macmini "yt-dlp ..."` returns
"command not found" while interactive SSH works fine.

Tests: 10 new cases covering env var read, whitespace stripping,
empty-value handling, command wrapping passthrough/active modes,
shlex quoting, is_ytdlp_installed short-circuit, and end-to-end
search_youtube wrapping. Full test suite: 0 new failures (the 14
pre-existing failures in test_store, test_watchlist, test_setup_openclaw,
test_safari_cookies, test_version_consistency are unchanged on main).

Verified live: 0 results → 4 real hits for "claude code" search from a
Hetzner VPS routed through a Mac mini exit node on Tailscale.
2026-05-16 22:17:29 -07:00
Trevin Chow 0e2059661a Merge pull request #378 from j-sperling/chore/gemini-3.1-flash-lite-ga
chore: migrate to gemini-3.1-flash-lite GA model
2026-05-16 22:12:20 -07:00
Trevin Chow b1c5f8db82 Merge pull request #382 from lustrousgorilla/bugfix/reddit-gaierror-retry
fix(http): expand retry budget + exponential backoff on DNS resolution failure
2026-05-16 22:11:56 -07:00
Trevin Chow 89c5cb9d5d Merge pull request #416 from tmchow/worktree-inherited-discovering-pebble
fix(ci): run full pytest suite, repair 13 rotted tests
2026-05-16 22:11:34 -07:00
Trevin Chow 5d4f9ef2c5 fix(store): use UTC for all date arithmetic against SQLite columns
Greptile flagged that _cli_query's --since parsing uses datetime.now()
(local time) while first_seen is stored via SQLite's datetime('now') (UTC).
The same bug exists in three other call sites that compare against either
first_seen or run_date (both UTC):

- get_daily_cost: "today" defaults to local date, returns wrong day's cost
  near UTC midnight
- get_stats: "7 days ago" cutoff for runs_7d / successful_7d
- get_trending: "N days ago" cutoff for finding activity ranking
- _cli_query: "N days ago" cutoff for --since flag (Greptile's flag)

All four now use datetime.now(timezone.utc). Same root cause and same fix
as the test_get_new_findings_filters_by_date repair in the previous commit.
2026-05-16 21:40:25 -07:00
Jeffrey Sperling 01262f78c6 chore: import GEMINI_FLASH_LITE in evaluate_search_quality
Address Greptile review nit. DEFAULT_JUDGE_MODEL now reuses the
constant from lib/providers.py instead of duplicating the literal,
so a future identifier change only needs one edit.
2026-05-16 21:40:22 -07:00