Commit Graph
100 Commits
Author SHA1 Message Date
Trevin Chow 84a19cf44d Merge pull request #438 from iliaal/refactor/github-search-parse-split 2026-05-22 08:10:40 -07:00
Trevin Chow 861462689e Merge pull request #444 from Yong-yuan-X/fix/centralize-test-path-setup 2026-05-22 08:09:20 -07:00
Trevin Chow d53121f035 Merge pull request #420 from hnshah/ren/watchlist-delta 2026-05-17 10:35:41 -07:00
Trevin Chow daca71f89e chore(release): v3.3.0
Release / build-and-release (push) Has been cancelled
~75 PRs merged since v3.2.0 plus 7 community fixes salvaged via PR triage.

Highlights:
- Install everywhere: npx skills add is canonical for Claude Code, Codex,
  Cursor, Gemini CLI, Copilot, Windsurf, and 50+ Agent Skills hosts.
- New emit mode: --emit=html for shareable HTML briefs.
- New source: Digg (auto-enabled when digg-pp-cli on PATH).
- New env vars: EXCLUDE_SOURCES, LAST30DAYS_YOUTUBE_SSH_HOST.
- New credential source: macOS Keychain.
- Reliability sweep: Reddit (4xx + URL prefix + multi-key auth), xAI
  error surfacing, Windows compatibility, YouTube/HN unblock,
  HTTP retries, planner gating, render fixes.
- Multi-harness reframe: AGENTS.md becomes canonical, CLAUDE.md points
  at it. SKILL_ROOT → SKILL_DIR substitution.

Breaking:
- .codex-plugin/plugin.json removed. Codex installs via npx skills add.
2026-05-17 09:25:46 -07:00
Trevin ChowandJonathan Oppenheim 170b570cbc fix(reddit): re-raise HTTP 402 so fallback chain triggers
The ScrapeCreators 402 (payment required / credits exhausted) status
was being swallowed by the broad except Exception handlers in
_global_search, _subreddit_search, and fetch_post_comments, returning
[] instead of propagating. That caused users with exhausted credits
to silently get zero Reddit results instead of falling through to
the OpenAI / public Reddit JSON fallback chain in _search_reddit_thread.
Add 402 to the existing 401/403 re-raise list across all three
ScrapeCreators call paths. Closes #170.

Co-authored-by: Jonathan Oppenheim <no-reply@postquantum.space>
2026-05-17 09:20:37 -07:00
Trevin ChowandFranco Carballar 4bae05e7fa fix(reddit): use browser-like headers to fix HTTP 403 from urllib
Reddit's public JSON endpoint returns 403 to requests carrying the
generic User-Agent and minimal header set urllib defaults to, while
matching curl requests succeed. Switch to a current-Chrome User-Agent
and add Accept-Language / Accept-Encoding / Connection headers so the
fingerprint matches a normal browser. Reddit now serves gzip when
Accept-Encoding includes it, so decompress the body before JSON parse.
Update the user-agent assertion in tests/test_reddit_public.py to match
the new browser-like string. Closes #199.

Co-authored-by: Franco Carballar <francocarballar@gmail.com>
2026-05-17 09:20:37 -07:00
Trevin ChowandEric Oberhofer a4f1f94802 fix(env): restore multi-key rotation for SCRAPECREATORS_API_KEY
Originally added in #268 to spread load across free-tier accounts when
SCRAPECREATORS_API_KEY is set to a comma-separated list. The 7-line block
was inadvertently dropped during the v3.0.6 consolidation (d14814a) even
though the changelog still advertised the feature. Re-apply the same
random.choice rotation in get_config() so user-facing behavior matches
the documented contract. Closes #287.

Co-authored-by: Eric Oberhofer <eric@oberhofer.io>
2026-05-17 09:20:37 -07:00
Trevin Chowandgujishh 16ce073d0c fix(cli): keep child cleanup working on Windows
_cleanup_children() called os.killpg unconditionally — Windows doesn't
have killpg as an attribute on os, so the call raised AttributeError
(not caught by the existing OSError-family handler) and aborted cleanup.
Guard with hasattr(os, "killpg") and fall back to os.kill(pid, SIGTERM)
on platforms without process-group APIs. Closes #226. Refs #110.

Co-authored-by: gujishh <baiaoshh@163.com>
2026-05-17 09:20:02 -07:00
Trevin ChowandAlex Key 5994b4f76a fix(reddit): use removeprefix("r/") for subreddit names, not lstrip("r/")
str.lstrip("r/") treats its argument as a character set, stripping
leading r and / repeatedly. Subreddits starting with 'r' (e.g. r/robotics,
r/ruby) were silently mangled to 'obotics' / 'uby'. Replace with
str.removeprefix("r/") at all four call sites. Python 3.9+ pattern is
safe here — project requires 3.12. Closes #288.

Co-authored-by: Alex Key <alexanderkey0508@gmail.com>
2026-05-17 09:20:02 -07:00
Trevin ChowandDave Morin bb5e6efbf9 fix(scripts): replace hardcoded developer paths in test-v1-vs-v2.sh
REPO_DIR now derives from the script's location (with env-var override)
and the Claude binary is looked up via PATH (with CLAUDE env-var override)
instead of hardcoded to /Users/mvanhorn/.local/bin/claude. Works on any
checkout. Closes #297.

Co-authored-by: Dave Morin <dave@morin.com>
2026-05-17 09:20:02 -07:00
Trevin Chow 76b8df40d3 Merge pull request #318 from flyingnobita/fix/gemini-claude-polyglot-hooks
fix: make hooks.json polyglot for Gemini CLI and Claude Code compatibility
2026-05-17 01:05:30 -07:00
Trevin Chow eb2d8b55e0 Merge pull request #344 from dzivkovi/feat/config-enablement
feat: configuration enablement — env-var defaults + source resilience
2026-05-17 01:05:08 -07:00
Trevin Chow 1a8ffd4847 fix(quality_nudge): also guard Instagram silent-failure on INCLUDE_SOURCES allowlist 2026-05-17 01:03:55 -07:00
Trevin Chow 1814bb1967 fix(quality_nudge,bluesky): gate Instagram nudge on EXCLUDE_SOURCES + anchor bluesky tests at resolver 2026-05-17 01:03:55 -07:00
Trevin Chow f236cff86a chore(pr-344): adapt to rebased base — fix test imports + memory-dir doc style
Rebased onto current main where:
- instagram.py uses unified http.get (not _requests fallback); tests now
  mock http.get and assert params/timeout kwargs.
- quality_nudge tests use lib.* import path with sys.path setup.
- README/CONFIGURATION.md memory-dir lines say "defaults to" so they pass
  test_no_stray_hardcoded_memory_dir_paths.
2026-05-17 01:03:55 -07:00
Trevin Chow d9a0ac31f2 Merge pull request #323 from GAOJIAN-0106/fix/openrouter-default-model-id
fix: correct invalid OPENROUTER_DEFAULT model ID
2026-05-17 01:01:54 -07:00
Trevin Chow 3e60c0817d Merge pull request #339 from dzivkovi/docs/configuration-md
docs: add CONFIGURATION.md + README pointers
2026-05-17 01:01:37 -07:00
Trevin Chow d530c90239 fix(hooks): remove timeout field per PR description intent 2026-05-17 00:59:41 -07:00
Trevin Chow aef6f35460 docs(providers): note why OpenRouter slug includes -preview suffix 2026-05-17 00:59:31 -07:00
Trevin Chow 71b1e8a411 Merge pull request #302 from nidhi-singh02/fix/github-repo-canonicalization
fix: Canonicalize ambiguous GitHub repo resolution for product comparisons
2026-05-17 00:57:05 -07:00
Trevin Chow 618458eb7e Merge pull request #320 from kaushikgopal/feat/brave-cookie-extraction
Add Brave browser support for X/Twitter cookie extraction
2026-05-17 00:56:44 -07:00
Trevin Chow 8cccd3e982 Merge pull request #334 from iamitp/codex/last-run-config-state
Preserve clean mode for last run state
2026-05-17 00:56:28 -07:00
Trevin Chow 17fb17222b fix(docs): use 'defaults to' wording for LAST30DAYS_MEMORY_DIR refs
The version_consistency test (test_no_stray_hardcoded_memory_dir_paths)
flagged 4 lines where `~/Documents/Last30Days/` appeared without the
canonical "defaults to" phrasing or the ${LAST30DAYS_MEMORY_DIR:-...}
literal form. Tightened the wording in the CONFIGURATION.md table +
footer paragraph, and the README "Where research files are saved"
section, so each path mention is anchored at the env-var override
contract rather than as a bare hardcoded default.
2026-05-17 00:54:49 -07:00
Trevin Chow 8ccd778366 fix(canonicalization): predicate-based call lookup + skip double-canon on auto-resolve
Two findings from Greptile review on PR #302:

1. tests/test_cli_v3.py:302 - The test asserted run_mock.call_args_list[0]
   was the main runner's invocation, but fanout.run_competitor_fanout
   submits main + competitors to a ThreadPoolExecutor and iterates with
   as_completed. With zero-latency mocks, thread scheduling determines
   which pipeline.run call lands first, so the competitor's call could
   take index [0] and flake CI. Replace [0] indexing with a predicate
   match on the canonicalized github_repos kwargs.

2. skills/last30days/scripts/last30days.py:662 - When auto_resolve returns
   github_repos, it has already run canonicalize_github_repos(cap=5) and
   ranked by relevance. The downstream block then re-canonicalized with
   cap=None, which can re-sort by topic-slug match and clobber the
   auto_resolve relevance order. Guard the second canonicalization with
   a repos_from_auto_resolve flag so it only fires for user-supplied
   --github-repo input.
2026-05-17 00:51:21 -07:00
Trevin Chow a3f173dc8a docs(readme): drop community-video link from "Going deeper" callout
The 6-min architecture walkthrough video is the contributor's own
work — not affiliated with the project. Removing the README pointer
to avoid implying endorsement. CONFIGURATION.md prose stands.
2026-05-17 00:51:14 -07:00
Trevin Chow 5a3ac8ca37 docs(config): document briefing.py show [--date DATE] subcommand 2026-05-17 00:50:36 -07:00
Trevin Chow e8eb15102f docs(agents): steer agents on maintaining CONFIGURATION.md 2026-05-17 00:50:36 -07:00
Trevin Chow 0f03a67166 Merge pull request #343 from Bortlesboat/codex/use-sandboxed-safari-cookie-path
fix: prefer sandboxed Safari cookie path
2026-05-17 00:47:32 -07:00
Trevin Chow 5a625fda9f Merge pull request #355 from dinakars777/test/cover-parallel-grounding-backend
test: cover parallel grounding backend
2026-05-17 00:47:20 -07:00
Trevin Chow b296a65515 fix(last-run): guard python3 absence + hoist datetime + use context manager 2026-05-17 00:44:30 -07:00
Trevin Chow f2737fc035 test(grounding): fix published_date → publish_date mock key mismatch 2026-05-17 00:42:07 -07:00
Trevin Chow 9ce7264d43 test(grounding): add serper>parallel priority + parallel empty-results coverage 2026-05-17 00:41:19 -07:00
Trevin Chow 8f565ee241 fix(chrome_cookies): sort Brave profiles by mtime, not alphabetically 2026-05-17 00:40:10 -07:00
Trevin Chow 5ab8c3ba76 Merge pull request #345 from dinakars777/docs/fix-stale-script-paths
docs: fix stale script paths
2026-05-17 00:38:17 -07:00
Trevin Chow 38bfb504e1 Merge pull request #349 from dinakars777/chore/sync-gemini-extension-version
chore: sync gemini extension version
2026-05-17 00:38:01 -07:00
Trevin Chow e8f23b4205 Merge pull request #340 from dzivkovi/fix/youtube-transcript-observability
fix(youtube): surface transcript-fetch ratio + add degraded nudge for stale yt-dlp
2026-05-17 00:31:37 -07:00
Trevin Chow b78ce34922 test(safari_cookies): add coverage for legacy fallback path 2026-05-17 00:31:09 -07:00
Trevin Chow 16a4fa9c39 Merge pull request #341 from flyingice/main
fix(grounding): align Parallel AI search with current API schema
2026-05-17 00:27:50 -07:00
Trevin Chow 3b75ff1537 Merge pull request #354 from dinakars777/fix/parallel-web-backend-source
fix: route parallel web backend through grounding
2026-05-17 00:26:50 -07:00
Trevin Chow 321975e144 Merge pull request #356 from dinakars777/fix/allow-threads-pinterest-search
fix: honor explicit optional source requests
2026-05-17 00:26:25 -07:00
Trevin Chow 5f9c637bbe Merge pull request #336 from davemorin/fix/319-xquik-source-capabilities
fix(planner): register xquik in SOURCE_CAPABILITIES (#319)
2026-05-17 00:26:08 -07:00
Trevin Chow 19132b0b5e Merge pull request #338 from dzivkovi/fix/windows-save-path-footer
fix(render): use forward slashes in save-path footer for Windows
2026-05-17 00:25:45 -07:00
Trevin Chow 1e4150ad78 Merge pull request #347 from dinakars777/docs/update-how-search-key-files
docs: update search key file paths
2026-05-17 00:24:55 -07:00
Trevin Chowandgreptile-apps[bot] 6a15afd8e8 Update skills/last30days/scripts/last30days.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-05-17 00:23:53 -07:00
Trevin Chow 0ab7051bc5 fix(planner): also register xquik in QUICK_SOURCE_PRIORITY 2026-05-17 00:22:04 -07:00
Trevin Chow ec0b126af6 test(pipeline): relax grounding assertion to stable source key 2026-05-17 00:21:57 -07:00
Trevin Chow d9e8a046ef docs(how-search): swap score.py->relevance.py at line 145 2026-05-17 00:21:36 -07:00
Trevin Chow 261ea5895c refactor(pipeline): remove dead threads-explicit-request branch 2026-05-17 00:20:56 -07:00
Trevin Chow 2692e0f4a2 Merge pull request #351 from dinakars777/docs/fix-changelog-skill-link-note
docs: correct changelog skill link note
2026-05-17 00:20:46 -07:00
Trevin Chow 10f35f82fe fix(grounding): guard parallel excerpts against None + cap at 500 chars 2026-05-17 00:20:27 -07:00
Trevin Chow 6b40d2c46f Merge pull request #346 from dinakars777/docs/fix-bug-report-repro-command
docs: fix bug report repro command
2026-05-17 00:20:21 -07:00
Trevin Chow 4a99c4f557 Merge pull request #337 from UncleMike1988/fix/path-quoting-spaces
Fix path-quoting in SessionStart check-config hook (handles spaces in…
2026-05-17 00:20:09 -07:00
Trevin Chow 0e353ae03f fix(render): apply as_posix to fallback branch + hoist shutil import 2026-05-17 00:19:49 -07:00
Trevin Chow 7214dd6051 Merge pull request #348 from dinakars777/docs/fix-readme-skill-link
docs: fix runtime skill spec link
2026-05-17 00:04:30 -07:00
Trevin Chow 6acf2fdbe2 Merge pull request #419 from mvanhorn/chore/remove-orphaned-spec-tasks
chore: remove orphaned SPEC.md and TASKS.md
2026-05-17 00:03:06 -07:00
Trevin Chow 07a3bdb3cf Merge pull request #364 from davemorin/fix/361-unsafe-eval-check-config
fix(hooks): replace unsafe eval with declare in check-config.sh
2026-05-17 00:02:44 -07:00
Trevin Chow f3df47c381 chore: remove orphaned SPEC.md and TASKS.md
Both files lived at the repo root as pre-plugin-layout artifacts. On
current main neither is referenced from README, SKILL.md, AGENTS.md,
CHANGELOG, or docs/ — no inbound links to break by removing. Git history
preserves the content for anyone who needs to dig it up.

Closes #352, #353. The PRs by @dinakars777 correctly flagged the drift;
deletion is the cleaner resolution than annotating them as historical.
2026-05-17 00:00:47 -07:00
Trevin Chow 46cf2328aa fix(hooks): use printf -v for bash 3.2 compat (declare -g is 4.2+)
macOS ships /bin/bash 3.2 and the script uses #!/bin/bash with
set -euo pipefail, so declare -g would abort the SessionStart hook
with "invalid option" on every Mac. printf -v writes via assignment
semantics (global from inside a function on 3.2+) — same scope
outcome, broader compatibility.
2026-05-16 23:49:44 -07:00
Trevin Chow 7506cbd542 fix(hooks): scope ENV_* to global (declare -g) so caller sees values 2026-05-16 23:49:05 -07:00
Trevin Chow aba6172032 Merge pull request #366 from davemorin/feat/324-reddit-json-fallback
feat(web): auto-enrich Reddit URLs from web search via JSON API
2026-05-16 23:41:57 -07:00
Trevin Chow d07e4698e3 Merge pull request #358 from dinakars777/fix/openclaw-poll-clock-init
fix: initialize OpenClaw poll timing once
2026-05-16 23:41:42 -07:00
Trevin Chow 4c0282dd55 Merge pull request #365 from davemorin/fix/284-version-metadata-drift
fix(version): replace hardcoded v3.0.0 with dynamic _skill_version()
2026-05-16 23:40:53 -07:00
Trevin Chow 99909fca67 Merge pull request #368 from hnshah/ren/advisory-security-workflow-252
ci: add advisory security workflow
2026-05-16 23:40:28 -07:00
Trevin Chow 36c43d50b7 test: drop side_effect padding to match collapsed time.time() call 2026-05-16 23:39:14 -07:00
Trevin Chow e2d9d705f6 review: fix selftext key path + break on RedditRateLimitError 2026-05-16 23:37:00 -07:00
Trevin Chow 9f39d10bc5 Merge pull request #373 from hnshah/ren/watchlist-sightings
feat(store): record per-run finding sightings
2026-05-16 23:24:09 -07:00
Trevin Chow 03043da407 Merge pull request #418 from tmchow/chore/greptile-config
chore: add greptile.json (triggerOnUpdates + statusCheck)
2026-05-16 23:23:48 -07:00
Trevin Chow 8bab997854 chore: add greptile.json to opt into update-triggered reviews + status check
Without this config, Greptile's documented default is `triggerOnUpdates: false`
(only the initial PR open triggers a review). Empirically Greptile has been
re-reviewing on force-push to this repo anyway, but documenting the intent
makes the behavior reliable across plan changes and any future config-source
shifts on Greptile's side.

`statusCheck: true` registers Greptile as a GitHub status check (not just a
PR comment). That gives maintainer-tooling a machine-readable heartbeat -
poll `GET /repos/.../commits/SHA/check-runs` and filter by app name to see
whether Greptile is `queued` / `in_progress` / `completed`. Without it the
only signal is "did a new Greptile comment appear" which is silently
ambiguous when Greptile re-reviews and finds nothing new.

If `statusCheck` is OSS-plan-restricted Greptile silently ignores the key,
which is fine - the rolling-summary comment with `Confidence Score: N/5`
remains the fallback signal.

Refs greptileai/skills `greploop` skill for the terminal-state pattern this
config enables.
2026-05-16 23:20:24 -07:00
Trevin Chow 791c0a57a0 review: gate web Reddit enrichment behind EXCLUDE_SOURCES
PR #366 routes Reddit URLs found in web-search results through the public
Reddit JSON API to recover thread body + top comments (the Claude Code
WebFetch tool blocks reddit.com directly). That bypass is sound and the
fixed problem is real - but the always-on shape ignores user intent on
source gating.

A user who sets EXCLUDE_SOURCES=reddit to suppress Reddit results would
still get Reddit content smuggled back in via web-search URLs that
happen to point at reddit.com threads. This contradicts the suppression
contract that EXCLUDE_SOURCES is supposed to provide (see
lib/pipeline.available_sources where the same env var gates the
top-level Reddit source).

Add a _reddit_excluded(config) check in web_search() that mirrors the
parsing pattern from lib/pipeline (comma-separated, case-insensitive,
whitespace-tolerant). When reddit is in EXCLUDE_SOURCES, skip the
enrichment pass entirely - the web results themselves still flow
through, but they're not augmented with Reddit body/comments.

Four new tests in test_grounding_v3.py cover:
- EXCLUDE_SOURCES=reddit skips enrichment
- case-insensitive parsing matches REDDIT/Reddit/whitespace-padded/csv
- Other sources in EXCLUDE_SOURCES don't trigger the gate
- Enrichment runs normally when reddit isn't excluded

19/19 grounding tests pass.
2026-05-16 22:52:06 -07:00
Trevin Chow d7b3995da1 Merge pull request #357 from dinakars777/fix/windows-env-permission-warning
fix: skip POSIX secret warning on Windows
2026-05-16 22:42:26 -07:00
Trevin Chow e217db77cc Merge pull request #369 from voidborne-d/fix/scrapecreators-100-credits
docs: correct ScrapeCreators free tier to 100 credits (closes #367)
2026-05-16 22:41:16 -07:00
Trevin Chow 2c2cfb9e7e Merge pull request #417 from tmchow/docs/eval-not-in-ci-solution
docs: capture eval-not-in-CI design decision under docs/solutions/
2026-05-16 22:31:39 -07:00
Trevin Chow 3276496f49 Merge pull request #376 from shoobee/feat/yt-dlp-ssh-routing
feat(youtube): route yt-dlp through SSH host for residential IP egress
2026-05-16 22:31:18 -07:00
Trevin Chowandhnshah 68ae74ff4f docs: capture eval-not-in-CI design decision under docs/solutions/
Closes #374 (adapted, not 1:1 merged).

@hnshah opened PR #374 proposing a docs/adr/ directory for architecture
decision records. The intent is right -- the "why is search-quality eval
manual?" reasoning drifts out of memory if it isn't written down -- but
the docs/adr/ convention doesn't fit alongside the existing
docs/solutions/ structure (compound-engineering ce-compound pattern with
frontmatter metadata, additive entries, no membership-contract test).

This commit adopts hnshah's ADR 002 content (search-quality eval is
manual by default) as a docs/solutions/architecture/ entry with the
canonical compound-style frontmatter (module, problem_type, applies_when,
related_components, tags). Drops the docs/adr/ directory pattern, the
README index, and the test_adr_docs.py contract test.

ADR 001 (multi-surface packaging) is intentionally not adopted here: it
referenced sync.sh as the deploy mechanism, but sync.sh was removed in
PR #405 in favor of `npx skills add . -g -y`. The multi-surface
packaging story is still real but has moved beyond what the original
ADR captured; a fresh "how we ship to multiple harnesses" entry would
make sense as a separate doc.

Co-authored-by: hnshah <hnshah@users.noreply.github.com>
2026-05-16 22:22:59 -07:00
Trevin Chow 27c90504c0 review: validate SSH host alias + rename LAST30DAYS_YT_SSH_HOST -> LAST30DAYS_YOUTUBE_SSH_HOST
Addresses two concerns surfaced during PR #376 review:

1. **SSH option-injection on the host value.** The original PR uses
   shlex.quote() on the remote command and added a `--` option terminator
   in front of the host, but neither one stops a hostile env var like
   `LAST30DAYS_YT_SSH_HOST=-oProxyCommand=...` from being read in the
   first place. Tighten `_ytdlp_ssh_host()` to validate the host against
   `^[a-zA-Z0-9._-]+$` (plain hostname/SSH-config-alias shape: letters,
   digits, dot, underscore, hyphen). Any value that doesn't match logs a
   warning to stderr and returns None, so the wrap function falls back to
   local execution. The `--` terminator stays as defense-in-depth for the
   case where a valid host happens to start with `-`, but the regex closes
   the door on the env var reaching ssh at all.

2. **Env var naming consistency.** Existing skill-internal config knobs
   spell out their domain: `LAST30DAYS_X_BACKEND`, `LAST30DAYS_X_MODEL`,
   `LAST30DAYS_PLANNER_MODEL`, `LAST30DAYS_RERANK_MODEL`, etc. The module
   is `youtube_yt.py`, the source key is `youtube`, the function family
   is `is_youtube_*()` — `YT` was the odd abbreviation out. Rename to
   `LAST30DAYS_YOUTUBE_SSH_HOST` so the variable matches the user mental
   model ("route YouTube fetches via residential IP") and the codebase's
   spelled-out convention.

Adds three new tests:
- test_host_alias_with_dash_prefix_is_rejected (validator rejects `-o...`)
- test_host_alias_with_shell_metacharacters_is_rejected (rejects spaces, ;, $, `, &)
- test_host_alias_validator_accepts_realistic_aliases (allows FQDNs, IPs, bare aliases)

The existing test_wrap_cmd_uses_option_terminator is rewritten to use a
valid host value (since an invalid one is now filtered upstream) and
continues to assert the `--` terminator placement as defense-in-depth.

44/44 youtube_yt tests pass (40 prior + 4 net new validator tests).
2026-05-16 22:19:35 -07:00
Trevin Chow 0e2059661a Merge pull request #378 from j-sperling/chore/gemini-3.1-flash-lite-ga
chore: migrate to gemini-3.1-flash-lite GA model
2026-05-16 22:12:20 -07:00
Trevin Chow b1c5f8db82 Merge pull request #382 from lustrousgorilla/bugfix/reddit-gaierror-retry
fix(http): expand retry budget + exponential backoff on DNS resolution failure
2026-05-16 22:11:56 -07:00
Trevin Chow 89c5cb9d5d Merge pull request #416 from tmchow/worktree-inherited-discovering-pebble
fix(ci): run full pytest suite, repair 13 rotted tests
2026-05-16 22:11:34 -07:00
Trevin Chow 5d4f9ef2c5 fix(store): use UTC for all date arithmetic against SQLite columns
Greptile flagged that _cli_query's --since parsing uses datetime.now()
(local time) while first_seen is stored via SQLite's datetime('now') (UTC).
The same bug exists in three other call sites that compare against either
first_seen or run_date (both UTC):

- get_daily_cost: "today" defaults to local date, returns wrong day's cost
  near UTC midnight
- get_stats: "7 days ago" cutoff for runs_7d / successful_7d
- get_trending: "N days ago" cutoff for finding activity ranking
- _cli_query: "N days ago" cutoff for --since flag (Greptile's flag)

All four now use datetime.now(timezone.utc). Same root cause and same fix
as the test_get_new_findings_filters_by_date repair in the previous commit.
2026-05-16 21:40:25 -07:00
Trevin Chow eb2d7a0f37 fix(ci): run full pytest suite, repair 13 rotted tests
CI was running only test_plugin_contract.py and test_version_consistency.py
(2 of 84 test files), masking 13 rotted tests across 4 clusters. The suite is
fully offline-safe (1402 tests in ~7s without network), so the narrow scope
wasn't gating integration flakiness; it was just stale. validate.yml now runs
`uv run pytest` against the full suite.

Engine fix: store.findings_from_report is rerank-first. ranked_candidates is
the primary persistence path; hackernews/polymarket are unconditionally
supplemented from items_by_source because they rank poorly but matter for
watchlists. When ranked_candidates was empty (rerank failed or skipped),
reddit, x, and every other source were silently dropped. The supplement loop
now falls back to all sources only when ranked_candidates is empty; the normal
path is unchanged.

Test repairs:
- test_store.py (6) + test_watchlist_commands.py (2): cascade from the engine fix
- test_get_new_findings_filters_by_date (latent): local-time vs SQLite UTC
  flake — switched to datetime.now(timezone.utc)
- TestPollDeviceAuth (3): mock_time.time side_effect lists too short after
  impl added a last_reminder call — padded timeout test, pinned others to
  return_value=0 (loops terminate via urlopen, not the clock)
- test_bare_run_emits_web_promo: engine reads ~/.config/last30days/.env, so
  a contributor's saved EXA/PARALLEL key made grounding "available" and
  suppressed the web promo. Also missing X made the "x" promo preempt "web".
  Set LAST30DAYS_CONFIG_DIR="", subprocess cwd=tmpdir, XAI_API_KEY stub.
2026-05-16 21:34:22 -07:00
Trevin Chow 719cdef2fb fix(http): contain DNS retry-budget widening to DNS path only
PR #382 introduced an `effective_retries` widening on the first gaierror,
but the widening leaked: every non-DNS error path (HTTPError, non-DNS
URLError, OSError) was gated on `effective_retries - 1` and so inherited
the expanded bound. A caller passing `retries=2` who hit DNS-then-non-DNS
got 3 attempts instead of 2 — contrary to the PR description and the
fail-fast intent of small retry budgets.

Fix:
- Gate every non-DNS sleep/retry decision on the caller's original
  `retries`, not the widened `effective_retries`.
- Add an explicit `break` in each non-DNS branch when the original
  budget is exhausted, so the (possibly widened) outer loop bound
  can't pull us into an extra attempt.

Adds two regression tests covering the DNS-then-non-DNS-URLError and
DNS-then-OSError sequences flagged in Greptile review on PR #382.
2026-05-16 21:13:17 -07:00
Trevin Chow ac04b56acc Merge pull request #383 from lustrousgorilla/bugfix/bird-x-json-decode-retry
fix(bird_x): retry subprocess on non-JSON stdout (HTML anti-bot interstitial)
2026-05-16 21:08:35 -07:00
Trevin Chow 42bfc6c76c Merge pull request #415 from tmchow/fix/sc-source-gating-consistency
fix(sources): align SC source gating between code and docs
2026-05-16 21:08:10 -07:00
Trevin Chow 9f08bb68b5 fix(sources): align SC source gating between code and docs
Two related drifts surfaced while reviewing PR #399 (EXCLUDE_SOURCES) —
docs claimed several SC-backed sources required INCLUDE_SOURCES opt-in
that the code didn't actually enforce, and threads was inconsistently
gated relative to its same-key siblings.

This commit picks the "code as source of truth + EXCLUDE_SOURCES as
suppression knob" model and aligns docs to match. It also promotes
threads to the same auto-on tier as tiktok and instagram, since all
three share the SC key and per-call cost shape — there was no real
product reason for threads being opt-in while the other two weren't.

The resulting source-gating model is three-tier and intentional:

  • **Auto-on if backing infra present** (suppress via EXCLUDE_SOURCES):
    reddit, HN, polymarket, X, youtube, github, bluesky, truthsocial,
    grounding, **tiktok, instagram, threads**

  • **INCLUDE_SOURCES persistent opt-in** (cost/billing reasons):
    perplexity (different paid API — OpenRouter),
    tiktok_comments / youtube_comments (N× extra SC calls per video)

  • **--search per-query opt-in** (relevance reasons):
    pinterest (visual pins, narrow utility),
    xiaohongshu (Chinese-market specific)

Changes:

- env.py: `is_threads_available()` drops the INCLUDE_SOURCES check,
  now mirrors tiktok/instagram (SC key → True). Docstring updated.
- tests/test_env_v3.py: new `ThreadsAvailabilityTests` class locks in
  the new contract and includes a regression guard ("INCLUDE_SOURCES
  should not be needed").
- SKILL.md: lines 333-338 rewritten so the model's "Build
  ACTIVE_SOURCES_LIST" checklist reflects what the engine actually
  runs. Drops false INCLUDE_SOURCES requirement for
  tiktok/instagram/threads; corrects pinterest to mention --search;
  adds missing INCLUDE_SOURCES=perplexity requirement.
- README: same alignment for the user-facing "Everything else in v3"
  section.

Note on EXCLUDE_SOURCES references in the new docs: the suppression
flag is wired up in PR #399. SKILL.md and README mention EXCLUDE_SOURCES
as the opt-out path; that prose is forward-looking until #399 lands.
The behavior changes in this PR (threads auto-on) are self-contained
and don't require #399 to function — but for users who want to suppress
the newly-auto-on threads source, #399 needs to land first.
2026-05-16 20:17:31 -07:00
Trevin Chow 602de1ebda Merge pull request #388 from bradferguson/fix/sc-youtube-and-hn-tokenization
fix(sources): unblock SC YouTube + multi-token HN searches
2026-05-16 19:41:12 -07:00
Trevin Chow bf3a82a87e Merge pull request #389 from kuyua9/fix/save-comparison-html-kuyua9
fix: save comparison HTML artifacts
2026-05-16 19:40:59 -07:00
Trevin Chow c010feb8f8 Merge pull request #399 from spiky02plateau/feat/exclude-sources-banner-and-pipeline
feat: honor EXCLUDE_SOURCES env var in source count + pipeline filter
2026-05-16 19:39:51 -07:00
Trevin Chow 4d4ac97ffb refactor: hoist comparison-html gate into a single condition (Greptile DRY)
Greptile flagged that `entity_reports and args.emit == "html"` appeared in
two places — once when computing the footer display path, again when calling
save_output. The else-branches differ between the two callsites (the display
needs `report.topic` as fallback; the save call needs `None` so save_output
falls back to the report's own topic), so collapsing into one shared
expression would be wrong, but hoisting just the condition into a single
`is_comparison_html` bool eliminates the risk of drift while keeping the
two callsites' fallback semantics distinct.
2026-05-16 19:32:49 -07:00
Trevin Chow 85255be350 Merge pull request #414 from mvanhorn/dependabot/uv/pytest-9.0.3
chore(deps-dev): bump pytest from 9.0.2 to 9.0.3
2026-05-16 19:31:46 -07:00
Trevin Chow 1aa120a420 Merge pull request #407 from DamienStevens/feat/macos-keychain-source
feat(env): macOS Keychain credential source
2026-05-16 19:31:34 -07:00
Trevin Chow 306d8c2d73 fix(env): wire EXCLUDE_SOURCES through get_config + SKILL.md integration
The original PR added EXCLUDE_SOURCES filtering to pipeline.available_sources()
and to the check-config.sh banner, but env.py::get_config() builds its config
dict from a hardcoded keys list that didn't include EXCLUDE_SOURCES. The
result: setting EXCLUDE_SOURCES in the environment silently no-op'd through
the Python pipeline. Only the bash hook (which reads shell env directly)
worked. The PR's unit tests didn't catch this because they construct config
dicts directly, bypassing get_config().

Changes:
- Add ('EXCLUDE_SOURCES', '') to env.py's keys list so the env var actually
  propagates into config.
- Add an end-to-end regression test that goes through get_config() rather
  than constructing config dicts directly.
- Document EXCLUDE_SOURCES in SKILL.md's source-list checklist so the model
  invoking the skill knows to subtract excluded sources before displaying
  the active-sources line. (Per AGENTS.md: engine flags without SKILL.md
  prose are incomplete — the agent invoking the skill won't know the flag
  exists.)
2026-05-16 19:30:27 -07:00
Trevin Chow d0dcf751f1 fix(keychain): single source of truth for key list + robust USER fallback
Addresses Greptile review on PR #407:

- P1: setup-keychain.sh ALL_KEYS was missing GOOGLE_GENAI_API_KEY and
  XIAOHONGSHU_API_BASE relative to _load_keychain's inline list, so users
  manually storing those keys would not see them in --list and the
  interactive prompt would never offer to set them.

  Hoist the canonical key list into lib/env.py::KEYCHAIN_KEYS, have
  get_config() pass it through, and add a parity test that parses
  ALL_KEYS out of setup-keychain.sh and asserts equality. Drift is now
  caught at CI time instead of after a user reports a missing key.

- P2: os.environ.get("USER", "") silently returned "" under sudo, in
  Docker without --env USER, or in CI runners that strip USER. The
  resulting `security find-generic-password -a ""` call would never
  match items stored by setup-keychain.sh, so all lookups silently
  returned nothing. Fall back to pwd.getpwuid(os.getuid()).pw_name when
  USER is absent.

The P2 process-listing comment ("secret visible briefly via ps because
security has no stdin path for -w") has no clean fix — the README
already documents the manual `security add-generic-password` invocation
as an alternative for users with strict secret hygiene.
2026-05-16 19:25:05 -07:00
Trevin Chow 14d8f62e02 Merge pull request #413 from tmchow/docs/compound-release-cascade-pattern
docs: compound learning on release-time consistency-test cascade failures
2026-05-16 19:12:31 -07:00
Trevin Chow 0fd532d249 docs: compound learning on release-time consistency-test cascade failures
Documents the cascade pattern surfaced during this session's install-modernization
arc: a `test_sync_cache_path_uses_skill_version` test asserted that a hardcoded
version pin in `sync.sh` matched the version frontmatter in SKILL.md. When a
release bumped SKILL.md, every open PR's CI failed simultaneously on the
unrelated stale-pin assertion. Affected at least 5 PRs across the 2026-05-13
to 2026-05-15 window (#400, #390, #392, and two others) plus required hotfix
PR #397 to unblock the queue.

The permanent fix shipped in PR #405 (deleted sync.sh + the test). This doc
captures the design lesson so the pattern doesn't reappear: don't write
consistency tests that read two files and assert one matches a value derived
from the other. Either derive at runtime from a single source of truth, or
self-skip / merge-base-scope the test so deletion is a non-event.

Created via /ce-compound. Includes:

- docs/solutions/workflow-issues/release-consistency-test-cascade-2026-05-16.md
  (the new learning — first entry under docs/solutions/)
- CONCEPTS.md (new — 4 entries: Skill, Engine, Harness, Beta channel,
  capturing project-specific vocabulary that surfaced across the session)
- AGENTS.md (added one-line Structure entries surfacing docs/solutions/ and
  CONCEPTS.md so fresh agents discover them)
- docs/plans/2026-04-22-{002,003,005,006}-*-plan.md (added deprecation banner
  to each, pointing readers at PR #405 and the new docs/solutions entry —
  these 4 historical plans still reference the deleted sync.sh inline)

Also: closed PR #379 (j-sperling's workaround for the same cascade,
superseded by PR #405's permanent fix).
2026-05-16 19:08:02 -07:00
Trevin Chow 8867a007ea Merge pull request #392 from Gujiassh/fix/openclaw-scrapecreators-optional-env
fix(openclaw): make ScrapeCreators key optional
2026-05-16 18:50:07 -07:00
Trevin Chow 01b5f3dc1e Merge pull request #363 from thinkun/pr/claim-contributor-entry
Claim contributor entry — @thinkun
2026-05-16 18:46:35 -07:00
Trevin Chow 2e39ee8ce4 Merge pull request #412 from tmchow/refactor/skill-meta-version-helper
refactor: consolidate SKILL.md version regex into lib/skill_meta.py
2026-05-16 18:42:49 -07:00
Trevin Chow 37033164da Merge pull request #410 from tmchow/docs/agents-orientation-multi-harness
docs: reframe as multi-harness Agent Skills package, flip CLAUDE.md ↔ AGENTS.md
2026-05-16 18:42:36 -07:00
Trevin Chowandgreptile-apps[bot] 9fe4b8f130 Update AGENTS.md
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-05-16 18:42:14 -07:00
Trevin Chow 73dc6b9996 refactor: consolidate SKILL.md version regex into lib/skill_meta.py
The same `^version:\s*"([^"]+)"\s*$` regex (or a slight variant) was
duplicated across three files: render.py inline, test_plugin_contract.py
local helper, test_version_consistency.py local helper. A future change
to the SKILL.md frontmatter version format would have needed to update
three places without any compile-time pressure to keep them in sync.

New skills/last30days/scripts/lib/skill_meta.py provides:
- `_VERSION_RE` private compiled pattern (accepts double-quoted,
  single-quoted, or unquoted YAML version scalars per the widening
  landed in 997708a)
- `read_skill_version(skill_md_path: Path) -> str | None` helper that
  catches OSError + UnicodeDecodeError and returns None on miss

Callers updated:
- render.py::_skill_version now calls skill_meta.read_skill_version
  inside the SKILL.md fallback loop, returning `read_skill_version(...) or "?"`.
  Semantically equivalent to the old break-after-first-SKILL.md logic.
- test_plugin_contract.py and test_version_consistency.py import the
  helper instead of defining the regex inline. Both files use the
  established sys.path.insert pattern.

Added tests/test_skill_meta.py with 6 direct unit tests covering the
helper's full contract: missing file, undecodable bytes, no-version-line,
and all three quoting styles (double, single, unquoted). Previously the
helper was only exercised transitively through render._skill_version().

Added test_skill_md_uses_double_quoted_version to
test_version_consistency.py — the old per-test regex incidentally
asserted "this repo's SKILL.md uses double-quotes" by being strict;
the shared helper accepts all three styles, so the assertion is now
explicit instead of implicit.

Code-reviewed by ce-code-review (8 reviewers); safe_auto fixes applied
inline (rename to _VERSION_RE, group or-chain instead of generator,
docstring tightened, dropped unnecessary `from __future__ import
annotations`, tightened signature to Path-only).

Conftest.py refactor for the sys.path.insert duplication across ~20 test
files filed as issue #411 — out of scope for this PR (touches many
files, separate concern).

Test results: 23 passed in the affected test set (16 prior + 6 new
test_skill_meta tests + 1 new double-quote assertion). Full suite shows
same 13 pre-existing failures as main; zero new failures.
2026-05-16 18:17:33 -07:00