Commit Graph

12 Commits

Author SHA1 Message Date
Ilia Alshanetsky 2acbf8a869 perf: batch store_findings, dedup source_items in O(1), remove dead code (#206)
1. N+1 queries in store.store_findings()
   The old loop ran one SELECT per finding to check existence, then one
   INSERT or UPDATE. 100 findings cost 200 serial SQLite roundtrips.
   Now: one batch SELECT with WHERE source_url IN (...) builds a lookup
   dict, then executemany() handles all inserts and updates. Query count
   stays constant regardless of batch size. Benchmark on 500 findings:
   ~30ms to ~20ms; gap widens on slower storage.

2. O(n^2) source_items dedup in fusion.weighted_rrf()
   Merging an item into an existing candidate ran any(existing.source ==
   ... for existing in candidate.source_items), linearly scanning a list
   that grew with each merge. At 40 candidates with 20 source_items each,
   fusion went quadratic. Now tracks (source, item_id) tuples in a
   per-candidate set for O(1) lookup. The source_items list itself is
   unchanged since other code iterates it.

3. Dead code removal
   - providers.GeminiClient.ground_search() and .url_context_json(): zero
     callers. Deleted.
   - render._top_comment_excerpt(): zero callers. Deleted.
   - env.is_reddit_available(): one-line wrapper around get_reddit_source.
     Callers can check get_reddit_source(config) is not None directly.
2026-04-25 14:17:17 -07:00
Ilia Alshanetsky e6b89f2644 perf: cache PreparedQuery per stream, skip double-normalize in dedupe (#282)
Scoring hot path (_normalize_score_dedupe) re-tokenized the same
ranking_query ~240x per stream: once per item for local_relevance,
plus ~5x per item across snippet windows. Query tokens are immutable
within a stream, so compute them once as relevance.PreparedQuery and
thread through signals.annotate_stream and snippet.extract_best_snippet.

dedupe._PreparedText called normalize_text twice: once in __init__ and
again via get_ngrams. Factor out _ngrams_of_normalized so the prepared
path skips the redundant pass while get_ngrams keeps its public contract.

Behavior unchanged.
2026-04-25 14:16:57 -07:00
Ilia Alshanetsky 2c2755b49c refactor(normalize): extract _join_comment_excerpts helper (#283)
_normalize_reddit, _normalize_hackernews, and _normalize_github inlined
the same 5-line comprehension to stringify and space-join the first 3
top_comments' excerpt field. Extract one helper, call it from all three.

The comment field name varies per source (Reddit/GitHub use 'excerpt',
HN uses 'text'), so it's passed as a parameter. Behavior unchanged.
2026-04-25 14:16:50 -07:00
Claire Novotny f03cb866aa fix: address plugin layout review feedback 2026-04-24 11:52:48 -04:00
Claire Novotny 72495c1c14 Restructure as Codex plugin 2026-04-23 20:15:02 -04:00
Matt Van Horn 361e9d6c13 fix: v3.0.8 - SKILL.md was too big and LAWs too deep - move to top + engine emits badge (#279)
Three independent Opus 4.7 self-debugs on 2026-04-18 converged on the same
root cause of the v3.0.6/v3.0.7 canonical-compliance regression: SKILL.md is
42,860 tokens / 1,478 lines, LAWs lived at line 1094+, every realistic reading
strategy failed to reach them before synthesis.

Unit 1 - Moved the BADGE MANDATORY block and VOICE CONTRACT LAW 1-5 (plus
the formatting-authority preface) from line ~1090 to line ~75 (right after
the SKILL CONTRACT preface, before HOW TO INVOKE THIS SKILL). Every reading
strategy now lands the LAWs in active context before synthesis.

Unit 2 - Engine now emits the badge as the first line of --emit=compact
stdout. Passing through the script output becomes the default-correct
behavior; emitting the badge no longer depends on model compliance. Reads
version from .claude-plugin/plugin.json at runtime with graceful fallback.

Unit 3 - Deleted skills/last30days/SKILL.md stub (231-line v3-spec file).
This was the wrong-file-capture hazard Ron Conway's self-debug identified:
model grabbed the first SKILL.md find surfaced and treated it as
authoritative. Only ONE SKILL.md in the plugin package now.

Diagnoses verbatim:
- Kanye thread: "I read lines 1-600 in chunks, jumped to 300-899, then
  stopped. File is 1478 lines. I never saw past ~900."
- Peter thread: "I tried Read once, hit the 25K token cap on a 42,860-token
  file, and bailed instead of chunked-reading with offset/limit. I never
  opened SKILL.md at all."
- Ron Conway thread: "I read one SKILL.md (231 lines)... the v3 spec stub.
  I never opened the operational SKILL.md sitting next to the script."

Validation: direct engine invocation confirms badge at line 1 of compact
output. Module imports clean.

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 12:50:10 -07:00
Matt Van Horn 8d8ca68781 chore: bump version to 3.0.1 + changelog entry
Atomic bump across all four manifests:
- SKILL.md (root)
- skills/last30days/SKILL.md (internal spec)
- .claude-plugin/plugin.json
- gemini-extension.json

CHANGELOG entry documents the skill-upload packaging fix, vendor/ removal,
legacy plans/ removal, and the new scripts/build-skill.sh builder.
2026-04-14 12:21:24 -04:00
Matt Van Horn 99b167d03a fix: resolve duplicate skill name causing marketplace validation failure (#204) (#214)
Two SKILL.md files declared `name: last30days` with `user-invocable: true`,
which caused strict marketplace validators to reject the plugin with "Some
plugins in this marketplace have validation errors":

- ./SKILL.md (canonical, also reachable via skills/last30days-nux/ symlink)
- ./skills/last30days/SKILL.md (v3 architecture spec, real file)

In v2.9.6, skills/last30days/SKILL.md was a symlink to ../../SKILL.md so
only one skill existed. Commit 0a9ff16 (v3.0.0) added a new real file at
skills/last30days-v3/SKILL.md, and commit 9be0780 then renamed that
directory to skills/last30days/, replacing the original symlink with a
different real file. The collision has been live since v3.0.0 shipped.

This change:
- Renames skills/last30days/SKILL.md to name: last30days-v3-spec and sets
  user-invocable: false. The file stays in place as internal architecture
  documentation, but it no longer competes with the canonical skill.
- Fixes README.md link that pointed to the deleted skills/last30days-v3/
  path (left over from the rename).
- Removes a stale variants/open/SKILL.md reference (variants/open was
  deleted in v3.0.0).

After the change, only one canonical name=last30days user-invocable=true
skill exists (the root SKILL.md, also reachable via the
skills/last30days-nux/ symlink, same inode).

Closes #204.

This contribution was developed with AI assistance (Codex).

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-04-11 02:36:55 -04:00
Pejman Pour-Moezzi bb7c956db5 fix: drop stale last30days-v3 argument hint 2026-04-09 21:37:36 -07:00
Pejman Pour-Moezzi 9be0780c46 fix: rename skills/last30days-v3 directory so plugin install resolves correctly 2026-04-09 21:37:00 -07:00
Matt Van Horn 0a9ff16dfc feat: v3.0.0 - intelligent search, GitHub person/project mode, ELI5, 13+ sources
v3 rewrites the search engine from the ground up:

- Intelligent pre-research: resolves X handles, GitHub repos, subreddits,
  TikTok hashtags, and YouTube channels before searching
- GitHub person-mode: PR velocity, top repos by stars, release notes
- GitHub project-mode: live star counts, README, releases, top issues
- ELI5 mode: plain language synthesis, no jargon
- 13+ sources: Reddit, X, YouTube, TikTok, Instagram, HN, Polymarket,
  GitHub, Threads, Pinterest, Perplexity, Bluesky, Web
- Free Reddit comments via public JSON (no API key needed)
- Fun judge v2: humor scoring baked into narrative
- Cookie consent before browser scanning
- 10,000 free ScrapeCreators calls
- 1,012 tests

Thank you to the community contributors whose issues and PRs shaped v3:
@uppinote20 (#143), @zerone0x (#134, #136), @thinkun (#116),
@thomasmktong (#124), @fanispoulinakisai-boop (#100), @pejmanjohn (#78),
@zl190 (#115), @hnshah (#84, #85, #86)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 10:52:23 -07:00
Matt Van Horn 2f16ff1ee8 feat(gemini): add Gemini CLI extension support
Add gemini-extension.json manifest with correct array-format settings,
symlink skills/last30days/SKILL.md to root SKILL.md for Gemini skill
discovery, add Gemini install paths to bash for-loop in both main and
open variant, and add Gemini CLI install instructions to README.

Incorporates the good parts of PR #53 (manifest, paths, README) while
avoiding duplicate SKILL.md, tool name scattering, and allowed-tools
pollution that would have created maintenance issues.

Closes #45

Co-Authored-By: Alex Ferrari <alex@thealexferrari.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 12:07:55 -07:00