docs: add perpetual monitoring mode plan (not building yet)

Explored adding scheduled re-runs and cumulative intelligence to last30days. Concluded that Claude Code's session-scoped scheduling (CronCreate/loop) can't support true perpetual monitoring since jobs die when the terminal closes. Plan documents the architecture, what exists, and why we're waiting for persistent background agents. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 09:37:18 -07:00
parent 28dff6e7b2
commit 4503da7920
1 changed files with 238 additions and 0 deletions
@@ -0,0 +1,238 @@
+---
+title: "feat: Perpetual Monitoring Mode"
+type: feat
+status: active
+date: 2026-03-08
+---
+
+# Perpetual Monitoring Mode
+
+## Overview
+
+Add a "perpetual monitoring" mode to last30days that lets users track topics over time with cumulative intelligence, delta reporting, and effortless re-runs. Designed for Claude Code's session model - not server cron.
+
+**User signal:** @JTDaly asked "Can your skill be used to do this perpetually?" in response to a Perplexity AI S&P 500 earnings dashboard that auto-refreshes quarterly.
+
+## Problem Statement
+
+last30days is one-shot: ask, get a briefing, done. But the most valuable research is longitudinal - how conversations evolve, when new voices enter, when engagement spikes. Users want "set it and watch" without re-prompting from scratch every time.
+
+## The Scheduling Reality in Claude Code
+
+| Mechanism | Persistence | Max Duration | Fires When |
+|-----------|------------|-------------|------------|
+| **CronCreate / `/loop`** | Session-only (RAM) | 3 days, auto-expires | REPL idle |
+| **System cron/launchd** | Permanent | Forever | Always |
+| **SQLite watchlist** | Permanent (disk) | Forever | On demand |
+
+**Key insight:** Don't build a scheduler. Build a **stateful watchlist** that composes with Claude Code's existing `/loop` for in-session automation, and survives across sessions via SQLite for manual re-runs. The `/loop` skill already exists and does scheduling perfectly - just make last30days a good citizen of it.
+
+## What Already Exists
+
+| Component | File | Status |
+|-----------|------|--------|
+| Topic CRUD + schedule fields | `scripts/watchlist.py` | Working |
+| SQLite persistence with URL dedup, sighting counts | `scripts/store.py` | Working |
+| Daily/weekly briefing generation | `scripts/briefing.py` | Working |
+| Budget tracking (daily cap) | `scripts/store.py` | Working |
+| FTS5 full-text search | `scripts/store.py` | Working |
+| WAL mode for concurrent access | `scripts/store.py` | Working |
+| `delivery_channel` setting in DB | `scripts/store.py` | Schema only |
+| SKILL.md watchlist commands | `SKILL.md` | Missing |
+
+## Proposed Solution
+
+### The Composable Pattern
+
+Instead of building scheduling into last30days, make last30days composable with Claude Code's existing tools:
+
+```
+# One-shot: research a topic and store findings
+/last30 "S&P 500 earnings" --watch
+
+# See what's new since last run (delta briefing)
+/last30 briefing
+
+# Automate with /loop (Claude Code native, session-scoped, 3-day max)
+/loop 4h /last30 briefing
+
+# Next session? Watchlist persists. Just re-loop or run manually.
+/last30 briefing
+```
+
+The user's watchlist lives in SQLite forever. The scheduling is ephemeral by design - you opt into it each session. This matches how people actually use Claude Code: sessions, not servers.
+
+### Phase 1: Watchlist + Briefing via SKILL.md (MVP)
+
+Wire up the existing `watchlist.py` and `briefing.py` infrastructure through the skill interface.
+
+**Tasks:**
+
+- [ ] **SKILL.md additions** - New command branches:
+  - `/last30 watch add "topic"` - Add topic to watchlist, run initial research with `--store`
+  - `/last30 watch list` - Show watched topics with last-run timestamps and finding counts
+  - `/last30 watch remove "topic"` - Remove from watchlist
+  - `/last30 briefing` - Generate delta briefing across all watched topics (what's new)
+  - `/last30 briefing --weekly` - Weekly digest with trend analysis
+  - `/last30 "topic" --watch` - One-shot research that also adds to watchlist
+
+- [ ] **`watchlist.py` updates:**
+  - `run-all` auto-adds `--store` flag so results persist
+  - `run-all` returns structured JSON (exit code + summary) for `/loop` consumption
+  - `run-one` returns per-topic summary for composability
+  - Add `--quick` default for watched topics (save API cost on recurring runs)
+
+- [ ] **`briefing.py` updates:**
+  - `generate` outputs compact format suitable for Claude synthesis (like main script's `--emit=compact`)
+  - Include "last run" timestamp per topic so user knows freshness
+  - Flag stale topics (not updated in 48+ hours)
+
+**Acceptance criteria:**
+- [ ] `/last30 watch add "AI earnings"` persists topic and runs initial research
+- [ ] `/last30 briefing` shows accumulated findings with "new since last briefing" markers
+- [ ] `/loop 4h /last30 briefing` works out of the box (composability)
+- [ ] Watchlist survives across Claude Code sessions (SQLite)
+- [ ] Budget cap enforced (default $5/day)
+
+### Phase 2: Delta Intelligence
+
+Make briefings show what *changed*, not just what exists.
+
+**Tasks:**
+
+- [ ] **`scripts/lib/delta.py`** (~150 lines):
+  - New findings since last briefing (URLs with `first_seen > last_briefing_time`)
+  - Engagement spikes (>2x increase in score since last sighting)
+  - New voices (new @handles appearing for first time in a topic)
+  - Gone quiet (topics with no new findings in 2+ runs)
+
+- [ ] **`briefing.py` delta integration:**
+  - Section: "Breaking" - high engagement + first seen this run
+  - Section: "Trending" - engagement increasing across runs
+  - Section: "New voices" - handles not seen before
+  - Section: "Gone quiet" - previously active, now silent
+
+- [ ] **Smart re-run logic in `watchlist.py`:**
+  - Skip topics that were updated < 4 hours ago (avoid redundant API calls)
+  - Prioritize topics with most engagement change potential
+  - `--force` flag to override skip logic
+
+**Acceptance criteria:**
+- [ ] Briefings clearly distinguish new vs. previously-seen findings
+- [ ] Engagement spikes flagged with specific metric ("upvotes 2.3x since yesterday")
+- [ ] Redundant API calls avoided via smart skip logic
+
+### Phase 3: Delivery + `/loop` Integration
+
+Make the monitoring truly hands-off during a session.
+
+**Tasks:**
+
+- [ ] **Slack delivery** (`scripts/lib/deliver.py` ~100 lines):
+  - Webhook URL config: `watchlist.py config delivery slack --webhook "https://..."`
+  - Format briefing as Slack Block Kit (topic sections, trend indicators)
+  - Auto-deliver after `run-all` if webhook configured
+
+- [ ] **`/last30 monitor` convenience command:**
+  - Shorthand that does: `run-all` + `briefing` + starts `/loop` automatically
+  - Prints: "Monitoring 5 topics every 4h. Auto-expires in 3 days. Run `/last30 monitor` again next session."
+  - Uses CronCreate directly (no `/loop` dependency) for tighter control
+
+- [ ] **Session resume hint:**
+  - On `/last30 briefing`, if watchlist has topics but no `/loop` active, suggest:
+    "You have 5 watched topics. Run `/loop 4h /last30 briefing` to auto-refresh, or `/last30 monitor` for hands-off mode."
+
+**Acceptance criteria:**
+- [ ] Slack webhook delivery works end-to-end
+- [ ] `/last30 monitor` starts automated loop with one command
+- [ ] Clear messaging about 3-day session limit and how to resume
+
+## Architecture: Why This Is Better Than Server Cron
+
+```
+Traditional approach (rejected):
+  System Cron -> watchlist.py run-all -> SQLite -> ??? deliver somehow
+
+Claude Code-native approach:
+  /last30 watch add "topic"   --> SQLite (persists forever)
+  /loop 4h /last30 briefing   --> CronCreate (session, 3-day max)
+       |
+       v
+  Claude reads briefing.py output
+  Claude synthesizes with LLM judgment
+  Claude delivers via Slack webhook
+  Claude answers follow-up questions in context
+```
+
+The Claude Code-native approach is better because:
+1. **LLM synthesis on every run** - not just raw data, but judgment ("this is unusual because...")
+2. **Conversational** - user can ask follow-ups ("tell me more about the META earnings spike")
+3. **Zero infrastructure** - no plist, no crontab, no daemon management
+4. **Portable** - works on any OS where Claude Code runs
+5. **Composable** - `/loop` is a general-purpose tool, not custom scheduling code
+
+## Alternative Approaches Considered
+
+### 1. Build custom scheduler.py with system cron/launchd
+**Rejected.** Doesn't work in Claude Code's model. Platform-specific. Requires root/sudo for some configs. Users of a Claude Code skill shouldn't need to manage system daemons.
+
+### 2. OpenClaw's persistent cron service
+**Not applicable.** OpenClaw has its own cron system with database-backed scheduling, but last30days is an open-source skill that should work without OpenClaw. Could be an optional integration later.
+
+### 3. Long-running Python daemon
+**Rejected.** Fragile, wastes resources, doesn't benefit from LLM synthesis on each run.
+
+### 4. Web dashboard with its own backend
+**Deferred to Phase 4.** Dramatically increases scope. The briefing-to-Slack pattern delivers 80% of the value with 10% of the effort.
+
+## Technical Considerations
+
+**No new services.** Pure Python scripts + SQLite + SKILL.md instructions. Zero infrastructure.
+
+**Cost control.** Each watched topic costs ~$0.05-0.30/run at `--quick` depth. 10 topics x 6 runs/day = $3-18/day. Budget cap in store.py already enforces limits.
+
+**Composability contract.** `briefing.py generate` must output clean, parseable text that Claude can synthesize. No interactive prompts, no side effects beyond SQLite writes.
+
+**Backwards compatibility.** All new. Existing `/last30 topic` unchanged. `--watch` flag is opt-in.
+
+## Dependencies & Risks
+
+| Risk | Likelihood | Mitigation |
+|------|-----------|------------|
+| Users expect "perpetual" to mean forever | High | Clear messaging: "3-day auto-expire per session, watchlist persists, re-run next session" |
+| `/loop` changes or breaks | Low | Composability means we don't depend on `/loop` internals - just CronCreate |
+| Budget overrun with many topics | Low | Budget cap already implemented in store.py |
+| SQLite grows large over months | Low | Add 90-day retention policy |
+| Slack webhook stops working | Low | Log failures, don't block pipeline, alert in next briefing |
+
+## Success Metrics
+
+- User can go from zero to monitoring in one command: `/last30 "S&P 500 earnings" --watch`
+- `/last30 briefing` surfaces genuinely new information with delta markers
+- The system composes cleanly with `/loop` - no special integration needed
+- Watchlist persists across sessions - user picks up where they left off
+- Clear, honest UX about session limits vs. persistent state
+
+## Scope Boundaries
+
+**In scope:**
+- Watchlist CRUD via SKILL.md
+- Delta-aware briefings
+- `/loop` composability (not custom scheduling)
+- Slack webhook delivery
+- `/last30 monitor` convenience command
+
+**Out of scope:**
+- System cron/launchd integration
+- Web dashboard
+- Email delivery (Slack webhook is simpler, covers most users)
+- Multi-user / team features
+- Custom NLP beyond what exists
+
+## Implementation Estimate
+
+- Phase 1 (Watchlist + Briefing): SKILL.md additions, minor `watchlist.py` and `briefing.py` updates
+- Phase 2 (Delta Intelligence): New `delta.py` module, `briefing.py` integration
+- Phase 3 (Delivery + Monitor): New `deliver.py`, SKILL.md `/last30 monitor` command
+
+Each phase ships independently. Phase 1 alone answers @JTDaly's question.