大海
85 lines
2.6 KiB
Markdown
85 lines
2.6 KiB
Markdown
---
|
|
name: compression
|
|
description: Compress tool outputs, logs, and JSON arrays before they enter LLM context. Saves 60-95% tokens.
|
|
metadata: { "openclaw": { "emoji": "📦" } }
|
|
---
|
|
|
|
# Compression — Context-Aware Output Compression
|
|
|
|
Compress large tool outputs before they consume LLM context. Content-type-aware: JSON arrays, build logs, search results, git diffs each get optimal compression.
|
|
|
|
## When to Use
|
|
|
|
- Tool output > 500 chars → compress before reading
|
|
- JSON arrays > 10 items → SmartCrusher
|
|
- Build/CI logs > 50 lines → template dedup
|
|
- Search/grep results > 30 lines → file dedup
|
|
- Git diffs with lock files → noise filter
|
|
- Need to retrieve original compressed data → CCR store
|
|
|
|
## Tools
|
|
|
|
### `ctx_compress` — one-stop compression
|
|
|
|
```bash
|
|
# Auto-detect type, compress stdin
|
|
cat huge_output.json | node src/compression/ctx-compress.js
|
|
|
|
# Specify type, bias
|
|
cat build.log | node src/compression/ctx-compress.js --type log --json
|
|
|
|
# Force JSON mode with query context for relevance
|
|
cat results.json | node src/compression/ctx-compress.js --type json --query "error timeout" --json
|
|
|
|
# Retrieve original from CCR store
|
|
node src/compression/ctx-compress.js --retrieve <ccr_hash>
|
|
```
|
|
|
|
### `smart-crusher` — direct JSON compression
|
|
|
|
```bash
|
|
node src/compression/smart-crusher.js --bias 0.7 --query "search terms" < input.json
|
|
```
|
|
|
|
### Content type detection
|
|
|
|
```bash
|
|
cat unknown_output.txt | node src/compression/detector.js
|
|
```
|
|
|
|
## Compression Strategies
|
|
|
|
| Content Type | Strategy | Savings |
|
|
|-------------|----------|:-------:|
|
|
| JSON array | lossless:csv or smart sample | 50-90% |
|
|
| Build log | template dedup | 90-99% |
|
|
| Search results | file dedup | 80-95% |
|
|
| Git diff | noise filter (lock files etc) | 30-70% |
|
|
|
|
## Key Feature: CCR (Compress-Cache-Retrieve)
|
|
|
|
Compressed output includes `<<ccr:HASH N_rows_offloaded>>` markers. If LLM needs original data, retrieve it:
|
|
|
|
```bash
|
|
node src/compression/ctx-compress.js --retrieve <hash>
|
|
```
|
|
|
|
## Integration Pattern
|
|
|
|
```
|
|
large_tool_output → ctx_compress → compressed → LLM context
|
|
↓
|
|
<<ccr:abc123>>
|
|
↓
|
|
LLM needs more? → ctx_compress --retrieve abc123
|
|
```
|
|
|
|
## File Paths
|
|
|
|
- `src/compression/ctx-compress.js` — main entry
|
|
- `src/compression/smart-crusher.js` — JSON compression engine
|
|
- `src/compression/detector.js` — content type detection
|
|
- `src/compression/ccr-store.js` — reversible storage
|
|
- `src/compression/ccr-backends.js` — SQLite/Redis persistence
|
|
- `src/compression/byte-surgery.js` — cache-safe byte replacement
|