Compare commits
170 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 1ee30ecf28 | |||
| 61d6536754 | |||
| 27b2cd2d02 | |||
| 7eae32d0b7 | |||
| 17624268a0 | |||
| 575cd3b479 | |||
| 0e21545bc5 | |||
| 28e6f07f8e | |||
| 0aee897ef4 | |||
| e36f737bb6 | |||
| a45147c53d | |||
| def2ce932e | |||
| a1cc229239 | |||
| e178470320 | |||
| 7acf3c1367 | |||
| 04fd645966 | |||
| d877c09cd1 | |||
| 65ff8d69d0 | |||
| 0f9cfe66ec | |||
| 23d4d5c611 | |||
| 75bc98a7eb | |||
| bc2d554e13 | |||
| 15f161e5b5 | |||
| 794455cc9f | |||
| 6548a50824 | |||
| 21214fd02a | |||
| c5a304dc20 | |||
| 66883060bc | |||
| f4b3758478 | |||
| d7c8ac81e9 | |||
| 1e0e6d1ae7 | |||
| e90fbfe194 | |||
| 3e90d51da9 | |||
| b82ab1ae10 | |||
| 9a8b245e39 | |||
| 5d046ecb61 | |||
| b4ed28e48f | |||
| f9b7241d47 | |||
| 526616e13f | |||
| e51bd91e82 | |||
| e575ddbcd1 | |||
| d793afaba6 | |||
| ca2e85520b | |||
| 3be2a64120 | |||
| 470c1288d0 | |||
| 881d9b96e2 | |||
| 15563051a1 | |||
| 0c6c45a010 | |||
| 7e66abe636 | |||
| e6406500f3 | |||
| afc8d0e3ee | |||
| 90bb4e0266 | |||
| 7ae0cd8c0a | |||
| c1af1cad14 | |||
| ded444000b | |||
| 5fb0e10167 | |||
| ef8134bba4 | |||
| be9f9ab838 | |||
| 8afd19f473 | |||
| 1f4154a897 | |||
| 1442852471 | |||
| 5461ca4030 | |||
| ab843071ef | |||
| 6c5346cfa4 | |||
| 77c62c76fb | |||
| 7c64579318 | |||
| 383004e06a | |||
| f52fb67232 | |||
| 59b594eccc | |||
| 3416c0f6d3 | |||
| 31f00b8d78 | |||
| ba565bd096 | |||
| 7018d77d58 | |||
| ce9b365491 | |||
| 70a0bb8af8 | |||
| 6737ba6dbd | |||
| f5f4dea075 | |||
| 233cc6fba8 | |||
| ae5812cf7f | |||
| 6af00ce4da | |||
| 3321bf7d34 | |||
| 00d80d2169 | |||
| 55efebd395 | |||
| 16733df4fc | |||
| f3daa5cf97 | |||
| 6ddcfe2b57 | |||
| 0d91d6834a | |||
| dc64f15839 | |||
| c912051173 | |||
| 22d7ad766a | |||
| 5895f3992d | |||
| 9129fc5f27 | |||
| 3fe100cd9a | |||
| c2cbccfed9 | |||
| 0a5c06633c | |||
| 7942f632e5 | |||
| fedbf95f61 | |||
| 1d3fd43fb3 | |||
| c02ffc59a5 | |||
| db20c5d5d3 | |||
| 3a3d38acce | |||
| 4b7d55111f | |||
| eda80b89b5 | |||
| 75a2213510 | |||
| 8a90e52fde | |||
| 9d0191d5eb | |||
| f3d11fe89d | |||
| df34436942 | |||
| cd94706d3b | |||
| 35073aed54 | |||
| 94b428b0b9 | |||
| 105b477a31 | |||
| 92963f0496 | |||
| e5e20a1154 | |||
| b4d189b536 | |||
| 27d3d545e6 | |||
| ca29c4fee5 | |||
| 49d05b7023 | |||
| 476935be92 | |||
| 4f4ad99d49 | |||
| 561b248a2b | |||
| a91a74d880 | |||
| 7434c3cb9f | |||
| 1a61a2cc81 | |||
| 932837ff9a | |||
| 4f52942260 | |||
| 3c1a04a587 | |||
| 6a28b215af | |||
| a881db4261 | |||
| 875f0987a6 | |||
| 0d1a33236e | |||
| e620a1db8f | |||
| 15a2961dfa | |||
| 4bc860c74d | |||
| 62aacf38b5 | |||
| d4e437bf2c | |||
| dbcc668f47 | |||
| 40f833d0d1 | |||
| dc4d7cd7a5 | |||
| 60f5ddf4f0 | |||
| a726aa7fe1 | |||
| 4b2e6f2ffb | |||
| 39f76ee4dc | |||
| a5682716ec | |||
| f269667a34 | |||
| 090b01c909 | |||
| 58b3a4dcb3 | |||
| 37b4cdeddc | |||
| bf6dc7084b | |||
| 50f2181fcc | |||
| b387a67b51 | |||
| a96c5cda14 | |||
| 9c6c04c800 | |||
| cf70302e92 | |||
| 80a08c8c10 | |||
| 606e8900ce | |||
| a37e9aa190 | |||
| 1cbf6a7b9c | |||
| 68319ec683 | |||
| f70711e75e | |||
| c3a9813b1c | |||
| d291d27664 | |||
| 0be4f097d0 | |||
| 12bd727e30 | |||
| b6d135c977 | |||
| 291d073223 | |||
| ef0889dd69 | |||
| 545a6c5761 | |||
| 00f3edd9cc | |||
| 2f674f1e45 |
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"permissions": {
|
||||||
|
"allow": [
|
||||||
|
"WebFetch(domain:community.groq.com)"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
name: ci
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
pull_request:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
test:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
strategy:
|
||||||
|
fail-fast: false
|
||||||
|
matrix:
|
||||||
|
python-version: ["3.10", "3.11", "3.12", "3.13"]
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Setup Python
|
||||||
|
uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: ${{ matrix.python-version }}
|
||||||
|
|
||||||
|
- name: Install package and test deps
|
||||||
|
run: |
|
||||||
|
python -m pip install --upgrade pip
|
||||||
|
pip install -c constraints.txt -e .[dev]
|
||||||
|
|
||||||
|
- name: Run tests
|
||||||
|
run: |
|
||||||
|
pytest -q
|
||||||
+44
-12
@@ -6,17 +6,49 @@ All notable changes to this project will be documented in this file.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## [1.3.1] - 2026-03-27
|
||||||
|
|
||||||
|
### 🐛 Bug Fixes / 修复
|
||||||
|
|
||||||
|
#### 📈 Xueqiu (雪球) — 全面修复
|
||||||
|
|
||||||
|
- **修复 400 错误根本原因:** `_ensure_cookies()` 仅访问首页只能获取 `acw_tc`(防 DDoS token),`xq_a_token` 由雪球前端 JS 动态生成,无法通过纯 HTTP 请求获取。新增三级 cookie 加载策略:① 读取 config 文件(`--from-browser` 保存的)→ ② 自动从本地 Chrome 浏览器提取(需安装 browser-cookie3)→ ③ homepage fallback
|
||||||
|
- **修复 User-Agent:** `"agent-reach/1.0"` 被雪球反爬系统识别拒绝,改为真实 Chrome UA
|
||||||
|
- **修复缺失 `Referer` 头:** 所有 API 请求加上 `Referer: https://xueqiu.com/`
|
||||||
|
- **修复 `get_hot_posts()` 端点:** 原端点 `/statuses/hot/listV3.json` 已废弃(返回空 body),改为 `/v4/statuses/public_timeline_by_category.json`,正确解析 `item.data` JSON 字符串获取 author/likes/text
|
||||||
|
- **修复 `urllib.request.quote` → `urllib.parse.quote`:** 明确使用正确模块
|
||||||
|
- **修复 `configure --from-browser` 不提取雪球 Cookie:** `PLATFORM_SPECS` 加入 Xueqiu,检测 `xq_a_token` 存在才保存
|
||||||
|
- **修正文档误导:** README/SKILL.md 中"无需配置"/"public API, no login required" → 准确描述需要 browser cookie
|
||||||
|
- **改善错误信息:** `check()` 失败时提示 `configure --from-browser chrome` 而非"可能需要代理"
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## [1.3.0] - 2026-03-12
|
||||||
|
|
||||||
|
### 🆕 New Channels / 新增渠道
|
||||||
|
|
||||||
|
#### 💻 V2EX
|
||||||
|
- Hot topics, node topics, topic detail + replies, user profile via public JSON API
|
||||||
|
- Zero config — no auth, no proxy, no API key required
|
||||||
|
- `get_hot_topics(limit)`, `get_node_topics(node_name, limit)`, `get_topic(id)`, `get_user(username)`
|
||||||
|
- 通过公开 JSON API 获取热门帖子、节点帖子、帖子详情+回复、用户信息
|
||||||
|
- 零配置,无需认证、无需代理、无需 API Key
|
||||||
|
|
||||||
|
### 📈 Improvements / 改进
|
||||||
|
|
||||||
|
- Channel count: 14 → 15
|
||||||
|
- 渠道数量:14 → 15
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## [1.1.0] - 2025-02-25
|
## [1.1.0] - 2025-02-25
|
||||||
|
|
||||||
### 🆕 New Channels / 新增渠道
|
### 🆕 New Channels / 新增渠道
|
||||||
|
|
||||||
#### 📷 Instagram
|
#### ~~📷 Instagram~~ (removed — upstream blocked)
|
||||||
- Read public posts and profiles via [instaloader](https://github.com/instaloader/instaloader)
|
- ~~Read public posts and profiles via [instaloader](https://github.com/instaloader/instaloader)~~
|
||||||
- Search via Exa (free, no API key)
|
- **Removed:** Instagram's aggressive anti-scraping measures broke all available open-source tools (instaloader, etc.). See [instaloader#2585](https://github.com/instaloader/instaloader/issues/2585). Will re-add when upstream recovers.
|
||||||
- Optional cookie login for private content
|
- **已移除:** Instagram 反爬封杀导致所有开源工具(instaloader 等)失效。上游恢复后会重新加回。
|
||||||
- 通过 instaloader 读取公开帖子和 Profile
|
|
||||||
- 搜索通过 Exa(免费,无需 API Key)
|
|
||||||
- 可选 Cookie 登录解锁私密内容
|
|
||||||
|
|
||||||
#### 💼 LinkedIn
|
#### 💼 LinkedIn
|
||||||
- Read person profiles, company pages, and job details via [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
- Read person profiles, company pages, and job details via [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||||
@@ -38,12 +70,12 @@ All notable changes to this project will be documented in this file.
|
|||||||
|
|
||||||
- Channel count: 9 → 12
|
- Channel count: 9 → 12
|
||||||
- `agent-reach doctor` now detects all 12 channels
|
- `agent-reach doctor` now detects all 12 channels
|
||||||
- CLI: added `search-instagram`, `search-linkedin`, `search-bosszhipin` subcommands
|
- CLI: added `search-linkedin`, `search-bosszhipin` subcommands
|
||||||
- Updated install guide with setup instructions for new channels
|
- Updated install guide with setup instructions for new channels
|
||||||
- 渠道数量:9 → 12
|
- 渠道数量:9 → 11
|
||||||
- `agent-reach doctor` 现在检测全部 12 个渠道
|
- `agent-reach doctor` 现在检测全部 11 个渠道
|
||||||
- CLI:新增 `search-instagram`、`search-linkedin`、`search-bosszhipin` 子命令
|
- CLI:新增 `search-linkedin`、`search-bosszhipin` 子命令
|
||||||
- 安装指南新增三个渠道的配置说明
|
- 安装指南新增渠道配置说明
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# CLAUDE.md
|
||||||
|
|
||||||
|
## Project
|
||||||
|
Agent Reach — Python CLI + library that gives AI agents read/search access to 14+ internet platforms.
|
||||||
|
Positioning: installer + doctor + config tool. NOT a wrapper — after install, agents call upstream tools directly.
|
||||||
|
Repo: github.com/Panniantong/Agent-Reach | License: MIT | Version: 1.3.0
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
- `pip install -e .` — Dev install
|
||||||
|
- `pytest tests/ -v` — All tests
|
||||||
|
- `pytest tests/test_cli.py -v` — CLI tests only
|
||||||
|
- `bash test.sh` — Full integration test (creates venv, installs, runs doctor + channel tests)
|
||||||
|
- `python -m agent_reach.cli doctor` — Run diagnostics
|
||||||
|
- `python -m agent_reach.cli install --env=auto` — Auto-configure
|
||||||
|
|
||||||
|
## Structure
|
||||||
|
- `agent_reach/cli.py` — CLI entry point (argparse)
|
||||||
|
- `agent_reach/core.py` — Core read/search routing logic
|
||||||
|
- `agent_reach/config.py` — Config management (YAML, env vars)
|
||||||
|
- `agent_reach/doctor.py` — Diagnostics engine
|
||||||
|
- `agent_reach/channels/` — One file per platform (twitter.py, reddit.py, youtube.py, etc.)
|
||||||
|
- `agent_reach/channels/base.py` — Base channel class (all channels inherit from this)
|
||||||
|
- `agent_reach/integrations/mcp_server.py` — MCP server integration
|
||||||
|
- `agent_reach/skill/` — OpenClaw skill files
|
||||||
|
- `agent_reach/guides/` — Usage guides
|
||||||
|
- `tests/` — pytest tests
|
||||||
|
- `config/mcporter.json` — MCP tool config
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
- Python 3.10+ with type hints
|
||||||
|
- Each channel is a single file in `channels/`, inherits from `BaseChannel`
|
||||||
|
- Channel contract: must implement `can_handle(url)`, `read(url)`, `search(query)`, `check()` methods
|
||||||
|
- Use `loguru` for logging, `rich` for CLI output
|
||||||
|
- Commit format: `type(scope): message` (one commit = one thing)
|
||||||
|
- All upstream tool calls go through public API/CLI, never hack internals
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
- NEVER modify upstream open source projects' source code
|
||||||
|
- Agent Reach is a "glue layer" — only route and call, don't reimagine
|
||||||
|
- Version in THREE places must match: `pyproject.toml`, `__init__.py`, `tests/test_cli.py`
|
||||||
|
- Always new branch for changes, PR to main, never push to main directly
|
||||||
|
- Run `pytest tests/ -v` before committing — all tests must pass
|
||||||
|
- Cookie-based auth (Twitter, XHS): use Cookie-Editor export method only, no QR scan
|
||||||
|
- XHS login: Cookie-Editor browser export only (QR will hang)
|
||||||
+107
@@ -0,0 +1,107 @@
|
|||||||
|
# Contributing to Agent Reach
|
||||||
|
|
||||||
|
Thank you for your interest in contributing to Agent Reach! This document provides guidelines and instructions for contributing.
|
||||||
|
|
||||||
|
## Getting Started
|
||||||
|
|
||||||
|
1. Fork the repository on GitHub
|
||||||
|
2. Clone your fork locally
|
||||||
|
3. Create a new branch for your contribution
|
||||||
|
4. Make your changes
|
||||||
|
5. Run tests and linting
|
||||||
|
6. Submit a pull request
|
||||||
|
|
||||||
|
## Development Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone your fork
|
||||||
|
git clone https://github.com/YOUR_USERNAME/Agent-Reach.git
|
||||||
|
cd Agent-Reach
|
||||||
|
|
||||||
|
# Install in development mode
|
||||||
|
pip install -e ".[dev]"
|
||||||
|
|
||||||
|
# Install pre-commit hooks (optional but recommended)
|
||||||
|
pre-commit install
|
||||||
|
```
|
||||||
|
|
||||||
|
## Code Style
|
||||||
|
|
||||||
|
We use the following tools to maintain code quality:
|
||||||
|
|
||||||
|
- **ruff**: Linting and import sorting
|
||||||
|
- **mypy**: Type checking
|
||||||
|
- **pytest**: Testing
|
||||||
|
|
||||||
|
Run all checks before submitting a PR:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Linting
|
||||||
|
ruff check agent_reach tests
|
||||||
|
ruff format agent_reach tests
|
||||||
|
|
||||||
|
# Type checking
|
||||||
|
mypy agent_reach
|
||||||
|
|
||||||
|
# Tests
|
||||||
|
pytest
|
||||||
|
```
|
||||||
|
|
||||||
|
## Adding New Channels
|
||||||
|
|
||||||
|
Agent Reach uses a unified channel interface. To add a new platform:
|
||||||
|
|
||||||
|
1. Create a new file in `agent_reach/channels/`
|
||||||
|
2. Implement the channel contract (see existing channels for examples)
|
||||||
|
3. Add tests in `tests/test_channels.py`
|
||||||
|
4. Update `agent_reach/doctor.py` to include the new channel
|
||||||
|
5. Update documentation
|
||||||
|
|
||||||
|
## Pull Request Guidelines
|
||||||
|
|
||||||
|
- **Small, focused changes** are preferred over large refactors
|
||||||
|
- Include tests for new functionality
|
||||||
|
- Update documentation if needed
|
||||||
|
- Follow existing code style
|
||||||
|
- Reference any related issues
|
||||||
|
|
||||||
|
## Reporting Issues
|
||||||
|
|
||||||
|
When reporting bugs, please include:
|
||||||
|
|
||||||
|
- Python version
|
||||||
|
- Operating system
|
||||||
|
- Steps to reproduce
|
||||||
|
- Expected vs actual behavior
|
||||||
|
- Any error messages
|
||||||
|
|
||||||
|
## Questions?
|
||||||
|
|
||||||
|
Feel free to open an issue for questions or join discussions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
感谢您对 Agent Reach 的贡献!本文档提供了贡献指南。
|
||||||
|
|
||||||
|
## 快速开始
|
||||||
|
|
||||||
|
1. 在 GitHub 上 fork 仓库
|
||||||
|
2. 本地 clone 您的 fork
|
||||||
|
3. 创建新分支
|
||||||
|
4. 提交更改
|
||||||
|
5. 运行测试和 lint
|
||||||
|
6. 提交 pull request
|
||||||
|
|
||||||
|
## 代码规范
|
||||||
|
|
||||||
|
- 使用 **ruff** 进行代码检查
|
||||||
|
- 使用 **mypy** 进行类型检查
|
||||||
|
- 使用 **pytest** 运行测试
|
||||||
|
|
||||||
|
## 添加新渠道
|
||||||
|
|
||||||
|
1. 在 `agent_reach/channels/` 创建新文件
|
||||||
|
2. 实现渠道接口
|
||||||
|
3. 添加测试
|
||||||
|
4. 更新 doctor 检测
|
||||||
|
5. 更新文档
|
||||||
@@ -11,7 +11,7 @@
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<a href="#30-秒上手">快速开始</a> · <a href="docs/README_en.md">English</a> · <a href="#支持的平台">支持平台</a> · <a href="#设计理念">设计理念</a>
|
<a href="#快速上手">快速开始</a> · <a href="docs/README_en.md">English</a> · <a href="docs/README_ja.md">日本語</a> · <a href="docs/README_ko.md">한국어</a> · <a href="#支持的平台">支持平台</a> · <a href="#设计理念">设计理念</a>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -42,6 +42,12 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|
|
||||||
复制给你的 Agent,几分钟后它就能读推特、搜 Reddit、看 YouTube、刷小红书了。
|
复制给你的 Agent,几分钟后它就能读推特、搜 Reddit、看 YouTube、刷小红书了。
|
||||||
|
|
||||||
|
**已经装过了?更新也是一句话:**
|
||||||
|
|
||||||
|
```
|
||||||
|
帮我更新 Agent Reach:https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
```
|
||||||
|
|
||||||
> ⭐ **Star 这个项目**,我们会持续追踪各平台的变化、接入新的渠道。你不用自己盯——平台封了我们修,有新渠道我们加。
|
> ⭐ **Star 这个项目**,我们会持续追踪各平台的变化、接入新的渠道。你不用自己盯——平台封了我们修,有新渠道我们加。
|
||||||
|
|
||||||
### ✅ 在你用之前,你可能想知道
|
### ✅ 在你用之前,你可能想知道
|
||||||
@@ -50,7 +56,7 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|---|---|
|
|---|---|
|
||||||
| 💰 **完全免费** | 所有工具开源、所有 API 免费。唯一可能花钱的是服务器代理($1/月),本地电脑不需要 |
|
| 💰 **完全免费** | 所有工具开源、所有 API 免费。唯一可能花钱的是服务器代理($1/月),本地电脑不需要 |
|
||||||
| 🔒 **隐私安全** | Cookie 只存在你本地,不上传不外传。代码完全开源,随时可审查 |
|
| 🔒 **隐私安全** | Cookie 只存在你本地,不上传不外传。代码完全开源,随时可审查 |
|
||||||
| 🔄 **持续更新** | 底层工具(yt-dlp、bird、Jina Reader 等)定期追踪更新到最新版,你不用自己盯 |
|
| 🔄 **持续更新** | 底层工具(yt-dlp、twitter-cli、rdt-cli、Jina Reader 等)定期追踪更新到最新版,你不用自己盯 |
|
||||||
| 🤖 **兼容所有 Agent** | Claude Code、OpenClaw、Cursor、Windsurf……任何能跑命令行的 Agent 都能用 |
|
| 🤖 **兼容所有 Agent** | Claude Code、OpenClaw、Cursor、Windsurf……任何能跑命令行的 Agent 都能用 |
|
||||||
| 🩺 **自带诊断** | `agent-reach doctor` 一条命令告诉你哪个通、哪个不通、怎么修 |
|
| 🩺 **自带诊断** | `agent-reach doctor` 一条命令告诉你哪个通、哪个不通、怎么修 |
|
||||||
|
|
||||||
@@ -67,15 +73,19 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
| 📦 **GitHub** | 读公开仓库 + 搜索 | 私有仓库、提 Issue/PR、Fork | 告诉 Agent「帮我登录 GitHub」 |
|
| 📦 **GitHub** | 读公开仓库 + 搜索 | 私有仓库、提 Issue/PR、Fork | 告诉 Agent「帮我登录 GitHub」 |
|
||||||
| 🐦 **Twitter/X** | 读单条推文 | 搜索推文、浏览时间线、发推 | 告诉 Agent「帮我配 Twitter」 |
|
| 🐦 **Twitter/X** | 读单条推文 | 搜索推文、浏览时间线、发推 | 告诉 Agent「帮我配 Twitter」 |
|
||||||
| 📺 **B站** | 本地:字幕提取 + 搜索 | 服务器也能用 | 告诉 Agent「帮我配代理」 |
|
| 📺 **B站** | 本地:字幕提取 + 搜索 | 服务器也能用 | 告诉 Agent「帮我配代理」 |
|
||||||
| 📖 **Reddit** | 搜索(通过 Exa 免费) | 读帖子和评论 | 告诉 Agent「帮我配代理」 |
|
| 📖 **Reddit** | 搜索 + 读帖子和评论(通过 rdt-cli) | Cookie | 需要登录认证(`rdt login`),详见 [rdt-cli](https://github.com/public-clis/rdt-cli) |
|
||||||
| 📕 **小红书** | — | 阅读、搜索、发帖、评论、点赞 | `docker run -d -p 18060:18060 xpzouying/xiaohongshu-mcp` 然后告诉 Agent「帮我配置小红书」 |
|
| 📕 **小红书** | — | 阅读、搜索、发帖、评论、点赞 | 告诉 Agent「帮我配小红书」 |
|
||||||
| 📷 **Instagram** | — | 读取帖子和 Profile | Cookie-Editor 导出 Cookie → `agent-reach configure instagram-cookies "..."` |
|
| 🎵 **抖音** | — | 视频解析、无水印下载链接获取 | 告诉 Agent「帮我配抖音」 |
|
||||||
| 💼 **LinkedIn** | Jina Reader 读公开页面 | Profile 详情、公司、职位搜索 | `linkedin-scraper-mcp --login`(需浏览器,服务器用 VNC) |
|
| 💼 **LinkedIn** | Jina Reader 读公开页面 | Profile 详情、公司页面、职位搜索 | 告诉 Agent「帮我配 LinkedIn」 |
|
||||||
| 🏢 **Boss直聘** | Jina Reader 读职位页 | 搜索职位、向 HR 打招呼 | 启动 mcp-bosszp → Boss直聘 App 扫码 |
|
| 💬 **微信公众号** | 搜索 + 阅读公众号文章(全文 Markdown) | — | 无需配置 |
|
||||||
|
| 📰 **微博** | 热搜、搜索内容/用户/话题、用户动态、评论 | — | 无需配置 |
|
||||||
|
| 💻 **V2EX** | 热门帖子、节点帖子、帖子详情+回复、用户信息 | — | 无需配置 |
|
||||||
|
| 📈 **雪球** | 股票行情、搜索股票、热门帖子、热门股票排行 | — | 告诉 Agent「帮我配雪球」 |
|
||||||
|
| 🎙️ **小宇宙播客** | — | 播客音频转文字(Whisper 转录,免费 Key) | 告诉 Agent「帮我配小宇宙播客」 |
|
||||||
|
|
||||||
> **不知道怎么配?不用查文档。** 直接告诉 Agent「帮我配 XXX」,它知道需要什么、会一步一步引导你。
|
> **不知道怎么配?不用查文档。** 直接告诉 Agent「帮我配 XXX」,它知道需要什么、会一步一步引导你。
|
||||||
>
|
>
|
||||||
> 🍪 需要 Cookie 的平台(Twitter、小红书等),建议使用 Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 一键导出。**服务器用户**没有浏览器界面,请在自己的电脑上登录对应网站后导出 Cookie,再发给 Agent 配置。
|
> 🍪 需要 Cookie 的平台(Twitter、小红书等),**优先使用** Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 导出 Cookie,发给 Agent 即可配置。流程统一:浏览器登录 → Cookie-Editor 导出 → 发给 Agent。比扫码更简单可靠。
|
||||||
>
|
>
|
||||||
> 🔒 Cookie 只存在你本地,不上传不外传。代码完全开源,随时可审查。
|
> 🔒 Cookie 只存在你本地,不上传不外传。代码完全开源,随时可审查。
|
||||||
> 💻 本地电脑不需要代理。代理只有部署在服务器上才需要(~$1/月)。
|
> 💻 本地电脑不需要代理。代理只有部署在服务器上才需要(~$1/月)。
|
||||||
@@ -84,6 +94,16 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|
|
||||||
## 快速上手
|
## 快速上手
|
||||||
|
|
||||||
|
> ⚠️ **OpenClaw 用户请先确认 exec 权限已开启**
|
||||||
|
>
|
||||||
|
> Agent Reach 依赖 Agent 执行 shell 命令(`pip install`、`mcporter`、`twitter` 等)。如果你的 OpenClaw 使用了默认的 `messaging` 工具配置,Agent 将无法执行命令。**安装前请先开启 exec 权限**:
|
||||||
|
>
|
||||||
|
> ```bash
|
||||||
|
> openclaw config set tools.profile "coding"
|
||||||
|
> ```
|
||||||
|
> 或在 `~/.openclaw/openclaw.json` 中设置 `"tools": { "profile": "coding" }`。
|
||||||
|
> 设置后重启 Gateway(`openclaw gateway restart`)并开启新对话即可。其他平台(Claude Code、Cursor、Windsurf 等)不受此限制。
|
||||||
|
|
||||||
复制这句话给你的 AI Agent(Claude Code、OpenClaw、Cursor 等):
|
复制这句话给你的 AI Agent(Claude Code、OpenClaw、Cursor 等):
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -92,6 +112,11 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|
|
||||||
就这一步。Agent 会自己完成剩下的所有事情。
|
就这一步。Agent 会自己完成剩下的所有事情。
|
||||||
|
|
||||||
|
> 🔄 **已安装过?** 更新也是一句话:
|
||||||
|
> ```
|
||||||
|
> 帮我更新 Agent Reach:https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
> ```
|
||||||
|
|
||||||
> 🛡️ **担心安全?** 可以用安全模式——不会自动装系统包,只告诉你需要什么:
|
> 🛡️ **担心安全?** 可以用安全模式——不会自动装系统包,只告诉你需要什么:
|
||||||
> ```
|
> ```
|
||||||
> 帮我安装 Agent Reach(安全模式):https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
> 帮我安装 Agent Reach(安全模式):https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
||||||
@@ -102,10 +127,10 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
<summary>它会做什么?(点击展开)</summary>
|
<summary>它会做什么?(点击展开)</summary>
|
||||||
|
|
||||||
1. **安装 CLI 工具** — `pip install` 装好 `agent-reach` 命令行
|
1. **安装 CLI 工具** — `pip install` 装好 `agent-reach` 命令行
|
||||||
2. **安装系统依赖** — 自动检测并安装 Node.js、gh CLI、mcporter、bird 等
|
2. **安装系统依赖** — 自动检测并安装 Node.js、gh CLI、mcporter、twitter-cli、rdt-cli 等
|
||||||
3. **配置搜索引擎** — 通过 MCP 接入 Exa(免费,无需 API Key)
|
3. **配置搜索引擎** — 通过 MCP 接入 Exa(免费,无需 API Key)
|
||||||
4. **检测环境** — 判断是本地电脑还是服务器,给出对应的配置建议
|
4. **检测环境** — 判断是本地电脑还是服务器,给出对应的配置建议
|
||||||
5. **注册 Skill** — 在 Agent 的 skills 目录安装 SKILL.md,以后 Agent 遇到"搜推特"、"看视频"这类需求,会自动知道调用 Agent Reach
|
5. **注册 SKILL.md** — 在 Agent 的 skills 目录安装使用指南,以后 Agent 遇到"搜推特"、"看视频"这类需求,会自动知道该调哪个上游工具
|
||||||
|
|
||||||
安装完之后,`agent-reach doctor` 一条命令告诉你每个渠道的状态。
|
安装完之后,`agent-reach doctor` 一条命令告诉你每个渠道的状态。
|
||||||
</details>
|
</details>
|
||||||
@@ -116,14 +141,14 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|
|
||||||
不需要任何配置,告诉 Agent 就行:
|
不需要任何配置,告诉 Agent 就行:
|
||||||
|
|
||||||
- "帮我看看这个链接" → 任意网页
|
- "帮我看看这个链接" → `curl https://r.jina.ai/URL` 读任意网页
|
||||||
- "这个 GitHub 仓库是做什么的" → GitHub 仓库、Issue、代码
|
- "这个 GitHub 仓库是做什么的" → `gh repo view owner/repo`
|
||||||
- "这个视频讲了什么" → YouTube / B站字幕提取
|
- "这个视频讲了什么" → `yt-dlp --dump-json URL` 提取字幕
|
||||||
- "帮我看看这条推文" → Twitter 推文
|
- "帮我看看这条推文" → `twitter tweet URL`
|
||||||
- "订阅这个 RSS" → RSS / Atom 源
|
- "订阅这个 RSS" → `feedparser` 解析
|
||||||
- "搜一下 GitHub 上有什么 LLM 框架" → GitHub 搜索
|
- "搜一下 GitHub 上有什么 LLM 框架" → `gh search repos "LLM framework"`
|
||||||
|
|
||||||
**不需要记命令。** Agent 自己知道该调什么。
|
**不需要记命令。** Agent 读了 SKILL.md 之后自己知道该调什么。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -135,44 +160,76 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|
|
||||||
Agent Reach 做的事情很简单:**帮你把这些选型和配置的活儿做完了。**
|
Agent Reach 做的事情很简单:**帮你把这些选型和配置的活儿做完了。**
|
||||||
|
|
||||||
|
安装完成后,Agent 直接调用上游工具(twitter-cli、rdt-cli、xhs-cli、yt-dlp、mcporter、gh CLI 等),不需要经过 Agent Reach 的包装层。
|
||||||
|
|
||||||
### 🔌 每个渠道都是可插拔的
|
### 🔌 每个渠道都是可插拔的
|
||||||
|
|
||||||
每个平台对应一个独立的 Python 文件,实现统一接口。**后端工具随时可以换**——哪天出了更好的工具,改一个文件就行,其他不用动。
|
每个平台背后是一个独立的上游工具。**不满意?换掉就行。**
|
||||||
|
|
||||||
```
|
```
|
||||||
channels/
|
channels/
|
||||||
├── web.py → Jina Reader ← 可以换成 Firecrawl、Crawl4AI……
|
├── web.py → Jina Reader ← 可以换成 Firecrawl、Crawl4AI……
|
||||||
├── twitter.py → bird ← 可以换成 Nitter、官方 API……
|
├── twitter.py → twitter-cli ← 可以换成官方 API……
|
||||||
├── youtube.py → yt-dlp ← 可以换成 YouTube API、Whisper……
|
├── youtube.py → yt-dlp ← 可以换成 YouTube API、Whisper……
|
||||||
├── github.py → gh CLI ← 可以换成 REST API、PyGithub……
|
├── github.py → gh CLI ← 可以换成 REST API、PyGithub……
|
||||||
├── bilibili.py → yt-dlp ← 可以换成 bilibili-api……
|
├── bilibili.py → yt-dlp ← 可以换成 bilibili-api……
|
||||||
├── reddit.py → JSON API + Exa ← 可以换成 PRAW、Pushshift……
|
├── reddit.py → rdt-cli ← 搜索+阅读,需 Cookie 认证
|
||||||
├── xiaohongshu.py → mcporter MCP ← 可以换成其他 XHS 工具……
|
├── xiaohongshu.py → mcporter MCP ← 可以换成其他 XHS 工具……
|
||||||
├── instagram.py → instaloader ← 可以换成 instagrapi、官方 API……
|
├── douyin.py → mcporter MCP ← 可以换成其他抖音工具……
|
||||||
├── linkedin.py → linkedin-mcp ← 可以换成 LinkedIn API……
|
├── linkedin.py → linkedin-mcp ← 可以换成 LinkedIn API……
|
||||||
├── bosszhipin.py → mcp-bosszp ← 可以换成其他招聘工具……
|
├── wechat.py → Exa (+ Camoufox) ← 搜索+阅读微信公众号文章
|
||||||
├── rss.py → feedparser ← 可以换成 atoma……
|
├── rss.py → feedparser ← 可以换成 atoma……
|
||||||
├── exa_search.py → mcporter MCP ← 可以换成 Tavily、SerpAPI……
|
├── exa_search.py → mcporter MCP ← 可以换成 Tavily、SerpAPI……
|
||||||
└── __init__.py → 渠道注册
|
└── __init__.py → 渠道注册(doctor 检测用)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
每个渠道文件只负责检测对应上游工具是否可用(`check()` 方法),给 `agent-reach doctor` 提供状态信息。实际的读取和搜索由 Agent 直接调用上游工具完成。
|
||||||
|
|
||||||
### 当前选型
|
### 当前选型
|
||||||
|
|
||||||
| 场景 | 选型 | 为什么选它 |
|
| 场景 | 选型 | 为什么选它 |
|
||||||
|------|------|-----------|
|
|------|------|-----------|
|
||||||
| 读网页 | [Jina Reader](https://github.com/jina-ai/reader) | 9.8K Star,免费,不需要 API Key |
|
| 读网页 | [Jina Reader](https://github.com/jina-ai/reader) | 9.8K Star,免费,不需要 API Key |
|
||||||
| 读推特 | [bird](https://www.npmjs.com/package/@steipete/bird) | Cookie 登录,免费。官方 API 按量付费(读一条 $0.005) |
|
| 读推特 | [twitter-cli](https://github.com/public-clis/twitter-cli) | 2.1K Star,Cookie 登录,搜索/读推文/时间线/长文 |
|
||||||
| 视频字幕 + 搜索 | [yt-dlp](https://github.com/yt-dlp/yt-dlp) | 148K Star,YouTube + B站 + 1800 站通吃 |
|
| Reddit | [rdt-cli](https://github.com/public-clis/rdt-cli) | 304 Star,Cookie 认证,搜索+全文+评论 |
|
||||||
| 搜全网 | [Exa](https://exa.ai) via [mcporter](https://github.com/steipete/mcporter) | AI 语义搜索,MCP 接入免 Key |
|
| 视频字幕 + 搜索 | [yt-dlp](https://github.com/yt-dlp/yt-dlp) | 154K Star,YouTube + B站 + 1800 站通吃 |
|
||||||
|
| B站增强 | [bili-cli](https://github.com/public-clis/bilibili-cli) | 590 Star,热门/排行/搜索/动态 |
|
||||||
|
| 搜全网 | [Exa](https://exa.ai) via [mcporter](https://github.com/nicobailon/mcporter) | AI 语义搜索,MCP 接入免 Key |
|
||||||
| GitHub | [gh CLI](https://cli.github.com) | 官方工具,认证后完整 API 能力 |
|
| GitHub | [gh CLI](https://cli.github.com) | 官方工具,认证后完整 API 能力 |
|
||||||
| 读 RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python 生态标准选择,2.3K Star |
|
| 读 RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python 生态标准选择,2.3K Star |
|
||||||
| 小红书 | [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) | ⭐9K+,Go 语言,Docker 一键部署 |
|
| 小红书 | [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) | 1.5K Star,pipx 一行安装,搜索/阅读/评论/发帖 |
|
||||||
| Instagram | [instaloader](https://github.com/instaloader/instaloader) | ⭐9.8K,Python CLI,Cookie 登录,免费 |
|
| 抖音 | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP 服务,无需登录,视频解析 + 无水印下载 |
|
||||||
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | ⭐900+,MCP 服务,浏览器自动化 |
|
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | ⭐1.2K,MCP 服务,浏览器自动化 |
|
||||||
| Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP 服务,支持职位搜索和打招呼 |
|
| 微信公众号 | [Exa](https://exa.ai)(搜索+阅读)+ [Camoufox](https://github.com/daijro/camoufox)(可选) | 零配置搜索+全文阅读,Camoufox 可选增强 |
|
||||||
|
|
||||||
> 📌 这些都是「当前选型」。不满意?换掉对应文件就行。这正是脚手架的意义。
|
> 📌 这些都是「当前选型」。不满意?换掉对应文件就行。这正是脚手架的意义。
|
||||||
|
|
||||||
|
### 抖音 / 小红书脚本提取的可选实现
|
||||||
|
|
||||||
|
如果你不只是想“解析抖音视频信息”,还想统一处理:
|
||||||
|
|
||||||
|
- 抖音视频脚本提取
|
||||||
|
- 小红书视频笔记脚本提取
|
||||||
|
- 小红书图文笔记正文 + 图片文字提取
|
||||||
|
- 固定输出 `script.md` 和 `info.json`
|
||||||
|
|
||||||
|
可以把 `douyin` 这个 mcporter alias 指向另一个兼容实现:
|
||||||
|
|
||||||
|
- [social-post-extractor-mcp](https://github.com/JNHFlow21/social-post-extractor-mcp)
|
||||||
|
|
||||||
|
这个实现保留了旧工具名兼容性:
|
||||||
|
|
||||||
|
- `parse_douyin_video_info`
|
||||||
|
- `get_douyin_download_link`
|
||||||
|
- `extract_douyin_text`
|
||||||
|
|
||||||
|
同时新增统一工具:
|
||||||
|
|
||||||
|
- `parse_social_post_info`
|
||||||
|
- `extract_social_post_script`
|
||||||
|
|
||||||
|
所以从 Agent Reach 的视角看,它依然只是一个 `mcporter` 里的 `douyin` server,只是能力更完整。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 安全性
|
## 安全性
|
||||||
@@ -189,7 +246,11 @@ Agent Reach 在设计上重视安全:
|
|||||||
|
|
||||||
### 🍪 Cookie 安全建议
|
### 🍪 Cookie 安全建议
|
||||||
|
|
||||||
需要 Cookie 的平台(Twitter、小红书、Instagram)建议使用**专用小号**,不要用主账号。Cookie 等同于完整登录权限,用小号可以在凭据泄露时限制影响范围。
|
> ⚠️ **封号风险提醒:** 使用 Cookie 登录的平台(Twitter、小红书等),通过脚本/API 调用**存在被平台检测并封号的风险**。请务必使用**专用小号**,不要用你的主账号。
|
||||||
|
|
||||||
|
需要 Cookie 的平台(Twitter、小红书)建议使用**专用小号**,不要用主账号。原因有二:
|
||||||
|
1. **封号风险** — 平台可能检测到非正常浏览器的 API 调用行为,导致账号被限制或封禁
|
||||||
|
2. **安全风险** — Cookie 等同于完整登录权限,用小号可以在凭据泄露时限制影响范围
|
||||||
|
|
||||||
### 📦 安装方式
|
### 📦 安装方式
|
||||||
|
|
||||||
@@ -199,6 +260,24 @@ Agent Reach 在设计上重视安全:
|
|||||||
| 安全模式 | `agent-reach install --env=auto --safe` | 生产服务器、多人共用机器 |
|
| 安全模式 | `agent-reach install --env=auto --safe` | 生产服务器、多人共用机器 |
|
||||||
| 仅预览 | `agent-reach install --env=auto --dry-run` | 先看看会做什么 |
|
| 仅预览 | `agent-reach install --env=auto --dry-run` | 先看看会做什么 |
|
||||||
|
|
||||||
|
### 🗑️ 卸载
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach uninstall
|
||||||
|
```
|
||||||
|
|
||||||
|
会清除:`~/.agent-reach/`(含所有 token/cookie)、各 Agent 的 skill 文件、mcporter 中的 MCP 配置。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 只预览,不实际删除
|
||||||
|
agent-reach uninstall --dry-run
|
||||||
|
|
||||||
|
# 只删 skill 文件,保留 token 配置(重装时用)
|
||||||
|
agent-reach uninstall --keep-config
|
||||||
|
```
|
||||||
|
|
||||||
|
卸载 Python 包本身:`pip uninstall agent-reach`
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 贡献
|
## 贡献
|
||||||
@@ -225,10 +304,91 @@ Agent Reach 在设计上重视安全:
|
|||||||
|
|
||||||
Star 一下,下次需要的时候能找到。⭐
|
Star 一下,下次需要的时候能找到。⭐
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 常见问题 / FAQ
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>AI Agent 怎么搜索 Twitter / X?不想付 API 费用</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach 使用 [twitter-cli](https://github.com/public-clis/twitter-cli) 通过 Cookie 认证访问 Twitter,完全免费。安装:`pipx install twitter-cli`,确保浏览器已登录 x.com,Agent 就可以用 `twitter search "关键词"` 搜索、`twitter tweet URL` 读推文了。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to search Twitter/X with AI agent for free (no API)?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach uses twitter-cli with cookie auth — zero API fees. Install with `pipx install twitter-cli`, make sure you're logged into x.com in your browser, then your agent can search with `twitter search "query"` and read tweets with `twitter tweet URL`.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Reddit 返回 403 怎么办?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach 使用 [rdt-cli](https://github.com/public-clis/rdt-cli) 访问 Reddit。Reddit 自 2024 年起要求认证,安装后需运行 `rdt login` 登录。安装:`pipx install rdt-cli`,然后 `rdt login`(自动从浏览器提取 Cookie)。之后 Agent 可以用 `rdt search "关键词"` 搜索、`rdt read POST_ID` 读帖子全文和评论。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to get YouTube video transcripts for AI?</strong></summary>
|
||||||
|
|
||||||
|
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"` extracts video metadata; `yt-dlp --write-sub --skip-download "URL"` extracts subtitles. Uses yt-dlp under the hood, supports multiple languages. No API key needed.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>怎么让 AI Agent 读小红书?</strong></summary>
|
||||||
|
|
||||||
|
安装 `pipx install xiaohongshu-cli`,然后 `xhs login`(自动从浏览器提取 Cookie)。之后 Agent 就能用 `xhs search "关键词"` 搜索笔记、`xhs read NOTE_ID` 阅读详情、`xhs comments NOTE_ID` 查看评论了。不需要 Docker。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>怎么让 AI Agent 解析抖音视频?</strong></summary>
|
||||||
|
|
||||||
|
安装 douyin-mcp-server 后,Agent 就能用 `mcporter call 'douyin.parse_douyin_video_info(share_link: "分享链接")'` 解析视频信息、获取无水印下载链接。不需要登录,把抖音分享链接发给 Agent 就行。详见 https://github.com/yzfly/douyin-mcp-server
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Compatible with Claude Code / Cursor / OpenClaw / Windsurf?</strong></summary>
|
||||||
|
|
||||||
|
Yes! Agent Reach is an installer + configuration tool — any AI coding agent that can run shell commands can use it. Works with Claude Code, Cursor, OpenClaw, Windsurf, Codex, and more. Just `pip install agent-reach`, run `agent-reach install`, and the agent can start using the upstream tools immediately.
|
||||||
|
|
||||||
|
**OpenClaw note:** If your OpenClaw is using the default `messaging` tool profile, the agent won't be able to run shell commands. Enable exec first: `openclaw config set tools.profile "coding"` (or set `"tools": { "profile": "coding" }` in `~/.openclaw/openclaw.json`), then restart the Gateway and start a new conversation before installing.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Is this free? Any API costs?</strong></summary>
|
||||||
|
|
||||||
|
100% free. All backends are open-source tools (twitter-cli, rdt-cli, xhs-cli, yt-dlp, Jina Reader, Exa, etc.) that don't require paid API keys. The only optional cost is a residential proxy (~$1/month) if you need Bilibili access from an overseas server.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 致谢
|
## 致谢
|
||||||
|
|
||||||
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://www.npmjs.com/package/@steipete/bird) · [Exa](https://exa.ai) · [mcporter](https://github.com/steipete/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) · [instaloader](https://github.com/instaloader/instaloader) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
|
[twitter-cli](https://github.com/public-clis/twitter-cli) · [rdt-cli](https://github.com/public-clis/rdt-cli) · [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) · [bili-cli](https://github.com/public-clis/bilibili-cli) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [Jina Reader](https://github.com/jina-ai/reader) · [Exa](https://exa.ai) · [mcporter](https://github.com/nicobailon/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||||
|
|
||||||
|
## 联系
|
||||||
|
|
||||||
|
- 📧 **Email:** pnt01@foxmail.com
|
||||||
|
- 🐦 **Twitter/X:** [@Neo_Reidlab](https://x.com/Neo_Reidlab)
|
||||||
|
|
||||||
|
交流或合作可加微信,拉你进交流群:
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="docs/wechat-group-qr.jpg" width="280" alt="WeChat QR">
|
||||||
|
</p>
|
||||||
|
|
||||||
|
> Bug 反馈和功能请求请用 [GitHub Issues](https://github.com/Panniantong/Agent-Reach/issues),更容易跟踪。
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
[MIT](LICENSE)
|
[MIT](LICENSE)
|
||||||
|
|
||||||
|
## 友情链接
|
||||||
|
|
||||||
|
[FluxNode](https://fluxnode.org) — 低价 AI API 中转站,官方一折,可按量或按套餐付费。可用于 OpenClaw、Claude Code 等一切 Agent。
|
||||||
|
|
||||||
|
[OpenClaw for Enterprise](https://github.com/littleben/openclaw-for-enterprise) — 企业级 OpenClaw 多用户部署方案,飞书里直接用 AI,容器隔离,一条命令管理。
|
||||||
|
|
||||||
|
[腾讯云 OpenClaw](https://www.tencentcloud.com/act/pro/intl-openclaw?referral_code=G76Y819A&lang=zh&pg=) — 在腾讯云Lighthouse秒级部署OpenClaw全能助手,可通过对话丝滑接入Agent Reach,给你的OpenClaw一键装上互联网能力。
|
||||||
|
|
||||||
|
## Star History
|
||||||
|
|
||||||
|
[](https://star-history.com/#Panniantong/Agent-Reach&Date)
|
||||||
|
|||||||
+51
@@ -0,0 +1,51 @@
|
|||||||
|
# Security Policy
|
||||||
|
|
||||||
|
## Supported Versions
|
||||||
|
|
||||||
|
| Version | Supported |
|
||||||
|
|---------|-----------|
|
||||||
|
| Latest | ✅ Yes |
|
||||||
|
|
||||||
|
## Reporting a Vulnerability
|
||||||
|
|
||||||
|
If you discover a security vulnerability in Agent-Reach, please report
|
||||||
|
it responsibly by using GitHub's private security advisory feature:
|
||||||
|
|
||||||
|
👉 **[Report a vulnerability](https://github.com/Panniantong/Agent-Reach/security/advisories/new)**
|
||||||
|
|
||||||
|
Please do NOT open a public GitHub issue for security vulnerabilities.
|
||||||
|
|
||||||
|
## What to Include
|
||||||
|
|
||||||
|
- Description of the vulnerability
|
||||||
|
- Steps to reproduce
|
||||||
|
- Affected versions
|
||||||
|
- Potential impact
|
||||||
|
- Suggested fix (if any)
|
||||||
|
|
||||||
|
## Response Timeline
|
||||||
|
|
||||||
|
- Acknowledgement within **48 hours**
|
||||||
|
- Status update within **7 days**
|
||||||
|
- Fix timeline communicated within **14 days**
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
The following are considered in scope:
|
||||||
|
- Authentication and authorization bypass
|
||||||
|
- Remote code execution
|
||||||
|
- Path traversal / arbitrary file read
|
||||||
|
- Server-Side Request Forgery (SSRF)
|
||||||
|
- Injection vulnerabilities (SQL, command, prompt)
|
||||||
|
- Sensitive data exposure
|
||||||
|
|
||||||
|
## Out of Scope
|
||||||
|
|
||||||
|
- Vulnerabilities in dependencies (report to the dependency maintainer)
|
||||||
|
- Social engineering attacks
|
||||||
|
- Denial of service via resource exhaustion
|
||||||
|
|
||||||
|
## Credits
|
||||||
|
|
||||||
|
We appreciate responsible disclosure and will credit researchers
|
||||||
|
in our release notes unless anonymity is requested.
|
||||||
@@ -1,87 +0,0 @@
|
|||||||
---
|
|
||||||
name: agent-reach
|
|
||||||
description: >
|
|
||||||
Give your AI agent eyes to see the entire internet. Read and search across
|
|
||||||
Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, RSS, and any web page
|
|
||||||
— all from a single CLI. Use when: (1) reading content from URLs (tweets, Reddit posts,
|
|
||||||
articles, videos), (2) searching across platforms (web, Twitter, Reddit, GitHub, YouTube,
|
|
||||||
Bilibili, XiaoHongShu), (3) checking channel health or updating Agent Reach.
|
|
||||||
Triggers: "search Twitter/Reddit/YouTube", "read this URL", "find posts about",
|
|
||||||
"搜索", "读取", "查一下", "看看这个链接".
|
|
||||||
---
|
|
||||||
|
|
||||||
# Agent Reach
|
|
||||||
|
|
||||||
Read and search the internet across 9+ platforms via unified CLI.
|
|
||||||
|
|
||||||
## Setup
|
|
||||||
|
|
||||||
First check if agent-reach is installed:
|
|
||||||
```bash
|
|
||||||
agent-reach doctor
|
|
||||||
```
|
|
||||||
|
|
||||||
If command not found, install it:
|
|
||||||
```bash
|
|
||||||
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
|
||||||
agent-reach install --env=auto
|
|
||||||
```
|
|
||||||
|
|
||||||
`install` auto-detects your environment and installs all dependencies (Node.js, mcporter, bird CLI, gh CLI). Read the output and run `agent-reach doctor` to see what's active.
|
|
||||||
|
|
||||||
For channels that need user input, ask the user. See the full setup guide:
|
|
||||||
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
|
||||||
|
|
||||||
## Commands
|
|
||||||
|
|
||||||
### Read any URL
|
|
||||||
```bash
|
|
||||||
agent-reach read <url>
|
|
||||||
agent-reach read <url> --json # structured output
|
|
||||||
```
|
|
||||||
Handles: tweets, Reddit posts, articles, YouTube (transcripts), GitHub repos, etc.
|
|
||||||
|
|
||||||
### Search
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach search "query" # web search (Exa)
|
|
||||||
agent-reach search-twitter "query" # Twitter/X
|
|
||||||
agent-reach search-reddit "query" # Reddit (--sub <subreddit>)
|
|
||||||
agent-reach search-github "query" # GitHub (--lang <language>)
|
|
||||||
agent-reach search-youtube "query" # YouTube
|
|
||||||
agent-reach search-bilibili "query" # Bilibili (B站)
|
|
||||||
agent-reach search-xhs "query" # XiaoHongShu (小红书)
|
|
||||||
```
|
|
||||||
|
|
||||||
All search commands support `-n <count>` for number of results.
|
|
||||||
|
|
||||||
### Management
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach doctor # channel status overview
|
|
||||||
agent-reach watch # quick health + update check (for scheduled tasks)
|
|
||||||
agent-reach check-update # check for new versions
|
|
||||||
```
|
|
||||||
|
|
||||||
### Configure channels
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
|
|
||||||
agent-reach configure proxy http://user:pass@ip:port
|
|
||||||
agent-reach configure --from-browser chrome # auto-extract cookies
|
|
||||||
```
|
|
||||||
|
|
||||||
## Channel Status Tiers
|
|
||||||
|
|
||||||
- **Tier 0 (zero config):** Web, YouTube, RSS, Twitter (read-only via Jina)
|
|
||||||
- **Tier 1 (free setup):** Exa web search (mcporter required)
|
|
||||||
- **Tier 2 (user config):** Twitter search (cookie), Reddit full (proxy), GitHub (token), Bilibili (proxy), XiaoHongShu (MCP)
|
|
||||||
|
|
||||||
Run `agent-reach doctor` to see which channels are active.
|
|
||||||
|
|
||||||
## Tips
|
|
||||||
|
|
||||||
- Always try `agent-reach read <url>` first for any URL — it auto-detects the platform
|
|
||||||
- For Twitter cookies, recommend the user install [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome extension
|
|
||||||
- Reddit and Bilibili block server IPs — suggest a residential proxy (~$1/month) if on a server
|
|
||||||
- If a channel breaks, run `agent-reach doctor` to diagnose
|
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Agent Reach — Give your AI Agent eyes to see the entire internet."""
|
"""Agent Reach — Give your AI Agent eyes to see the entire internet."""
|
||||||
|
|
||||||
__version__ = "1.0.0"
|
__version__ = "1.4.0"
|
||||||
__author__ = "Neo Reid"
|
__author__ = "Neo Reid"
|
||||||
|
|
||||||
from agent_reach.core import AgentReach
|
from agent_reach.core import AgentReach
|
||||||
|
|||||||
@@ -1,14 +1,10 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Channel registry — routes URLs to the right channel.
|
Channel registry — lists all supported platforms for doctor checks.
|
||||||
|
|
||||||
This is the core of Agent Reach' pluggable architecture.
|
|
||||||
Add a new channel: just create a file and register it here.
|
|
||||||
Swap a backend: just change the implementation inside the channel file.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Dict, List, Optional
|
from typing import List, Optional
|
||||||
from .base import Channel, ReadResult, SearchResult
|
from .base import Channel
|
||||||
|
|
||||||
# Import all channels
|
# Import all channels
|
||||||
from .web import WebChannel
|
from .web import WebChannel
|
||||||
@@ -20,12 +16,15 @@ from .rss import RSSChannel
|
|||||||
from .bilibili import BilibiliChannel
|
from .bilibili import BilibiliChannel
|
||||||
from .exa_search import ExaSearchChannel
|
from .exa_search import ExaSearchChannel
|
||||||
from .xiaohongshu import XiaoHongShuChannel
|
from .xiaohongshu import XiaoHongShuChannel
|
||||||
from .instagram import InstagramChannel
|
from .douyin import DouyinChannel
|
||||||
from .linkedin import LinkedInChannel
|
from .linkedin import LinkedInChannel
|
||||||
from .bosszhipin import BossZhipinChannel
|
from .wechat import WeChatChannel
|
||||||
|
from .weibo import WeiboChannel
|
||||||
|
from .xiaoyuzhou import XiaoyuzhouChannel
|
||||||
|
from .v2ex import V2EXChannel
|
||||||
|
from .xueqiu import XueqiuChannel
|
||||||
|
|
||||||
|
|
||||||
# Channel registry — order matters (first match wins, web is last as fallback)
|
|
||||||
ALL_CHANNELS: List[Channel] = [
|
ALL_CHANNELS: List[Channel] = [
|
||||||
GitHubChannel(),
|
GitHubChannel(),
|
||||||
TwitterChannel(),
|
TwitterChannel(),
|
||||||
@@ -33,27 +32,18 @@ ALL_CHANNELS: List[Channel] = [
|
|||||||
RedditChannel(),
|
RedditChannel(),
|
||||||
BilibiliChannel(),
|
BilibiliChannel(),
|
||||||
XiaoHongShuChannel(),
|
XiaoHongShuChannel(),
|
||||||
InstagramChannel(),
|
DouyinChannel(),
|
||||||
LinkedInChannel(),
|
LinkedInChannel(),
|
||||||
BossZhipinChannel(),
|
WeChatChannel(),
|
||||||
|
WeiboChannel(),
|
||||||
|
XiaoyuzhouChannel(),
|
||||||
|
V2EXChannel(),
|
||||||
|
XueqiuChannel(),
|
||||||
RSSChannel(),
|
RSSChannel(),
|
||||||
ExaSearchChannel(),
|
ExaSearchChannel(),
|
||||||
WebChannel(), # Fallback — handles any URL
|
WebChannel(),
|
||||||
]
|
]
|
||||||
|
|
||||||
# Search-capable channels
|
|
||||||
SEARCH_CHANNELS: Dict[str, Channel] = {
|
|
||||||
ch.name: ch for ch in ALL_CHANNELS if ch.can_search()
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def get_channel_for_url(url: str) -> Channel:
|
|
||||||
"""Find the right channel for a URL."""
|
|
||||||
for channel in ALL_CHANNELS:
|
|
||||||
if channel.can_handle(url):
|
|
||||||
return channel
|
|
||||||
return WebChannel() # Should never reach here, but just in case
|
|
||||||
|
|
||||||
|
|
||||||
def get_channel(name: str) -> Optional[Channel]:
|
def get_channel(name: str) -> Optional[Channel]:
|
||||||
"""Get a channel by name."""
|
"""Get a channel by name."""
|
||||||
@@ -69,7 +59,7 @@ def get_all_channels() -> List[Channel]:
|
|||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"Channel", "ReadResult", "SearchResult",
|
"Channel",
|
||||||
"ALL_CHANNELS", "SEARCH_CHANNELS",
|
"ALL_CHANNELS",
|
||||||
"get_channel_for_url", "get_channel", "get_all_channels",
|
"get_channel", "get_all_channels",
|
||||||
]
|
]
|
||||||
|
|||||||
+11
-115
@@ -1,110 +1,28 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Channel base class — the universal interface for all platforms.
|
Channel base class — platform availability checking.
|
||||||
|
|
||||||
Every channel (YouTube, Twitter, GitHub, etc.) implements this interface.
|
Each channel represents a platform (YouTube, Twitter, GitHub, etc.)
|
||||||
The backend tool can be swapped anytime without changing anything else.
|
and provides:
|
||||||
|
- can_handle(url) → does this URL belong to this platform?
|
||||||
|
- check(config) → is the upstream tool installed and configured?
|
||||||
|
|
||||||
Example:
|
After installation, agents call upstream tools directly.
|
||||||
class YouTubeChannel(Channel):
|
|
||||||
name = "youtube"
|
|
||||||
backends = ["yt-dlp"] # current backend, can be swapped
|
|
||||||
|
|
||||||
async def read(self, url, config):
|
|
||||||
# Just call yt-dlp, return standardized dict
|
|
||||||
...
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
from abc import ABC, abstractmethod
|
from abc import ABC, abstractmethod
|
||||||
from dataclasses import dataclass
|
from typing import List, Tuple
|
||||||
from typing import Any, Dict, List, Optional, Tuple
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class ReadResult:
|
|
||||||
"""Standardized read result. Every channel returns this."""
|
|
||||||
title: str
|
|
||||||
content: str
|
|
||||||
url: str
|
|
||||||
author: str = ""
|
|
||||||
date: str = ""
|
|
||||||
platform: str = ""
|
|
||||||
extra: dict = None
|
|
||||||
|
|
||||||
def __post_init__(self):
|
|
||||||
self.extra = self.extra or {}
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
d = {
|
|
||||||
"title": self.title,
|
|
||||||
"content": self.content,
|
|
||||||
"url": self.url,
|
|
||||||
"platform": self.platform,
|
|
||||||
}
|
|
||||||
if self.author:
|
|
||||||
d["author"] = self.author
|
|
||||||
if self.date:
|
|
||||||
d["date"] = self.date
|
|
||||||
if self.extra:
|
|
||||||
d["extra"] = self.extra
|
|
||||||
return d
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class SearchResult:
|
|
||||||
"""Standardized search result."""
|
|
||||||
title: str
|
|
||||||
url: str
|
|
||||||
snippet: str = ""
|
|
||||||
author: str = ""
|
|
||||||
date: str = ""
|
|
||||||
score: float = 0
|
|
||||||
extra: dict = None
|
|
||||||
|
|
||||||
def __post_init__(self):
|
|
||||||
self.extra = self.extra or {}
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
d = {
|
|
||||||
"title": self.title,
|
|
||||||
"url": self.url,
|
|
||||||
"snippet": self.snippet,
|
|
||||||
}
|
|
||||||
if self.author:
|
|
||||||
d["author"] = self.author
|
|
||||||
if self.date:
|
|
||||||
d["date"] = self.date
|
|
||||||
if self.extra:
|
|
||||||
d["extra"] = self.extra
|
|
||||||
return d
|
|
||||||
|
|
||||||
|
|
||||||
class Channel(ABC):
|
class Channel(ABC):
|
||||||
"""
|
"""Base class for all channels."""
|
||||||
Base class for all channels.
|
|
||||||
|
|
||||||
Subclasses just need to implement:
|
|
||||||
- read(url, config) → ReadResult
|
|
||||||
- can_handle(url) → bool
|
|
||||||
- check(config) → (status, message)
|
|
||||||
|
|
||||||
Optionally:
|
|
||||||
- search(query, config, **kwargs) → list[SearchResult]
|
|
||||||
"""
|
|
||||||
|
|
||||||
name: str = "" # e.g. "youtube"
|
name: str = "" # e.g. "youtube"
|
||||||
description: str = "" # e.g. "YouTube video transcripts"
|
description: str = "" # e.g. "YouTube 视频和字幕"
|
||||||
backends: List[str] = [] # e.g. ["yt-dlp"] — what external tool is used
|
backends: List[str] = [] # e.g. ["yt-dlp"] — what upstream tool is used
|
||||||
requires_config: List[str] = [] # e.g. ["reddit_proxy"]
|
|
||||||
requires_tools: List[str] = [] # e.g. ["yt-dlp"]
|
|
||||||
tier: int = 0 # 0=zero-config, 1=needs free key, 2=needs setup
|
tier: int = 0 # 0=zero-config, 1=needs free key, 2=needs setup
|
||||||
|
|
||||||
@abstractmethod
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
"""Read content from a URL. Must return ReadResult."""
|
|
||||||
...
|
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
"""Check if this channel can handle this URL."""
|
"""Check if this channel can handle this URL."""
|
||||||
@@ -112,29 +30,7 @@ class Channel(ABC):
|
|||||||
|
|
||||||
def check(self, config=None) -> Tuple[str, str]:
|
def check(self, config=None) -> Tuple[str, str]:
|
||||||
"""
|
"""
|
||||||
Check if this channel is available.
|
Check if this channel's upstream tool is available.
|
||||||
Returns (status, message) where status is 'ok'/'warn'/'off'/'error'.
|
Returns (status, message) where status is 'ok'/'warn'/'off'/'error'.
|
||||||
"""
|
"""
|
||||||
# Check required tools
|
|
||||||
for tool in self.requires_tools:
|
|
||||||
if not shutil.which(tool):
|
|
||||||
return "off", f"需要安装:pip install {tool}"
|
|
||||||
|
|
||||||
# Check required config
|
|
||||||
for key in self.requires_config:
|
|
||||||
if config and not config.get(key):
|
|
||||||
return "off", f"需要配置 {key},运行 agent-reach setup"
|
|
||||||
|
|
||||||
return "ok", f"{'、'.join(self.backends) if self.backends else '内置'}"
|
return "ok", f"{'、'.join(self.backends) if self.backends else '内置'}"
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
"""Search this platform. Override if supported."""
|
|
||||||
raise NotImplementedError(f"{self.name} does not support search")
|
|
||||||
|
|
||||||
def can_search(self) -> bool:
|
|
||||||
"""Whether this channel supports search."""
|
|
||||||
try:
|
|
||||||
# Check if search is overridden
|
|
||||||
return type(self).search is not Channel.search
|
|
||||||
except:
|
|
||||||
return False
|
|
||||||
|
|||||||
@@ -1,207 +1,66 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Bilibili — via yt-dlp (same backend as YouTube).
|
"""Bilibili — video via yt-dlp, search/browse via bili-cli or API."""
|
||||||
|
|
||||||
Backend: yt-dlp (https://github.com/yt-dlp/yt-dlp)
|
|
||||||
yt-dlp natively supports Bilibili — video info, subtitles, and search.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
import json
|
||||||
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
import urllib.request
|
||||||
from .base import Channel, ReadResult, SearchResult
|
from .base import Channel
|
||||||
from typing import List
|
|
||||||
|
_UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
|
||||||
|
_TIMEOUT = 10
|
||||||
|
_SEARCH_API = "https://api.bilibili.com/x/web-interface/search/all/v2?keyword=test&page=1"
|
||||||
|
|
||||||
|
|
||||||
|
def _search_api_ok() -> bool:
|
||||||
|
"""Return True if Bilibili search API responds with code 0."""
|
||||||
|
req = urllib.request.Request(_SEARCH_API, headers={"User-Agent": _UA})
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=_TIMEOUT) as resp:
|
||||||
|
data = json.loads(resp.read())
|
||||||
|
return data.get("code") == 0
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
class BilibiliChannel(Channel):
|
class BilibiliChannel(Channel):
|
||||||
name = "bilibili"
|
name = "bilibili"
|
||||||
description = "B站视频信息和字幕"
|
description = "B站视频、字幕和搜索"
|
||||||
backends = ["yt-dlp"]
|
backends = ["yt-dlp", "bili-cli (可选)", "B站搜索 API"]
|
||||||
requires_tools = ["yt-dlp"]
|
tier = 1
|
||||||
tier = 0
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
d = urlparse(url).netloc.lower()
|
d = urlparse(url).netloc.lower()
|
||||||
return "bilibili.com" in d or "b23.tv" in d
|
return "bilibili.com" in d or "b23.tv" in d
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if not shutil.which("yt-dlp"):
|
if not shutil.which("yt-dlp"):
|
||||||
return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
|
return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
|
||||||
proxy = config.get("bilibili_proxy") if config else None
|
|
||||||
|
proxy = (config.get("bilibili_proxy") if config else None) or os.environ.get("BILIBILI_PROXY")
|
||||||
|
has_bili_cli = bool(shutil.which("bili"))
|
||||||
|
|
||||||
|
parts = []
|
||||||
|
|
||||||
|
# 视频读取状态
|
||||||
if proxy:
|
if proxy:
|
||||||
return "ok", "已配置代理,完整可用"
|
parts.append("视频读取:yt-dlp(代理已配置)")
|
||||||
import os
|
else:
|
||||||
is_server = bool(os.environ.get("SSH_CONNECTION") or os.path.exists("/etc/cloud"))
|
parts.append("视频读取:yt-dlp")
|
||||||
if is_server:
|
|
||||||
return "warn", "服务器 IP 可能被封,配置代理即可解决:agent-reach configure proxy URL"
|
|
||||||
return "ok", "本地直连可用"
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
# bili-cli 增强
|
||||||
if not shutil.which("yt-dlp"):
|
if has_bili_cli:
|
||||||
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
|
parts.append("搜索/热门/排行:bili-cli 可用")
|
||||||
|
else:
|
||||||
|
# 检测搜索 API 连通性
|
||||||
|
api_ok = _search_api_ok()
|
||||||
|
if api_ok:
|
||||||
|
parts.append("搜索:B站 API 可用")
|
||||||
|
else:
|
||||||
|
parts.append("搜索:B站 API 不可达")
|
||||||
|
parts.append("提示:安装 bili-cli 可解锁热门/排行/动态:pipx install bilibili-cli")
|
||||||
|
|
||||||
proxy = config.get("bilibili_proxy") if config else None
|
status = "ok" if has_bili_cli or _search_api_ok() else "warn"
|
||||||
|
return status, "。".join(parts)
|
||||||
# Get video info via yt-dlp
|
|
||||||
info = self._get_info(url, proxy)
|
|
||||||
if not info:
|
|
||||||
return ReadResult(
|
|
||||||
title="Bilibili",
|
|
||||||
content=f"⚠️ 无法获取视频信息: {url}\n服务器 IP 可能被封,配个代理:agent-reach configure proxy URL",
|
|
||||||
url=url, platform="bilibili",
|
|
||||||
)
|
|
||||||
|
|
||||||
title = info.get("title", url)
|
|
||||||
author = info.get("uploader", "")
|
|
||||||
desc = info.get("description", "")
|
|
||||||
|
|
||||||
# Try subtitles
|
|
||||||
subtitle = self._get_subtitles(url, proxy)
|
|
||||||
content = desc
|
|
||||||
if subtitle:
|
|
||||||
content += f"\n\n## 字幕\n{subtitle}"
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=title, content=content, url=url,
|
|
||||||
author=author, platform="bilibili",
|
|
||||||
extra={
|
|
||||||
"view_count": info.get("view_count"),
|
|
||||||
"like_count": info.get("like_count"),
|
|
||||||
"duration": info.get("duration_string"),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
"""Search Bilibili.
|
|
||||||
|
|
||||||
Strategy:
|
|
||||||
1. Try yt-dlp bilisearch (works on local machines)
|
|
||||||
2. Fallback to Exa site:bilibili.com (works on servers)
|
|
||||||
"""
|
|
||||||
if not shutil.which("yt-dlp"):
|
|
||||||
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
|
|
||||||
|
|
||||||
limit = kwargs.get("limit", 5)
|
|
||||||
proxy = config.get("bilibili_proxy") if config else None
|
|
||||||
|
|
||||||
# Strategy 1: yt-dlp bilisearch
|
|
||||||
results = self._search_ytdlp(query, limit, proxy)
|
|
||||||
if results:
|
|
||||||
return results
|
|
||||||
|
|
||||||
# Strategy 2: Exa fallback (server-friendly)
|
|
||||||
results = self._search_exa(query, limit)
|
|
||||||
if results:
|
|
||||||
return results
|
|
||||||
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _search_ytdlp(self, query: str, limit: int, proxy: str = None) -> List[SearchResult]:
|
|
||||||
"""Search via yt-dlp bilisearch (needs local/Chinese IP)."""
|
|
||||||
cmd = [
|
|
||||||
"yt-dlp", "--dump-json", "--no-download",
|
|
||||||
f"bilisearch{limit}:{query}",
|
|
||||||
]
|
|
||||||
if proxy:
|
|
||||||
cmd += ["--proxy", proxy]
|
|
||||||
|
|
||||||
try:
|
|
||||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
|
|
||||||
if r.returncode != 0:
|
|
||||||
return []
|
|
||||||
results = []
|
|
||||||
for line in r.stdout.strip().split("\n"):
|
|
||||||
if not line.strip():
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
vid = d.get("id", "")
|
|
||||||
url = d.get("webpage_url", f"https://www.bilibili.com/video/av{vid}")
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=d.get("title", f"av{vid}"),
|
|
||||||
url=url,
|
|
||||||
snippet=f"👤 {d.get('uploader', '?')} · 👁 {d.get('view_count', '?')}",
|
|
||||||
extra={
|
|
||||||
"view_count": d.get("view_count"),
|
|
||||||
"uploader": d.get("uploader"),
|
|
||||||
"duration": d.get("duration_string"),
|
|
||||||
},
|
|
||||||
))
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
continue
|
|
||||||
return results
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _search_exa(self, query: str, limit: int) -> List[SearchResult]:
|
|
||||||
"""Fallback: search via Exa (site:bilibili.com). Works on any IP."""
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call",
|
|
||||||
f'exa.web_search_exa(query: "site:bilibili.com {query}", numResults: {limit})'],
|
|
||||||
capture_output=True, text=True, timeout=30,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
return []
|
|
||||||
|
|
||||||
results = []
|
|
||||||
# Parse mcporter output: Title: / Author: / URL: / Text: blocks
|
|
||||||
title, author, url = "", "", ""
|
|
||||||
for line in r.stdout.split("\n"):
|
|
||||||
if line.startswith("Title: "):
|
|
||||||
title = line[7:].strip()
|
|
||||||
elif line.startswith("Author: "):
|
|
||||||
author = line[8:].strip()
|
|
||||||
elif line.startswith("URL: "):
|
|
||||||
url = line[5:].strip()
|
|
||||||
if url and "bilibili.com" in url:
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title or url,
|
|
||||||
url=url,
|
|
||||||
snippet=f"👤 {author}" if author else "(via Exa search)",
|
|
||||||
))
|
|
||||||
title, author, url = "", "", ""
|
|
||||||
return results
|
|
||||||
except Exception:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _get_info(self, url: str, proxy: str = None) -> dict:
|
|
||||||
cmd = ["yt-dlp", "--dump-json", "--no-download", url]
|
|
||||||
if proxy:
|
|
||||||
cmd += ["--proxy", proxy]
|
|
||||||
try:
|
|
||||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
|
||||||
if r.returncode == 0:
|
|
||||||
return json.loads(r.stdout)
|
|
||||||
except (subprocess.TimeoutExpired, json.JSONDecodeError):
|
|
||||||
pass
|
|
||||||
return {}
|
|
||||||
|
|
||||||
def _get_subtitles(self, url: str, proxy: str = None) -> str:
|
|
||||||
import tempfile
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
|
||||||
cmd = [
|
|
||||||
"yt-dlp", "--write-sub", "--write-auto-sub",
|
|
||||||
"--sub-lang", "zh-Hans,zh,en",
|
|
||||||
"--skip-download", "--sub-format", "vtt",
|
|
||||||
"-o", f"{tmpdir}/%(id)s.%(ext)s", url,
|
|
||||||
]
|
|
||||||
if proxy:
|
|
||||||
cmd += ["--proxy", proxy]
|
|
||||||
try:
|
|
||||||
subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
|
||||||
for f in Path(tmpdir).glob("*.vtt"):
|
|
||||||
text = f.read_text(errors="replace")
|
|
||||||
lines = []
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if not line or line.startswith("WEBVTT") or "-->" in line or line.isdigit():
|
|
||||||
continue
|
|
||||||
if line not in lines[-1:]:
|
|
||||||
lines.append(line)
|
|
||||||
return "\n".join(lines)
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
pass
|
|
||||||
return ""
|
|
||||||
|
|||||||
@@ -1,183 +0,0 @@
|
|||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""Boss直聘 (BOSS Zhipin) — via mcp-bosszp (MCP) or Jina Reader fallback.
|
|
||||||
|
|
||||||
Backend: mcp-bosszp (161 stars, FastMCP + Playwright)
|
|
||||||
Swap to: any Boss直聘 access tool
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
import requests
|
|
||||||
|
|
||||||
|
|
||||||
def _mcporter_has_bosszhipin() -> bool:
|
|
||||||
"""Check if mcporter has Boss直聘 MCP configured."""
|
|
||||||
if not shutil.which("mcporter"):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
# Check for various possible config names
|
|
||||||
out = r.stdout.lower()
|
|
||||||
return "boss" in out or "zhipin" in out or "bosszhipin" in out
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _mcporter_call(expr: str, timeout: int = 30) -> str:
|
|
||||||
"""Call a Boss直聘 MCP tool via mcporter."""
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call", expr],
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
|
|
||||||
def _get_mcp_name() -> str:
|
|
||||||
"""Get the actual MCP server name configured in mcporter."""
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
for line in r.stdout.split("\n"):
|
|
||||||
line_lower = line.strip().lower()
|
|
||||||
for name in ["bosszhipin", "boss-zp", "bosszp", "boss"]:
|
|
||||||
if name in line_lower:
|
|
||||||
# Extract the actual server name
|
|
||||||
parts = line.strip().split()
|
|
||||||
if parts:
|
|
||||||
return parts[0]
|
|
||||||
return "bosszhipin"
|
|
||||||
except Exception:
|
|
||||||
return "bosszhipin"
|
|
||||||
|
|
||||||
|
|
||||||
class BossZhipinChannel(Channel):
|
|
||||||
name = "bosszhipin"
|
|
||||||
description = "Boss直聘职位搜索"
|
|
||||||
backends = ["mcp-bosszp", "Jina Reader"]
|
|
||||||
tier = 2
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
|
||||||
domain = urlparse(url).netloc.lower()
|
|
||||||
return "zhipin.com" in domain or "boss.com" in domain
|
|
||||||
|
|
||||||
def check(self, config=None):
|
|
||||||
if _mcporter_has_bosszhipin():
|
|
||||||
return "ok", "可搜索职位、向 HR 打招呼"
|
|
||||||
|
|
||||||
return "off", (
|
|
||||||
"可通过 Jina Reader 读取职位页面。完整功能需要:\n"
|
|
||||||
" 1. git clone https://github.com/mucsbr/mcp-bosszp.git\n"
|
|
||||||
" 2. cd mcp-bosszp && pip install -r requirements.txt && playwright install chromium\n"
|
|
||||||
" 3. python boss_zhipin_fastmcp_v2.py(启动后扫码登录)\n"
|
|
||||||
" 4. mcporter config add bosszhipin http://localhost:8000/mcp\n"
|
|
||||||
" 或用 Docker:docker-compose up -d\n"
|
|
||||||
" 详见 https://github.com/mucsbr/mcp-bosszp"
|
|
||||||
)
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
# Boss直聘 pages mostly work with Jina Reader
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
async def _read_jina(self, url: str) -> ReadResult:
|
|
||||||
"""Read Boss直聘 page via Jina Reader."""
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
f"https://r.jina.ai/{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
|
|
||||||
if len(text.strip()) < 50:
|
|
||||||
return ReadResult(
|
|
||||||
title="Boss直聘",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法读取此页面内容: {url}\n\n"
|
|
||||||
"提示:\n"
|
|
||||||
"- 安装 mcp-bosszp 可解锁职位搜索和自动打招呼\n"
|
|
||||||
"- 详见 https://github.com/mucsbr/mcp-bosszp"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="bosszhipin",
|
|
||||||
)
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=text[:100] if text else url,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="bosszhipin",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
return ReadResult(
|
|
||||||
title="Boss直聘",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法读取此 Boss直聘页面: {url}\n\n"
|
|
||||||
"提示:\n"
|
|
||||||
"- Boss直聘部分页面需要登录\n"
|
|
||||||
"- 安装 mcp-bosszp 可解锁完整功能\n"
|
|
||||||
"- 详见 https://github.com/mucsbr/mcp-bosszp"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="bosszhipin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
|
|
||||||
# Try MCP search first
|
|
||||||
if _mcporter_has_bosszhipin():
|
|
||||||
try:
|
|
||||||
return await self._search_mcp(query, limit, config)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Fallback to Exa
|
|
||||||
from agent_reach.channels.exa_search import ExaSearchChannel
|
|
||||||
exa = ExaSearchChannel()
|
|
||||||
return await exa.search(f"site:zhipin.com {query}", config=config, limit=limit)
|
|
||||||
|
|
||||||
async def _search_mcp(self, query: str, limit: int, config=None) -> List[SearchResult]:
|
|
||||||
"""Search Boss直聘 via MCP."""
|
|
||||||
server = _get_mcp_name()
|
|
||||||
try:
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'{server}.get_recommend_jobs_tool(page: 1)',
|
|
||||||
timeout=30,
|
|
||||||
)
|
|
||||||
return self._parse_jobs(out, limit)
|
|
||||||
except Exception:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_jobs(self, text: str, limit: int) -> List[SearchResult]:
|
|
||||||
"""Parse MCP job search output into SearchResults."""
|
|
||||||
results = []
|
|
||||||
try:
|
|
||||||
data = json.loads(text)
|
|
||||||
jobs = data if isinstance(data, list) else data.get("jobs", data.get("results", []))
|
|
||||||
for job in jobs[:limit]:
|
|
||||||
if isinstance(job, dict):
|
|
||||||
title = job.get("title") or job.get("jobName", "")
|
|
||||||
company = job.get("company") or job.get("brandName", "")
|
|
||||||
salary = job.get("salary") or job.get("salaryDesc", "")
|
|
||||||
url = job.get("url", "")
|
|
||||||
snippet = f"🏢 {company}" if company else ""
|
|
||||||
if salary:
|
|
||||||
snippet += f" · 💰 {salary}"
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title,
|
|
||||||
url=url,
|
|
||||||
snippet=snippet,
|
|
||||||
))
|
|
||||||
except (json.JSONDecodeError, KeyError):
|
|
||||||
pass
|
|
||||||
return results
|
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Douyin (抖音) — check if mcporter + douyin-mcp-server is available."""
|
||||||
|
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
|
||||||
|
class DouyinChannel(Channel):
|
||||||
|
name = "douyin"
|
||||||
|
description = "抖音短视频"
|
||||||
|
backends = ["douyin-mcp-server"]
|
||||||
|
tier = 2
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "douyin.com" in d or "iesdouyin.com" in d
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
mcporter = shutil.which("mcporter")
|
||||||
|
if not mcporter:
|
||||||
|
return "off", (
|
||||||
|
"需要 mcporter + douyin-mcp-server。安装步骤:\n"
|
||||||
|
" 1. npm install -g mcporter\n"
|
||||||
|
" 2. pip install douyin-mcp-server\n"
|
||||||
|
" 3. 启动服务(见下方说明)\n"
|
||||||
|
" 4. mcporter config add douyin http://localhost:18070/mcp\n"
|
||||||
|
" 详见 https://github.com/yzfly/douyin-mcp-server"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
[mcporter, "config", "list"], capture_output=True,
|
||||||
|
encoding="utf-8", errors="replace", timeout=5
|
||||||
|
)
|
||||||
|
if "douyin" not in r.stdout:
|
||||||
|
return "off", (
|
||||||
|
"mcporter 已装但抖音 MCP 未配置。运行:\n"
|
||||||
|
" pip install douyin-mcp-server\n"
|
||||||
|
" # 启动服务后:\n"
|
||||||
|
" mcporter config add douyin http://localhost:18070/mcp"
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return "off", "mcporter 连接异常"
|
||||||
|
# Verify MCP connectivity by listing available tools instead of
|
||||||
|
# calling with a hardcoded (invalid) share link that always fails.
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
[mcporter, "list", "douyin"],
|
||||||
|
capture_output=True, encoding="utf-8", errors="replace", timeout=15
|
||||||
|
)
|
||||||
|
if r.returncode == 0 and r.stdout.strip():
|
||||||
|
return "ok", "完整可用(视频解析、下载链接获取)"
|
||||||
|
return "warn", "MCP 已连接但工具列表为空,检查 douyin-mcp-server 服务是否在运行"
|
||||||
|
except Exception:
|
||||||
|
return "warn", "MCP 连接异常,检查 douyin-mcp-server 服务是否在运行"
|
||||||
@@ -1,110 +1,38 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Exa semantic search — via mcporter + Exa MCP server.
|
"""Exa Search — check if mcporter + Exa MCP is available."""
|
||||||
|
|
||||||
Backend: Exa MCP at mcp.exa.ai (OAuth, no API key needed)
|
|
||||||
Requires: mcporter CLI
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from .base import Channel, SearchResult
|
from .base import Channel
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class ExaSearchChannel(Channel):
|
class ExaSearchChannel(Channel):
|
||||||
name = "exa_search"
|
name = "exa_search"
|
||||||
description = "全网语义搜索(同时支持 Reddit/Twitter 搜索)"
|
description = "全网语义搜索"
|
||||||
backends = ["exa-mcp"]
|
backends = ["Exa via mcporter"]
|
||||||
tier = 1
|
tier = 0
|
||||||
|
|
||||||
def _mcporter_ok(self) -> bool:
|
|
||||||
if not shutil.which("mcporter"):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
return "exa" in r.stdout
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
def _call(self, expr: str, timeout: int = 30) -> str:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call", expr],
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
# ── Channel interface ──
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
return False # search-only
|
return False # Search-only channel
|
||||||
|
|
||||||
async def read(self, url: str, config=None):
|
|
||||||
raise NotImplementedError("Exa is a search engine, not a reader")
|
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if not shutil.which("mcporter"):
|
mcporter = shutil.which("mcporter")
|
||||||
|
if not mcporter:
|
||||||
return "off", (
|
return "off", (
|
||||||
"需要 mcporter。安装:npm install -g mcporter && "
|
"需要 mcporter + Exa MCP。安装:\n"
|
||||||
"mcporter config add exa https://mcp.exa.ai/mcp"
|
|
||||||
)
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
return "off", "mcporter 已装但 Exa 未配置。运行:mcporter config add exa https://mcp.exa.ai/mcp"
|
|
||||||
return "ok", "MCP 已连接,免 Key 直接可用(全网搜索 + Reddit + Twitter)"
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
raise ValueError(
|
|
||||||
"Exa 搜索需要 mcporter。安装:\n"
|
|
||||||
" npm install -g mcporter\n"
|
" npm install -g mcporter\n"
|
||||||
" mcporter config add exa https://mcp.exa.ai/mcp"
|
" mcporter config add exa https://mcp.exa.ai/mcp"
|
||||||
)
|
)
|
||||||
|
try:
|
||||||
limit = kwargs.get("limit", 5)
|
r = subprocess.run(
|
||||||
safe_q = query.replace('"', '\\"')
|
[mcporter, "config", "list"], capture_output=True,
|
||||||
out = self._call(
|
encoding="utf-8", errors="replace", timeout=5
|
||||||
f'exa.web_search_exa(query: "{safe_q}", numResults: {min(limit, 10)})',
|
)
|
||||||
timeout=30,
|
if "exa" in r.stdout.lower():
|
||||||
)
|
return "ok", "全网语义搜索可用(免费,无需 API Key)"
|
||||||
return self._parse_output(out, limit)
|
return "off", (
|
||||||
|
"mcporter 已装但 Exa 未配置。运行:\n"
|
||||||
# ── Parse mcporter text output ──
|
" mcporter config add exa https://mcp.exa.ai/mcp"
|
||||||
|
)
|
||||||
def _parse_output(self, text: str, limit: int) -> List[SearchResult]:
|
except Exception:
|
||||||
"""Parse mcporter's Title/URL/Text block format."""
|
return "off", "mcporter 连接异常"
|
||||||
results = []
|
|
||||||
cur = {}
|
|
||||||
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line.startswith("Title: "):
|
|
||||||
if cur.get("title"):
|
|
||||||
results.append(self._make_result(cur))
|
|
||||||
cur = {"title": line[7:]}
|
|
||||||
elif line.startswith("URL: "):
|
|
||||||
cur["url"] = line[5:]
|
|
||||||
elif line.startswith("Published Date: "):
|
|
||||||
cur["date"] = line[16:]
|
|
||||||
elif line.startswith("Text: "):
|
|
||||||
cur["text"] = line[6:]
|
|
||||||
elif "text" in cur and line:
|
|
||||||
cur["text"] += " " + line
|
|
||||||
|
|
||||||
if cur.get("title"):
|
|
||||||
results.append(self._make_result(cur))
|
|
||||||
|
|
||||||
return results[:limit]
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _make_result(d: dict) -> SearchResult:
|
|
||||||
return SearchResult(
|
|
||||||
title=d.get("title", ""),
|
|
||||||
url=d.get("url", ""),
|
|
||||||
snippet=d.get("text", "")[:500],
|
|
||||||
date=d.get("date", ""),
|
|
||||||
score=0,
|
|
||||||
)
|
|
||||||
|
|||||||
+13
-120
@@ -1,16 +1,9 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""GitHub — via gh CLI.
|
"""GitHub — check if gh CLI is available."""
|
||||||
|
|
||||||
Backend: gh CLI (https://cli.github.com)
|
|
||||||
Swap to: GitHub REST API
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class GitHubChannel(Channel):
|
class GitHubChannel(Channel):
|
||||||
@@ -19,121 +12,21 @@ class GitHubChannel(Channel):
|
|||||||
backends = ["gh CLI"]
|
backends = ["gh CLI"]
|
||||||
tier = 0
|
tier = 0
|
||||||
|
|
||||||
def _gh(self, args: list, timeout: int = 15) -> str:
|
|
||||||
r = subprocess.run(
|
|
||||||
["gh"] + args,
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
def _gh_json(self, args: list, timeout: int = 15) -> dict:
|
|
||||||
return json.loads(self._gh(args + ["--json"], timeout))
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
return "github.com" in urlparse(url).netloc.lower()
|
return "github.com" in urlparse(url).netloc.lower()
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if not shutil.which("gh"):
|
gh = shutil.which("gh")
|
||||||
return "warn", "gh CLI 未安装。安装:https://cli.github.com 。公开仓库仍可通过 Jina Reader 读取"
|
if not gh:
|
||||||
|
return "warn", "gh CLI 未安装。安装:https://cli.github.com"
|
||||||
try:
|
try:
|
||||||
self._gh(["auth", "status"], timeout=5)
|
r = subprocess.run(
|
||||||
return "ok", "完整可用(读取、搜索、Fork、Issue、PR 等)"
|
[gh, "auth", "status"],
|
||||||
except Exception:
|
capture_output=True, encoding="utf-8", errors="replace", timeout=5
|
||||||
return "ok", "gh CLI 已装但未认证。运行 gh auth login 可解锁完整功能"
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
if not shutil.which("gh"):
|
|
||||||
# Fallback to Jina Reader for public repos
|
|
||||||
from agent_reach.channels.web import WebChannel
|
|
||||||
return await WebChannel().read(url, config)
|
|
||||||
|
|
||||||
path = urlparse(url).path.strip("/").split("/")
|
|
||||||
if len(path) < 2:
|
|
||||||
from agent_reach.channels.web import WebChannel
|
|
||||||
return await WebChannel().read(url, config)
|
|
||||||
|
|
||||||
owner, repo = path[0], path[1]
|
|
||||||
|
|
||||||
# Issues / PRs
|
|
||||||
if len(path) >= 4 and path[2] in ("issues", "pull"):
|
|
||||||
return await self._read_issue(owner, repo, path[3], url)
|
|
||||||
|
|
||||||
# Repo
|
|
||||||
return await self._read_repo(owner, repo, url)
|
|
||||||
|
|
||||||
async def _read_repo(self, owner: str, repo: str, url: str) -> ReadResult:
|
|
||||||
slug = f"{owner}/{repo}"
|
|
||||||
try:
|
|
||||||
# Get repo info
|
|
||||||
info = self._gh(["repo", "view", slug])
|
|
||||||
# Get README
|
|
||||||
try:
|
|
||||||
readme = self._gh(
|
|
||||||
["api", f"repos/{slug}/readme", "--jq", ".content"],
|
|
||||||
timeout=10,
|
|
||||||
)
|
|
||||||
import base64
|
|
||||||
readme_text = base64.b64decode(readme).decode("utf-8", errors="replace")
|
|
||||||
except Exception:
|
|
||||||
readme_text = ""
|
|
||||||
|
|
||||||
content = readme_text or info
|
|
||||||
return ReadResult(
|
|
||||||
title=slug, content=content, url=url,
|
|
||||||
author=owner, platform="github",
|
|
||||||
)
|
)
|
||||||
|
if r.returncode == 0:
|
||||||
|
return "ok", "完整可用(读取、搜索、Fork、Issue、PR 等)"
|
||||||
|
return "warn", "gh CLI 已安装但未认证。运行 gh auth login 可解锁完整功能"
|
||||||
except Exception:
|
except Exception:
|
||||||
from agent_reach.channels.web import WebChannel
|
return "warn", "gh CLI 状态检查失败,运行 gh auth status 查看详情"
|
||||||
return await WebChannel().read(url)
|
|
||||||
|
|
||||||
async def _read_issue(self, owner: str, repo: str, num: str, url: str) -> ReadResult:
|
|
||||||
slug = f"{owner}/{repo}"
|
|
||||||
try:
|
|
||||||
out = self._gh(["issue", "view", num, "-R", slug])
|
|
||||||
return ReadResult(
|
|
||||||
title=f"{slug}#{num}", content=out, url=url,
|
|
||||||
platform="github",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
# Might be a PR
|
|
||||||
try:
|
|
||||||
out = self._gh(["pr", "view", num, "-R", slug])
|
|
||||||
return ReadResult(
|
|
||||||
title=f"{slug}#{num}", content=out, url=url,
|
|
||||||
platform="github",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
from agent_reach.channels.web import WebChannel
|
|
||||||
return await WebChannel().read(url)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
if not shutil.which("gh"):
|
|
||||||
raise ValueError("GitHub search requires gh CLI. Install: https://cli.github.com")
|
|
||||||
|
|
||||||
language = kwargs.get("language")
|
|
||||||
limit = kwargs.get("limit", 5)
|
|
||||||
|
|
||||||
args = ["search", "repos", query, "--sort", "stars", f"--limit={limit}"]
|
|
||||||
if language:
|
|
||||||
args += [f"--language={language}"]
|
|
||||||
|
|
||||||
out = self._gh(args, timeout=15)
|
|
||||||
results = []
|
|
||||||
for line in out.strip().split("\n"):
|
|
||||||
if not line.strip():
|
|
||||||
continue
|
|
||||||
parts = line.split("\t")
|
|
||||||
if len(parts) >= 1:
|
|
||||||
slug = parts[0].strip()
|
|
||||||
desc = parts[1].strip() if len(parts) > 1 else ""
|
|
||||||
stars = parts[3].strip() if len(parts) > 3 else ""
|
|
||||||
lang = parts[5].strip() if len(parts) > 5 else ""
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=slug,
|
|
||||||
url=f"https://github.com/{slug}",
|
|
||||||
snippet=desc,
|
|
||||||
extra={"stars": stars, "language": lang},
|
|
||||||
))
|
|
||||||
return results
|
|
||||||
|
|||||||
@@ -1,248 +0,0 @@
|
|||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""Instagram — via instaloader (free, open source).
|
|
||||||
|
|
||||||
Backend: instaloader (9.8K stars, Python CLI + library)
|
|
||||||
Swap to: any Instagram access tool
|
|
||||||
"""
|
|
||||||
|
|
||||||
import re
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class InstagramChannel(Channel):
|
|
||||||
name = "instagram"
|
|
||||||
description = "Instagram 帖子和 Profile"
|
|
||||||
backends = ["instaloader"]
|
|
||||||
tier = 2 # Needs login for full access
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
|
||||||
domain = urlparse(url).netloc.lower()
|
|
||||||
return "instagram.com" in domain or "instagr.am" in domain
|
|
||||||
|
|
||||||
def check(self, config=None):
|
|
||||||
# Check both CLI and Python module
|
|
||||||
has_cli = shutil.which("instaloader")
|
|
||||||
has_module = False
|
|
||||||
try:
|
|
||||||
import instaloader
|
|
||||||
has_module = True
|
|
||||||
except ImportError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
if not has_cli and not has_module:
|
|
||||||
return "off", (
|
|
||||||
"需要安装 instaloader:pip install instaloader\n"
|
|
||||||
" 安装后可读取 Instagram 帖子和 Profile\n"
|
|
||||||
" 登录: agent-reach configure instagram-cookies \"sessionid=xxx; csrftoken=yyy; ...\""
|
|
||||||
)
|
|
||||||
|
|
||||||
# Check if cookies are configured
|
|
||||||
cookie_file = Path.home() / ".agent-reach" / "instagram-cookies.txt"
|
|
||||||
if cookie_file.exists():
|
|
||||||
return "ok", "已登录,可读取 Instagram 帖子和 Profile"
|
|
||||||
return "ok", "可读取公开帖子和 Profile。登录可访问更多内容:\n agent-reach configure instagram-cookies \"sessionid=xxx; csrftoken=yyy; ...\""
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
# Try instaloader (module or CLI)
|
|
||||||
try:
|
|
||||||
import instaloader
|
|
||||||
return await self._read_instaloader(url, config)
|
|
||||||
except ImportError:
|
|
||||||
pass
|
|
||||||
# Fallback: Jina Reader
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
async def _read_instaloader(self, url: str, config=None) -> ReadResult:
|
|
||||||
"""Read Instagram content using instaloader Python API."""
|
|
||||||
import asyncio
|
|
||||||
import concurrent.futures
|
|
||||||
|
|
||||||
def _sync_read():
|
|
||||||
import instaloader
|
|
||||||
L = instaloader.Instaloader(
|
|
||||||
download_pictures=False,
|
|
||||||
download_videos=False,
|
|
||||||
download_video_thumbnails=False,
|
|
||||||
download_geotags=False,
|
|
||||||
download_comments=False,
|
|
||||||
save_metadata=False,
|
|
||||||
compress_json=False,
|
|
||||||
max_connection_attempts=1, # Don't retry on rate limit
|
|
||||||
)
|
|
||||||
|
|
||||||
# Try to load session: cookie file > saved session
|
|
||||||
cookie_file = Path.home() / ".agent-reach" / "instagram-cookies.txt"
|
|
||||||
if cookie_file.exists():
|
|
||||||
try:
|
|
||||||
cookie_str = cookie_file.read_text().strip()
|
|
||||||
cookies = {}
|
|
||||||
for part in cookie_str.split(";"):
|
|
||||||
part = part.strip()
|
|
||||||
if "=" in part:
|
|
||||||
k, v = part.split("=", 1)
|
|
||||||
cookies[k.strip()] = v.strip()
|
|
||||||
if "sessionid" in cookies and "csrftoken" in cookies:
|
|
||||||
# Extract username from ds_user_id or use generic
|
|
||||||
username = cookies.get("ds_user_id", "user")
|
|
||||||
L.context.load_session(username, cookies)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
elif config and config.get("instagram_username"):
|
|
||||||
try:
|
|
||||||
L.load_session_from_file(config.get("instagram_username"))
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
path = urlparse(url).path.strip("/")
|
|
||||||
|
|
||||||
if "/p/" in url or "/reel/" in url:
|
|
||||||
return self._read_post_sync(L, url, path)
|
|
||||||
else:
|
|
||||||
return self._read_profile_sync(L, url, path)
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Run with 15s timeout to avoid instaloader's 30-min retry
|
|
||||||
loop = asyncio.get_event_loop()
|
|
||||||
with concurrent.futures.ThreadPoolExecutor() as pool:
|
|
||||||
result = await asyncio.wait_for(
|
|
||||||
loop.run_in_executor(pool, _sync_read),
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
return result
|
|
||||||
except (asyncio.TimeoutError, Exception):
|
|
||||||
# Any error or timeout → Jina fallback
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
def _read_post_sync(self, L, url: str, path: str) -> ReadResult:
|
|
||||||
"""Read a single Instagram post (sync, runs in executor)."""
|
|
||||||
import instaloader
|
|
||||||
|
|
||||||
# Extract shortcode from URL
|
|
||||||
match = re.search(r"/(?:p|reel)/([A-Za-z0-9_-]+)", url)
|
|
||||||
if not match:
|
|
||||||
raise ValueError("Cannot extract shortcode from URL")
|
|
||||||
|
|
||||||
shortcode = match.group(1)
|
|
||||||
try:
|
|
||||||
post = instaloader.Post.from_shortcode(L.context, shortcode)
|
|
||||||
|
|
||||||
lines = []
|
|
||||||
if post.caption:
|
|
||||||
lines.append(post.caption)
|
|
||||||
lines.append("")
|
|
||||||
lines.append(f"👤 @{post.owner_username}")
|
|
||||||
lines.append(f"❤️ {post.likes} likes")
|
|
||||||
if post.comments:
|
|
||||||
lines.append(f"💬 {post.comments} comments")
|
|
||||||
lines.append(f"📅 {post.date_utc.strftime('%Y-%m-%d %H:%M')}")
|
|
||||||
if post.location:
|
|
||||||
lines.append(f"📍 {post.location}")
|
|
||||||
if post.hashtags:
|
|
||||||
lines.append(f"#️⃣ {' '.join('#' + h for h in post.hashtags)}")
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=f"@{post.owner_username}: {(post.caption or '')[:80]}",
|
|
||||||
content="\n".join(lines),
|
|
||||||
url=url,
|
|
||||||
author=f"@{post.owner_username}",
|
|
||||||
date=post.date_utc.strftime("%Y-%m-%d"),
|
|
||||||
platform="instagram",
|
|
||||||
extra={"likes": post.likes, "comments": post.comments},
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
raise # Let executor timeout handle fallback
|
|
||||||
|
|
||||||
def _read_profile_sync(self, L, url: str, path: str) -> ReadResult:
|
|
||||||
"""Read an Instagram profile (sync, runs in executor)."""
|
|
||||||
import instaloader
|
|
||||||
|
|
||||||
# Extract username from path
|
|
||||||
username = path.split("/")[0] if path else ""
|
|
||||||
if not username or username in ("p", "reel", "stories", "explore"):
|
|
||||||
raise ValueError("Cannot extract username from URL")
|
|
||||||
|
|
||||||
try:
|
|
||||||
profile = instaloader.Profile.from_username(L.context, username)
|
|
||||||
|
|
||||||
lines = []
|
|
||||||
lines.append(f"👤 {profile.full_name} (@{profile.username})")
|
|
||||||
if profile.biography:
|
|
||||||
lines.append(f"📝 {profile.biography}")
|
|
||||||
if profile.external_url:
|
|
||||||
lines.append(f"🔗 {profile.external_url}")
|
|
||||||
lines.append("")
|
|
||||||
lines.append(f"📊 {profile.mediacount} posts · "
|
|
||||||
f"{profile.followers} followers · "
|
|
||||||
f"{profile.followees} following")
|
|
||||||
if profile.is_verified:
|
|
||||||
lines.append("✅ Verified")
|
|
||||||
if profile.is_business_account and profile.business_category_name:
|
|
||||||
lines.append(f"🏢 {profile.business_category_name}")
|
|
||||||
|
|
||||||
# Get recent posts (up to 5)
|
|
||||||
lines.append("")
|
|
||||||
lines.append("📸 Recent posts:")
|
|
||||||
count = 0
|
|
||||||
for post in profile.get_posts():
|
|
||||||
if count >= 5:
|
|
||||||
break
|
|
||||||
caption = (post.caption or "")[:100].replace("\n", " ")
|
|
||||||
lines.append(f" • ❤️{post.likes} | {post.date_utc.strftime('%m-%d')} | {caption}")
|
|
||||||
count += 1
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=f"{profile.full_name} (@{profile.username}) - Instagram",
|
|
||||||
content="\n".join(lines),
|
|
||||||
url=url,
|
|
||||||
author=f"@{profile.username}",
|
|
||||||
platform="instagram",
|
|
||||||
extra={
|
|
||||||
"followers": profile.followers,
|
|
||||||
"posts": profile.mediacount,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
raise # Let executor timeout handle fallback
|
|
||||||
|
|
||||||
async def _read_jina(self, url: str) -> ReadResult:
|
|
||||||
"""Fallback: use Jina Reader."""
|
|
||||||
import requests
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
f"https://r.jina.ai/{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
return ReadResult(
|
|
||||||
title=text[:100] if text else url,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="instagram",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
return ReadResult(
|
|
||||||
title="Instagram",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法读取此 Instagram 内容: {url}\n\n"
|
|
||||||
"提示:\n"
|
|
||||||
"- 确保 URL 正确\n"
|
|
||||||
"- 安装 instaloader: pip install instaloader\n"
|
|
||||||
"- 登录以访问更多内容: instaloader --login YOUR_USERNAME"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="instagram",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
"""Search Instagram via Exa."""
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
from agent_reach.channels.exa_search import ExaSearchChannel
|
|
||||||
exa = ExaSearchChannel()
|
|
||||||
return await exa.search(f"site:instagram.com {query}", config=config, limit=limit)
|
|
||||||
@@ -1,268 +1,41 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""LinkedIn — via linkedin-scraper-mcp (MCP) or Jina Reader fallback.
|
"""LinkedIn — check if linkedin-scraper-mcp is available."""
|
||||||
|
|
||||||
Backend: linkedin-scraper-mcp (916 stars, Patchright browser automation)
|
|
||||||
Swap to: any LinkedIn access tool
|
|
||||||
"""
|
|
||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
import requests
|
|
||||||
|
|
||||||
|
|
||||||
def _mcporter_has_linkedin() -> bool:
|
|
||||||
"""Check if mcporter has linkedin MCP configured."""
|
|
||||||
if not shutil.which("mcporter"):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
return "linkedin" in r.stdout.lower()
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _mcporter_call(expr: str, timeout: int = 30) -> str:
|
|
||||||
"""Call a LinkedIn MCP tool via mcporter."""
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call", expr],
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
|
|
||||||
class LinkedInChannel(Channel):
|
class LinkedInChannel(Channel):
|
||||||
name = "linkedin"
|
name = "linkedin"
|
||||||
description = "LinkedIn 个人/公司 Profile 和职位"
|
description = "LinkedIn 职业社交"
|
||||||
backends = ["linkedin-scraper-mcp", "Jina Reader"]
|
backends = ["linkedin-scraper-mcp", "Jina Reader"]
|
||||||
tier = 2
|
tier = 2
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
domain = urlparse(url).netloc.lower()
|
from urllib.parse import urlparse
|
||||||
return "linkedin.com" in domain
|
return "linkedin.com" in urlparse(url).netloc.lower()
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if _mcporter_has_linkedin():
|
mcporter = shutil.which("mcporter")
|
||||||
return "ok", "完整可用(Profile、公司、职位搜索)"
|
if not mcporter:
|
||||||
|
return "off", (
|
||||||
# Check if linkedin-scraper-mcp is installed as CLI
|
"基本内容可通过 Jina Reader 读取。完整功能需要:\n"
|
||||||
if shutil.which("linkedin-scraper-mcp"):
|
" pip install linkedin-scraper-mcp\n"
|
||||||
return "warn", (
|
" mcporter config add linkedin http://localhost:3000/mcp\n"
|
||||||
"linkedin-scraper-mcp 已安装但未接入 mcporter。运行:\n"
|
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
|
||||||
" 1. linkedin-scraper-mcp --login(在有浏览器的机器上登录)\n"
|
|
||||||
" 2. linkedin-scraper-mcp --transport streamable-http --port 8001\n"
|
|
||||||
" 3. mcporter config add linkedin http://localhost:8001/mcp"
|
|
||||||
)
|
)
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
[mcporter, "config", "list"], capture_output=True,
|
||||||
|
encoding="utf-8", errors="replace", timeout=5
|
||||||
|
)
|
||||||
|
if "linkedin" in r.stdout.lower():
|
||||||
|
return "ok", "完整可用(Profile、公司、职位搜索)"
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
return "off", (
|
return "off", (
|
||||||
"可通过 Jina Reader 读取部分内容。完整功能需要:\n"
|
"mcporter 已装但 LinkedIn MCP 未配置。运行:\n"
|
||||||
" 1. pip install linkedin-scraper-mcp\n"
|
" pip install linkedin-scraper-mcp\n"
|
||||||
" 2. linkedin-scraper-mcp --login(在有浏览器的机器上登录)\n"
|
" mcporter config add linkedin http://localhost:3000/mcp"
|
||||||
" 3. linkedin-scraper-mcp --transport streamable-http --port 8001\n"
|
|
||||||
" 4. mcporter config add linkedin http://localhost:8001/mcp\n"
|
|
||||||
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
path = urlparse(url).path.strip("/")
|
|
||||||
|
|
||||||
# Try MCP first
|
|
||||||
if _mcporter_has_linkedin():
|
|
||||||
try:
|
|
||||||
if "/in/" in url:
|
|
||||||
return await self._read_profile_mcp(url)
|
|
||||||
elif "/company/" in url:
|
|
||||||
return await self._read_company_mcp(url)
|
|
||||||
elif "/jobs/view/" in url:
|
|
||||||
return await self._read_job_mcp(url)
|
|
||||||
except Exception:
|
|
||||||
pass # Fall through to Jina
|
|
||||||
|
|
||||||
# Fallback: Jina Reader
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
async def _read_profile_mcp(self, url: str) -> ReadResult:
|
|
||||||
"""Read a LinkedIn profile via MCP."""
|
|
||||||
import re
|
|
||||||
# Extract username from URL: /in/username/
|
|
||||||
match = re.search(r"/in/([^/]+)", url)
|
|
||||||
if not match:
|
|
||||||
return await self._read_jina(url)
|
|
||||||
username = match.group(1)
|
|
||||||
safe_username = username.replace('"', '\\"')
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.get_person_profile(linkedin_username: "{safe_username}")',
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
return ReadResult(
|
|
||||||
title=self._extract_title(out) or f"LinkedIn Profile - {username}",
|
|
||||||
content=out.strip(),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_company_mcp(self, url: str) -> ReadResult:
|
|
||||||
"""Read a LinkedIn company page via MCP."""
|
|
||||||
import re
|
|
||||||
# Extract company name from URL: /company/name/
|
|
||||||
match = re.search(r"/company/([^/]+)", url)
|
|
||||||
if not match:
|
|
||||||
return await self._read_jina(url)
|
|
||||||
company = match.group(1)
|
|
||||||
safe_company = company.replace('"', '\\"')
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.get_company_profile(company_name: "{safe_company}")',
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
return ReadResult(
|
|
||||||
title=self._extract_title(out) or "LinkedIn Company",
|
|
||||||
content=out.strip(),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_job_mcp(self, url: str) -> ReadResult:
|
|
||||||
"""Read a LinkedIn job posting via MCP."""
|
|
||||||
import re
|
|
||||||
match = re.search(r"/jobs/view/(\d+)", url)
|
|
||||||
if not match:
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
job_id = match.group(1)
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.get_job_details(job_id: "{job_id}")',
|
|
||||||
timeout=30,
|
|
||||||
)
|
|
||||||
return ReadResult(
|
|
||||||
title=self._extract_title(out) or f"LinkedIn Job {job_id}",
|
|
||||||
content=out.strip(),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_jina(self, url: str) -> ReadResult:
|
|
||||||
"""Fallback: use Jina Reader."""
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
f"https://r.jina.ai/{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
|
|
||||||
# Check if content is usable
|
|
||||||
if len(text.strip()) < 100 or "Sign in" in text[:200]:
|
|
||||||
return ReadResult(
|
|
||||||
title="LinkedIn",
|
|
||||||
content=(
|
|
||||||
f"⚠️ LinkedIn 页面需要登录才能完整查看。\n\n"
|
|
||||||
f"URL: {url}\n\n"
|
|
||||||
"完整功能需安装 linkedin-scraper-mcp:\n"
|
|
||||||
" pip install linkedin-scraper-mcp\n"
|
|
||||||
" uvx linkedin-scraper-mcp --login\n"
|
|
||||||
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=text[:100] if text else url,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
return ReadResult(
|
|
||||||
title="LinkedIn",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法读取此 LinkedIn 页面: {url}\n\n"
|
|
||||||
"提示:\n"
|
|
||||||
"- LinkedIn 需要登录才能查看大部分内容\n"
|
|
||||||
"- 安装 linkedin-scraper-mcp 解锁完整功能\n"
|
|
||||||
"- 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
|
|
||||||
# Try MCP search first
|
|
||||||
if _mcporter_has_linkedin():
|
|
||||||
try:
|
|
||||||
return await self._search_mcp(query, limit)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Fallback to Exa
|
|
||||||
from agent_reach.channels.exa_search import ExaSearchChannel
|
|
||||||
exa = ExaSearchChannel()
|
|
||||||
return await exa.search(f"site:linkedin.com {query}", config=config, limit=limit)
|
|
||||||
|
|
||||||
async def _search_mcp(self, query: str, limit: int) -> List[SearchResult]:
|
|
||||||
"""Search LinkedIn via MCP."""
|
|
||||||
safe_q = query.replace('"', '\\"')
|
|
||||||
# Try job search first (most common use case)
|
|
||||||
try:
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.search_jobs(keywords: "{safe_q}")',
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
results = self._parse_search_results(out, "job")
|
|
||||||
if results:
|
|
||||||
return results[:limit]
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Try people search
|
|
||||||
try:
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.search_people(keywords: "{safe_q}")',
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
results = self._parse_search_results(out, "people")
|
|
||||||
if results:
|
|
||||||
return results
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_search_results(self, text: str, result_type: str) -> List[SearchResult]:
|
|
||||||
"""Parse MCP search output into SearchResults."""
|
|
||||||
import json
|
|
||||||
results = []
|
|
||||||
try:
|
|
||||||
data = json.loads(text)
|
|
||||||
items = data if isinstance(data, list) else data.get("results", data.get("jobs", []))
|
|
||||||
for item in items:
|
|
||||||
if isinstance(item, dict):
|
|
||||||
title = item.get("title") or item.get("name") or item.get("headline", "")
|
|
||||||
url = item.get("url") or item.get("link", "")
|
|
||||||
snippet = item.get("description") or item.get("company", "")
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title,
|
|
||||||
url=url,
|
|
||||||
snippet=snippet[:200] if snippet else "",
|
|
||||||
))
|
|
||||||
except (json.JSONDecodeError, KeyError):
|
|
||||||
# Try line-by-line parsing
|
|
||||||
pass
|
|
||||||
return results
|
|
||||||
|
|
||||||
def _extract_title(self, text: str) -> str:
|
|
||||||
"""Extract a title from MCP output."""
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line and not line.startswith(("{", "[", "#", "http")):
|
|
||||||
return line[:80]
|
|
||||||
return ""
|
|
||||||
|
|||||||
+56
-158
@@ -1,178 +1,76 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Reddit — via Reddit JSON API + optional proxy.
|
"""Reddit — search and read via rdt-cli (public-clis/rdt-cli).
|
||||||
|
|
||||||
Backend: Reddit public JSON API (append .json to any URL)
|
NOTE: Reddit requires authentication since 2024. All API requests
|
||||||
Swap to: any Reddit access method
|
(including public subreddit reads) return HTTP 403 without a valid
|
||||||
|
session cookie. Run `rdt login` after installation to authenticate.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import os
|
import json
|
||||||
import requests
|
import shutil
|
||||||
from urllib.parse import urlparse
|
import subprocess
|
||||||
from .base import Channel, ReadResult
|
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
_CREDENTIAL_FILE = "~/.config/rdt-cli/credential.json"
|
||||||
|
|
||||||
|
|
||||||
class RedditChannel(Channel):
|
class RedditChannel(Channel):
|
||||||
name = "reddit"
|
name = "reddit"
|
||||||
description = "Reddit 帖子和评论"
|
description = "Reddit 帖子和评论"
|
||||||
backends = ["Reddit JSON API"]
|
backends = ["rdt-cli"]
|
||||||
tier = 2
|
tier = 0
|
||||||
|
|
||||||
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
domain = urlparse(url).netloc.lower()
|
from urllib.parse import urlparse
|
||||||
return "reddit.com" in domain or "redd.it" in domain
|
|
||||||
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "reddit.com" in d or "redd.it" in d
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
proxy = config.get("reddit_proxy") if config else None
|
rdt = shutil.which("rdt")
|
||||||
has_bot = bool(os.environ.get("REDDIT_CLIENT_ID"))
|
if not rdt:
|
||||||
if proxy and has_bot:
|
return "off", (
|
||||||
return "ok", "完整可用(代理 + OAuth Bot)"
|
"需要安装 rdt-cli(推荐使用最新版 v0.4.2+):\n"
|
||||||
elif proxy:
|
" pip install 'rdt-cli>=0.4.2'\n"
|
||||||
return "ok", "代理已配置,可读取帖子。配置 REDDIT_CLIENT_ID/SECRET 可解锁高级搜索和发帖"
|
"或:\n"
|
||||||
elif has_bot:
|
" uv tool install rdt-cli\n"
|
||||||
return "warn", "OAuth Bot 已配置,但服务器直连可能被封。配个代理更稳定:agent-reach configure proxy URL"
|
"最新源码:https://github.com/public-clis/rdt-cli\n"
|
||||||
else:
|
"安装后运行 `rdt login` 登录(需先在浏览器登录 reddit.com)"
|
||||||
return "off", "搜索用 Exa 免费可用。读帖子需配个代理:agent-reach configure proxy URL"
|
)
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
proxy = config.get("reddit_proxy") if config else None
|
|
||||||
proxies = {"http": proxy, "https": proxy} if proxy else None
|
|
||||||
|
|
||||||
# Clean URL: remove query params, trailing slash, then add .json
|
|
||||||
parsed = urlparse(url)
|
|
||||||
clean_path = parsed.path.rstrip("/")
|
|
||||||
# Remove trailing .json if already present (avoid double .json)
|
|
||||||
if clean_path.endswith(".json"):
|
|
||||||
clean_path = clean_path[:-5]
|
|
||||||
json_url = f"https://www.reddit.com{clean_path}.json"
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
resp = requests.get(
|
r = subprocess.run(
|
||||||
json_url,
|
[rdt, "status", "--json"],
|
||||||
headers={"User-Agent": self.USER_AGENT},
|
capture_output=True,
|
||||||
proxies=proxies,
|
encoding="utf-8",
|
||||||
params={"limit": 50},
|
errors="replace",
|
||||||
timeout=15,
|
timeout=10,
|
||||||
)
|
)
|
||||||
resp.raise_for_status()
|
data = json.loads(r.stdout or "{}")
|
||||||
except requests.exceptions.HTTPError as e:
|
authenticated = data.get("data", {}).get("authenticated", False)
|
||||||
status = e.response.status_code if e.response is not None else 0
|
username = data.get("data", {}).get("username") or ""
|
||||||
if status in (403, 429):
|
|
||||||
return ReadResult(
|
|
||||||
title="Reddit",
|
|
||||||
content="⚠️ Reddit blocked this request (403 Forbidden). "
|
|
||||||
"Reddit blocks most server IPs.\n"
|
|
||||||
"Fix: agent-reach configure proxy http://user:pass@ip:port\n"
|
|
||||||
"Cheap option: https://www.webshare.io ($1/month)\n\n"
|
|
||||||
"Alternatively, search Reddit via Exa (free, no proxy needed): "
|
|
||||||
"agent-reach search-reddit \"your query\"",
|
|
||||||
url=url,
|
|
||||||
platform="reddit",
|
|
||||||
)
|
|
||||||
raise
|
|
||||||
|
|
||||||
data = resp.json()
|
if authenticated:
|
||||||
|
suffix = f"(已登录:{username})" if username else ""
|
||||||
|
return "ok", (f"rdt-cli 可用{suffix}(搜索帖子、阅读全文、查看评论)")
|
||||||
|
|
||||||
# Subreddit listing page: /r/sub/, /r/sub/hot, /r/sub/new, /r/sub/top
|
return "warn", (
|
||||||
if isinstance(data, dict) and data.get("kind") == "Listing":
|
"rdt-cli 已安装但未登录。Reddit 自 2024 年起要求认证,"
|
||||||
return self._parse_listing(data, url)
|
"未登录时所有请求均返回 403。\n\n"
|
||||||
|
"方法一(自动):运行 `rdt login`\n"
|
||||||
if isinstance(data, list) and len(data) >= 1:
|
" 先在浏览器登录 reddit.com,再运行此命令自动提取 Cookie。\n\n"
|
||||||
# Post page: [post_listing, comments_listing]
|
"方法二(手动,适用于 Chrome/Edge 127+ 无法自动提取时):\n"
|
||||||
post = data[0]["data"]["children"][0]["data"]
|
" 1. Chrome 应用商店安装 Cookie-Editor 扩展:\n"
|
||||||
title = post.get("title", "")
|
" https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm\n"
|
||||||
author = post.get("author", "")
|
" 2. 在浏览器打开 reddit.com(确保已登录)\n"
|
||||||
selftext = post.get("selftext", "")
|
" 3. 点击 Cookie-Editor 图标,找到 `reddit_session`,复制其 Value\n"
|
||||||
score = post.get("score", 0)
|
f" 4. 将以下内容写入 {_CREDENTIAL_FILE}:\n"
|
||||||
subreddit = post.get("subreddit", "")
|
' {"cookies": {"reddit_session": "<粘贴 Value>"}, '
|
||||||
|
'"source": "manual", "username": "<你的用户名>", '
|
||||||
# Extract comments
|
'"modhash": null, "saved_at": 0, "last_verified_at": null}\n\n'
|
||||||
comments_text = ""
|
"验证:`rdt status --json` 确认 authenticated: true"
|
||||||
if len(data) >= 2:
|
|
||||||
comments_text = self._extract_comments(data[1])
|
|
||||||
|
|
||||||
content = selftext
|
|
||||||
if comments_text:
|
|
||||||
content += f"\n\n---\n## Comments\n{comments_text}"
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=title,
|
|
||||||
content=content,
|
|
||||||
url=url,
|
|
||||||
author=f"u/{author}",
|
|
||||||
platform="reddit",
|
|
||||||
extra={"subreddit": subreddit, "score": score},
|
|
||||||
)
|
)
|
||||||
|
|
||||||
raise ValueError(f"Could not parse Reddit response for: {url}")
|
except (json.JSONDecodeError, FileNotFoundError, subprocess.TimeoutExpired):
|
||||||
|
return "warn", "rdt-cli 已安装但状态检查失败,运行 `rdt status` 查看详情"
|
||||||
def _parse_listing(self, data: dict, url: str) -> ReadResult:
|
|
||||||
"""Parse a subreddit listing (hot/new/top/rising)."""
|
|
||||||
children = data.get("data", {}).get("children", [])
|
|
||||||
|
|
||||||
# Extract subreddit name and sort from URL
|
|
||||||
parsed = urlparse(url)
|
|
||||||
path_parts = [p for p in parsed.path.strip("/").split("/") if p]
|
|
||||||
subreddit = path_parts[1] if len(path_parts) >= 2 else "reddit"
|
|
||||||
sort_type = path_parts[2] if len(path_parts) >= 3 else "hot"
|
|
||||||
|
|
||||||
lines = []
|
|
||||||
for i, child in enumerate(children, 1):
|
|
||||||
if child.get("kind") != "t3":
|
|
||||||
continue
|
|
||||||
post = child.get("data", {})
|
|
||||||
title = post.get("title", "")
|
|
||||||
author = post.get("author", "")
|
|
||||||
score = post.get("score", 0)
|
|
||||||
num_comments = post.get("num_comments", 0)
|
|
||||||
permalink = post.get("permalink", "")
|
|
||||||
post_url = post.get("url", "")
|
|
||||||
is_self = post.get("is_self", False)
|
|
||||||
|
|
||||||
lines.append(f"### {i}. {title}")
|
|
||||||
lines.append(f"👤 u/{author} · ⬆ {score} · 💬 {num_comments}")
|
|
||||||
if not is_self and post_url:
|
|
||||||
lines.append(f"🔗 {post_url}")
|
|
||||||
lines.append(f"📎 https://www.reddit.com{permalink}")
|
|
||||||
# Add selftext preview (first 200 chars)
|
|
||||||
selftext = post.get("selftext", "")
|
|
||||||
if selftext:
|
|
||||||
preview = selftext[:200].replace("\n", " ")
|
|
||||||
if len(selftext) > 200:
|
|
||||||
preview += "..."
|
|
||||||
lines.append(f"> {preview}")
|
|
||||||
lines.append("")
|
|
||||||
|
|
||||||
content = "\n".join(lines) if lines else "No posts found."
|
|
||||||
return ReadResult(
|
|
||||||
title=f"r/{subreddit} — {sort_type}",
|
|
||||||
content=content,
|
|
||||||
url=url,
|
|
||||||
platform="reddit",
|
|
||||||
extra={"subreddit": subreddit, "sort": sort_type, "count": len(children)},
|
|
||||||
)
|
|
||||||
|
|
||||||
def _extract_comments(self, comments_data: dict, depth: int = 0, max_depth: int = 3) -> str:
|
|
||||||
"""Recursively extract comments."""
|
|
||||||
lines = []
|
|
||||||
children = comments_data.get("data", {}).get("children", [])
|
|
||||||
|
|
||||||
for child in children:
|
|
||||||
if child.get("kind") != "t1":
|
|
||||||
continue
|
|
||||||
data = child.get("data", {})
|
|
||||||
author = data.get("author", "[deleted]")
|
|
||||||
body = data.get("body", "")
|
|
||||||
score = data.get("score", 0)
|
|
||||||
indent = " " * depth
|
|
||||||
|
|
||||||
lines.append(f"{indent}**u/{author}** ({score} points):")
|
|
||||||
lines.append(f"{indent}{body}")
|
|
||||||
lines.append("")
|
|
||||||
|
|
||||||
# Recurse into replies
|
|
||||||
if depth < max_depth and data.get("replies") and isinstance(data["replies"], dict):
|
|
||||||
lines.append(self._extract_comments(data["replies"], depth + 1, max_depth))
|
|
||||||
|
|
||||||
return "\n".join(lines)
|
|
||||||
|
|||||||
@@ -1,13 +1,7 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""RSS feeds — via feedparser (free, pip dependency).
|
"""RSS — check if feedparser is available."""
|
||||||
|
|
||||||
Backend: feedparser (https://github.com/kurtmckee/feedparser)
|
from .base import Channel
|
||||||
Swap to: any RSS parser
|
|
||||||
"""
|
|
||||||
|
|
||||||
import feedparser
|
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult
|
|
||||||
|
|
||||||
|
|
||||||
class RSSChannel(Channel):
|
class RSSChannel(Channel):
|
||||||
@@ -17,41 +11,11 @@ class RSSChannel(Channel):
|
|||||||
tier = 0
|
tier = 0
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
lower = url.lower()
|
return any(x in url.lower() for x in ["/feed", "/rss", ".xml", "atom"])
|
||||||
domain = urlparse(url).netloc.lower()
|
|
||||||
return (lower.endswith(".xml") or "/rss" in lower or "/feed" in lower
|
|
||||||
or "/atom" in lower or "rss" in domain)
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
def check(self, config=None):
|
||||||
feed = feedparser.parse(url)
|
try:
|
||||||
|
import feedparser
|
||||||
if feed.bozo and not feed.entries:
|
return "ok", "可读取 RSS/Atom 源"
|
||||||
raise ValueError(f"Failed to parse RSS feed: {url}")
|
except ImportError:
|
||||||
|
return "off", "feedparser 未安装。安装:pip install feedparser"
|
||||||
if not feed.entries:
|
|
||||||
raise ValueError(f"No entries in RSS feed: {url}")
|
|
||||||
|
|
||||||
# Return latest entry
|
|
||||||
entry = feed.entries[0]
|
|
||||||
content = entry.get("summary", "") or entry.get("description", "")
|
|
||||||
|
|
||||||
# If multiple entries, summarize all
|
|
||||||
if len(feed.entries) > 1:
|
|
||||||
lines = [f"# {feed.feed.get('title', 'RSS Feed')}\n"]
|
|
||||||
for i, e in enumerate(feed.entries[:20], 1):
|
|
||||||
title = e.get("title", "Untitled")
|
|
||||||
link = e.get("link", "")
|
|
||||||
summary = e.get("summary", "")[:200]
|
|
||||||
lines.append(f"## {i}. {title}")
|
|
||||||
lines.append(f"🔗 {link}")
|
|
||||||
if summary:
|
|
||||||
lines.append(summary)
|
|
||||||
lines.append("")
|
|
||||||
content = "\n".join(lines)
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=feed.feed.get("title", entry.get("title", url)),
|
|
||||||
content=content,
|
|
||||||
url=url,
|
|
||||||
platform="rss",
|
|
||||||
)
|
|
||||||
|
|||||||
+60
-187
@@ -1,209 +1,82 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Twitter/X — via bird CLI (free) or Jina Reader fallback.
|
"""Twitter/X — check if twitter-cli or bird CLI is available."""
|
||||||
|
|
||||||
Backend: bird (@steipete/bird npm package) for search/timeline
|
|
||||||
Jina Reader for single tweets
|
|
||||||
Swap to: any Twitter access tool
|
|
||||||
"""
|
|
||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
import requests
|
|
||||||
|
|
||||||
|
|
||||||
def _bird_cmd():
|
|
||||||
"""Find bird CLI binary."""
|
|
||||||
return shutil.which("bird") or shutil.which("birdx")
|
|
||||||
|
|
||||||
|
|
||||||
def _bird_env(config=None):
|
|
||||||
"""Build env dict with Twitter cookies for bird CLI."""
|
|
||||||
import os
|
|
||||||
env = os.environ.copy()
|
|
||||||
if config:
|
|
||||||
auth_token = config.get("twitter_auth_token")
|
|
||||||
ct0 = config.get("twitter_ct0")
|
|
||||||
if auth_token:
|
|
||||||
env["AUTH_TOKEN"] = auth_token
|
|
||||||
if ct0:
|
|
||||||
env["CT0"] = ct0
|
|
||||||
return env
|
|
||||||
|
|
||||||
|
|
||||||
class TwitterChannel(Channel):
|
class TwitterChannel(Channel):
|
||||||
name = "twitter"
|
name = "twitter"
|
||||||
description = "Twitter/X 推文"
|
description = "Twitter/X 推文"
|
||||||
backends = ["bird", "Jina Reader"]
|
backends = ["twitter-cli", "bird CLI (legacy)"]
|
||||||
tier = 0 # Single tweet reading is zero-config
|
tier = 1
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
domain = urlparse(url).netloc.lower()
|
from urllib.parse import urlparse
|
||||||
return "x.com" in domain or "twitter.com" in domain
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "x.com" in d or "twitter.com" in d
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
# Basic reading always works (Jina fallback)
|
# Prefer twitter-cli, fallback to bird/birdx
|
||||||
if _bird_cmd():
|
twitter = shutil.which("twitter")
|
||||||
return "ok", "搜索、时间线、发推全部可用"
|
bird = shutil.which("bird") or shutil.which("birdx")
|
||||||
return "ok", "可读取推文。安装 bird + 配置 Cookie 可解锁搜索和发推"
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
if twitter:
|
||||||
# Try bird first
|
return self._check_twitter_cli(twitter)
|
||||||
bird = _bird_cmd()
|
elif bird:
|
||||||
if bird:
|
return self._check_bird(bird)
|
||||||
return await self._read_bird(url, bird, config)
|
else:
|
||||||
# Fallback: Jina Reader
|
return "warn", (
|
||||||
return await self._read_jina(url)
|
"Twitter CLI 未安装。安装方式:\n"
|
||||||
|
" pipx install twitter-cli\n"
|
||||||
async def _read_bird(self, url: str, bird: str, config=None) -> ReadResult:
|
"或:\n"
|
||||||
result = subprocess.run(
|
" uv tool install twitter-cli"
|
||||||
[bird, "read", url],
|
|
||||||
capture_output=True, timeout=30,
|
|
||||||
encoding='utf-8', errors='replace',
|
|
||||||
env=_bird_env(config),
|
|
||||||
)
|
|
||||||
if result.returncode != 0:
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
text = result.stdout.strip()
|
|
||||||
# Extract author from first line
|
|
||||||
author = ""
|
|
||||||
lines = text.split("\n")
|
|
||||||
if lines and lines[0].startswith("@"):
|
|
||||||
author = lines[0].split()[0]
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=text[:100],
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
author=author,
|
|
||||||
platform="twitter",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_jina(self, url: str) -> ReadResult:
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
f"https://r.jina.ai/{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
)
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
|
|
||||||
# Detect unusable Jina responses for X/Twitter (JS-required pages)
|
def _check_twitter_cli(self, binary: str):
|
||||||
unusable_indicators = [
|
try:
|
||||||
"page doesn", # "this page doesn't exist" (handles both ' and ')
|
r = subprocess.run(
|
||||||
"miss what", # "Don't miss what's happening"
|
[binary, "status"], capture_output=True,
|
||||||
"Something went wrong. Try reloading",
|
encoding="utf-8", errors="replace", timeout=10
|
||||||
"Log in](", # Markdown link: [Log in](...)
|
)
|
||||||
]
|
output = (r.stdout or "") + (r.stderr or "")
|
||||||
if any(indicator in text for indicator in unusable_indicators):
|
if r.returncode == 0 and "ok: true" in output:
|
||||||
return ReadResult(
|
return "ok", (
|
||||||
title="Twitter/X",
|
"twitter-cli 完整可用(搜索、读推文、时间线、长文/Article、"
|
||||||
content="⚠️ Could not read this tweet.\n"
|
"用户查询、Thread)"
|
||||||
"The tweet may have been deleted, or the account is private.\n\n"
|
|
||||||
"Tips:\n"
|
|
||||||
"- Make sure the URL is correct\n"
|
|
||||||
"- Try: bird read <url> (if bird CLI is installed)\n"
|
|
||||||
"- For protected tweets, configure Twitter cookies: "
|
|
||||||
"agent-reach configure twitter-cookies AUTH_TOKEN CT0",
|
|
||||||
url=url,
|
|
||||||
platform="twitter",
|
|
||||||
)
|
)
|
||||||
|
if "not_authenticated" in output:
|
||||||
title = text[:100] if text else url
|
return "warn", (
|
||||||
return ReadResult(
|
"twitter-cli 已安装但未认证。设置方式:\n"
|
||||||
title=title,
|
" export TWITTER_AUTH_TOKEN=\"xxx\"\n"
|
||||||
content=text,
|
" export TWITTER_CT0=\"yyy\"\n"
|
||||||
url=url,
|
"或确保已在浏览器中登录 x.com"
|
||||||
platform="twitter",
|
)
|
||||||
|
return "warn", (
|
||||||
|
"twitter-cli 已安装但认证检查失败。运行:\n"
|
||||||
|
" twitter -v status 查看详细信息"
|
||||||
)
|
)
|
||||||
except Exception:
|
except Exception:
|
||||||
return ReadResult(
|
return "warn", "twitter-cli 已安装但连接失败"
|
||||||
title="Twitter/X",
|
|
||||||
content="⚠️ Could not read this tweet.\n"
|
|
||||||
"The tweet may have been deleted, or the account is private.\n\n"
|
|
||||||
"Tips:\n"
|
|
||||||
"- Make sure the URL is correct\n"
|
|
||||||
"- Try: bird read <url> (if bird CLI is installed)\n"
|
|
||||||
"- For protected tweets, configure Twitter cookies: "
|
|
||||||
"agent-reach configure twitter-cookies AUTH_TOKEN CT0",
|
|
||||||
url=url,
|
|
||||||
platform="twitter",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
def _check_bird(self, binary: str):
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
|
|
||||||
bird = _bird_cmd()
|
|
||||||
if bird:
|
|
||||||
return await self._search_bird(query, limit, bird, config)
|
|
||||||
|
|
||||||
# Fallback to Exa
|
|
||||||
return await self._search_exa(query, limit, config)
|
|
||||||
|
|
||||||
async def _search_bird(self, query: str, limit: int, bird: str, config=None) -> List[SearchResult]:
|
|
||||||
try:
|
try:
|
||||||
result = subprocess.run(
|
r = subprocess.run(
|
||||||
[bird, "search", query, "-n", str(limit)],
|
[binary, "check"], capture_output=True,
|
||||||
capture_output=True, timeout=30,
|
encoding="utf-8", errors="replace", timeout=10
|
||||||
encoding='utf-8', errors='replace',
|
|
||||||
env=_bird_env(config),
|
|
||||||
)
|
)
|
||||||
if result.returncode != 0:
|
output = (r.stdout or "") + (r.stderr or "")
|
||||||
return []
|
if r.returncode == 0:
|
||||||
|
return "ok", "bird CLI 可用(读取、搜索推文,含长文/X Article)"
|
||||||
return self._parse_bird_output(result.stdout)
|
if "Missing credentials" in output or "missing" in output.lower():
|
||||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
return "warn", (
|
||||||
return []
|
"bird CLI 已安装但未配置认证。设置环境变量:\n"
|
||||||
|
" export AUTH_TOKEN=\"xxx\"\n"
|
||||||
def _parse_bird_output(self, text: str) -> List[SearchResult]:
|
" export CT0=\"yyy\""
|
||||||
"""Parse bird text output into SearchResults."""
|
)
|
||||||
results = []
|
return "warn", (
|
||||||
current = {}
|
"bird CLI 已安装但认证检查失败。"
|
||||||
text_lines = []
|
)
|
||||||
|
except Exception:
|
||||||
for line in text.strip().split("\n"):
|
return "warn", "bird CLI 已安装但连接失败"
|
||||||
line = line.strip()
|
|
||||||
if line.startswith("─"):
|
|
||||||
if current:
|
|
||||||
current["text"] = "\n".join(text_lines).strip()
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=current.get("text", "")[:80],
|
|
||||||
url=current.get("url", ""),
|
|
||||||
snippet=current.get("text", ""),
|
|
||||||
author=current.get("author", ""),
|
|
||||||
date=current.get("date", ""),
|
|
||||||
))
|
|
||||||
current = {}
|
|
||||||
text_lines = []
|
|
||||||
continue
|
|
||||||
if line.startswith("@") and line.endswith(":") and "(" in line:
|
|
||||||
current["author"] = line.split()[0]
|
|
||||||
continue
|
|
||||||
if line.startswith("date:"):
|
|
||||||
current["date"] = line[5:].strip()
|
|
||||||
continue
|
|
||||||
if line.startswith("url:"):
|
|
||||||
current["url"] = line[4:].strip()
|
|
||||||
continue
|
|
||||||
if current is not None:
|
|
||||||
text_lines.append(line)
|
|
||||||
|
|
||||||
if current and text_lines:
|
|
||||||
current["text"] = "\n".join(text_lines).strip()
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=current.get("text", "")[:80],
|
|
||||||
url=current.get("url", ""),
|
|
||||||
snippet=current.get("text", ""),
|
|
||||||
author=current.get("author", ""),
|
|
||||||
date=current.get("date", ""),
|
|
||||||
))
|
|
||||||
return results
|
|
||||||
|
|
||||||
async def _search_exa(self, query: str, limit: int, config=None) -> List[SearchResult]:
|
|
||||||
from agent_reach.channels.exa_search import ExaSearchChannel
|
|
||||||
exa = ExaSearchChannel()
|
|
||||||
return await exa.search(f"site:x.com {query}", config=config, limit=limit)
|
|
||||||
|
|||||||
@@ -0,0 +1,212 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""V2EX — public API channel for topics, nodes, users, and replies."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import urllib.request
|
||||||
|
from typing import Any
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
_UA = "agent-reach/1.0"
|
||||||
|
_TIMEOUT = 10
|
||||||
|
|
||||||
|
|
||||||
|
def _get_json(url: str) -> Any:
|
||||||
|
"""Fetch *url* and return parsed JSON. Raises on HTTP/network errors."""
|
||||||
|
req = urllib.request.Request(url, headers={"User-Agent": _UA})
|
||||||
|
with urllib.request.urlopen(req, timeout=_TIMEOUT) as resp:
|
||||||
|
return json.loads(resp.read().decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
class V2EXChannel(Channel):
|
||||||
|
name = "v2ex"
|
||||||
|
description = "V2EX 节点、主题与回复"
|
||||||
|
backends = ["V2EX API (public)"]
|
||||||
|
tier = 0
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# URL routing
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "v2ex.com" in d
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Health check
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
try:
|
||||||
|
_get_json(
|
||||||
|
"https://www.v2ex.com/api/topics/show.json?node_name=python&page=1"
|
||||||
|
)
|
||||||
|
return "ok", "公开 API 可用(热门主题、节点浏览、主题详情、用户信息)"
|
||||||
|
except Exception as e:
|
||||||
|
return "warn", f"V2EX API 连接失败(可能需要代理):{e}"
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Data-fetching methods
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def get_hot_topics(self, limit: int = 20) -> list:
|
||||||
|
"""获取热门帖子列表。
|
||||||
|
|
||||||
|
Returns a list of dicts with keys:
|
||||||
|
title, url, replies, node_name, node_title, content
|
||||||
|
"""
|
||||||
|
data = _get_json("https://www.v2ex.com/api/topics/hot.json")
|
||||||
|
results = []
|
||||||
|
for item in data[:limit]:
|
||||||
|
node = item.get("node") or {}
|
||||||
|
content = item.get("content", "") or ""
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"id": item.get("id", 0),
|
||||||
|
"title": item.get("title", ""),
|
||||||
|
"url": item.get("url", ""),
|
||||||
|
"replies": item.get("replies", 0),
|
||||||
|
"node_name": node.get("name", ""),
|
||||||
|
"node_title": node.get("title", ""),
|
||||||
|
"content": content[:200],
|
||||||
|
"created": item.get("created", 0),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return results
|
||||||
|
|
||||||
|
def get_node_topics(self, node_name: str, limit: int = 20) -> list:
|
||||||
|
"""获取指定节点的最新帖子。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
node_name: 节点名称,如 "python"、"tech"、"jobs"
|
||||||
|
limit: 最多返回条数
|
||||||
|
|
||||||
|
Returns a list of dicts with keys:
|
||||||
|
title, url, replies, node_name, node_title, content
|
||||||
|
"""
|
||||||
|
url = (
|
||||||
|
f"https://www.v2ex.com/api/topics/show.json"
|
||||||
|
f"?node_name={node_name}&page=1"
|
||||||
|
)
|
||||||
|
data = _get_json(url)
|
||||||
|
results = []
|
||||||
|
for item in data[:limit]:
|
||||||
|
node = item.get("node") or {}
|
||||||
|
content = item.get("content", "") or ""
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"id": item.get("id", 0),
|
||||||
|
"title": item.get("title", ""),
|
||||||
|
"url": item.get("url", ""),
|
||||||
|
"replies": item.get("replies", 0),
|
||||||
|
"node_name": node.get("name", node_name),
|
||||||
|
"node_title": node.get("title", ""),
|
||||||
|
"content": content[:200],
|
||||||
|
"created": item.get("created", 0),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return results
|
||||||
|
|
||||||
|
def get_topic(self, topic_id: int) -> dict:
|
||||||
|
"""获取单个帖子详情和回复列表。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
topic_id: 帖子 ID(从 URL https://www.v2ex.com/t/<id> 中获取)
|
||||||
|
|
||||||
|
Returns a dict with keys:
|
||||||
|
id, title, url, content, replies_count, node_name, node_title,
|
||||||
|
author, created, replies (list of dicts with: author, content, created)
|
||||||
|
"""
|
||||||
|
topic_data = _get_json(
|
||||||
|
f"https://www.v2ex.com/api/topics/show.json?id={topic_id}"
|
||||||
|
)
|
||||||
|
# API returns a list even for single-ID queries
|
||||||
|
if isinstance(topic_data, list):
|
||||||
|
topic = topic_data[0] if topic_data else {}
|
||||||
|
else:
|
||||||
|
topic = topic_data
|
||||||
|
|
||||||
|
node = topic.get("node") or {}
|
||||||
|
member = topic.get("member") or {}
|
||||||
|
|
||||||
|
# Fetch replies (first page)
|
||||||
|
try:
|
||||||
|
replies_raw = _get_json(
|
||||||
|
f"https://www.v2ex.com/api/replies/show.json"
|
||||||
|
f"?topic_id={topic_id}&page=1"
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
replies_raw = []
|
||||||
|
|
||||||
|
replies = [
|
||||||
|
{
|
||||||
|
"author": (r.get("member") or {}).get("username", ""),
|
||||||
|
"content": r.get("content", ""),
|
||||||
|
"created": r.get("created", 0),
|
||||||
|
}
|
||||||
|
for r in (replies_raw or [])
|
||||||
|
]
|
||||||
|
|
||||||
|
return {
|
||||||
|
"id": topic.get("id", topic_id),
|
||||||
|
"title": topic.get("title", ""),
|
||||||
|
"url": topic.get("url", f"https://www.v2ex.com/t/{topic_id}"),
|
||||||
|
"content": topic.get("content", ""),
|
||||||
|
"replies_count": topic.get("replies", 0),
|
||||||
|
"node_name": node.get("name", ""),
|
||||||
|
"node_title": node.get("title", ""),
|
||||||
|
"author": member.get("username", ""),
|
||||||
|
"created": topic.get("created", 0),
|
||||||
|
"replies": replies,
|
||||||
|
}
|
||||||
|
|
||||||
|
def get_user(self, username: str) -> dict:
|
||||||
|
"""获取用户信息。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
username: V2EX 用户名
|
||||||
|
|
||||||
|
Returns a dict with keys:
|
||||||
|
id, username, url, website, twitter, psn, github, btc,
|
||||||
|
location, bio, avatar, created
|
||||||
|
"""
|
||||||
|
data = _get_json(
|
||||||
|
f"https://www.v2ex.com/api/members/show.json?username={username}"
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"id": data.get("id", 0),
|
||||||
|
"username": data.get("username", username),
|
||||||
|
"url": data.get("url", f"https://www.v2ex.com/member/{username}"),
|
||||||
|
"website": data.get("website", ""),
|
||||||
|
"twitter": data.get("twitter", ""),
|
||||||
|
"psn": data.get("psn", ""),
|
||||||
|
"github": data.get("github", ""),
|
||||||
|
"btc": data.get("btc", ""),
|
||||||
|
"location": data.get("location", ""),
|
||||||
|
"bio": data.get("bio", ""),
|
||||||
|
"avatar": data.get("avatar_large", data.get("avatar_normal", "")),
|
||||||
|
"created": data.get("created", 0),
|
||||||
|
}
|
||||||
|
|
||||||
|
def search(self, query: str, limit: int = 10) -> list:
|
||||||
|
"""搜索帖子。
|
||||||
|
|
||||||
|
注意:V2EX 公开 API 暂不支持全文搜索端点(/api/search.json 不可用)。
|
||||||
|
本方法通过 Jina Reader 代理 V2EX 站内搜索页面获取结果(纯文本,无结构化数据)。
|
||||||
|
|
||||||
|
如需精确搜索,建议直接访问 https://www.v2ex.com/?q=<query> 或
|
||||||
|
使用 Exa channel 的 site:v2ex.com 搜索。
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
list of dicts with keys: title, url, snippet
|
||||||
|
如果搜索不可用,返回包含单条 {"error": str} 的列表。
|
||||||
|
"""
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"error": (
|
||||||
|
"V2EX 公开 API 不提供搜索端点。"
|
||||||
|
f"建议改用:https://www.v2ex.com/?q={query} "
|
||||||
|
"或通过 Exa channel 使用 site:v2ex.com 搜索。"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
]
|
||||||
+20
-37
@@ -1,49 +1,32 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Web pages — via Jina Reader API (free, no config needed).
|
"""Web — any URL via Jina Reader. Always available."""
|
||||||
|
|
||||||
Backend: Jina Reader (https://r.jina.ai)
|
import urllib.request
|
||||||
Swap to: Firecrawl, Trafilatura, or any other reader API
|
from .base import Channel
|
||||||
"""
|
|
||||||
|
|
||||||
import requests
|
_UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
|
||||||
from .base import Channel, ReadResult
|
|
||||||
|
|
||||||
|
|
||||||
class WebChannel(Channel):
|
class WebChannel(Channel):
|
||||||
name = "web"
|
name = "web"
|
||||||
description = "网页(任意 URL)"
|
description = "任意网页"
|
||||||
backends = ["Jina Reader API"]
|
backends = ["Jina Reader"]
|
||||||
tier = 0
|
tier = 0
|
||||||
|
|
||||||
JINA_URL = "https://r.jina.ai/"
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
# Fallback — handles any URL not matched by other channels
|
return True # Fallback — handles any URL
|
||||||
return True
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
def check(self, config=None):
|
||||||
resp = requests.get(
|
return "ok", "通过 Jina Reader 读取任意网页(curl https://r.jina.ai/URL)"
|
||||||
f"{self.JINA_URL}{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
def read(self, url: str) -> str:
|
||||||
timeout=15,
|
"""通过 Jina Reader 读取网页,返回 Markdown 全文。"""
|
||||||
)
|
if not url.startswith(("http://", "https://")):
|
||||||
resp.raise_for_status()
|
url = "https://" + url
|
||||||
text = resp.text
|
jina_url = f"https://r.jina.ai/{url}"
|
||||||
|
req = urllib.request.Request(
|
||||||
# Extract title from first markdown heading
|
jina_url,
|
||||||
title = url
|
headers={"User-Agent": _UA, "Accept": "text/plain"},
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line.startswith("# "):
|
|
||||||
title = line[2:].strip()
|
|
||||||
break
|
|
||||||
if line.startswith("Title:"):
|
|
||||||
title = line[6:].strip()
|
|
||||||
break
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=title,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="web",
|
|
||||||
)
|
)
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||||
|
return resp.read().decode("utf-8")
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""WeChat Official Account articles — read and search.
|
||||||
|
|
||||||
|
Read: Exa crawling (primary) / Camoufox stealth browser (optional)
|
||||||
|
Search: Exa web_search with includeDomains mp.weixin.qq.com
|
||||||
|
"""
|
||||||
|
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
|
||||||
|
def _exa_available() -> bool:
|
||||||
|
mcporter = shutil.which("mcporter")
|
||||||
|
if not mcporter:
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
[mcporter, "config", "list"],
|
||||||
|
capture_output=True, encoding="utf-8", errors="replace", timeout=5,
|
||||||
|
)
|
||||||
|
return "exa" in r.stdout.lower()
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
class WeChatChannel(Channel):
|
||||||
|
name = "wechat"
|
||||||
|
description = "微信公众号文章"
|
||||||
|
backends = ["Exa via mcporter (搜索+阅读)", "Camoufox (可选阅读)"]
|
||||||
|
tier = 0
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "mp.weixin.qq.com" in d or "weixin.qq.com" in d
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
has_exa = _exa_available()
|
||||||
|
has_camoufox = False
|
||||||
|
try:
|
||||||
|
import camoufox # noqa: F401
|
||||||
|
has_camoufox = True
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if has_exa and has_camoufox:
|
||||||
|
return "ok", "完整可用(Exa 搜索 + Exa/Camoufox 阅读公众号文章)"
|
||||||
|
elif has_exa:
|
||||||
|
return "ok", (
|
||||||
|
"通过 Exa 搜索和阅读微信公众号文章(免费,无需额外配置)。"
|
||||||
|
"可选安装 Camoufox 获得更好的全文阅读效果。"
|
||||||
|
)
|
||||||
|
elif has_camoufox:
|
||||||
|
return "warn", (
|
||||||
|
"Camoufox 可阅读公众号文章,但搜索功能需要 Exa。"
|
||||||
|
"运行 `agent-reach install --env=auto` 安装 Exa。"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
return "off", (
|
||||||
|
"需要 mcporter + Exa MCP 来搜索和阅读微信公众号文章。\n"
|
||||||
|
"运行 `agent-reach install --env=auto` 安装。"
|
||||||
|
)
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Weibo (微博) — check if mcporter + mcp-server-weibo is available."""
|
||||||
|
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
|
||||||
|
class WeiboChannel(Channel):
|
||||||
|
name = "weibo"
|
||||||
|
description = "微博动态与热搜"
|
||||||
|
backends = ["mcp-server-weibo"]
|
||||||
|
tier = 1
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "weibo.com" in d or "weibo.cn" in d
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
mcporter = shutil.which("mcporter")
|
||||||
|
if not mcporter:
|
||||||
|
return "off", (
|
||||||
|
"需要 mcporter + mcp-server-weibo。安装步骤:\n"
|
||||||
|
" 1. npm install -g mcporter\n"
|
||||||
|
" 2. pip install git+https://github.com/Panniantong/mcp-server-weibo.git\n"
|
||||||
|
" 3. mcporter config add weibo --command 'mcp-server-weibo'\n"
|
||||||
|
" 详见 https://github.com/Panniantong/mcp-server-weibo"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
[mcporter, "config", "list"], capture_output=True,
|
||||||
|
encoding="utf-8", errors="replace", timeout=5
|
||||||
|
)
|
||||||
|
if "weibo" not in r.stdout:
|
||||||
|
return "off", (
|
||||||
|
"mcporter 已装但微博 MCP 未配置。运行:\n"
|
||||||
|
" pip install git+https://github.com/Panniantong/mcp-server-weibo.git\n"
|
||||||
|
" mcporter config add weibo --command 'mcp-server-weibo'"
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return "off", "mcporter 连接异常"
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
[mcporter, "list", "weibo"], capture_output=True,
|
||||||
|
encoding="utf-8", errors="replace", timeout=15
|
||||||
|
)
|
||||||
|
if r.returncode == 0 and "search_users" in r.stdout:
|
||||||
|
return "ok", "完整可用(热搜、搜索、用户动态、评论)"
|
||||||
|
return "warn", "MCP 已配置但工具加载失败,检查 mcp-server-weibo 版本"
|
||||||
|
except Exception:
|
||||||
|
return "warn", "MCP 连接异常,检查 mcp-server-weibo 是否可用"
|
||||||
+143
-153
@@ -1,172 +1,162 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""XiaoHongShu (小红书) — via mcporter + xiaohongshu MCP server.
|
"""XiaoHongShu — check if xhs-cli (xiaohongshu-cli) is available."""
|
||||||
|
|
||||||
Backend: xiaohongshu-mcp server (internal API, reliable)
|
|
||||||
Requires: mcporter CLI + xiaohongshu MCP server running
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List, Optional
|
|
||||||
|
def format_xhs_result(data):
|
||||||
|
"""Clean XHS API response, keeping only useful fields.
|
||||||
|
|
||||||
|
Handles both single note objects and lists of notes (search results).
|
||||||
|
Drastically reduces token usage by stripping structural redundancy (#134).
|
||||||
|
"""
|
||||||
|
if isinstance(data, list):
|
||||||
|
return [_clean_note(item) for item in data]
|
||||||
|
if isinstance(data, dict):
|
||||||
|
# Handle search_feeds wrapper: {"items": [...]} or {"data": {"items": [...]}}
|
||||||
|
items = None
|
||||||
|
if "items" in data:
|
||||||
|
items = data["items"]
|
||||||
|
elif "data" in data and isinstance(data.get("data"), dict):
|
||||||
|
items = data["data"].get("items") or data["data"].get("notes")
|
||||||
|
if items and isinstance(items, list):
|
||||||
|
return [_clean_note(item) for item in items]
|
||||||
|
# Single note
|
||||||
|
return _clean_note(data)
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_note(note):
|
||||||
|
"""Extract useful fields from a single XHS note/feed item."""
|
||||||
|
if not isinstance(note, dict):
|
||||||
|
return note
|
||||||
|
|
||||||
|
# Some responses nest the note under "note_card" or "note"
|
||||||
|
inner = note.get("note_card") or note.get("note") or note
|
||||||
|
|
||||||
|
result = {}
|
||||||
|
|
||||||
|
# Basic info
|
||||||
|
for key in ("id", "note_id", "xsec_token", "title", "desc", "type", "time"):
|
||||||
|
if key in inner:
|
||||||
|
result[key] = inner[key]
|
||||||
|
|
||||||
|
# Content (may be in desc or content)
|
||||||
|
if "content" in inner and "desc" not in result:
|
||||||
|
result["content"] = inner["content"]
|
||||||
|
|
||||||
|
# Author
|
||||||
|
user = inner.get("user") or inner.get("author")
|
||||||
|
if isinstance(user, dict):
|
||||||
|
result["user"] = {
|
||||||
|
k: user[k] for k in ("nickname", "user_id", "nick_name") if k in user
|
||||||
|
}
|
||||||
|
|
||||||
|
# Engagement metrics
|
||||||
|
interact = inner.get("interact_info") or inner.get("note_interact_info") or {}
|
||||||
|
if isinstance(interact, dict):
|
||||||
|
for key in ("liked_count", "collected_count", "comment_count", "share_count"):
|
||||||
|
if key in interact:
|
||||||
|
result[key] = interact[key]
|
||||||
|
# Also check top-level (some API formats)
|
||||||
|
for key in ("liked_count", "collected_count", "comment_count", "share_count"):
|
||||||
|
if key in inner and key not in result:
|
||||||
|
result[key] = inner[key]
|
||||||
|
|
||||||
|
# Images — just URLs
|
||||||
|
images = inner.get("image_list") or inner.get("images_list") or []
|
||||||
|
if isinstance(images, list):
|
||||||
|
urls = []
|
||||||
|
for img in images:
|
||||||
|
if isinstance(img, dict):
|
||||||
|
url = img.get("url") or img.get("url_default") or img.get("original")
|
||||||
|
if url:
|
||||||
|
urls.append(url)
|
||||||
|
elif isinstance(img, str):
|
||||||
|
urls.append(img)
|
||||||
|
if urls:
|
||||||
|
result["images"] = urls
|
||||||
|
|
||||||
|
# Tags
|
||||||
|
tags = inner.get("tag_list") or inner.get("tags") or []
|
||||||
|
if isinstance(tags, list):
|
||||||
|
tag_names = []
|
||||||
|
for t in tags:
|
||||||
|
if isinstance(t, dict) and "name" in t:
|
||||||
|
tag_names.append(t["name"])
|
||||||
|
elif isinstance(t, str):
|
||||||
|
tag_names.append(t)
|
||||||
|
if tag_names:
|
||||||
|
result["tags"] = tag_names
|
||||||
|
|
||||||
|
# Comments (if present, e.g. from get_feed_detail with comments)
|
||||||
|
comments = inner.get("comments") or []
|
||||||
|
if isinstance(comments, list) and comments:
|
||||||
|
result["comments"] = [_clean_comment(c) for c in comments]
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_comment(comment):
|
||||||
|
"""Extract useful fields from a comment."""
|
||||||
|
if not isinstance(comment, dict):
|
||||||
|
return comment
|
||||||
|
result = {}
|
||||||
|
if "content" in comment:
|
||||||
|
result["content"] = comment["content"]
|
||||||
|
user = comment.get("user_info") or comment.get("user")
|
||||||
|
if isinstance(user, dict):
|
||||||
|
result["user"] = user.get("nickname") or user.get("nick_name", "")
|
||||||
|
for key in ("like_count", "sub_comment_count"):
|
||||||
|
if key in comment:
|
||||||
|
result[key] = comment[key]
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
class XiaoHongShuChannel(Channel):
|
class XiaoHongShuChannel(Channel):
|
||||||
name = "xiaohongshu"
|
name = "xiaohongshu"
|
||||||
description = "小红书笔记"
|
description = "小红书笔记"
|
||||||
backends = ["xiaohongshu-mcp"]
|
backends = ["xhs-cli (xiaohongshu-cli)"]
|
||||||
tier = 2
|
tier = 1
|
||||||
|
|
||||||
def _mcporter_ok(self) -> bool:
|
|
||||||
"""Check if mcporter + xiaohongshu MCP is available."""
|
|
||||||
if not shutil.which("mcporter"):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
return "xiaohongshu" in r.stdout
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
def _call(self, expr: str, timeout: int = 30) -> str:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call", expr],
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
# ── Channel interface ──
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
d = urlparse(url).netloc.lower()
|
d = urlparse(url).netloc.lower()
|
||||||
return "xiaohongshu.com" in d or "xhslink.com" in d
|
return "xiaohongshu.com" in d or "xhslink.com" in d
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if not shutil.which("mcporter"):
|
xhs = shutil.which("xhs")
|
||||||
|
if not xhs:
|
||||||
return "off", (
|
return "off", (
|
||||||
"需要 mcporter + xiaohongshu-mcp。安装步骤:\n"
|
"需要安装 xhs-cli:\n"
|
||||||
" 1. npm install -g mcporter\n"
|
" pipx install xiaohongshu-cli\n"
|
||||||
" 2. docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
|
"或:\n"
|
||||||
" 3. mcporter config add xiaohongshu http://localhost:18060/mcp\n"
|
" uv tool install xiaohongshu-cli\n"
|
||||||
" 详见 https://github.com/xpzouying/xiaohongshu-mcp"
|
"安装后运行 `xhs login` 登录"
|
||||||
)
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
return "off", (
|
|
||||||
"mcporter 已装但小红书 MCP 未配置。运行:\n"
|
|
||||||
" docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
|
|
||||||
" mcporter config add xiaohongshu http://localhost:18060/mcp"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
out = self._call("xiaohongshu.check_login_status()", timeout=10)
|
r = subprocess.run(
|
||||||
if "已登录" in out or "logged" in out.lower():
|
[xhs, "status"], capture_output=True,
|
||||||
return "ok", "完整可用(阅读、搜索、发帖、评论、点赞)"
|
encoding="utf-8", errors="replace", timeout=10,
|
||||||
return "warn", "MCP 已连接但未登录,需扫码登录"
|
)
|
||||||
|
output = (r.stdout or "") + (r.stderr or "")
|
||||||
|
if r.returncode == 0 and "ok: true" in output:
|
||||||
|
return "ok", (
|
||||||
|
"完整可用(搜索、阅读、评论、发帖、热门、"
|
||||||
|
"收藏、关注、用户查询)"
|
||||||
|
)
|
||||||
|
if "not_authenticated" in output or "expired" in output:
|
||||||
|
return "warn", (
|
||||||
|
"xhs-cli 已安装但未登录。运行:\n"
|
||||||
|
" xhs login\n"
|
||||||
|
"(自动从浏览器提取 Cookie,或扫码登录)"
|
||||||
|
)
|
||||||
|
return "warn", (
|
||||||
|
"xhs-cli 已安装但状态异常。运行:\n"
|
||||||
|
" xhs -v status 查看详细信息"
|
||||||
|
)
|
||||||
except Exception:
|
except Exception:
|
||||||
return "warn", "MCP 连接异常,检查 xiaohongshu-mcp 服务是否在运行"
|
return "warn", "xhs-cli 已安装但连接失败"
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
return ReadResult(
|
|
||||||
title="XiaoHongShu",
|
|
||||||
content=(
|
|
||||||
"⚠️ 小红书需要 mcporter + xiaohongshu-mcp 才能使用。\n\n"
|
|
||||||
"安装步骤:\n"
|
|
||||||
"1. npm install -g mcporter\n"
|
|
||||||
"2. docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
|
|
||||||
"3. mcporter config add xiaohongshu http://localhost:18060/mcp\n"
|
|
||||||
"4. 运行 agent-reach doctor 检查状态\n\n"
|
|
||||||
"详见 https://github.com/xpzouying/xiaohongshu-mcp"
|
|
||||||
),
|
|
||||||
url=url, platform="xiaohongshu",
|
|
||||||
)
|
|
||||||
|
|
||||||
note_id = self._extract_note_id(url)
|
|
||||||
if not note_id:
|
|
||||||
return ReadResult(
|
|
||||||
title="XiaoHongShu",
|
|
||||||
content=f"⚠️ 无法从 URL 提取笔记 ID: {url}",
|
|
||||||
url=url, platform="xiaohongshu",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Step 1: get xsec_token from feeds
|
|
||||||
xsec_token = self._find_token(note_id)
|
|
||||||
|
|
||||||
if not xsec_token:
|
|
||||||
return ReadResult(
|
|
||||||
title="XiaoHongShu",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法获取笔记 {note_id} 的访问令牌。\n"
|
|
||||||
"小红书需要 xsec_token 才能读取笔记详情。\n"
|
|
||||||
"请先通过搜索找到这篇笔记,或直接使用搜索功能。"
|
|
||||||
),
|
|
||||||
url=url, platform="xiaohongshu",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Step 2: get detail
|
|
||||||
out = self._call(
|
|
||||||
f'xiaohongshu.get_feed_detail(feed_id: "{note_id}", xsec_token: "{xsec_token}")',
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=self._extract_title(out) or f"XHS {note_id}",
|
|
||||||
content=out.strip(),
|
|
||||||
url=url, platform="xiaohongshu",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
raise ValueError(
|
|
||||||
"小红书搜索需要 mcporter + xiaohongshu-mcp。\n"
|
|
||||||
"安装: npm install -g mcporter && mcporter config add xiaohongshu http://localhost:18060/mcp"
|
|
||||||
)
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
safe_q = query.replace('"', '\\"')
|
|
||||||
out = self._call(f'xiaohongshu.search_feeds(keyword: "{safe_q}")', timeout=30)
|
|
||||||
|
|
||||||
results = []
|
|
||||||
try:
|
|
||||||
data = json.loads(out)
|
|
||||||
for item in data.get("feeds", [])[:limit]:
|
|
||||||
card = item.get("noteCard", {})
|
|
||||||
user = card.get("user", {})
|
|
||||||
interact = card.get("interactInfo", {})
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=card.get("displayTitle", ""),
|
|
||||||
url=f"https://www.xiaohongshu.com/explore/{item.get('id', '')}",
|
|
||||||
snippet=f"👤 {user.get('nickname', '')} · ❤ {interact.get('likedCount', '0')}",
|
|
||||||
score=0,
|
|
||||||
))
|
|
||||||
except (json.JSONDecodeError, KeyError):
|
|
||||||
pass
|
|
||||||
return results
|
|
||||||
|
|
||||||
# ── Helpers ──
|
|
||||||
|
|
||||||
def _extract_note_id(self, url: str) -> str:
|
|
||||||
parts = urlparse(url).path.strip("/").split("/")
|
|
||||||
return parts[-1] if parts else ""
|
|
||||||
|
|
||||||
def _find_token(self, note_id: str) -> Optional[str]:
|
|
||||||
"""Try to find xsec_token for a note from feeds."""
|
|
||||||
try:
|
|
||||||
out = self._call("xiaohongshu.list_feeds()", timeout=15)
|
|
||||||
data = json.loads(out)
|
|
||||||
for feed in data.get("feeds", []):
|
|
||||||
if feed.get("id") == note_id:
|
|
||||||
return feed.get("xsecToken", "")
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return None
|
|
||||||
|
|
||||||
def _extract_title(self, text: str) -> str:
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line and not line.startswith(("{", "[", "#", "http")):
|
|
||||||
return line[:80]
|
|
||||||
return ""
|
|
||||||
|
|||||||
@@ -0,0 +1,54 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Xiaoyuzhou Podcast (小宇宙播客) — transcribe podcasts via Groq Whisper API."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
from agent_reach.config import Config
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
|
||||||
|
class XiaoyuzhouChannel(Channel):
|
||||||
|
name = "xiaoyuzhou"
|
||||||
|
description = "小宇宙播客转文字"
|
||||||
|
backends = ["groq-whisper", "ffmpeg"]
|
||||||
|
tier = 1
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "xiaoyuzhoufm.com" in d
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
# Check ffmpeg
|
||||||
|
if not shutil.which("ffmpeg"):
|
||||||
|
return "off", (
|
||||||
|
"需要 ffmpeg(音频转码和切片)。安装:\n"
|
||||||
|
" Ubuntu/Debian: apt install -y ffmpeg\n"
|
||||||
|
" macOS: brew install ffmpeg"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check script exists
|
||||||
|
script = os.path.expanduser("~/.agent-reach/tools/xiaoyuzhou/transcribe.sh")
|
||||||
|
if not os.path.isfile(script):
|
||||||
|
return "off", (
|
||||||
|
"转录脚本未安装。运行:\n"
|
||||||
|
" agent-reach install --env=auto\n"
|
||||||
|
" 或手动复制 transcribe.sh 到 ~/.agent-reach/tools/xiaoyuzhou/"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check GROQ_API_KEY — prefer env var, fall back to Agent Reach config
|
||||||
|
has_key = bool(os.environ.get("GROQ_API_KEY"))
|
||||||
|
if not has_key:
|
||||||
|
try:
|
||||||
|
cfg = config if config is not None else Config()
|
||||||
|
has_key = bool(cfg.get("groq_api_key"))
|
||||||
|
except Exception:
|
||||||
|
has_key = False
|
||||||
|
if not has_key:
|
||||||
|
return "warn", (
|
||||||
|
"需要配置 Groq API Key(免费)。步骤:\n"
|
||||||
|
" 1. 注册 https://console.groq.com\n"
|
||||||
|
" 2. 运行: agent-reach configure groq-key gsk_xxxxx"
|
||||||
|
)
|
||||||
|
|
||||||
|
return "ok", "完整可用(播客下载 + Whisper 转录)"
|
||||||
@@ -0,0 +1,314 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Xueqiu (雪球) — stock quotes, search, trending posts & hot stocks."""
|
||||||
|
|
||||||
|
import http.cookiejar
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
_UA = (
|
||||||
|
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
|
||||||
|
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||||
|
"Chrome/120.0.0.0 Safari/537.36"
|
||||||
|
)
|
||||||
|
_REFERER = "https://xueqiu.com/"
|
||||||
|
_TIMEOUT = 10
|
||||||
|
_XUEQIU_HOME = "https://xueqiu.com"
|
||||||
|
|
||||||
|
# --------------- cookie-aware HTTP helpers --------------- #
|
||||||
|
|
||||||
|
_cookie_jar = http.cookiejar.CookieJar()
|
||||||
|
_opener = urllib.request.build_opener(
|
||||||
|
urllib.request.HTTPCookieProcessor(_cookie_jar),
|
||||||
|
)
|
||||||
|
_cookies_initialized = False
|
||||||
|
|
||||||
|
|
||||||
|
def _inject_cookie_string(cookie_str: str) -> None:
|
||||||
|
"""Parse a 'name=value; name2=value2' string and inject into the cookie jar."""
|
||||||
|
for pair in cookie_str.split(";"):
|
||||||
|
pair = pair.strip()
|
||||||
|
if "=" not in pair:
|
||||||
|
continue
|
||||||
|
name, _, value = pair.partition("=")
|
||||||
|
cookie = http.cookiejar.Cookie(
|
||||||
|
version=0,
|
||||||
|
name=name.strip(),
|
||||||
|
value=value.strip(),
|
||||||
|
port=None,
|
||||||
|
port_specified=False,
|
||||||
|
domain=".xueqiu.com",
|
||||||
|
domain_specified=True,
|
||||||
|
domain_initial_dot=True,
|
||||||
|
path="/",
|
||||||
|
path_specified=True,
|
||||||
|
secure=True,
|
||||||
|
expires=None,
|
||||||
|
discard=True,
|
||||||
|
comment=None,
|
||||||
|
comment_url=None,
|
||||||
|
rest={},
|
||||||
|
)
|
||||||
|
_cookie_jar.set_cookie(cookie)
|
||||||
|
|
||||||
|
|
||||||
|
def _load_cookies_from_config() -> bool:
|
||||||
|
"""Try to load Xueqiu cookies from agent-reach config file (xueqiu_cookie key)."""
|
||||||
|
try:
|
||||||
|
from ..config import Config
|
||||||
|
|
||||||
|
cfg = Config()
|
||||||
|
cookie_str = cfg.get("xueqiu_cookie")
|
||||||
|
if not cookie_str:
|
||||||
|
return False
|
||||||
|
_inject_cookie_string(cookie_str)
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _load_cookies_from_browser() -> bool:
|
||||||
|
"""Try to silently load Xueqiu cookies from the local Chrome browser.
|
||||||
|
|
||||||
|
Only succeeds when browser_cookie3 is installed AND the user is logged in
|
||||||
|
(xq_a_token present). Failures are silently ignored so that agents without
|
||||||
|
a local browser keep working.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
try:
|
||||||
|
import rookiepy
|
||||||
|
cookies = rookiepy.chrome([".xueqiu.com"])
|
||||||
|
if not any(c.get("name") == "xq_a_token" for c in cookies):
|
||||||
|
return False
|
||||||
|
for c in cookies:
|
||||||
|
_cookie_jar.set(c["name"], c["value"], domain=c.get("domain", ".xueqiu.com"))
|
||||||
|
return True
|
||||||
|
except ImportError:
|
||||||
|
import browser_cookie3
|
||||||
|
cookies = list(browser_cookie3.chrome(domain_name=".xueqiu.com"))
|
||||||
|
if not any(c.name == "xq_a_token" for c in cookies):
|
||||||
|
return False
|
||||||
|
for c in cookies:
|
||||||
|
_cookie_jar.set_cookie(c)
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_cookies() -> None:
|
||||||
|
"""Populate session cookies using the best available source.
|
||||||
|
|
||||||
|
Priority order:
|
||||||
|
1. Saved cookie string in ~/.agent-reach/config.yaml (set by configure --from-browser)
|
||||||
|
2. Live Chrome browser cookies via rookiepy/browser_cookie3 (if installed + logged in)
|
||||||
|
3. Homepage visit fallback (only yields anti-DDoS acw_tc,
|
||||||
|
not enough for stock APIs)
|
||||||
|
"""
|
||||||
|
global _cookies_initialized
|
||||||
|
if _cookies_initialized:
|
||||||
|
return
|
||||||
|
if _load_cookies_from_config():
|
||||||
|
_cookies_initialized = True
|
||||||
|
return
|
||||||
|
if _load_cookies_from_browser():
|
||||||
|
_cookies_initialized = True
|
||||||
|
return
|
||||||
|
# Fallback: visit homepage to pick up acw_tc anti-DDoS cookie.
|
||||||
|
# This is not sufficient for authenticated APIs but avoids hard failures
|
||||||
|
# on public endpoints that only need the session cookie.
|
||||||
|
req = urllib.request.Request(_XUEQIU_HOME, headers={"User-Agent": _UA})
|
||||||
|
_opener.open(req, timeout=_TIMEOUT)
|
||||||
|
_cookies_initialized = True
|
||||||
|
|
||||||
|
|
||||||
|
def _get_json(url: str) -> Any:
|
||||||
|
"""Fetch *url* with Xueqiu session cookies and return parsed JSON."""
|
||||||
|
_ensure_cookies()
|
||||||
|
req = urllib.request.Request(
|
||||||
|
url, headers={"User-Agent": _UA, "Referer": _REFERER}
|
||||||
|
)
|
||||||
|
with _opener.open(req, timeout=_TIMEOUT) as resp:
|
||||||
|
return json.loads(resp.read().decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_html(text: str) -> str:
|
||||||
|
"""Remove HTML tags and decode common entities."""
|
||||||
|
text = re.sub(r"<[^>]+>", "", text)
|
||||||
|
for entity, char in ((" ", " "), ("&", "&"), ("<", "<"), (">", ">")):
|
||||||
|
text = text.replace(entity, char)
|
||||||
|
return text.strip()
|
||||||
|
|
||||||
|
|
||||||
|
class XueqiuChannel(Channel):
|
||||||
|
name = "xueqiu"
|
||||||
|
description = "雪球股票行情与社区动态"
|
||||||
|
backends = ["Xueqiu API (需要登录 Cookie)"]
|
||||||
|
tier = 1
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# URL routing
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
d = urllib.parse.urlparse(url).netloc.lower()
|
||||||
|
return "xueqiu.com" in d
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Health check
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
try:
|
||||||
|
data = _get_json(
|
||||||
|
"https://stock.xueqiu.com/v5/stock/batch/quote.json?symbol=SH000001"
|
||||||
|
)
|
||||||
|
items = (data.get("data") or {}).get("items") or []
|
||||||
|
if items:
|
||||||
|
return "ok", "公开 API 可用(行情、搜索、热帖、热股)"
|
||||||
|
return "warn", "API 响应异常(返回数据为空)"
|
||||||
|
except Exception as e:
|
||||||
|
return "warn", (
|
||||||
|
f"Xueqiu API 连接失败:{e}。"
|
||||||
|
"请先登录雪球后运行:agent-reach configure --from-browser chrome"
|
||||||
|
)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Data-fetching methods
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def get_stock_quote(self, symbol: str) -> dict:
|
||||||
|
"""获取实时股票行情。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
symbol: 股票代码,如 SH600519(沪)、SZ000858(深)、AAPL(美)、00700(港)
|
||||||
|
|
||||||
|
Returns a dict with keys:
|
||||||
|
symbol, name, current, percent, chg, high, low, open, last_close,
|
||||||
|
volume, amount, market_capital, turnover_rate, pe_ttm, timestamp
|
||||||
|
"""
|
||||||
|
data = _get_json(
|
||||||
|
f"https://stock.xueqiu.com/v5/stock/batch/quote.json?symbol={symbol}"
|
||||||
|
)
|
||||||
|
items = (data.get("data") or {}).get("items") or []
|
||||||
|
q = (items[0].get("quote") or {}) if items else {}
|
||||||
|
return {
|
||||||
|
"symbol": q.get("symbol", symbol),
|
||||||
|
"name": q.get("name", ""),
|
||||||
|
"current": q.get("current"),
|
||||||
|
"percent": q.get("percent"),
|
||||||
|
"chg": q.get("chg"),
|
||||||
|
"high": q.get("high"),
|
||||||
|
"low": q.get("low"),
|
||||||
|
"open": q.get("open"),
|
||||||
|
"last_close": q.get("last_close"),
|
||||||
|
"volume": q.get("volume"),
|
||||||
|
"amount": q.get("amount"),
|
||||||
|
"market_capital": q.get("market_capital"),
|
||||||
|
"turnover_rate": q.get("turnover_rate"),
|
||||||
|
"pe_ttm": q.get("pe_ttm"),
|
||||||
|
"timestamp": q.get("timestamp"),
|
||||||
|
}
|
||||||
|
|
||||||
|
def search_stock(self, query: str, limit: int = 10) -> list:
|
||||||
|
"""搜索股票。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
query: 股票代码或中文名称,如 "茅台"、"600519"
|
||||||
|
limit: 最多返回条数
|
||||||
|
|
||||||
|
Returns a list of dicts with keys:
|
||||||
|
symbol, name, exchange
|
||||||
|
"""
|
||||||
|
data = _get_json(
|
||||||
|
f"https://xueqiu.com/stock/search.json"
|
||||||
|
f"?code={urllib.parse.quote(query)}&size={limit}"
|
||||||
|
)
|
||||||
|
stocks = data.get("stocks") or []
|
||||||
|
results = []
|
||||||
|
for s in stocks[:limit]:
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"symbol": s.get("code", ""),
|
||||||
|
"name": s.get("name", ""),
|
||||||
|
"exchange": s.get("exchange", ""),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return results
|
||||||
|
|
||||||
|
def get_hot_posts(self, limit: int = 20) -> list:
|
||||||
|
"""获取雪球热门帖子。
|
||||||
|
|
||||||
|
Uses the v4 public timeline endpoint which returns posts in a `list`
|
||||||
|
array. Each item carries a JSON-encoded `data` field containing the
|
||||||
|
actual post payload (title, description, user, like_count, target).
|
||||||
|
|
||||||
|
Args:
|
||||||
|
limit: 最多返回条数(上限 50)
|
||||||
|
|
||||||
|
Returns a list of dicts with keys:
|
||||||
|
id, title, text, author, likes, url
|
||||||
|
"""
|
||||||
|
data = _get_json(
|
||||||
|
"https://xueqiu.com/v4/statuses/public_timeline_by_category.json"
|
||||||
|
"?since_id=-1&max_id=-1&count=20&category=-1"
|
||||||
|
)
|
||||||
|
items = data.get("list") or []
|
||||||
|
results = []
|
||||||
|
for item in items[:limit]:
|
||||||
|
# Each item.data is a JSON string containing the real post payload
|
||||||
|
try:
|
||||||
|
post = (
|
||||||
|
json.loads(item["data"])
|
||||||
|
if isinstance(item.get("data"), str)
|
||||||
|
else {}
|
||||||
|
)
|
||||||
|
except (json.JSONDecodeError, KeyError):
|
||||||
|
post = {}
|
||||||
|
user = post.get("user") or {}
|
||||||
|
text = _strip_html(
|
||||||
|
post.get("text") or post.get("description") or ""
|
||||||
|
)
|
||||||
|
target = post.get("target", "")
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"id": post.get("id", 0),
|
||||||
|
"title": post.get("title") or "",
|
||||||
|
"text": text[:200],
|
||||||
|
"author": user.get("screen_name", ""),
|
||||||
|
"likes": post.get("like_count", 0),
|
||||||
|
"url": f"https://xueqiu.com{target}" if target else "",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return results
|
||||||
|
|
||||||
|
def get_hot_stocks(self, limit: int = 10, stock_type: int = 10) -> list:
|
||||||
|
"""获取热门股票排行。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
limit: 最多返回条数(上限 50)
|
||||||
|
stock_type: 10=人气榜(默认),12=关注榜
|
||||||
|
|
||||||
|
Returns a list of dicts with keys:
|
||||||
|
symbol, name, current, percent, rank
|
||||||
|
"""
|
||||||
|
data = _get_json(
|
||||||
|
f"https://stock.xueqiu.com/v5/stock/hot_stock/list.json"
|
||||||
|
f"?size={limit}&type={stock_type}"
|
||||||
|
)
|
||||||
|
items = (data.get("data") or {}).get("items") or []
|
||||||
|
results = []
|
||||||
|
for idx, item in enumerate(items[:limit], 1):
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"symbol": item.get("code") or item.get("symbol", ""),
|
||||||
|
"name": item.get("name", ""),
|
||||||
|
"current": item.get("current"),
|
||||||
|
"percent": item.get("percent"),
|
||||||
|
"rank": idx,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return results
|
||||||
+30
-109
@@ -1,125 +1,46 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""YouTube — via yt-dlp (video info, subtitles, and search).
|
"""YouTube — check if yt-dlp is available with JS runtime."""
|
||||||
|
|
||||||
Backend: yt-dlp (https://github.com/yt-dlp/yt-dlp)
|
|
||||||
Supports: read (info + subtitles), search (ytsearch)
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
|
||||||
import tempfile
|
from agent_reach.utils.paths import get_ytdlp_config_path, render_ytdlp_fix_command
|
||||||
from pathlib import Path
|
from agent_reach.utils.text import read_utf8_text
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult, SearchResult
|
from .base import Channel
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class YouTubeChannel(Channel):
|
class YouTubeChannel(Channel):
|
||||||
name = "youtube"
|
name = "youtube"
|
||||||
description = "YouTube 视频字幕"
|
description = "YouTube 视频和字幕"
|
||||||
backends = ["yt-dlp"]
|
backends = ["yt-dlp"]
|
||||||
requires_tools = ["yt-dlp"]
|
|
||||||
tier = 0
|
tier = 0
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
d = urlparse(url).netloc.lower()
|
d = urlparse(url).netloc.lower()
|
||||||
return "youtube.com" in d or "youtu.be" in d
|
return "youtube.com" in d or "youtu.be" in d
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
def check(self, config=None):
|
||||||
if not shutil.which("yt-dlp"):
|
if not shutil.which("yt-dlp"):
|
||||||
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
|
return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
|
||||||
|
# Check JS runtime
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
has_js = shutil.which("deno") or shutil.which("node")
|
||||||
info = self._get_info(url)
|
if not has_js:
|
||||||
title = info.get("title", url)
|
return "warn", (
|
||||||
author = info.get("uploader", "")
|
"yt-dlp 已安装但缺少 JS runtime(YouTube 必须)。\n"
|
||||||
|
" 安装 Node.js 或 deno,然后运行:agent-reach install"
|
||||||
transcript = self._get_subtitles(url, tmpdir)
|
|
||||||
if not transcript:
|
|
||||||
transcript = f"[Video: {title}]\n[No subtitles available.]"
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=title, content=transcript, url=url,
|
|
||||||
author=author, platform="youtube",
|
|
||||||
extra={
|
|
||||||
"duration": info.get("duration_string"),
|
|
||||||
"view_count": info.get("view_count"),
|
|
||||||
"upload_date": info.get("upload_date"),
|
|
||||||
},
|
|
||||||
)
|
)
|
||||||
|
# Check yt-dlp config for --js-runtimes
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
# Deno works out of the box; Node.js requires explicit config
|
||||||
"""Search YouTube via yt-dlp's ytsearch."""
|
has_deno = shutil.which("deno")
|
||||||
if not shutil.which("yt-dlp"):
|
if not has_deno:
|
||||||
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
|
ytdlp_config = get_ytdlp_config_path()
|
||||||
|
has_js_config = False
|
||||||
limit = kwargs.get("limit", 10)
|
if ytdlp_config.exists():
|
||||||
|
has_js_config = "--js-runtimes" in read_utf8_text(ytdlp_config)
|
||||||
try:
|
if not has_js_config:
|
||||||
r = subprocess.run(
|
return "warn", (
|
||||||
["yt-dlp", "--dump-json", "--flat-playlist",
|
"yt-dlp 已安装但未配置 JS runtime。运行:\n"
|
||||||
f"ytsearch{limit}:{query}"],
|
f" {render_ytdlp_fix_command()}"
|
||||||
capture_output=True, text=True, timeout=30,
|
)
|
||||||
)
|
return "ok", "可提取视频信息和字幕"
|
||||||
results = []
|
|
||||||
for line in r.stdout.strip().split("\n"):
|
|
||||||
if not line.strip():
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
vid = d.get("id", "")
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=d.get("title", ""),
|
|
||||||
url=f"https://youtube.com/watch?v={vid}" if vid else "",
|
|
||||||
snippet=(
|
|
||||||
f"👤 {d.get('channel', '?')} · "
|
|
||||||
f"⏱ {d.get('duration_string', '?')} · "
|
|
||||||
f"👁 {d.get('view_count', '?')}"
|
|
||||||
),
|
|
||||||
extra={
|
|
||||||
"channel": d.get("channel"),
|
|
||||||
"duration": d.get("duration_string"),
|
|
||||||
"view_count": d.get("view_count"),
|
|
||||||
},
|
|
||||||
))
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
continue
|
|
||||||
return results
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _get_info(self, url: str) -> dict:
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["yt-dlp", "--dump-json", "--no-download", url],
|
|
||||||
capture_output=True, text=True, timeout=30,
|
|
||||||
)
|
|
||||||
if r.returncode == 0:
|
|
||||||
return json.loads(r.stdout)
|
|
||||||
except (subprocess.TimeoutExpired, json.JSONDecodeError):
|
|
||||||
pass
|
|
||||||
return {}
|
|
||||||
|
|
||||||
def _get_subtitles(self, url: str, tmpdir: str) -> str:
|
|
||||||
try:
|
|
||||||
subprocess.run(
|
|
||||||
["yt-dlp", "--write-auto-sub", "--write-sub",
|
|
||||||
"--sub-lang", "en,zh-Hans,zh",
|
|
||||||
"--skip-download", "--sub-format", "vtt",
|
|
||||||
"-o", f"{tmpdir}/%(id)s.%(ext)s", url],
|
|
||||||
capture_output=True, text=True, timeout=30,
|
|
||||||
)
|
|
||||||
for f in Path(tmpdir).glob("*.vtt"):
|
|
||||||
text = f.read_text(errors="replace")
|
|
||||||
lines = []
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if not line or line.startswith("WEBVTT") or "-->" in line or line.isdigit():
|
|
||||||
continue
|
|
||||||
if line not in lines[-1:]:
|
|
||||||
lines.append(line)
|
|
||||||
return "\n".join(lines)
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
pass
|
|
||||||
return ""
|
|
||||||
|
|||||||
+1166
-514
File diff suppressed because it is too large
Load Diff
+15
-8
@@ -21,8 +21,7 @@ class Config:
|
|||||||
# Feature → required config keys
|
# Feature → required config keys
|
||||||
FEATURE_REQUIREMENTS = {
|
FEATURE_REQUIREMENTS = {
|
||||||
"exa_search": ["exa_api_key"],
|
"exa_search": ["exa_api_key"],
|
||||||
"reddit_proxy": ["reddit_proxy"],
|
"twitter_xreach": ["twitter_auth_token", "twitter_ct0"], # legacy key name; used by bird CLI
|
||||||
"twitter_bird": ["twitter_auth_token", "twitter_ct0"],
|
|
||||||
"groq_whisper": ["groq_api_key"],
|
"groq_whisper": ["groq_api_key"],
|
||||||
"github_token": ["github_token"],
|
"github_token": ["github_token"],
|
||||||
}
|
}
|
||||||
@@ -41,7 +40,7 @@ class Config:
|
|||||||
def load(self):
|
def load(self):
|
||||||
"""Load config from YAML file."""
|
"""Load config from YAML file."""
|
||||||
if self.config_path.exists():
|
if self.config_path.exists():
|
||||||
with open(self.config_path, "r") as f:
|
with open(self.config_path, "r", encoding="utf-8") as f:
|
||||||
self.data = yaml.safe_load(f) or {}
|
self.data = yaml.safe_load(f) or {}
|
||||||
else:
|
else:
|
||||||
self.data = {}
|
self.data = {}
|
||||||
@@ -49,14 +48,22 @@ class Config:
|
|||||||
def save(self):
|
def save(self):
|
||||||
"""Save config to YAML file."""
|
"""Save config to YAML file."""
|
||||||
self._ensure_dir()
|
self._ensure_dir()
|
||||||
with open(self.config_path, "w") as f:
|
# Create file with restricted permissions from the start to avoid
|
||||||
yaml.dump(self.data, f, default_flow_style=False, allow_unicode=True)
|
# a race window where credentials are briefly world-readable.
|
||||||
# Restrict permissions — config may contain credentials
|
|
||||||
try:
|
try:
|
||||||
import stat
|
import stat
|
||||||
self.config_path.chmod(stat.S_IRUSR | stat.S_IWUSR) # 0o600
|
fd = os.open(
|
||||||
|
str(self.config_path),
|
||||||
|
os.O_WRONLY | os.O_CREAT | os.O_TRUNC,
|
||||||
|
stat.S_IRUSR | stat.S_IWUSR, # 0o600
|
||||||
|
)
|
||||||
|
with os.fdopen(fd, "w", encoding="utf-8") as f:
|
||||||
|
yaml.dump(self.data, f, default_flow_style=False, allow_unicode=True)
|
||||||
except OSError:
|
except OSError:
|
||||||
pass # Windows or permission edge cases
|
# Fallback for Windows or other edge cases where os.open flags
|
||||||
|
# are not fully supported.
|
||||||
|
with open(self.config_path, "w", encoding="utf-8") as f:
|
||||||
|
yaml.dump(self.data, f, default_flow_style=False, allow_unicode=True)
|
||||||
|
|
||||||
def get(self, key: str, default: Any = None) -> Any:
|
def get(self, key: str, default: Any = None) -> Any:
|
||||||
"""Get a config value. Also checks environment variables (uppercase)."""
|
"""Get a config value. Also checks environment variables (uppercase)."""
|
||||||
|
|||||||
+124
-23
@@ -32,6 +32,12 @@ PLATFORM_SPECS = [
|
|||||||
"cookies": ["SESSDATA", "bili_jct"],
|
"cookies": ["SESSDATA", "bili_jct"],
|
||||||
"config_key": "bilibili",
|
"config_key": "bilibili",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"name": "Xueqiu",
|
||||||
|
"domains": [".xueqiu.com", "xueqiu.com"],
|
||||||
|
"cookies": None, # grab all — xq_a_token + session cookies required
|
||||||
|
"config_key": "xueqiu",
|
||||||
|
},
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
@@ -46,36 +52,66 @@ def extract_all(browser: str = "chrome") -> Dict[str, dict]:
|
|||||||
"bilibili": {"SESSDATA": "xxx", "bili_jct": "yyy"},
|
"bilibili": {"SESSDATA": "xxx", "bili_jct": "yyy"},
|
||||||
}
|
}
|
||||||
"""
|
"""
|
||||||
|
# Try rookiepy first (Rust-based, more stable), fallback to browser_cookie3
|
||||||
|
use_rookiepy = False
|
||||||
try:
|
try:
|
||||||
import browser_cookie3
|
import rookiepy
|
||||||
|
use_rookiepy = True
|
||||||
except ImportError:
|
except ImportError:
|
||||||
raise RuntimeError(
|
try:
|
||||||
"browser_cookie3 not installed. Run: pip install browser-cookie3"
|
import browser_cookie3
|
||||||
)
|
except ImportError:
|
||||||
|
raise RuntimeError(
|
||||||
# Get browser cookie jar
|
"Cookie extraction requires rookiepy or browser_cookie3.\n"
|
||||||
browser_funcs = {
|
"Install: pip install rookiepy (recommended)\n"
|
||||||
"chrome": browser_cookie3.chrome,
|
" or: pip install browser-cookie3"
|
||||||
"firefox": browser_cookie3.firefox,
|
)
|
||||||
"edge": browser_cookie3.edge,
|
|
||||||
"brave": browser_cookie3.brave,
|
|
||||||
"opera": browser_cookie3.opera,
|
|
||||||
}
|
|
||||||
|
|
||||||
browser = browser.lower()
|
browser = browser.lower()
|
||||||
if browser not in browser_funcs:
|
supported = ["chrome", "firefox", "edge", "brave", "opera"]
|
||||||
|
if browser not in supported:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"Unsupported browser: {browser}. "
|
f"Unsupported browser: {browser}. Supported: {', '.join(supported)}"
|
||||||
f"Supported: {', '.join(browser_funcs.keys())}"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
try:
|
if use_rookiepy:
|
||||||
cookie_jar = browser_funcs[browser]()
|
# rookiepy returns list of dicts with name/value/domain/path keys
|
||||||
except Exception as e:
|
try:
|
||||||
raise RuntimeError(
|
browser_funcs = {
|
||||||
f"Could not read {browser} cookies: {e}\n"
|
"chrome": rookiepy.chrome,
|
||||||
f"Make sure {browser} is closed and you have permission to read its data."
|
"firefox": rookiepy.firefox,
|
||||||
)
|
"edge": rookiepy.edge,
|
||||||
|
"brave": rookiepy.brave,
|
||||||
|
"opera": rookiepy.opera,
|
||||||
|
}
|
||||||
|
raw_cookies = browser_funcs[browser]()
|
||||||
|
# Wrap into objects with .name, .value, .domain for compatibility
|
||||||
|
class _Cookie:
|
||||||
|
def __init__(self, d):
|
||||||
|
self.name = d.get("name", "")
|
||||||
|
self.value = d.get("value", "")
|
||||||
|
self.domain = d.get("domain", "")
|
||||||
|
cookie_jar = [_Cookie(c) for c in raw_cookies]
|
||||||
|
except Exception as e:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Could not read {browser} cookies via rookiepy: {e}\n"
|
||||||
|
f"Make sure {browser} is closed and you have permission."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
browser_funcs = {
|
||||||
|
"chrome": browser_cookie3.chrome,
|
||||||
|
"firefox": browser_cookie3.firefox,
|
||||||
|
"edge": browser_cookie3.edge,
|
||||||
|
"brave": browser_cookie3.brave,
|
||||||
|
"opera": browser_cookie3.opera,
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
cookie_jar = browser_funcs[browser]()
|
||||||
|
except Exception as e:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Could not read {browser} cookies: {e}\n"
|
||||||
|
f"Make sure {browser} is closed and you have permission."
|
||||||
|
)
|
||||||
|
|
||||||
results = {}
|
results = {}
|
||||||
|
|
||||||
@@ -112,6 +148,57 @@ def extract_all(browser: str = "chrome") -> Dict[str, dict]:
|
|||||||
return results
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def _sync_xfetch_session(auth_token: str, ct0: str) -> None:
|
||||||
|
"""Sync Twitter credentials to ~/.config/xfetch/session.json (legacy xreach compat)."""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
try:
|
||||||
|
xfetch_dir = os.path.join(os.path.expanduser("~"), ".config", "xfetch")
|
||||||
|
os.makedirs(xfetch_dir, exist_ok=True)
|
||||||
|
session_path = os.path.join(xfetch_dir, "session.json")
|
||||||
|
session_data: dict = {}
|
||||||
|
if os.path.exists(session_path):
|
||||||
|
try:
|
||||||
|
with open(session_path, "r", encoding="utf-8") as sf:
|
||||||
|
session_data = json.load(sf)
|
||||||
|
except (json.JSONDecodeError, OSError):
|
||||||
|
session_data = {}
|
||||||
|
session_data["authToken"] = auth_token
|
||||||
|
session_data["ct0"] = ct0
|
||||||
|
with open(session_path, "w", encoding="utf-8") as sf:
|
||||||
|
json.dump(session_data, sf, indent=2)
|
||||||
|
os.chmod(session_path, 0o600)
|
||||||
|
except Exception:
|
||||||
|
# Non-fatal: agent-reach config is the source of truth, xfetch sync is best-effort
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _sync_bird_env(auth_token: str, ct0: str) -> None:
|
||||||
|
"""Write Twitter credentials to ~/.config/bird/credentials.env for bird CLI.
|
||||||
|
|
||||||
|
bird reads AUTH_TOKEN and CT0 from environment variables. This writes a
|
||||||
|
shell-sourceable file so users can `source ~/.config/bird/credentials.env`.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
|
||||||
|
try:
|
||||||
|
bird_dir = os.path.join(os.path.expanduser("~"), ".config", "bird")
|
||||||
|
os.makedirs(bird_dir, exist_ok=True)
|
||||||
|
env_path = os.path.join(bird_dir, "credentials.env")
|
||||||
|
with open(env_path, "w", encoding="utf-8") as f:
|
||||||
|
f.write(f'AUTH_TOKEN="{auth_token}"\n')
|
||||||
|
f.write(f'CT0="{ct0}"\n')
|
||||||
|
os.chmod(env_path, 0o600)
|
||||||
|
except Exception:
|
||||||
|
# Non-fatal: agent-reach config is the source of truth, bird env sync is best-effort
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# Alias for callers expecting the name _sync_bird_credentials
|
||||||
|
_sync_bird_credentials = _sync_bird_env
|
||||||
|
|
||||||
|
|
||||||
def configure_from_browser(browser: str, config) -> List[Tuple[str, bool, str]]:
|
def configure_from_browser(browser: str, config) -> List[Tuple[str, bool, str]]:
|
||||||
"""
|
"""
|
||||||
Extract cookies and configure all found platforms.
|
Extract cookies and configure all found platforms.
|
||||||
@@ -136,6 +223,8 @@ def configure_from_browser(browser: str, config) -> List[Tuple[str, bool, str]]:
|
|||||||
if "auth_token" in tc and "ct0" in tc:
|
if "auth_token" in tc and "ct0" in tc:
|
||||||
config.set("twitter_auth_token", tc["auth_token"])
|
config.set("twitter_auth_token", tc["auth_token"])
|
||||||
config.set("twitter_ct0", tc["ct0"])
|
config.set("twitter_ct0", tc["ct0"])
|
||||||
|
# Legacy sync (best-effort)
|
||||||
|
_sync_xfetch_session(tc["auth_token"], tc["ct0"])
|
||||||
results_list.append(("Twitter/X", True, "auth_token + ct0"))
|
results_list.append(("Twitter/X", True, "auth_token + ct0"))
|
||||||
else:
|
else:
|
||||||
found = ", ".join(tc.keys())
|
found = ", ".join(tc.keys())
|
||||||
@@ -163,4 +252,16 @@ def configure_from_browser(browser: str, config) -> List[Tuple[str, bool, str]]:
|
|||||||
results_list.append(("Bilibili", False,
|
results_list.append(("Bilibili", False,
|
||||||
f"No SESSDATA found. Make sure you're logged into bilibili.com in {browser}."))
|
f"No SESSDATA found. Make sure you're logged into bilibili.com in {browser}."))
|
||||||
|
|
||||||
|
if "xueqiu" in extracted:
|
||||||
|
cookie_str = extracted["xueqiu"].get("cookie_string", "")
|
||||||
|
# Only save if xq_a_token is present — anonymous cookies are useless
|
||||||
|
if cookie_str and "xq_a_token" in cookie_str:
|
||||||
|
config.set("xueqiu_cookie", cookie_str)
|
||||||
|
n_cookies = len(cookie_str.split(";"))
|
||||||
|
results_list.append(("Xueqiu", True, f"{n_cookies} cookies (含 xq_a_token)"))
|
||||||
|
elif cookie_str:
|
||||||
|
results_list.append(("Xueqiu", False,
|
||||||
|
f"找到 {len(cookie_str.split(';'))} 个 Cookie 但缺少 xq_a_token,"
|
||||||
|
f"请先在 {browser} 中登录 xueqiu.com"))
|
||||||
|
|
||||||
return results_list
|
return results_list
|
||||||
|
|||||||
+16
-116
@@ -1,126 +1,36 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
AgentReach — the unified entry point.
|
AgentReach — installer, doctor, and configuration tool.
|
||||||
|
|
||||||
Pure glue: routes URLs to the right channel, routes searches to the right engine.
|
Agent Reach helps AI agents install and configure upstream platform tools
|
||||||
Every channel is a thin wrapper around an external tool. Swap any backend anytime.
|
(twitter-cli, yt-dlp, mcporter, gh CLI, etc.). After installation, agents
|
||||||
|
call the upstream tools directly — no wrapper layer needed.
|
||||||
|
|
||||||
Usage:
|
Usage:
|
||||||
from agent_reach import AgentReach
|
from agent_reach.doctor import check_all, format_report
|
||||||
|
from agent_reach.config import Config
|
||||||
|
|
||||||
eyes = AgentReach()
|
config = Config()
|
||||||
content = await eyes.read("https://github.com/openai/gpt-4")
|
results = check_all(config)
|
||||||
results = await eyes.search("AI agent framework")
|
print(format_report(results))
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
from typing import Dict, Optional
|
||||||
from typing import Any, Dict, List, Optional
|
|
||||||
|
|
||||||
from agent_reach.config import Config
|
from agent_reach.config import Config
|
||||||
from agent_reach.channels import get_channel_for_url, get_channel, get_all_channels
|
|
||||||
|
|
||||||
|
|
||||||
class AgentReach:
|
class AgentReach:
|
||||||
"""Give your AI Agent eyes to see the entire internet."""
|
"""Give your AI Agent eyes to see the entire internet.
|
||||||
|
|
||||||
|
This class provides health-check functionality.
|
||||||
|
For reading/searching, use the upstream tools directly
|
||||||
|
(see SKILL.md for commands).
|
||||||
|
"""
|
||||||
|
|
||||||
def __init__(self, config: Optional[Config] = None):
|
def __init__(self, config: Optional[Config] = None):
|
||||||
self.config = config or Config()
|
self.config = config or Config()
|
||||||
|
|
||||||
# ── Reading ─────────────────────────────────────────
|
|
||||||
|
|
||||||
async def read(self, url: str) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Read content from any URL. Auto-detects platform.
|
|
||||||
|
|
||||||
Supported: Web, GitHub, Reddit, Twitter, YouTube,
|
|
||||||
Bilibili, RSS, and more.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict with title, content, url, author, platform, etc.
|
|
||||||
"""
|
|
||||||
if not url.startswith(("http://", "https://")):
|
|
||||||
url = f"https://{url}"
|
|
||||||
|
|
||||||
channel = get_channel_for_url(url)
|
|
||||||
result = await channel.read(url, config=self.config)
|
|
||||||
return result.to_dict()
|
|
||||||
|
|
||||||
async def read_batch(self, urls: List[str]) -> List[Dict[str, Any]]:
|
|
||||||
"""Read multiple URLs concurrently."""
|
|
||||||
tasks = [self.read(url) for url in urls]
|
|
||||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
|
||||||
return [r for r in results if not isinstance(r, Exception)]
|
|
||||||
|
|
||||||
def detect_platform(self, url: str) -> str:
|
|
||||||
"""Detect what platform a URL belongs to."""
|
|
||||||
channel = get_channel_for_url(url)
|
|
||||||
return channel.name
|
|
||||||
|
|
||||||
# ── Searching ───────────────────────────────────────
|
|
||||||
|
|
||||||
async def search(self, query: str, num_results: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Semantic web search via Exa."""
|
|
||||||
ch = get_channel("exa_search")
|
|
||||||
results = await ch.search(query, config=self.config, limit=num_results)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_reddit(self, query: str, subreddit: Optional[str] = None, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Reddit via Exa (bypasses IP blocks)."""
|
|
||||||
ch = get_channel("exa_search")
|
|
||||||
q = f"site:reddit.com/r/{subreddit} {query}" if subreddit else f"site:reddit.com {query}"
|
|
||||||
results = await ch.search(q, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_github(self, query: str, language: Optional[str] = None, limit: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Search GitHub repositories."""
|
|
||||||
ch = get_channel("github")
|
|
||||||
results = await ch.search(query, config=self.config, language=language, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_twitter(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Twitter. Uses bird CLI if available, else Exa."""
|
|
||||||
ch = get_channel("twitter")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_youtube(self, query: str, limit: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Search YouTube via yt-dlp."""
|
|
||||||
ch = get_channel("youtube")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_bilibili(self, query: str, limit: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Bilibili. Tries yt-dlp first, falls back to Exa."""
|
|
||||||
ch = get_channel("bilibili")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_xhs(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search XiaoHongShu via mcporter."""
|
|
||||||
ch = get_channel("xiaohongshu")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_instagram(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Instagram via Exa."""
|
|
||||||
ch = get_channel("instagram")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_linkedin(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search LinkedIn via MCP or Exa."""
|
|
||||||
ch = get_channel("linkedin")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_bosszhipin(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Boss直聘 via MCP or Exa."""
|
|
||||||
ch = get_channel("bosszhipin")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
# ── Health ──────────────────────────────────────────
|
|
||||||
|
|
||||||
def doctor(self) -> Dict[str, dict]:
|
def doctor(self) -> Dict[str, dict]:
|
||||||
"""Check all channel availability."""
|
"""Check all channel availability."""
|
||||||
from agent_reach.doctor import check_all
|
from agent_reach.doctor import check_all
|
||||||
@@ -130,13 +40,3 @@ class AgentReach:
|
|||||||
"""Get formatted health report."""
|
"""Get formatted health report."""
|
||||||
from agent_reach.doctor import check_all, format_report
|
from agent_reach.doctor import check_all, format_report
|
||||||
return format_report(check_all(self.config))
|
return format_report(check_all(self.config))
|
||||||
|
|
||||||
# ── Sync wrappers ───────────────────────────────────
|
|
||||||
|
|
||||||
def read_sync(self, url: str) -> Dict[str, Any]:
|
|
||||||
"""Synchronous version of read()."""
|
|
||||||
return asyncio.run(self.read(url))
|
|
||||||
|
|
||||||
def search_sync(self, query: str, num_results: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Synchronous version of search()."""
|
|
||||||
return asyncio.run(self.search(query, num_results))
|
|
||||||
|
|||||||
+60
-29
@@ -25,53 +25,84 @@ def check_all(config: Config) -> Dict[str, dict]:
|
|||||||
|
|
||||||
|
|
||||||
def format_report(results: Dict[str, dict]) -> str:
|
def format_report(results: Dict[str, dict]) -> str:
|
||||||
"""Format results as a readable text report."""
|
"""Format results as a readable text report (with Rich markup)."""
|
||||||
|
try:
|
||||||
|
from rich.markup import escape
|
||||||
|
except ImportError:
|
||||||
|
escape = lambda x: x
|
||||||
|
|
||||||
lines = []
|
lines = []
|
||||||
lines.append("👁️ Agent Reach 状态")
|
lines.append("[bold cyan]Agent Reach 状态[/bold cyan]")
|
||||||
lines.append("=" * 40)
|
lines.append("[cyan]" + "=" * 40 + "[/cyan]")
|
||||||
|
|
||||||
ok_count = sum(1 for r in results.values() if r["status"] == "ok")
|
ok_count = sum(1 for r in results.values() if r["status"] == "ok")
|
||||||
total = len(results)
|
total = len(results)
|
||||||
|
|
||||||
# Tier 0 — zero config
|
# Tier 0 — zero config
|
||||||
lines.append("")
|
lines.append("")
|
||||||
lines.append("✅ 装好即用:")
|
lines.append("[bold]✅ 装好即用:[/bold]")
|
||||||
for key, r in results.items():
|
for key, r in results.items():
|
||||||
if r["tier"] == 0:
|
if r["tier"] == 0:
|
||||||
|
name_msg = f"[bold]{escape(r['name'])}[/bold] — {escape(r['message'])}"
|
||||||
if r["status"] == "ok":
|
if r["status"] == "ok":
|
||||||
lines.append(f" ✅ {r['name']} — {r['message']}")
|
lines.append(f" [green]✅[/green] {name_msg}")
|
||||||
elif r["status"] == "warn":
|
elif r["status"] == "warn":
|
||||||
lines.append(f" ⚠️ {r['name']} — {r['message']}")
|
lines.append(f" [yellow][!][/yellow] {name_msg}")
|
||||||
elif r["status"] in ("off", "error"):
|
elif r["status"] in ("off", "error"):
|
||||||
lines.append(f" ❌ {r['name']} — {r['message']}")
|
lines.append(f" [red][X][/red] {name_msg}")
|
||||||
|
|
||||||
# Tier 1 — needs free key
|
# Tier 1 — needs free key / login
|
||||||
tier1 = {k: r for k, r in results.items() if r["tier"] == 1}
|
tier1 = {k: r for k, r in results.items() if r["tier"] == 1}
|
||||||
if tier1:
|
tier1_active = {k: r for k, r in tier1.items() if r["status"] == "ok"}
|
||||||
|
tier1_inactive = {k: r for k, r in tier1.items() if r["status"] != "ok"}
|
||||||
|
if tier1_active:
|
||||||
lines.append("")
|
lines.append("")
|
||||||
lines.append("🔍 搜索(mcporter 即可解锁):")
|
lines.append("[bold]可选渠道(已安装):[/bold]")
|
||||||
for key, r in tier1.items():
|
for key, r in tier1_active.items():
|
||||||
if r["status"] == "ok":
|
name_msg = f"[bold]{escape(r['name'])}[/bold] — {escape(r['message'])}"
|
||||||
lines.append(f" ✅ {r['name']} — {r['message']}")
|
lines.append(f" [green]✅[/green] {name_msg}")
|
||||||
else:
|
|
||||||
lines.append(f" ⬜ {r['name']} — {r['message']}")
|
|
||||||
|
|
||||||
# Tier 2 — optional setup
|
# Tier 2 — optional complex setup
|
||||||
tier2 = {k: r for k, r in results.items() if r["tier"] == 2}
|
tier2 = {k: r for k, r in results.items() if r["tier"] == 2}
|
||||||
if tier2:
|
tier2_active = {k: r for k, r in tier2.items() if r["status"] == "ok"}
|
||||||
lines.append("")
|
tier2_inactive = {k: r for k, r in tier2.items() if r["status"] != "ok"}
|
||||||
lines.append("🔧 配置后可用:")
|
if tier2_active:
|
||||||
for key, r in tier2.items():
|
if not tier1_active:
|
||||||
if r["status"] == "ok":
|
lines.append("")
|
||||||
lines.append(f" ✅ {r['name']} — {r['message']}")
|
lines.append("[bold]可选渠道(已安装):[/bold]")
|
||||||
elif r["status"] == "warn":
|
for key, r in tier2_active.items():
|
||||||
lines.append(f" ⚠️ {r['name']} — {r['message']}")
|
name_msg = f"[bold]{escape(r['name'])}[/bold] — {escape(r['message'])}"
|
||||||
else:
|
lines.append(f" [green]✅[/green] {name_msg}")
|
||||||
lines.append(f" ⬜ {r['name']} — {r['message']}")
|
|
||||||
|
|
||||||
lines.append("")
|
lines.append("")
|
||||||
lines.append(f"状态:{ok_count}/{total} 个渠道可用")
|
status_color = "green" if ok_count == total else ("yellow" if ok_count > 0 else "red")
|
||||||
if ok_count < total:
|
lines.append(f"状态:[{status_color}]{ok_count}/{total}[/{status_color}] 个渠道可用")
|
||||||
lines.append("运行 `agent-reach setup` 解锁更多渠道")
|
|
||||||
|
# Summarize inactive optional channels in one line instead of listing each
|
||||||
|
all_inactive = list(tier1_inactive.values()) + list(tier2_inactive.values())
|
||||||
|
if all_inactive:
|
||||||
|
names = [r["name"] for r in all_inactive]
|
||||||
|
lines.append(
|
||||||
|
f"还有 {len(names)} 个可选渠道可以解锁({'、'.join(names)}),"
|
||||||
|
"告诉你的 Agent「帮我装 XXX」即可"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Security check: config file permissions (Unix only)
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
import sys
|
||||||
|
|
||||||
|
config_path = Config.CONFIG_DIR / "config.yaml"
|
||||||
|
if config_path.exists() and sys.platform != "win32":
|
||||||
|
try:
|
||||||
|
mode = config_path.stat().st_mode
|
||||||
|
if mode & (stat.S_IRGRP | stat.S_IROTH):
|
||||||
|
lines.append("")
|
||||||
|
lines.append(
|
||||||
|
"[bold red][!] 安全提示:config.yaml 权限过宽(其他用户可读)[/bold red]"
|
||||||
|
)
|
||||||
|
lines.append(" 修复:chmod 600 ~/.agent-reach/config.yaml")
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|||||||
@@ -22,8 +22,8 @@ mcporter config add exa https://mcp.exa.ai/mcp
|
|||||||
|
|
||||||
### 3. 验证
|
### 3. 验证
|
||||||
```bash
|
```bash
|
||||||
agent-reach doctor | grep "Web Search"
|
agent-reach doctor | grep "Search"
|
||||||
agent-reach search "test" -n 1
|
mcporter call 'exa.web_search_exa(query: "test", numResults: 1)'
|
||||||
```
|
```
|
||||||
|
|
||||||
## 需要用户手动做的步骤
|
## 需要用户手动做的步骤
|
||||||
|
|||||||
@@ -1,60 +1,54 @@
|
|||||||
# Reddit 代理配置指南
|
# Reddit 配置指南
|
||||||
|
|
||||||
## 功能说明
|
## 功能说明
|
||||||
Reddit 封锁了大量服务器 IP,直接访问会返回 403。需要 ISP 代理(Residential/ISP Proxy)才能读取完整的 Reddit 帖子和评论。
|
|
||||||
|
|
||||||
**注意**:即使没有代理,也可以通过 Exa 搜索找到 Reddit 内容(只需要配 Exa API key)。代理只是用于读取完整帖子+评论。
|
Reddit 封锁了几乎所有非浏览器的直接访问(包括数据中心和 ISP 代理 IP),JSON API 返回 403。
|
||||||
|
|
||||||
|
Agent Reach 通过 **rdt-cli** 实现 Reddit 的搜索和阅读功能:
|
||||||
|
- **搜索**:`rdt search "关键词"`
|
||||||
|
- **阅读完整帖子+评论**:`rdt read POST_ID`
|
||||||
|
|
||||||
|
免费,无需代理,无需 API Key,无需登录。
|
||||||
|
|
||||||
## Agent 可自动完成的步骤
|
## Agent 可自动完成的步骤
|
||||||
|
|
||||||
1. 检查当前状态:
|
1. 检查 rdt-cli 是否可用:
|
||||||
```bash
|
```bash
|
||||||
agent-reach doctor | grep "Reddit"
|
which rdt && echo "installed" || echo "not installed"
|
||||||
```
|
```
|
||||||
|
|
||||||
2. 如果用户提供了代理,测试连通性:
|
2. 如果未安装,自动安装:
|
||||||
```bash
|
```bash
|
||||||
curl -s --proxy "用户提供的代理" \
|
pipx install rdt-cli
|
||||||
-H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" \
|
|
||||||
"https://www.reddit.com/r/test.json?limit=1" \
|
|
||||||
-o /dev/null -w "%{http_code}"
|
|
||||||
```
|
```
|
||||||
返回 200 = 可用,403 = 代理被封,其他 = 配置错误
|
|
||||||
|
|
||||||
3. 写入配置:
|
或一键安装:
|
||||||
```python
|
```bash
|
||||||
from agent_reach.config import Config
|
agent-reach install --env=auto --channels=reddit
|
||||||
c = Config()
|
```
|
||||||
c.set("reddit_proxy", "http://用户名:密码@IP:端口")
|
|
||||||
|
## 使用示例
|
||||||
|
|
||||||
|
搜索 Reddit 内容:
|
||||||
|
```bash
|
||||||
|
rdt search "python best practices" -n 5
|
||||||
|
```
|
||||||
|
|
||||||
|
阅读完整帖子和评论:
|
||||||
|
```bash
|
||||||
|
rdt read POST_ID
|
||||||
```
|
```
|
||||||
|
|
||||||
## 需要用户手动做的步骤
|
## 需要用户手动做的步骤
|
||||||
|
|
||||||
请告诉用户:
|
无。rdt-cli 通过 `agent-reach install --env=auto` 自动安装。
|
||||||
|
|
||||||
> 要完整阅读 Reddit 帖子和评论,需要一个 ISP 代理(约 $3-10/月)。
|
## Fallback:Exa 搜索
|
||||||
>
|
|
||||||
> 推荐代理服务商(选一个就行):
|
|
||||||
> 1. **Smartproxy** (https://smartproxy.com) — ISP 代理,按流量计费
|
|
||||||
> 2. **Bright Data** (https://brightdata.com) — 大厂,ISP 代理
|
|
||||||
> 3. **IPRoyal** (https://iproyal.com) — 便宜,适合入门
|
|
||||||
> 4. **ProxyEmpire** (https://proxyempire.io) — 有 Reddit 专用代理
|
|
||||||
>
|
|
||||||
> 购买时选择:
|
|
||||||
> - 类型:**ISP Proxy**(不要选 Datacenter,会被封)
|
|
||||||
> - 地区:**美国**
|
|
||||||
> - 协议:**HTTP**
|
|
||||||
>
|
|
||||||
> 购买后你会得到一个代理地址,格式类似:
|
|
||||||
> `http://用户名:密码@IP地址:端口号`
|
|
||||||
>
|
|
||||||
> 把这个地址发给我就行。
|
|
||||||
>
|
|
||||||
> ⚠️ 如果不想花钱,可以跳过。我仍然可以通过搜索引擎找到 Reddit 上的内容,只是不能读完整的帖子和评论。
|
|
||||||
|
|
||||||
## Agent 收到代理后的操作
|
如果你已经配置了 Exa(通过 mcporter),也可以通过 Exa 搜索 Reddit 内容:
|
||||||
|
|
||||||
1. 测试代理:用 curl 测试 reddit.com 是否返回 200
|
```bash
|
||||||
2. 如果成功,写入配置:`config.set("reddit_proxy", proxy_url)`
|
mcporter call 'exa.web_search_exa(query: "python best practices", numResults: 5, includeDomains: ["reddit.com"])'
|
||||||
3. 反馈:"✅ Reddit 完整阅读已开启!现在我可以读取 Reddit 帖子和所有评论了。"
|
```
|
||||||
4. 如果失败,告诉用户:"❌ 这个代理无法访问 Reddit,请检查代理是否有效,或换一个试试。"
|
|
||||||
|
rdt-cli 是当前推荐方案,无需额外配置即可使用。
|
||||||
|
|||||||
@@ -1,67 +1,84 @@
|
|||||||
# Twitter 高级功能配置指南(bird CLI)
|
# Twitter 高级功能配置指南(twitter-cli)
|
||||||
|
|
||||||
## 功能说明
|
Twitter 基础阅读通过 Jina Reader 免费可用,无需配置。
|
||||||
基础 Twitter 功能(搜索+读单条推文)无需配置,开箱即用。
|
|
||||||
|
|
||||||
高级功能需要 bird CLI:
|
高级功能需要 twitter-cli(@public-clis/twitter-cli):
|
||||||
- 查看用户时间线
|
|
||||||
- 深度搜索(更精确、更多结果)
|
|
||||||
- 读取完整线程(thread)
|
|
||||||
- 查看关注列表推文
|
|
||||||
|
|
||||||
bird 是免费开源工具(npm 包 @steipete/bird),但需要你的 Twitter 账号 cookie。
|
- 搜索推文(`twitter search`)
|
||||||
|
- 读取完整推文和对话链(`twitter tweet`、`twitter thread`)
|
||||||
|
- 用户时间线(`twitter timeline`)
|
||||||
|
- 长文阅读(`twitter article`)
|
||||||
|
|
||||||
## Agent 可自动完成的步骤
|
twitter-cli 是免费开源工具(pipx 安装),但需要你的 Twitter 账号 cookie。
|
||||||
|
|
||||||
|
## 快速配置
|
||||||
|
|
||||||
|
1. 检查 twitter-cli 是否安装:
|
||||||
|
|
||||||
1. 检查 bird 是否安装:
|
|
||||||
```bash
|
```bash
|
||||||
which bird && echo "installed" || echo "not installed"
|
which twitter && echo "installed" || echo "not installed"
|
||||||
```
|
```
|
||||||
|
|
||||||
2. 安装 bird:
|
2. 安装 twitter-cli:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm install -g @steipete/bird
|
pipx install twitter-cli
|
||||||
```
|
```
|
||||||
|
|
||||||
3. 测试(需要先配置 cookie):
|
3. 测试是否配置好:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
AUTH_TOKEN="xxx" CT0="yyy" bird search "test" -n 1
|
twitter search "test" -n 1
|
||||||
```
|
```
|
||||||
|
|
||||||
4. 用 agent-reach 配置 cookie(推荐):
|
## 获取 Cookie(Cookie-Editor 方式,推荐)
|
||||||
|
|
||||||
|
1. 安装 [Cookie-Editor](https://cookie-editor.com/) 浏览器扩展
|
||||||
|
2. 登录 x.com
|
||||||
|
3. 点击 Cookie-Editor 图标 → Export → 复制全部
|
||||||
|
4. 运行配置命令:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
|
agent-reach configure twitter-cookies "粘贴的 cookie JSON"
|
||||||
```
|
```
|
||||||
|
|
||||||
## 需要用户手动做的步骤
|
这会自动提取 `auth_token` 和 `ct0`,并写入环境变量。
|
||||||
|
|
||||||
请告诉用户:
|
## 手动设置 Cookie
|
||||||
|
|
||||||
> Twitter 高级功能需要你的 Twitter 账号 cookie(完全免费)。
|
如果你已经知道 `auth_token` 和 `ct0`:
|
||||||
>
|
|
||||||
> **最简单的方式:**
|
|
||||||
> 1. 安装 Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm)
|
|
||||||
> 2. 打开 https://x.com 并确保已登录
|
|
||||||
> 3. 点击 Cookie-Editor 插件图标 → Export → Header String
|
|
||||||
> 4. 把导出的内容发给我
|
|
||||||
>
|
|
||||||
> **手动方式:**
|
|
||||||
> 1. 用 Chrome 打开 https://x.com 并确保你已登录
|
|
||||||
> 2. 按 **F12** 打开开发者工具(Mac 按 Cmd+Option+I)
|
|
||||||
> 3. 点击顶部的 **Application**(应用)标签
|
|
||||||
> 4. 左侧找到 **Cookies** → **https://x.com**
|
|
||||||
> 5. 在列表中找到以下两个值,双击复制:
|
|
||||||
> - **auth_token** — 一串字母数字
|
|
||||||
> - **ct0** — 一串字母数字
|
|
||||||
> 6. 把这两个值发给我
|
|
||||||
>
|
|
||||||
> ⚠️ 这些 cookie 让我能以你的身份读取推文(只读)。我不会发推、点赞或做任何操作。
|
|
||||||
> ⚠️ cookie 大约 1-3 个月会过期,届时需要重新导出。
|
|
||||||
|
|
||||||
## Agent 收到 cookie 后的操作
|
1. 安装 twitter-cli(如果没装):`pipx install twitter-cli`
|
||||||
|
|
||||||
1. 安装 bird(如果没装):`npm install -g @steipete/bird`
|
2. 设置环境变量:
|
||||||
2. 配置 cookie:`agent-reach configure twitter-cookies "粘贴的内容"`
|
|
||||||
3. 测试:运行 `agent-reach doctor` 确认 Twitter 状态
|
```bash
|
||||||
4. 反馈:"✅ Twitter 高级功能已开启!现在可以搜索推文、查看时间线了。"
|
export AUTH_TOKEN="你的auth_token"
|
||||||
5. 如果失败:"❌ Cookie 无效或已过期,请重新导出。"
|
export CT0="你的ct0"
|
||||||
|
```
|
||||||
|
|
||||||
|
3. 测试:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
twitter search "test" -n 1
|
||||||
|
```
|
||||||
|
|
||||||
|
## 代理配置
|
||||||
|
|
||||||
|
> twitter-cli 支持通过环境变量设置代理:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export HTTP_PROXY="http://user:pass@host:port"
|
||||||
|
export HTTPS_PROXY="http://user:pass@host:port"
|
||||||
|
twitter search "test" -n 1
|
||||||
|
```
|
||||||
|
|
||||||
|
也可以使用全局代理工具:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
proxychains twitter search "test" -n 1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fallback:bird CLI
|
||||||
|
|
||||||
|
如果你已经安装了 [bird CLI](https://www.npmjs.com/package/@steipete/bird)(`npm install -g @steipete/bird`),它也能正常工作。Agent Reach 会自动检测并使用已安装的 bird。两者功能类似,twitter-cli 是当前推荐方案。
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ playwright install chromium
|
|||||||
|
|
||||||
3. 安装完成后测试:
|
3. 安装完成后测试:
|
||||||
```bash
|
```bash
|
||||||
agent-reach read "https://mp.weixin.qq.com/s/一个测试链接"
|
curl -s "https://r.jina.ai/https://mp.weixin.qq.com/s/一个测试链接" -H "Accept: text/markdown"
|
||||||
```
|
```
|
||||||
|
|
||||||
## 需要用户手动做的步骤
|
## 需要用户手动做的步骤
|
||||||
|
|||||||
@@ -1,71 +1,90 @@
|
|||||||
# 小红书配置指南
|
# 小红书配置指南
|
||||||
|
|
||||||
## 功能说明
|
## 功能说明
|
||||||
读取和搜索小红书笔记。通过 [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) 实现(⭐9K+,Go 语言,内置 Chrome 浏览器)。
|
读取和搜索小红书笔记。通过 [xhs-cli](https://github.com/jackwener/xiaohongshu-cli)(⭐1.5K,pipx 一行安装)实现。
|
||||||
|
|
||||||
## 前置条件
|
## 前置条件
|
||||||
- Docker(用来运行 xiaohongshu-mcp 服务)
|
- Python 3.10+(pipx 安装)
|
||||||
- mcporter CLI(MCP 协议桥接工具)
|
- 浏览器已登录 xiaohongshu.com(用于导出 Cookie)
|
||||||
|
|
||||||
## Agent 可自动完成的步骤
|
## Agent 可自动完成的步骤
|
||||||
|
|
||||||
### 1. 安装 mcporter
|
### 1. 安装 xhs-cli
|
||||||
```bash
|
```bash
|
||||||
npm install -g mcporter
|
pipx install xiaohongshu-cli
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. 启动 xiaohongshu-mcp 服务
|
### 2. 登录(从浏览器提取 Cookie)
|
||||||
```bash
|
```bash
|
||||||
docker run -d \
|
xhs login
|
||||||
--name xiaohongshu-mcp \
|
|
||||||
-p 18060:18060 \
|
|
||||||
xpzouying/xiaohongshu-mcp
|
|
||||||
```
|
```
|
||||||
|
|
||||||
> 如需代理(服务器部署推荐):
|
> 这会自动从浏览器提取 Cookie。如果自动提取失败,可以手动导入(见下方)。
|
||||||
> ```bash
|
|
||||||
> docker run -d \
|
|
||||||
> --name xiaohongshu-mcp \
|
|
||||||
> -p 18060:18060 \
|
|
||||||
> -e XHS_PROXY=http://user:pass@ip:port \
|
|
||||||
> xpzouying/xiaohongshu-mcp
|
|
||||||
> ```
|
|
||||||
|
|
||||||
### 3. 注册到 mcporter
|
### 3. 验证
|
||||||
```bash
|
|
||||||
mcporter config add xiaohongshu http://localhost:18060/mcp
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. 验证
|
|
||||||
```bash
|
```bash
|
||||||
agent-reach doctor
|
agent-reach doctor
|
||||||
```
|
```
|
||||||
|
|
||||||
应该看到小红书显示为 ✅ 或 ⚠️(MCP 已连接但未登录)。
|
应该看到小红书显示为 ✅。
|
||||||
|
|
||||||
## 需要用户手动做的步骤
|
## 需要用户手动做的步骤
|
||||||
|
|
||||||
如果 doctor 显示"MCP 已连接但未登录":
|
如果 `xhs login` 自动提取失败,需要手动导入 cookies:
|
||||||
|
|
||||||
> 小红书需要登录一次(之后会记住你的登录状态)。
|
> **推荐方式:Cookie-Editor 浏览器导出(最可靠)**
|
||||||
>
|
>
|
||||||
> 打开 http://localhost:18060 ,用手机小红书 App 扫描二维码登录。
|
> 1. 在 Chrome 中安装 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 扩展
|
||||||
> 登录后 cookie 会自动保存在 Docker 容器内,大约 1-3 个月有效。
|
> 2. 浏览器登录 xiaohongshu.com
|
||||||
|
> 3. 点击 Cookie-Editor 图标 → Export → Header String
|
||||||
|
> 4. 把导出的字符串发给 Agent,运行:`agent-reach configure xhs-cookies "导出的cookie字符串"`
|
||||||
|
>
|
||||||
|
> **注意**:不要依赖 QR 扫码登录,Cookie-Editor 导出方式最简单可靠。这个方式也适合 WSL、SSH、容器等无法直接读取桌面浏览器 Cookie 的环境。
|
||||||
|
|
||||||
|
`agent-reach configure xhs-cookies` 会把 Cookie 同步到 `~/.xiaohongshu-cli/cookies.json`,供 `xhs status/search/read` 直接使用;如果你还在用旧的 `xiaohongshu-mcp` Docker 方案,也会保留兼容导入文件。
|
||||||
|
|
||||||
|
## 使用示例
|
||||||
|
|
||||||
|
搜索笔记:
|
||||||
|
```bash
|
||||||
|
xhs search "关键词"
|
||||||
|
```
|
||||||
|
|
||||||
|
阅读笔记详情:
|
||||||
|
```bash
|
||||||
|
xhs read NOTE_ID
|
||||||
|
```
|
||||||
|
|
||||||
|
查看评论:
|
||||||
|
```bash
|
||||||
|
xhs comments NOTE_ID
|
||||||
|
```
|
||||||
|
|
||||||
## 常见问题
|
## 常见问题
|
||||||
|
|
||||||
**Q: Docker 容器重启后 cookie 丢了?**
|
**Q: Cookie 过期了?**
|
||||||
A: 挂载数据卷持久化:
|
A: 重新运行 `xhs login` 或通过 Cookie-Editor 重新导出。
|
||||||
|
|
||||||
|
**Q: 小红书提示 IP 风险?**
|
||||||
|
A: 推荐使用住宅代理:`export HTTP_PROXY="http://user:pass@ip:port"`。
|
||||||
|
|
||||||
|
**Q: xhs-cli 不支持我的系统?**
|
||||||
|
A: 确保 Python 3.10+ 和 pipx 已安装。运行 `pipx install xiaohongshu-cli` 即可。
|
||||||
|
|
||||||
|
**Q: WSL 里 `xhs login` 读不到 Windows 浏览器怎么办?**
|
||||||
|
A: 这是常见限制。WSL 里没有直接可读的 Linux 浏览器 Cookie 时,`xhs login` 自动提取会失败。推荐在 Windows 浏览器用 Cookie-Editor 导出 `xiaohongshu.com` 的 Header String,然后在 WSL 里运行 `agent-reach configure xhs-cookies "..."`,再用 `xhs status` 验证。
|
||||||
|
|
||||||
|
## 备选方案:Docker MCP
|
||||||
|
|
||||||
|
如果你已经在使用 [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) Docker 方案,它也能正常工作:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker run -d \
|
docker run -d \
|
||||||
--name xiaohongshu-mcp \
|
--name xiaohongshu-mcp \
|
||||||
-p 18060:18060 \
|
-p 18060:18060 \
|
||||||
-v xhs-data:/app/data \
|
|
||||||
xpzouying/xiaohongshu-mcp
|
xpzouying/xiaohongshu-mcp
|
||||||
|
|
||||||
|
mcporter config add xiaohongshu http://localhost:18060/mcp
|
||||||
```
|
```
|
||||||
|
|
||||||
**Q: 服务器上小红书提示 IP 风险?**
|
xhs-cli 是当前推荐方案,不需要 Docker,安装更简单。
|
||||||
A: 加代理参数 `-e XHS_PROXY=http://user:pass@ip:port`,推荐住宅代理。
|
|
||||||
|
|
||||||
**Q: 我不想用 Docker?**
|
|
||||||
A: 可以从源码编译:https://github.com/xpzouying/xiaohongshu-mcp
|
|
||||||
|
|||||||
@@ -1,10 +1,11 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Agent Reach MCP Server — expose all capabilities as MCP tools.
|
Agent Reach MCP Server — expose doctor/status as MCP tool.
|
||||||
|
|
||||||
Run: python -m agent_reach.integrations.mcp_server
|
Run: python -m agent_reach.integrations.mcp_server
|
||||||
|
|
||||||
8 tools for any MCP-compatible AI Agent.
|
Agent Reach is an installer + doctor tool. For actual reading/searching,
|
||||||
|
agents should call upstream tools directly (twitter-cli, yt-dlp, mcporter, etc.).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
@@ -35,50 +36,15 @@ def create_server():
|
|||||||
@server.list_tools()
|
@server.list_tools()
|
||||||
async def list_tools():
|
async def list_tools():
|
||||||
return [
|
return [
|
||||||
Tool(name="read_url",
|
|
||||||
description="Read content from any URL. Supports: web, GitHub, Reddit, Twitter, YouTube, Bilibili, RSS.",
|
|
||||||
inputSchema={"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}),
|
|
||||||
Tool(name="read_batch",
|
|
||||||
description="Read multiple URLs concurrently.",
|
|
||||||
inputSchema={"type": "object", "properties": {"urls": {"type": "array", "items": {"type": "string"}}}, "required": ["urls"]}),
|
|
||||||
Tool(name="detect_platform",
|
|
||||||
description="Detect what platform a URL belongs to.",
|
|
||||||
inputSchema={"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}),
|
|
||||||
Tool(name="search",
|
|
||||||
description="Semantic web search via Exa.",
|
|
||||||
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "num_results": {"type": "integer", "default": 5}}, "required": ["query"]}),
|
|
||||||
Tool(name="search_reddit",
|
|
||||||
description="Search Reddit posts.",
|
|
||||||
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "subreddit": {"type": "string"}, "limit": {"type": "integer", "default": 10}}, "required": ["query"]}),
|
|
||||||
Tool(name="search_github",
|
|
||||||
description="Search GitHub repositories.",
|
|
||||||
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "language": {"type": "string"}, "limit": {"type": "integer", "default": 5}}, "required": ["query"]}),
|
|
||||||
Tool(name="search_twitter",
|
|
||||||
description="Search Twitter/X posts.",
|
|
||||||
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "limit": {"type": "integer", "default": 10}}, "required": ["query"]}),
|
|
||||||
Tool(name="get_status",
|
Tool(name="get_status",
|
||||||
description="Get Agent Reach status: which channels are active.",
|
description="Get Agent Reach status: which channels are installed and active.",
|
||||||
inputSchema={"type": "object", "properties": {}}),
|
inputSchema={"type": "object", "properties": {}}),
|
||||||
]
|
]
|
||||||
|
|
||||||
@server.call_tool()
|
@server.call_tool()
|
||||||
async def call_tool(name: str, arguments: dict):
|
async def call_tool(name: str, arguments: dict):
|
||||||
try:
|
try:
|
||||||
if name == "read_url":
|
if name == "get_status":
|
||||||
result = await eyes.read(arguments["url"])
|
|
||||||
elif name == "read_batch":
|
|
||||||
result = await eyes.read_batch(arguments["urls"])
|
|
||||||
elif name == "detect_platform":
|
|
||||||
result = eyes.detect_platform(arguments["url"])
|
|
||||||
elif name == "search":
|
|
||||||
result = await eyes.search(arguments["query"], arguments.get("num_results", 5))
|
|
||||||
elif name == "search_reddit":
|
|
||||||
result = await eyes.search_reddit(arguments["query"], arguments.get("subreddit"), arguments.get("limit", 10))
|
|
||||||
elif name == "search_github":
|
|
||||||
result = await eyes.search_github(arguments["query"], arguments.get("language"), arguments.get("limit", 5))
|
|
||||||
elif name == "search_twitter":
|
|
||||||
result = await eyes.search_twitter(arguments["query"], arguments.get("limit", 10))
|
|
||||||
elif name == "get_status":
|
|
||||||
result = eyes.doctor_report()
|
result = eyes.doctor_report()
|
||||||
else:
|
else:
|
||||||
result = f"Unknown tool: {name}"
|
result = f"Unknown tool: {name}"
|
||||||
|
|||||||
@@ -1,65 +0,0 @@
|
|||||||
---
|
|
||||||
name: agent-reach
|
|
||||||
description: "Give your AI Agent eyes to see the entire internet. Search + read 10+ platforms with one install."
|
|
||||||
install: pip install git+https://github.com/Panniantong/agent-reach.git
|
|
||||||
---
|
|
||||||
|
|
||||||
# Agent Reach — 给 Agent 装上眼睛 👁️
|
|
||||||
|
|
||||||
安装后你就能搜索和阅读全网内容。
|
|
||||||
|
|
||||||
## 安装
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install git+https://github.com/Panniantong/agent-reach.git
|
|
||||||
```
|
|
||||||
|
|
||||||
## 装完直接能用的(零配置)
|
|
||||||
|
|
||||||
- `agent-reach read <URL>` — 读取任意网页、GitHub、B站、YouTube、RSS
|
|
||||||
- `agent-reach search-github "query"` — 搜索 GitHub 仓库
|
|
||||||
- `agent-reach doctor` — 查看所有平台状态
|
|
||||||
|
|
||||||
## 解锁更多功能
|
|
||||||
|
|
||||||
运行 `agent-reach configure` 进行交互式配置,或者读取以下指南:
|
|
||||||
|
|
||||||
| 功能 | 指南文件 |
|
|
||||||
|------|---------|
|
|
||||||
| 全网搜索 + Reddit搜索 + Twitter搜索 | `agent_reach/guides/setup-exa.md` |
|
|
||||||
| Reddit 完整阅读 | `agent_reach/guides/setup-reddit.md` |
|
|
||||||
| Twitter 高级(时间线/线程) | `agent_reach/guides/setup-twitter.md` |
|
|
||||||
| 小红书 | `agent_reach/guides/setup-xiaohongshu.md` |
|
|
||||||
| 微信公众号 | `agent_reach/guides/setup-wechat.md` |
|
|
||||||
| 视频语音转文字 | `agent_reach/guides/setup-groq.md` |
|
|
||||||
|
|
||||||
每个指南里有详细的步骤说明,你(Agent)可以照着做,只有需要人类操作的部分(登录、复制 key)才需要问用户。
|
|
||||||
|
|
||||||
## MCP Server
|
|
||||||
|
|
||||||
如果你的 Agent 平台支持 MCP:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install agent-reach[mcp]
|
|
||||||
python -m agent_reach.integrations.mcp_server
|
|
||||||
```
|
|
||||||
|
|
||||||
提供 8 个工具:read_url, read_batch, detect_platform, search, search_reddit, search_github, search_twitter, get_status
|
|
||||||
|
|
||||||
## Python API
|
|
||||||
|
|
||||||
```python
|
|
||||||
from agent_reach import AgentReach
|
|
||||||
import asyncio
|
|
||||||
|
|
||||||
eyes = AgentReach()
|
|
||||||
|
|
||||||
# 读取
|
|
||||||
result = asyncio.run(eyes.read("https://github.com/openai/gpt-4"))
|
|
||||||
|
|
||||||
# 搜索
|
|
||||||
results = asyncio.run(eyes.search("AI agent framework"))
|
|
||||||
|
|
||||||
# 健康检查
|
|
||||||
print(eyes.doctor_report())
|
|
||||||
```
|
|
||||||
Executable
+167
@@ -0,0 +1,167 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# 小宇宙播客转文字脚本
|
||||||
|
# 用法: bash transcribe.sh <小宇宙链接> [输出文件路径]
|
||||||
|
# 环境变量: GROQ_API_KEY (必须)
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
URL="${1:?用法: bash transcribe.sh <小宇宙链接> [输出文件路径]}"
|
||||||
|
OUTPUT="${2:-/tmp/podcast_transcript.txt}"
|
||||||
|
TMPDIR="/tmp/xiaoyuzhou_$$"
|
||||||
|
|
||||||
|
# Try env var first, then agent-reach config.yaml
|
||||||
|
if [ -z "$GROQ_API_KEY" ]; then
|
||||||
|
CONFIG_FILE="$HOME/.agent-reach/config.yaml"
|
||||||
|
if [ -f "$CONFIG_FILE" ]; then
|
||||||
|
GROQ_API_KEY=$(python3 -c "import yaml; print((yaml.safe_load(open('$CONFIG_FILE')) or {}).get('groq_api_key',''))" 2>/dev/null || true)
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
GROQ_API_KEY="${GROQ_API_KEY:?请设置 GROQ_API_KEY 环境变量或运行 agent-reach configure groq-key}"
|
||||||
|
|
||||||
|
# Groq API 限制: 25MB per file
|
||||||
|
MAX_CHUNK_SIZE_MB=20
|
||||||
|
AUDIO_BITRATE="64k"
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
rm -rf "$TMPDIR"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
mkdir -p "$TMPDIR"
|
||||||
|
|
||||||
|
echo "📻 小宇宙播客转文字"
|
||||||
|
echo "===================="
|
||||||
|
|
||||||
|
# Step 1: 提取音频 URL 和标题
|
||||||
|
echo "🔍 正在解析页面..."
|
||||||
|
PAGE=$(curl -s "$URL")
|
||||||
|
AUDIO_URL=$(echo "$PAGE" | grep -oP 'https://media\.xyzcdn\.net/[^"]*\.(m4a|mp3)' | head -1)
|
||||||
|
TITLE=$(echo "$PAGE" | grep -oP '"title":"[^"]*"' | head -1 | sed 's/"title":"//;s/"//')
|
||||||
|
|
||||||
|
if [ -z "$AUDIO_URL" ]; then
|
||||||
|
echo "❌ 无法从页面提取音频链接"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "📝 标题: $TITLE"
|
||||||
|
echo "🔗 音频: $AUDIO_URL"
|
||||||
|
|
||||||
|
# Step 2: 下载音频
|
||||||
|
echo "⬇️ 正在下载音频..."
|
||||||
|
EXT="${AUDIO_URL##*.}"
|
||||||
|
curl -sL -o "$TMPDIR/original.$EXT" "$AUDIO_URL"
|
||||||
|
FILE_SIZE=$(ls -lh "$TMPDIR/original.$EXT" | awk '{print $5}')
|
||||||
|
echo "📦 文件大小: $FILE_SIZE"
|
||||||
|
|
||||||
|
# Step 3: 获取时长
|
||||||
|
DURATION=$(ffprobe -v quiet -show_entries format=duration -of csv=p=0 "$TMPDIR/original.$EXT" 2>/dev/null | cut -d. -f1)
|
||||||
|
DURATION_MIN=$((DURATION / 60))
|
||||||
|
DURATION_SEC=$((DURATION % 60))
|
||||||
|
echo "⏱️ 时长: ${DURATION_MIN}分${DURATION_SEC}秒"
|
||||||
|
|
||||||
|
# Step 4: 转为低码率单声道 MP3
|
||||||
|
echo "🔄 正在转码..."
|
||||||
|
ffmpeg -y -i "$TMPDIR/original.$EXT" -b:a "$AUDIO_BITRATE" -ac 1 "$TMPDIR/mono.mp3" 2>/dev/null
|
||||||
|
MONO_SIZE=$(stat -c%s "$TMPDIR/mono.mp3" 2>/dev/null || stat -f%z "$TMPDIR/mono.mp3")
|
||||||
|
echo "📦 转码后: $(echo "$MONO_SIZE / 1024 / 1024" | bc)MB"
|
||||||
|
|
||||||
|
# Step 5: 按大小切片
|
||||||
|
MAX_BYTES=$((MAX_CHUNK_SIZE_MB * 1024 * 1024))
|
||||||
|
|
||||||
|
if [ "$MONO_SIZE" -le "$MAX_BYTES" ]; then
|
||||||
|
# 不需要切片
|
||||||
|
cp "$TMPDIR/mono.mp3" "$TMPDIR/chunk_0.mp3"
|
||||||
|
NUM_CHUNKS=1
|
||||||
|
echo "📎 无需切片"
|
||||||
|
else
|
||||||
|
# 计算需要几个 chunk
|
||||||
|
NUM_CHUNKS=$(( (MONO_SIZE / MAX_BYTES) + 1 ))
|
||||||
|
CHUNK_DURATION=$(( DURATION / NUM_CHUNKS + 10 )) # 加 10 秒缓冲
|
||||||
|
echo "✂️ 切分为 $NUM_CHUNKS 段 (每段约 $((CHUNK_DURATION / 60)) 分钟)..."
|
||||||
|
|
||||||
|
for i in $(seq 0 $((NUM_CHUNKS - 1))); do
|
||||||
|
START=$((i * CHUNK_DURATION))
|
||||||
|
ffmpeg -y -i "$TMPDIR/mono.mp3" -ss "$START" -t "$CHUNK_DURATION" -c copy "$TMPDIR/chunk_${i}.mp3" 2>/dev/null
|
||||||
|
CHUNK_SIZE=$(ls -lh "$TMPDIR/chunk_${i}.mp3" | awk '{print $5}')
|
||||||
|
echo " 段 $((i+1))/$NUM_CHUNKS: $CHUNK_SIZE"
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Step 6: 调用 Groq Whisper API 转录
|
||||||
|
echo "🎙️ 正在转录 (Groq Whisper large-v3)..."
|
||||||
|
|
||||||
|
for i in $(seq 0 $((NUM_CHUNKS - 1))); do
|
||||||
|
echo -n " 段 $((i+1))/$NUM_CHUNKS... "
|
||||||
|
|
||||||
|
RESPONSE=$(curl -s -w "\n%{http_code}" \
|
||||||
|
https://api.groq.com/openai/v1/audio/transcriptions \
|
||||||
|
-H "Authorization: Bearer $GROQ_API_KEY" \
|
||||||
|
-F file="@$TMPDIR/chunk_${i}.mp3" \
|
||||||
|
-F model="whisper-large-v3" \
|
||||||
|
-F language="zh" \
|
||||||
|
-F response_format="text")
|
||||||
|
|
||||||
|
HTTP_CODE=$(echo "$RESPONSE" | tail -1)
|
||||||
|
BODY=$(echo "$RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$HTTP_CODE" != "200" ]; then
|
||||||
|
echo "❌ API 错误 (HTTP $HTTP_CODE)"
|
||||||
|
echo "$BODY"
|
||||||
|
|
||||||
|
# 如果是速率限制,等待后重试
|
||||||
|
if [ "$HTTP_CODE" = "429" ]; then
|
||||||
|
# 从错误信息中提取等待时间,默认 120 秒
|
||||||
|
WAIT_SEC=$(echo "$BODY" | grep -oP 'in \K[0-9]+m' | sed 's/m//' | head -1)
|
||||||
|
WAIT_SEC=${WAIT_SEC:-2}
|
||||||
|
WAIT_SEC=$((WAIT_SEC * 60 + 30))
|
||||||
|
echo " ⏳ 速率限制,等待 ${WAIT_SEC} 秒后重试..."
|
||||||
|
sleep "$WAIT_SEC"
|
||||||
|
RESPONSE=$(curl -s -w "\n%{http_code}" \
|
||||||
|
https://api.groq.com/openai/v1/audio/transcriptions \
|
||||||
|
-H "Authorization: Bearer $GROQ_API_KEY" \
|
||||||
|
-F file="@$TMPDIR/chunk_${i}.mp3" \
|
||||||
|
-F model="whisper-large-v3" \
|
||||||
|
-F language="zh" \
|
||||||
|
-F response_format="text")
|
||||||
|
HTTP_CODE=$(echo "$RESPONSE" | tail -1)
|
||||||
|
BODY=$(echo "$RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$HTTP_CODE" != "200" ]; then
|
||||||
|
echo " ❌ 重试失败"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$BODY" > "$TMPDIR/transcript_${i}.txt"
|
||||||
|
CHARS=$(wc -m < "$TMPDIR/transcript_${i}.txt")
|
||||||
|
echo "✅ ($CHARS 字)"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 7: 合并输出
|
||||||
|
echo "📄 正在合并文字稿..."
|
||||||
|
|
||||||
|
{
|
||||||
|
echo "# $TITLE"
|
||||||
|
echo ""
|
||||||
|
echo "来源: $URL"
|
||||||
|
echo "时长: ${DURATION_MIN}分${DURATION_SEC}秒"
|
||||||
|
echo "转录时间: $(date '+%Y-%m-%d %H:%M')"
|
||||||
|
echo ""
|
||||||
|
echo "---"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
for i in $(seq 0 $((NUM_CHUNKS - 1))); do
|
||||||
|
cat "$TMPDIR/transcript_${i}.txt"
|
||||||
|
echo ""
|
||||||
|
done
|
||||||
|
} > "$OUTPUT"
|
||||||
|
|
||||||
|
TOTAL_CHARS=$(wc -m < "$OUTPUT")
|
||||||
|
echo ""
|
||||||
|
echo "✅ 完成!"
|
||||||
|
echo "📄 输出: $OUTPUT"
|
||||||
|
echo "📊 总字数: $TOTAL_CHARS"
|
||||||
|
echo "===================="
|
||||||
+90
-71
@@ -1,87 +1,106 @@
|
|||||||
---
|
---
|
||||||
name: agent-reach
|
name: agent-reach
|
||||||
description: >
|
description: >
|
||||||
Give your AI agent eyes to see the entire internet. Read and search across
|
Give your AI agent eyes to see the entire internet.
|
||||||
Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, RSS, and any web page
|
17 platforms via CLI, MCP, curl, and Python scripts.
|
||||||
— all from a single CLI. Use when: (1) reading content from URLs (tweets, Reddit posts,
|
Zero config for 8 channels.
|
||||||
articles, videos), (2) searching across platforms (web, Twitter, Reddit, GitHub, YouTube,
|
|
||||||
Bilibili, XiaoHongShu), (3) checking channel health or updating Agent Reach.
|
【路由方式】SKILL.md 包含路由表和常用命令,复杂场景需按需阅读对应分类的 references/*.md。
|
||||||
Triggers: "search Twitter/Reddit/YouTube", "read this URL", "find posts about",
|
分类:search / social (小红书/抖音/微博/推特/B站/V2EX/Reddit) / career(LinkedIn) / dev(github) / web(网页/文章/公众号/RSS) / video(YouTube/B站/播客).
|
||||||
"搜索", "读取", "查一下", "看看这个链接".
|
|
||||||
|
Use when user asks to search, read, or interact on any supported platform,
|
||||||
|
shares a URL, or asks to search the web.
|
||||||
|
triggers:
|
||||||
|
- search: 搜/查/找/search/搜索/查一下/帮我搜
|
||||||
|
- social:
|
||||||
|
- 小红书: xiaohongshu/xhs/小红书/红书
|
||||||
|
- 抖音: douyin/抖音
|
||||||
|
- Twitter: twitter/推特/x.com/推文
|
||||||
|
- 微博: weibo/微博
|
||||||
|
- B站: bilibili/b站/哔哩哔哩
|
||||||
|
- V2EX: v2ex
|
||||||
|
- Reddit: reddit
|
||||||
|
- career: 招聘/职位/求职/linkedin/领英/找工作
|
||||||
|
- dev: github/代码/仓库/gh/issue/pr/分支/commit
|
||||||
|
- web: 网页/链接/文章/公众号/微信文章/rss/读一下/打开这个
|
||||||
|
- video: youtube/视频/播客/字幕/小宇宙/转录/yt
|
||||||
|
- finance: 雪球/股票/stock/xueqiu/行情/基金
|
||||||
|
metadata:
|
||||||
|
openclaw:
|
||||||
|
homepage: https://github.com/Panniantong/Agent-Reach
|
||||||
---
|
---
|
||||||
|
|
||||||
# Agent Reach
|
# Agent Reach — 路由器
|
||||||
|
|
||||||
Read and search the internet across 9+ platforms via unified CLI.
|
17 平台工具集合。根据用户意图选择对应分类。
|
||||||
|
|
||||||
## Setup
|
## 路由表
|
||||||
|
|
||||||
|
| 用户意图 | 分类 | 详细文档 |
|
||||||
|
|---------|------|---------|
|
||||||
|
| 网页搜索/代码搜索 | search | [references/search.md](references/search.md) |
|
||||||
|
| 小红书/抖音/微博/推特/B站/V2EX/Reddit | social | [references/social.md](references/social.md) |
|
||||||
|
| 招聘/职位/LinkedIn | career | [references/career.md](references/career.md) |
|
||||||
|
| GitHub/代码 | dev | [references/dev.md](references/dev.md) |
|
||||||
|
| 网页/文章/公众号/RSS | web | [references/web.md](references/web.md) |
|
||||||
|
| YouTube/B站/播客字幕 | video | [references/video.md](references/video.md) |
|
||||||
|
|
||||||
|
## 零配置快速命令
|
||||||
|
|
||||||
First check if agent-reach is installed:
|
|
||||||
```bash
|
```bash
|
||||||
|
# Exa 网页搜索
|
||||||
|
mcporter call 'exa.web_search_exa(query: "query", numResults: 5)'
|
||||||
|
|
||||||
|
# 通用网页阅读
|
||||||
|
curl -s "https://r.jina.ai/URL"
|
||||||
|
|
||||||
|
# GitHub 搜索
|
||||||
|
gh search repos "query" --sort stars --limit 10
|
||||||
|
|
||||||
|
# Twitter 搜索
|
||||||
|
twitter search "query" --limit 10
|
||||||
|
|
||||||
|
# YouTube/B站字幕
|
||||||
|
yt-dlp --write-sub --skip-download -o "/tmp/%(id)s" "URL"
|
||||||
|
|
||||||
|
# Reddit 搜索
|
||||||
|
rdt search "query" --limit 10
|
||||||
|
|
||||||
|
# Reddit 读帖 + 评论
|
||||||
|
rdt read POST_ID
|
||||||
|
|
||||||
|
# V2EX 热门
|
||||||
|
curl -s "https://www.v2ex.com/api/topics/hot.json" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
## 环境检查
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 检查可用 channel
|
||||||
agent-reach doctor
|
agent-reach doctor
|
||||||
|
|
||||||
|
# 查看所有 MCP 服务
|
||||||
|
mcporter_list_servers()
|
||||||
```
|
```
|
||||||
|
|
||||||
If command not found, install it:
|
## 工作区规则
|
||||||
```bash
|
|
||||||
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
|
||||||
agent-reach install --env=auto
|
|
||||||
```
|
|
||||||
|
|
||||||
`install` auto-detects your environment and installs all dependencies (Node.js, mcporter, bird CLI, gh CLI). Read the output and run `agent-reach doctor` to see what's active.
|
**不要在 agent workspace 创建文件。** 使用 `/tmp/` 存放临时输出,`~/.agent-reach/` 存放持久数据。
|
||||||
|
|
||||||
For channels that need user input, ask the user. See the full setup guide:
|
## 详细文档
|
||||||
|
|
||||||
|
根据用户需求,阅读对应的详细文档:
|
||||||
|
|
||||||
|
- [搜索工具](references/search.md) — Exa AI 搜索
|
||||||
|
- [社交媒体](references/social.md) — 小红书, 抖音, Twitter, B站, V2EX, Reddit
|
||||||
|
- [职场招聘](references/career.md) — LinkedIn
|
||||||
|
- [开发工具](references/dev.md) — GitHub CLI
|
||||||
|
- [网页阅读](references/web.md) — Jina Reader, 微信公众号, RSS
|
||||||
|
- [视频播客](references/video.md) — YouTube, B站, 小宇宙
|
||||||
|
|
||||||
|
## 配置渠道
|
||||||
|
|
||||||
|
如果某个 channel 需要配置,获取安装指南:
|
||||||
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
||||||
|
|
||||||
## Commands
|
用户只需提供 cookies,其他配置由 agent 完成。
|
||||||
|
|
||||||
### Read any URL
|
|
||||||
```bash
|
|
||||||
agent-reach read <url>
|
|
||||||
agent-reach read <url> --json # structured output
|
|
||||||
```
|
|
||||||
Handles: tweets, Reddit posts, articles, YouTube (transcripts), GitHub repos, etc.
|
|
||||||
|
|
||||||
### Search
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach search "query" # web search (Exa)
|
|
||||||
agent-reach search-twitter "query" # Twitter/X
|
|
||||||
agent-reach search-reddit "query" # Reddit (--sub <subreddit>)
|
|
||||||
agent-reach search-github "query" # GitHub (--lang <language>)
|
|
||||||
agent-reach search-youtube "query" # YouTube
|
|
||||||
agent-reach search-bilibili "query" # Bilibili (B站)
|
|
||||||
agent-reach search-xhs "query" # XiaoHongShu (小红书)
|
|
||||||
```
|
|
||||||
|
|
||||||
All search commands support `-n <count>` for number of results.
|
|
||||||
|
|
||||||
### Management
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach doctor # channel status overview
|
|
||||||
agent-reach watch # quick health + update check (for scheduled tasks)
|
|
||||||
agent-reach check-update # check for new versions
|
|
||||||
```
|
|
||||||
|
|
||||||
### Configure channels
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
|
|
||||||
agent-reach configure proxy http://user:pass@ip:port
|
|
||||||
agent-reach configure --from-browser chrome # auto-extract cookies
|
|
||||||
```
|
|
||||||
|
|
||||||
## Channel Status Tiers
|
|
||||||
|
|
||||||
- **Tier 0 (zero config):** Web, YouTube, RSS, Twitter (read-only via Jina)
|
|
||||||
- **Tier 1 (free setup):** Exa web search (mcporter required)
|
|
||||||
- **Tier 2 (user config):** Twitter search (cookie), Reddit full (proxy), GitHub (token), Bilibili (proxy), XiaoHongShu (MCP)
|
|
||||||
|
|
||||||
Run `agent-reach doctor` to see which channels are active.
|
|
||||||
|
|
||||||
## Tips
|
|
||||||
|
|
||||||
- Always try `agent-reach read <url>` first for any URL — it auto-detects the platform
|
|
||||||
- For Twitter cookies, recommend the user install [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome extension
|
|
||||||
- Reddit and Bilibili block server IPs — suggest a residential proxy (~$1/month) if on a server
|
|
||||||
- If a channel breaks, run `agent-reach doctor` to diagnose
|
|
||||||
|
|||||||
@@ -0,0 +1,312 @@
|
|||||||
|
---
|
||||||
|
name: agent-reach
|
||||||
|
description: >
|
||||||
|
Give your AI agent eyes to see the entire internet.
|
||||||
|
Search and read 17 platforms: Twitter/X, Reddit, YouTube, GitHub, Bilibili,
|
||||||
|
XiaoHongShu, Douyin, Weibo, WeChat Articles, Xiaoyuzhou Podcast, LinkedIn,
|
||||||
|
V2EX, Xueqiu, RSS, Exa web search, and any web page.
|
||||||
|
Zero config for 8 channels. Use when the user asks to search, read, or interact
|
||||||
|
on any supported platform, shares a URL, or asks to search the web.
|
||||||
|
Triggers: "search twitter", "search xiaohongshu", "watch this video",
|
||||||
|
"search the web", "look this up", "research", "youtube transcript",
|
||||||
|
"search reddit", "read this link", "bilibili", "douyin video",
|
||||||
|
"wechat article", "wechat official account", "weibo", "V2EX",
|
||||||
|
"xiaoyuzhou", "podcast", "xueqiu", "stock quote",
|
||||||
|
"install agent reach".
|
||||||
|
metadata:
|
||||||
|
openclaw:
|
||||||
|
homepage: https://github.com/Panniantong/Agent-Reach
|
||||||
|
---
|
||||||
|
|
||||||
|
# Agent Reach — Usage Guide
|
||||||
|
|
||||||
|
Upstream tools for 17 platforms. Call them directly.
|
||||||
|
|
||||||
|
Run `agent-reach doctor` to check which channels are available.
|
||||||
|
|
||||||
|
## ⚠️ Workspace Rules
|
||||||
|
|
||||||
|
**Never create files in the agent workspace.** Use `/tmp/` for temporary output and `~/.agent-reach/` for persistent data.
|
||||||
|
|
||||||
|
## Web — Any URL
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "https://r.jina.ai/URL"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Web Search (Exa)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mcporter call 'exa.web_search_exa(query: "query", numResults: 5)'
|
||||||
|
mcporter call 'exa.get_code_context_exa(query: "code question", tokensNum: 3000)'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Twitter/X (bird)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bird search "query" -n 10 # search
|
||||||
|
bird read URL_OR_ID # read tweet (supports /status/ and /article/ URLs)
|
||||||
|
bird user-tweets @username -n 20 # user timeline
|
||||||
|
bird thread URL_OR_ID # full thread
|
||||||
|
```
|
||||||
|
|
||||||
|
## YouTube (yt-dlp)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yt-dlp --dump-json "URL" # video metadata
|
||||||
|
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --skip-download -o "/tmp/%(id)s" "URL"
|
||||||
|
# download subtitles, then read the .vtt file
|
||||||
|
yt-dlp --dump-json "ytsearch5:query" # search
|
||||||
|
```
|
||||||
|
|
||||||
|
## Bilibili (yt-dlp)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yt-dlp --dump-json "https://www.bilibili.com/video/BVxxx"
|
||||||
|
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --convert-subs vtt --skip-download -o "/tmp/%(id)s" "URL"
|
||||||
|
```
|
||||||
|
|
||||||
|
> Server IPs may get 412. Use `--cookies-from-browser chrome` or configure a proxy.
|
||||||
|
|
||||||
|
## Reddit
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "https://www.reddit.com/r/SUBREDDIT/hot.json?limit=10" -H "User-Agent: agent-reach/1.0"
|
||||||
|
curl -s "https://www.reddit.com/search.json?q=QUERY&limit=10" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
> Server IPs may get 403. Search via Exa instead, or configure a proxy.
|
||||||
|
|
||||||
|
## GitHub (gh CLI)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
gh search repos "query" --sort stars --limit 10
|
||||||
|
gh repo view owner/repo
|
||||||
|
gh search code "query" --language python
|
||||||
|
gh issue list -R owner/repo --state open
|
||||||
|
gh issue view 123 -R owner/repo
|
||||||
|
```
|
||||||
|
|
||||||
|
## XiaoHongShu (mcporter)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mcporter call 'xiaohongshu.search_feeds(keyword: "query")'
|
||||||
|
mcporter call 'xiaohongshu.get_feed_detail(feed_id: "xxx", xsec_token: "yyy")'
|
||||||
|
mcporter call 'xiaohongshu.get_feed_detail(feed_id: "xxx", xsec_token: "yyy", load_all_comments: true)'
|
||||||
|
mcporter call 'xiaohongshu.publish_content(title: "Title", content: "Body text", images: ["/path/img.jpg"], tags: ["tag"])'
|
||||||
|
```
|
||||||
|
|
||||||
|
> Requires login. Use Cookie-Editor to import cookies.
|
||||||
|
|
||||||
|
> **Tip: Clean bloated output.** The XHS API returns large JSON with many unused fields.
|
||||||
|
> Pipe through the formatter to save context:
|
||||||
|
> ```bash
|
||||||
|
> mcporter call 'xiaohongshu.search_feeds(keyword: "query")' | agent-reach format xhs
|
||||||
|
> ```
|
||||||
|
> This keeps only: title, content, author, engagement counts, image URLs, and tags.
|
||||||
|
|
||||||
|
## Douyin (mcporter)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mcporter call 'douyin.parse_douyin_video_info(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
mcporter call 'douyin.get_douyin_download_link(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
```
|
||||||
|
|
||||||
|
> No login needed.
|
||||||
|
|
||||||
|
## WeChat Articles
|
||||||
|
|
||||||
|
**Search** (`miku_ai`):
|
||||||
|
```bash
|
||||||
|
# miku_ai is installed inside the agent-reach Python environment.
|
||||||
|
# Use the same interpreter that runs agent-reach (handles pipx / venv installs):
|
||||||
|
AGENT_REACH_PYTHON=$(python3 -c "import agent_reach, sys; print(sys.executable)" 2>/dev/null || echo python3)
|
||||||
|
$AGENT_REACH_PYTHON -c "
|
||||||
|
import asyncio
|
||||||
|
from miku_ai import get_wexin_article
|
||||||
|
async def s():
|
||||||
|
for a in await get_wexin_article('query', 5):
|
||||||
|
print(f'{a[\"title\"]} | {a[\"url\"]}')
|
||||||
|
asyncio.run(s())
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Read** (Camoufox — bypasses WeChat anti-bot):
|
||||||
|
```bash
|
||||||
|
cd ~/.agent-reach/tools/wechat-article-for-ai && python3 main.py "https://mp.weixin.qq.com/s/ARTICLE_ID"
|
||||||
|
```
|
||||||
|
|
||||||
|
> WeChat articles cannot be read with Jina Reader or curl. Use Camoufox.
|
||||||
|
|
||||||
|
## Weibo (mcporter)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Trending topics
|
||||||
|
mcporter call 'weibo.get_trendings(limit: 20)'
|
||||||
|
|
||||||
|
# Search users
|
||||||
|
mcporter call 'weibo.search_users(keyword: "Lei Jun", limit: 10)'
|
||||||
|
|
||||||
|
# Get a user profile
|
||||||
|
mcporter call 'weibo.get_profile(uid: "1195230310")'
|
||||||
|
|
||||||
|
# Get a user's feed
|
||||||
|
mcporter call 'weibo.get_feeds(uid: "1195230310", limit: 20)'
|
||||||
|
|
||||||
|
# Get a user's hot posts
|
||||||
|
mcporter call 'weibo.get_hot_feeds(uid: "1195230310", limit: 10)'
|
||||||
|
|
||||||
|
# Search post content
|
||||||
|
mcporter call 'weibo.search_content(keyword: "artificial intelligence", limit: 20)'
|
||||||
|
|
||||||
|
# Search topics
|
||||||
|
mcporter call 'weibo.search_topics(keyword: "AI", limit: 10)'
|
||||||
|
|
||||||
|
# Get post comments
|
||||||
|
mcporter call 'weibo.get_comments(mid: "5099916367123456", limit: 50)'
|
||||||
|
|
||||||
|
# Get fans
|
||||||
|
mcporter call 'weibo.get_fans(uid: "1195230310", limit: 20)'
|
||||||
|
|
||||||
|
# Get followings
|
||||||
|
mcporter call 'weibo.get_followers(uid: "1195230310", limit: 20)'
|
||||||
|
```
|
||||||
|
|
||||||
|
> Zero config. No login needed. Uses the mobile API with auto-generated visitor cookies.
|
||||||
|
|
||||||
|
## Xiaoyuzhou Podcast (groq-whisper + ffmpeg)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Transcribe a single podcast episode (outputs text to /tmp/)
|
||||||
|
~/.agent-reach/tools/xiaoyuzhou/transcribe.sh "https://www.xiaoyuzhoufm.com/episode/EPISODE_ID"
|
||||||
|
```
|
||||||
|
|
||||||
|
> Requires `ffmpeg` and a Groq API key (free).
|
||||||
|
> Configure the key with `agent-reach configure groq-key YOUR_KEY`.
|
||||||
|
> On first run, install the tools with `agent-reach install --env=auto`.
|
||||||
|
> Run `agent-reach doctor` to check status.
|
||||||
|
> Output Markdown files are saved to `/tmp/` by default.
|
||||||
|
|
||||||
|
## LinkedIn (mcporter)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mcporter call 'linkedin.get_person_profile(linkedin_url: "https://linkedin.com/in/username")'
|
||||||
|
mcporter call 'linkedin.search_people(keyword: "AI engineer", limit: 10)'
|
||||||
|
```
|
||||||
|
|
||||||
|
Fallback: `curl -s "https://r.jina.ai/https://linkedin.com/in/username"`
|
||||||
|
|
||||||
|
## V2EX (public API)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Hot topics
|
||||||
|
curl -s "https://www.v2ex.com/api/topics/hot.json" -H "User-Agent: agent-reach/1.0"
|
||||||
|
|
||||||
|
# Topics in a node (node_name examples: python, tech, jobs, qna)
|
||||||
|
curl -s "https://www.v2ex.com/api/topics/show.json?node_name=python&page=1" -H "User-Agent: agent-reach/1.0"
|
||||||
|
|
||||||
|
# Topic details (extract topic_id from URLs like https://www.v2ex.com/t/1234567)
|
||||||
|
curl -s "https://www.v2ex.com/api/topics/show.json?id=TOPIC_ID" -H "User-Agent: agent-reach/1.0"
|
||||||
|
|
||||||
|
# Topic replies
|
||||||
|
curl -s "https://www.v2ex.com/api/replies/show.json?topic_id=TOPIC_ID&page=1" -H "User-Agent: agent-reach/1.0"
|
||||||
|
|
||||||
|
# User profile
|
||||||
|
curl -s "https://www.v2ex.com/api/members/show.json?username=USERNAME" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
Python example (`V2EXChannel`):
|
||||||
|
|
||||||
|
```python
|
||||||
|
from agent_reach.channels.v2ex import V2EXChannel
|
||||||
|
|
||||||
|
ch = V2EXChannel()
|
||||||
|
|
||||||
|
# Get hot topics (default 20 items)
|
||||||
|
# Returned fields: id, title, url, replies, node_name, node_title, content(first 200 chars), created
|
||||||
|
topics = ch.get_hot_topics(limit=10)
|
||||||
|
for t in topics:
|
||||||
|
print(f"[{t['node_title']}] {t['title']} ({t['replies']} replies) {t['url']}")
|
||||||
|
print(f" id={t['id']} created={t['created']}")
|
||||||
|
|
||||||
|
# Get latest topics for a specific node
|
||||||
|
# Returned fields: id, title, url, replies, node_name, node_title, content(first 200 chars), created
|
||||||
|
node_topics = ch.get_node_topics("python", limit=5)
|
||||||
|
for t in node_topics:
|
||||||
|
print(t["id"], t["title"], t["url"])
|
||||||
|
|
||||||
|
# Get one topic plus replies
|
||||||
|
# Returned fields: id, title, url, content, replies_count, node_name, node_title,
|
||||||
|
# author, created, replies (list of {author, content, created})
|
||||||
|
topic = ch.get_topic(1234567)
|
||||||
|
print(topic["title"], "—", topic["author"])
|
||||||
|
for r in topic["replies"]:
|
||||||
|
print(f" {r['author']}: {r['content'][:80]}")
|
||||||
|
|
||||||
|
# Get user info
|
||||||
|
# Returned fields: id, username, url, website, twitter, psn, github, btc, location, bio, avatar, created
|
||||||
|
user = ch.get_user("Livid")
|
||||||
|
print(user["username"], user["bio"], user["github"])
|
||||||
|
|
||||||
|
# Search (not supported by the public V2EX API; returns guidance instead)
|
||||||
|
result = ch.search("asyncio")
|
||||||
|
print(result[0]["error"]) # Use built-in site search or the Exa channel instead
|
||||||
|
```
|
||||||
|
|
||||||
|
> No auth required. Results are public JSON. V2EX node names are listed at https://www.v2ex.com/planes
|
||||||
|
|
||||||
|
## Xueqiu (public API)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from agent_reach.channels.xueqiu import XueqiuChannel
|
||||||
|
|
||||||
|
ch = XueqiuChannel()
|
||||||
|
|
||||||
|
# Get stock quotes (symbol examples: SH600519 mainland China, SZ000858 Shenzhen, AAPL US, 00700 HK)
|
||||||
|
# Returned fields: symbol, name, current, percent, chg, high, low, open, last_close,
|
||||||
|
# volume, amount, market_capital, turnover_rate, pe_ttm, timestamp
|
||||||
|
quote = ch.get_stock_quote("AAPL")
|
||||||
|
print(f"{quote['name']} ({quote['symbol']}): {quote['current']} ({quote['percent']}%)")
|
||||||
|
|
||||||
|
# Search stocks
|
||||||
|
# Returned fields: symbol, name, exchange
|
||||||
|
stocks = ch.search_stock("Apple", limit=5)
|
||||||
|
for s in stocks:
|
||||||
|
print(f"{s['name']} ({s['symbol']}) - {s['exchange']}")
|
||||||
|
|
||||||
|
# Hot posts
|
||||||
|
# Returned fields: id, title, text(first 200 chars), author, likes, url
|
||||||
|
posts = ch.get_hot_posts(limit=10)
|
||||||
|
for p in posts:
|
||||||
|
print(f"{p['author']}: {p['text'][:50]}... ({p['likes']} likes)")
|
||||||
|
|
||||||
|
# Hot stocks (stock_type=10 popularity ranking, stock_type=12 watchlist ranking)
|
||||||
|
# Returned fields: symbol, name, current, percent, rank
|
||||||
|
hot = ch.get_hot_stocks(limit=10, stock_type=10)
|
||||||
|
for s in hot:
|
||||||
|
print(f"#{s['rank']} {s['name']} ({s['symbol']}): {s['current']} ({s['percent']}%)")
|
||||||
|
```
|
||||||
|
|
||||||
|
> No login required. Agent Reach auto-fetches session cookies, and all public APIs can be used directly.
|
||||||
|
|
||||||
|
## RSS (feedparser)
|
||||||
|
|
||||||
|
```python
|
||||||
|
python3 -c "
|
||||||
|
import feedparser
|
||||||
|
for e in feedparser.parse('FEED_URL').entries[:5]:
|
||||||
|
print(f'{e.title} — {e.link}')
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- **Channel not working?** Run `agent-reach doctor` — it shows status and fix instructions.
|
||||||
|
- **Twitter fetch failed?** Ensure `undici` is installed: `npm install -g undici`. Configure a proxy if needed: `agent-reach configure proxy URL`.
|
||||||
|
|
||||||
|
## Setting Up a Channel ("help me configure XXX")
|
||||||
|
|
||||||
|
If a channel needs setup (cookies, Docker, etc.), fetch the install guide:
|
||||||
|
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
||||||
|
|
||||||
|
The user only provides cookies. Everything else is your job.
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# 职场招聘
|
||||||
|
|
||||||
|
LinkedIn。
|
||||||
|
|
||||||
|
## LinkedIn
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 获取个人资料
|
||||||
|
mcporter call 'linkedin-scraper.get_person_profile(linkedin_url: "https://linkedin.com/in/username")'
|
||||||
|
|
||||||
|
# 搜索人才
|
||||||
|
mcporter call 'linkedin-scraper.search_people(keyword: "AI engineer", limit: 10)'
|
||||||
|
|
||||||
|
# 获取公司资料
|
||||||
|
mcporter call 'linkedin-scraper.get_company_profile(linkedin_url: "https://linkedin.com/company/xxx")'
|
||||||
|
|
||||||
|
# 搜索职位
|
||||||
|
mcporter call 'linkedin-scraper.search_jobs(keyword: "software engineer", limit: 10)'
|
||||||
|
```
|
||||||
|
|
||||||
|
> **需要登录**: LinkedIn scraper 需要有效的登录态。
|
||||||
|
|
||||||
|
### Fallback 方案
|
||||||
|
|
||||||
|
如果 MCP 不可用,可以用 Jina Reader:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "https://r.jina.ai/https://linkedin.com/in/username"
|
||||||
|
```
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
# 开发工具
|
||||||
|
|
||||||
|
GitHub CLI
|
||||||
|
|
||||||
|
## GitHub (gh CLI)
|
||||||
|
|
||||||
|
GitHub 官方命令行工具,用于仓库、Issue、PR、Actions、Release 以及 API 访问。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 认证
|
||||||
|
gh auth login
|
||||||
|
gh auth status
|
||||||
|
|
||||||
|
# 搜索
|
||||||
|
gh search repos "query" --sort stars --limit 10
|
||||||
|
gh search code "query" --language python
|
||||||
|
|
||||||
|
# 仓库
|
||||||
|
gh repo view owner/repo
|
||||||
|
gh repo clone owner/repo
|
||||||
|
gh repo create my-repo --private
|
||||||
|
gh repo fork owner/repo
|
||||||
|
gh repo fork owner/repo --clone
|
||||||
|
gh repo sync owner/repo
|
||||||
|
|
||||||
|
# Issues
|
||||||
|
gh issue list -R owner/repo --state open
|
||||||
|
gh issue view 123 -R owner/repo
|
||||||
|
gh issue create -R owner/repo --title "Title" --body "Body"
|
||||||
|
|
||||||
|
# Pull Requests
|
||||||
|
gh pr list -R owner/repo --state open
|
||||||
|
gh pr view 123 -R owner/repo
|
||||||
|
gh pr create -R owner/repo --title "Title" --body "Body"
|
||||||
|
gh pr checks 123 --repo owner/repo
|
||||||
|
|
||||||
|
# Actions / CI
|
||||||
|
gh run list --repo owner/repo --limit 10
|
||||||
|
gh run view <run-id> --repo owner/repo
|
||||||
|
gh run view <run-id> --repo owner/repo --log-failed
|
||||||
|
gh workflow list --repo owner/repo
|
||||||
|
|
||||||
|
# Releases
|
||||||
|
gh release list -R owner/repo
|
||||||
|
gh release create v1.0.0
|
||||||
|
|
||||||
|
# API
|
||||||
|
gh api /user
|
||||||
|
gh api repos/owner/repo
|
||||||
|
|
||||||
|
# JSON 输出
|
||||||
|
gh issue list --repo owner/repo --json number,title --jq '.[] | "\(.number): \(.title)"'
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## 选择指南
|
||||||
|
|
||||||
|
| 工具 | 来源 | 用途 |
|
||||||
|
|-----|------|------|
|
||||||
|
| gh CLI | agent-reach | Git 操作 |
|
||||||
|
| zread | my-mcp-tools | 读仓库内容 |
|
||||||
|
| context7 | my-mcp-tools | 查技术文档 |
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# 搜索工具
|
||||||
|
|
||||||
|
Exa AI 搜索引擎。
|
||||||
|
|
||||||
|
## Exa AI 搜索
|
||||||
|
|
||||||
|
高质量 AI 搜索引擎,擅长技术和代码搜索。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mcporter call 'exa.web_search_exa(query: "query", numResults: 5)'
|
||||||
|
mcporter call 'exa.get_code_context_exa(query: "code question", tokensNum: 3000)'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 使用场景
|
||||||
|
|
||||||
|
| 场景 | 参数 |
|
||||||
|
|-----|------|
|
||||||
|
| 网页搜索 | `web_search_exa(query: "...", numResults: 5)` |
|
||||||
|
| 代码搜索 | `get_code_context_exa(query: "...", tokensNum: 3000)` |
|
||||||
|
|
||||||
|
### 特点
|
||||||
|
|
||||||
|
- 擅长英文内容和技术文档
|
||||||
|
- 支持代码上下文搜索
|
||||||
|
- 结果质量高
|
||||||
|
|
||||||
|
## 与其他搜索工具对比
|
||||||
|
|
||||||
|
| 工具 | 来源 | 适用场景 |
|
||||||
|
|-----|------|---------|
|
||||||
|
| Exa | agent-reach | 英文/技术/代码搜索 |
|
||||||
|
| 智谱搜索 | my-mcp-tools | 中文搜索 |
|
||||||
|
| GitHub 搜索 | agent-reach (dev.md) | 仓库/代码搜索 |
|
||||||
@@ -0,0 +1,228 @@
|
|||||||
|
# 社交媒体 & 社区
|
||||||
|
|
||||||
|
小红书、抖音、Twitter/X、微博、B站、V2EX、Reddit。
|
||||||
|
|
||||||
|
## 小红书 / XiaoHongShu (xhs-cli)
|
||||||
|
|
||||||
|
### 稳定可用的命令
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 搜索笔记(推荐入口)
|
||||||
|
xhs search "query"
|
||||||
|
|
||||||
|
# 阅读笔记详情(必须用搜索结果中的 URL 或 ID,不能裸 note_id)
|
||||||
|
xhs read NOTE_ID_OR_URL
|
||||||
|
|
||||||
|
# 查看评论
|
||||||
|
xhs comments NOTE_ID_OR_URL
|
||||||
|
|
||||||
|
# 浏览热门
|
||||||
|
xhs hot
|
||||||
|
|
||||||
|
# 推荐 feed
|
||||||
|
xhs feed
|
||||||
|
```
|
||||||
|
|
||||||
|
### 已知不稳定的命令(v0.6.4)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 以下命令当前可能返回 API error,谨慎使用:
|
||||||
|
xhs user USER_ID # 可能返回 {code: -1}
|
||||||
|
xhs user-posts USER_ID # 可能返回 {code: -1}
|
||||||
|
xhs favorites # 可能返回 API error
|
||||||
|
```
|
||||||
|
|
||||||
|
### 重要注意事项
|
||||||
|
|
||||||
|
> **安装**: `pipx install xiaohongshu-cli`,然后 `xhs login`(自动从浏览器提取 Cookie)。
|
||||||
|
>
|
||||||
|
> **xsec_token 限制**: 小红书强制 xsec_token 机制,**不能直接用裸 note_id 去读**。正确流程是:先 `xhs search` 或 `xhs feed` 获取结果,再用结果中的 URL/ID 去 `xhs read`。直接构造 note_id 会被拦截。
|
||||||
|
>
|
||||||
|
> **频率控制**: 高频请求(批量搜索、深翻评论)会触发验证码,这是平台限制无法绕过。建议每次操作间隔 2-3 秒。
|
||||||
|
>
|
||||||
|
> **POST 操作风险**: 发帖(post)、评论(comment)、点赞(like) 等写操作在 v0.6.x 可能因签名问题返回 406。如需使用,建议降级到 v0.3.5 (`pipx install xiaohongshu-cli==0.3.5`)。
|
||||||
|
|
||||||
|
## 抖音 / Douyin
|
||||||
|
|
||||||
|
### 安装与配置
|
||||||
|
|
||||||
|
`douyin-mcp-server` 是 **stdio 模式**的 MCP server,需先安装再注册到 mcporter:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. 安装
|
||||||
|
pipx install douyin-mcp-server
|
||||||
|
|
||||||
|
# 2. 查找安装路径
|
||||||
|
pipx runpip douyin-mcp-server show -f 2>/dev/null | grep "Location" \
|
||||||
|
|| find ~/.local -name "douyin-mcp-server" 2>/dev/null | head -1
|
||||||
|
|
||||||
|
# 3. 注册到 mcporter(使用 stdio 模式,将路径替换为上一步的输出)
|
||||||
|
mcporter config add douyin --command "/path/to/douyin-mcp-server" --scope home
|
||||||
|
```
|
||||||
|
|
||||||
|
> **注意**:`agent-reach install --channels douyin` 暂不支持抖音渠道(抖音在"可选渠道待解锁"列表)。
|
||||||
|
> HTTP 模式(`mcporter config add douyin http://localhost:18070/mcp`)**无法正常工作**,请使用上方 stdio 方式。
|
||||||
|
|
||||||
|
### 用法
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 解析视频信息
|
||||||
|
mcporter call 'douyin.parse_douyin_video_info(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
|
||||||
|
# 获取无水印下载链接
|
||||||
|
mcporter call 'douyin.get_douyin_download_link(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
|
||||||
|
# 提取视频文案
|
||||||
|
mcporter call 'douyin.extract_douyin_text(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
```
|
||||||
|
|
||||||
|
> **无需登录**
|
||||||
|
|
||||||
|
## Twitter/X (twitter-cli)
|
||||||
|
|
||||||
|
### 稳定命令
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 首页时间线(最稳定)
|
||||||
|
twitter feed -n 20
|
||||||
|
|
||||||
|
# 读取单条推文(含回复)
|
||||||
|
twitter tweet URL_OR_ID
|
||||||
|
|
||||||
|
# 读取长文 / X Article
|
||||||
|
twitter article URL_OR_ID
|
||||||
|
|
||||||
|
# 用户时间线
|
||||||
|
twitter user-posts @username -n 20
|
||||||
|
|
||||||
|
# 用户资料
|
||||||
|
twitter user @username
|
||||||
|
```
|
||||||
|
|
||||||
|
### 可能不稳定的命令
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 搜索推文(Twitter 频繁改 GraphQL 端点,可能 404)
|
||||||
|
twitter search "query" -n 10
|
||||||
|
# 如果 search 返回 404,升级 twitter-cli:pipx upgrade twitter-cli
|
||||||
|
|
||||||
|
# likes(2024 年后只能看自己的,平台限制)
|
||||||
|
twitter likes
|
||||||
|
```
|
||||||
|
|
||||||
|
### 重要注意事项
|
||||||
|
|
||||||
|
> **安装**: `pipx install twitter-cli`(确保 v0.8.5+)
|
||||||
|
>
|
||||||
|
> **认证**: 推荐用 Cookie-Editor 导出后设置环境变量 `TWITTER_AUTH_TOKEN` + `TWITTER_CT0`。自动提取在 SSH/Docker/无头环境不可用。
|
||||||
|
>
|
||||||
|
> **IP 风控**: 不要在 VPS/数据中心 IP 上频繁调用,尤其是 followers/following,有封号风险。使用住宅代理或本地环境。
|
||||||
|
>
|
||||||
|
> **search 可能失效**: Twitter 频繁修改 GraphQL API,search 命令可能随时返回 404。如遇到,先 `pipx upgrade twitter-cli`。如果最新版仍不行,说明上游还没跟上 Twitter 的改动,用 `twitter feed` 替代。
|
||||||
|
>
|
||||||
|
> **输出格式**: 建议用 `--yaml` 或 `--json` 获得结构化输出,对 AI agent 更友好。
|
||||||
|
|
||||||
|
## 微博 / Weibo
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 使用 Jina Reader 读取
|
||||||
|
curl -s "https://r.jina.ai/https://weibo.com/USER_ID/POST_ID"
|
||||||
|
```
|
||||||
|
|
||||||
|
> 微博主要通过网页抓取,推荐使用通用网页读取方式。
|
||||||
|
|
||||||
|
## B站 / Bilibili
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 获取视频元数据
|
||||||
|
yt-dlp --dump-json "https://www.bilibili.com/video/BVxxx"
|
||||||
|
|
||||||
|
# 下载字幕
|
||||||
|
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --convert-subs vtt --skip-download -o "/tmp/%(id)s" "URL"
|
||||||
|
```
|
||||||
|
|
||||||
|
> **注意**: 服务器 IP 可能遇到 412 错误。使用 `--cookies-from-browser chrome` 或配置代理。
|
||||||
|
|
||||||
|
## V2EX (公开 API)
|
||||||
|
|
||||||
|
无需认证,直接调用公开 API。
|
||||||
|
|
||||||
|
### 热门主题
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "https://www.v2ex.com/api/topics/hot.json" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 节点主题
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# node_name 如: python, tech, jobs, qna, programmers
|
||||||
|
curl -s "https://www.v2ex.com/api/topics/show.json?node_name=python&page=1" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 主题详情
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# topic_id 从 URL 获取,如 https://www.v2ex.com/t/1234567
|
||||||
|
curl -s "https://www.v2ex.com/api/topics/show.json?id=TOPIC_ID" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 主题回复
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "https://www.v2ex.com/api/replies/show.json?topic_id=TOPIC_ID&page=1" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 用户信息
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "https://www.v2ex.com/api/members/show.json?username=USERNAME" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Python 调用示例
|
||||||
|
|
||||||
|
```python
|
||||||
|
from agent_reach.channels.v2ex import V2EXChannel
|
||||||
|
|
||||||
|
ch = V2EXChannel()
|
||||||
|
|
||||||
|
# 获取热门帖子
|
||||||
|
topics = ch.get_hot_topics(limit=10)
|
||||||
|
for t in topics:
|
||||||
|
print(f"[{t['node_title']}] {t['title']} ({t['replies']} 回复)")
|
||||||
|
|
||||||
|
# 获取节点帖子
|
||||||
|
node_topics = ch.get_node_topics("python", limit=5)
|
||||||
|
|
||||||
|
# 获取帖子详情 + 回复
|
||||||
|
topic = ch.get_topic(1234567)
|
||||||
|
print(topic["title"], "—", topic["author"])
|
||||||
|
|
||||||
|
# 获取用户信息
|
||||||
|
user = ch.get_user("Livid")
|
||||||
|
```
|
||||||
|
|
||||||
|
> **节点列表**: https://www.v2ex.com/planes
|
||||||
|
|
||||||
|
## Reddit (rdt-cli)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 搜索帖子
|
||||||
|
rdt search "query" --limit 10
|
||||||
|
|
||||||
|
# 读帖子全文 + 评论
|
||||||
|
rdt read POST_ID
|
||||||
|
|
||||||
|
# 浏览 subreddit
|
||||||
|
rdt sub python --limit 20
|
||||||
|
|
||||||
|
# 浏览热门
|
||||||
|
rdt popular --limit 10
|
||||||
|
|
||||||
|
# 浏览 /r/all
|
||||||
|
rdt all --limit 10
|
||||||
|
```
|
||||||
|
|
||||||
|
> **安装**: `pipx install rdt-cli`(确保 v0.4.2+)。无需登录即可搜索和阅读。
|
||||||
|
> 需要登录的功能:`rdt feed --subs-only`(订阅列表)、`rdt saved`(收藏)。
|
||||||
|
> 建议使用 `--yaml` 输出,对 AI agent 更友好。
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
# 视频/播客
|
||||||
|
|
||||||
|
YouTube、B站、小宇宙播客的字幕和转录。
|
||||||
|
|
||||||
|
## YouTube (yt-dlp)
|
||||||
|
|
||||||
|
### 获取视频元数据
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yt-dlp --dump-json "URL"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 下载字幕
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 下载字幕 (不下载视频)
|
||||||
|
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --skip-download -o "/tmp/%(id)s" "URL"
|
||||||
|
|
||||||
|
# 然后读取 .vtt 文件
|
||||||
|
cat /tmp/VIDEO_ID.*.vtt
|
||||||
|
```
|
||||||
|
|
||||||
|
### 获取评论
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 提取评论(best-effort,不保证完整)
|
||||||
|
yt-dlp --write-comments --skip-download --write-info-json \
|
||||||
|
--extractor-args "youtube:max_comments=20" \
|
||||||
|
-o "/tmp/%(id)s" "URL"
|
||||||
|
# 评论在 .info.json 的 comments 字段中
|
||||||
|
```
|
||||||
|
|
||||||
|
### 搜索视频
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yt-dlp --dump-json "ytsearch5:query"
|
||||||
|
```
|
||||||
|
|
||||||
|
> **字幕注意**: 手动上传的字幕提取可靠;自动生成字幕可能存在行间重复,需后处理。
|
||||||
|
> **评论注意**: `--write-comments` 基于网页抓取(非 YouTube Data API),部分评论可能丢失。
|
||||||
|
|
||||||
|
## B站 / Bilibili (yt-dlp + bili-cli)
|
||||||
|
|
||||||
|
### 视频元数据 (yt-dlp)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yt-dlp --dump-json "https://www.bilibili.com/video/BVxxx"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 字幕 (yt-dlp)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --convert-subs vtt --skip-download -o "/tmp/%(id)s" "URL"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 搜索/热门/排行 (bili-cli)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 搜索视频
|
||||||
|
bili search "query" --type video -n 5
|
||||||
|
|
||||||
|
# 热门视频
|
||||||
|
bili hot -n 10
|
||||||
|
|
||||||
|
# 排行榜
|
||||||
|
bili rank -n 10
|
||||||
|
```
|
||||||
|
|
||||||
|
> **412 风控**: 海外 IP 必须提供 Cookie(`--cookies-from-browser chrome` 或 `--cookies /path/to/cookies.txt`),国内 IP 一般不受影响。
|
||||||
|
> **安装 bili-cli**: `pipx install bilibili-cli`,然后 `bili login` 扫码登录。
|
||||||
|
|
||||||
|
## 小宇宙播客 / Xiaoyuzhou Podcast
|
||||||
|
|
||||||
|
### 转录单集播客
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 输出 Markdown 文件到 /tmp/
|
||||||
|
~/.agent-reach/tools/xiaoyuzhou/transcribe.sh "https://www.xiaoyuzhoufm.com/episode/EPISODE_ID"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 前置要求
|
||||||
|
|
||||||
|
1. **ffmpeg**: `brew install ffmpeg`
|
||||||
|
2. **Groq API Key** (免费): https://console.groq.com/keys
|
||||||
|
3. **配置 Key**: `agent-reach configure groq-key YOUR_KEY`
|
||||||
|
4. **首次运行**: `agent-reach install --env=auto` 安装工具
|
||||||
|
|
||||||
|
### 检查状态
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach doctor
|
||||||
|
```
|
||||||
|
|
||||||
|
> 输出 Markdown 文件默认保存到 `/tmp/`。
|
||||||
|
|
||||||
|
## 抖音视频解析
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 解析视频信息
|
||||||
|
mcporter call 'douyin.parse_douyin_video_info(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
|
||||||
|
# 获取无水印下载链接
|
||||||
|
mcporter call 'douyin.get_douyin_download_link(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
```
|
||||||
|
|
||||||
|
> 详见 [social.md](social.md#抖音--douyin)
|
||||||
|
|
||||||
|
## 选择指南
|
||||||
|
|
||||||
|
| 场景 | 推荐工具 |
|
||||||
|
|-----|---------|
|
||||||
|
| YouTube 字幕 | yt-dlp |
|
||||||
|
| B站字幕 | yt-dlp |
|
||||||
|
| 播客转录 | 小宇宙 transcribe.sh |
|
||||||
|
| 抖音视频解析 | douyin MCP |
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
# 网页阅读
|
||||||
|
|
||||||
|
通用网页、微信公众号、RSS。
|
||||||
|
|
||||||
|
## 通用网页 (Jina Reader)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 读取任意网页内容
|
||||||
|
curl -s "https://r.jina.ai/URL"
|
||||||
|
|
||||||
|
# 示例
|
||||||
|
curl -s "https://r.jina.ai/https://example.com/article"
|
||||||
|
```
|
||||||
|
|
||||||
|
**适用场景**: 大多数网页可以直接用 Jina Reader 读取。
|
||||||
|
|
||||||
|
## Web Reader (MCP)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 读取网页内容 (Markdown 格式)
|
||||||
|
mcporter call 'web-reader.webReader(url: "https://example.com")'
|
||||||
|
|
||||||
|
# 保留图片
|
||||||
|
mcporter call 'web-reader.webReader(url: "https://example.com", retain_images: true)'
|
||||||
|
|
||||||
|
# 纯文本格式
|
||||||
|
mcporter call 'web-reader.webReader(url: "https://example.com", return_format: "text")'
|
||||||
|
```
|
||||||
|
|
||||||
|
**适用场景**: 需要更精确控制输出格式时使用。
|
||||||
|
|
||||||
|
## 微信公众号 / WeChat Articles
|
||||||
|
|
||||||
|
### 搜索公众号文章(通过 Exa)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 搜索微信公众号文章
|
||||||
|
mcporter call 'exa.web_search_exa(query: "搜索关键词", numResults: 5, includeDomains: ["mp.weixin.qq.com"])'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 阅读公众号文章全文(通过 Exa)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 抓取文章全文
|
||||||
|
mcporter call 'exa.crawling_exa(urls: ["https://mp.weixin.qq.com/s/ARTICLE_ID"], maxCharacters: 10000)'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 可选:Camoufox 阅读(反爬更强)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/.agent-reach/tools/wechat-article-for-ai && python3 main.py "https://mp.weixin.qq.com/s/ARTICLE_ID"
|
||||||
|
```
|
||||||
|
|
||||||
|
> **注意**: Jina Reader 无法读取微信文章(被 CAPTCHA 拦截),推荐用 Exa。
|
||||||
|
|
||||||
|
## RSS (feedparser)
|
||||||
|
|
||||||
|
```python
|
||||||
|
python3 -c "
|
||||||
|
import feedparser
|
||||||
|
for e in feedparser.parse('FEED_URL').entries[:5]:
|
||||||
|
print(f'{e.title} — {e.link}')
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
**适用场景**: 订阅博客、新闻源、播客等 RSS feed。
|
||||||
|
|
||||||
|
## 选择指南
|
||||||
|
|
||||||
|
| 场景 | 推荐工具 |
|
||||||
|
|-----|---------|
|
||||||
|
| 通用网页 | Jina Reader (`curl r.jina.ai`) |
|
||||||
|
| 需要图片/格式控制 | web-reader MCP |
|
||||||
|
| 微信公众号 | Exa (搜索+阅读) / Camoufox (可选阅读) |
|
||||||
|
| RSS 订阅 | feedparser |
|
||||||
|
| 微博/知乎等 | Jina Reader |
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
"""Cross-platform path and remediation helpers for yt-dlp."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def get_ytdlp_config_dir() -> Path:
|
||||||
|
"""Return the recommended yt-dlp user config directory for this OS."""
|
||||||
|
|
||||||
|
if sys.platform == "win32":
|
||||||
|
appdata = os.environ.get("APPDATA")
|
||||||
|
if appdata:
|
||||||
|
return Path(appdata) / "yt-dlp"
|
||||||
|
return Path.home() / "AppData" / "Roaming" / "yt-dlp"
|
||||||
|
if sys.platform == "darwin":
|
||||||
|
return Path.home() / "Library" / "Application Support" / "yt-dlp"
|
||||||
|
return Path.home() / ".config" / "yt-dlp"
|
||||||
|
|
||||||
|
|
||||||
|
def get_ytdlp_config_path() -> Path:
|
||||||
|
"""Return the yt-dlp user config file path for this OS."""
|
||||||
|
|
||||||
|
return get_ytdlp_config_dir() / "config"
|
||||||
|
|
||||||
|
|
||||||
|
def render_ytdlp_fix_command() -> str:
|
||||||
|
"""Return an OS-appropriate command to enable Node.js as yt-dlp JS runtime."""
|
||||||
|
|
||||||
|
config_path = get_ytdlp_config_path()
|
||||||
|
if sys.platform == "win32":
|
||||||
|
return (
|
||||||
|
f"$cfg = '{config_path}'\n"
|
||||||
|
"New-Item -ItemType Directory -Force -Path (Split-Path $cfg) | Out-Null\n"
|
||||||
|
"if (-not (Test-Path $cfg) -or -not (Select-String -Path $cfg -Pattern '--js-runtimes' -Quiet)) {\n"
|
||||||
|
" Add-Content -Path $cfg -Value '--js-runtimes node'\n"
|
||||||
|
"}"
|
||||||
|
)
|
||||||
|
return (
|
||||||
|
f"mkdir -p '{config_path.parent}' && "
|
||||||
|
f"grep -qxF -- '--js-runtimes node' '{config_path}' 2>/dev/null || "
|
||||||
|
f"printf '%s\n' '--js-runtimes node' >> '{config_path}'"
|
||||||
|
)
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
"""UTF-8-safe text helpers for cross-platform file operations."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def read_utf8_text(path: str | Path, default: str = "") -> str:
|
||||||
|
"""Read text as UTF-8 with replacement semantics."""
|
||||||
|
|
||||||
|
target = Path(path)
|
||||||
|
if not target.exists():
|
||||||
|
return default
|
||||||
|
return target.read_text(encoding="utf-8", errors="replace")
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# Agent Reach tested dependency set
|
||||||
|
# Usage:
|
||||||
|
# pip install -c constraints.txt -e .[dev]
|
||||||
|
|
||||||
|
requests==2.32.5
|
||||||
|
feedparser==6.0.12
|
||||||
|
python-dotenv==1.2.1
|
||||||
|
loguru==0.7.3
|
||||||
|
PyYAML==6.0.3
|
||||||
|
rich==14.3.2
|
||||||
|
yt-dlp==2025.5.22
|
||||||
|
|
||||||
|
pytest==8.0.0
|
||||||
|
ruff==0.15.1
|
||||||
|
mypy==1.19.1
|
||||||
|
types-requests==2.32.4.20260107
|
||||||
|
types-PyYAML==6.0.12.20250915
|
||||||
+168
-35
@@ -11,7 +11,7 @@
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<a href="#quick-start">Quick Start</a> · <a href="../README.md">中文</a> · <a href="#supported-platforms">Platforms</a> · <a href="#design-philosophy">Philosophy</a>
|
<a href="#quick-start">Quick Start</a> · <a href="../README.md">中文</a> · <a href="README_ja.md">日本語</a> · <a href="README_ko.md">한국어</a> · <a href="#supported-platforms">Platforms</a> · <a href="#design-philosophy">Philosophy</a>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -39,13 +39,19 @@ Install Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/m
|
|||||||
|
|
||||||
Copy that to your Agent. A few minutes later, it can read tweets, search Reddit, and watch Bilibili.
|
Copy that to your Agent. A few minutes later, it can read tweets, search Reddit, and watch Bilibili.
|
||||||
|
|
||||||
|
**Already installed? Update in one command:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
```
|
||||||
|
|
||||||
### ✅ Before you start, you might want to know
|
### ✅ Before you start, you might want to know
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
|---|---|
|
|---|---|
|
||||||
| 💰 **Completely free** | All tools are open source, all APIs are free. The only possible cost is a server proxy ($1/month) — local computers don't need one |
|
| 💰 **Completely free** | All tools are open source, all APIs are free. The only possible cost is a server proxy ($1/month) — local computers don't need one |
|
||||||
| 🔒 **Privacy safe** | Cookies stay local. Never uploaded. Fully open source — audit anytime |
|
| 🔒 **Privacy safe** | Cookies stay local. Never uploaded. Fully open source — audit anytime |
|
||||||
| 🔄 **Kept up to date** | Upstream tools (yt-dlp, bird, Jina Reader, etc.) are tracked and updated regularly |
|
| 🔄 **Kept up to date** | Upstream tools (yt-dlp, twitter-cli, rdt-cli, Jina Reader, etc.) are tracked and updated regularly |
|
||||||
| 🤖 **Works with any Agent** | Claude Code, OpenClaw, Cursor, Windsurf… any Agent that can run commands |
|
| 🤖 **Works with any Agent** | Claude Code, OpenClaw, Cursor, Windsurf… any Agent that can run commands |
|
||||||
| 🩺 **Built-in diagnostics** | `agent-reach doctor` — one command shows what works, what doesn't, and how to fix it |
|
| 🩺 **Built-in diagnostics** | `agent-reach doctor` — one command shows what works, what doesn't, and how to fix it |
|
||||||
|
|
||||||
@@ -56,17 +62,21 @@ Copy that to your Agent. A few minutes later, it can read tweets, search Reddit,
|
|||||||
| Platform | Capabilities | Setup | Notes |
|
| Platform | Capabilities | Setup | Notes |
|
||||||
|----------|-------------|:-----:|-------|
|
|----------|-------------|:-----:|-------|
|
||||||
| 🌐 **Web** | Read | Zero config | Any URL → clean Markdown ([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) |
|
| 🌐 **Web** | Read | Zero config | Any URL → clean Markdown ([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) |
|
||||||
| 🐦 **Twitter/X** | Read · Search | Zero config / Cookie | Single tweets readable out of the box. Cookie unlocks search, timeline, posting ([bird](https://github.com/steipete/bird)) |
|
| 🐦 **Twitter/X** | Read · Search | Cookie | Cookie unlocks search, timeline, tweet reading, articles ([twitter-cli](https://github.com/public-clis/twitter-cli)) |
|
||||||
| 📕 **XiaoHongShu** | Read · Search · **Post · Comment · Like** | mcporter | Via [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) internal API, install and go |
|
| 📕 **XiaoHongShu** | Read · Search · **Post · Comment · Like** | Cookie | `pipx install xiaohongshu-cli` + `xhs login` ([xhs-cli](https://github.com/jackwener/xiaohongshu-cli)) |
|
||||||
| 📷 **Instagram** | Read · Search | instaloader | Posts, profiles, hashtags ([instaloader](https://github.com/instaloader/instaloader) ⭐9.8K) |
|
| 🎵 **Douyin** | Video parsing · Watermark-free download | mcporter | Via [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server), no login needed |
|
||||||
| 💼 **LinkedIn** | Read · Search | mcporter / Jina | Profiles, companies, job search ([linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) ⭐900+) |
|
| 💼 **LinkedIn** | Jina Reader (public pages) | Full profiles, companies, job search | Tell your Agent "help me set up LinkedIn" |
|
||||||
| 🏢 **Boss直聘** | Read · Search | mcporter / Jina | Job search, greet recruiters ([mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)) |
|
| 💬 **WeChat Articles** | Search + Read | Zero config | Search + read WeChat Official Account articles via Exa (zero config) + optional [Camoufox](https://github.com/daijro/camoufox) |
|
||||||
|
| 📰 **Weibo** | Trending · Search · Feeds · Comments | Zero config | Hot search, content/user/topic search, feeds, comments ([mcp-server-weibo](https://github.com/Panniantong/mcp-server-weibo)) |
|
||||||
|
| 💻 **V2EX** | Hot topics · Node topics · Topic detail + replies · User profile | Zero config | Public JSON API, no auth required. Great for tech community content |
|
||||||
|
| 📈 **Xueqiu (雪球)** | Stock quotes · Search · Hot posts · Hot stocks | Browser cookie | Tell your Agent "help me set up Xueqiu" |
|
||||||
|
| 🎙️ **Xiaoyuzhou Podcast** | Transcription | Free API key | Podcast audio → full text transcript via Groq Whisper (free) |
|
||||||
| 🔍 **Web Search** | Search | Auto-configured | Auto-configured during install, free, no API key ([Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter)) |
|
| 🔍 **Web Search** | Search | Auto-configured | Auto-configured during install, free, no API key ([Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter)) |
|
||||||
| 📦 **GitHub** | Read · Search | Zero config | [gh CLI](https://cli.github.com) powered. Public repos work immediately. `gh auth login` unlocks Fork, Issue, PR |
|
| 📦 **GitHub** | Read · Search | Zero config | [gh CLI](https://cli.github.com) powered. Public repos work immediately. `gh auth login` unlocks Fork, Issue, PR |
|
||||||
| 📺 **YouTube** | Read · **Search** | Zero config | Subtitles + search across 1800+ video sites ([yt-dlp](https://github.com/yt-dlp/yt-dlp) ⭐148K) |
|
| 📺 **YouTube** | Read · **Search** | Zero config | Subtitles + search across 1800+ video sites ([yt-dlp](https://github.com/yt-dlp/yt-dlp) ⭐148K) |
|
||||||
| 📺 **Bilibili** | Read · **Search** | Zero config / Proxy | Video info + subtitles + search. Local works directly, servers need a proxy ([yt-dlp](https://github.com/yt-dlp/yt-dlp)) |
|
| 📺 **Bilibili** | Read · **Search** | Zero config / Proxy | Video info + subtitles + search. Local works directly, servers need a proxy ([yt-dlp](https://github.com/yt-dlp/yt-dlp)) |
|
||||||
| 📡 **RSS** | Read | Zero config | Any RSS/Atom feed ([feedparser](https://github.com/kurtmckee/feedparser) ⭐2.3K) |
|
| 📡 **RSS** | Read | Zero config | Any RSS/Atom feed ([feedparser](https://github.com/kurtmckee/feedparser) ⭐2.3K) |
|
||||||
| 📖 **Reddit** | Search · Read | Free / Proxy | Search via Exa (free). Reading posts needs a proxy on servers |
|
| 📖 **Reddit** | Search · Read | Cookie | Requires auth since 2024 — `rdt login` after install ([rdt-cli](https://github.com/public-clis/rdt-cli)) |
|
||||||
|
|
||||||
> **Setup levels:** Zero config = install and go · Auto-configured = handled during install · mcporter = needs MCP service · Cookie = export from browser · Proxy = $1/month
|
> **Setup levels:** Zero config = install and go · Auto-configured = handled during install · mcporter = needs MCP service · Cookie = export from browser · Proxy = $1/month
|
||||||
|
|
||||||
@@ -82,6 +92,11 @@ Install Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/m
|
|||||||
|
|
||||||
The Agent auto-installs, detects your environment, and tells you what's ready.
|
The Agent auto-installs, detects your environment, and tells you what's ready.
|
||||||
|
|
||||||
|
> 🔄 **Already installed?** Update in one command:
|
||||||
|
> ```
|
||||||
|
> Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
> ```
|
||||||
|
|
||||||
<details>
|
<details>
|
||||||
<summary>Manual install</summary>
|
<summary>Manual install</summary>
|
||||||
|
|
||||||
@@ -101,6 +116,11 @@ npx skills add Panniantong/Agent-Reach@agent-reach
|
|||||||
After the Skill is installed, the Agent will auto-detect whether `agent-reach` CLI is available and install it if needed.
|
After the Skill is installed, the Agent will auto-detect whether `agent-reach` CLI is available and install it if needed.
|
||||||
|
|
||||||
> If you install via `agent-reach install`, the skill is registered automatically — no extra steps needed.
|
> If you install via `agent-reach install`, the skill is registered automatically — no extra steps needed.
|
||||||
|
>
|
||||||
|
> Prefer an English-only skill file? Set an English locale or export `AGENT_REACH_LANG=en`
|
||||||
|
> before running `agent-reach install --env=auto` or `agent-reach skill --install`.
|
||||||
|
> The installed file is always written as `SKILL.md`, so switching languages means rerunning
|
||||||
|
> the install command with the new locale and replacing the previously installed skill file.
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -109,14 +129,14 @@ After the Skill is installed, the Agent will auto-detect whether `agent-reach` C
|
|||||||
|
|
||||||
No configuration needed — just tell your Agent:
|
No configuration needed — just tell your Agent:
|
||||||
|
|
||||||
- "Read this link" → any web page
|
- "Read this link" → `curl https://r.jina.ai/URL` for any web page
|
||||||
- "What's this GitHub repo about?" → repos, issues, code
|
- "What's this GitHub repo about?" → `gh repo view owner/repo`
|
||||||
- "What does this video cover?" → YouTube / Bilibili subtitles
|
- "What does this video cover?" → `yt-dlp --dump-json URL` for subtitles
|
||||||
- "Read this tweet" → Twitter posts
|
- "Read this tweet" → `twitter tweet URL`
|
||||||
- "Subscribe to this RSS" → RSS / Atom feeds
|
- "Subscribe to this RSS" → `feedparser` to parse feeds
|
||||||
- "Search GitHub for LLM frameworks" → GitHub search
|
- "Search GitHub for LLM frameworks" → `gh search repos "LLM framework"`
|
||||||
|
|
||||||
**No commands to remember.** The Agent knows what to call.
|
**No commands to remember.** The Agent reads SKILL.md and knows what to call.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -130,9 +150,9 @@ Tell your Agent "help me configure Twitter cookies" — it'll guide you through
|
|||||||
|
|
||||||
### 🌐 Proxy — $1/month, servers only
|
### 🌐 Proxy — $1/month, servers only
|
||||||
|
|
||||||
Reddit and Bilibili block server IPs. Get a proxy ([Webshare](https://webshare.io) recommended, $1/month) and send the address to your Agent.
|
Bilibili blocks server IPs. Get a proxy ([Webshare](https://webshare.io) recommended, $1/month) and send the address to your Agent.
|
||||||
|
|
||||||
> Local computers don't need a proxy. Reddit search works free via Exa even without one.
|
> Reddit now works free via rdt-cli without any proxy. Local computers don't need a proxy for Bilibili either.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -156,7 +176,7 @@ $ agent-reach doctor
|
|||||||
⬜ Web semantic search — sign up at exa.ai for free key
|
⬜ Web semantic search — sign up at exa.ai for free key
|
||||||
|
|
||||||
🔧 Configurable:
|
🔧 Configurable:
|
||||||
⬜ Reddit posts and comments — search via Exa (free). Reading needs proxy
|
✅ Reddit posts and comments — search and read via rdt-cli (free, no proxy)
|
||||||
⬜ XiaoHongShu notes — needs cookie. Export from browser
|
⬜ XiaoHongShu notes — needs cookie. Export from browser
|
||||||
|
|
||||||
Status: 6/9 channels available
|
Status: 6/9 channels available
|
||||||
@@ -172,41 +192,48 @@ Every time you spin up a new Agent, you spend time finding tools, installing dep
|
|||||||
|
|
||||||
Agent Reach does one simple thing: **it makes those tool selection and configuration decisions for you.**
|
Agent Reach does one simple thing: **it makes those tool selection and configuration decisions for you.**
|
||||||
|
|
||||||
|
After installation, your Agent calls the upstream tools directly (twitter-cli, rdt-cli, xhs-cli, yt-dlp, mcporter, gh CLI, etc.) — no wrapper layer in between.
|
||||||
|
|
||||||
### 🔌 Every Channel is Pluggable
|
### 🔌 Every Channel is Pluggable
|
||||||
|
|
||||||
Each platform is a single Python file implementing a unified interface. **Backends can be swapped anytime** — when a better tool comes along, change one file and nothing else breaks.
|
Each platform maps to an upstream tool. **Don't like one? Swap it out.**
|
||||||
|
|
||||||
```
|
```
|
||||||
channels/
|
channels/
|
||||||
├── web.py → Jina Reader ← swap to Firecrawl, Crawl4AI…
|
├── web.py → Jina Reader ← swap to Firecrawl, Crawl4AI…
|
||||||
├── twitter.py → bird ← swap to Nitter, official API…
|
├── twitter.py → twitter-cli ← swap to official API…
|
||||||
├── youtube.py → yt-dlp ← swap to YouTube API, Whisper…
|
├── youtube.py → yt-dlp ← swap to YouTube API, Whisper…
|
||||||
├── github.py → gh CLI ← swap to REST API, PyGithub…
|
├── github.py → gh CLI ← swap to REST API, PyGithub…
|
||||||
├── bilibili.py → yt-dlp ← swap to bilibili-api…
|
├── bilibili.py → yt-dlp ← swap to bilibili-api…
|
||||||
├── reddit.py → JSON API + Exa ← swap to PRAW, Pushshift…
|
├── reddit.py → rdt-cli ← search + read, cookie auth required
|
||||||
├── xiaohongshu.py → mcporter MCP ← swap to other XHS tools…
|
├── xiaohongshu.py → mcporter MCP ← swap to other XHS tools…
|
||||||
├── instagram.py → instaloader ← swap to instagrapi, official API…
|
├── douyin.py → mcporter MCP ← swap to other Douyin tools…
|
||||||
├── linkedin.py → linkedin-mcp ← swap to LinkedIn API…
|
├── linkedin.py → linkedin-mcp ← swap to LinkedIn API…
|
||||||
├── bosszhipin.py → mcp-bosszp ← swap to other job tools…
|
├── rss.py → feedparser ← swap to atoma…
|
||||||
├── rss.py → feedparser ← swap to atoma…
|
|
||||||
├── exa_search.py → mcporter MCP ← swap to Tavily, SerpAPI…
|
├── exa_search.py → mcporter MCP ← swap to Tavily, SerpAPI…
|
||||||
└── __init__.py → Channel registry
|
└── __init__.py → Channel registry (for doctor checks)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Each channel file only checks whether its upstream tool is installed and working (`check()` method for `agent-reach doctor`). The actual reading and searching is done by calling the upstream tools directly.
|
||||||
|
|
||||||
### Current Tool Choices
|
### Current Tool Choices
|
||||||
|
|
||||||
| Scenario | Tool | Why |
|
| Scenario | Tool | Why |
|
||||||
|----------|------|-----|
|
|----------|------|-----|
|
||||||
| Read web pages | [Jina Reader](https://github.com/jina-ai/reader) | 9.8K stars, free, no API key needed |
|
| Read web pages | [Jina Reader](https://github.com/jina-ai/reader) | 9.8K stars, free, no API key needed |
|
||||||
| Read tweets | [bird](https://github.com/steipete/bird) | Cookie auth, free. Official API is pay-per-use ($0.005/post read) |
|
| Read tweets | [twitter-cli](https://github.com/public-clis/twitter-cli) | 2.1K stars, cookie auth, search/read/timeline/articles |
|
||||||
| Video subtitles + search | [yt-dlp](https://github.com/yt-dlp/yt-dlp) | 148K stars, YouTube + Bilibili + 1800 sites |
|
| Reddit | [rdt-cli](https://github.com/public-clis/rdt-cli) | 304 stars, cookie auth, search + full posts + comments |
|
||||||
| Search the web | [Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter) | AI semantic search, MCP integration, no API key |
|
| Video subtitles + search | [yt-dlp](https://github.com/yt-dlp/yt-dlp) | 154K stars, YouTube + Bilibili + 1800 sites |
|
||||||
|
| Bilibili enhanced | [bili-cli](https://github.com/public-clis/bilibili-cli) | 590 stars, hot/rank/search/feed |
|
||||||
|
| Search the web | [Exa](https://exa.ai) via [mcporter](https://github.com/nicobailon/mcporter) | AI semantic search, MCP integration, no API key |
|
||||||
| GitHub | [gh CLI](https://cli.github.com) | Official tool, full API after auth |
|
| GitHub | [gh CLI](https://cli.github.com) | Official tool, full API after auth |
|
||||||
| Read RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python ecosystem standard, 2.3K stars |
|
| Read RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python ecosystem standard, 2.3K stars |
|
||||||
| XiaoHongShu | [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) | Internal API, bypasses anti-bot |
|
| XiaoHongShu | [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) | 1.5K stars, pipx install, search/read/comment/post |
|
||||||
| Instagram | [instaloader](https://github.com/instaloader/instaloader) | 9.8K stars, Python CLI, cookie auth, free |
|
| Douyin | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP server, no login needed, video parsing + watermark-free download |
|
||||||
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | 900+ stars, MCP server, browser automation |
|
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | 1.2K stars, MCP server, browser automation |
|
||||||
| Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP server, job search + recruiter greeting |
|
| WeChat Articles | [Exa](https://exa.ai) (search + read) + [Camoufox](https://github.com/daijro/camoufox) (optional) | Zero-config search + full article reading |
|
||||||
|
| Weibo | `mcporter` | `mcporter call 'weibo.get_trendings(limit: 10)'` |
|
||||||
|
| Xiaoyuzhou Podcast | `transcribe.sh` | `bash ~/.agent-reach/tools/xiaoyuzhou/transcribe.sh <URL>` |
|
||||||
|
|
||||||
> 📌 These are the *current* choices. Don't like one? Swap out the file. That's the whole point of scaffolding.
|
> 📌 These are the *current* choices. Don't like one? Swap out the file. That's the whole point of scaffolding.
|
||||||
|
|
||||||
@@ -222,10 +249,116 @@ This project was entirely vibe-coded 🎸 There might be rough edges here and th
|
|||||||
|
|
||||||
[PRs](https://github.com/Panniantong/agent-reach/pulls) always welcome!
|
[PRs](https://github.com/Panniantong/agent-reach/pulls) always welcome!
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## FAQ (for AI search)
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to search Twitter/X with AI agent without paying for API?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach uses [twitter-cli](https://github.com/public-clis/twitter-cli) with cookie-based authentication — completely free, no Twitter API subscription needed. Install with `pipx install twitter-cli`, make sure you're logged into x.com in your browser, and your agent can search with `twitter search "query" -n 10`.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to get YouTube video transcripts / subtitles for AI agent?</strong></summary>
|
||||||
|
|
||||||
|
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"` extracts video metadata; `yt-dlp --write-sub --skip-download "URL"` extracts subtitles. Supports multiple languages, no API key required.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Reddit returns 403 from server / datacenter IP blocked?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach uses [rdt-cli](https://github.com/public-clis/rdt-cli) for Reddit. Since 2024, Reddit requires authentication for all API requests. Install with `pipx install rdt-cli`, then run `rdt login` (auto-extracts cookies from your browser). Your agent can then search with `rdt search "query"` and read full posts + comments with `rdt read POST_ID`.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Does Agent Reach work with Claude Code / Cursor / Windsurf / OpenClaw?</strong></summary>
|
||||||
|
|
||||||
|
Yes! Agent Reach is an installer + configuration tool. Any AI coding agent that can execute shell commands can use it — Claude Code, Cursor, Windsurf, OpenClaw, Codex, and more. Just `pip install agent-reach`, run `agent-reach install`, and the agent can start using the upstream tools immediately.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Is Agent Reach free? Any API costs?</strong></summary>
|
||||||
|
|
||||||
|
100% free and open source. All backends (twitter-cli, rdt-cli, xhs-cli, yt-dlp, Jina Reader, Exa) are free tools that don't require paid API keys. The only optional cost is a residential proxy (~$1/month) if you need Bilibili access from a server. Reddit works free via rdt-cli without any proxy.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Free alternative to Twitter API for web scraping?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach uses twitter-cli which accesses Twitter via cookie auth — same as your browser session. No API fees, no rate limit tiers, no developer account needed. Supports search, read tweets, read profiles, and timelines.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to read XiaoHongShu / 小红书 content programmatically?</strong></summary>
|
||||||
|
|
||||||
|
Install `pipx install xiaohongshu-cli`, then `xhs login` (auto-extracts cookies from browser). Your agent can then use `xhs search "query"` to search notes, `xhs read NOTE_ID` to read details, `xhs comments NOTE_ID` to view comments. No Docker needed.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to parse Douyin / 抖音 videos with AI agent?</strong></summary>
|
||||||
|
|
||||||
|
Install douyin-mcp-server, then your agent can use `mcporter call 'douyin.parse_douyin_video_info(share_link: "share_url")'` to parse video info and get watermark-free download links. No login required — just share the Douyin link. See https://github.com/yzfly/douyin-mcp-server
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to extract scripts from both Douyin and XiaoHongShu with one MCP?</strong></summary>
|
||||||
|
|
||||||
|
If you want one MCP server that can handle:
|
||||||
|
|
||||||
|
- Douyin videos
|
||||||
|
- XiaoHongShu video notes
|
||||||
|
- XiaoHongShu image notes
|
||||||
|
|
||||||
|
and directly write `script.md` + `info.json`, you can point the existing `douyin` mcporter alias at:
|
||||||
|
|
||||||
|
- https://github.com/JNHFlow21/social-post-extractor-mcp
|
||||||
|
|
||||||
|
It keeps backward compatibility with:
|
||||||
|
|
||||||
|
- `parse_douyin_video_info`
|
||||||
|
- `get_douyin_download_link`
|
||||||
|
- `extract_douyin_text`
|
||||||
|
|
||||||
|
and adds unified tools:
|
||||||
|
|
||||||
|
- `parse_social_post_info`
|
||||||
|
- `extract_social_post_script`
|
||||||
|
|
||||||
|
This is useful when your agent workflow is “paste a link, get a script file”.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Credits
|
## Credits
|
||||||
|
|
||||||
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://github.com/steipete/bird) · [Exa](https://exa.ai) · [feedparser](https://github.com/kurtmckee/feedparser) · [instaloader](https://github.com/instaloader/instaloader) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
|
[twitter-cli](https://github.com/public-clis/twitter-cli) · [rdt-cli](https://github.com/public-clis/rdt-cli) · [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) · [bili-cli](https://github.com/public-clis/bilibili-cli) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [Jina Reader](https://github.com/jina-ai/reader) · [Exa](https://exa.ai) · [mcporter](https://github.com/nicobailon/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||||
|
|
||||||
|
## Contact
|
||||||
|
|
||||||
|
- 📧 **Email:** pnt01@foxmail.com
|
||||||
|
- 🐦 **Twitter/X:** [@Neo_Reidlab](https://x.com/Neo_Reidlab)
|
||||||
|
|
||||||
|
For collaboration or questions, add me on WeChat — I'll invite you to the community group:
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="wechat-group-qr.jpg" width="280" alt="WeChat QR">
|
||||||
|
</p>
|
||||||
|
|
||||||
|
> For bug reports and feature requests, please use [GitHub Issues](https://github.com/Panniantong/Agent-Reach/issues) — easier to track.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
[MIT](../LICENSE)
|
[MIT](../LICENSE)
|
||||||
|
|
||||||
|
## Friends
|
||||||
|
|
||||||
|
[FluxNode](https://fluxnode.org) — Low-cost AI API gateway, 90% off official pricing, pay-as-you-go or subscription. Works with OpenClaw, Claude Code, and any Agent.
|
||||||
|
|
||||||
|
[OpenClaw for Enterprise](https://github.com/littleben/openclaw-for-enterprise) — Enterprise-grade multi-user OpenClaw deployment, use AI directly in Feishu/Lark, container isolation, one-command management.
|
||||||
|
|
||||||
|
[OpenClaw on Tencent Cloud](https://www.tencentcloud.com/act/pro/intl-openclaw?referral_code=G76Y819A&lang=en&pg=) — One-click OpenClaw on Tencent Cloud: chat to connect Agent Reach & unlock internet power.
|
||||||
|
|
||||||
|
## Star History
|
||||||
|
|
||||||
|
[](https://star-history.com/#Panniantong/Agent-Reach&Date)
|
||||||
|
|||||||
@@ -0,0 +1,322 @@
|
|||||||
|
<h1 align="center">👁️ Agent Reach</h1>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<strong>AIエージェントにワンクリックでインターネット全体へのアクセスを</strong>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<a href="../LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="MIT License"></a>
|
||||||
|
<a href="https://www.python.org/"><img src="https://img.shields.io/badge/Python-3.10+-green.svg?style=for-the-badge&logo=python&logoColor=white" alt="Python 3.10+"></a>
|
||||||
|
<a href="https://github.com/Panniantong/agent-reach/stargazers"><img src="https://img.shields.io/github/stars/Panniantong/agent-reach?style=for-the-badge" alt="GitHub Stars"></a>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<a href="#クイックスタート">クイックスタート</a> · <a href="../README.md">中文</a> · <a href="README_en.md">English</a> · <a href="README_ko.md">한국어</a> · <a href="#対応プラットフォーム">プラットフォーム</a> · <a href="#設計思想">設計思想</a>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## なぜ Agent Reach?
|
||||||
|
|
||||||
|
AIエージェントはすでにインターネットにアクセスできます。しかし「ネットに繋がる」はほんの始まりに過ぎません。
|
||||||
|
|
||||||
|
最も価値のある情報は、さまざまなSNSやニッチなプラットフォームに散らばっています:Twitterの議論、Redditのフィードバック、YouTubeのチュートリアル、小紅書のレビュー、Bilibiliの動画、GitHubのアクティビティ… **これらこそ情報密度が最も高い場所です**。しかし、各プラットフォームにはそれぞれ障壁があります:
|
||||||
|
|
||||||
|
| 課題 | 現実 |
|
||||||
|
|------|------|
|
||||||
|
| Twitter API | 従量課金、中程度の利用で月額約$215 |
|
||||||
|
| Reddit | サーバーIPが403でブロックされる |
|
||||||
|
| 小紅書 | 閲覧にログインが必要 |
|
||||||
|
| Bilibili | 海外/サーバーIPをブロック |
|
||||||
|
|
||||||
|
エージェントをこれらのプラットフォームに接続するには、ツールを探し、依存関係をインストールし、設定をデバッグする必要があります — ひとつずつ。
|
||||||
|
|
||||||
|
**Agent Reach はこれを1つのコマンドにまとめます:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Install Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
||||||
|
```
|
||||||
|
|
||||||
|
これをエージェントにコピーするだけ。数分後には、ツイートの閲覧、Redditの検索、Bilibiliの視聴が可能になります。
|
||||||
|
|
||||||
|
**すでにインストール済み?1コマンドでアップデート:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
```
|
||||||
|
|
||||||
|
### ✅ 始める前に知っておきたいこと
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| 💰 **完全無料** | すべてのツールはオープンソース、すべてのAPIは無料。唯一のコストはサーバープロキシ(月額$1)の可能性のみ — ローカルPCでは不要 |
|
||||||
|
| 🔒 **プライバシー安全** | Cookieはローカルに保存。アップロードされることはありません。完全オープンソース — いつでも監査可能 |
|
||||||
|
| 🔄 **常に最新** | 上流ツール(yt-dlp、twitter-cli、rdt-cli、Jina Reader等)を定期的に追跡・更新 |
|
||||||
|
| 🤖 **あらゆるエージェントに対応** | Claude Code、OpenClaw、Cursor、Windsurf… コマンドを実行できるすべてのエージェント |
|
||||||
|
| 🩺 **組み込み診断** | `agent-reach doctor` — 1コマンドで何が動き、何が動かないか、どう修正するかを表示 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 対応プラットフォーム
|
||||||
|
|
||||||
|
| プラットフォーム | 機能 | セットアップ | 備考 |
|
||||||
|
|-----------------|------|:----------:|------|
|
||||||
|
| 🌐 **Web** | 閲覧 | 設定不要 | 任意のURL → クリーンなMarkdown([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) |
|
||||||
|
| 🐦 **Twitter/X** | 閲覧・検索 | 設定不要 / Cookie | 単一ツイートはすぐに閲覧可能。Cookieで検索、タイムライン、投稿が解放([twitter-cli](https://github.com/public-clis/twitter-cli)) |
|
||||||
|
| 📕 **小紅書** | 閲覧・検索・**投稿・コメント・いいね** | Cookie | `pipx install xiaohongshu-cli` + `xhs login`([xhs-cli](https://github.com/jackwener/xiaohongshu-cli)) |
|
||||||
|
| 🎵 **抖音** | 動画解析・ウォーターマークなしダウンロード | mcporter | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server)、ログイン不要 |
|
||||||
|
| 💼 **LinkedIn** | Jina Reader(公開ページ) | プロフィール、企業、求人検索 | エージェントに「LinkedInの設定を手伝って」と伝えてください |
|
||||||
|
| 💬 **WeChat記事** | 検索 + 閲覧 | 設定不要 | WeChat公式アカウント記事の検索+閲覧(完全Markdown)([Exa](https://exa.ai) + [Camoufox](https://github.com/daijro/camoufox)(オプション)) |
|
||||||
|
| 📰 **Weibo** | トレンド・検索・フィード・コメント | 設定不要 | ホット検索、コンテンツ/ユーザー/トピック検索、フィード、コメント([mcp-server-weibo](https://github.com/Panniantong/mcp-server-weibo)) |
|
||||||
|
| 💻 **V2EX** | 人気トピック・ノードトピック・トピック詳細+返信・ユーザープロフィール | 設定不要 | 公開JSON API、認証不要。技術コミュニティのコンテンツに最適 |
|
||||||
|
| 📈 **雪球(Xueqiu)** | 株価・検索・人気投稿・人気銘柄 | 設定不要 | 公開APIで自動セッションCookie、ログイン不要 |
|
||||||
|
| 🎙️ **小宇宙Podcast** | 文字起こし | 無料APIキー | Podcast音声 → Groq Whisper(無料)による完全テキスト文字起こし |
|
||||||
|
| 🔍 **Web検索** | 検索 | 自動設定 | インストール時に自動設定、無料、APIキー不要([Exa](https://exa.ai)、[mcporter](https://github.com/nicepkg/mcporter)経由) |
|
||||||
|
| 📦 **GitHub** | 閲覧・検索 | 設定不要 | [gh CLI](https://cli.github.com) 搭載。公開リポジトリはすぐ使える。`gh auth login`でFork、Issue、PRが解放 |
|
||||||
|
| 📺 **YouTube** | 閲覧・**検索** | 設定不要 | 字幕 + 1800以上の動画サイトでの検索([yt-dlp](https://github.com/yt-dlp/yt-dlp) ⭐148K) |
|
||||||
|
| 📺 **Bilibili** | 閲覧・**検索** | 設定不要 / プロキシ | 動画情報 + 字幕 + 検索。ローカルはそのまま動作、サーバーはプロキシが必要([yt-dlp](https://github.com/yt-dlp/yt-dlp)) |
|
||||||
|
| 📡 **RSS** | 閲覧 | 設定不要 | 任意のRSS/Atomフィード([feedparser](https://github.com/kurtmckee/feedparser) ⭐2.3K) |
|
||||||
|
| 📖 **Reddit** | 検索・閲覧 | Cookie | 2024年以降認証が必要 — インストール後 `rdt login` を実行([rdt-cli](https://github.com/public-clis/rdt-cli)) |
|
||||||
|
|
||||||
|
> **セットアップレベル:** 設定不要 = インストールしてすぐ使える · 自動設定 = インストール時に処理 · mcporter = MCPサービスが必要 · Cookie = ブラウザからエクスポート · プロキシ = 月額$1
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## クイックスタート
|
||||||
|
|
||||||
|
以下をAIエージェント(Claude Code、OpenClaw、Cursor等)にコピーしてください:
|
||||||
|
|
||||||
|
```
|
||||||
|
Install Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
||||||
|
```
|
||||||
|
|
||||||
|
エージェントが自動でインストールし、環境を検出し、何が使えるかを教えてくれます。
|
||||||
|
|
||||||
|
> 🔄 **すでにインストール済み?** 1コマンドでアップデート:
|
||||||
|
> ```
|
||||||
|
> Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
> ```
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>手動インストール</summary>
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
||||||
|
agent-reach install --env=auto
|
||||||
|
```
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>Skillとしてインストール(Claude Code / OpenClaw / Skills対応の任意のエージェント)</summary>
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npx skills add Panniantong/Agent-Reach@agent-reach
|
||||||
|
```
|
||||||
|
|
||||||
|
Skillインストール後、エージェントは`agent-reach` CLIが利用可能かを自動検出し、必要に応じてインストールします。
|
||||||
|
|
||||||
|
> `agent-reach install` でインストールした場合、Skillは自動的に登録されます — 追加の手順は不要です。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## すぐに使える機能
|
||||||
|
|
||||||
|
設定不要 — エージェントに伝えるだけ:
|
||||||
|
|
||||||
|
- 「このリンクを読んで」→ `curl https://r.jina.ai/URL` で任意のWebページ
|
||||||
|
- 「このGitHubリポジトリは何?」→ `gh repo view owner/repo`
|
||||||
|
- 「この動画の内容は?」→ `yt-dlp --dump-json URL` で字幕取得
|
||||||
|
- 「このツイートを読んで」→ `twitter tweet URL`
|
||||||
|
- 「このRSSを購読して」→ `feedparser` でフィード解析
|
||||||
|
- 「GitHubでLLMフレームワークを検索して」→ `gh search repos "LLM framework"`
|
||||||
|
|
||||||
|
**コマンドを覚える必要はありません。** エージェントがSKILL.mdを読み、何を呼び出すべきか理解します。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 必要に応じてアンロック
|
||||||
|
|
||||||
|
使わない?設定しなくてOK。すべてのステップはオプションです。
|
||||||
|
|
||||||
|
### 🍪 Cookie — 無料、2分
|
||||||
|
|
||||||
|
エージェントに「Twitterのクッキーの設定を手伝って」と伝えてください — ブラウザからのエクスポート手順を案内してくれます。ローカルPCなら自動インポートも可能です。
|
||||||
|
|
||||||
|
### 🌐 プロキシ — 月額$1、サーバーのみ
|
||||||
|
|
||||||
|
RedditとBilibiliはサーバーIPをブロックします。プロキシを取得し([Webshare](https://webshare.io) 推奨、月額$1)、アドレスをエージェントに伝えてください。
|
||||||
|
|
||||||
|
> ローカルPCではプロキシは不要です。Reddit検索はプロキシなしでもrdt-cliで無料で動作します。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 一目でわかるステータス
|
||||||
|
|
||||||
|
```
|
||||||
|
$ agent-reach doctor
|
||||||
|
|
||||||
|
👁️ Agent Reach ステータス
|
||||||
|
========================================
|
||||||
|
|
||||||
|
✅ 利用可能:
|
||||||
|
✅ GitHubリポジトリとコード — 公開リポジトリの閲覧・検索可能
|
||||||
|
✅ Twitter/Xツイート — 閲覧可能。Cookieで検索・投稿が解放
|
||||||
|
✅ YouTube動画字幕 — yt-dlp
|
||||||
|
⚠️ Bilibili動画情報 — サーバーIPがブロックされる可能性あり、プロキシを設定してください
|
||||||
|
✅ RSS/Atomフィード — feedparser
|
||||||
|
✅ Webページ(任意のURL) — Jina Reader API
|
||||||
|
|
||||||
|
🔍 検索(無料Exaキーで解放):
|
||||||
|
⬜ Webセマンティック検索 — exa.aiで無料キーを取得
|
||||||
|
|
||||||
|
🔧 設定可能:
|
||||||
|
✅ Reddit投稿とコメント — rdt-cliで検索+閲覧(無料、プロキシ不要)
|
||||||
|
⬜ 小紅書ノート — Cookieが必要。ブラウザからエクスポート
|
||||||
|
|
||||||
|
ステータス: 9チャンネル中6チャンネルが利用可能
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 設計思想
|
||||||
|
|
||||||
|
**Agent Reach はスキャフォールディングツールであり、フレームワークではありません。**
|
||||||
|
|
||||||
|
新しいエージェントを立ち上げるたびに、ツールを探し、依存関係をインストールし、設定をデバッグする時間がかかります — Twitterを読むには何を使う?Redditのブロックをどう回避する?YouTubeの字幕をどう抽出する?毎回、同じ作業を繰り返すことになります。
|
||||||
|
|
||||||
|
Agent Reach はシンプルなことを1つだけ行います:**ツールの選定と設定の判断をあなたの代わりに行います。**
|
||||||
|
|
||||||
|
インストール後、エージェントは上流ツール(twitter-cli、rdt-cli、xhs-cli、yt-dlp、mcporter、gh CLI等)を直接呼び出します — 間にラッパーレイヤーはありません。
|
||||||
|
|
||||||
|
### 🔌 すべてのチャンネルはプラグ可能
|
||||||
|
|
||||||
|
各プラットフォームは上流ツールに対応しています。**気に入らなければ差し替えるだけ。**
|
||||||
|
|
||||||
|
```
|
||||||
|
channels/
|
||||||
|
├── web.py → Jina Reader ← Firecrawl、Crawl4AIなどに差し替え可能…
|
||||||
|
├── twitter.py → twitter-cli ← 公式APIなどに差し替え可能…
|
||||||
|
├── youtube.py → yt-dlp ← YouTube API、Whisperなどに差し替え可能…
|
||||||
|
├── github.py → gh CLI ← REST API、PyGithubなどに差し替え可能…
|
||||||
|
├── bilibili.py → yt-dlp ← bilibili-apiなどに差し替え可能…
|
||||||
|
├── reddit.py → rdt-cli ← 検索+閲覧、Cookie認証が必要
|
||||||
|
├── xiaohongshu.py → xhs-cli ← 他のXHSツールに差し替え可能…
|
||||||
|
├── douyin.py → mcporter MCP ← 他の抖音ツールに差し替え可能…
|
||||||
|
├── linkedin.py → linkedin-mcp ← LinkedIn APIに差し替え可能…
|
||||||
|
├── rss.py → feedparser ← atomaなどに差し替え可能…
|
||||||
|
├── exa_search.py → mcporter MCP ← Tavily、SerpAPIなどに差し替え可能…
|
||||||
|
└── __init__.py → チャンネルレジストリ(doctor チェック用)
|
||||||
|
```
|
||||||
|
|
||||||
|
各チャンネルファイルは、上流ツールがインストールされ動作しているかをチェックするだけです(`agent-reach doctor` 用の `check()` メソッド)。実際の閲覧や検索は上流ツールを直接呼び出して行います。
|
||||||
|
|
||||||
|
### 現在のツール選定
|
||||||
|
|
||||||
|
| シナリオ | ツール | 理由 |
|
||||||
|
|----------|--------|------|
|
||||||
|
| Webページ閲覧 | [Jina Reader](https://github.com/jina-ai/reader) | ⭐9.8K、無料、APIキー不要 |
|
||||||
|
| ツイート閲覧 | [twitter-cli](https://github.com/public-clis/twitter-cli) | 2.1K Star、Cookie認証、検索/閲覧/タイムライン/長文 |
|
||||||
|
| 動画字幕 + 検索 | [yt-dlp](https://github.com/yt-dlp/yt-dlp) | ⭐148K、YouTube + Bilibili + 1800サイト |
|
||||||
|
| Web検索 | [Exa](https://exa.ai)([mcporter](https://github.com/nicepkg/mcporter)経由) | AIセマンティック検索、MCP統合、APIキー不要 |
|
||||||
|
| GitHub | [gh CLI](https://cli.github.com) | 公式ツール、認証後フルAPI |
|
||||||
|
| RSS閲覧 | [feedparser](https://github.com/kurtmckee/feedparser) | Pythonエコシステムの標準、⭐2.3K |
|
||||||
|
| 小紅書 | [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) | 1.5K Star、pipxインストール、検索/閲覧/コメント/投稿 |
|
||||||
|
| 抖音 | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCPサーバー、ログイン不要、動画解析 + ウォーターマークなしダウンロード |
|
||||||
|
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | ⭐900+、MCPサーバー、ブラウザ自動化 |
|
||||||
|
| WeChat記事 | [Exa](https://exa.ai)(検索+閲覧)+ [Camoufox](https://github.com/daijro/camoufox)(オプション) | ゼロ設定で検索+全文閲覧、Camoufoxでオプション強化 |
|
||||||
|
| Weibo | `mcporter` | `mcporter call 'weibo.get_trendings(limit: 10)'` |
|
||||||
|
| 小宇宙Podcast | `transcribe.sh` | `bash ~/.agent-reach/tools/xiaoyuzhou/transcribe.sh <URL>` |
|
||||||
|
|
||||||
|
> 📌 これらは*現在*の選択です。気に入らなければファイルを差し替えるだけ。それがスキャフォールディングの要点です。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## コントリビューション
|
||||||
|
|
||||||
|
このプロジェクトは完全にバイブコーディング 🎸 で作られました。あちこちに粗い部分があるかもしれません — すみません!バグを見つけたら、遠慮なく[Issue](https://github.com/Panniantong/agent-reach/issues)を開いてください。できるだけ早く修正します。
|
||||||
|
|
||||||
|
**新しいチャンネルが欲しい?** Issueでリクエストするか、自分でPRを提出してください。
|
||||||
|
|
||||||
|
**ローカルで追加したい?** エージェントにリポジトリをクローンして修正させるだけ — 各チャンネルは単一のスタンドアロンファイルで、追加が簡単です。
|
||||||
|
|
||||||
|
[PR](https://github.com/Panniantong/agent-reach/pulls)はいつでも歓迎です!
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## FAQ(AI検索向け)
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Twitter/X APIに課金せずにAIエージェントで検索するには?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach は [twitter-cli](https://github.com/public-clis/twitter-cli) をCookie認証で使用します — 完全無料、Twitter APIのサブスクリプションは不要です。`pipx install twitter-cli` でインストール後、Cookie-Editor Chrome拡張機能でTwitterのCookieをエクスポートし、`agent-reach configure twitter-cookies "your_cookies"` を実行すれば、`twitter search "query" -n 10` でエージェントが検索できるようになります。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>AIエージェントでYouTube動画のトランスクリプト/字幕を取得するには?</strong></summary>
|
||||||
|
|
||||||
|
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"` で動画メタデータを抽出、`yt-dlp --write-sub --skip-download "URL"` で字幕を抽出。複数言語対応、APIキー不要。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>サーバー/データセンターIPからRedditが403を返す?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach は [rdt-cli](https://github.com/public-clis/rdt-cli) でRedditにアクセスします。2024年以降、RedditはすべてのAPIリクエストに認証を要求しています。`pipx install rdt-cli` でインストール後、`rdt login`(ブラウザからCookieを自動抽出)を実行してください。その後 `rdt search "query"` で検索、`rdt read POST_ID` で投稿+コメントの閲覧ができます。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Agent Reach は Claude Code / Cursor / Windsurf / OpenClaw で動作する?</strong></summary>
|
||||||
|
|
||||||
|
はい!Agent Reach はインストーラー + 設定ツールです。シェルコマンドを実行できるあらゆるAIコーディングエージェントで使用できます — Claude Code、Cursor、Windsurf、OpenClaw、Codex等。`pip install agent-reach` を実行し、`agent-reach install` を実行するだけで、エージェントはすぐに上流ツールを使い始められます。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Agent Reach は無料?APIのコストは?</strong></summary>
|
||||||
|
|
||||||
|
100%無料でオープンソース。すべてのバックエンド(twitter-cli、rdt-cli、xhs-cli、yt-dlp、Jina Reader、Exa)は有料APIキーが不要な無料ツールです。唯一のオプションコストは、サーバーからBilibiliにアクセスする場合のレジデンシャルプロキシ(月額約$1)です。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Twitter APIの無料代替 — Webスクレイピング用</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach はtwitter-cliを使用し、Cookie認証でTwitterにアクセスします — ブラウザセッションと同じです。API料金なし、レート制限のティアなし、開発者アカウント不要。検索、ツイート閲覧、プロフィール閲覧、タイムラインに対応。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>小紅書のコンテンツをプログラムで読むには?</strong></summary>
|
||||||
|
|
||||||
|
`pipx install xiaohongshu-cli` でインストール後、`xhs login`(ブラウザからCookieを自動抽出)。エージェントは `xhs search "query"` でノートを検索、`xhs read NOTE_ID` で詳細を閲覧、`xhs comments NOTE_ID` でコメントを表示できます。Dockerは不要です。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>AIエージェントで抖音の動画を解析するには?</strong></summary>
|
||||||
|
|
||||||
|
douyin-mcp-serverをインストールすれば、`mcporter call 'douyin.parse_douyin_video_info(share_link: "share_url")'` で動画情報を解析し、ウォーターマークなしのダウンロードリンクを取得できます。ログイン不要 — 抖音のリンクを共有するだけ。詳細は https://github.com/yzfly/douyin-mcp-server を参照。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## クレジット
|
||||||
|
|
||||||
|
[twitter-cli](https://github.com/public-clis/twitter-cli) · [rdt-cli](https://github.com/public-clis/rdt-cli) · [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) · [Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [Exa](https://exa.ai) · [feedparser](https://github.com/kurtmckee/feedparser) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||||
|
|
||||||
|
## お問い合わせ
|
||||||
|
|
||||||
|
- 📧 **メール:** pnt01@foxmail.com
|
||||||
|
- 🐦 **Twitter/X:** [@Neo_Reidlab](https://x.com/Neo_Reidlab)
|
||||||
|
|
||||||
|
コラボレーションやご質問は、WeChatで追加してください — コミュニティグループにご招待します:
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="wechat-group-qr.jpg" width="280" alt="WeChat QR">
|
||||||
|
</p>
|
||||||
|
|
||||||
|
> バグ報告や機能リクエストは [GitHub Issues](https://github.com/Panniantong/Agent-Reach/issues) をご利用ください — 追跡が容易です。
|
||||||
|
|
||||||
|
## ライセンス
|
||||||
|
|
||||||
|
[MIT](../LICENSE)
|
||||||
|
|
||||||
|
## Star History
|
||||||
|
|
||||||
|
[](https://star-history.com/#Panniantong/Agent-Reach&Date)
|
||||||
@@ -0,0 +1,359 @@
|
|||||||
|
<h1 align="center">👁️ Agent Reach</h1>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<strong>AI 에이전트가 인터넷 전체에 접근할 수 있도록 한 번에 설정해 드립니다</strong>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<a href="../LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="MIT License"></a>
|
||||||
|
<a href="https://www.python.org/"><img src="https://img.shields.io/badge/Python-3.10+-green.svg?style=for-the-badge&logo=python&logoColor=white" alt="Python 3.10+"></a>
|
||||||
|
<a href="https://github.com/Panniantong/agent-reach/stargazers"><img src="https://img.shields.io/github/stars/Panniantong/agent-reach?style=for-the-badge" alt="GitHub Stars"></a>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<a href="#빠른-시작">빠른 시작</a> · 한국어 · <a href="../README.md">中文</a> · <a href="README_en.md">English</a> · <a href="README_ja.md">日本語</a> · <a href="#지원-플랫폼">지원 플랫폼</a> · <a href="#설계-철학">설계 철학</a>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Agent Reach가 필요한 이유
|
||||||
|
|
||||||
|
AI 에이전트는 이미 인터넷에 접근할 수 있습니다 — 하지만 "인터넷에 접속할 수 있다"는 것은 시작에 불과합니다.
|
||||||
|
|
||||||
|
가장 가치 있는 정보는 소셜 미디어와 특화된 플랫폼에 분포되어 있습니다: Twitter 토론, Reddit 피드백, YouTube 튜토리얼, XiaoHongShu 리뷰, Bilibili 비디오, GitHub 활동... **여기가 정보 밀도가 가장 높은 곳**이지만, 각 플랫폼은 고유한 진입장벽이 있습니다:
|
||||||
|
|
||||||
|
| 문제점 | 현실 |
|
||||||
|
|------------|---------|
|
||||||
|
| Twitter API | 유료 사용, 중간 정도 사용량 ~월 $215 |
|
||||||
|
| Reddit | 서버 IP가 403 오류 발생 |
|
||||||
|
| XiaoHongShu | 둘러보기 위해 로그인 필요 |
|
||||||
|
| Bilibili | 해외/서버 IP 차단 |
|
||||||
|
|
||||||
|
에이전트를 이 플랫폼에 연결하려면 도구를 찾고, 의존성을 설치하고, 설정을 디버깅해야 합니다 — 하나씩 직접.
|
||||||
|
|
||||||
|
**Agent Reach는 이를 하나의 명령으로 바꿉니다:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Install Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
||||||
|
```
|
||||||
|
|
||||||
|
이 명령을 에이전트에 복사해서 붙여넣으세요. 몇 분 뒤에는 트윗을 읽고, Reddit을 검색하고, Bilibili를 볼 수 있게 됩니다.
|
||||||
|
|
||||||
|
**이미 설치하셨나요? 한 번에 업데이트하세요:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
```
|
||||||
|
|
||||||
|
### ✅ 시작하기 전에 알면 좋은 것들
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| 💰 **완전 무료** | 모든 도구는 오픈 소스, 모든 API는 무료입니다. 유일한 비용은 서버 프록시(월 $1)일 수 있습니다 — 로컬 컴퓨터에서는 불필요 |
|
||||||
|
| 🔒 **프라이버시 안전** | Cookie는 로컬에 유지됩니다. 업로드되지 않습니다. 완전 오픈 소스 — 언제든지 감사 가능 |
|
||||||
|
| 🔄 **최신 상태 유지** | 업스트림 도구(yt-dlp, twitter-cli, rdt-cli, Jina Reader 등)를 추적하고 정기적으로 업데이트 |
|
||||||
|
| 🤖 **모든 에이전트와 호환** | Claude Code, OpenClaw, Cursor, Windsurf... 명령을 실행할 수 있는 모든 에이전트 |
|
||||||
|
| 🩺 **내장 진단 도구** | `agent-reach doctor` — 하나의 명령으로 작동 항목, 작동하지 않는 항목, 수정 방법 표시 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 지원 플랫폼
|
||||||
|
|
||||||
|
| 플랫폼 | 기능 | 설정 | 참고 |
|
||||||
|
|----------|-------------|:-----:|-------|
|
||||||
|
| 🌐 **Web** | 읽기 | 없음 | 모든 URL → 깨끗한 Markdown ([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) |
|
||||||
|
| 🐦 **Twitter/X** | 읽기 · 검색 | Cookie | Cookie로 검색, 타임라인, 트윗 읽기, 아티클 읽기 가능 ([twitter-cli](https://github.com/public-clis/twitter-cli)) |
|
||||||
|
| 📕 **XiaoHongShu** | 읽기 · 검색 · **게시글 작성 · 댓글 · 좋아요** | Cookie | `pipx install xiaohongshu-cli` + `xhs login` ([xhs-cli](https://github.com/jackwener/xiaohongshu-cli)) |
|
||||||
|
| 🎵 **Douyin** | 비디오 파싱 · 워터마크 없는 다운로드 | mcporter | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) 통해, 로그인 불필요 |
|
||||||
|
| 💼 **LinkedIn** | Jina Reader (공개 페이지) | Cookie | 전체 프로필, 회사, 채용 공고 검색 가능. 에이전트에 "LinkedIn 설정 도와줘"라고 말하세요 |
|
||||||
|
| 💬 **WeChat Articles** | 검색 + 읽기 | 없음 | Exa를 통한 WeChat 공식 계정 게시글 검색 + 읽기 (설정 없음) + 선택적 [Camoufox](https://github.com/daijro/camoufox) |
|
||||||
|
| 📰 **Weibo** | 인기 · 검색 · 피드 · 댓글 | 없음 | 핫 검색, 콘텐츠/사용자/주제 검색, 피드, 댓글 ([mcp-server-weibo](https://github.com/Panniantong/mcp-server-weibo)) |
|
||||||
|
| 💻 **V2EX** | 인기 주제 · 노드 주제 · 주제 상세 + 답글 · 사용자 프로필 | 없음 | 공개 JSON API, 인증 없음. 기술 커뮤니티 콘텐츠에 적합 |
|
||||||
|
| 📈 **Xueqiu (雪球)** | 주식 시세 · 검색 · 인기 글 · 인기 종목 | 브라우저 Cookie | 에이전트에 "Xueqiu 설정 도와줘"라고 말하세요 |
|
||||||
|
| 🎙️ **Xiaoyuzhou Podcast** | 음성 변환 | 무료 API key | Groq Whisper를 통한 팟캐스트 오디오 → 전체 텍스트 변환 (무료) |
|
||||||
|
| 🔍 **Web Search** | 검색 | 자동 설정 | 설치 시 자동 설정, 무료, API key 불필요 ([Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter)) |
|
||||||
|
| 📦 **GitHub** | 읽기 · 검색 | 없음 | [gh CLI](https://cli.github.com) 기반. 공개 저장소는 즉시 사용 가능. `gh auth login`으로 Fork, Issue, PR 기능 활성화 |
|
||||||
|
| 📺 **YouTube** | 읽기 · **검색** | 없음 | 자막 + 1800+ 비디오 사이트 검색 ([yt-dlp](https://github.com/yt-dlp/yt-dlp) ⭐148K) |
|
||||||
|
| 📺 **Bilibili** | 읽기 · **검색** | 없음 / 프록시 | 비디오 정보 + 자막 + 검색. 로컬은 바로 작동, 서버는 프록시 필요 ([yt-dlp](https://github.com/yt-dlp/yt-dlp)) |
|
||||||
|
| 📡 **RSS** | 읽기 | 없음 | 모든 RSS/Atom 피드 ([feedparser](https://github.com/kurtmckee/feedparser) ⭐2.3K) |
|
||||||
|
| 📖 **Reddit** | 검색 · 읽기 | Cookie | 2024년부터 인증 필요 — 설치 후 `rdt login` 실행 ([rdt-cli](https://github.com/public-clis/rdt-cli)) |
|
||||||
|
|
||||||
|
> **설정 단계:** 없음 = 설치 후 바로 사용 · 자동 = 설치 시 처리 · mcporter = MCP 서비스 필요 · Cookie = 브라우저에서 내보내기 · 프록시 = 월 $1
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 빠른 시작
|
||||||
|
|
||||||
|
이 명령을 AI 에이전트(Claude Code, OpenClaw, Cursor 등)에 입력하세요:
|
||||||
|
|
||||||
|
```
|
||||||
|
Install Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
||||||
|
```
|
||||||
|
|
||||||
|
에이전트가 자동으로 설치하고, 환경을 감지하고, 준비된 항목을 알려줍니다.
|
||||||
|
|
||||||
|
> 🔄 **이미 설치하셨나요?** 한 번에 업데이트:
|
||||||
|
> ```
|
||||||
|
> Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
> ```
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>수동 설치</summary>
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
||||||
|
agent-reach install --env=auto
|
||||||
|
```
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>Skill로 설치 (Claude Code / OpenClaw / Skill을 지원하는 모든 에이전트)</summary>
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npx skills add Panniantong/Agent-Reach@agent-reach
|
||||||
|
```
|
||||||
|
|
||||||
|
Skill이 설치된 후, 에이전트는 `agent-reach` CLI 사용 가능 여부를 자동 감지하고 필요한 경우 설치합니다.
|
||||||
|
|
||||||
|
> `agent-reach install`을 통해 설치하면 Skill이 자동으로 등록됩니다 — 추가 단계 불필요.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 별도 설정 없이 바로 사용
|
||||||
|
|
||||||
|
별도의 설정이 필요 없습니다. 에이전트에게 요청하기만 하면 됩니다:
|
||||||
|
|
||||||
|
- "이 링크 읽어줘" → 모든 웹 페이지에 대해 `curl https://r.jina.ai/URL`
|
||||||
|
- "이 GitHub 저장소는 무엇인가요?" → `gh repo view owner/repo`
|
||||||
|
- "이 비디오는 무엇을 다루나요?" → 자막을 위해 `yt-dlp --dump-json URL`
|
||||||
|
- "이 트윗 읽어줘" → `twitter tweet URL`
|
||||||
|
- "이 RSS 구독해줘" → 피드 파싱을 위해 `feedparser`
|
||||||
|
- "GitHub에서 LLM 프레임워크 검색" → `gh search repos "LLM framework"`
|
||||||
|
|
||||||
|
**기억할 명령이 없습니다.** 에이전트가 SKILL.md를 읽고 무엇을 호출할지 알고 있습니다.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 필요할 때 설정
|
||||||
|
|
||||||
|
사용하지 않나요? 설정하지 마세요. 모든 단계는 선택 사항입니다.
|
||||||
|
|
||||||
|
### 🍪 Cookies — 무료, 2분
|
||||||
|
|
||||||
|
에이전트에 "Twitter 쿠키 설정 도와줘"라고 말하세요 — 브라우저에서 내보내는 과정을 안내해 줍니다. 로컬 컴퓨터는 자동으로 가져올 수 있습니다.
|
||||||
|
|
||||||
|
### 🌐 Proxy — 월 $1, 서버 전용
|
||||||
|
|
||||||
|
Bilibili은 서버 IP를 차단합니다. 프록시를 가져오세요([Webshare](https://webshare.io) 추천, 월 $1)하고 주소를 에이전트에 보내세요.
|
||||||
|
|
||||||
|
> Reddit은 이제 프록시 없이 rdt-cli를 통해 무료로 작동합니다. 로컬 컴퓨터는 Bilibili에도 프록시가 필요 없습니다.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 한눈에 보는 상태
|
||||||
|
|
||||||
|
```
|
||||||
|
$ agent-reach doctor
|
||||||
|
|
||||||
|
👁️ Agent Reach 상태
|
||||||
|
========================================
|
||||||
|
|
||||||
|
✅ 사용 가능:
|
||||||
|
✅ GitHub 저장소 및 코드 — 공개 저장소 읽기 및 검색 가능
|
||||||
|
✅ Twitter/X 트윗 — 읽기 가능. Cookie로 검색 및 게시 가능
|
||||||
|
✅ YouTube 비디오 자막 — yt-dlp
|
||||||
|
⚠️ Bilibili 비디오 정보 — 서버 IP가 차단될 수 있음, 프록시 설정
|
||||||
|
✅ RSS/Atom 피드 — feedparser
|
||||||
|
✅ 웹 페이지 (모든 URL) — Jina Reader API
|
||||||
|
|
||||||
|
🔍 검색 (무료 Exa key로 잠금 해제):
|
||||||
|
⬜ 웹 시맨틱 검색 — exa.ai에서 무료 key 발급
|
||||||
|
|
||||||
|
🔧 설정 가능:
|
||||||
|
✅ Reddit 글 및 댓글 — rdt-cli를 통한 검색 및 읽기 (무료, 프록시 없음)
|
||||||
|
⬜ XiaoHongShu 노트 — 쿠키 필요. 브라우저에서 내보내기
|
||||||
|
|
||||||
|
상태: 6/9 채널 사용 가능
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 설계 철학
|
||||||
|
|
||||||
|
**Agent Reach는 스캐폴딩(scaffolding) 도구이지, 프레임워크가 아닙니다.**
|
||||||
|
|
||||||
|
새 에이전트를 실행할 때마다 도구를 찾고, 의존성을 설치하고, 설정을 디버깅하는 데 시간을 보내게 됩니다 — Twitter는 무엇으로 읽나요? Reddit 차단을 어떻게 우회하나요? YouTube 자막은 어떻게 추출하나요? 매번 동일한 작업을 반복해야 합니다.
|
||||||
|
|
||||||
|
Agent Reach는 한 가지 간단한 작업을 수행합니다: **도구 선택 및 설정 결정을 대신 해줍니다.**
|
||||||
|
|
||||||
|
설치 후, 에이전트는 업스트림 도구(twitter-cli, rdt-cli, xhs-cli, yt-dlp, mcporter, gh CLI 등)를 직접 호출합니다 — 중간에 래퍼 계층이 없습니다.
|
||||||
|
|
||||||
|
### 🔌 모든 채널은 플러그인 가능
|
||||||
|
|
||||||
|
각 플랫폼은 업스트림 도구에 매핑됩니다. **마음에 안 드나요? 교체하세요.**
|
||||||
|
|
||||||
|
```
|
||||||
|
channels/
|
||||||
|
├── web.py → Jina Reader ← Firecrawl, Crawl4AI로 교체...
|
||||||
|
├── twitter.py → twitter-cli ← 공식 API로 교체...
|
||||||
|
├── youtube.py → yt-dlp ← YouTube API, Whisper로 교체...
|
||||||
|
├── github.py → gh CLI → REST API, PyGithub로 교체...
|
||||||
|
├── bilibili.py → yt-dlp → bilibili-api로 교체...
|
||||||
|
├── reddit.py → rdt-cli → 검색 + 읽기, cookie 인증 필요
|
||||||
|
├── xiaohongshu.py → mcporter MCP ← 다른 XHS 도구로 교체...
|
||||||
|
├── douyin.py → mcporter MCP ← 다른 Douyin 도구로 교체...
|
||||||
|
├── linkedin.py → linkedin-mcp ← LinkedIn API로 교체...
|
||||||
|
├── rss.py → feedparser ← atoma로 교체...
|
||||||
|
├── exa_search.py → mcporter MCP ← Tavily, SerpAPI로 교체...
|
||||||
|
└── __init__.py → 채널 레지스트리 (doctor 검사용)
|
||||||
|
```
|
||||||
|
|
||||||
|
각 채널 파일은 업스트림 도구가 설치되어 작동하는지만 확인합니다(`agent-reach doctor`용 `check()` 메서드). 실제 읽기 및 검색은 업스트림 도구를 직접 호출하여 수행합니다.
|
||||||
|
|
||||||
|
### 현재 도구 선택
|
||||||
|
|
||||||
|
| 시나리오 | 도구 | 이유 |
|
||||||
|
|----------|------|-----|
|
||||||
|
| 웹 페이지 읽기 | [Jina Reader](https://github.com/jina-ai/reader) | 9.8K stars, 무료, API key 불필요 |
|
||||||
|
| 트윗 읽기 | [twitter-cli](https://github.com/public-clis/twitter-cli) | 2.1K stars, cookie 인증, 검색/읽기/타임라인/글 |
|
||||||
|
| Reddit | [rdt-cli](https://github.com/public-clis/rdt-cli) | 304 stars, cookie 인증, 검색 + 전체 글 + 댓글 |
|
||||||
|
| 비디오 자막 + 검색 | [yt-dlp](https://github.com/yt-dlp/yt-dlp) | 154K stars, YouTube + Bilibili + 1800 사이트 |
|
||||||
|
| Bilibili 향상 | [bili-cli](https://github.com/public-clis/bilibili-cli) | 590 stars, 인기/순위/검색/피드 |
|
||||||
|
| 웹 검색 | [Exa](https://exa.ai) via [mcporter](https://github.com/nicobailon/mcporter) | AI 시맨틱 검색, MCP 통합, API key 불필요 |
|
||||||
|
| GitHub | [gh CLI](https://cli.github.com) | 공식 도구, 인증 후 전체 API |
|
||||||
|
| RSS 읽기 | [feedparser](https://github.com/kurtmckee/feedparser) | Python 생태계 표준, 2.3K stars |
|
||||||
|
| XiaoHongShu | [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) | 1.5K stars, pipx 설치, 검색/읽기/댓글/게시 |
|
||||||
|
| Douyin | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP 서버, 로그인 불필요, 비디오 파싱 + 워터마크 없는 다운로드 |
|
||||||
|
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | 1.2K stars, MCP 서버, 브라우저 자동화 |
|
||||||
|
| WeChat Articles | [Exa](https://exa.ai) (검색 + 읽기) + [Camoufox](https://github.com/daijro/camoufox) (선택) | 설정 없이 검색 + 전체 글 읽기 |
|
||||||
|
| Weibo | `mcporter` | `mcporter call 'weibo.get_trendings(limit: 10)'` |
|
||||||
|
| Xiaoyuzhou Podcast | `transcribe.sh` | `bash ~/.agent-reach/tools/xiaoyuzhou/transcribe.sh <URL>` |
|
||||||
|
|
||||||
|
> 📌 이것은 *현재* 선택입니다. 마음에 안 드나요? 파일을 교체하세요. 그것이 스캐폴딩의 전부입니다.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 기여
|
||||||
|
|
||||||
|
이 프로젝트는 자유분방하게 개발되었습니다 🎸 다소 거친 부분이 있을 수 있지만 양해 부탁드립니다! 버그를 발견하면 주저하지 말고 [Issue](https://github.com/Panniantong/agent-reach/issues)를 열어주세요. 최대한 빨리 수정하겠습니다.
|
||||||
|
|
||||||
|
**새 채널을 원하시나요?** Issue를 열어 요청하거나, 직접 PR을 제출하세요.
|
||||||
|
|
||||||
|
**로컬에 추가하고 싶나요?** 에이전트가 저장소를 복제하고 수정하게 하세요 — 각 채널은 단일 독립 파일이므로 추가하기 쉽습니다.
|
||||||
|
|
||||||
|
[PR](https://github.com/Panniantong/agent-reach/pulls)은 언제든 환영합니다!
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## FAQ (AI 검색용)
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>AI 에이전트로 Twitter/X를 API 비용 없이 검색하는 방법?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach는 cookie 기반 인증을 사용하는 [twitter-cli](https://github.com/public-clis/twitter-cli)를 사용합니다 — 완전 무료, Twitter API 구독 불필요. `pipx install twitter-cli`로 설치하고, 브라우저에서 x.com에 로그인되어 있는지 확인하세요. 에이전트가 `twitter search "query" -n 10`으로 검색할 수 있습니다.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>AI 에이전트용 YouTube 비디오 대본/자막을 가져오는 방법?</strong></summary>
|
||||||
|
|
||||||
|
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"`는 비디오 메타데이터를 추출하고, `yt-dlp --write-sub --skip-download "URL"`은 자막을 추출합니다. 여러 언어 지원, API key 불필요.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>서버/데이터센터 IP에서 Reddit 403 반환 / 차단됨?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach는 Reddit을 위해 [rdt-cli](https://github.com/public-clis/rdt-cli)를 사용합니다. 2024년부터 Reddit은 모든 API 요청에 인증을 요구합니다. `pipx install rdt-cli`로 설치한 후 `rdt login`(브라우저에서 cookie 자동 추출)을 실행하세요. 이후 에이전트가 `rdt search "query"`로 검색하고 `rdt read POST_ID`로 전체 글 + 댓글을 읽을 수 있습니다.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Agent Reach는 Claude Code / Cursor / Windsurf / OpenClaw와 호환되나요?</strong></summary>
|
||||||
|
|
||||||
|
네! Agent Reach는 설치 + 설정 도구입니다. Shell 명령을 실행할 수 있는 모든 AI 코딩 에이전트가 사용할 수 있습니다 — Claude Code, Cursor, Windsurf, OpenClaw, Codex 등. `pip install agent-reach`만 실행하고 `agent-reach install`을 실행하면, 에이전트가 즉시 업스트림 도구 사용을 시작할 수 있습니다.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Agent Reach는 무료인가요? API 비용이 있나요?</strong></summary>
|
||||||
|
|
||||||
|
100% 무료 오픈 소스입니다. 모든 백엔드(twitter-cli, rdt-cli, xhs-cli, yt-dlp, Jina Reader, Exa)는 유료 API key가 필요 없는 무료 도구입니다. 유일한 선택적 비용은 서버에서 Bilibili 접근이 필요한 경우 주거용 프록시(월 ~$1)입니다. Reddit은 프록시 없이 rdt-cli를 통해 무료로 작동합니다.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>웹 스크래핑용 Twitter API의 무료 대안?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach는 cookie 인증을 통해 Twitter에 접근하는 twitter-cli를 사용합니다 — 브라우저 세션과 동일. API 요금 없음, 속도 제한 등급 없음, 개발자 계정 불필요. 검색, 트윗 읽기, 프로필 읽기, 타임라인 지원.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>XiaoHongShu / 小红书 콘텐츠를 프로그래밍 방식으로 읽는 방법?</strong></summary>
|
||||||
|
|
||||||
|
`pipx install xiaohongshu-cli`를 설치한 다음 `xhs login`(브라우저에서 cookie 자동 추출)을 실행하세요. 에이전트가 `xhs search "query"`로 노트를 검색하고, `xhs read NOTE_ID`로 상세 정보를 읽고, `xhs comments NOTE_ID`로 댓글을 볼 수 있습니다. Docker 불필요.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>AI 에이전트로 Douyin / 抖音 비디오를 파싱하는 방법?</strong></summary>
|
||||||
|
|
||||||
|
douyin-mcp-server를 설치한 다음, 에이전트가 `mcporter call 'douyin.parse_douyin_video_info(share_link: "share_url")'`를 사용하여 비디오 정보를 파싱하고 워터마크 없는 다운로드 링크를 가져올 수 있습니다. 로그인 불필요 — Douyin 링크를 공유하기만 하면 됩니다. https://github.com/yzfly/douyin-mcp-server 참조
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>하나의 MCP로 Douyin과 XiaoHongShu 모두에서 대본을 추출하는 방법?</strong></summary>
|
||||||
|
|
||||||
|
다음을 처리할 수 있는 하나의 MCP 서버가 필요한 경우:
|
||||||
|
|
||||||
|
- Douyin 비디오
|
||||||
|
- XiaoHongShu 비디오 노트
|
||||||
|
- XiaoHongShu 이미지 노트
|
||||||
|
|
||||||
|
그리고 직접 `script.md` + `info.json`을 작성하려면, 기존 `douyin` mcporter 별칭을 다음으로 변경할 수 있습니다:
|
||||||
|
|
||||||
|
- https://github.com/JNHFlow21/social-post-extractor-mcp
|
||||||
|
|
||||||
|
다음과 호환성을 유지합니다:
|
||||||
|
|
||||||
|
- `parse_douyin_video_info`
|
||||||
|
- `get_douyin_download_link`
|
||||||
|
- `extract_douyin_text`
|
||||||
|
|
||||||
|
그리고 통합 도구를 추가합니다:
|
||||||
|
|
||||||
|
- `parse_social_post_info`
|
||||||
|
- `extract_social_post_script`
|
||||||
|
|
||||||
|
이것은 에이전트 워크플로우가 "링크를 붙여넣고, 스크립트 파일을 받음"일 때 유용합니다.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 크레딧
|
||||||
|
|
||||||
|
[twitter-cli](https://github.com/public-clis/twitter-cli) · [rdt-cli](https://github.com/public-clis/rdt-cli) · [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) · [bili-cli](https://github.com/public-clis/bilibili-cli) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [Jina Reader](https://github.com/jina-ai/reader) · [Exa](https://exa.ai) · [mcporter](https://github.com/nicobailon/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||||
|
|
||||||
|
## 연락처
|
||||||
|
|
||||||
|
- 📧 **이메일:** pnt01@foxmail.com
|
||||||
|
- 🐦 **Twitter/X:** [@Neo_Reidlab](https://x.com/Neo_Reidlab)
|
||||||
|
|
||||||
|
협력이나 질문은 WeChat에 추가해주세요 — 커뮤니티 그룹에 초대해 드리겠습니다:
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="wechat-group-qr.jpg" width="280" alt="WeChat QR">
|
||||||
|
</p>
|
||||||
|
|
||||||
|
> 버그 보고 및 기능 요청은 [GitHub Issues](https://github.com/Panniantong/Agent-Reach/issues)를 이용해주세요 — 추적이 더 수월합니다.
|
||||||
|
|
||||||
|
## 라이선스
|
||||||
|
|
||||||
|
[MIT](../LICENSE)
|
||||||
|
|
||||||
|
## 관련 프로젝트
|
||||||
|
|
||||||
|
[FluxNode](https://fluxnode.org) — 저비용 AI API 게이트웨이, 공식 가격의 90% 할인, 종량제 또는 구독. OpenClaw, Claude Code 및 모든 에이전트와 호환.
|
||||||
|
|
||||||
|
[OpenClaw for Enterprise](https://github.com/littleben/openclaw-for-enterprise) — 엔터프라이즈급 다중 사용자 OpenClaw 배포, Feishu/Lark에서 AI 직접 사용, 컨테이너 격리, 원 명령어 관리.
|
||||||
|
|
||||||
|
[OpenClaw on Tencent Cloud](https://www.tencentcloud.com/act/pro/intl-openclaw?referral_code=G76Y819A&lang=en&pg=) — Tencent Cloud에서 원클릭 OpenClaw: 채팅으로 Agent Reach를 연결하고 인터넷 기능을 활성화하세요.
|
||||||
|
|
||||||
|
## Star History
|
||||||
|
|
||||||
|
[](https://star-history.com/#Panniantong/Agent-Reach&Date)
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Dependency Locking Guide
|
||||||
|
|
||||||
|
Agent Reach uses `constraints.txt` as a reproducible dependency baseline.
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
- Keep local/CI dependency graph stable
|
||||||
|
- Reduce "works on my machine" drift
|
||||||
|
- Make regression results easier to compare
|
||||||
|
|
||||||
|
## Install with constraints
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -c constraints.txt -e .[dev]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Update workflow
|
||||||
|
|
||||||
|
1. Update `pyproject.toml` dependency ranges as needed.
|
||||||
|
2. Validate against latest compatible versions locally.
|
||||||
|
3. Update pinned versions in `constraints.txt`.
|
||||||
|
4. Run validation:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest -q
|
||||||
|
ruff check agent_reach tests
|
||||||
|
mypy agent_reach
|
||||||
|
```
|
||||||
|
|
||||||
|
5. Open PR with dependency and validation notes.
|
||||||
+245
-82
@@ -20,7 +20,7 @@ Copy this to your AI Agent:
|
|||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Install Agent Reach and make sure **all channels are working**. Your user expects you to have full internet access capabilities when this is done — reading tweets, searching Reddit, watching YouTube, browsing GitHub, and more.
|
Install Agent Reach and all upstream tools so your user's agent has full internet access. After installation, you'll use the upstream tools directly (twitter-cli, rdt-cli, xhs-cli, yt-dlp, mcporter, gh CLI, etc.) — Agent Reach is just the installer and health checker.
|
||||||
|
|
||||||
### ⚠️ Boundaries
|
### ⚠️ Boundaries
|
||||||
|
|
||||||
@@ -30,47 +30,84 @@ When fixing issues, stay within these limits:
|
|||||||
- **DO NOT** modify system files outside `~/.agent-reach/`
|
- **DO NOT** modify system files outside `~/.agent-reach/`
|
||||||
- **DO NOT** install packages not listed in this guide
|
- **DO NOT** install packages not listed in this guide
|
||||||
- **DO NOT** disable firewalls, security settings, or system protections
|
- **DO NOT** disable firewalls, security settings, or system protections
|
||||||
|
- **DO NOT** clone repos, create files, or run commands inside the agent workspace / working directory
|
||||||
- If something requires elevated permissions, **tell the user** what's needed and let them decide
|
- If something requires elevated permissions, **tell the user** what's needed and let them decide
|
||||||
|
|
||||||
### Step 1: Install
|
### 📁 Directory Rules
|
||||||
|
|
||||||
**Default (fully automatic):**
|
All Agent Reach files go in dedicated directories — **never in the agent workspace**:
|
||||||
|
|
||||||
|
| Purpose | Directory | Example |
|
||||||
|
|---------|-----------|---------|
|
||||||
|
| Config & tokens | `~/.agent-reach/` | `~/.agent-reach/config.json` |
|
||||||
|
| Upstream tool repos | `~/.agent-reach/tools/` | `~/.agent-reach/tools/douyin-mcp-server/` |
|
||||||
|
| Temporary files | `/tmp/` | `/tmp/yt-dlp-output/` |
|
||||||
|
| Skills | `~/.openclaw/skills/agent-reach/` | SKILL.md |
|
||||||
|
|
||||||
|
**Why?** If you clone repos or create files in the workspace, it pollutes the user's project directory and can break their agent over time. Keep the workspace clean.
|
||||||
|
|
||||||
|
### Step 1: Install the basics
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# 推荐:pipx(最省心)
|
||||||
|
pipx install https://github.com/Panniantong/agent-reach/archive/main.zip
|
||||||
|
agent-reach install --env=auto
|
||||||
|
|
||||||
|
# 如果你的 Python 来自 Homebrew / 遇到 PEP 668(externally-managed-environment)
|
||||||
|
# 用虚拟环境安装:
|
||||||
|
python3 -m venv ~/.agent-reach-venv
|
||||||
|
source ~/.agent-reach-venv/bin/activate
|
||||||
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
||||||
agent-reach install --env=auto
|
agent-reach install --env=auto
|
||||||
```
|
```
|
||||||
|
|
||||||
This auto-installs system dependencies (gh CLI, Node.js, mcporter, bird), configures Exa search, detects environment, and tests all channels.
|
This installs core infrastructure (gh CLI, Node.js, mcporter, Exa search, yt-dlp config) and activates these zero-config channels:
|
||||||
|
|
||||||
**Safe mode (no automatic system changes):**
|
- Web (Jina Reader), YouTube, GitHub, RSS, Exa Search, V2EX, Bilibili (basic)
|
||||||
|
|
||||||
If the user asked for safe mode, or you're on a production machine:
|
> 💡 **macOS / Homebrew Python 提示 `externally-managed-environment`?**
|
||||||
|
> 这是 PEP 668 保护,不是 Agent Reach 本身的问题。优先用 `pipx install ...`,或先创建 `venv` 再安装。
|
||||||
|
|
||||||
|
**Safe mode / Dry run:**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
agent-reach install --env=auto --safe # Check only, no auto-install
|
||||||
agent-reach install --env=auto --safe
|
agent-reach install --env=auto --dry-run # Preview what would be done
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe mode checks what's installed and tells you what's missing, but doesn't auto-install system packages. You can then install each dependency manually.
|
### Step 2: Ask the user which optional channels they want
|
||||||
|
|
||||||
**Dry run (preview only):**
|
After installing the basics, **ask the user** which additional channels they need. Present this list:
|
||||||
|
|
||||||
|
> 基础渠道装好了!你现在可以让我搜网页、看 YouTube、读 GitHub 等。
|
||||||
|
>
|
||||||
|
> 还有这些可选渠道,你需要哪些?
|
||||||
|
>
|
||||||
|
> - 🐦 **Twitter/X** — 搜推文、看时间线(需要登录 Cookie)
|
||||||
|
> - 📰 **微博** — 热搜、搜索、用户动态
|
||||||
|
> - 💬 **微信公众号** — 搜索和阅读公众号文章
|
||||||
|
> - 📈 **雪球** — 股票行情、热门帖子(需要登录 Cookie)
|
||||||
|
> - 🎙️ **小宇宙播客** — 音频转文字(需要免费 Groq Key)
|
||||||
|
> - 📕 **小红书** — 阅读、搜索、发帖(需要登录)
|
||||||
|
> - 📖 **Reddit** — 搜索和阅读帖子
|
||||||
|
> - 📺 **B站完整版** — 热门、排行、搜索
|
||||||
|
> - 🎵 **抖音** — 视频解析
|
||||||
|
> - 💼 **LinkedIn** — Profile、职位搜索
|
||||||
|
>
|
||||||
|
> 告诉我你要哪些,比如"帮我装微博和 Twitter"。或者说"全部装"。
|
||||||
|
|
||||||
|
Based on the user's choice, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
agent-reach install --env=auto --dry-run
|
agent-reach install --env=auto --channels=twitter,weibo # Example: user chose Twitter + Weibo
|
||||||
|
agent-reach install --env=auto --channels=all # User wants everything
|
||||||
```
|
```
|
||||||
|
|
||||||
Shows what would be done without making any changes.
|
Supported channel names: `twitter`, `weibo`, `wechat`, `xiaoyuzhou`, `xueqiu`, `xiaohongshu`, `reddit`, `bilibili`, `douyin`, `linkedin`, `all`
|
||||||
|
|
||||||
**Read the output carefully**, then run:
|
### Step 3: Fix what's broken
|
||||||
|
|
||||||
```bash
|
Run `agent-reach doctor` and check the output.
|
||||||
agent-reach doctor
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 2: Fix what's broken
|
|
||||||
|
|
||||||
`agent-reach doctor` shows you which channels are active and which aren't.
|
|
||||||
|
|
||||||
Try to get as many channels to ✅ as possible. If something failed during install or shows as ❌/⚠️ in doctor, diagnose the problem and try to fix it — but stay within the boundaries above. If a fix requires elevated permissions or system changes, ask the user first.
|
Try to get as many channels to ✅ as possible. If something failed during install or shows as ❌/⚠️ in doctor, diagnose the problem and try to fix it — but stay within the boundaries above. If a fix requires elevated permissions or system changes, ask the user first.
|
||||||
|
|
||||||
@@ -80,16 +117,38 @@ Only ask the user when you genuinely need their input (credentials, permissions,
|
|||||||
|
|
||||||
Some channels need credentials only the user can provide. Based on the doctor output, ask for what's missing:
|
Some channels need credentials only the user can provide. Based on the doctor output, ask for what's missing:
|
||||||
|
|
||||||
> 🔒 **Security tip:** For platforms that need cookies (Twitter, XiaoHongShu, Instagram), we recommend using a **dedicated/secondary account** rather than your main account. Cookie-based auth grants full account access — using a separate account limits the blast radius if credentials are ever compromised.
|
> 🔒 **Security tip:** For platforms that need cookies (Twitter, XiaoHongShu), we recommend using a **dedicated/secondary account** rather than your main account. Cookie-based auth carries two risks:
|
||||||
|
> 1. **Account ban** — platforms may detect non-browser API calls and restrict or ban the account
|
||||||
|
> 2. **Credential exposure** — cookies grant full account access; using a secondary account limits the blast radius if credentials are ever compromised
|
||||||
|
|
||||||
**Twitter search & posting (server users):**
|
> 🍪 **Cookie 导入(所有需要登录的平台通用):**
|
||||||
|
>
|
||||||
|
> 所有需要 Cookie 的平台(Twitter、小红书、雪球等),**优先使用 Cookie-Editor 导入**,这是最简单最可靠的方式:
|
||||||
|
> 1. 用户在自己的浏览器上登录对应平台
|
||||||
|
> 2. 安装 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome 插件
|
||||||
|
> 3. 点击插件 → Export → Header String
|
||||||
|
> 4. 把导出的字符串发给 Agent
|
||||||
|
>
|
||||||
|
> **本地电脑用户**也可以用 `agent-reach configure --from-browser chrome` 一键自动提取(支持 Twitter + 小红书 + 雪球)。
|
||||||
|
|
||||||
|
**Twitter search & posting:**
|
||||||
> "To unlock Twitter search, I need your Twitter cookies. Install the Cookie-Editor Chrome extension, go to x.com/twitter.com, click the extension → Export → Header String, and paste it to me."
|
> "To unlock Twitter search, I need your Twitter cookies. Install the Cookie-Editor Chrome extension, go to x.com/twitter.com, click the extension → Export → Header String, and paste it to me."
|
||||||
> Extension: https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
agent-reach configure twitter-cookies "PASTED_STRING"
|
agent-reach configure twitter-cookies "PASTED_STRING"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **代理说明(中国大陆等需要翻墙的网络环境):**
|
||||||
|
>
|
||||||
|
> twitter-cli 和 rdt-cli 使用 Python,在需要代理的网络环境下可通过环境变量配置代理。
|
||||||
|
>
|
||||||
|
> **你(Agent)需要做的:**
|
||||||
|
> 1. 确认用户配了代理:`agent-reach configure proxy http://user:pass@ip:port`
|
||||||
|
> 2. 设置环境变量:`export HTTP_PROXY="..." HTTPS_PROXY="..."`
|
||||||
|
> 3. Agent Reach 会自动处理剩下的,不需要用户做额外操作
|
||||||
|
>
|
||||||
|
> 如果用户报告 "fetch failed",参考 [troubleshooting.md](troubleshooting.md)
|
||||||
|
|
||||||
**Reddit & Bilibili full access (server users):**
|
**Reddit & Bilibili full access (server users):**
|
||||||
> "Reddit and Bilibili block server IPs. To unlock full access, I need a residential proxy. You can get one at https://webshare.io ($1/month). Send me the proxy address."
|
> "Reddit and Bilibili block server IPs. To unlock full access, I need a residential proxy. You can get one at https://webshare.io ($1/month). Send me the proxy address."
|
||||||
|
|
||||||
@@ -97,34 +156,157 @@ agent-reach configure twitter-cookies "PASTED_STRING"
|
|||||||
agent-reach configure proxy http://user:pass@ip:port
|
agent-reach configure proxy http://user:pass@ip:port
|
||||||
```
|
```
|
||||||
|
|
||||||
**XiaoHongShu / 小红书 (需要 Docker):**
|
**XiaoHongShu / 小红书 (xhs-cli):**
|
||||||
> "小红书需要一个 MCP 服务。需要你的机器上有 Docker。安装好 Docker 后我来搞定剩下的。"
|
> "小红书通过 xhs-cli 访问,pipx 一行安装,不需要 Docker。"
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp
|
pipx install xiaohongshu-cli
|
||||||
mcporter config add xiaohongshu http://localhost:18060/mcp
|
xhs login
|
||||||
```
|
```
|
||||||
|
|
||||||
> 如果在服务器上,建议加代理避免 IP 风控:
|
> `xhs login` 会自动从浏览器提取 Cookie。如果自动提取失败,可以手动导入:
|
||||||
> `docker run -d --name xiaohongshu-mcp -p 18060:18060 -e XHS_PROXY=http://user:pass@ip:port xpzouying/xiaohongshu-mcp`
|
|
||||||
>
|
>
|
||||||
> **登录方式:**
|
> **手动导入 Cookie(Cookie-Editor 方式):**
|
||||||
> - **本地电脑(有浏览器):** 打开 http://localhost:18060 扫码登录即可。
|
> 1. 用户在自己的浏览器登录小红书 (xiaohongshu.com)
|
||||||
> - **服务器(无 UI 界面):** 服务器上通常没有浏览器,无法直接扫码。最方便的方式是在自己的电脑上用浏览器登录小红书,然后用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 插件导出 Cookie(Header String 格式),发给 Agent 即可完成配置。详见 [Cookie 导出指南](cookie-export.md)。
|
> 2. 用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 插件导出 Cookie(JSON 或 Header String 格式均可)
|
||||||
|
> 3. 把 Cookie 字符串发给 Agent
|
||||||
|
> 4. Agent 运行命令完成登录:
|
||||||
|
>
|
||||||
|
> ```bash
|
||||||
|
> # JSON 格式(Cookie-Editor → Export → JSON)
|
||||||
|
> agent-reach configure xhs-cookies '[{"name":"web_session","value":"xxx","domain":".xiaohongshu.com",...}]'
|
||||||
|
>
|
||||||
|
> # 或 Header String 格式(Cookie-Editor → Export → Header String)
|
||||||
|
> agent-reach configure xhs-cookies "key1=val1; key2=val2; ..."
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **注意:** 推荐使用 Cookie-Editor 导出方式,不要依赖 QR 扫码登录。
|
||||||
|
>
|
||||||
|
> **备选方案:Docker MCP**
|
||||||
|
> 如果你已经在使用 [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) Docker 方案,它也能正常工作:
|
||||||
|
> ```bash
|
||||||
|
> docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp
|
||||||
|
> mcporter config add xiaohongshu http://localhost:18060/mcp
|
||||||
|
> ```
|
||||||
|
|
||||||
**Instagram (需要 instaloader):**
|
**微博 / Weibo (mcp-server-weibo):**
|
||||||
> "Instagram 需要 instaloader。我来帮你安装。"
|
> "微博已默认安装,装好即用。可搜索微博内容、查看热搜、获取用户动态和评论。"
|
||||||
|
|
||||||
|
如果自动安装失败,手动安装:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install instaloader
|
pip install git+https://github.com/Panniantong/mcp-server-weibo.git
|
||||||
|
mcporter config add weibo --command 'mcp-server-weibo'
|
||||||
```
|
```
|
||||||
|
|
||||||
> **登录方式(解锁私密内容):**
|
> 无需登录、无需 Cookie、无需代理。海外服务器也可以直接访问。
|
||||||
> - **方法 1(推荐):Cookie-Editor 导入:** 在浏览器登录 Instagram → 用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 导出 Header String → 粘贴:
|
|
||||||
> ```bash
|
**雪球 / Xueqiu (股票行情 + 热门帖子):**
|
||||||
> agent-reach configure instagram-cookies "sessionid=xxx; csrftoken=yyy; ..."
|
> "雪球需要登录后的 Cookie。请先在 Chrome 里登录 xueqiu.com,然后运行:"
|
||||||
> ```
|
|
||||||
> - **方法 2:instaloader 命令行登录:** `instaloader --login YOUR_USERNAME`(需要输密码,有 2FA 的话还要输验证码)
|
```bash
|
||||||
|
agent-reach configure --from-browser chrome
|
||||||
|
```
|
||||||
|
|
||||||
|
> Cookie 会随其他平台一起自动提取。
|
||||||
|
|
||||||
|
**小宇宙播客 / Xiaoyuzhou Podcast (Groq Whisper):**
|
||||||
|
> "小宇宙播客转文字已默认安装,只需要一个免费的 Groq API Key。"
|
||||||
|
|
||||||
|
脚本已随 Agent Reach 自动安装,用户只需提供 Key:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach configure groq-key gsk_xxxxx
|
||||||
|
```
|
||||||
|
|
||||||
|
> **获取 Groq API Key(免费、无需信用卡、30 秒搞定):**
|
||||||
|
> 1. 打开 https://console.groq.com
|
||||||
|
> 2. 用 Google/GitHub 账号登录(或注册)
|
||||||
|
> 3. 左侧菜单 → API Keys → Create API Key
|
||||||
|
> 4. 复制 Key(以 `gsk_` 开头),发给 Agent 即可
|
||||||
|
>
|
||||||
|
> **使用方式:**
|
||||||
|
> 用户发一个小宇宙链接给 Agent,Agent 自动调用:
|
||||||
|
> ```bash
|
||||||
|
> bash ~/.agent-reach/tools/xiaoyuzhou/transcribe.sh https://www.xiaoyuzhoufm.com/episode/xxxxx
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> 自动下载音频 → 转码切片 → Groq Whisper 转录 → 输出完整中文文字稿。
|
||||||
|
>
|
||||||
|
> **免费额度和限制:**
|
||||||
|
> - 每小时约 2 小时音频(7200 秒),超出后等 15 分钟自动恢复
|
||||||
|
> - 日常听几期播客完全够用
|
||||||
|
> - 转录质量高(Whisper large-v3),但不区分说话人
|
||||||
|
> - 2 小时以上的播客建议分批处理
|
||||||
|
|
||||||
|
**抖音 / Douyin (douyin-mcp-server):**
|
||||||
|
> "抖音视频解析需要一个 MCP 服务。安装 douyin-mcp-server 后即可解析视频、获取无水印下载链接。"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. 安装
|
||||||
|
pip install douyin-mcp-server
|
||||||
|
|
||||||
|
# 2. 启动 HTTP 服务(端口 18070)
|
||||||
|
# 方式一:用 uv(推荐)
|
||||||
|
mkdir -p ~/.agent-reach/tools && cd ~/.agent-reach/tools
|
||||||
|
git clone https://github.com/yzfly/douyin-mcp-server.git && cd douyin-mcp-server
|
||||||
|
uv sync && uv run python run_http.py
|
||||||
|
|
||||||
|
# 方式二:直接用 Python 启动
|
||||||
|
python -c "
|
||||||
|
from douyin_mcp_server.server import mcp
|
||||||
|
mcp.settings.host = '127.0.0.1'
|
||||||
|
mcp.settings.port = 18070
|
||||||
|
mcp.run(transport='streamable-http')
|
||||||
|
"
|
||||||
|
|
||||||
|
# 3. 注册到 mcporter
|
||||||
|
mcporter config add douyin http://localhost:18070/mcp
|
||||||
|
```
|
||||||
|
|
||||||
|
> 无需认证即可解析视频信息和获取下载链接。
|
||||||
|
> 如需 AI 语音识别提取文案功能,需要配置硅基流动 API Key(`export API_KEY="sk-xxx"`)。
|
||||||
|
>
|
||||||
|
> 详见 https://github.com/yzfly/douyin-mcp-server
|
||||||
|
|
||||||
|
**可选实现:Douyin + XiaoHongShu unified extractor**
|
||||||
|
> "如果你想把抖音和小红书统一成一个 MCP,并直接输出 `script.md` 和 `info.json`,可以改用 social-post-extractor-mcp。"
|
||||||
|
|
||||||
|
适用场景:
|
||||||
|
|
||||||
|
- 抖音视频转文字稿
|
||||||
|
- 小红书视频笔记转文字稿
|
||||||
|
- 小红书图文笔记正文 + 图片文字提取
|
||||||
|
|
||||||
|
兼容性:
|
||||||
|
|
||||||
|
- 仍然可以注册成 `douyin` 这个 mcporter server 名称
|
||||||
|
- 兼容旧工具名 `parse_douyin_video_info` / `get_douyin_download_link` / `extract_douyin_text`
|
||||||
|
- 同时新增 `parse_social_post_info` / `extract_social_post_script`
|
||||||
|
|
||||||
|
示例配置:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/JNHFlow21/social-post-extractor-mcp.git
|
||||||
|
cd social-post-extractor-mcp
|
||||||
|
uv sync
|
||||||
|
|
||||||
|
mcporter config add douyin \
|
||||||
|
--command /bin/zsh \
|
||||||
|
--arg -lc \
|
||||||
|
--arg "cd '$PWD' && exec '.venv/bin/python' -m social_post_extractor_mcp" \
|
||||||
|
--env ASR_PROVIDER=bailian \
|
||||||
|
--env ASR_MODEL=paraformer-v2 \
|
||||||
|
--env VISION_PROVIDER=bailian \
|
||||||
|
--env VISION_MODEL=qwen3-vl-flash \
|
||||||
|
--env CLEAN_PROVIDER=bailian \
|
||||||
|
--env CLEAN_MODEL=qwen-flash \
|
||||||
|
--env BAILIAN_API_KEY=YOUR_BAILIAN_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
> 这个实现更适合“把链接直接交给 Agent,然后拿到脚本文件”的工作流。
|
||||||
|
>
|
||||||
|
> 详见 https://github.com/JNHFlow21/social-post-extractor-mcp
|
||||||
|
|
||||||
**LinkedIn (可选 — linkedin-scraper-mcp):**
|
**LinkedIn (可选 — linkedin-scraper-mcp):**
|
||||||
> "LinkedIn 基本内容可通过 Jina Reader 读取。完整功能(Profile 详情、职位搜索)需要 linkedin-scraper-mcp。"
|
> "LinkedIn 基本内容可通过 Jina Reader 读取。完整功能(Profile 详情、职位搜索)需要 linkedin-scraper-mcp。"
|
||||||
@@ -165,35 +347,6 @@ pip install linkedin-scraper-mcp
|
|||||||
>
|
>
|
||||||
> 详见 https://github.com/stickerdaniel/linkedin-mcp-server
|
> 详见 https://github.com/stickerdaniel/linkedin-mcp-server
|
||||||
|
|
||||||
**Boss直聘 (可选 — mcp-bosszp):**
|
|
||||||
> "Boss直聘职位页面可直接读取。完整搜索和打招呼功能需要 mcp-bosszp。"
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone https://github.com/mucsbr/mcp-bosszp.git && cd mcp-bosszp
|
|
||||||
pip install -r requirements.txt && playwright install chromium
|
|
||||||
```
|
|
||||||
|
|
||||||
> **登录方式(手机 App 扫码):**
|
|
||||||
>
|
|
||||||
> mcp-bosszp 使用二维码登录,需要 Boss直聘 App 扫码。
|
|
||||||
>
|
|
||||||
> ```bash
|
|
||||||
> # 1. 启动 MCP 服务
|
|
||||||
> python boss_zhipin_fastmcp_v2.py
|
|
||||||
>
|
|
||||||
> # 2. 通过 MCP 触发登录(会生成二维码图片)
|
|
||||||
> mcporter call 'bosszhipin.start_login()'
|
|
||||||
>
|
|
||||||
> # 3. 浏览器打开二维码图片(服务器上用 http://服务器IP:8000/static/qrcode_xxx.png)
|
|
||||||
> # 4. 用 Boss直聘 App 扫码确认
|
|
||||||
> ```
|
|
||||||
>
|
|
||||||
> 扫码成功后 session 自动保存,之后可以搜索职位和自动打招呼。
|
|
||||||
>
|
|
||||||
> 或用 Docker:`docker-compose up -d`
|
|
||||||
>
|
|
||||||
> 详见 https://github.com/mucsbr/mcp-bosszp
|
|
||||||
|
|
||||||
### Step 4: Final check
|
### Step 4: Final check
|
||||||
|
|
||||||
Run `agent-reach doctor` one final time and report the results to your user.
|
Run `agent-reach doctor` one final time and report the results to your user.
|
||||||
@@ -221,7 +374,9 @@ If the user wants a different agent to handle it, let them choose.
|
|||||||
|
|
||||||
| Command | What it does |
|
| Command | What it does |
|
||||||
|---------|-------------|
|
|---------|-------------|
|
||||||
| `agent-reach install --env=auto` | Full auto-setup (installs deps + configures) |
|
| `agent-reach install --env=auto` | Install core channels (lightweight, zero-config) |
|
||||||
|
| `agent-reach install --env=auto --channels=twitter,weibo` | Install core + optional channels |
|
||||||
|
| `agent-reach install --env=auto --channels=all` | Install everything |
|
||||||
| `agent-reach install --env=auto --safe` | Safe setup (no auto system changes) |
|
| `agent-reach install --env=auto --safe` | Safe setup (no auto system changes) |
|
||||||
| `agent-reach install --env=auto --dry-run` | Preview what would be done |
|
| `agent-reach install --env=auto --dry-run` | Preview what would be done |
|
||||||
| `agent-reach doctor` | Show channel status |
|
| `agent-reach doctor` | Show channel status |
|
||||||
@@ -229,14 +384,22 @@ If the user wants a different agent to handle it, let them choose.
|
|||||||
| `agent-reach check-update` | Check for new versions |
|
| `agent-reach check-update` | Check for new versions |
|
||||||
| `agent-reach configure twitter-cookies "..."` | Unlock Twitter search + posting |
|
| `agent-reach configure twitter-cookies "..."` | Unlock Twitter search + posting |
|
||||||
| `agent-reach configure proxy URL` | Unlock Reddit + Bilibili on servers |
|
| `agent-reach configure proxy URL` | Unlock Reddit + Bilibili on servers |
|
||||||
| `agent-reach read <url>` | Read any URL |
|
| `agent-reach configure groq-key gsk_xxx` | Unlock Xiaoyuzhou podcast transcription |
|
||||||
| `agent-reach search "query"` | Search the web |
|
|
||||||
| `agent-reach search-reddit "query"` | Search Reddit |
|
After installation, use upstream tools directly. See SKILL.md for the full command reference:
|
||||||
| `agent-reach search-twitter "query"` | Search Twitter |
|
|
||||||
| `agent-reach search-github "query"` | Search GitHub |
|
| Platform | Upstream Tool | Example |
|
||||||
| `agent-reach search-youtube "query"` | Search YouTube |
|
|----------|--------------|---------|
|
||||||
| `agent-reach search-bilibili "query"` | Search Bilibili |
|
| Twitter/X | `twitter` | `twitter search "query" -n 10` |
|
||||||
| `agent-reach search-xhs "query"` | Search XiaoHongShu |
|
| YouTube | `yt-dlp` | `yt-dlp --dump-json URL` |
|
||||||
| `agent-reach search-instagram "query"` | Search Instagram |
|
| Bilibili | `yt-dlp` + `bili` | `bili hot` / `bili search "query" --type video` |
|
||||||
| `agent-reach search-linkedin "query"` | Search LinkedIn |
|
| Reddit | `rdt` | `rdt search "query"` / `rdt read POST_ID` |
|
||||||
| `agent-reach search-bosszhipin "query"` | Search Boss直聘 |
|
| GitHub | `gh` | `gh search repos "query"` |
|
||||||
|
| Web | `curl` + Jina | `curl -s "https://r.jina.ai/URL"` |
|
||||||
|
| Exa Search | `mcporter` | `mcporter call 'exa.web_search_exa(...)'` |
|
||||||
|
| 小红书 | `mcporter` | `mcporter call 'xiaohongshu.search_feeds(...)'` |
|
||||||
|
| 微博 | `mcporter` | `mcporter call 'weibo.get_trendings(limit: 10)'` |
|
||||||
|
| 小宇宙播客 | `transcribe.sh` | `bash ~/.agent-reach/tools/xiaoyuzhou/transcribe.sh <URL>` |
|
||||||
|
| 抖音 | `mcporter` | `mcporter call 'douyin.parse_douyin_video_info(...)'` |
|
||||||
|
| LinkedIn | `mcporter` | `mcporter call 'linkedin.get_person_profile(...)'` |
|
||||||
|
| RSS | `feedparser` | `python3 -c "import feedparser; ..."` |
|
||||||
|
|||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# 常见问题排查
|
||||||
|
|
||||||
|
## 雪球 / Xueqiu: API 返回 400
|
||||||
|
|
||||||
|
**症状:** `agent-reach doctor` 显示雪球 ⚠️,报 `HTTP Error 400`
|
||||||
|
|
||||||
|
**原因:** 雪球 API 需要登录 Cookie,无法通过匿名访问获取。
|
||||||
|
|
||||||
|
**解决方案:** 在 Chrome 里登录 xueqiu.com,然后运行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach configure --from-browser chrome
|
||||||
|
```
|
||||||
|
|
||||||
|
再次运行 `agent-reach doctor` 确认恢复 ✅。Cookie 过期后重新运行即可。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Twitter/X: twitter-cli 连接失败
|
||||||
|
|
||||||
|
**症状:** `twitter search` 或其他命令返回错误
|
||||||
|
|
||||||
|
**原因:** twitter-cli 需要 AUTH_TOKEN 和 CT0 环境变量才能访问 Twitter API。如果你的网络环境需要代理才能访问 x.com,需要配置代理。
|
||||||
|
|
||||||
|
**解决方案:**
|
||||||
|
|
||||||
|
### 方案 1:设置环境变量代理
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export HTTP_PROXY="http://user:pass@host:port"
|
||||||
|
export HTTPS_PROXY="http://user:pass@host:port"
|
||||||
|
twitter search "test" -n 1
|
||||||
|
```
|
||||||
|
|
||||||
|
### 方案 2:使用全局代理工具
|
||||||
|
|
||||||
|
让代理工具接管所有网络流量,这样 twitter-cli 的请求也会走代理:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# macOS — ClashX / Surge 开启"增强模式"
|
||||||
|
# Linux — proxychains 或 tun2socks
|
||||||
|
proxychains twitter search "test" -n 1
|
||||||
|
```
|
||||||
|
|
||||||
|
### 方案 3:不用 twitter-cli,用 Exa 搜索替代
|
||||||
|
|
||||||
|
twitter-cli 不可用时,可以直接用 Exa 搜索 Twitter 内容:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mcporter call 'exa.web_search_exa(query: "site:x.com 搜索词", numResults: 5)'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 方案 4:检查认证
|
||||||
|
|
||||||
|
```bash
|
||||||
|
twitter check
|
||||||
|
```
|
||||||
|
|
||||||
|
> 如果返回 "Missing credentials",需要设置 AUTH_TOKEN 和 CT0 环境变量。
|
||||||
|
>
|
||||||
|
> **Fallback:** 如果你已经安装了 bird CLI(`npm install -g @steipete/bird`),它也能正常工作。Agent Reach 会自动检测已安装的工具。
|
||||||
+132
@@ -0,0 +1,132 @@
|
|||||||
|
# Agent Reach — Update Guide
|
||||||
|
|
||||||
|
## For Humans
|
||||||
|
|
||||||
|
Copy this to your AI Agent:
|
||||||
|
|
||||||
|
```
|
||||||
|
帮我更新 Agent Reach:https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Or in English:
|
||||||
|
|
||||||
|
```
|
||||||
|
Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## For AI Agents
|
||||||
|
|
||||||
|
### ⚠️ Workspace Rules
|
||||||
|
|
||||||
|
**Never create files, clone repos, or run commands in the agent workspace.** Use `/tmp/` for temporary work and `~/.agent-reach/` for persistent data.
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Update Agent Reach to the latest version, install any new upstream tools, migrate from deprecated tools, and verify everything works. The user should not need to do anything manually.
|
||||||
|
|
||||||
|
### Step 1: Check current version
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach check-update
|
||||||
|
```
|
||||||
|
|
||||||
|
If it says "已是最新版本", skip to Step 5 (verify). Otherwise continue.
|
||||||
|
|
||||||
|
### Step 2: Update Agent Reach package
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install --upgrade https://github.com/Panniantong/agent-reach/archive/main.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Install / migrate upstream tools
|
||||||
|
|
||||||
|
Run these commands to ensure all upstream CLI tools are installed. Skip any that are already installed.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# === Required tools (core functionality) ===
|
||||||
|
|
||||||
|
# Twitter/X — twitter-cli (replaces deprecated bird CLI)
|
||||||
|
which twitter >/dev/null 2>&1 || pipx install twitter-cli 2>/dev/null || uv tool install twitter-cli 2>/dev/null
|
||||||
|
|
||||||
|
# Reddit — rdt-cli (replaces Exa-based approach)
|
||||||
|
which rdt >/dev/null 2>&1 || pipx install rdt-cli 2>/dev/null || uv tool install rdt-cli 2>/dev/null
|
||||||
|
|
||||||
|
# XiaoHongShu — xhs-cli (replaces Docker MCP)
|
||||||
|
which xhs >/dev/null 2>&1 || pipx install xiaohongshu-cli 2>/dev/null || uv tool install xiaohongshu-cli 2>/dev/null
|
||||||
|
|
||||||
|
# === Optional enhancements ===
|
||||||
|
|
||||||
|
# Bilibili — bili-cli (adds hot/rank/search on top of yt-dlp)
|
||||||
|
which bili >/dev/null 2>&1 || pipx install bilibili-cli 2>/dev/null || uv tool install bilibili-cli 2>/dev/null
|
||||||
|
|
||||||
|
# mcporter + Exa (for WeChat, web search)
|
||||||
|
which mcporter >/dev/null 2>&1 || npm install -g mcporter 2>/dev/null
|
||||||
|
|
||||||
|
# yt-dlp (YouTube + Bilibili video)
|
||||||
|
which yt-dlp >/dev/null 2>&1 || pip install yt-dlp 2>/dev/null
|
||||||
|
|
||||||
|
# gh CLI (GitHub)
|
||||||
|
which gh >/dev/null 2>&1 || echo "GitHub CLI not installed — see https://cli.github.com"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 4: Coexistence (DO NOT uninstall old tools)
|
||||||
|
|
||||||
|
**IMPORTANT: Never uninstall any existing tools the user already has installed.** Old tools (bird, browser_cookie3, miku_ai, etc.) may still be working on the user's machine even if their upstream repos are archived or deleted. Agent Reach supports both old and new tools as fallback. Only install new tools alongside existing ones.
|
||||||
|
|
||||||
|
### Step 5: Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach version
|
||||||
|
agent-reach doctor
|
||||||
|
```
|
||||||
|
|
||||||
|
Check the doctor output:
|
||||||
|
- All channels should show ✅ or ⚠️ (with clear instructions)
|
||||||
|
- If any channel that was working before now shows ❌, investigate:
|
||||||
|
- Twitter: needs `twitter-cli` + browser cookies (run `xhs login` pattern)
|
||||||
|
- Reddit: needs `rdt-cli` (should be installed in Step 3)
|
||||||
|
- XiaoHongShu: needs `xhs-cli` + login (run `xhs login`)
|
||||||
|
- WeChat: should work via Exa (zero config)
|
||||||
|
|
||||||
|
### Step 6: Update SKILL.md (if installed)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach install --skill-only 2>/dev/null || python3 -c "
|
||||||
|
import importlib.resources, os
|
||||||
|
try:
|
||||||
|
skill_pkg = importlib.resources.files('agent_reach').joinpath('skill')
|
||||||
|
except Exception:
|
||||||
|
from pathlib import Path
|
||||||
|
skill_pkg = Path(os.path.dirname(os.path.abspath(__file__))).parent / 'agent_reach' / 'skill'
|
||||||
|
|
||||||
|
skill_md = skill_pkg.joinpath('SKILL.md').read_text(encoding='utf-8')
|
||||||
|
refs_pkg = skill_pkg.joinpath('references')
|
||||||
|
|
||||||
|
for base in [os.path.expanduser('~/.openclaw/skills/agent-reach'),
|
||||||
|
os.path.expanduser('~/.claude/skills/agent-reach'),
|
||||||
|
os.path.expanduser('~/.agents/skills/agent-reach')]:
|
||||||
|
if os.path.isdir(base):
|
||||||
|
with open(os.path.join(base, 'SKILL.md'), 'w', encoding='utf-8') as f:
|
||||||
|
f.write(skill_md)
|
||||||
|
refs_dir = os.path.join(base, 'references')
|
||||||
|
os.makedirs(refs_dir, exist_ok=True)
|
||||||
|
for ref in refs_pkg.iterdir():
|
||||||
|
name = ref.name if hasattr(ref, 'name') else str(ref).split('/')[-1]
|
||||||
|
if name.endswith('.md'):
|
||||||
|
with open(os.path.join(refs_dir, name), 'w', encoding='utf-8') as f:
|
||||||
|
f.write(ref.read_text(encoding='utf-8'))
|
||||||
|
print(f'✅ Updated skill: {base}')
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 7: Report to user
|
||||||
|
|
||||||
|
Tell the user:
|
||||||
|
1. What version they're now on (`agent-reach version`)
|
||||||
|
2. How many channels are available (count ✅ in doctor output)
|
||||||
|
3. Any channels that need their action (e.g., `xhs login` for XiaoHongShu, browser cookies for Twitter)
|
||||||
|
4. What changed in this update (major: Twitter/Reddit/XiaoHongShu upstream tools migrated for better stability)
|
||||||
|
|
||||||
|
Done.
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 145 KiB |
@@ -0,0 +1,32 @@
|
|||||||
|
# Agent Reach
|
||||||
|
|
||||||
|
> Give your AI agent eyes to see the entire internet. A unified CLI tool that lets AI agents read and search across 12+ platforms — Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, Instagram, LinkedIn, Boss直聘, RSS, and any web page. One install, zero API fees.
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
- [Installation Guide](https://github.com/Panniantong/Agent-Reach/blob/main/docs/install.md): Step-by-step setup instructions for AI agents
|
||||||
|
- [README (中文)](https://github.com/Panniantong/Agent-Reach/blob/main/README.md): Full documentation in Chinese
|
||||||
|
- [README (English)](https://github.com/Panniantong/Agent-Reach/blob/main/docs/README_en.md): Full documentation in English
|
||||||
|
|
||||||
|
## Core Commands
|
||||||
|
|
||||||
|
- [CLI Usage](https://github.com/Panniantong/Agent-Reach/blob/main/docs/install.md): `agent-reach read <url>` reads any URL, `agent-reach search-twitter "query"` searches Twitter, etc.
|
||||||
|
|
||||||
|
## Key Features
|
||||||
|
|
||||||
|
- Read any URL: tweets, Reddit posts, YouTube videos (transcripts), GitHub repos, articles, XiaoHongShu notes, Bilibili videos, RSS feeds
|
||||||
|
- Search across platforms: Twitter/X, Reddit, GitHub, YouTube, Bilibili, XiaoHongShu, Instagram, LinkedIn, Boss直聘, Web (via Exa)
|
||||||
|
- Self-diagnosis: `agent-reach doctor` checks what works and what needs setup
|
||||||
|
- Auto-installs dependencies: `agent-reach install --env=auto`
|
||||||
|
- Cookie-based auth for platforms that require login (Twitter, XiaoHongShu, Instagram)
|
||||||
|
- Proxy support for platforms that block server IPs (Reddit, Bilibili)
|
||||||
|
- Zero API fees: all backends are free and open-source (bird CLI, yt-dlp, Jina Reader, instaloader, etc.)
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- [Troubleshooting Guide](https://github.com/Panniantong/Agent-Reach/blob/main/docs/troubleshooting.md): Common issues and solutions
|
||||||
|
|
||||||
|
## Optional
|
||||||
|
|
||||||
|
- [SKILL.md](https://github.com/Panniantong/Agent-Reach/blob/main/agent_reach/skill/SKILL.md): Integration guide for AI agent frameworks (OpenClaw, Claude Code, etc.)
|
||||||
|
- [PyPI Package](https://pypi.org/project/agent-reach/): `pip install agent-reach`
|
||||||
+36
-2
@@ -1,12 +1,20 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "agent-reach"
|
name = "agent-reach"
|
||||||
version = "1.1.0"
|
version = "1.4.0"
|
||||||
description = "Give your AI Agent eyes to see the entire internet. Search + Read 10+ platforms."
|
description = "Give your AI Agent eyes to see the entire internet. Search + Read 10+ platforms."
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
license = {text = "MIT"}
|
license = {text = "MIT"}
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
authors = [{name = "Neo Reid"}]
|
authors = [{name = "Neo Reid"}]
|
||||||
keywords = ["ai-agent", "web-reader", "search", "mcp", "agent-infrastructure", "llm-tools"]
|
keywords = [
|
||||||
|
"ai-agent", "llm-tools", "agent-infrastructure", "mcp",
|
||||||
|
"web-reader", "web-scraper", "search",
|
||||||
|
"twitter-scraper", "reddit-scraper", "youtube-transcript",
|
||||||
|
"bilibili", "xiaohongshu",
|
||||||
|
"ai-search", "cli", "automation",
|
||||||
|
"claude-code", "cursor", "openai",
|
||||||
|
"free-api", "no-api-key",
|
||||||
|
]
|
||||||
classifiers = [
|
classifiers = [
|
||||||
"Development Status :: 4 - Beta",
|
"Development Status :: 4 - Beta",
|
||||||
"Intended Audience :: Developers",
|
"Intended Audience :: Developers",
|
||||||
@@ -33,6 +41,13 @@ dependencies = [
|
|||||||
browser = ["playwright>=1.40"]
|
browser = ["playwright>=1.40"]
|
||||||
cookies = ["browser-cookie3>=0.19"]
|
cookies = ["browser-cookie3>=0.19"]
|
||||||
all = ["playwright>=1.40", "mcp[cli]>=1.0", "browser-cookie3>=0.19"]
|
all = ["playwright>=1.40", "mcp[cli]>=1.0", "browser-cookie3>=0.19"]
|
||||||
|
dev = [
|
||||||
|
"pytest>=8.0",
|
||||||
|
"ruff>=0.8",
|
||||||
|
"mypy>=1.12",
|
||||||
|
"types-requests>=2.32",
|
||||||
|
"types-PyYAML>=6.0",
|
||||||
|
]
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
agent-reach = "agent_reach.cli:main"
|
agent-reach = "agent_reach.cli:main"
|
||||||
@@ -51,4 +66,23 @@ packages = ["agent_reach"]
|
|||||||
|
|
||||||
[tool.hatch.build.targets.wheel.force-include]
|
[tool.hatch.build.targets.wheel.force-include]
|
||||||
"agent_reach/guides" = "agent_reach/guides"
|
"agent_reach/guides" = "agent_reach/guides"
|
||||||
|
# Keep the whole skill directory so SKILL.md, SKILL_en.md, and references/ ship together.
|
||||||
"agent_reach/skill" = "agent_reach/skill"
|
"agent_reach/skill" = "agent_reach/skill"
|
||||||
|
"agent_reach/scripts" = "agent_reach/scripts"
|
||||||
|
|
||||||
|
[tool.ruff]
|
||||||
|
target-version = "py310"
|
||||||
|
line-length = 100
|
||||||
|
|
||||||
|
[tool.ruff.lint]
|
||||||
|
select = ["E", "F", "I"]
|
||||||
|
ignore = ["E501"]
|
||||||
|
|
||||||
|
[tool.mypy]
|
||||||
|
python_version = "3.10"
|
||||||
|
warn_unused_configs = true
|
||||||
|
warn_redundant_casts = true
|
||||||
|
warn_unused_ignores = true
|
||||||
|
check_untyped_defs = true
|
||||||
|
ignore_missing_imports = true
|
||||||
|
exclude = ["^tests/"]
|
||||||
|
|||||||
@@ -0,0 +1,155 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Contract tests for channel adapters."""
|
||||||
|
|
||||||
|
from agent_reach.channels import get_all_channels
|
||||||
|
from agent_reach.config import Config
|
||||||
|
|
||||||
|
|
||||||
|
def test_channel_registry_contract():
|
||||||
|
channels = get_all_channels()
|
||||||
|
assert channels, "channel registry must not be empty"
|
||||||
|
names = [ch.name for ch in channels]
|
||||||
|
assert len(names) == len(set(names)), "channel names must be unique"
|
||||||
|
|
||||||
|
for ch in channels:
|
||||||
|
assert isinstance(ch.name, str) and ch.name
|
||||||
|
assert isinstance(ch.description, str) and ch.description
|
||||||
|
assert isinstance(ch.backends, list)
|
||||||
|
assert ch.tier in {0, 1, 2}
|
||||||
|
|
||||||
|
|
||||||
|
def test_channel_check_contract_with_minimal_runtime(monkeypatch, tmp_path):
|
||||||
|
# Keep contract tests deterministic by simulating "deps mostly absent".
|
||||||
|
monkeypatch.setattr("shutil.which", lambda _cmd: None)
|
||||||
|
config = Config(config_path=tmp_path / "config.yaml")
|
||||||
|
|
||||||
|
for ch in get_all_channels():
|
||||||
|
status, message = ch.check(config)
|
||||||
|
assert status in {"ok", "warn", "off", "error"}
|
||||||
|
assert isinstance(message, str) and message.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def test_youtube_warns_when_node_only_and_no_config(monkeypatch, tmp_path):
|
||||||
|
"""YouTube should warn when only Node.js is installed but no yt-dlp config exists."""
|
||||||
|
from agent_reach.channels.youtube import YouTubeChannel
|
||||||
|
|
||||||
|
def fake_which(cmd):
|
||||||
|
if cmd == "yt-dlp":
|
||||||
|
return "/usr/bin/yt-dlp"
|
||||||
|
if cmd == "node":
|
||||||
|
return "/usr/bin/node"
|
||||||
|
return None # deno not installed
|
||||||
|
|
||||||
|
monkeypatch.setattr("shutil.which", fake_which)
|
||||||
|
# Point to a non-existent config file
|
||||||
|
monkeypatch.setattr("os.path.expanduser", lambda p: str(tmp_path / ".config/yt-dlp/config"))
|
||||||
|
|
||||||
|
ch = YouTubeChannel()
|
||||||
|
status, message = ch.check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "--js-runtimes" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_youtube_warns_with_windows_specific_fix_command(monkeypatch, tmp_path):
|
||||||
|
"""Windows guidance should use a PowerShell-style yt-dlp config command."""
|
||||||
|
from agent_reach.channels.youtube import YouTubeChannel
|
||||||
|
|
||||||
|
def fake_which(cmd):
|
||||||
|
if cmd == "yt-dlp":
|
||||||
|
return "C:/yt-dlp.exe"
|
||||||
|
if cmd == "node":
|
||||||
|
return "C:/node.exe"
|
||||||
|
return None
|
||||||
|
|
||||||
|
monkeypatch.setattr("shutil.which", fake_which)
|
||||||
|
monkeypatch.setattr("agent_reach.utils.paths.sys.platform", "win32")
|
||||||
|
monkeypatch.setenv("APPDATA", str(tmp_path / "AppData" / "Roaming"))
|
||||||
|
|
||||||
|
ch = YouTubeChannel()
|
||||||
|
status, message = ch.check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "Select-String" in message
|
||||||
|
assert "--js-runtimes node" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_youtube_ok_when_deno_installed(monkeypatch):
|
||||||
|
"""YouTube should return ok when Deno is installed (no config needed)."""
|
||||||
|
from agent_reach.channels.youtube import YouTubeChannel
|
||||||
|
|
||||||
|
def fake_which(cmd):
|
||||||
|
if cmd == "yt-dlp":
|
||||||
|
return "/usr/bin/yt-dlp"
|
||||||
|
if cmd == "deno":
|
||||||
|
return "/usr/bin/deno"
|
||||||
|
return None
|
||||||
|
|
||||||
|
monkeypatch.setattr("shutil.which", fake_which)
|
||||||
|
|
||||||
|
ch = YouTubeChannel()
|
||||||
|
status, _msg = ch.check()
|
||||||
|
assert status == "ok"
|
||||||
|
|
||||||
|
|
||||||
|
def test_douyin_check_does_not_call_with_invalid_url(monkeypatch, tmp_path):
|
||||||
|
"""Douyin check should use 'mcporter list' instead of calling with a hardcoded URL."""
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
from agent_reach.channels.douyin import DouyinChannel
|
||||||
|
|
||||||
|
calls = []
|
||||||
|
original_run = subprocess.run
|
||||||
|
|
||||||
|
def tracking_run(cmd, **kwargs):
|
||||||
|
calls.append(cmd)
|
||||||
|
# Simulate mcporter config list returning douyin
|
||||||
|
if "config" in cmd and "list" in cmd:
|
||||||
|
|
||||||
|
class R:
|
||||||
|
stdout = "douyin http://localhost:18070/mcp"
|
||||||
|
returncode = 0
|
||||||
|
|
||||||
|
return R()
|
||||||
|
# Simulate mcporter list douyin returning tools
|
||||||
|
if "list" in cmd and "douyin" in cmd:
|
||||||
|
|
||||||
|
class R:
|
||||||
|
stdout = "parse_douyin_video_info"
|
||||||
|
returncode = 0
|
||||||
|
|
||||||
|
return R()
|
||||||
|
return original_run(cmd, **kwargs)
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"shutil.which", lambda cmd: "/usr/bin/mcporter" if cmd == "mcporter" else None
|
||||||
|
)
|
||||||
|
monkeypatch.setattr("subprocess.run", tracking_run)
|
||||||
|
|
||||||
|
ch = DouyinChannel()
|
||||||
|
status, _msg = ch.check()
|
||||||
|
|
||||||
|
# Should NOT contain any hardcoded douyin.com URL in subprocess calls
|
||||||
|
for call in calls:
|
||||||
|
call_str = " ".join(call) if isinstance(call, list) else str(call)
|
||||||
|
assert "https://www.douyin.com" not in call_str
|
||||||
|
|
||||||
|
|
||||||
|
def test_channel_can_handle_contract():
|
||||||
|
url_samples = {
|
||||||
|
"github": "https://github.com/panniantong/agent-reach",
|
||||||
|
"twitter": "https://x.com/user/status/1",
|
||||||
|
"youtube": "https://youtube.com/watch?v=abc",
|
||||||
|
"reddit": "https://reddit.com/r/python",
|
||||||
|
"bilibili": "https://www.bilibili.com/video/BV1xx411",
|
||||||
|
"xiaohongshu": "https://www.xiaohongshu.com/explore/123",
|
||||||
|
"douyin": "https://www.douyin.com/video/123",
|
||||||
|
"linkedin": "https://www.linkedin.com/in/test",
|
||||||
|
"weibo": "https://weibo.com/u/1749127163",
|
||||||
|
"rss": "https://example.com/feed.xml",
|
||||||
|
"xueqiu": "https://xueqiu.com/S/SH600519",
|
||||||
|
"exa_search": "https://example.com",
|
||||||
|
"web": "https://example.com",
|
||||||
|
}
|
||||||
|
for ch in get_all_channels():
|
||||||
|
sample = url_samples.get(ch.name, "https://example.com")
|
||||||
|
result = ch.can_handle(sample)
|
||||||
|
assert isinstance(result, bool)
|
||||||
+714
-84
@@ -1,114 +1,744 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Tests for the channel system."""
|
"""Tests for channel registry basics and health checks."""
|
||||||
|
|
||||||
import pytest
|
import json
|
||||||
from unittest.mock import patch, MagicMock
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
from urllib.error import URLError
|
||||||
|
|
||||||
from agent_reach.channels import get_channel_for_url, get_channel, get_all_channels
|
from agent_reach.channels import get_all_channels, get_channel
|
||||||
from agent_reach.channels.base import ReadResult, SearchResult
|
from agent_reach.channels.v2ex import V2EXChannel
|
||||||
|
from agent_reach.channels.xiaohongshu import XiaoHongShuChannel
|
||||||
|
from agent_reach.channels.xueqiu import XueqiuChannel
|
||||||
|
|
||||||
|
|
||||||
class TestChannelRouting:
|
class TestChannelRegistry:
|
||||||
def test_github_url(self):
|
|
||||||
ch = get_channel_for_url("https://github.com/openai/gpt-4")
|
|
||||||
assert ch.name == "github"
|
|
||||||
|
|
||||||
def test_twitter_url(self):
|
|
||||||
ch = get_channel_for_url("https://x.com/elonmusk/status/123")
|
|
||||||
assert ch.name == "twitter"
|
|
||||||
|
|
||||||
def test_youtube_url(self):
|
|
||||||
ch = get_channel_for_url("https://youtube.com/watch?v=abc")
|
|
||||||
assert ch.name == "youtube"
|
|
||||||
|
|
||||||
def test_reddit_url(self):
|
|
||||||
ch = get_channel_for_url("https://reddit.com/r/test")
|
|
||||||
assert ch.name == "reddit"
|
|
||||||
|
|
||||||
def test_bilibili_url(self):
|
|
||||||
ch = get_channel_for_url("https://bilibili.com/video/BV1xx")
|
|
||||||
assert ch.name == "bilibili"
|
|
||||||
|
|
||||||
def test_rss_url(self):
|
|
||||||
ch = get_channel_for_url("https://example.com/feed.xml")
|
|
||||||
assert ch.name == "rss"
|
|
||||||
|
|
||||||
def test_generic_url_fallback(self):
|
|
||||||
ch = get_channel_for_url("https://example.com")
|
|
||||||
assert ch.name == "web"
|
|
||||||
|
|
||||||
def test_get_channel_by_name(self):
|
def test_get_channel_by_name(self):
|
||||||
ch = get_channel("github")
|
ch = get_channel("github")
|
||||||
assert ch is not None
|
assert ch is not None
|
||||||
assert ch.name == "github"
|
assert ch.name == "github"
|
||||||
|
|
||||||
|
def test_get_unknown_channel_returns_none(self):
|
||||||
|
assert get_channel("not-exists") is None
|
||||||
|
|
||||||
def test_all_channels_registered(self):
|
def test_all_channels_registered(self):
|
||||||
channels = get_all_channels()
|
channels = get_all_channels()
|
||||||
names = [ch.name for ch in channels]
|
names = [ch.name for ch in channels]
|
||||||
assert "web" in names
|
assert "web" in names
|
||||||
assert "github" in names
|
assert "github" in names
|
||||||
assert "twitter" in names
|
assert "twitter" in names
|
||||||
|
assert "v2ex" in names
|
||||||
|
|
||||||
|
|
||||||
class TestReadResult:
|
class TestV2EXChannel:
|
||||||
def test_to_dict(self):
|
def test_can_handle_v2ex_urls(self):
|
||||||
r = ReadResult(title="Test", content="Body", url="https://example.com", platform="web")
|
ch = V2EXChannel()
|
||||||
d = r.to_dict()
|
assert ch.can_handle("https://www.v2ex.com/t/1234567")
|
||||||
assert d["title"] == "Test"
|
assert ch.can_handle("https://v2ex.com/go/python")
|
||||||
assert d["content"] == "Body"
|
assert not ch.can_handle("https://github.com/user/repo")
|
||||||
assert d["platform"] == "web"
|
assert not ch.can_handle("https://reddit.com/r/Python")
|
||||||
|
|
||||||
def test_to_dict_optional_fields(self):
|
def test_check_ok_when_api_reachable(self, monkeypatch):
|
||||||
r = ReadResult(title="T", content="C", url="u", author="A", date="2025-01-01")
|
import urllib.request
|
||||||
d = r.to_dict()
|
|
||||||
assert d["author"] == "A"
|
|
||||||
assert d["date"] == "2025-01-01"
|
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
status = 200
|
||||||
|
|
||||||
class TestSearchResult:
|
def __enter__(self):
|
||||||
def test_to_dict(self):
|
return self
|
||||||
r = SearchResult(title="Test", url="https://example.com", snippet="A snippet")
|
|
||||||
d = r.to_dict()
|
|
||||||
assert d["title"] == "Test"
|
|
||||||
assert d["snippet"] == "A snippet"
|
|
||||||
|
|
||||||
|
def __exit__(self, *args):
|
||||||
|
pass
|
||||||
|
|
||||||
class TestGitHubChannel:
|
def read(self):
|
||||||
@patch("agent_reach.channels.github.requests.get")
|
return b"[]"
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_search(self, mock_get):
|
monkeypatch.setattr(
|
||||||
mock_resp = MagicMock()
|
urllib.request,
|
||||||
mock_resp.json.return_value = {
|
"urlopen",
|
||||||
"items": [{"full_name": "test/repo", "html_url": "https://github.com/test/repo",
|
lambda req, timeout=None: FakeResponse(),
|
||||||
"description": "A test", "stargazers_count": 100, "forks_count": 10,
|
)
|
||||||
"language": "Python", "updated_at": "2025-01-01"}]
|
status, msg = V2EXChannel().check()
|
||||||
|
assert status == "ok"
|
||||||
|
assert "公开 API 可用" in msg
|
||||||
|
|
||||||
|
def test_check_warn_when_api_unreachable(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
def raise_error(req, timeout=None):
|
||||||
|
raise URLError("connection refused")
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", raise_error)
|
||||||
|
status, msg = V2EXChannel().check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "失败" in msg
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_hot_topics
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_hot_topics_returns_list(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
fake_data = [
|
||||||
|
{
|
||||||
|
"id": 111,
|
||||||
|
"title": "Python 3.13 发布了",
|
||||||
|
"url": "https://www.v2ex.com/t/111",
|
||||||
|
"replies": 42,
|
||||||
|
"content": "发布公告内容",
|
||||||
|
"created": 1700000000,
|
||||||
|
"node": {"name": "python", "title": "Python"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 222,
|
||||||
|
"title": "Rust 好学吗",
|
||||||
|
"url": "https://www.v2ex.com/t/222",
|
||||||
|
"replies": 10,
|
||||||
|
"content": "",
|
||||||
|
"created": 1700000001,
|
||||||
|
"node": {"name": "rust", "title": "Rust"},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
status = 200
|
||||||
|
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", lambda req, timeout=None: FakeResponse())
|
||||||
|
topics = V2EXChannel().get_hot_topics(limit=5)
|
||||||
|
assert len(topics) == 2
|
||||||
|
assert topics[0]["id"] == 111
|
||||||
|
assert topics[0]["title"] == "Python 3.13 发布了"
|
||||||
|
assert topics[0]["replies"] == 42
|
||||||
|
assert topics[0]["node_name"] == "python"
|
||||||
|
assert topics[0]["node_title"] == "Python"
|
||||||
|
assert topics[0]["created"] == 1700000000
|
||||||
|
|
||||||
|
def test_get_hot_topics_respects_limit(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
fake_data = [
|
||||||
|
{"id": i, "title": f"Topic {i}", "url": f"https://v2ex.com/t/{i}", "replies": i,
|
||||||
|
"content": "", "created": 1700000000 + i, "node": {"name": "tech", "title": "Tech"}}
|
||||||
|
for i in range(10)
|
||||||
|
]
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", lambda req, timeout=None: FakeResponse())
|
||||||
|
topics = V2EXChannel().get_hot_topics(limit=3)
|
||||||
|
assert len(topics) == 3
|
||||||
|
|
||||||
|
def test_get_hot_topics_truncates_content(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
long_content = "A" * 300
|
||||||
|
fake_data = [
|
||||||
|
{"id": 1, "title": "Long post", "url": "https://v2ex.com/t/1", "replies": 0,
|
||||||
|
"content": long_content, "created": 1700000000, "node": {"name": "tech", "title": "Tech"}}
|
||||||
|
]
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", lambda req, timeout=None: FakeResponse())
|
||||||
|
topics = V2EXChannel().get_hot_topics(limit=1)
|
||||||
|
assert len(topics[0]["content"]) == 200
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_node_topics
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_node_topics(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
fake_data = [
|
||||||
|
{
|
||||||
|
"id": 333,
|
||||||
|
"title": "Flask 部署问题",
|
||||||
|
"url": "https://www.v2ex.com/t/333",
|
||||||
|
"replies": 5,
|
||||||
|
"content": "求帮助",
|
||||||
|
"created": 1710000000,
|
||||||
|
"node": {"name": "python", "title": "Python"},
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", lambda req, timeout=None: FakeResponse())
|
||||||
|
topics = V2EXChannel().get_node_topics("python")
|
||||||
|
assert len(topics) == 1
|
||||||
|
assert topics[0]["id"] == 333
|
||||||
|
assert topics[0]["node_name"] == "python"
|
||||||
|
assert topics[0]["title"] == "Flask 部署问题"
|
||||||
|
assert topics[0]["created"] == 1710000000
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_topic
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_topic_returns_detail_and_replies(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
topic_data = [
|
||||||
|
{
|
||||||
|
"id": 999,
|
||||||
|
"title": "测试帖子",
|
||||||
|
"url": "https://www.v2ex.com/t/999",
|
||||||
|
"content": "帖子正文",
|
||||||
|
"replies": 2,
|
||||||
|
"node": {"name": "qna", "title": "问与答"},
|
||||||
|
"member": {"username": "alice"},
|
||||||
|
"created": 1700000000,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
replies_data = [
|
||||||
|
{
|
||||||
|
"member": {"username": "bob"},
|
||||||
|
"content": "第一条回复",
|
||||||
|
"created": 1700000100,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"member": {"username": "carol"},
|
||||||
|
"content": "第二条回复",
|
||||||
|
"created": 1700000200,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __init__(self, payload):
|
||||||
|
self._payload = payload
|
||||||
|
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return json.dumps(self._payload).encode()
|
||||||
|
|
||||||
|
def fake_urlopen(req, timeout=None):
|
||||||
|
url = req.full_url
|
||||||
|
if "replies" in url:
|
||||||
|
return FakeResponse(replies_data)
|
||||||
|
return FakeResponse(topic_data)
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
|
||||||
|
result = V2EXChannel().get_topic(999)
|
||||||
|
|
||||||
|
assert result["id"] == 999
|
||||||
|
assert result["title"] == "测试帖子"
|
||||||
|
assert result["author"] == "alice"
|
||||||
|
assert result["node_name"] == "qna"
|
||||||
|
assert len(result["replies"]) == 2
|
||||||
|
assert result["replies"][0]["author"] == "bob"
|
||||||
|
assert result["replies"][1]["content"] == "第二条回复"
|
||||||
|
|
||||||
|
def test_get_topic_handles_empty_replies(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
topic_data = [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"title": "孤独帖子",
|
||||||
|
"url": "https://www.v2ex.com/t/1",
|
||||||
|
"content": "",
|
||||||
|
"replies": 0,
|
||||||
|
"node": {"name": "offtopic", "title": "水"},
|
||||||
|
"member": {"username": "dave"},
|
||||||
|
"created": 0,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __init__(self, payload): self._payload = payload
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return json.dumps(self._payload).encode()
|
||||||
|
|
||||||
|
def fake_urlopen(req, timeout=None):
|
||||||
|
if "replies" in req.full_url:
|
||||||
|
return FakeResponse([])
|
||||||
|
return FakeResponse(topic_data)
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
|
||||||
|
result = V2EXChannel().get_topic(1)
|
||||||
|
assert result["replies"] == []
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_user
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_user_returns_profile(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
fake_user = {
|
||||||
|
"id": 42,
|
||||||
|
"username": "alice",
|
||||||
|
"url": "https://www.v2ex.com/member/alice",
|
||||||
|
"website": "https://alice.dev",
|
||||||
|
"twitter": "alice_tw",
|
||||||
|
"psn": "",
|
||||||
|
"github": "alice",
|
||||||
|
"btc": "",
|
||||||
|
"location": "Shanghai",
|
||||||
|
"bio": "Python dev",
|
||||||
|
"avatar_large": "https://cdn.v2ex.com/avatars/alice_large.png",
|
||||||
|
"created": 1500000000,
|
||||||
}
|
}
|
||||||
mock_resp.raise_for_status = MagicMock()
|
|
||||||
mock_get.return_value = mock_resp
|
|
||||||
|
|
||||||
ch = get_channel("github")
|
class FakeResponse:
|
||||||
results = await ch.search("test query")
|
def __enter__(self): return self
|
||||||
assert len(results) == 1
|
def __exit__(self, *_): pass
|
||||||
assert results[0].title == "test/repo"
|
def read(self): return json.dumps(fake_user).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", lambda req, timeout=None: FakeResponse())
|
||||||
|
user = V2EXChannel().get_user("alice")
|
||||||
|
|
||||||
|
assert user["id"] == 42
|
||||||
|
assert user["username"] == "alice"
|
||||||
|
assert user["github"] == "alice"
|
||||||
|
assert user["location"] == "Shanghai"
|
||||||
|
assert "alice_large.png" in user["avatar"]
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# search
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_search_returns_unavailable_notice(self):
|
||||||
|
result = V2EXChannel().search("python asyncio")
|
||||||
|
assert len(result) == 1
|
||||||
|
assert "error" in result[0]
|
||||||
|
assert "V2EX" in result[0]["error"]
|
||||||
|
|
||||||
|
|
||||||
class TestExaSearch:
|
class TestXueqiuChannel:
|
||||||
@patch("agent_reach.channels.exa_search.requests.post")
|
def test_can_handle_xueqiu_urls(self):
|
||||||
@pytest.mark.asyncio
|
ch = XueqiuChannel()
|
||||||
async def test_search(self, mock_post):
|
assert ch.can_handle("https://xueqiu.com/S/SH600519")
|
||||||
from agent_reach.config import Config
|
assert ch.can_handle("https://stock.xueqiu.com/v5/stock/batch/quote.json")
|
||||||
config = Config(config_path="/tmp/test-exa-config.yaml")
|
assert ch.can_handle("https://www.xueqiu.com/1234567890/12345")
|
||||||
config.set("exa_api_key", "test-key")
|
assert not ch.can_handle("https://github.com/user/repo")
|
||||||
|
assert not ch.can_handle("https://v2ex.com/t/123")
|
||||||
|
|
||||||
mock_resp = MagicMock()
|
def test_check_ok_when_api_reachable(self, monkeypatch):
|
||||||
mock_resp.json.return_value = {
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
"results": [{"title": "Result", "url": "https://example.com",
|
|
||||||
"text": "snippet", "publishedDate": "", "score": 0.9}]
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
fake_response_data = {
|
||||||
|
"data": {
|
||||||
|
"items": [
|
||||||
|
{"quote": {"symbol": "SH000001", "name": "上证指数", "current": 3200.0}}
|
||||||
|
]
|
||||||
|
}
|
||||||
}
|
}
|
||||||
mock_resp.raise_for_status = MagicMock()
|
|
||||||
mock_post.return_value = mock_resp
|
|
||||||
|
|
||||||
ch = get_channel("exa_search")
|
class FakeResponse:
|
||||||
results = await ch.search("test", config=config)
|
def __enter__(self):
|
||||||
assert len(results) == 1
|
return self
|
||||||
assert results[0].title == "Result"
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_response_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", lambda req, timeout=None: FakeResponse())
|
||||||
|
status, msg = XueqiuChannel().check()
|
||||||
|
assert status == "ok"
|
||||||
|
assert "公开 API 可用" in msg
|
||||||
|
|
||||||
|
def test_check_warn_when_api_unreachable(self, monkeypatch):
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
def raise_error(req, timeout=None):
|
||||||
|
raise URLError("connection refused")
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", raise_error)
|
||||||
|
status, msg = XueqiuChannel().check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "失败" in msg
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_stock_quote
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_stock_quote(self, monkeypatch):
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
fake_data = {
|
||||||
|
"data": {
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"quote": {
|
||||||
|
"symbol": "SH600519",
|
||||||
|
"name": "贵州茅台",
|
||||||
|
"current": 1800.0,
|
||||||
|
"percent": 1.5,
|
||||||
|
"chg": 26.6,
|
||||||
|
"high": 1810.0,
|
||||||
|
"low": 1770.0,
|
||||||
|
"open": 1775.0,
|
||||||
|
"last_close": 1773.4,
|
||||||
|
"volume": 12345678,
|
||||||
|
"amount": 22000000000,
|
||||||
|
"market_capital": 2260000000000,
|
||||||
|
"turnover_rate": 0.098,
|
||||||
|
"pe_ttm": 30.5,
|
||||||
|
"timestamp": 1700000000000,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", lambda req, timeout=None: FakeResponse())
|
||||||
|
quote = XueqiuChannel().get_stock_quote("SH600519")
|
||||||
|
assert quote["symbol"] == "SH600519"
|
||||||
|
assert quote["name"] == "贵州茅台"
|
||||||
|
assert quote["current"] == 1800.0
|
||||||
|
assert quote["percent"] == 1.5
|
||||||
|
assert quote["volume"] == 12345678
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# search_stock
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_search_stock(self, monkeypatch):
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
fake_data = {
|
||||||
|
"stocks": [
|
||||||
|
{"code": "SH600519", "name": "贵州茅台", "exchange": "SHA"},
|
||||||
|
{"code": "SZ000858", "name": "五粮液", "exchange": "SZA"},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", lambda req, timeout=None: FakeResponse())
|
||||||
|
results = XueqiuChannel().search_stock("茅台", limit=5)
|
||||||
|
assert len(results) == 2
|
||||||
|
assert results[0]["symbol"] == "SH600519"
|
||||||
|
assert results[0]["name"] == "贵州茅台"
|
||||||
|
assert results[1]["exchange"] == "SZA"
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_hot_posts
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_hot_posts_returns_list(self, monkeypatch):
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
# v4 timeline: each item has a JSON-encoded `data` field
|
||||||
|
def make_item(id_, title, text, author, likes, target):
|
||||||
|
post = {
|
||||||
|
"id": id_,
|
||||||
|
"title": title,
|
||||||
|
"text": text,
|
||||||
|
"user": {"screen_name": author},
|
||||||
|
"like_count": likes,
|
||||||
|
"target": target,
|
||||||
|
}
|
||||||
|
return {"data": json.dumps(post), "original_status": None}
|
||||||
|
|
||||||
|
fake_data = {
|
||||||
|
"list": [
|
||||||
|
make_item(111, "市场分析", "<p>今天大盘走势&分析</p>", "投资者A", 42, "/1234567890/111"),
|
||||||
|
make_item(222, "", "短评", "投资者B", 10, "/9876543210/222"),
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", lambda req, timeout=None: FakeResponse())
|
||||||
|
posts = XueqiuChannel().get_hot_posts(limit=10)
|
||||||
|
assert len(posts) == 2
|
||||||
|
assert posts[0]["id"] == 111
|
||||||
|
assert posts[0]["author"] == "投资者A"
|
||||||
|
assert posts[0]["likes"] == 42
|
||||||
|
assert "今天大盘走势&分析" in posts[0]["text"] # HTML stripped
|
||||||
|
assert "<p>" not in posts[0]["text"]
|
||||||
|
assert posts[0]["url"] == "https://xueqiu.com/1234567890/111"
|
||||||
|
|
||||||
|
def test_get_hot_posts_respects_limit(self, monkeypatch):
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
fake_data = {
|
||||||
|
"list": [
|
||||||
|
{
|
||||||
|
"data": json.dumps({
|
||||||
|
"id": i,
|
||||||
|
"title": f"Post {i}",
|
||||||
|
"text": f"Content {i}",
|
||||||
|
"user": {"screen_name": f"User {i}"},
|
||||||
|
"like_count": i,
|
||||||
|
"target": f"/user/{i}",
|
||||||
|
}),
|
||||||
|
"original_status": None,
|
||||||
|
}
|
||||||
|
for i in range(10)
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", lambda req, timeout=None: FakeResponse())
|
||||||
|
posts = XueqiuChannel().get_hot_posts(limit=3)
|
||||||
|
assert len(posts) == 3
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_hot_stocks
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_hot_stocks(self, monkeypatch):
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
fake_data = {
|
||||||
|
"data": {
|
||||||
|
"items": [
|
||||||
|
{"code": "SH600519", "name": "贵州茅台", "current": 1800.0, "percent": 1.5},
|
||||||
|
{"code": "SZ000858", "name": "五粮液", "current": 160.0, "percent": -0.8},
|
||||||
|
{"code": "SH601318", "name": "中国平安", "current": 45.0, "percent": 0.3},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", lambda req, timeout=None: FakeResponse())
|
||||||
|
stocks = XueqiuChannel().get_hot_stocks(limit=10, stock_type=10)
|
||||||
|
assert len(stocks) == 3
|
||||||
|
assert stocks[0]["symbol"] == "SH600519"
|
||||||
|
assert stocks[0]["rank"] == 1
|
||||||
|
assert stocks[1]["percent"] == -0.8
|
||||||
|
assert stocks[2]["rank"] == 3
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Cookie loading
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_ensure_cookies_loads_from_config(self, monkeypatch, tmp_path):
|
||||||
|
"""_ensure_cookies() should inject cookies from the config file."""
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", False)
|
||||||
|
|
||||||
|
# Provide a fake Config that returns a cookie string with xq_a_token
|
||||||
|
class FakeConfig:
|
||||||
|
def get(self, key, default=None):
|
||||||
|
if key == "xueqiu_cookie":
|
||||||
|
return "xq_a_token=TESTTOKEN; xq_is_login=1"
|
||||||
|
return default
|
||||||
|
|
||||||
|
import agent_reach.channels.xueqiu as xq_mod
|
||||||
|
monkeypatch.setattr(
|
||||||
|
xq_mod,
|
||||||
|
"_load_cookies_from_config",
|
||||||
|
lambda: (xq_mod._inject_cookie_string("xq_a_token=TESTTOKEN; xq_is_login=1") or True),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(xq_mod, "_load_cookies_from_browser", lambda: False)
|
||||||
|
|
||||||
|
# Patch opener so no real HTTP call is made
|
||||||
|
class FakeResp:
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return b'{"data":{"items":[]}}'
|
||||||
|
|
||||||
|
monkeypatch.setattr(xq_mod._opener, "open", lambda req, timeout=None: FakeResp())
|
||||||
|
|
||||||
|
xq_mod._ensure_cookies()
|
||||||
|
assert xq_mod._cookies_initialized is True
|
||||||
|
cookie_names = {c.name for c in xq_mod._cookie_jar}
|
||||||
|
assert "xq_a_token" in cookie_names
|
||||||
|
|
||||||
|
def test_get_json_sends_referer_and_browser_ua(self, monkeypatch):
|
||||||
|
"""_get_json() must send Referer and a browser-like User-Agent."""
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
class FakeResp:
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return b'{"data":{"items":[]}}'
|
||||||
|
|
||||||
|
def fake_open(req, timeout=None):
|
||||||
|
captured["ua"] = req.get_header("User-agent")
|
||||||
|
captured["referer"] = req.get_header("Referer")
|
||||||
|
return FakeResp()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", fake_open)
|
||||||
|
xueqiu_mod._get_json("https://stock.xueqiu.com/v5/stock/batch/quote.json?symbol=SH000001")
|
||||||
|
|
||||||
|
assert captured["referer"] == "https://xueqiu.com/"
|
||||||
|
assert "Mozilla" in captured["ua"]
|
||||||
|
assert "agent-reach" not in captured["ua"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestRedditChannel:
|
||||||
|
def test_reports_off_when_not_installed(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(shutil, "which", lambda _: None)
|
||||||
|
from agent_reach.channels.reddit import RedditChannel
|
||||||
|
status, msg = RedditChannel().check()
|
||||||
|
assert status == "off"
|
||||||
|
assert "rdt-cli" in msg
|
||||||
|
assert "public-clis/rdt-cli" in msg
|
||||||
|
|
||||||
|
def test_reports_ok_when_authenticated(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(shutil, "which", lambda _: "/usr/local/bin/rdt")
|
||||||
|
fake_output = json.dumps({
|
||||||
|
"ok": True,
|
||||||
|
"schema_version": "1",
|
||||||
|
"data": {"authenticated": True, "username": "testuser", "cookie_count": 1},
|
||||||
|
})
|
||||||
|
|
||||||
|
def fake_run(cmd, **kwargs):
|
||||||
|
return subprocess.CompletedProcess(cmd, 0, fake_output, "")
|
||||||
|
|
||||||
|
monkeypatch.setattr(subprocess, "run", fake_run)
|
||||||
|
from agent_reach.channels.reddit import RedditChannel
|
||||||
|
status, msg = RedditChannel().check()
|
||||||
|
assert status == "ok"
|
||||||
|
assert "testuser" in msg
|
||||||
|
|
||||||
|
def test_reports_warn_when_not_authenticated(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(shutil, "which", lambda _: "/usr/local/bin/rdt")
|
||||||
|
fake_output = json.dumps({
|
||||||
|
"ok": True,
|
||||||
|
"schema_version": "1",
|
||||||
|
"data": {"authenticated": False, "username": None, "cookie_count": 0},
|
||||||
|
})
|
||||||
|
|
||||||
|
def fake_run(cmd, **kwargs):
|
||||||
|
return subprocess.CompletedProcess(cmd, 0, fake_output, "")
|
||||||
|
|
||||||
|
monkeypatch.setattr(subprocess, "run", fake_run)
|
||||||
|
from agent_reach.channels.reddit import RedditChannel
|
||||||
|
status, msg = RedditChannel().check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "403" in msg
|
||||||
|
assert "rdt login" in msg
|
||||||
|
assert "Cookie-Editor" in msg
|
||||||
|
assert "chromewebstore.google.com" in msg
|
||||||
|
|
||||||
|
def test_reports_warn_when_status_check_fails(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(shutil, "which", lambda _: "/usr/local/bin/rdt")
|
||||||
|
|
||||||
|
def fake_run(cmd, **kwargs):
|
||||||
|
return subprocess.CompletedProcess(cmd, 1, "not valid json{{{", "")
|
||||||
|
|
||||||
|
monkeypatch.setattr(subprocess, "run", fake_run)
|
||||||
|
from agent_reach.channels.reddit import RedditChannel
|
||||||
|
status, msg = RedditChannel().check()
|
||||||
|
assert status == "warn"
|
||||||
|
|
||||||
|
def test_can_handle_reddit_urls(self):
|
||||||
|
from agent_reach.channels.reddit import RedditChannel
|
||||||
|
ch = RedditChannel()
|
||||||
|
assert ch.can_handle("https://www.reddit.com/r/python/comments/abc123/")
|
||||||
|
assert ch.can_handle("https://redd.it/abc123")
|
||||||
|
assert not ch.can_handle("https://github.com/user/repo")
|
||||||
|
assert not ch.can_handle("https://v2ex.com/t/123")
|
||||||
|
|
||||||
|
|
||||||
|
class TestXiaoHongShuChannel:
|
||||||
|
def test_reports_ok_when_cli_authenticated(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(shutil, "which", lambda _: "/usr/local/bin/xhs")
|
||||||
|
|
||||||
|
def fake_run(cmd, **kwargs):
|
||||||
|
return subprocess.CompletedProcess(cmd, 0, "ok: true\nusername: testuser\n", "")
|
||||||
|
|
||||||
|
monkeypatch.setattr(subprocess, "run", fake_run)
|
||||||
|
|
||||||
|
status, msg = XiaoHongShuChannel().check()
|
||||||
|
assert status == "ok"
|
||||||
|
assert "完整可用" in msg
|
||||||
|
|
||||||
|
def test_reports_warn_when_not_authenticated(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(shutil, "which", lambda _: "/usr/local/bin/xhs")
|
||||||
|
|
||||||
|
def fake_run(cmd, **kwargs):
|
||||||
|
return subprocess.CompletedProcess(cmd, 1, "", "ok: false\nerror:\n code: not_authenticated\n")
|
||||||
|
|
||||||
|
monkeypatch.setattr(subprocess, "run", fake_run)
|
||||||
|
|
||||||
|
status, msg = XiaoHongShuChannel().check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "xhs login" in msg
|
||||||
|
|
||||||
|
def test_reports_off_when_not_installed(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(shutil, "which", lambda _: None)
|
||||||
|
status, msg = XiaoHongShuChannel().check()
|
||||||
|
assert status == "off"
|
||||||
|
assert "xiaohongshu-cli" in msg
|
||||||
|
|||||||
@@ -1,8 +1,13 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Tests for Agent Reach CLI."""
|
"""Tests for Agent Reach CLI."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
import requests
|
||||||
from unittest.mock import patch
|
from unittest.mock import patch
|
||||||
|
import agent_reach.cli as cli
|
||||||
from agent_reach.cli import main
|
from agent_reach.cli import main
|
||||||
|
|
||||||
|
|
||||||
@@ -27,3 +32,116 @@ class TestCLI:
|
|||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "Agent Reach" in captured.out
|
assert "Agent Reach" in captured.out
|
||||||
assert "✅" in captured.out
|
assert "✅" in captured.out
|
||||||
|
|
||||||
|
def test_parse_twitter_cookie_input_separate_values(self):
|
||||||
|
auth_token, ct0 = cli._parse_twitter_cookie_input("token123 ct0abc")
|
||||||
|
assert auth_token == "token123"
|
||||||
|
assert ct0 == "ct0abc"
|
||||||
|
|
||||||
|
def test_parse_twitter_cookie_input_cookie_header(self):
|
||||||
|
auth_token, ct0 = cli._parse_twitter_cookie_input(
|
||||||
|
"auth_token=token123; ct0=ct0abc; other=value"
|
||||||
|
)
|
||||||
|
assert auth_token == "token123"
|
||||||
|
assert ct0 == "ct0abc"
|
||||||
|
|
||||||
|
def test_configure_xhs_cookies_writes_xhs_cli_cookie_file(self, tmp_path, monkeypatch, capsys):
|
||||||
|
monkeypatch.setenv("HOME", str(tmp_path))
|
||||||
|
monkeypatch.setattr("shutil.which", lambda _name: None)
|
||||||
|
|
||||||
|
cli._configure_xhs_cookies("a1=token123; web_session=session456; other=value")
|
||||||
|
|
||||||
|
cookie_path = tmp_path / ".xiaohongshu-cli" / "cookies.json"
|
||||||
|
data = json.loads(cookie_path.read_text())
|
||||||
|
assert data["a1"] == "token123"
|
||||||
|
assert data["web_session"] == "session456"
|
||||||
|
assert "saved_at" in data
|
||||||
|
assert oct(os.stat(cookie_path).st_mode & 0o777) == "0o600"
|
||||||
|
|
||||||
|
legacy_path = tmp_path / ".agent-reach" / "xhs-cookies.json"
|
||||||
|
assert legacy_path.exists()
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert "xhs-cli cookies saved" in captured.out
|
||||||
|
|
||||||
|
|
||||||
|
class TestCheckUpdateRetry:
|
||||||
|
def test_retry_timeout_classification(self):
|
||||||
|
sleeps = []
|
||||||
|
|
||||||
|
def fake_sleep(seconds):
|
||||||
|
sleeps.append(seconds)
|
||||||
|
|
||||||
|
with patch("requests.get", side_effect=requests.exceptions.Timeout("timed out")):
|
||||||
|
resp, err, attempts = cli._github_get_with_retry(
|
||||||
|
"https://api.github.com/test",
|
||||||
|
timeout=1,
|
||||||
|
retries=3,
|
||||||
|
sleeper=fake_sleep,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert resp is None
|
||||||
|
assert err == "timeout"
|
||||||
|
assert attempts == 3
|
||||||
|
assert sleeps == [1, 2]
|
||||||
|
|
||||||
|
def test_retry_dns_classification(self):
|
||||||
|
error = requests.exceptions.ConnectionError("getaddrinfo failed for api.github.com")
|
||||||
|
with patch("requests.get", side_effect=error):
|
||||||
|
resp, err, attempts = cli._github_get_with_retry(
|
||||||
|
"https://api.github.com/test",
|
||||||
|
retries=1,
|
||||||
|
sleeper=lambda _x: None,
|
||||||
|
)
|
||||||
|
assert resp is None
|
||||||
|
assert err == "dns"
|
||||||
|
assert attempts == 1
|
||||||
|
|
||||||
|
def test_retry_rate_limit_then_success(self):
|
||||||
|
sleeps = []
|
||||||
|
|
||||||
|
class R:
|
||||||
|
def __init__(self, code, payload=None, headers=None):
|
||||||
|
self.status_code = code
|
||||||
|
self._payload = payload or {}
|
||||||
|
self.headers = headers or {}
|
||||||
|
|
||||||
|
def json(self):
|
||||||
|
return self._payload
|
||||||
|
|
||||||
|
sequence = [
|
||||||
|
R(429, headers={"Retry-After": "3"}),
|
||||||
|
R(200, payload={"tag_name": "v1.4.0"}),
|
||||||
|
]
|
||||||
|
|
||||||
|
with patch("requests.get", side_effect=sequence):
|
||||||
|
resp, err, attempts = cli._github_get_with_retry(
|
||||||
|
"https://api.github.com/test",
|
||||||
|
retries=3,
|
||||||
|
sleeper=lambda s: sleeps.append(s),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert err is None
|
||||||
|
assert resp is not None
|
||||||
|
assert resp.status_code == 200
|
||||||
|
assert attempts == 2
|
||||||
|
assert sleeps == [3.0]
|
||||||
|
|
||||||
|
def test_classify_rate_limit_from_403(self):
|
||||||
|
class R:
|
||||||
|
status_code = 403
|
||||||
|
headers = {"X-RateLimit-Remaining": "0"}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def json():
|
||||||
|
return {"message": "API rate limit exceeded"}
|
||||||
|
|
||||||
|
assert cli._classify_github_response_error(R()) == "rate_limit"
|
||||||
|
|
||||||
|
def test_check_update_reports_classified_error(self, capsys):
|
||||||
|
with patch("agent_reach.cli._github_get_with_retry", return_value=(None, "timeout", 3)):
|
||||||
|
result = cli._cmd_check_update()
|
||||||
|
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert result == "error"
|
||||||
|
assert "网络超时" in captured.out
|
||||||
|
assert "已重试 3 次" in captured.out
|
||||||
|
|||||||
+13
-5
@@ -61,11 +61,6 @@ class TestConfig:
|
|||||||
tmp_config.set("exa_api_key", "test-key")
|
tmp_config.set("exa_api_key", "test-key")
|
||||||
assert tmp_config.is_configured("exa_search")
|
assert tmp_config.is_configured("exa_search")
|
||||||
|
|
||||||
def test_is_configured_reddit(self, tmp_config):
|
|
||||||
assert not tmp_config.is_configured("reddit_proxy")
|
|
||||||
tmp_config.set("reddit_proxy", "http://user:pass@ip:port")
|
|
||||||
assert tmp_config.is_configured("reddit_proxy")
|
|
||||||
|
|
||||||
def test_get_configured_features(self, tmp_config):
|
def test_get_configured_features(self, tmp_config):
|
||||||
features = tmp_config.get_configured_features()
|
features = tmp_config.get_configured_features()
|
||||||
assert isinstance(features, dict)
|
assert isinstance(features, dict)
|
||||||
@@ -78,3 +73,16 @@ class TestConfig:
|
|||||||
masked = tmp_config.to_dict()
|
masked = tmp_config.to_dict()
|
||||||
assert masked["exa_api_key"] == "super-se..."
|
assert masked["exa_api_key"] == "super-se..."
|
||||||
assert masked["normal_setting"] == "visible"
|
assert masked["normal_setting"] == "visible"
|
||||||
|
|
||||||
|
def test_save_creates_file_with_restricted_permissions(self, tmp_path):
|
||||||
|
import stat
|
||||||
|
import sys
|
||||||
|
config_file = tmp_path / "secure_config.yaml"
|
||||||
|
config = Config(config_path=config_file)
|
||||||
|
config.set("secret_key", "my-secret")
|
||||||
|
|
||||||
|
if sys.platform != "win32":
|
||||||
|
mode = config_file.stat().st_mode
|
||||||
|
# File should be owner-only read/write (0o600)
|
||||||
|
assert not (mode & stat.S_IRGRP), "group read should not be set"
|
||||||
|
assert not (mode & stat.S_IROTH), "other read should not be set"
|
||||||
|
|||||||
+1
-8
@@ -2,6 +2,7 @@
|
|||||||
"""Tests for AgentReach core class."""
|
"""Tests for AgentReach core class."""
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from agent_reach.config import Config
|
from agent_reach.config import Config
|
||||||
from agent_reach.core import AgentReach
|
from agent_reach.core import AgentReach
|
||||||
|
|
||||||
@@ -16,14 +17,6 @@ class TestAgentReach:
|
|||||||
def test_init(self, eyes):
|
def test_init(self, eyes):
|
||||||
assert eyes.config is not None
|
assert eyes.config is not None
|
||||||
|
|
||||||
def test_detect_platform(self, eyes):
|
|
||||||
assert eyes.detect_platform("https://github.com/test/repo") == "github"
|
|
||||||
assert eyes.detect_platform("https://reddit.com/r/test") == "reddit"
|
|
||||||
assert eyes.detect_platform("https://x.com/user/status/123") == "twitter"
|
|
||||||
assert eyes.detect_platform("https://youtube.com/watch?v=abc") == "youtube"
|
|
||||||
assert eyes.detect_platform("https://bilibili.com/video/BV1xx") == "bilibili"
|
|
||||||
assert eyes.detect_platform("https://example.com") == "web"
|
|
||||||
|
|
||||||
def test_doctor(self, eyes):
|
def test_doctor(self, eyes):
|
||||||
results = eyes.doctor()
|
results = eyes.doctor()
|
||||||
assert isinstance(results, dict)
|
assert isinstance(results, dict)
|
||||||
|
|||||||
+84
-20
@@ -2,8 +2,22 @@
|
|||||||
"""Tests for doctor module."""
|
"""Tests for doctor module."""
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
|
import agent_reach.doctor as doctor
|
||||||
from agent_reach.config import Config
|
from agent_reach.config import Config
|
||||||
from agent_reach.doctor import check_all, format_report
|
|
||||||
|
|
||||||
|
class _StubChannel:
|
||||||
|
def __init__(self, name, description, tier, status, message, backends=None):
|
||||||
|
self.name = name
|
||||||
|
self.description = description
|
||||||
|
self.tier = tier
|
||||||
|
self._status = status
|
||||||
|
self._message = message
|
||||||
|
self.backends = backends or []
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
return self._status, self._message
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
@@ -12,25 +26,75 @@ def tmp_config(tmp_path):
|
|||||||
|
|
||||||
|
|
||||||
class TestDoctor:
|
class TestDoctor:
|
||||||
def test_zero_config_channels_ok(self, tmp_config):
|
def test_check_all_collects_channel_results(self, tmp_config, monkeypatch):
|
||||||
results = check_all(tmp_config)
|
monkeypatch.setattr(
|
||||||
assert results["web"]["status"] == "ok"
|
doctor,
|
||||||
assert results["github"]["status"] == "ok"
|
"get_all_channels",
|
||||||
assert results["bilibili"]["status"] in ("ok", "warn") # warn on servers
|
lambda: [
|
||||||
assert results["rss"]["status"] == "ok"
|
_StubChannel("web", "网页", 0, "ok", "可抓取网页", ["requests"]),
|
||||||
|
_StubChannel("github", "GitHub", 0, "warn", "gh 未安装", ["gh"]),
|
||||||
|
_StubChannel("exa_search", "全网语义搜索", 1, "off", "mcporter 未配置", ["Exa"]),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
def test_exa_off_without_key(self, tmp_config):
|
results = doctor.check_all(tmp_config)
|
||||||
results = check_all(tmp_config)
|
|
||||||
assert results["exa_search"]["status"] == "off"
|
|
||||||
|
|
||||||
def test_exa_on_with_key(self, tmp_config):
|
assert results == {
|
||||||
tmp_config.set("exa_api_key", "test-key")
|
"web": {
|
||||||
results = check_all(tmp_config)
|
"status": "ok",
|
||||||
assert results["exa_search"]["status"] == "ok"
|
"name": "网页",
|
||||||
|
"message": "可抓取网页",
|
||||||
|
"tier": 0,
|
||||||
|
"backends": ["requests"],
|
||||||
|
},
|
||||||
|
"github": {
|
||||||
|
"status": "warn",
|
||||||
|
"name": "GitHub",
|
||||||
|
"message": "gh 未安装",
|
||||||
|
"tier": 0,
|
||||||
|
"backends": ["gh"],
|
||||||
|
},
|
||||||
|
"exa_search": {
|
||||||
|
"status": "off",
|
||||||
|
"name": "全网语义搜索",
|
||||||
|
"message": "mcporter 未配置",
|
||||||
|
"tier": 1,
|
||||||
|
"backends": ["Exa"],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
def test_format_report(self, tmp_config):
|
def test_format_report(self):
|
||||||
results = check_all(tmp_config)
|
report = doctor.format_report(
|
||||||
report = format_report(results)
|
{
|
||||||
assert "Agent Reach" in report
|
"web": {
|
||||||
assert "✅" in report
|
"status": "ok",
|
||||||
assert "渠道可用" in report
|
"name": "网页",
|
||||||
|
"message": "可抓取网页",
|
||||||
|
"tier": 0,
|
||||||
|
"backends": ["requests"],
|
||||||
|
},
|
||||||
|
"exa_search": {
|
||||||
|
"status": "off",
|
||||||
|
"name": "全网语义搜索",
|
||||||
|
"message": "mcporter 未配置",
|
||||||
|
"tier": 1,
|
||||||
|
"backends": ["Exa"],
|
||||||
|
},
|
||||||
|
"xiaohongshu": {
|
||||||
|
"status": "warn",
|
||||||
|
"name": "小红书",
|
||||||
|
"message": "MCP 已配置,但健康检查超时",
|
||||||
|
"tier": 2,
|
||||||
|
"backends": ["mcporter"],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Strip Rich markup tags for assertion (PR #170 added [bold], [yellow] etc.)
|
||||||
|
import re
|
||||||
|
plain = re.sub(r"\[[^\]]*\]", "", report)
|
||||||
|
assert "Agent Reach" in plain
|
||||||
|
assert "装好即用:" in plain
|
||||||
|
assert "1/3 个渠道可用" in plain
|
||||||
|
# Inactive optional channels should be summarized in one line
|
||||||
|
assert "可选渠道可以解锁" in plain
|
||||||
|
|||||||
@@ -0,0 +1,128 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Tests for 'agent-reach skill' command and _install_skill / _uninstall_skill."""
|
||||||
|
|
||||||
|
import importlib.resources
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from agent_reach.cli import _install_skill, _uninstall_skill
|
||||||
|
|
||||||
|
|
||||||
|
class TestSkillCommand(unittest.TestCase):
|
||||||
|
"""Test skill install and uninstall via CLI helpers."""
|
||||||
|
|
||||||
|
def test_skill_resources_include_both_locales(self):
|
||||||
|
"""Package resources should expose both default and English skill markdown files."""
|
||||||
|
skill_dir = importlib.resources.files("agent_reach").joinpath("skill")
|
||||||
|
|
||||||
|
default_skill = skill_dir.joinpath("SKILL.md").read_text(encoding="utf-8")
|
||||||
|
english_skill = skill_dir.joinpath("SKILL_en.md").read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
self.assertTrue(default_skill.strip())
|
||||||
|
self.assertTrue(english_skill.strip())
|
||||||
|
|
||||||
|
def test_install_skill_creates_skill_md(self):
|
||||||
|
"""_install_skill should create SKILL.md in the first available skill dir."""
|
||||||
|
with tempfile.TemporaryDirectory() as tmpdir:
|
||||||
|
skill_dir = os.path.join(tmpdir, "skills")
|
||||||
|
os.makedirs(skill_dir)
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"agent_reach.cli.os.path.expanduser",
|
||||||
|
side_effect=lambda p: p.replace("~", tmpdir),
|
||||||
|
), patch.dict(os.environ, {}, clear=False):
|
||||||
|
# Remove OPENCLAW_HOME to avoid interference
|
||||||
|
env = os.environ.copy()
|
||||||
|
env.pop("OPENCLAW_HOME", None)
|
||||||
|
with patch.dict(os.environ, env, clear=True):
|
||||||
|
_install_skill()
|
||||||
|
|
||||||
|
target = os.path.join(skill_dir, "agent-reach", "SKILL.md")
|
||||||
|
# Check at least one known skill dir pattern
|
||||||
|
found = False
|
||||||
|
for dirpath, _, filenames in os.walk(tmpdir):
|
||||||
|
if "SKILL.md" in filenames:
|
||||||
|
found = True
|
||||||
|
# Verify content is non-empty
|
||||||
|
with open(os.path.join(dirpath, "SKILL.md")) as f:
|
||||||
|
content = f.read()
|
||||||
|
self.assertIn("Agent Reach", content)
|
||||||
|
# _install_skill may or may not find dirs depending on mock; just ensure no crash
|
||||||
|
# The important test is that the function runs without error
|
||||||
|
|
||||||
|
def test_uninstall_skill_removes_dir(self):
|
||||||
|
"""_uninstall_skill should remove skill directories."""
|
||||||
|
with tempfile.TemporaryDirectory() as tmpdir:
|
||||||
|
# Create a fake skill installation
|
||||||
|
skill_path = os.path.join(tmpdir, ".openclaw", "skills", "agent-reach")
|
||||||
|
os.makedirs(skill_path)
|
||||||
|
with open(os.path.join(skill_path, "SKILL.md"), "w") as f:
|
||||||
|
f.write("test")
|
||||||
|
|
||||||
|
self.assertTrue(os.path.exists(skill_path))
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"agent_reach.cli.os.path.expanduser",
|
||||||
|
side_effect=lambda p: p.replace("~", tmpdir),
|
||||||
|
), patch.dict(os.environ, {}, clear=False):
|
||||||
|
env = os.environ.copy()
|
||||||
|
env.pop("OPENCLAW_HOME", None)
|
||||||
|
with patch.dict(os.environ, env, clear=True):
|
||||||
|
_uninstall_skill()
|
||||||
|
|
||||||
|
self.assertFalse(os.path.exists(skill_path))
|
||||||
|
|
||||||
|
def test_install_creates_dir_if_parent_exists(self):
|
||||||
|
"""_install_skill should create agent-reach dir inside existing skill dir."""
|
||||||
|
with tempfile.TemporaryDirectory() as tmpdir:
|
||||||
|
# Create the .openclaw/skills parent but not agent-reach subdir
|
||||||
|
skill_parent = os.path.join(tmpdir, ".openclaw", "skills")
|
||||||
|
os.makedirs(skill_parent)
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"agent_reach.cli.os.path.expanduser",
|
||||||
|
side_effect=lambda p: p.replace("~", tmpdir),
|
||||||
|
), patch.dict(os.environ, {}, clear=False):
|
||||||
|
env = os.environ.copy()
|
||||||
|
env.pop("OPENCLAW_HOME", None)
|
||||||
|
with patch.dict(os.environ, env, clear=True):
|
||||||
|
_install_skill()
|
||||||
|
|
||||||
|
target = os.path.join(skill_parent, "agent-reach", "SKILL.md")
|
||||||
|
self.assertTrue(os.path.exists(target))
|
||||||
|
with open(target) as f:
|
||||||
|
content = f.read()
|
||||||
|
self.assertIn("Agent Reach", content)
|
||||||
|
|
||||||
|
def test_install_uses_english_skill_for_english_locale(self):
|
||||||
|
"""_install_skill should install the English skill file for English locales."""
|
||||||
|
with tempfile.TemporaryDirectory() as tmpdir:
|
||||||
|
skill_parent = os.path.join(tmpdir, ".openclaw", "skills")
|
||||||
|
os.makedirs(skill_parent)
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"agent_reach.cli.os.path.expanduser",
|
||||||
|
side_effect=lambda p: p.replace("~", tmpdir),
|
||||||
|
):
|
||||||
|
env = os.environ.copy()
|
||||||
|
env.pop("OPENCLAW_HOME", None)
|
||||||
|
env["LANG"] = "en_US.UTF-8"
|
||||||
|
with patch.dict(os.environ, env, clear=True):
|
||||||
|
_install_skill()
|
||||||
|
|
||||||
|
target = os.path.join(skill_parent, "agent-reach", "SKILL.md")
|
||||||
|
self.assertTrue(os.path.exists(target))
|
||||||
|
with open(target) as f:
|
||||||
|
content = f.read()
|
||||||
|
self.assertTrue(content.strip())
|
||||||
|
self.assertIn("Give your AI agent eyes to see the entire internet.", content)
|
||||||
|
self.assertNotIn("搜推特", content)
|
||||||
|
self.assertTrue(
|
||||||
|
os.path.exists(os.path.join(skill_parent, "agent-reach", "references"))
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
from unittest.mock import patch, Mock
|
||||||
|
|
||||||
|
from agent_reach.channels.twitter import TwitterChannel
|
||||||
|
|
||||||
|
|
||||||
|
def _cp(stdout="", stderr="", returncode=0):
|
||||||
|
m = Mock()
|
||||||
|
m.stdout = stdout
|
||||||
|
m.stderr = stderr
|
||||||
|
m.returncode = returncode
|
||||||
|
return m
|
||||||
|
|
||||||
|
|
||||||
|
# --- twitter-cli tests ---
|
||||||
|
|
||||||
|
def test_check_twitter_cli_found_and_auth_ok():
|
||||||
|
"""twitter-cli found + twitter status ok → ok."""
|
||||||
|
channel = TwitterChannel()
|
||||||
|
with patch("shutil.which", side_effect=lambda name: "/usr/local/bin/twitter" if name == "twitter" else None), patch(
|
||||||
|
"subprocess.run",
|
||||||
|
return_value=_cp(stdout="ok: true\nusername: testuser\n", returncode=0),
|
||||||
|
):
|
||||||
|
status, message = channel.check()
|
||||||
|
assert status == "ok"
|
||||||
|
assert "twitter-cli" in message
|
||||||
|
assert "完整可用" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_check_twitter_cli_found_auth_missing():
|
||||||
|
"""twitter-cli found + not_authenticated → warn about auth."""
|
||||||
|
channel = TwitterChannel()
|
||||||
|
with patch("shutil.which", side_effect=lambda name: "/usr/local/bin/twitter" if name == "twitter" else None), patch(
|
||||||
|
"subprocess.run",
|
||||||
|
return_value=_cp(
|
||||||
|
stderr="ok: false\nerror:\n code: not_authenticated\n",
|
||||||
|
returncode=1,
|
||||||
|
),
|
||||||
|
):
|
||||||
|
status, message = channel.check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "未认证" in message
|
||||||
|
|
||||||
|
|
||||||
|
# --- bird CLI fallback tests ---
|
||||||
|
|
||||||
|
def test_check_bird_fallback_auth_ok():
|
||||||
|
"""No twitter-cli, but bird found + bird check ok → ok."""
|
||||||
|
channel = TwitterChannel()
|
||||||
|
def which_side_effect(name):
|
||||||
|
if name == "bird":
|
||||||
|
return "/usr/local/bin/bird"
|
||||||
|
return None
|
||||||
|
with patch("shutil.which", side_effect=which_side_effect), patch(
|
||||||
|
"subprocess.run",
|
||||||
|
return_value=_cp(stdout="Authenticated as @user\n", returncode=0),
|
||||||
|
):
|
||||||
|
status, message = channel.check()
|
||||||
|
assert status == "ok"
|
||||||
|
assert "bird" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_check_bird_fallback_auth_missing():
|
||||||
|
"""No twitter-cli, bird found but Missing credentials → warn."""
|
||||||
|
channel = TwitterChannel()
|
||||||
|
def which_side_effect(name):
|
||||||
|
if name == "bird":
|
||||||
|
return "/usr/local/bin/bird"
|
||||||
|
return None
|
||||||
|
with patch("shutil.which", side_effect=which_side_effect), patch(
|
||||||
|
"subprocess.run",
|
||||||
|
return_value=_cp(stderr="Missing credentials\n", returncode=1),
|
||||||
|
):
|
||||||
|
status, message = channel.check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "未配置认证" in message
|
||||||
|
|
||||||
|
|
||||||
|
# --- neither installed ---
|
||||||
|
|
||||||
|
def test_check_nothing_installed():
|
||||||
|
"""Neither twitter-cli nor bird → warn with install hint."""
|
||||||
|
channel = TwitterChannel()
|
||||||
|
with patch("shutil.which", return_value=None):
|
||||||
|
status, message = channel.check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "twitter-cli" in message
|
||||||
|
|
||||||
|
|
||||||
|
# --- twitter-cli preferred over bird ---
|
||||||
|
|
||||||
|
def test_twitter_cli_preferred_over_bird():
|
||||||
|
"""When both are installed, twitter-cli is used."""
|
||||||
|
channel = TwitterChannel()
|
||||||
|
def which_side_effect(name):
|
||||||
|
if name == "twitter":
|
||||||
|
return "/usr/local/bin/twitter"
|
||||||
|
if name == "bird":
|
||||||
|
return "/usr/local/bin/bird"
|
||||||
|
return None
|
||||||
|
with patch("shutil.which", side_effect=which_side_effect), patch(
|
||||||
|
"subprocess.run",
|
||||||
|
return_value=_cp(stdout="ok: true\n", returncode=0),
|
||||||
|
):
|
||||||
|
status, message = channel.check()
|
||||||
|
assert status == "ok"
|
||||||
|
assert "twitter-cli" in message
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Tests for XiaoHongShu output formatter (issue #134)."""
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
from agent_reach.channels.xiaohongshu import format_xhs_result
|
||||||
|
|
||||||
|
|
||||||
|
class TestFormatXhsResult(unittest.TestCase):
|
||||||
|
"""Test format_xhs_result strips redundant fields."""
|
||||||
|
|
||||||
|
SAMPLE_NOTE = {
|
||||||
|
"id": "abc123",
|
||||||
|
"title": "测试笔记",
|
||||||
|
"desc": "这是正文内容",
|
||||||
|
"type": "normal",
|
||||||
|
"xsec_token": "tok_xxx",
|
||||||
|
"user": {
|
||||||
|
"nickname": "小红",
|
||||||
|
"user_id": "u123",
|
||||||
|
"avatar": "https://example.com/avatar.jpg",
|
||||||
|
"extra_field": "should be dropped",
|
||||||
|
},
|
||||||
|
"interact_info": {
|
||||||
|
"liked_count": "100",
|
||||||
|
"collected_count": "50",
|
||||||
|
"comment_count": "20",
|
||||||
|
"share_count": "10",
|
||||||
|
"sticky_count": "0",
|
||||||
|
"relation": "none",
|
||||||
|
},
|
||||||
|
"image_list": [
|
||||||
|
{
|
||||||
|
"url": "https://img.example.com/1.jpg",
|
||||||
|
"info_list": [{"url": "https://img.example.com/1_small.jpg", "image_scene": "WB_DFT"}],
|
||||||
|
"width": 1080,
|
||||||
|
"height": 1440,
|
||||||
|
"trace_id": "tr_123",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"url": "https://img.example.com/2.jpg",
|
||||||
|
"info_list": [{"url": "https://img.example.com/2_small.jpg"}],
|
||||||
|
"width": 1080,
|
||||||
|
"height": 1080,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"tag_list": [
|
||||||
|
{"id": "t1", "name": "旅行", "type": "topic"},
|
||||||
|
{"id": "t2", "name": "美食", "type": "topic"},
|
||||||
|
],
|
||||||
|
"at_user_list": [],
|
||||||
|
"geo_info": {"latitude": 0, "longitude": 0},
|
||||||
|
"audit_info": {"audit_status": 0},
|
||||||
|
"model_type": None,
|
||||||
|
"note_flow_source": "search",
|
||||||
|
}
|
||||||
|
|
||||||
|
def test_single_note_keeps_useful_fields(self):
|
||||||
|
result = format_xhs_result(self.SAMPLE_NOTE)
|
||||||
|
self.assertEqual(result["id"], "abc123")
|
||||||
|
self.assertEqual(result["title"], "测试笔记")
|
||||||
|
self.assertEqual(result["desc"], "这是正文内容")
|
||||||
|
self.assertEqual(result["type"], "normal")
|
||||||
|
self.assertEqual(result["user"]["nickname"], "小红")
|
||||||
|
self.assertEqual(result["liked_count"], "100")
|
||||||
|
self.assertEqual(result["collected_count"], "50")
|
||||||
|
self.assertEqual(result["images"], [
|
||||||
|
"https://img.example.com/1.jpg",
|
||||||
|
"https://img.example.com/2.jpg",
|
||||||
|
])
|
||||||
|
self.assertEqual(result["tags"], ["旅行", "美食"])
|
||||||
|
|
||||||
|
def test_single_note_drops_useless_fields(self):
|
||||||
|
result = format_xhs_result(self.SAMPLE_NOTE)
|
||||||
|
self.assertNotIn("at_user_list", result)
|
||||||
|
self.assertNotIn("geo_info", result)
|
||||||
|
self.assertNotIn("audit_info", result)
|
||||||
|
self.assertNotIn("model_type", result)
|
||||||
|
self.assertNotIn("note_flow_source", result)
|
||||||
|
# User should not have extra fields
|
||||||
|
self.assertNotIn("avatar", result.get("user", {}))
|
||||||
|
self.assertNotIn("extra_field", result.get("user", {}))
|
||||||
|
|
||||||
|
def test_search_results_wrapper(self):
|
||||||
|
"""Handle {"items": [...]} wrapper from search_feeds."""
|
||||||
|
wrapped = {"items": [self.SAMPLE_NOTE, self.SAMPLE_NOTE]}
|
||||||
|
result = format_xhs_result(wrapped)
|
||||||
|
self.assertIsInstance(result, list)
|
||||||
|
self.assertEqual(len(result), 2)
|
||||||
|
self.assertEqual(result[0]["title"], "测试笔记")
|
||||||
|
|
||||||
|
def test_list_input(self):
|
||||||
|
result = format_xhs_result([self.SAMPLE_NOTE])
|
||||||
|
self.assertIsInstance(result, list)
|
||||||
|
self.assertEqual(len(result), 1)
|
||||||
|
self.assertEqual(result[0]["title"], "测试笔记")
|
||||||
|
|
||||||
|
def test_note_card_wrapper(self):
|
||||||
|
"""Handle notes nested under 'note_card'."""
|
||||||
|
wrapped = {"note_card": self.SAMPLE_NOTE}
|
||||||
|
result = format_xhs_result(wrapped)
|
||||||
|
self.assertEqual(result["title"], "测试笔记")
|
||||||
|
|
||||||
|
def test_with_comments(self):
|
||||||
|
note = dict(self.SAMPLE_NOTE)
|
||||||
|
note["comments"] = [
|
||||||
|
{
|
||||||
|
"content": "写得好!",
|
||||||
|
"user_info": {"nickname": "路人甲", "user_id": "u456"},
|
||||||
|
"like_count": 5,
|
||||||
|
"sub_comment_count": 1,
|
||||||
|
"ip_location": "上海",
|
||||||
|
"status": 0,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
result = format_xhs_result(note)
|
||||||
|
self.assertEqual(len(result["comments"]), 1)
|
||||||
|
self.assertEqual(result["comments"][0]["content"], "写得好!")
|
||||||
|
self.assertEqual(result["comments"][0]["user"], "路人甲")
|
||||||
|
self.assertEqual(result["comments"][0]["like_count"], 5)
|
||||||
|
self.assertNotIn("ip_location", result["comments"][0])
|
||||||
|
|
||||||
|
def test_empty_input(self):
|
||||||
|
self.assertEqual(format_xhs_result({}), {})
|
||||||
|
self.assertEqual(format_xhs_result([]), [])
|
||||||
|
|
||||||
|
def test_non_dict_passthrough(self):
|
||||||
|
self.assertEqual(format_xhs_result("hello"), "hello")
|
||||||
|
self.assertIsNone(format_xhs_result(None))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import agent_reach.cli as cli
|
||||||
|
|
||||||
|
|
||||||
|
class _DummyConfig:
|
||||||
|
def get(self, _key):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def test_install_xiaoyuzhou_deps_does_not_raise_when_no_groq_key(capsys):
|
||||||
|
with patch("agent_reach.config.Config", return_value=_DummyConfig()), \
|
||||||
|
patch("os.path.isfile", side_effect=lambda p: True if str(p).endswith("transcribe.sh") else False), \
|
||||||
|
patch("shutil.which", return_value=None):
|
||||||
|
cli._install_xiaoyuzhou_deps()
|
||||||
|
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "Xiaoyuzhou" in out
|
||||||
|
assert "Groq API key not set" in out
|
||||||
Reference in New Issue
Block a user