Compare commits
151 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 23d4d5c611 | |||
| 75bc98a7eb | |||
| bc2d554e13 | |||
| 15f161e5b5 | |||
| 794455cc9f | |||
| 6548a50824 | |||
| 21214fd02a | |||
| c5a304dc20 | |||
| 66883060bc | |||
| f4b3758478 | |||
| d7c8ac81e9 | |||
| 1e0e6d1ae7 | |||
| e90fbfe194 | |||
| 3e90d51da9 | |||
| b82ab1ae10 | |||
| 9a8b245e39 | |||
| 5d046ecb61 | |||
| b4ed28e48f | |||
| f9b7241d47 | |||
| 526616e13f | |||
| e51bd91e82 | |||
| e575ddbcd1 | |||
| d793afaba6 | |||
| ca2e85520b | |||
| 3be2a64120 | |||
| 470c1288d0 | |||
| 881d9b96e2 | |||
| 15563051a1 | |||
| 0c6c45a010 | |||
| 7e66abe636 | |||
| e6406500f3 | |||
| afc8d0e3ee | |||
| 90bb4e0266 | |||
| 7ae0cd8c0a | |||
| c1af1cad14 | |||
| ded444000b | |||
| 5fb0e10167 | |||
| ef8134bba4 | |||
| be9f9ab838 | |||
| 8afd19f473 | |||
| 1f4154a897 | |||
| 1442852471 | |||
| 5461ca4030 | |||
| ab843071ef | |||
| 6c5346cfa4 | |||
| 77c62c76fb | |||
| 7c64579318 | |||
| 383004e06a | |||
| f52fb67232 | |||
| 59b594eccc | |||
| 3416c0f6d3 | |||
| 31f00b8d78 | |||
| ba565bd096 | |||
| 7018d77d58 | |||
| ce9b365491 | |||
| 70a0bb8af8 | |||
| 6737ba6dbd | |||
| f5f4dea075 | |||
| 233cc6fba8 | |||
| ae5812cf7f | |||
| 6af00ce4da | |||
| 3321bf7d34 | |||
| 00d80d2169 | |||
| 55efebd395 | |||
| 16733df4fc | |||
| f3daa5cf97 | |||
| 6ddcfe2b57 | |||
| 0d91d6834a | |||
| dc64f15839 | |||
| c912051173 | |||
| 22d7ad766a | |||
| 5895f3992d | |||
| 9129fc5f27 | |||
| 3fe100cd9a | |||
| c2cbccfed9 | |||
| 0a5c06633c | |||
| 7942f632e5 | |||
| fedbf95f61 | |||
| 1d3fd43fb3 | |||
| c02ffc59a5 | |||
| db20c5d5d3 | |||
| 3a3d38acce | |||
| 4b7d55111f | |||
| eda80b89b5 | |||
| 75a2213510 | |||
| 8a90e52fde | |||
| 9d0191d5eb | |||
| f3d11fe89d | |||
| df34436942 | |||
| cd94706d3b | |||
| 35073aed54 | |||
| 94b428b0b9 | |||
| 105b477a31 | |||
| 92963f0496 | |||
| e5e20a1154 | |||
| b4d189b536 | |||
| 27d3d545e6 | |||
| ca29c4fee5 | |||
| 49d05b7023 | |||
| 476935be92 | |||
| 4f4ad99d49 | |||
| 561b248a2b | |||
| a91a74d880 | |||
| 7434c3cb9f | |||
| 1a61a2cc81 | |||
| 932837ff9a | |||
| 4f52942260 | |||
| 3c1a04a587 | |||
| 6a28b215af | |||
| a881db4261 | |||
| 875f0987a6 | |||
| 0d1a33236e | |||
| e620a1db8f | |||
| 15a2961dfa | |||
| 4bc860c74d | |||
| 62aacf38b5 | |||
| d4e437bf2c | |||
| dbcc668f47 | |||
| 40f833d0d1 | |||
| dc4d7cd7a5 | |||
| 60f5ddf4f0 | |||
| a726aa7fe1 | |||
| 4b2e6f2ffb | |||
| 39f76ee4dc | |||
| a5682716ec | |||
| f269667a34 | |||
| 090b01c909 | |||
| 58b3a4dcb3 | |||
| 37b4cdeddc | |||
| bf6dc7084b | |||
| 50f2181fcc | |||
| b387a67b51 | |||
| a96c5cda14 | |||
| 9c6c04c800 | |||
| cf70302e92 | |||
| 80a08c8c10 | |||
| 606e8900ce | |||
| a37e9aa190 | |||
| 1cbf6a7b9c | |||
| 68319ec683 | |||
| f70711e75e | |||
| c3a9813b1c | |||
| d291d27664 | |||
| 0be4f097d0 | |||
| 12bd727e30 | |||
| b6d135c977 | |||
| 291d073223 | |||
| ef0889dd69 | |||
| 545a6c5761 | |||
| 00f3edd9cc | |||
| 2f674f1e45 |
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"permissions": {
|
||||||
|
"allow": [
|
||||||
|
"WebFetch(domain:community.groq.com)"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
name: ci
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
pull_request:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
test:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
strategy:
|
||||||
|
fail-fast: false
|
||||||
|
matrix:
|
||||||
|
python-version: ["3.10", "3.11", "3.12", "3.13"]
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Setup Python
|
||||||
|
uses: actions/setup-python@v5
|
||||||
|
with:
|
||||||
|
python-version: ${{ matrix.python-version }}
|
||||||
|
|
||||||
|
- name: Install package and test deps
|
||||||
|
run: |
|
||||||
|
python -m pip install --upgrade pip
|
||||||
|
pip install -c constraints.txt -e .[dev]
|
||||||
|
|
||||||
|
- name: Run tests
|
||||||
|
run: |
|
||||||
|
pytest -q
|
||||||
+44
-12
@@ -6,17 +6,49 @@ All notable changes to this project will be documented in this file.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## [1.3.1] - 2026-03-27
|
||||||
|
|
||||||
|
### 🐛 Bug Fixes / 修复
|
||||||
|
|
||||||
|
#### 📈 Xueqiu (雪球) — 全面修复
|
||||||
|
|
||||||
|
- **修复 400 错误根本原因:** `_ensure_cookies()` 仅访问首页只能获取 `acw_tc`(防 DDoS token),`xq_a_token` 由雪球前端 JS 动态生成,无法通过纯 HTTP 请求获取。新增三级 cookie 加载策略:① 读取 config 文件(`--from-browser` 保存的)→ ② 自动从本地 Chrome 浏览器提取(需安装 browser-cookie3)→ ③ homepage fallback
|
||||||
|
- **修复 User-Agent:** `"agent-reach/1.0"` 被雪球反爬系统识别拒绝,改为真实 Chrome UA
|
||||||
|
- **修复缺失 `Referer` 头:** 所有 API 请求加上 `Referer: https://xueqiu.com/`
|
||||||
|
- **修复 `get_hot_posts()` 端点:** 原端点 `/statuses/hot/listV3.json` 已废弃(返回空 body),改为 `/v4/statuses/public_timeline_by_category.json`,正确解析 `item.data` JSON 字符串获取 author/likes/text
|
||||||
|
- **修复 `urllib.request.quote` → `urllib.parse.quote`:** 明确使用正确模块
|
||||||
|
- **修复 `configure --from-browser` 不提取雪球 Cookie:** `PLATFORM_SPECS` 加入 Xueqiu,检测 `xq_a_token` 存在才保存
|
||||||
|
- **修正文档误导:** README/SKILL.md 中"无需配置"/"public API, no login required" → 准确描述需要 browser cookie
|
||||||
|
- **改善错误信息:** `check()` 失败时提示 `configure --from-browser chrome` 而非"可能需要代理"
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## [1.3.0] - 2026-03-12
|
||||||
|
|
||||||
|
### 🆕 New Channels / 新增渠道
|
||||||
|
|
||||||
|
#### 💻 V2EX
|
||||||
|
- Hot topics, node topics, topic detail + replies, user profile via public JSON API
|
||||||
|
- Zero config — no auth, no proxy, no API key required
|
||||||
|
- `get_hot_topics(limit)`, `get_node_topics(node_name, limit)`, `get_topic(id)`, `get_user(username)`
|
||||||
|
- 通过公开 JSON API 获取热门帖子、节点帖子、帖子详情+回复、用户信息
|
||||||
|
- 零配置,无需认证、无需代理、无需 API Key
|
||||||
|
|
||||||
|
### 📈 Improvements / 改进
|
||||||
|
|
||||||
|
- Channel count: 14 → 15
|
||||||
|
- 渠道数量:14 → 15
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## [1.1.0] - 2025-02-25
|
## [1.1.0] - 2025-02-25
|
||||||
|
|
||||||
### 🆕 New Channels / 新增渠道
|
### 🆕 New Channels / 新增渠道
|
||||||
|
|
||||||
#### 📷 Instagram
|
#### ~~📷 Instagram~~ (removed — upstream blocked)
|
||||||
- Read public posts and profiles via [instaloader](https://github.com/instaloader/instaloader)
|
- ~~Read public posts and profiles via [instaloader](https://github.com/instaloader/instaloader)~~
|
||||||
- Search via Exa (free, no API key)
|
- **Removed:** Instagram's aggressive anti-scraping measures broke all available open-source tools (instaloader, etc.). See [instaloader#2585](https://github.com/instaloader/instaloader/issues/2585). Will re-add when upstream recovers.
|
||||||
- Optional cookie login for private content
|
- **已移除:** Instagram 反爬封杀导致所有开源工具(instaloader 等)失效。上游恢复后会重新加回。
|
||||||
- 通过 instaloader 读取公开帖子和 Profile
|
|
||||||
- 搜索通过 Exa(免费,无需 API Key)
|
|
||||||
- 可选 Cookie 登录解锁私密内容
|
|
||||||
|
|
||||||
#### 💼 LinkedIn
|
#### 💼 LinkedIn
|
||||||
- Read person profiles, company pages, and job details via [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
- Read person profiles, company pages, and job details via [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||||
@@ -38,12 +70,12 @@ All notable changes to this project will be documented in this file.
|
|||||||
|
|
||||||
- Channel count: 9 → 12
|
- Channel count: 9 → 12
|
||||||
- `agent-reach doctor` now detects all 12 channels
|
- `agent-reach doctor` now detects all 12 channels
|
||||||
- CLI: added `search-instagram`, `search-linkedin`, `search-bosszhipin` subcommands
|
- CLI: added `search-linkedin`, `search-bosszhipin` subcommands
|
||||||
- Updated install guide with setup instructions for new channels
|
- Updated install guide with setup instructions for new channels
|
||||||
- 渠道数量:9 → 12
|
- 渠道数量:9 → 11
|
||||||
- `agent-reach doctor` 现在检测全部 12 个渠道
|
- `agent-reach doctor` 现在检测全部 11 个渠道
|
||||||
- CLI:新增 `search-instagram`、`search-linkedin`、`search-bosszhipin` 子命令
|
- CLI:新增 `search-linkedin`、`search-bosszhipin` 子命令
|
||||||
- 安装指南新增三个渠道的配置说明
|
- 安装指南新增渠道配置说明
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# CLAUDE.md
|
||||||
|
|
||||||
|
## Project
|
||||||
|
Agent Reach — Python CLI + library that gives AI agents read/search access to 14+ internet platforms.
|
||||||
|
Positioning: installer + doctor + config tool. NOT a wrapper — after install, agents call upstream tools directly.
|
||||||
|
Repo: github.com/Panniantong/Agent-Reach | License: MIT | Version: 1.3.0
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
- `pip install -e .` — Dev install
|
||||||
|
- `pytest tests/ -v` — All tests
|
||||||
|
- `pytest tests/test_cli.py -v` — CLI tests only
|
||||||
|
- `bash test.sh` — Full integration test (creates venv, installs, runs doctor + channel tests)
|
||||||
|
- `python -m agent_reach.cli doctor` — Run diagnostics
|
||||||
|
- `python -m agent_reach.cli install --env=auto` — Auto-configure
|
||||||
|
|
||||||
|
## Structure
|
||||||
|
- `agent_reach/cli.py` — CLI entry point (argparse)
|
||||||
|
- `agent_reach/core.py` — Core read/search routing logic
|
||||||
|
- `agent_reach/config.py` — Config management (YAML, env vars)
|
||||||
|
- `agent_reach/doctor.py` — Diagnostics engine
|
||||||
|
- `agent_reach/channels/` — One file per platform (twitter.py, reddit.py, youtube.py, etc.)
|
||||||
|
- `agent_reach/channels/base.py` — Base channel class (all channels inherit from this)
|
||||||
|
- `agent_reach/integrations/mcp_server.py` — MCP server integration
|
||||||
|
- `agent_reach/skill/` — OpenClaw skill files
|
||||||
|
- `agent_reach/guides/` — Usage guides
|
||||||
|
- `tests/` — pytest tests
|
||||||
|
- `config/mcporter.json` — MCP tool config
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
- Python 3.10+ with type hints
|
||||||
|
- Each channel is a single file in `channels/`, inherits from `BaseChannel`
|
||||||
|
- Channel contract: must implement `can_handle(url)`, `read(url)`, `search(query)`, `check()` methods
|
||||||
|
- Use `loguru` for logging, `rich` for CLI output
|
||||||
|
- Commit format: `type(scope): message` (one commit = one thing)
|
||||||
|
- All upstream tool calls go through public API/CLI, never hack internals
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
- NEVER modify upstream open source projects' source code
|
||||||
|
- Agent Reach is a "glue layer" — only route and call, don't reimagine
|
||||||
|
- Version in THREE places must match: `pyproject.toml`, `__init__.py`, `tests/test_cli.py`
|
||||||
|
- Always new branch for changes, PR to main, never push to main directly
|
||||||
|
- Run `pytest tests/ -v` before committing — all tests must pass
|
||||||
|
- Cookie-based auth (Twitter, XHS): use Cookie-Editor export method only, no QR scan
|
||||||
|
- XHS login: Cookie-Editor browser export only (QR will hang)
|
||||||
+107
@@ -0,0 +1,107 @@
|
|||||||
|
# Contributing to Agent Reach
|
||||||
|
|
||||||
|
Thank you for your interest in contributing to Agent Reach! This document provides guidelines and instructions for contributing.
|
||||||
|
|
||||||
|
## Getting Started
|
||||||
|
|
||||||
|
1. Fork the repository on GitHub
|
||||||
|
2. Clone your fork locally
|
||||||
|
3. Create a new branch for your contribution
|
||||||
|
4. Make your changes
|
||||||
|
5. Run tests and linting
|
||||||
|
6. Submit a pull request
|
||||||
|
|
||||||
|
## Development Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone your fork
|
||||||
|
git clone https://github.com/YOUR_USERNAME/Agent-Reach.git
|
||||||
|
cd Agent-Reach
|
||||||
|
|
||||||
|
# Install in development mode
|
||||||
|
pip install -e ".[dev]"
|
||||||
|
|
||||||
|
# Install pre-commit hooks (optional but recommended)
|
||||||
|
pre-commit install
|
||||||
|
```
|
||||||
|
|
||||||
|
## Code Style
|
||||||
|
|
||||||
|
We use the following tools to maintain code quality:
|
||||||
|
|
||||||
|
- **ruff**: Linting and import sorting
|
||||||
|
- **mypy**: Type checking
|
||||||
|
- **pytest**: Testing
|
||||||
|
|
||||||
|
Run all checks before submitting a PR:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Linting
|
||||||
|
ruff check agent_reach tests
|
||||||
|
ruff format agent_reach tests
|
||||||
|
|
||||||
|
# Type checking
|
||||||
|
mypy agent_reach
|
||||||
|
|
||||||
|
# Tests
|
||||||
|
pytest
|
||||||
|
```
|
||||||
|
|
||||||
|
## Adding New Channels
|
||||||
|
|
||||||
|
Agent Reach uses a unified channel interface. To add a new platform:
|
||||||
|
|
||||||
|
1. Create a new file in `agent_reach/channels/`
|
||||||
|
2. Implement the channel contract (see existing channels for examples)
|
||||||
|
3. Add tests in `tests/test_channels.py`
|
||||||
|
4. Update `agent_reach/doctor.py` to include the new channel
|
||||||
|
5. Update documentation
|
||||||
|
|
||||||
|
## Pull Request Guidelines
|
||||||
|
|
||||||
|
- **Small, focused changes** are preferred over large refactors
|
||||||
|
- Include tests for new functionality
|
||||||
|
- Update documentation if needed
|
||||||
|
- Follow existing code style
|
||||||
|
- Reference any related issues
|
||||||
|
|
||||||
|
## Reporting Issues
|
||||||
|
|
||||||
|
When reporting bugs, please include:
|
||||||
|
|
||||||
|
- Python version
|
||||||
|
- Operating system
|
||||||
|
- Steps to reproduce
|
||||||
|
- Expected vs actual behavior
|
||||||
|
- Any error messages
|
||||||
|
|
||||||
|
## Questions?
|
||||||
|
|
||||||
|
Feel free to open an issue for questions or join discussions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
感谢您对 Agent Reach 的贡献!本文档提供了贡献指南。
|
||||||
|
|
||||||
|
## 快速开始
|
||||||
|
|
||||||
|
1. 在 GitHub 上 fork 仓库
|
||||||
|
2. 本地 clone 您的 fork
|
||||||
|
3. 创建新分支
|
||||||
|
4. 提交更改
|
||||||
|
5. 运行测试和 lint
|
||||||
|
6. 提交 pull request
|
||||||
|
|
||||||
|
## 代码规范
|
||||||
|
|
||||||
|
- 使用 **ruff** 进行代码检查
|
||||||
|
- 使用 **mypy** 进行类型检查
|
||||||
|
- 使用 **pytest** 运行测试
|
||||||
|
|
||||||
|
## 添加新渠道
|
||||||
|
|
||||||
|
1. 在 `agent_reach/channels/` 创建新文件
|
||||||
|
2. 实现渠道接口
|
||||||
|
3. 添加测试
|
||||||
|
4. 更新 doctor 检测
|
||||||
|
5. 更新文档
|
||||||
@@ -11,7 +11,7 @@
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<a href="#30-秒上手">快速开始</a> · <a href="docs/README_en.md">English</a> · <a href="#支持的平台">支持平台</a> · <a href="#设计理念">设计理念</a>
|
<a href="#快速上手">快速开始</a> · <a href="docs/README_en.md">English</a> · <a href="docs/README_ja.md">日本語</a> · <a href="#支持的平台">支持平台</a> · <a href="#设计理念">设计理念</a>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -42,6 +42,12 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|
|
||||||
复制给你的 Agent,几分钟后它就能读推特、搜 Reddit、看 YouTube、刷小红书了。
|
复制给你的 Agent,几分钟后它就能读推特、搜 Reddit、看 YouTube、刷小红书了。
|
||||||
|
|
||||||
|
**已经装过了?更新也是一句话:**
|
||||||
|
|
||||||
|
```
|
||||||
|
帮我更新 Agent Reach:https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
```
|
||||||
|
|
||||||
> ⭐ **Star 这个项目**,我们会持续追踪各平台的变化、接入新的渠道。你不用自己盯——平台封了我们修,有新渠道我们加。
|
> ⭐ **Star 这个项目**,我们会持续追踪各平台的变化、接入新的渠道。你不用自己盯——平台封了我们修,有新渠道我们加。
|
||||||
|
|
||||||
### ✅ 在你用之前,你可能想知道
|
### ✅ 在你用之前,你可能想知道
|
||||||
@@ -67,15 +73,19 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
| 📦 **GitHub** | 读公开仓库 + 搜索 | 私有仓库、提 Issue/PR、Fork | 告诉 Agent「帮我登录 GitHub」 |
|
| 📦 **GitHub** | 读公开仓库 + 搜索 | 私有仓库、提 Issue/PR、Fork | 告诉 Agent「帮我登录 GitHub」 |
|
||||||
| 🐦 **Twitter/X** | 读单条推文 | 搜索推文、浏览时间线、发推 | 告诉 Agent「帮我配 Twitter」 |
|
| 🐦 **Twitter/X** | 读单条推文 | 搜索推文、浏览时间线、发推 | 告诉 Agent「帮我配 Twitter」 |
|
||||||
| 📺 **B站** | 本地:字幕提取 + 搜索 | 服务器也能用 | 告诉 Agent「帮我配代理」 |
|
| 📺 **B站** | 本地:字幕提取 + 搜索 | 服务器也能用 | 告诉 Agent「帮我配代理」 |
|
||||||
| 📖 **Reddit** | 搜索(通过 Exa 免费) | 读帖子和评论 | 告诉 Agent「帮我配代理」 |
|
| 📖 **Reddit** | 搜索 + 读帖子和评论(通过 Exa) | — | 无需配置(自动通过 Exa) |
|
||||||
| 📕 **小红书** | — | 阅读、搜索、发帖、评论、点赞 | `docker run -d -p 18060:18060 xpzouying/xiaohongshu-mcp` 然后告诉 Agent「帮我配置小红书」 |
|
| 📕 **小红书** | — | 阅读、搜索、发帖、评论、点赞 | 告诉 Agent「帮我配小红书」 |
|
||||||
| 📷 **Instagram** | — | 读取帖子和 Profile | Cookie-Editor 导出 Cookie → `agent-reach configure instagram-cookies "..."` |
|
| 🎵 **抖音** | — | 视频解析、无水印下载链接获取 | 告诉 Agent「帮我配抖音」 |
|
||||||
| 💼 **LinkedIn** | Jina Reader 读公开页面 | Profile 详情、公司、职位搜索 | `linkedin-scraper-mcp --login`(需浏览器,服务器用 VNC) |
|
| 💼 **LinkedIn** | Jina Reader 读公开页面 | Profile 详情、公司页面、职位搜索 | 告诉 Agent「帮我配 LinkedIn」 |
|
||||||
| 🏢 **Boss直聘** | Jina Reader 读职位页 | 搜索职位、向 HR 打招呼 | 启动 mcp-bosszp → Boss直聘 App 扫码 |
|
| 💬 **微信公众号** | 搜索 + 阅读公众号文章(全文 Markdown) | — | 无需配置 |
|
||||||
|
| 📰 **微博** | 热搜、搜索内容/用户/话题、用户动态、评论 | — | 无需配置 |
|
||||||
|
| 💻 **V2EX** | 热门帖子、节点帖子、帖子详情+回复、用户信息 | — | 无需配置 |
|
||||||
|
| 📈 **雪球** | 股票行情、搜索股票、热门帖子、热门股票排行 | — | 告诉 Agent「帮我配雪球」 |
|
||||||
|
| 🎙️ **小宇宙播客** | — | 播客音频转文字(Whisper 转录,免费 Key) | 告诉 Agent「帮我配小宇宙播客」 |
|
||||||
|
|
||||||
> **不知道怎么配?不用查文档。** 直接告诉 Agent「帮我配 XXX」,它知道需要什么、会一步一步引导你。
|
> **不知道怎么配?不用查文档。** 直接告诉 Agent「帮我配 XXX」,它知道需要什么、会一步一步引导你。
|
||||||
>
|
>
|
||||||
> 🍪 需要 Cookie 的平台(Twitter、小红书等),建议使用 Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 一键导出。**服务器用户**没有浏览器界面,请在自己的电脑上登录对应网站后导出 Cookie,再发给 Agent 配置。
|
> 🍪 需要 Cookie 的平台(Twitter、小红书等),**优先使用** Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 导出 Cookie,发给 Agent 即可配置。流程统一:浏览器登录 → Cookie-Editor 导出 → 发给 Agent。比扫码更简单可靠。
|
||||||
>
|
>
|
||||||
> 🔒 Cookie 只存在你本地,不上传不外传。代码完全开源,随时可审查。
|
> 🔒 Cookie 只存在你本地,不上传不外传。代码完全开源,随时可审查。
|
||||||
> 💻 本地电脑不需要代理。代理只有部署在服务器上才需要(~$1/月)。
|
> 💻 本地电脑不需要代理。代理只有部署在服务器上才需要(~$1/月)。
|
||||||
@@ -84,6 +94,16 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|
|
||||||
## 快速上手
|
## 快速上手
|
||||||
|
|
||||||
|
> ⚠️ **OpenClaw 用户请先确认 exec 权限已开启**
|
||||||
|
>
|
||||||
|
> Agent Reach 依赖 Agent 执行 shell 命令(`pip install`、`mcporter`、`bird` 等)。如果你的 OpenClaw 使用了默认的 `messaging` 工具配置,Agent 将无法执行命令。**安装前请先开启 exec 权限**:
|
||||||
|
>
|
||||||
|
> ```bash
|
||||||
|
> openclaw config set tools.profile "coding"
|
||||||
|
> ```
|
||||||
|
> 或在 `~/.openclaw/openclaw.json` 中设置 `"tools": { "profile": "coding" }`。
|
||||||
|
> 设置后重启 Gateway(`openclaw gateway restart`)并开启新对话即可。其他平台(Claude Code、Cursor、Windsurf 等)不受此限制。
|
||||||
|
|
||||||
复制这句话给你的 AI Agent(Claude Code、OpenClaw、Cursor 等):
|
复制这句话给你的 AI Agent(Claude Code、OpenClaw、Cursor 等):
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -92,6 +112,11 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|
|
||||||
就这一步。Agent 会自己完成剩下的所有事情。
|
就这一步。Agent 会自己完成剩下的所有事情。
|
||||||
|
|
||||||
|
> 🔄 **已安装过?** 更新也是一句话:
|
||||||
|
> ```
|
||||||
|
> 帮我更新 Agent Reach:https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
> ```
|
||||||
|
|
||||||
> 🛡️ **担心安全?** 可以用安全模式——不会自动装系统包,只告诉你需要什么:
|
> 🛡️ **担心安全?** 可以用安全模式——不会自动装系统包,只告诉你需要什么:
|
||||||
> ```
|
> ```
|
||||||
> 帮我安装 Agent Reach(安全模式):https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
> 帮我安装 Agent Reach(安全模式):https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
||||||
@@ -105,7 +130,7 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
2. **安装系统依赖** — 自动检测并安装 Node.js、gh CLI、mcporter、bird 等
|
2. **安装系统依赖** — 自动检测并安装 Node.js、gh CLI、mcporter、bird 等
|
||||||
3. **配置搜索引擎** — 通过 MCP 接入 Exa(免费,无需 API Key)
|
3. **配置搜索引擎** — 通过 MCP 接入 Exa(免费,无需 API Key)
|
||||||
4. **检测环境** — 判断是本地电脑还是服务器,给出对应的配置建议
|
4. **检测环境** — 判断是本地电脑还是服务器,给出对应的配置建议
|
||||||
5. **注册 Skill** — 在 Agent 的 skills 目录安装 SKILL.md,以后 Agent 遇到"搜推特"、"看视频"这类需求,会自动知道调用 Agent Reach
|
5. **注册 SKILL.md** — 在 Agent 的 skills 目录安装使用指南,以后 Agent 遇到"搜推特"、"看视频"这类需求,会自动知道该调哪个上游工具
|
||||||
|
|
||||||
安装完之后,`agent-reach doctor` 一条命令告诉你每个渠道的状态。
|
安装完之后,`agent-reach doctor` 一条命令告诉你每个渠道的状态。
|
||||||
</details>
|
</details>
|
||||||
@@ -116,14 +141,14 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|
|
||||||
不需要任何配置,告诉 Agent 就行:
|
不需要任何配置,告诉 Agent 就行:
|
||||||
|
|
||||||
- "帮我看看这个链接" → 任意网页
|
- "帮我看看这个链接" → `curl https://r.jina.ai/URL` 读任意网页
|
||||||
- "这个 GitHub 仓库是做什么的" → GitHub 仓库、Issue、代码
|
- "这个 GitHub 仓库是做什么的" → `gh repo view owner/repo`
|
||||||
- "这个视频讲了什么" → YouTube / B站字幕提取
|
- "这个视频讲了什么" → `yt-dlp --dump-json URL` 提取字幕
|
||||||
- "帮我看看这条推文" → Twitter 推文
|
- "帮我看看这条推文" → `bird read URL`
|
||||||
- "订阅这个 RSS" → RSS / Atom 源
|
- "订阅这个 RSS" → `feedparser` 解析
|
||||||
- "搜一下 GitHub 上有什么 LLM 框架" → GitHub 搜索
|
- "搜一下 GitHub 上有什么 LLM 框架" → `gh search repos "LLM framework"`
|
||||||
|
|
||||||
**不需要记命令。** Agent 自己知道该调什么。
|
**不需要记命令。** Agent 读了 SKILL.md 之后自己知道该调什么。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -135,27 +160,31 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|
|
||||||
Agent Reach 做的事情很简单:**帮你把这些选型和配置的活儿做完了。**
|
Agent Reach 做的事情很简单:**帮你把这些选型和配置的活儿做完了。**
|
||||||
|
|
||||||
|
安装完成后,Agent 直接调用上游工具(bird CLI、yt-dlp、mcporter、gh CLI 等),不需要经过 Agent Reach 的包装层。
|
||||||
|
|
||||||
### 🔌 每个渠道都是可插拔的
|
### 🔌 每个渠道都是可插拔的
|
||||||
|
|
||||||
每个平台对应一个独立的 Python 文件,实现统一接口。**后端工具随时可以换**——哪天出了更好的工具,改一个文件就行,其他不用动。
|
每个平台背后是一个独立的上游工具。**不满意?换掉就行。**
|
||||||
|
|
||||||
```
|
```
|
||||||
channels/
|
channels/
|
||||||
├── web.py → Jina Reader ← 可以换成 Firecrawl、Crawl4AI……
|
├── web.py → Jina Reader ← 可以换成 Firecrawl、Crawl4AI……
|
||||||
├── twitter.py → bird ← 可以换成 Nitter、官方 API……
|
├── twitter.py → bird ← 可以换成 Nitter、官方 API……
|
||||||
├── youtube.py → yt-dlp ← 可以换成 YouTube API、Whisper……
|
├── youtube.py → yt-dlp ← 可以换成 YouTube API、Whisper……
|
||||||
├── github.py → gh CLI ← 可以换成 REST API、PyGithub……
|
├── github.py → gh CLI ← 可以换成 REST API、PyGithub……
|
||||||
├── bilibili.py → yt-dlp ← 可以换成 bilibili-api……
|
├── bilibili.py → yt-dlp ← 可以换成 bilibili-api……
|
||||||
├── reddit.py → JSON API + Exa ← 可以换成 PRAW、Pushshift……
|
├── reddit.py → Exa ← 搜索+阅读,无需代理
|
||||||
├── xiaohongshu.py → mcporter MCP ← 可以换成其他 XHS 工具……
|
├── xiaohongshu.py → mcporter MCP ← 可以换成其他 XHS 工具……
|
||||||
├── instagram.py → instaloader ← 可以换成 instagrapi、官方 API……
|
├── douyin.py → mcporter MCP ← 可以换成其他抖音工具……
|
||||||
├── linkedin.py → linkedin-mcp ← 可以换成 LinkedIn API……
|
├── linkedin.py → linkedin-mcp ← 可以换成 LinkedIn API……
|
||||||
├── bosszhipin.py → mcp-bosszp ← 可以换成其他招聘工具……
|
├── wechat.py → camoufox+miku ← 搜索+阅读微信公众号文章
|
||||||
├── rss.py → feedparser ← 可以换成 atoma……
|
├── rss.py → feedparser ← 可以换成 atoma……
|
||||||
├── exa_search.py → mcporter MCP ← 可以换成 Tavily、SerpAPI……
|
├── exa_search.py → mcporter MCP ← 可以换成 Tavily、SerpAPI……
|
||||||
└── __init__.py → 渠道注册
|
└── __init__.py → 渠道注册(doctor 检测用)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
每个渠道文件只负责检测对应上游工具是否可用(`check()` 方法),给 `agent-reach doctor` 提供状态信息。实际的读取和搜索由 Agent 直接调用上游工具完成。
|
||||||
|
|
||||||
### 当前选型
|
### 当前选型
|
||||||
|
|
||||||
| 场景 | 选型 | 为什么选它 |
|
| 场景 | 选型 | 为什么选它 |
|
||||||
@@ -167,12 +196,38 @@ channels/
|
|||||||
| GitHub | [gh CLI](https://cli.github.com) | 官方工具,认证后完整 API 能力 |
|
| GitHub | [gh CLI](https://cli.github.com) | 官方工具,认证后完整 API 能力 |
|
||||||
| 读 RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python 生态标准选择,2.3K Star |
|
| 读 RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python 生态标准选择,2.3K Star |
|
||||||
| 小红书 | [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) | ⭐9K+,Go 语言,Docker 一键部署 |
|
| 小红书 | [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) | ⭐9K+,Go 语言,Docker 一键部署 |
|
||||||
| Instagram | [instaloader](https://github.com/instaloader/instaloader) | ⭐9.8K,Python CLI,Cookie 登录,免费 |
|
| 抖音 | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP 服务,无需登录,视频解析 + 无水印下载 |
|
||||||
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | ⭐900+,MCP 服务,浏览器自动化 |
|
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | ⭐900+,MCP 服务,浏览器自动化 |
|
||||||
| Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP 服务,支持职位搜索和打招呼 |
|
| 微信公众号 | [wechat-article-for-ai](https://github.com/Panniantong/wechat-article-for-ai) + [miku_ai](https://github.com/GobinFan/Miku_Spider) | Camoufox 隐身浏览器读全文 + 搜狗搜索 |
|
||||||
|
|
||||||
> 📌 这些都是「当前选型」。不满意?换掉对应文件就行。这正是脚手架的意义。
|
> 📌 这些都是「当前选型」。不满意?换掉对应文件就行。这正是脚手架的意义。
|
||||||
|
|
||||||
|
### 抖音 / 小红书脚本提取的可选实现
|
||||||
|
|
||||||
|
如果你不只是想“解析抖音视频信息”,还想统一处理:
|
||||||
|
|
||||||
|
- 抖音视频脚本提取
|
||||||
|
- 小红书视频笔记脚本提取
|
||||||
|
- 小红书图文笔记正文 + 图片文字提取
|
||||||
|
- 固定输出 `script.md` 和 `info.json`
|
||||||
|
|
||||||
|
可以把 `douyin` 这个 mcporter alias 指向另一个兼容实现:
|
||||||
|
|
||||||
|
- [social-post-extractor-mcp](https://github.com/JNHFlow21/social-post-extractor-mcp)
|
||||||
|
|
||||||
|
这个实现保留了旧工具名兼容性:
|
||||||
|
|
||||||
|
- `parse_douyin_video_info`
|
||||||
|
- `get_douyin_download_link`
|
||||||
|
- `extract_douyin_text`
|
||||||
|
|
||||||
|
同时新增统一工具:
|
||||||
|
|
||||||
|
- `parse_social_post_info`
|
||||||
|
- `extract_social_post_script`
|
||||||
|
|
||||||
|
所以从 Agent Reach 的视角看,它依然只是一个 `mcporter` 里的 `douyin` server,只是能力更完整。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 安全性
|
## 安全性
|
||||||
@@ -189,7 +244,11 @@ Agent Reach 在设计上重视安全:
|
|||||||
|
|
||||||
### 🍪 Cookie 安全建议
|
### 🍪 Cookie 安全建议
|
||||||
|
|
||||||
需要 Cookie 的平台(Twitter、小红书、Instagram)建议使用**专用小号**,不要用主账号。Cookie 等同于完整登录权限,用小号可以在凭据泄露时限制影响范围。
|
> ⚠️ **封号风险提醒:** 使用 Cookie 登录的平台(Twitter、小红书等),通过脚本/API 调用**存在被平台检测并封号的风险**。请务必使用**专用小号**,不要用你的主账号。
|
||||||
|
|
||||||
|
需要 Cookie 的平台(Twitter、小红书)建议使用**专用小号**,不要用主账号。原因有二:
|
||||||
|
1. **封号风险** — 平台可能检测到非正常浏览器的 API 调用行为,导致账号被限制或封禁
|
||||||
|
2. **安全风险** — Cookie 等同于完整登录权限,用小号可以在凭据泄露时限制影响范围
|
||||||
|
|
||||||
### 📦 安装方式
|
### 📦 安装方式
|
||||||
|
|
||||||
@@ -199,6 +258,24 @@ Agent Reach 在设计上重视安全:
|
|||||||
| 安全模式 | `agent-reach install --env=auto --safe` | 生产服务器、多人共用机器 |
|
| 安全模式 | `agent-reach install --env=auto --safe` | 生产服务器、多人共用机器 |
|
||||||
| 仅预览 | `agent-reach install --env=auto --dry-run` | 先看看会做什么 |
|
| 仅预览 | `agent-reach install --env=auto --dry-run` | 先看看会做什么 |
|
||||||
|
|
||||||
|
### 🗑️ 卸载
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach uninstall
|
||||||
|
```
|
||||||
|
|
||||||
|
会清除:`~/.agent-reach/`(含所有 token/cookie)、各 Agent 的 skill 文件、mcporter 中的 MCP 配置。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 只预览,不实际删除
|
||||||
|
agent-reach uninstall --dry-run
|
||||||
|
|
||||||
|
# 只删 skill 文件,保留 token 配置(重装时用)
|
||||||
|
agent-reach uninstall --keep-config
|
||||||
|
```
|
||||||
|
|
||||||
|
卸载 Python 包本身:`pip uninstall agent-reach`
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 贡献
|
## 贡献
|
||||||
@@ -225,10 +302,91 @@ Agent Reach 在设计上重视安全:
|
|||||||
|
|
||||||
Star 一下,下次需要的时候能找到。⭐
|
Star 一下,下次需要的时候能找到。⭐
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 常见问题 / FAQ
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>AI Agent 怎么搜索 Twitter / X?不想付 API 费用</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach 使用 [bird CLI](https://www.npmjs.com/package/@steipete/bird) 通过 Cookie 认证访问 Twitter,完全免费。安装 Agent Reach 后,用 Cookie-Editor 导出你的 Twitter Cookie,运行 `agent-reach configure twitter-cookies "your_cookies"` 即可。之后 Agent 就可以用 `bird search "关键词"` 搜索推文了。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to search Twitter/X with AI agent for free (no API)?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach uses the bird CLI with cookie auth — zero API fees. After installing, export your Twitter cookies with the Cookie-Editor extension, run `agent-reach configure twitter-cookies "your_cookies"`, then your agent can search with `bird search "query"`.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Reddit 返回 403 怎么办?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach 已改为通过 Exa 搜索和阅读 Reddit 内容,完全绕过 Reddit API 的 IP 封锁。无需代理,无需额外配置。运行 `agent-reach install --env=auto` 自动安装 Exa。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to get YouTube video transcripts for AI?</strong></summary>
|
||||||
|
|
||||||
|
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"` extracts video metadata; `yt-dlp --write-sub --skip-download "URL"` extracts subtitles. Uses yt-dlp under the hood, supports multiple languages. No API key needed.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>怎么让 AI Agent 读小红书?</strong></summary>
|
||||||
|
|
||||||
|
小红书需要通过 Docker 运行一个 MCP 服务。安装 Docker 后,运行 `agent-reach install` 会自动配置。之后 Agent 就能用 `mcporter call 'xiaohongshu.get_feed_detail(...)'` 读取笔记或 `mcporter call 'xiaohongshu.search_feeds(keyword: "关键词")'` 搜索了。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>怎么让 AI Agent 解析抖音视频?</strong></summary>
|
||||||
|
|
||||||
|
安装 douyin-mcp-server 后,Agent 就能用 `mcporter call 'douyin.parse_douyin_video_info(share_link: "分享链接")'` 解析视频信息、获取无水印下载链接。不需要登录,把抖音分享链接发给 Agent 就行。详见 https://github.com/yzfly/douyin-mcp-server
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Compatible with Claude Code / Cursor / OpenClaw / Windsurf?</strong></summary>
|
||||||
|
|
||||||
|
Yes! Agent Reach is an installer + configuration tool — any AI coding agent that can run shell commands can use it. Works with Claude Code, Cursor, OpenClaw, Windsurf, Codex, and more. Just `pip install agent-reach`, run `agent-reach install`, and the agent can start using the upstream tools immediately.
|
||||||
|
|
||||||
|
**OpenClaw note:** If your OpenClaw is using the default `messaging` tool profile, the agent won't be able to run shell commands. Enable exec first: `openclaw config set tools.profile "coding"` (or set `"tools": { "profile": "coding" }` in `~/.openclaw/openclaw.json`), then restart the Gateway and start a new conversation before installing.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Is this free? Any API costs?</strong></summary>
|
||||||
|
|
||||||
|
100% free. All backends are open-source tools (bird CLI, yt-dlp, Jina Reader, Exa, etc.) that don't require paid API keys. The only optional cost is a residential proxy (~$1/month) if you need Bilibili access from a server. Reddit now works free via Exa without any proxy.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 致谢
|
## 致谢
|
||||||
|
|
||||||
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://www.npmjs.com/package/@steipete/bird) · [Exa](https://exa.ai) · [mcporter](https://github.com/steipete/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) · [instaloader](https://github.com/instaloader/instaloader) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
|
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://www.npmjs.com/package/@steipete/bird) · [Exa](https://exa.ai) · [mcporter](https://github.com/steipete/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||||
|
|
||||||
|
## 联系
|
||||||
|
|
||||||
|
- 📧 **Email:** pnt01@foxmail.com
|
||||||
|
- 🐦 **Twitter/X:** [@Neo_Reidlab](https://x.com/Neo_Reidlab)
|
||||||
|
|
||||||
|
交流或合作可加微信,拉你进交流群:
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="docs/wechat-group-qr.jpg" width="280" alt="WeChat QR">
|
||||||
|
</p>
|
||||||
|
|
||||||
|
> Bug 反馈和功能请求请用 [GitHub Issues](https://github.com/Panniantong/Agent-Reach/issues),更容易跟踪。
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
[MIT](LICENSE)
|
[MIT](LICENSE)
|
||||||
|
|
||||||
|
## 友情链接
|
||||||
|
|
||||||
|
[FluxNode](https://fluxnode.org) — 低价 AI API 中转站,官方一折,可按量或按套餐付费。可用于 OpenClaw、Claude Code 等一切 Agent。
|
||||||
|
|
||||||
|
[OpenClaw for Enterprise](https://github.com/littleben/openclaw-for-enterprise) — 企业级 OpenClaw 多用户部署方案,飞书里直接用 AI,容器隔离,一条命令管理。
|
||||||
|
|
||||||
|
[腾讯云 OpenClaw](https://www.tencentcloud.com/act/pro/intl-openclaw?referral_code=G76Y819A&lang=zh&pg=) — 在腾讯云Lighthouse秒级部署OpenClaw全能助手,可通过对话丝滑接入Agent Reach,给你的OpenClaw一键装上互联网能力。
|
||||||
|
|
||||||
|
## Star History
|
||||||
|
|
||||||
|
[](https://star-history.com/#Panniantong/Agent-Reach&Date)
|
||||||
|
|||||||
@@ -1,87 +0,0 @@
|
|||||||
---
|
|
||||||
name: agent-reach
|
|
||||||
description: >
|
|
||||||
Give your AI agent eyes to see the entire internet. Read and search across
|
|
||||||
Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, RSS, and any web page
|
|
||||||
— all from a single CLI. Use when: (1) reading content from URLs (tweets, Reddit posts,
|
|
||||||
articles, videos), (2) searching across platforms (web, Twitter, Reddit, GitHub, YouTube,
|
|
||||||
Bilibili, XiaoHongShu), (3) checking channel health or updating Agent Reach.
|
|
||||||
Triggers: "search Twitter/Reddit/YouTube", "read this URL", "find posts about",
|
|
||||||
"搜索", "读取", "查一下", "看看这个链接".
|
|
||||||
---
|
|
||||||
|
|
||||||
# Agent Reach
|
|
||||||
|
|
||||||
Read and search the internet across 9+ platforms via unified CLI.
|
|
||||||
|
|
||||||
## Setup
|
|
||||||
|
|
||||||
First check if agent-reach is installed:
|
|
||||||
```bash
|
|
||||||
agent-reach doctor
|
|
||||||
```
|
|
||||||
|
|
||||||
If command not found, install it:
|
|
||||||
```bash
|
|
||||||
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
|
||||||
agent-reach install --env=auto
|
|
||||||
```
|
|
||||||
|
|
||||||
`install` auto-detects your environment and installs all dependencies (Node.js, mcporter, bird CLI, gh CLI). Read the output and run `agent-reach doctor` to see what's active.
|
|
||||||
|
|
||||||
For channels that need user input, ask the user. See the full setup guide:
|
|
||||||
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
|
||||||
|
|
||||||
## Commands
|
|
||||||
|
|
||||||
### Read any URL
|
|
||||||
```bash
|
|
||||||
agent-reach read <url>
|
|
||||||
agent-reach read <url> --json # structured output
|
|
||||||
```
|
|
||||||
Handles: tweets, Reddit posts, articles, YouTube (transcripts), GitHub repos, etc.
|
|
||||||
|
|
||||||
### Search
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach search "query" # web search (Exa)
|
|
||||||
agent-reach search-twitter "query" # Twitter/X
|
|
||||||
agent-reach search-reddit "query" # Reddit (--sub <subreddit>)
|
|
||||||
agent-reach search-github "query" # GitHub (--lang <language>)
|
|
||||||
agent-reach search-youtube "query" # YouTube
|
|
||||||
agent-reach search-bilibili "query" # Bilibili (B站)
|
|
||||||
agent-reach search-xhs "query" # XiaoHongShu (小红书)
|
|
||||||
```
|
|
||||||
|
|
||||||
All search commands support `-n <count>` for number of results.
|
|
||||||
|
|
||||||
### Management
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach doctor # channel status overview
|
|
||||||
agent-reach watch # quick health + update check (for scheduled tasks)
|
|
||||||
agent-reach check-update # check for new versions
|
|
||||||
```
|
|
||||||
|
|
||||||
### Configure channels
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
|
|
||||||
agent-reach configure proxy http://user:pass@ip:port
|
|
||||||
agent-reach configure --from-browser chrome # auto-extract cookies
|
|
||||||
```
|
|
||||||
|
|
||||||
## Channel Status Tiers
|
|
||||||
|
|
||||||
- **Tier 0 (zero config):** Web, YouTube, RSS, Twitter (read-only via Jina)
|
|
||||||
- **Tier 1 (free setup):** Exa web search (mcporter required)
|
|
||||||
- **Tier 2 (user config):** Twitter search (cookie), Reddit full (proxy), GitHub (token), Bilibili (proxy), XiaoHongShu (MCP)
|
|
||||||
|
|
||||||
Run `agent-reach doctor` to see which channels are active.
|
|
||||||
|
|
||||||
## Tips
|
|
||||||
|
|
||||||
- Always try `agent-reach read <url>` first for any URL — it auto-detects the platform
|
|
||||||
- For Twitter cookies, recommend the user install [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome extension
|
|
||||||
- Reddit and Bilibili block server IPs — suggest a residential proxy (~$1/month) if on a server
|
|
||||||
- If a channel breaks, run `agent-reach doctor` to diagnose
|
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Agent Reach — Give your AI Agent eyes to see the entire internet."""
|
"""Agent Reach — Give your AI Agent eyes to see the entire internet."""
|
||||||
|
|
||||||
__version__ = "1.0.0"
|
__version__ = "1.4.0"
|
||||||
__author__ = "Neo Reid"
|
__author__ = "Neo Reid"
|
||||||
|
|
||||||
from agent_reach.core import AgentReach
|
from agent_reach.core import AgentReach
|
||||||
|
|||||||
@@ -1,14 +1,10 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Channel registry — routes URLs to the right channel.
|
Channel registry — lists all supported platforms for doctor checks.
|
||||||
|
|
||||||
This is the core of Agent Reach' pluggable architecture.
|
|
||||||
Add a new channel: just create a file and register it here.
|
|
||||||
Swap a backend: just change the implementation inside the channel file.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Dict, List, Optional
|
from typing import List, Optional
|
||||||
from .base import Channel, ReadResult, SearchResult
|
from .base import Channel
|
||||||
|
|
||||||
# Import all channels
|
# Import all channels
|
||||||
from .web import WebChannel
|
from .web import WebChannel
|
||||||
@@ -20,12 +16,15 @@ from .rss import RSSChannel
|
|||||||
from .bilibili import BilibiliChannel
|
from .bilibili import BilibiliChannel
|
||||||
from .exa_search import ExaSearchChannel
|
from .exa_search import ExaSearchChannel
|
||||||
from .xiaohongshu import XiaoHongShuChannel
|
from .xiaohongshu import XiaoHongShuChannel
|
||||||
from .instagram import InstagramChannel
|
from .douyin import DouyinChannel
|
||||||
from .linkedin import LinkedInChannel
|
from .linkedin import LinkedInChannel
|
||||||
from .bosszhipin import BossZhipinChannel
|
from .wechat import WeChatChannel
|
||||||
|
from .weibo import WeiboChannel
|
||||||
|
from .xiaoyuzhou import XiaoyuzhouChannel
|
||||||
|
from .v2ex import V2EXChannel
|
||||||
|
from .xueqiu import XueqiuChannel
|
||||||
|
|
||||||
|
|
||||||
# Channel registry — order matters (first match wins, web is last as fallback)
|
|
||||||
ALL_CHANNELS: List[Channel] = [
|
ALL_CHANNELS: List[Channel] = [
|
||||||
GitHubChannel(),
|
GitHubChannel(),
|
||||||
TwitterChannel(),
|
TwitterChannel(),
|
||||||
@@ -33,27 +32,18 @@ ALL_CHANNELS: List[Channel] = [
|
|||||||
RedditChannel(),
|
RedditChannel(),
|
||||||
BilibiliChannel(),
|
BilibiliChannel(),
|
||||||
XiaoHongShuChannel(),
|
XiaoHongShuChannel(),
|
||||||
InstagramChannel(),
|
DouyinChannel(),
|
||||||
LinkedInChannel(),
|
LinkedInChannel(),
|
||||||
BossZhipinChannel(),
|
WeChatChannel(),
|
||||||
|
WeiboChannel(),
|
||||||
|
XiaoyuzhouChannel(),
|
||||||
|
V2EXChannel(),
|
||||||
|
XueqiuChannel(),
|
||||||
RSSChannel(),
|
RSSChannel(),
|
||||||
ExaSearchChannel(),
|
ExaSearchChannel(),
|
||||||
WebChannel(), # Fallback — handles any URL
|
WebChannel(),
|
||||||
]
|
]
|
||||||
|
|
||||||
# Search-capable channels
|
|
||||||
SEARCH_CHANNELS: Dict[str, Channel] = {
|
|
||||||
ch.name: ch for ch in ALL_CHANNELS if ch.can_search()
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def get_channel_for_url(url: str) -> Channel:
|
|
||||||
"""Find the right channel for a URL."""
|
|
||||||
for channel in ALL_CHANNELS:
|
|
||||||
if channel.can_handle(url):
|
|
||||||
return channel
|
|
||||||
return WebChannel() # Should never reach here, but just in case
|
|
||||||
|
|
||||||
|
|
||||||
def get_channel(name: str) -> Optional[Channel]:
|
def get_channel(name: str) -> Optional[Channel]:
|
||||||
"""Get a channel by name."""
|
"""Get a channel by name."""
|
||||||
@@ -69,7 +59,7 @@ def get_all_channels() -> List[Channel]:
|
|||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"Channel", "ReadResult", "SearchResult",
|
"Channel",
|
||||||
"ALL_CHANNELS", "SEARCH_CHANNELS",
|
"ALL_CHANNELS",
|
||||||
"get_channel_for_url", "get_channel", "get_all_channels",
|
"get_channel", "get_all_channels",
|
||||||
]
|
]
|
||||||
|
|||||||
+11
-115
@@ -1,110 +1,28 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Channel base class — the universal interface for all platforms.
|
Channel base class — platform availability checking.
|
||||||
|
|
||||||
Every channel (YouTube, Twitter, GitHub, etc.) implements this interface.
|
Each channel represents a platform (YouTube, Twitter, GitHub, etc.)
|
||||||
The backend tool can be swapped anytime without changing anything else.
|
and provides:
|
||||||
|
- can_handle(url) → does this URL belong to this platform?
|
||||||
|
- check(config) → is the upstream tool installed and configured?
|
||||||
|
|
||||||
Example:
|
After installation, agents call upstream tools directly.
|
||||||
class YouTubeChannel(Channel):
|
|
||||||
name = "youtube"
|
|
||||||
backends = ["yt-dlp"] # current backend, can be swapped
|
|
||||||
|
|
||||||
async def read(self, url, config):
|
|
||||||
# Just call yt-dlp, return standardized dict
|
|
||||||
...
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
from abc import ABC, abstractmethod
|
from abc import ABC, abstractmethod
|
||||||
from dataclasses import dataclass
|
from typing import List, Tuple
|
||||||
from typing import Any, Dict, List, Optional, Tuple
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class ReadResult:
|
|
||||||
"""Standardized read result. Every channel returns this."""
|
|
||||||
title: str
|
|
||||||
content: str
|
|
||||||
url: str
|
|
||||||
author: str = ""
|
|
||||||
date: str = ""
|
|
||||||
platform: str = ""
|
|
||||||
extra: dict = None
|
|
||||||
|
|
||||||
def __post_init__(self):
|
|
||||||
self.extra = self.extra or {}
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
d = {
|
|
||||||
"title": self.title,
|
|
||||||
"content": self.content,
|
|
||||||
"url": self.url,
|
|
||||||
"platform": self.platform,
|
|
||||||
}
|
|
||||||
if self.author:
|
|
||||||
d["author"] = self.author
|
|
||||||
if self.date:
|
|
||||||
d["date"] = self.date
|
|
||||||
if self.extra:
|
|
||||||
d["extra"] = self.extra
|
|
||||||
return d
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class SearchResult:
|
|
||||||
"""Standardized search result."""
|
|
||||||
title: str
|
|
||||||
url: str
|
|
||||||
snippet: str = ""
|
|
||||||
author: str = ""
|
|
||||||
date: str = ""
|
|
||||||
score: float = 0
|
|
||||||
extra: dict = None
|
|
||||||
|
|
||||||
def __post_init__(self):
|
|
||||||
self.extra = self.extra or {}
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
d = {
|
|
||||||
"title": self.title,
|
|
||||||
"url": self.url,
|
|
||||||
"snippet": self.snippet,
|
|
||||||
}
|
|
||||||
if self.author:
|
|
||||||
d["author"] = self.author
|
|
||||||
if self.date:
|
|
||||||
d["date"] = self.date
|
|
||||||
if self.extra:
|
|
||||||
d["extra"] = self.extra
|
|
||||||
return d
|
|
||||||
|
|
||||||
|
|
||||||
class Channel(ABC):
|
class Channel(ABC):
|
||||||
"""
|
"""Base class for all channels."""
|
||||||
Base class for all channels.
|
|
||||||
|
|
||||||
Subclasses just need to implement:
|
|
||||||
- read(url, config) → ReadResult
|
|
||||||
- can_handle(url) → bool
|
|
||||||
- check(config) → (status, message)
|
|
||||||
|
|
||||||
Optionally:
|
|
||||||
- search(query, config, **kwargs) → list[SearchResult]
|
|
||||||
"""
|
|
||||||
|
|
||||||
name: str = "" # e.g. "youtube"
|
name: str = "" # e.g. "youtube"
|
||||||
description: str = "" # e.g. "YouTube video transcripts"
|
description: str = "" # e.g. "YouTube 视频和字幕"
|
||||||
backends: List[str] = [] # e.g. ["yt-dlp"] — what external tool is used
|
backends: List[str] = [] # e.g. ["yt-dlp"] — what upstream tool is used
|
||||||
requires_config: List[str] = [] # e.g. ["reddit_proxy"]
|
|
||||||
requires_tools: List[str] = [] # e.g. ["yt-dlp"]
|
|
||||||
tier: int = 0 # 0=zero-config, 1=needs free key, 2=needs setup
|
tier: int = 0 # 0=zero-config, 1=needs free key, 2=needs setup
|
||||||
|
|
||||||
@abstractmethod
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
"""Read content from a URL. Must return ReadResult."""
|
|
||||||
...
|
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
"""Check if this channel can handle this URL."""
|
"""Check if this channel can handle this URL."""
|
||||||
@@ -112,29 +30,7 @@ class Channel(ABC):
|
|||||||
|
|
||||||
def check(self, config=None) -> Tuple[str, str]:
|
def check(self, config=None) -> Tuple[str, str]:
|
||||||
"""
|
"""
|
||||||
Check if this channel is available.
|
Check if this channel's upstream tool is available.
|
||||||
Returns (status, message) where status is 'ok'/'warn'/'off'/'error'.
|
Returns (status, message) where status is 'ok'/'warn'/'off'/'error'.
|
||||||
"""
|
"""
|
||||||
# Check required tools
|
|
||||||
for tool in self.requires_tools:
|
|
||||||
if not shutil.which(tool):
|
|
||||||
return "off", f"需要安装:pip install {tool}"
|
|
||||||
|
|
||||||
# Check required config
|
|
||||||
for key in self.requires_config:
|
|
||||||
if config and not config.get(key):
|
|
||||||
return "off", f"需要配置 {key},运行 agent-reach setup"
|
|
||||||
|
|
||||||
return "ok", f"{'、'.join(self.backends) if self.backends else '内置'}"
|
return "ok", f"{'、'.join(self.backends) if self.backends else '内置'}"
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
"""Search this platform. Override if supported."""
|
|
||||||
raise NotImplementedError(f"{self.name} does not support search")
|
|
||||||
|
|
||||||
def can_search(self) -> bool:
|
|
||||||
"""Whether this channel supports search."""
|
|
||||||
try:
|
|
||||||
# Check if search is overridden
|
|
||||||
return type(self).search is not Channel.search
|
|
||||||
except:
|
|
||||||
return False
|
|
||||||
|
|||||||
@@ -1,207 +1,66 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Bilibili — via yt-dlp (same backend as YouTube).
|
"""Bilibili — video via yt-dlp, search/browse via bili-cli or API."""
|
||||||
|
|
||||||
Backend: yt-dlp (https://github.com/yt-dlp/yt-dlp)
|
|
||||||
yt-dlp natively supports Bilibili — video info, subtitles, and search.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
import json
|
||||||
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
import urllib.request
|
||||||
from .base import Channel, ReadResult, SearchResult
|
from .base import Channel
|
||||||
from typing import List
|
|
||||||
|
_UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
|
||||||
|
_TIMEOUT = 10
|
||||||
|
_SEARCH_API = "https://api.bilibili.com/x/web-interface/search/all/v2?keyword=test&page=1"
|
||||||
|
|
||||||
|
|
||||||
|
def _search_api_ok() -> bool:
|
||||||
|
"""Return True if Bilibili search API responds with code 0."""
|
||||||
|
req = urllib.request.Request(_SEARCH_API, headers={"User-Agent": _UA})
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=_TIMEOUT) as resp:
|
||||||
|
data = json.loads(resp.read())
|
||||||
|
return data.get("code") == 0
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
class BilibiliChannel(Channel):
|
class BilibiliChannel(Channel):
|
||||||
name = "bilibili"
|
name = "bilibili"
|
||||||
description = "B站视频信息和字幕"
|
description = "B站视频、字幕和搜索"
|
||||||
backends = ["yt-dlp"]
|
backends = ["yt-dlp", "bili-cli (可选)", "B站搜索 API"]
|
||||||
requires_tools = ["yt-dlp"]
|
tier = 1
|
||||||
tier = 0
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
d = urlparse(url).netloc.lower()
|
d = urlparse(url).netloc.lower()
|
||||||
return "bilibili.com" in d or "b23.tv" in d
|
return "bilibili.com" in d or "b23.tv" in d
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if not shutil.which("yt-dlp"):
|
if not shutil.which("yt-dlp"):
|
||||||
return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
|
return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
|
||||||
proxy = config.get("bilibili_proxy") if config else None
|
|
||||||
|
proxy = (config.get("bilibili_proxy") if config else None) or os.environ.get("BILIBILI_PROXY")
|
||||||
|
has_bili_cli = bool(shutil.which("bili"))
|
||||||
|
|
||||||
|
parts = []
|
||||||
|
|
||||||
|
# 视频读取状态
|
||||||
if proxy:
|
if proxy:
|
||||||
return "ok", "已配置代理,完整可用"
|
parts.append("视频读取:yt-dlp(代理已配置)")
|
||||||
import os
|
else:
|
||||||
is_server = bool(os.environ.get("SSH_CONNECTION") or os.path.exists("/etc/cloud"))
|
parts.append("视频读取:yt-dlp")
|
||||||
if is_server:
|
|
||||||
return "warn", "服务器 IP 可能被封,配置代理即可解决:agent-reach configure proxy URL"
|
|
||||||
return "ok", "本地直连可用"
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
# bili-cli 增强
|
||||||
if not shutil.which("yt-dlp"):
|
if has_bili_cli:
|
||||||
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
|
parts.append("搜索/热门/排行:bili-cli 可用")
|
||||||
|
else:
|
||||||
|
# 检测搜索 API 连通性
|
||||||
|
api_ok = _search_api_ok()
|
||||||
|
if api_ok:
|
||||||
|
parts.append("搜索:B站 API 可用")
|
||||||
|
else:
|
||||||
|
parts.append("搜索:B站 API 不可达")
|
||||||
|
parts.append("提示:安装 bili-cli 可解锁热门/排行/动态:pipx install bilibili-cli")
|
||||||
|
|
||||||
proxy = config.get("bilibili_proxy") if config else None
|
status = "ok" if has_bili_cli or _search_api_ok() else "warn"
|
||||||
|
return status, "。".join(parts)
|
||||||
# Get video info via yt-dlp
|
|
||||||
info = self._get_info(url, proxy)
|
|
||||||
if not info:
|
|
||||||
return ReadResult(
|
|
||||||
title="Bilibili",
|
|
||||||
content=f"⚠️ 无法获取视频信息: {url}\n服务器 IP 可能被封,配个代理:agent-reach configure proxy URL",
|
|
||||||
url=url, platform="bilibili",
|
|
||||||
)
|
|
||||||
|
|
||||||
title = info.get("title", url)
|
|
||||||
author = info.get("uploader", "")
|
|
||||||
desc = info.get("description", "")
|
|
||||||
|
|
||||||
# Try subtitles
|
|
||||||
subtitle = self._get_subtitles(url, proxy)
|
|
||||||
content = desc
|
|
||||||
if subtitle:
|
|
||||||
content += f"\n\n## 字幕\n{subtitle}"
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=title, content=content, url=url,
|
|
||||||
author=author, platform="bilibili",
|
|
||||||
extra={
|
|
||||||
"view_count": info.get("view_count"),
|
|
||||||
"like_count": info.get("like_count"),
|
|
||||||
"duration": info.get("duration_string"),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
"""Search Bilibili.
|
|
||||||
|
|
||||||
Strategy:
|
|
||||||
1. Try yt-dlp bilisearch (works on local machines)
|
|
||||||
2. Fallback to Exa site:bilibili.com (works on servers)
|
|
||||||
"""
|
|
||||||
if not shutil.which("yt-dlp"):
|
|
||||||
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
|
|
||||||
|
|
||||||
limit = kwargs.get("limit", 5)
|
|
||||||
proxy = config.get("bilibili_proxy") if config else None
|
|
||||||
|
|
||||||
# Strategy 1: yt-dlp bilisearch
|
|
||||||
results = self._search_ytdlp(query, limit, proxy)
|
|
||||||
if results:
|
|
||||||
return results
|
|
||||||
|
|
||||||
# Strategy 2: Exa fallback (server-friendly)
|
|
||||||
results = self._search_exa(query, limit)
|
|
||||||
if results:
|
|
||||||
return results
|
|
||||||
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _search_ytdlp(self, query: str, limit: int, proxy: str = None) -> List[SearchResult]:
|
|
||||||
"""Search via yt-dlp bilisearch (needs local/Chinese IP)."""
|
|
||||||
cmd = [
|
|
||||||
"yt-dlp", "--dump-json", "--no-download",
|
|
||||||
f"bilisearch{limit}:{query}",
|
|
||||||
]
|
|
||||||
if proxy:
|
|
||||||
cmd += ["--proxy", proxy]
|
|
||||||
|
|
||||||
try:
|
|
||||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
|
|
||||||
if r.returncode != 0:
|
|
||||||
return []
|
|
||||||
results = []
|
|
||||||
for line in r.stdout.strip().split("\n"):
|
|
||||||
if not line.strip():
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
vid = d.get("id", "")
|
|
||||||
url = d.get("webpage_url", f"https://www.bilibili.com/video/av{vid}")
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=d.get("title", f"av{vid}"),
|
|
||||||
url=url,
|
|
||||||
snippet=f"👤 {d.get('uploader', '?')} · 👁 {d.get('view_count', '?')}",
|
|
||||||
extra={
|
|
||||||
"view_count": d.get("view_count"),
|
|
||||||
"uploader": d.get("uploader"),
|
|
||||||
"duration": d.get("duration_string"),
|
|
||||||
},
|
|
||||||
))
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
continue
|
|
||||||
return results
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _search_exa(self, query: str, limit: int) -> List[SearchResult]:
|
|
||||||
"""Fallback: search via Exa (site:bilibili.com). Works on any IP."""
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call",
|
|
||||||
f'exa.web_search_exa(query: "site:bilibili.com {query}", numResults: {limit})'],
|
|
||||||
capture_output=True, text=True, timeout=30,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
return []
|
|
||||||
|
|
||||||
results = []
|
|
||||||
# Parse mcporter output: Title: / Author: / URL: / Text: blocks
|
|
||||||
title, author, url = "", "", ""
|
|
||||||
for line in r.stdout.split("\n"):
|
|
||||||
if line.startswith("Title: "):
|
|
||||||
title = line[7:].strip()
|
|
||||||
elif line.startswith("Author: "):
|
|
||||||
author = line[8:].strip()
|
|
||||||
elif line.startswith("URL: "):
|
|
||||||
url = line[5:].strip()
|
|
||||||
if url and "bilibili.com" in url:
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title or url,
|
|
||||||
url=url,
|
|
||||||
snippet=f"👤 {author}" if author else "(via Exa search)",
|
|
||||||
))
|
|
||||||
title, author, url = "", "", ""
|
|
||||||
return results
|
|
||||||
except Exception:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _get_info(self, url: str, proxy: str = None) -> dict:
|
|
||||||
cmd = ["yt-dlp", "--dump-json", "--no-download", url]
|
|
||||||
if proxy:
|
|
||||||
cmd += ["--proxy", proxy]
|
|
||||||
try:
|
|
||||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
|
||||||
if r.returncode == 0:
|
|
||||||
return json.loads(r.stdout)
|
|
||||||
except (subprocess.TimeoutExpired, json.JSONDecodeError):
|
|
||||||
pass
|
|
||||||
return {}
|
|
||||||
|
|
||||||
def _get_subtitles(self, url: str, proxy: str = None) -> str:
|
|
||||||
import tempfile
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
|
||||||
cmd = [
|
|
||||||
"yt-dlp", "--write-sub", "--write-auto-sub",
|
|
||||||
"--sub-lang", "zh-Hans,zh,en",
|
|
||||||
"--skip-download", "--sub-format", "vtt",
|
|
||||||
"-o", f"{tmpdir}/%(id)s.%(ext)s", url,
|
|
||||||
]
|
|
||||||
if proxy:
|
|
||||||
cmd += ["--proxy", proxy]
|
|
||||||
try:
|
|
||||||
subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
|
||||||
for f in Path(tmpdir).glob("*.vtt"):
|
|
||||||
text = f.read_text(errors="replace")
|
|
||||||
lines = []
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if not line or line.startswith("WEBVTT") or "-->" in line or line.isdigit():
|
|
||||||
continue
|
|
||||||
if line not in lines[-1:]:
|
|
||||||
lines.append(line)
|
|
||||||
return "\n".join(lines)
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
pass
|
|
||||||
return ""
|
|
||||||
|
|||||||
@@ -1,183 +0,0 @@
|
|||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""Boss直聘 (BOSS Zhipin) — via mcp-bosszp (MCP) or Jina Reader fallback.
|
|
||||||
|
|
||||||
Backend: mcp-bosszp (161 stars, FastMCP + Playwright)
|
|
||||||
Swap to: any Boss直聘 access tool
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
import requests
|
|
||||||
|
|
||||||
|
|
||||||
def _mcporter_has_bosszhipin() -> bool:
|
|
||||||
"""Check if mcporter has Boss直聘 MCP configured."""
|
|
||||||
if not shutil.which("mcporter"):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
# Check for various possible config names
|
|
||||||
out = r.stdout.lower()
|
|
||||||
return "boss" in out or "zhipin" in out or "bosszhipin" in out
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _mcporter_call(expr: str, timeout: int = 30) -> str:
|
|
||||||
"""Call a Boss直聘 MCP tool via mcporter."""
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call", expr],
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
|
|
||||||
def _get_mcp_name() -> str:
|
|
||||||
"""Get the actual MCP server name configured in mcporter."""
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
for line in r.stdout.split("\n"):
|
|
||||||
line_lower = line.strip().lower()
|
|
||||||
for name in ["bosszhipin", "boss-zp", "bosszp", "boss"]:
|
|
||||||
if name in line_lower:
|
|
||||||
# Extract the actual server name
|
|
||||||
parts = line.strip().split()
|
|
||||||
if parts:
|
|
||||||
return parts[0]
|
|
||||||
return "bosszhipin"
|
|
||||||
except Exception:
|
|
||||||
return "bosszhipin"
|
|
||||||
|
|
||||||
|
|
||||||
class BossZhipinChannel(Channel):
|
|
||||||
name = "bosszhipin"
|
|
||||||
description = "Boss直聘职位搜索"
|
|
||||||
backends = ["mcp-bosszp", "Jina Reader"]
|
|
||||||
tier = 2
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
|
||||||
domain = urlparse(url).netloc.lower()
|
|
||||||
return "zhipin.com" in domain or "boss.com" in domain
|
|
||||||
|
|
||||||
def check(self, config=None):
|
|
||||||
if _mcporter_has_bosszhipin():
|
|
||||||
return "ok", "可搜索职位、向 HR 打招呼"
|
|
||||||
|
|
||||||
return "off", (
|
|
||||||
"可通过 Jina Reader 读取职位页面。完整功能需要:\n"
|
|
||||||
" 1. git clone https://github.com/mucsbr/mcp-bosszp.git\n"
|
|
||||||
" 2. cd mcp-bosszp && pip install -r requirements.txt && playwright install chromium\n"
|
|
||||||
" 3. python boss_zhipin_fastmcp_v2.py(启动后扫码登录)\n"
|
|
||||||
" 4. mcporter config add bosszhipin http://localhost:8000/mcp\n"
|
|
||||||
" 或用 Docker:docker-compose up -d\n"
|
|
||||||
" 详见 https://github.com/mucsbr/mcp-bosszp"
|
|
||||||
)
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
# Boss直聘 pages mostly work with Jina Reader
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
async def _read_jina(self, url: str) -> ReadResult:
|
|
||||||
"""Read Boss直聘 page via Jina Reader."""
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
f"https://r.jina.ai/{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
|
|
||||||
if len(text.strip()) < 50:
|
|
||||||
return ReadResult(
|
|
||||||
title="Boss直聘",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法读取此页面内容: {url}\n\n"
|
|
||||||
"提示:\n"
|
|
||||||
"- 安装 mcp-bosszp 可解锁职位搜索和自动打招呼\n"
|
|
||||||
"- 详见 https://github.com/mucsbr/mcp-bosszp"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="bosszhipin",
|
|
||||||
)
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=text[:100] if text else url,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="bosszhipin",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
return ReadResult(
|
|
||||||
title="Boss直聘",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法读取此 Boss直聘页面: {url}\n\n"
|
|
||||||
"提示:\n"
|
|
||||||
"- Boss直聘部分页面需要登录\n"
|
|
||||||
"- 安装 mcp-bosszp 可解锁完整功能\n"
|
|
||||||
"- 详见 https://github.com/mucsbr/mcp-bosszp"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="bosszhipin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
|
|
||||||
# Try MCP search first
|
|
||||||
if _mcporter_has_bosszhipin():
|
|
||||||
try:
|
|
||||||
return await self._search_mcp(query, limit, config)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Fallback to Exa
|
|
||||||
from agent_reach.channels.exa_search import ExaSearchChannel
|
|
||||||
exa = ExaSearchChannel()
|
|
||||||
return await exa.search(f"site:zhipin.com {query}", config=config, limit=limit)
|
|
||||||
|
|
||||||
async def _search_mcp(self, query: str, limit: int, config=None) -> List[SearchResult]:
|
|
||||||
"""Search Boss直聘 via MCP."""
|
|
||||||
server = _get_mcp_name()
|
|
||||||
try:
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'{server}.get_recommend_jobs_tool(page: 1)',
|
|
||||||
timeout=30,
|
|
||||||
)
|
|
||||||
return self._parse_jobs(out, limit)
|
|
||||||
except Exception:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_jobs(self, text: str, limit: int) -> List[SearchResult]:
|
|
||||||
"""Parse MCP job search output into SearchResults."""
|
|
||||||
results = []
|
|
||||||
try:
|
|
||||||
data = json.loads(text)
|
|
||||||
jobs = data if isinstance(data, list) else data.get("jobs", data.get("results", []))
|
|
||||||
for job in jobs[:limit]:
|
|
||||||
if isinstance(job, dict):
|
|
||||||
title = job.get("title") or job.get("jobName", "")
|
|
||||||
company = job.get("company") or job.get("brandName", "")
|
|
||||||
salary = job.get("salary") or job.get("salaryDesc", "")
|
|
||||||
url = job.get("url", "")
|
|
||||||
snippet = f"🏢 {company}" if company else ""
|
|
||||||
if salary:
|
|
||||||
snippet += f" · 💰 {salary}"
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title,
|
|
||||||
url=url,
|
|
||||||
snippet=snippet,
|
|
||||||
))
|
|
||||||
except (json.JSONDecodeError, KeyError):
|
|
||||||
pass
|
|
||||||
return results
|
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Douyin (抖音) — check if mcporter + douyin-mcp-server is available."""
|
||||||
|
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
|
||||||
|
class DouyinChannel(Channel):
|
||||||
|
name = "douyin"
|
||||||
|
description = "抖音短视频"
|
||||||
|
backends = ["douyin-mcp-server"]
|
||||||
|
tier = 2
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "douyin.com" in d or "iesdouyin.com" in d
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
mcporter = shutil.which("mcporter")
|
||||||
|
if not mcporter:
|
||||||
|
return "off", (
|
||||||
|
"需要 mcporter + douyin-mcp-server。安装步骤:\n"
|
||||||
|
" 1. npm install -g mcporter\n"
|
||||||
|
" 2. pip install douyin-mcp-server\n"
|
||||||
|
" 3. 启动服务(见下方说明)\n"
|
||||||
|
" 4. mcporter config add douyin http://localhost:18070/mcp\n"
|
||||||
|
" 详见 https://github.com/yzfly/douyin-mcp-server"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
[mcporter, "config", "list"], capture_output=True,
|
||||||
|
encoding="utf-8", errors="replace", timeout=5
|
||||||
|
)
|
||||||
|
if "douyin" not in r.stdout:
|
||||||
|
return "off", (
|
||||||
|
"mcporter 已装但抖音 MCP 未配置。运行:\n"
|
||||||
|
" pip install douyin-mcp-server\n"
|
||||||
|
" # 启动服务后:\n"
|
||||||
|
" mcporter config add douyin http://localhost:18070/mcp"
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return "off", "mcporter 连接异常"
|
||||||
|
# Verify MCP connectivity by listing available tools instead of
|
||||||
|
# calling with a hardcoded (invalid) share link that always fails.
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
[mcporter, "list", "douyin"],
|
||||||
|
capture_output=True, encoding="utf-8", errors="replace", timeout=15
|
||||||
|
)
|
||||||
|
if r.returncode == 0 and r.stdout.strip():
|
||||||
|
return "ok", "完整可用(视频解析、下载链接获取)"
|
||||||
|
return "warn", "MCP 已连接但工具列表为空,检查 douyin-mcp-server 服务是否在运行"
|
||||||
|
except Exception:
|
||||||
|
return "warn", "MCP 连接异常,检查 douyin-mcp-server 服务是否在运行"
|
||||||
@@ -1,110 +1,38 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Exa semantic search — via mcporter + Exa MCP server.
|
"""Exa Search — check if mcporter + Exa MCP is available."""
|
||||||
|
|
||||||
Backend: Exa MCP at mcp.exa.ai (OAuth, no API key needed)
|
|
||||||
Requires: mcporter CLI
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from .base import Channel, SearchResult
|
from .base import Channel
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class ExaSearchChannel(Channel):
|
class ExaSearchChannel(Channel):
|
||||||
name = "exa_search"
|
name = "exa_search"
|
||||||
description = "全网语义搜索(同时支持 Reddit/Twitter 搜索)"
|
description = "全网语义搜索"
|
||||||
backends = ["exa-mcp"]
|
backends = ["Exa via mcporter"]
|
||||||
tier = 1
|
tier = 0
|
||||||
|
|
||||||
def _mcporter_ok(self) -> bool:
|
|
||||||
if not shutil.which("mcporter"):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
return "exa" in r.stdout
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
def _call(self, expr: str, timeout: int = 30) -> str:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call", expr],
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
# ── Channel interface ──
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
return False # search-only
|
return False # Search-only channel
|
||||||
|
|
||||||
async def read(self, url: str, config=None):
|
|
||||||
raise NotImplementedError("Exa is a search engine, not a reader")
|
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if not shutil.which("mcporter"):
|
mcporter = shutil.which("mcporter")
|
||||||
|
if not mcporter:
|
||||||
return "off", (
|
return "off", (
|
||||||
"需要 mcporter。安装:npm install -g mcporter && "
|
"需要 mcporter + Exa MCP。安装:\n"
|
||||||
"mcporter config add exa https://mcp.exa.ai/mcp"
|
|
||||||
)
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
return "off", "mcporter 已装但 Exa 未配置。运行:mcporter config add exa https://mcp.exa.ai/mcp"
|
|
||||||
return "ok", "MCP 已连接,免 Key 直接可用(全网搜索 + Reddit + Twitter)"
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
raise ValueError(
|
|
||||||
"Exa 搜索需要 mcporter。安装:\n"
|
|
||||||
" npm install -g mcporter\n"
|
" npm install -g mcporter\n"
|
||||||
" mcporter config add exa https://mcp.exa.ai/mcp"
|
" mcporter config add exa https://mcp.exa.ai/mcp"
|
||||||
)
|
)
|
||||||
|
try:
|
||||||
limit = kwargs.get("limit", 5)
|
r = subprocess.run(
|
||||||
safe_q = query.replace('"', '\\"')
|
[mcporter, "config", "list"], capture_output=True,
|
||||||
out = self._call(
|
encoding="utf-8", errors="replace", timeout=5
|
||||||
f'exa.web_search_exa(query: "{safe_q}", numResults: {min(limit, 10)})',
|
)
|
||||||
timeout=30,
|
if "exa" in r.stdout.lower():
|
||||||
)
|
return "ok", "全网语义搜索可用(免费,无需 API Key)"
|
||||||
return self._parse_output(out, limit)
|
return "off", (
|
||||||
|
"mcporter 已装但 Exa 未配置。运行:\n"
|
||||||
# ── Parse mcporter text output ──
|
" mcporter config add exa https://mcp.exa.ai/mcp"
|
||||||
|
)
|
||||||
def _parse_output(self, text: str, limit: int) -> List[SearchResult]:
|
except Exception:
|
||||||
"""Parse mcporter's Title/URL/Text block format."""
|
return "off", "mcporter 连接异常"
|
||||||
results = []
|
|
||||||
cur = {}
|
|
||||||
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line.startswith("Title: "):
|
|
||||||
if cur.get("title"):
|
|
||||||
results.append(self._make_result(cur))
|
|
||||||
cur = {"title": line[7:]}
|
|
||||||
elif line.startswith("URL: "):
|
|
||||||
cur["url"] = line[5:]
|
|
||||||
elif line.startswith("Published Date: "):
|
|
||||||
cur["date"] = line[16:]
|
|
||||||
elif line.startswith("Text: "):
|
|
||||||
cur["text"] = line[6:]
|
|
||||||
elif "text" in cur and line:
|
|
||||||
cur["text"] += " " + line
|
|
||||||
|
|
||||||
if cur.get("title"):
|
|
||||||
results.append(self._make_result(cur))
|
|
||||||
|
|
||||||
return results[:limit]
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _make_result(d: dict) -> SearchResult:
|
|
||||||
return SearchResult(
|
|
||||||
title=d.get("title", ""),
|
|
||||||
url=d.get("url", ""),
|
|
||||||
snippet=d.get("text", "")[:500],
|
|
||||||
date=d.get("date", ""),
|
|
||||||
score=0,
|
|
||||||
)
|
|
||||||
|
|||||||
+13
-120
@@ -1,16 +1,9 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""GitHub — via gh CLI.
|
"""GitHub — check if gh CLI is available."""
|
||||||
|
|
||||||
Backend: gh CLI (https://cli.github.com)
|
|
||||||
Swap to: GitHub REST API
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class GitHubChannel(Channel):
|
class GitHubChannel(Channel):
|
||||||
@@ -19,121 +12,21 @@ class GitHubChannel(Channel):
|
|||||||
backends = ["gh CLI"]
|
backends = ["gh CLI"]
|
||||||
tier = 0
|
tier = 0
|
||||||
|
|
||||||
def _gh(self, args: list, timeout: int = 15) -> str:
|
|
||||||
r = subprocess.run(
|
|
||||||
["gh"] + args,
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
def _gh_json(self, args: list, timeout: int = 15) -> dict:
|
|
||||||
return json.loads(self._gh(args + ["--json"], timeout))
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
return "github.com" in urlparse(url).netloc.lower()
|
return "github.com" in urlparse(url).netloc.lower()
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if not shutil.which("gh"):
|
gh = shutil.which("gh")
|
||||||
return "warn", "gh CLI 未安装。安装:https://cli.github.com 。公开仓库仍可通过 Jina Reader 读取"
|
if not gh:
|
||||||
|
return "warn", "gh CLI 未安装。安装:https://cli.github.com"
|
||||||
try:
|
try:
|
||||||
self._gh(["auth", "status"], timeout=5)
|
r = subprocess.run(
|
||||||
return "ok", "完整可用(读取、搜索、Fork、Issue、PR 等)"
|
[gh, "auth", "status"],
|
||||||
except Exception:
|
capture_output=True, encoding="utf-8", errors="replace", timeout=5
|
||||||
return "ok", "gh CLI 已装但未认证。运行 gh auth login 可解锁完整功能"
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
if not shutil.which("gh"):
|
|
||||||
# Fallback to Jina Reader for public repos
|
|
||||||
from agent_reach.channels.web import WebChannel
|
|
||||||
return await WebChannel().read(url, config)
|
|
||||||
|
|
||||||
path = urlparse(url).path.strip("/").split("/")
|
|
||||||
if len(path) < 2:
|
|
||||||
from agent_reach.channels.web import WebChannel
|
|
||||||
return await WebChannel().read(url, config)
|
|
||||||
|
|
||||||
owner, repo = path[0], path[1]
|
|
||||||
|
|
||||||
# Issues / PRs
|
|
||||||
if len(path) >= 4 and path[2] in ("issues", "pull"):
|
|
||||||
return await self._read_issue(owner, repo, path[3], url)
|
|
||||||
|
|
||||||
# Repo
|
|
||||||
return await self._read_repo(owner, repo, url)
|
|
||||||
|
|
||||||
async def _read_repo(self, owner: str, repo: str, url: str) -> ReadResult:
|
|
||||||
slug = f"{owner}/{repo}"
|
|
||||||
try:
|
|
||||||
# Get repo info
|
|
||||||
info = self._gh(["repo", "view", slug])
|
|
||||||
# Get README
|
|
||||||
try:
|
|
||||||
readme = self._gh(
|
|
||||||
["api", f"repos/{slug}/readme", "--jq", ".content"],
|
|
||||||
timeout=10,
|
|
||||||
)
|
|
||||||
import base64
|
|
||||||
readme_text = base64.b64decode(readme).decode("utf-8", errors="replace")
|
|
||||||
except Exception:
|
|
||||||
readme_text = ""
|
|
||||||
|
|
||||||
content = readme_text or info
|
|
||||||
return ReadResult(
|
|
||||||
title=slug, content=content, url=url,
|
|
||||||
author=owner, platform="github",
|
|
||||||
)
|
)
|
||||||
|
if r.returncode == 0:
|
||||||
|
return "ok", "完整可用(读取、搜索、Fork、Issue、PR 等)"
|
||||||
|
return "warn", "gh CLI 已安装但未认证。运行 gh auth login 可解锁完整功能"
|
||||||
except Exception:
|
except Exception:
|
||||||
from agent_reach.channels.web import WebChannel
|
return "warn", "gh CLI 状态检查失败,运行 gh auth status 查看详情"
|
||||||
return await WebChannel().read(url)
|
|
||||||
|
|
||||||
async def _read_issue(self, owner: str, repo: str, num: str, url: str) -> ReadResult:
|
|
||||||
slug = f"{owner}/{repo}"
|
|
||||||
try:
|
|
||||||
out = self._gh(["issue", "view", num, "-R", slug])
|
|
||||||
return ReadResult(
|
|
||||||
title=f"{slug}#{num}", content=out, url=url,
|
|
||||||
platform="github",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
# Might be a PR
|
|
||||||
try:
|
|
||||||
out = self._gh(["pr", "view", num, "-R", slug])
|
|
||||||
return ReadResult(
|
|
||||||
title=f"{slug}#{num}", content=out, url=url,
|
|
||||||
platform="github",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
from agent_reach.channels.web import WebChannel
|
|
||||||
return await WebChannel().read(url)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
if not shutil.which("gh"):
|
|
||||||
raise ValueError("GitHub search requires gh CLI. Install: https://cli.github.com")
|
|
||||||
|
|
||||||
language = kwargs.get("language")
|
|
||||||
limit = kwargs.get("limit", 5)
|
|
||||||
|
|
||||||
args = ["search", "repos", query, "--sort", "stars", f"--limit={limit}"]
|
|
||||||
if language:
|
|
||||||
args += [f"--language={language}"]
|
|
||||||
|
|
||||||
out = self._gh(args, timeout=15)
|
|
||||||
results = []
|
|
||||||
for line in out.strip().split("\n"):
|
|
||||||
if not line.strip():
|
|
||||||
continue
|
|
||||||
parts = line.split("\t")
|
|
||||||
if len(parts) >= 1:
|
|
||||||
slug = parts[0].strip()
|
|
||||||
desc = parts[1].strip() if len(parts) > 1 else ""
|
|
||||||
stars = parts[3].strip() if len(parts) > 3 else ""
|
|
||||||
lang = parts[5].strip() if len(parts) > 5 else ""
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=slug,
|
|
||||||
url=f"https://github.com/{slug}",
|
|
||||||
snippet=desc,
|
|
||||||
extra={"stars": stars, "language": lang},
|
|
||||||
))
|
|
||||||
return results
|
|
||||||
|
|||||||
@@ -1,248 +0,0 @@
|
|||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""Instagram — via instaloader (free, open source).
|
|
||||||
|
|
||||||
Backend: instaloader (9.8K stars, Python CLI + library)
|
|
||||||
Swap to: any Instagram access tool
|
|
||||||
"""
|
|
||||||
|
|
||||||
import re
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class InstagramChannel(Channel):
|
|
||||||
name = "instagram"
|
|
||||||
description = "Instagram 帖子和 Profile"
|
|
||||||
backends = ["instaloader"]
|
|
||||||
tier = 2 # Needs login for full access
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
|
||||||
domain = urlparse(url).netloc.lower()
|
|
||||||
return "instagram.com" in domain or "instagr.am" in domain
|
|
||||||
|
|
||||||
def check(self, config=None):
|
|
||||||
# Check both CLI and Python module
|
|
||||||
has_cli = shutil.which("instaloader")
|
|
||||||
has_module = False
|
|
||||||
try:
|
|
||||||
import instaloader
|
|
||||||
has_module = True
|
|
||||||
except ImportError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
if not has_cli and not has_module:
|
|
||||||
return "off", (
|
|
||||||
"需要安装 instaloader:pip install instaloader\n"
|
|
||||||
" 安装后可读取 Instagram 帖子和 Profile\n"
|
|
||||||
" 登录: agent-reach configure instagram-cookies \"sessionid=xxx; csrftoken=yyy; ...\""
|
|
||||||
)
|
|
||||||
|
|
||||||
# Check if cookies are configured
|
|
||||||
cookie_file = Path.home() / ".agent-reach" / "instagram-cookies.txt"
|
|
||||||
if cookie_file.exists():
|
|
||||||
return "ok", "已登录,可读取 Instagram 帖子和 Profile"
|
|
||||||
return "ok", "可读取公开帖子和 Profile。登录可访问更多内容:\n agent-reach configure instagram-cookies \"sessionid=xxx; csrftoken=yyy; ...\""
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
# Try instaloader (module or CLI)
|
|
||||||
try:
|
|
||||||
import instaloader
|
|
||||||
return await self._read_instaloader(url, config)
|
|
||||||
except ImportError:
|
|
||||||
pass
|
|
||||||
# Fallback: Jina Reader
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
async def _read_instaloader(self, url: str, config=None) -> ReadResult:
|
|
||||||
"""Read Instagram content using instaloader Python API."""
|
|
||||||
import asyncio
|
|
||||||
import concurrent.futures
|
|
||||||
|
|
||||||
def _sync_read():
|
|
||||||
import instaloader
|
|
||||||
L = instaloader.Instaloader(
|
|
||||||
download_pictures=False,
|
|
||||||
download_videos=False,
|
|
||||||
download_video_thumbnails=False,
|
|
||||||
download_geotags=False,
|
|
||||||
download_comments=False,
|
|
||||||
save_metadata=False,
|
|
||||||
compress_json=False,
|
|
||||||
max_connection_attempts=1, # Don't retry on rate limit
|
|
||||||
)
|
|
||||||
|
|
||||||
# Try to load session: cookie file > saved session
|
|
||||||
cookie_file = Path.home() / ".agent-reach" / "instagram-cookies.txt"
|
|
||||||
if cookie_file.exists():
|
|
||||||
try:
|
|
||||||
cookie_str = cookie_file.read_text().strip()
|
|
||||||
cookies = {}
|
|
||||||
for part in cookie_str.split(";"):
|
|
||||||
part = part.strip()
|
|
||||||
if "=" in part:
|
|
||||||
k, v = part.split("=", 1)
|
|
||||||
cookies[k.strip()] = v.strip()
|
|
||||||
if "sessionid" in cookies and "csrftoken" in cookies:
|
|
||||||
# Extract username from ds_user_id or use generic
|
|
||||||
username = cookies.get("ds_user_id", "user")
|
|
||||||
L.context.load_session(username, cookies)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
elif config and config.get("instagram_username"):
|
|
||||||
try:
|
|
||||||
L.load_session_from_file(config.get("instagram_username"))
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
path = urlparse(url).path.strip("/")
|
|
||||||
|
|
||||||
if "/p/" in url or "/reel/" in url:
|
|
||||||
return self._read_post_sync(L, url, path)
|
|
||||||
else:
|
|
||||||
return self._read_profile_sync(L, url, path)
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Run with 15s timeout to avoid instaloader's 30-min retry
|
|
||||||
loop = asyncio.get_event_loop()
|
|
||||||
with concurrent.futures.ThreadPoolExecutor() as pool:
|
|
||||||
result = await asyncio.wait_for(
|
|
||||||
loop.run_in_executor(pool, _sync_read),
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
return result
|
|
||||||
except (asyncio.TimeoutError, Exception):
|
|
||||||
# Any error or timeout → Jina fallback
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
def _read_post_sync(self, L, url: str, path: str) -> ReadResult:
|
|
||||||
"""Read a single Instagram post (sync, runs in executor)."""
|
|
||||||
import instaloader
|
|
||||||
|
|
||||||
# Extract shortcode from URL
|
|
||||||
match = re.search(r"/(?:p|reel)/([A-Za-z0-9_-]+)", url)
|
|
||||||
if not match:
|
|
||||||
raise ValueError("Cannot extract shortcode from URL")
|
|
||||||
|
|
||||||
shortcode = match.group(1)
|
|
||||||
try:
|
|
||||||
post = instaloader.Post.from_shortcode(L.context, shortcode)
|
|
||||||
|
|
||||||
lines = []
|
|
||||||
if post.caption:
|
|
||||||
lines.append(post.caption)
|
|
||||||
lines.append("")
|
|
||||||
lines.append(f"👤 @{post.owner_username}")
|
|
||||||
lines.append(f"❤️ {post.likes} likes")
|
|
||||||
if post.comments:
|
|
||||||
lines.append(f"💬 {post.comments} comments")
|
|
||||||
lines.append(f"📅 {post.date_utc.strftime('%Y-%m-%d %H:%M')}")
|
|
||||||
if post.location:
|
|
||||||
lines.append(f"📍 {post.location}")
|
|
||||||
if post.hashtags:
|
|
||||||
lines.append(f"#️⃣ {' '.join('#' + h for h in post.hashtags)}")
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=f"@{post.owner_username}: {(post.caption or '')[:80]}",
|
|
||||||
content="\n".join(lines),
|
|
||||||
url=url,
|
|
||||||
author=f"@{post.owner_username}",
|
|
||||||
date=post.date_utc.strftime("%Y-%m-%d"),
|
|
||||||
platform="instagram",
|
|
||||||
extra={"likes": post.likes, "comments": post.comments},
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
raise # Let executor timeout handle fallback
|
|
||||||
|
|
||||||
def _read_profile_sync(self, L, url: str, path: str) -> ReadResult:
|
|
||||||
"""Read an Instagram profile (sync, runs in executor)."""
|
|
||||||
import instaloader
|
|
||||||
|
|
||||||
# Extract username from path
|
|
||||||
username = path.split("/")[0] if path else ""
|
|
||||||
if not username or username in ("p", "reel", "stories", "explore"):
|
|
||||||
raise ValueError("Cannot extract username from URL")
|
|
||||||
|
|
||||||
try:
|
|
||||||
profile = instaloader.Profile.from_username(L.context, username)
|
|
||||||
|
|
||||||
lines = []
|
|
||||||
lines.append(f"👤 {profile.full_name} (@{profile.username})")
|
|
||||||
if profile.biography:
|
|
||||||
lines.append(f"📝 {profile.biography}")
|
|
||||||
if profile.external_url:
|
|
||||||
lines.append(f"🔗 {profile.external_url}")
|
|
||||||
lines.append("")
|
|
||||||
lines.append(f"📊 {profile.mediacount} posts · "
|
|
||||||
f"{profile.followers} followers · "
|
|
||||||
f"{profile.followees} following")
|
|
||||||
if profile.is_verified:
|
|
||||||
lines.append("✅ Verified")
|
|
||||||
if profile.is_business_account and profile.business_category_name:
|
|
||||||
lines.append(f"🏢 {profile.business_category_name}")
|
|
||||||
|
|
||||||
# Get recent posts (up to 5)
|
|
||||||
lines.append("")
|
|
||||||
lines.append("📸 Recent posts:")
|
|
||||||
count = 0
|
|
||||||
for post in profile.get_posts():
|
|
||||||
if count >= 5:
|
|
||||||
break
|
|
||||||
caption = (post.caption or "")[:100].replace("\n", " ")
|
|
||||||
lines.append(f" • ❤️{post.likes} | {post.date_utc.strftime('%m-%d')} | {caption}")
|
|
||||||
count += 1
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=f"{profile.full_name} (@{profile.username}) - Instagram",
|
|
||||||
content="\n".join(lines),
|
|
||||||
url=url,
|
|
||||||
author=f"@{profile.username}",
|
|
||||||
platform="instagram",
|
|
||||||
extra={
|
|
||||||
"followers": profile.followers,
|
|
||||||
"posts": profile.mediacount,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
raise # Let executor timeout handle fallback
|
|
||||||
|
|
||||||
async def _read_jina(self, url: str) -> ReadResult:
|
|
||||||
"""Fallback: use Jina Reader."""
|
|
||||||
import requests
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
f"https://r.jina.ai/{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
return ReadResult(
|
|
||||||
title=text[:100] if text else url,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="instagram",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
return ReadResult(
|
|
||||||
title="Instagram",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法读取此 Instagram 内容: {url}\n\n"
|
|
||||||
"提示:\n"
|
|
||||||
"- 确保 URL 正确\n"
|
|
||||||
"- 安装 instaloader: pip install instaloader\n"
|
|
||||||
"- 登录以访问更多内容: instaloader --login YOUR_USERNAME"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="instagram",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
"""Search Instagram via Exa."""
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
from agent_reach.channels.exa_search import ExaSearchChannel
|
|
||||||
exa = ExaSearchChannel()
|
|
||||||
return await exa.search(f"site:instagram.com {query}", config=config, limit=limit)
|
|
||||||
@@ -1,268 +1,41 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""LinkedIn — via linkedin-scraper-mcp (MCP) or Jina Reader fallback.
|
"""LinkedIn — check if linkedin-scraper-mcp is available."""
|
||||||
|
|
||||||
Backend: linkedin-scraper-mcp (916 stars, Patchright browser automation)
|
|
||||||
Swap to: any LinkedIn access tool
|
|
||||||
"""
|
|
||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
import requests
|
|
||||||
|
|
||||||
|
|
||||||
def _mcporter_has_linkedin() -> bool:
|
|
||||||
"""Check if mcporter has linkedin MCP configured."""
|
|
||||||
if not shutil.which("mcporter"):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
return "linkedin" in r.stdout.lower()
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _mcporter_call(expr: str, timeout: int = 30) -> str:
|
|
||||||
"""Call a LinkedIn MCP tool via mcporter."""
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call", expr],
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
|
|
||||||
class LinkedInChannel(Channel):
|
class LinkedInChannel(Channel):
|
||||||
name = "linkedin"
|
name = "linkedin"
|
||||||
description = "LinkedIn 个人/公司 Profile 和职位"
|
description = "LinkedIn 职业社交"
|
||||||
backends = ["linkedin-scraper-mcp", "Jina Reader"]
|
backends = ["linkedin-scraper-mcp", "Jina Reader"]
|
||||||
tier = 2
|
tier = 2
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
domain = urlparse(url).netloc.lower()
|
from urllib.parse import urlparse
|
||||||
return "linkedin.com" in domain
|
return "linkedin.com" in urlparse(url).netloc.lower()
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if _mcporter_has_linkedin():
|
mcporter = shutil.which("mcporter")
|
||||||
return "ok", "完整可用(Profile、公司、职位搜索)"
|
if not mcporter:
|
||||||
|
return "off", (
|
||||||
# Check if linkedin-scraper-mcp is installed as CLI
|
"基本内容可通过 Jina Reader 读取。完整功能需要:\n"
|
||||||
if shutil.which("linkedin-scraper-mcp"):
|
" pip install linkedin-scraper-mcp\n"
|
||||||
return "warn", (
|
" mcporter config add linkedin http://localhost:3000/mcp\n"
|
||||||
"linkedin-scraper-mcp 已安装但未接入 mcporter。运行:\n"
|
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
|
||||||
" 1. linkedin-scraper-mcp --login(在有浏览器的机器上登录)\n"
|
|
||||||
" 2. linkedin-scraper-mcp --transport streamable-http --port 8001\n"
|
|
||||||
" 3. mcporter config add linkedin http://localhost:8001/mcp"
|
|
||||||
)
|
)
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
[mcporter, "config", "list"], capture_output=True,
|
||||||
|
encoding="utf-8", errors="replace", timeout=5
|
||||||
|
)
|
||||||
|
if "linkedin" in r.stdout.lower():
|
||||||
|
return "ok", "完整可用(Profile、公司、职位搜索)"
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
return "off", (
|
return "off", (
|
||||||
"可通过 Jina Reader 读取部分内容。完整功能需要:\n"
|
"mcporter 已装但 LinkedIn MCP 未配置。运行:\n"
|
||||||
" 1. pip install linkedin-scraper-mcp\n"
|
" pip install linkedin-scraper-mcp\n"
|
||||||
" 2. linkedin-scraper-mcp --login(在有浏览器的机器上登录)\n"
|
" mcporter config add linkedin http://localhost:3000/mcp"
|
||||||
" 3. linkedin-scraper-mcp --transport streamable-http --port 8001\n"
|
|
||||||
" 4. mcporter config add linkedin http://localhost:8001/mcp\n"
|
|
||||||
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
path = urlparse(url).path.strip("/")
|
|
||||||
|
|
||||||
# Try MCP first
|
|
||||||
if _mcporter_has_linkedin():
|
|
||||||
try:
|
|
||||||
if "/in/" in url:
|
|
||||||
return await self._read_profile_mcp(url)
|
|
||||||
elif "/company/" in url:
|
|
||||||
return await self._read_company_mcp(url)
|
|
||||||
elif "/jobs/view/" in url:
|
|
||||||
return await self._read_job_mcp(url)
|
|
||||||
except Exception:
|
|
||||||
pass # Fall through to Jina
|
|
||||||
|
|
||||||
# Fallback: Jina Reader
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
async def _read_profile_mcp(self, url: str) -> ReadResult:
|
|
||||||
"""Read a LinkedIn profile via MCP."""
|
|
||||||
import re
|
|
||||||
# Extract username from URL: /in/username/
|
|
||||||
match = re.search(r"/in/([^/]+)", url)
|
|
||||||
if not match:
|
|
||||||
return await self._read_jina(url)
|
|
||||||
username = match.group(1)
|
|
||||||
safe_username = username.replace('"', '\\"')
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.get_person_profile(linkedin_username: "{safe_username}")',
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
return ReadResult(
|
|
||||||
title=self._extract_title(out) or f"LinkedIn Profile - {username}",
|
|
||||||
content=out.strip(),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_company_mcp(self, url: str) -> ReadResult:
|
|
||||||
"""Read a LinkedIn company page via MCP."""
|
|
||||||
import re
|
|
||||||
# Extract company name from URL: /company/name/
|
|
||||||
match = re.search(r"/company/([^/]+)", url)
|
|
||||||
if not match:
|
|
||||||
return await self._read_jina(url)
|
|
||||||
company = match.group(1)
|
|
||||||
safe_company = company.replace('"', '\\"')
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.get_company_profile(company_name: "{safe_company}")',
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
return ReadResult(
|
|
||||||
title=self._extract_title(out) or "LinkedIn Company",
|
|
||||||
content=out.strip(),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_job_mcp(self, url: str) -> ReadResult:
|
|
||||||
"""Read a LinkedIn job posting via MCP."""
|
|
||||||
import re
|
|
||||||
match = re.search(r"/jobs/view/(\d+)", url)
|
|
||||||
if not match:
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
job_id = match.group(1)
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.get_job_details(job_id: "{job_id}")',
|
|
||||||
timeout=30,
|
|
||||||
)
|
|
||||||
return ReadResult(
|
|
||||||
title=self._extract_title(out) or f"LinkedIn Job {job_id}",
|
|
||||||
content=out.strip(),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_jina(self, url: str) -> ReadResult:
|
|
||||||
"""Fallback: use Jina Reader."""
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
f"https://r.jina.ai/{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
|
|
||||||
# Check if content is usable
|
|
||||||
if len(text.strip()) < 100 or "Sign in" in text[:200]:
|
|
||||||
return ReadResult(
|
|
||||||
title="LinkedIn",
|
|
||||||
content=(
|
|
||||||
f"⚠️ LinkedIn 页面需要登录才能完整查看。\n\n"
|
|
||||||
f"URL: {url}\n\n"
|
|
||||||
"完整功能需安装 linkedin-scraper-mcp:\n"
|
|
||||||
" pip install linkedin-scraper-mcp\n"
|
|
||||||
" uvx linkedin-scraper-mcp --login\n"
|
|
||||||
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=text[:100] if text else url,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
return ReadResult(
|
|
||||||
title="LinkedIn",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法读取此 LinkedIn 页面: {url}\n\n"
|
|
||||||
"提示:\n"
|
|
||||||
"- LinkedIn 需要登录才能查看大部分内容\n"
|
|
||||||
"- 安装 linkedin-scraper-mcp 解锁完整功能\n"
|
|
||||||
"- 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
|
|
||||||
# Try MCP search first
|
|
||||||
if _mcporter_has_linkedin():
|
|
||||||
try:
|
|
||||||
return await self._search_mcp(query, limit)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Fallback to Exa
|
|
||||||
from agent_reach.channels.exa_search import ExaSearchChannel
|
|
||||||
exa = ExaSearchChannel()
|
|
||||||
return await exa.search(f"site:linkedin.com {query}", config=config, limit=limit)
|
|
||||||
|
|
||||||
async def _search_mcp(self, query: str, limit: int) -> List[SearchResult]:
|
|
||||||
"""Search LinkedIn via MCP."""
|
|
||||||
safe_q = query.replace('"', '\\"')
|
|
||||||
# Try job search first (most common use case)
|
|
||||||
try:
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.search_jobs(keywords: "{safe_q}")',
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
results = self._parse_search_results(out, "job")
|
|
||||||
if results:
|
|
||||||
return results[:limit]
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Try people search
|
|
||||||
try:
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.search_people(keywords: "{safe_q}")',
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
results = self._parse_search_results(out, "people")
|
|
||||||
if results:
|
|
||||||
return results
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_search_results(self, text: str, result_type: str) -> List[SearchResult]:
|
|
||||||
"""Parse MCP search output into SearchResults."""
|
|
||||||
import json
|
|
||||||
results = []
|
|
||||||
try:
|
|
||||||
data = json.loads(text)
|
|
||||||
items = data if isinstance(data, list) else data.get("results", data.get("jobs", []))
|
|
||||||
for item in items:
|
|
||||||
if isinstance(item, dict):
|
|
||||||
title = item.get("title") or item.get("name") or item.get("headline", "")
|
|
||||||
url = item.get("url") or item.get("link", "")
|
|
||||||
snippet = item.get("description") or item.get("company", "")
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title,
|
|
||||||
url=url,
|
|
||||||
snippet=snippet[:200] if snippet else "",
|
|
||||||
))
|
|
||||||
except (json.JSONDecodeError, KeyError):
|
|
||||||
# Try line-by-line parsing
|
|
||||||
pass
|
|
||||||
return results
|
|
||||||
|
|
||||||
def _extract_title(self, text: str) -> str:
|
|
||||||
"""Extract a title from MCP output."""
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line and not line.startswith(("{", "[", "#", "http")):
|
|
||||||
return line[:80]
|
|
||||||
return ""
|
|
||||||
|
|||||||
+18
-165
@@ -1,178 +1,31 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Reddit — via Reddit JSON API + optional proxy.
|
"""Reddit — search and read via rdt-cli (public-clis/rdt-cli)."""
|
||||||
|
|
||||||
Backend: Reddit public JSON API (append .json to any URL)
|
import shutil
|
||||||
Swap to: any Reddit access method
|
import subprocess
|
||||||
"""
|
from .base import Channel
|
||||||
|
|
||||||
import os
|
|
||||||
import requests
|
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult
|
|
||||||
|
|
||||||
|
|
||||||
class RedditChannel(Channel):
|
class RedditChannel(Channel):
|
||||||
name = "reddit"
|
name = "reddit"
|
||||||
description = "Reddit 帖子和评论"
|
description = "Reddit 帖子和评论"
|
||||||
backends = ["Reddit JSON API"]
|
backends = ["rdt-cli"]
|
||||||
tier = 2
|
tier = 0
|
||||||
|
|
||||||
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
domain = urlparse(url).netloc.lower()
|
from urllib.parse import urlparse
|
||||||
return "reddit.com" in domain or "redd.it" in domain
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "reddit.com" in d or "redd.it" in d
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
proxy = config.get("reddit_proxy") if config else None
|
rdt = shutil.which("rdt")
|
||||||
has_bot = bool(os.environ.get("REDDIT_CLIENT_ID"))
|
if rdt:
|
||||||
if proxy and has_bot:
|
return "ok", (
|
||||||
return "ok", "完整可用(代理 + OAuth Bot)"
|
"rdt-cli 可用(搜索帖子、阅读全文、查看评论,无需登录)"
|
||||||
elif proxy:
|
|
||||||
return "ok", "代理已配置,可读取帖子。配置 REDDIT_CLIENT_ID/SECRET 可解锁高级搜索和发帖"
|
|
||||||
elif has_bot:
|
|
||||||
return "warn", "OAuth Bot 已配置,但服务器直连可能被封。配个代理更稳定:agent-reach configure proxy URL"
|
|
||||||
else:
|
|
||||||
return "off", "搜索用 Exa 免费可用。读帖子需配个代理:agent-reach configure proxy URL"
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
proxy = config.get("reddit_proxy") if config else None
|
|
||||||
proxies = {"http": proxy, "https": proxy} if proxy else None
|
|
||||||
|
|
||||||
# Clean URL: remove query params, trailing slash, then add .json
|
|
||||||
parsed = urlparse(url)
|
|
||||||
clean_path = parsed.path.rstrip("/")
|
|
||||||
# Remove trailing .json if already present (avoid double .json)
|
|
||||||
if clean_path.endswith(".json"):
|
|
||||||
clean_path = clean_path[:-5]
|
|
||||||
json_url = f"https://www.reddit.com{clean_path}.json"
|
|
||||||
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
json_url,
|
|
||||||
headers={"User-Agent": self.USER_AGENT},
|
|
||||||
proxies=proxies,
|
|
||||||
params={"limit": 50},
|
|
||||||
timeout=15,
|
|
||||||
)
|
)
|
||||||
resp.raise_for_status()
|
return "off", (
|
||||||
except requests.exceptions.HTTPError as e:
|
"需要安装 rdt-cli:\n"
|
||||||
status = e.response.status_code if e.response is not None else 0
|
" pipx install rdt-cli\n"
|
||||||
if status in (403, 429):
|
"或:\n"
|
||||||
return ReadResult(
|
" uv tool install rdt-cli"
|
||||||
title="Reddit",
|
|
||||||
content="⚠️ Reddit blocked this request (403 Forbidden). "
|
|
||||||
"Reddit blocks most server IPs.\n"
|
|
||||||
"Fix: agent-reach configure proxy http://user:pass@ip:port\n"
|
|
||||||
"Cheap option: https://www.webshare.io ($1/month)\n\n"
|
|
||||||
"Alternatively, search Reddit via Exa (free, no proxy needed): "
|
|
||||||
"agent-reach search-reddit \"your query\"",
|
|
||||||
url=url,
|
|
||||||
platform="reddit",
|
|
||||||
)
|
|
||||||
raise
|
|
||||||
|
|
||||||
data = resp.json()
|
|
||||||
|
|
||||||
# Subreddit listing page: /r/sub/, /r/sub/hot, /r/sub/new, /r/sub/top
|
|
||||||
if isinstance(data, dict) and data.get("kind") == "Listing":
|
|
||||||
return self._parse_listing(data, url)
|
|
||||||
|
|
||||||
if isinstance(data, list) and len(data) >= 1:
|
|
||||||
# Post page: [post_listing, comments_listing]
|
|
||||||
post = data[0]["data"]["children"][0]["data"]
|
|
||||||
title = post.get("title", "")
|
|
||||||
author = post.get("author", "")
|
|
||||||
selftext = post.get("selftext", "")
|
|
||||||
score = post.get("score", 0)
|
|
||||||
subreddit = post.get("subreddit", "")
|
|
||||||
|
|
||||||
# Extract comments
|
|
||||||
comments_text = ""
|
|
||||||
if len(data) >= 2:
|
|
||||||
comments_text = self._extract_comments(data[1])
|
|
||||||
|
|
||||||
content = selftext
|
|
||||||
if comments_text:
|
|
||||||
content += f"\n\n---\n## Comments\n{comments_text}"
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=title,
|
|
||||||
content=content,
|
|
||||||
url=url,
|
|
||||||
author=f"u/{author}",
|
|
||||||
platform="reddit",
|
|
||||||
extra={"subreddit": subreddit, "score": score},
|
|
||||||
)
|
|
||||||
|
|
||||||
raise ValueError(f"Could not parse Reddit response for: {url}")
|
|
||||||
|
|
||||||
def _parse_listing(self, data: dict, url: str) -> ReadResult:
|
|
||||||
"""Parse a subreddit listing (hot/new/top/rising)."""
|
|
||||||
children = data.get("data", {}).get("children", [])
|
|
||||||
|
|
||||||
# Extract subreddit name and sort from URL
|
|
||||||
parsed = urlparse(url)
|
|
||||||
path_parts = [p for p in parsed.path.strip("/").split("/") if p]
|
|
||||||
subreddit = path_parts[1] if len(path_parts) >= 2 else "reddit"
|
|
||||||
sort_type = path_parts[2] if len(path_parts) >= 3 else "hot"
|
|
||||||
|
|
||||||
lines = []
|
|
||||||
for i, child in enumerate(children, 1):
|
|
||||||
if child.get("kind") != "t3":
|
|
||||||
continue
|
|
||||||
post = child.get("data", {})
|
|
||||||
title = post.get("title", "")
|
|
||||||
author = post.get("author", "")
|
|
||||||
score = post.get("score", 0)
|
|
||||||
num_comments = post.get("num_comments", 0)
|
|
||||||
permalink = post.get("permalink", "")
|
|
||||||
post_url = post.get("url", "")
|
|
||||||
is_self = post.get("is_self", False)
|
|
||||||
|
|
||||||
lines.append(f"### {i}. {title}")
|
|
||||||
lines.append(f"👤 u/{author} · ⬆ {score} · 💬 {num_comments}")
|
|
||||||
if not is_self and post_url:
|
|
||||||
lines.append(f"🔗 {post_url}")
|
|
||||||
lines.append(f"📎 https://www.reddit.com{permalink}")
|
|
||||||
# Add selftext preview (first 200 chars)
|
|
||||||
selftext = post.get("selftext", "")
|
|
||||||
if selftext:
|
|
||||||
preview = selftext[:200].replace("\n", " ")
|
|
||||||
if len(selftext) > 200:
|
|
||||||
preview += "..."
|
|
||||||
lines.append(f"> {preview}")
|
|
||||||
lines.append("")
|
|
||||||
|
|
||||||
content = "\n".join(lines) if lines else "No posts found."
|
|
||||||
return ReadResult(
|
|
||||||
title=f"r/{subreddit} — {sort_type}",
|
|
||||||
content=content,
|
|
||||||
url=url,
|
|
||||||
platform="reddit",
|
|
||||||
extra={"subreddit": subreddit, "sort": sort_type, "count": len(children)},
|
|
||||||
)
|
)
|
||||||
|
|
||||||
def _extract_comments(self, comments_data: dict, depth: int = 0, max_depth: int = 3) -> str:
|
|
||||||
"""Recursively extract comments."""
|
|
||||||
lines = []
|
|
||||||
children = comments_data.get("data", {}).get("children", [])
|
|
||||||
|
|
||||||
for child in children:
|
|
||||||
if child.get("kind") != "t1":
|
|
||||||
continue
|
|
||||||
data = child.get("data", {})
|
|
||||||
author = data.get("author", "[deleted]")
|
|
||||||
body = data.get("body", "")
|
|
||||||
score = data.get("score", 0)
|
|
||||||
indent = " " * depth
|
|
||||||
|
|
||||||
lines.append(f"{indent}**u/{author}** ({score} points):")
|
|
||||||
lines.append(f"{indent}{body}")
|
|
||||||
lines.append("")
|
|
||||||
|
|
||||||
# Recurse into replies
|
|
||||||
if depth < max_depth and data.get("replies") and isinstance(data["replies"], dict):
|
|
||||||
lines.append(self._extract_comments(data["replies"], depth + 1, max_depth))
|
|
||||||
|
|
||||||
return "\n".join(lines)
|
|
||||||
|
|||||||
@@ -1,13 +1,7 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""RSS feeds — via feedparser (free, pip dependency).
|
"""RSS — check if feedparser is available."""
|
||||||
|
|
||||||
Backend: feedparser (https://github.com/kurtmckee/feedparser)
|
from .base import Channel
|
||||||
Swap to: any RSS parser
|
|
||||||
"""
|
|
||||||
|
|
||||||
import feedparser
|
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult
|
|
||||||
|
|
||||||
|
|
||||||
class RSSChannel(Channel):
|
class RSSChannel(Channel):
|
||||||
@@ -17,41 +11,11 @@ class RSSChannel(Channel):
|
|||||||
tier = 0
|
tier = 0
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
lower = url.lower()
|
return any(x in url.lower() for x in ["/feed", "/rss", ".xml", "atom"])
|
||||||
domain = urlparse(url).netloc.lower()
|
|
||||||
return (lower.endswith(".xml") or "/rss" in lower or "/feed" in lower
|
|
||||||
or "/atom" in lower or "rss" in domain)
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
def check(self, config=None):
|
||||||
feed = feedparser.parse(url)
|
try:
|
||||||
|
import feedparser
|
||||||
if feed.bozo and not feed.entries:
|
return "ok", "可读取 RSS/Atom 源"
|
||||||
raise ValueError(f"Failed to parse RSS feed: {url}")
|
except ImportError:
|
||||||
|
return "off", "feedparser 未安装。安装:pip install feedparser"
|
||||||
if not feed.entries:
|
|
||||||
raise ValueError(f"No entries in RSS feed: {url}")
|
|
||||||
|
|
||||||
# Return latest entry
|
|
||||||
entry = feed.entries[0]
|
|
||||||
content = entry.get("summary", "") or entry.get("description", "")
|
|
||||||
|
|
||||||
# If multiple entries, summarize all
|
|
||||||
if len(feed.entries) > 1:
|
|
||||||
lines = [f"# {feed.feed.get('title', 'RSS Feed')}\n"]
|
|
||||||
for i, e in enumerate(feed.entries[:20], 1):
|
|
||||||
title = e.get("title", "Untitled")
|
|
||||||
link = e.get("link", "")
|
|
||||||
summary = e.get("summary", "")[:200]
|
|
||||||
lines.append(f"## {i}. {title}")
|
|
||||||
lines.append(f"🔗 {link}")
|
|
||||||
if summary:
|
|
||||||
lines.append(summary)
|
|
||||||
lines.append("")
|
|
||||||
content = "\n".join(lines)
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=feed.feed.get("title", entry.get("title", url)),
|
|
||||||
content=content,
|
|
||||||
url=url,
|
|
||||||
platform="rss",
|
|
||||||
)
|
|
||||||
|
|||||||
+35
-191
@@ -1,209 +1,53 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Twitter/X — via bird CLI (free) or Jina Reader fallback.
|
"""Twitter/X — check if twitter-cli (public-clis/twitter-cli) is available."""
|
||||||
|
|
||||||
Backend: bird (@steipete/bird npm package) for search/timeline
|
|
||||||
Jina Reader for single tweets
|
|
||||||
Swap to: any Twitter access tool
|
|
||||||
"""
|
|
||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
import requests
|
|
||||||
|
|
||||||
|
|
||||||
def _bird_cmd():
|
|
||||||
"""Find bird CLI binary."""
|
|
||||||
return shutil.which("bird") or shutil.which("birdx")
|
|
||||||
|
|
||||||
|
|
||||||
def _bird_env(config=None):
|
|
||||||
"""Build env dict with Twitter cookies for bird CLI."""
|
|
||||||
import os
|
|
||||||
env = os.environ.copy()
|
|
||||||
if config:
|
|
||||||
auth_token = config.get("twitter_auth_token")
|
|
||||||
ct0 = config.get("twitter_ct0")
|
|
||||||
if auth_token:
|
|
||||||
env["AUTH_TOKEN"] = auth_token
|
|
||||||
if ct0:
|
|
||||||
env["CT0"] = ct0
|
|
||||||
return env
|
|
||||||
|
|
||||||
|
|
||||||
class TwitterChannel(Channel):
|
class TwitterChannel(Channel):
|
||||||
name = "twitter"
|
name = "twitter"
|
||||||
description = "Twitter/X 推文"
|
description = "Twitter/X 推文"
|
||||||
backends = ["bird", "Jina Reader"]
|
backends = ["twitter-cli"]
|
||||||
tier = 0 # Single tweet reading is zero-config
|
tier = 1
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
domain = urlparse(url).netloc.lower()
|
from urllib.parse import urlparse
|
||||||
return "x.com" in domain or "twitter.com" in domain
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "x.com" in d or "twitter.com" in d
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
# Basic reading always works (Jina fallback)
|
twitter = shutil.which("twitter")
|
||||||
if _bird_cmd():
|
if not twitter:
|
||||||
return "ok", "搜索、时间线、发推全部可用"
|
return "warn", (
|
||||||
return "ok", "可读取推文。安装 bird + 配置 Cookie 可解锁搜索和发推"
|
"twitter-cli 未安装。安装方式:\n"
|
||||||
|
" pipx install twitter-cli\n"
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
"或:\n"
|
||||||
# Try bird first
|
" uv tool install twitter-cli"
|
||||||
bird = _bird_cmd()
|
|
||||||
if bird:
|
|
||||||
return await self._read_bird(url, bird, config)
|
|
||||||
# Fallback: Jina Reader
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
async def _read_bird(self, url: str, bird: str, config=None) -> ReadResult:
|
|
||||||
result = subprocess.run(
|
|
||||||
[bird, "read", url],
|
|
||||||
capture_output=True, timeout=30,
|
|
||||||
encoding='utf-8', errors='replace',
|
|
||||||
env=_bird_env(config),
|
|
||||||
)
|
|
||||||
if result.returncode != 0:
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
text = result.stdout.strip()
|
|
||||||
# Extract author from first line
|
|
||||||
author = ""
|
|
||||||
lines = text.split("\n")
|
|
||||||
if lines and lines[0].startswith("@"):
|
|
||||||
author = lines[0].split()[0]
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=text[:100],
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
author=author,
|
|
||||||
platform="twitter",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_jina(self, url: str) -> ReadResult:
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
f"https://r.jina.ai/{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
)
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
|
|
||||||
# Detect unusable Jina responses for X/Twitter (JS-required pages)
|
try:
|
||||||
unusable_indicators = [
|
r = subprocess.run(
|
||||||
"page doesn", # "this page doesn't exist" (handles both ' and ')
|
[twitter, "status"], capture_output=True,
|
||||||
"miss what", # "Don't miss what's happening"
|
encoding="utf-8", errors="replace", timeout=10
|
||||||
"Something went wrong. Try reloading",
|
)
|
||||||
"Log in](", # Markdown link: [Log in](...)
|
output = (r.stdout or "") + (r.stderr or "")
|
||||||
]
|
if r.returncode == 0 and "ok: true" in output:
|
||||||
if any(indicator in text for indicator in unusable_indicators):
|
return "ok", (
|
||||||
return ReadResult(
|
"完整可用(搜索、读推文、时间线、长文/Article、"
|
||||||
title="Twitter/X",
|
"用户查询、Thread)"
|
||||||
content="⚠️ Could not read this tweet.\n"
|
|
||||||
"The tweet may have been deleted, or the account is private.\n\n"
|
|
||||||
"Tips:\n"
|
|
||||||
"- Make sure the URL is correct\n"
|
|
||||||
"- Try: bird read <url> (if bird CLI is installed)\n"
|
|
||||||
"- For protected tweets, configure Twitter cookies: "
|
|
||||||
"agent-reach configure twitter-cookies AUTH_TOKEN CT0",
|
|
||||||
url=url,
|
|
||||||
platform="twitter",
|
|
||||||
)
|
)
|
||||||
|
if "not_authenticated" in output:
|
||||||
title = text[:100] if text else url
|
return "warn", (
|
||||||
return ReadResult(
|
"twitter-cli 已安装但未认证。设置方式:\n"
|
||||||
title=title,
|
" export TWITTER_AUTH_TOKEN=\"xxx\"\n"
|
||||||
content=text,
|
" export TWITTER_CT0=\"yyy\"\n"
|
||||||
url=url,
|
"或确保已在浏览器中登录 x.com"
|
||||||
platform="twitter",
|
)
|
||||||
|
return "warn", (
|
||||||
|
"twitter-cli 已安装但认证检查失败。运行:\n"
|
||||||
|
" twitter -v status 查看详细信息"
|
||||||
)
|
)
|
||||||
except Exception:
|
except Exception:
|
||||||
return ReadResult(
|
return "warn", "twitter-cli 已安装但连接失败"
|
||||||
title="Twitter/X",
|
|
||||||
content="⚠️ Could not read this tweet.\n"
|
|
||||||
"The tweet may have been deleted, or the account is private.\n\n"
|
|
||||||
"Tips:\n"
|
|
||||||
"- Make sure the URL is correct\n"
|
|
||||||
"- Try: bird read <url> (if bird CLI is installed)\n"
|
|
||||||
"- For protected tweets, configure Twitter cookies: "
|
|
||||||
"agent-reach configure twitter-cookies AUTH_TOKEN CT0",
|
|
||||||
url=url,
|
|
||||||
platform="twitter",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
|
|
||||||
bird = _bird_cmd()
|
|
||||||
if bird:
|
|
||||||
return await self._search_bird(query, limit, bird, config)
|
|
||||||
|
|
||||||
# Fallback to Exa
|
|
||||||
return await self._search_exa(query, limit, config)
|
|
||||||
|
|
||||||
async def _search_bird(self, query: str, limit: int, bird: str, config=None) -> List[SearchResult]:
|
|
||||||
try:
|
|
||||||
result = subprocess.run(
|
|
||||||
[bird, "search", query, "-n", str(limit)],
|
|
||||||
capture_output=True, timeout=30,
|
|
||||||
encoding='utf-8', errors='replace',
|
|
||||||
env=_bird_env(config),
|
|
||||||
)
|
|
||||||
if result.returncode != 0:
|
|
||||||
return []
|
|
||||||
|
|
||||||
return self._parse_bird_output(result.stdout)
|
|
||||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_bird_output(self, text: str) -> List[SearchResult]:
|
|
||||||
"""Parse bird text output into SearchResults."""
|
|
||||||
results = []
|
|
||||||
current = {}
|
|
||||||
text_lines = []
|
|
||||||
|
|
||||||
for line in text.strip().split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line.startswith("─"):
|
|
||||||
if current:
|
|
||||||
current["text"] = "\n".join(text_lines).strip()
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=current.get("text", "")[:80],
|
|
||||||
url=current.get("url", ""),
|
|
||||||
snippet=current.get("text", ""),
|
|
||||||
author=current.get("author", ""),
|
|
||||||
date=current.get("date", ""),
|
|
||||||
))
|
|
||||||
current = {}
|
|
||||||
text_lines = []
|
|
||||||
continue
|
|
||||||
if line.startswith("@") and line.endswith(":") and "(" in line:
|
|
||||||
current["author"] = line.split()[0]
|
|
||||||
continue
|
|
||||||
if line.startswith("date:"):
|
|
||||||
current["date"] = line[5:].strip()
|
|
||||||
continue
|
|
||||||
if line.startswith("url:"):
|
|
||||||
current["url"] = line[4:].strip()
|
|
||||||
continue
|
|
||||||
if current is not None:
|
|
||||||
text_lines.append(line)
|
|
||||||
|
|
||||||
if current and text_lines:
|
|
||||||
current["text"] = "\n".join(text_lines).strip()
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=current.get("text", "")[:80],
|
|
||||||
url=current.get("url", ""),
|
|
||||||
snippet=current.get("text", ""),
|
|
||||||
author=current.get("author", ""),
|
|
||||||
date=current.get("date", ""),
|
|
||||||
))
|
|
||||||
return results
|
|
||||||
|
|
||||||
async def _search_exa(self, query: str, limit: int, config=None) -> List[SearchResult]:
|
|
||||||
from agent_reach.channels.exa_search import ExaSearchChannel
|
|
||||||
exa = ExaSearchChannel()
|
|
||||||
return await exa.search(f"site:x.com {query}", config=config, limit=limit)
|
|
||||||
|
|||||||
@@ -0,0 +1,212 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""V2EX — public API channel for topics, nodes, users, and replies."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import urllib.request
|
||||||
|
from typing import Any
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
_UA = "agent-reach/1.0"
|
||||||
|
_TIMEOUT = 10
|
||||||
|
|
||||||
|
|
||||||
|
def _get_json(url: str) -> Any:
|
||||||
|
"""Fetch *url* and return parsed JSON. Raises on HTTP/network errors."""
|
||||||
|
req = urllib.request.Request(url, headers={"User-Agent": _UA})
|
||||||
|
with urllib.request.urlopen(req, timeout=_TIMEOUT) as resp:
|
||||||
|
return json.loads(resp.read().decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
class V2EXChannel(Channel):
|
||||||
|
name = "v2ex"
|
||||||
|
description = "V2EX 节点、主题与回复"
|
||||||
|
backends = ["V2EX API (public)"]
|
||||||
|
tier = 0
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# URL routing
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "v2ex.com" in d
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Health check
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
try:
|
||||||
|
_get_json(
|
||||||
|
"https://www.v2ex.com/api/topics/show.json?node_name=python&page=1"
|
||||||
|
)
|
||||||
|
return "ok", "公开 API 可用(热门主题、节点浏览、主题详情、用户信息)"
|
||||||
|
except Exception as e:
|
||||||
|
return "warn", f"V2EX API 连接失败(可能需要代理):{e}"
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Data-fetching methods
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def get_hot_topics(self, limit: int = 20) -> list:
|
||||||
|
"""获取热门帖子列表。
|
||||||
|
|
||||||
|
Returns a list of dicts with keys:
|
||||||
|
title, url, replies, node_name, node_title, content
|
||||||
|
"""
|
||||||
|
data = _get_json("https://www.v2ex.com/api/topics/hot.json")
|
||||||
|
results = []
|
||||||
|
for item in data[:limit]:
|
||||||
|
node = item.get("node") or {}
|
||||||
|
content = item.get("content", "") or ""
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"id": item.get("id", 0),
|
||||||
|
"title": item.get("title", ""),
|
||||||
|
"url": item.get("url", ""),
|
||||||
|
"replies": item.get("replies", 0),
|
||||||
|
"node_name": node.get("name", ""),
|
||||||
|
"node_title": node.get("title", ""),
|
||||||
|
"content": content[:200],
|
||||||
|
"created": item.get("created", 0),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return results
|
||||||
|
|
||||||
|
def get_node_topics(self, node_name: str, limit: int = 20) -> list:
|
||||||
|
"""获取指定节点的最新帖子。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
node_name: 节点名称,如 "python"、"tech"、"jobs"
|
||||||
|
limit: 最多返回条数
|
||||||
|
|
||||||
|
Returns a list of dicts with keys:
|
||||||
|
title, url, replies, node_name, node_title, content
|
||||||
|
"""
|
||||||
|
url = (
|
||||||
|
f"https://www.v2ex.com/api/topics/show.json"
|
||||||
|
f"?node_name={node_name}&page=1"
|
||||||
|
)
|
||||||
|
data = _get_json(url)
|
||||||
|
results = []
|
||||||
|
for item in data[:limit]:
|
||||||
|
node = item.get("node") or {}
|
||||||
|
content = item.get("content", "") or ""
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"id": item.get("id", 0),
|
||||||
|
"title": item.get("title", ""),
|
||||||
|
"url": item.get("url", ""),
|
||||||
|
"replies": item.get("replies", 0),
|
||||||
|
"node_name": node.get("name", node_name),
|
||||||
|
"node_title": node.get("title", ""),
|
||||||
|
"content": content[:200],
|
||||||
|
"created": item.get("created", 0),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return results
|
||||||
|
|
||||||
|
def get_topic(self, topic_id: int) -> dict:
|
||||||
|
"""获取单个帖子详情和回复列表。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
topic_id: 帖子 ID(从 URL https://www.v2ex.com/t/<id> 中获取)
|
||||||
|
|
||||||
|
Returns a dict with keys:
|
||||||
|
id, title, url, content, replies_count, node_name, node_title,
|
||||||
|
author, created, replies (list of dicts with: author, content, created)
|
||||||
|
"""
|
||||||
|
topic_data = _get_json(
|
||||||
|
f"https://www.v2ex.com/api/topics/show.json?id={topic_id}"
|
||||||
|
)
|
||||||
|
# API returns a list even for single-ID queries
|
||||||
|
if isinstance(topic_data, list):
|
||||||
|
topic = topic_data[0] if topic_data else {}
|
||||||
|
else:
|
||||||
|
topic = topic_data
|
||||||
|
|
||||||
|
node = topic.get("node") or {}
|
||||||
|
member = topic.get("member") or {}
|
||||||
|
|
||||||
|
# Fetch replies (first page)
|
||||||
|
try:
|
||||||
|
replies_raw = _get_json(
|
||||||
|
f"https://www.v2ex.com/api/replies/show.json"
|
||||||
|
f"?topic_id={topic_id}&page=1"
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
replies_raw = []
|
||||||
|
|
||||||
|
replies = [
|
||||||
|
{
|
||||||
|
"author": (r.get("member") or {}).get("username", ""),
|
||||||
|
"content": r.get("content", ""),
|
||||||
|
"created": r.get("created", 0),
|
||||||
|
}
|
||||||
|
for r in (replies_raw or [])
|
||||||
|
]
|
||||||
|
|
||||||
|
return {
|
||||||
|
"id": topic.get("id", topic_id),
|
||||||
|
"title": topic.get("title", ""),
|
||||||
|
"url": topic.get("url", f"https://www.v2ex.com/t/{topic_id}"),
|
||||||
|
"content": topic.get("content", ""),
|
||||||
|
"replies_count": topic.get("replies", 0),
|
||||||
|
"node_name": node.get("name", ""),
|
||||||
|
"node_title": node.get("title", ""),
|
||||||
|
"author": member.get("username", ""),
|
||||||
|
"created": topic.get("created", 0),
|
||||||
|
"replies": replies,
|
||||||
|
}
|
||||||
|
|
||||||
|
def get_user(self, username: str) -> dict:
|
||||||
|
"""获取用户信息。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
username: V2EX 用户名
|
||||||
|
|
||||||
|
Returns a dict with keys:
|
||||||
|
id, username, url, website, twitter, psn, github, btc,
|
||||||
|
location, bio, avatar, created
|
||||||
|
"""
|
||||||
|
data = _get_json(
|
||||||
|
f"https://www.v2ex.com/api/members/show.json?username={username}"
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"id": data.get("id", 0),
|
||||||
|
"username": data.get("username", username),
|
||||||
|
"url": data.get("url", f"https://www.v2ex.com/member/{username}"),
|
||||||
|
"website": data.get("website", ""),
|
||||||
|
"twitter": data.get("twitter", ""),
|
||||||
|
"psn": data.get("psn", ""),
|
||||||
|
"github": data.get("github", ""),
|
||||||
|
"btc": data.get("btc", ""),
|
||||||
|
"location": data.get("location", ""),
|
||||||
|
"bio": data.get("bio", ""),
|
||||||
|
"avatar": data.get("avatar_large", data.get("avatar_normal", "")),
|
||||||
|
"created": data.get("created", 0),
|
||||||
|
}
|
||||||
|
|
||||||
|
def search(self, query: str, limit: int = 10) -> list:
|
||||||
|
"""搜索帖子。
|
||||||
|
|
||||||
|
注意:V2EX 公开 API 暂不支持全文搜索端点(/api/search.json 不可用)。
|
||||||
|
本方法通过 Jina Reader 代理 V2EX 站内搜索页面获取结果(纯文本,无结构化数据)。
|
||||||
|
|
||||||
|
如需精确搜索,建议直接访问 https://www.v2ex.com/?q=<query> 或
|
||||||
|
使用 Exa channel 的 site:v2ex.com 搜索。
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
list of dicts with keys: title, url, snippet
|
||||||
|
如果搜索不可用,返回包含单条 {"error": str} 的列表。
|
||||||
|
"""
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"error": (
|
||||||
|
"V2EX 公开 API 不提供搜索端点。"
|
||||||
|
f"建议改用:https://www.v2ex.com/?q={query} "
|
||||||
|
"或通过 Exa channel 使用 site:v2ex.com 搜索。"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -1,49 +1,17 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Web pages — via Jina Reader API (free, no config needed).
|
"""Web — any URL via Jina Reader. Always available."""
|
||||||
|
|
||||||
Backend: Jina Reader (https://r.jina.ai)
|
from .base import Channel
|
||||||
Swap to: Firecrawl, Trafilatura, or any other reader API
|
|
||||||
"""
|
|
||||||
|
|
||||||
import requests
|
|
||||||
from .base import Channel, ReadResult
|
|
||||||
|
|
||||||
|
|
||||||
class WebChannel(Channel):
|
class WebChannel(Channel):
|
||||||
name = "web"
|
name = "web"
|
||||||
description = "网页(任意 URL)"
|
description = "任意网页"
|
||||||
backends = ["Jina Reader API"]
|
backends = ["Jina Reader"]
|
||||||
tier = 0
|
tier = 0
|
||||||
|
|
||||||
JINA_URL = "https://r.jina.ai/"
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
# Fallback — handles any URL not matched by other channels
|
return True # Fallback — handles any URL
|
||||||
return True
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
def check(self, config=None):
|
||||||
resp = requests.get(
|
return "ok", "通过 Jina Reader 读取任意网页(curl https://r.jina.ai/URL)"
|
||||||
f"{self.JINA_URL}{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
|
|
||||||
# Extract title from first markdown heading
|
|
||||||
title = url
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line.startswith("# "):
|
|
||||||
title = line[2:].strip()
|
|
||||||
break
|
|
||||||
if line.startswith("Title:"):
|
|
||||||
title = line[6:].strip()
|
|
||||||
break
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=title,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="web",
|
|
||||||
)
|
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""WeChat Official Account articles — read and search.
|
||||||
|
|
||||||
|
Read: Exa crawling (primary) / Camoufox stealth browser (optional)
|
||||||
|
Search: Exa web_search with includeDomains mp.weixin.qq.com
|
||||||
|
"""
|
||||||
|
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
|
||||||
|
def _exa_available() -> bool:
|
||||||
|
mcporter = shutil.which("mcporter")
|
||||||
|
if not mcporter:
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
[mcporter, "config", "list"],
|
||||||
|
capture_output=True, text=True, timeout=5,
|
||||||
|
)
|
||||||
|
return "exa" in r.stdout.lower()
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
class WeChatChannel(Channel):
|
||||||
|
name = "wechat"
|
||||||
|
description = "微信公众号文章"
|
||||||
|
backends = ["Exa via mcporter (搜索+阅读)", "Camoufox (可选阅读)"]
|
||||||
|
tier = 0
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "mp.weixin.qq.com" in d or "weixin.qq.com" in d
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
has_exa = _exa_available()
|
||||||
|
has_camoufox = False
|
||||||
|
try:
|
||||||
|
import camoufox # noqa: F401
|
||||||
|
has_camoufox = True
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if has_exa and has_camoufox:
|
||||||
|
return "ok", "完整可用(Exa 搜索 + Exa/Camoufox 阅读公众号文章)"
|
||||||
|
elif has_exa:
|
||||||
|
return "ok", (
|
||||||
|
"通过 Exa 搜索和阅读微信公众号文章(免费,无需额外配置)。"
|
||||||
|
"可选安装 Camoufox 获得更好的全文阅读效果。"
|
||||||
|
)
|
||||||
|
elif has_camoufox:
|
||||||
|
return "warn", (
|
||||||
|
"Camoufox 可阅读公众号文章,但搜索功能需要 Exa。"
|
||||||
|
"运行 `agent-reach install --env=auto` 安装 Exa。"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
return "off", (
|
||||||
|
"需要 mcporter + Exa MCP 来搜索和阅读微信公众号文章。\n"
|
||||||
|
"运行 `agent-reach install --env=auto` 安装。"
|
||||||
|
)
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Weibo (微博) — check if mcporter + mcp-server-weibo is available."""
|
||||||
|
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
|
||||||
|
class WeiboChannel(Channel):
|
||||||
|
name = "weibo"
|
||||||
|
description = "微博动态与热搜"
|
||||||
|
backends = ["mcp-server-weibo"]
|
||||||
|
tier = 1
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "weibo.com" in d or "weibo.cn" in d
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
mcporter = shutil.which("mcporter")
|
||||||
|
if not mcporter:
|
||||||
|
return "off", (
|
||||||
|
"需要 mcporter + mcp-server-weibo。安装步骤:\n"
|
||||||
|
" 1. npm install -g mcporter\n"
|
||||||
|
" 2. pip install git+https://github.com/Panniantong/mcp-server-weibo.git\n"
|
||||||
|
" 3. mcporter config add weibo --command 'mcp-server-weibo'\n"
|
||||||
|
" 详见 https://github.com/Panniantong/mcp-server-weibo"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
[mcporter, "config", "list"], capture_output=True,
|
||||||
|
encoding="utf-8", errors="replace", timeout=5
|
||||||
|
)
|
||||||
|
if "weibo" not in r.stdout:
|
||||||
|
return "off", (
|
||||||
|
"mcporter 已装但微博 MCP 未配置。运行:\n"
|
||||||
|
" pip install git+https://github.com/Panniantong/mcp-server-weibo.git\n"
|
||||||
|
" mcporter config add weibo --command 'mcp-server-weibo'"
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return "off", "mcporter 连接异常"
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
[mcporter, "list", "weibo"], capture_output=True,
|
||||||
|
encoding="utf-8", errors="replace", timeout=15
|
||||||
|
)
|
||||||
|
if r.returncode == 0 and "search_users" in r.stdout:
|
||||||
|
return "ok", "完整可用(热搜、搜索、用户动态、评论)"
|
||||||
|
return "warn", "MCP 已配置但工具加载失败,检查 mcp-server-weibo 版本"
|
||||||
|
except Exception:
|
||||||
|
return "warn", "MCP 连接异常,检查 mcp-server-weibo 是否可用"
|
||||||
+143
-153
@@ -1,172 +1,162 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""XiaoHongShu (小红书) — via mcporter + xiaohongshu MCP server.
|
"""XiaoHongShu — check if xhs-cli (xiaohongshu-cli) is available."""
|
||||||
|
|
||||||
Backend: xiaohongshu-mcp server (internal API, reliable)
|
|
||||||
Requires: mcporter CLI + xiaohongshu MCP server running
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List, Optional
|
|
||||||
|
def format_xhs_result(data):
|
||||||
|
"""Clean XHS API response, keeping only useful fields.
|
||||||
|
|
||||||
|
Handles both single note objects and lists of notes (search results).
|
||||||
|
Drastically reduces token usage by stripping structural redundancy (#134).
|
||||||
|
"""
|
||||||
|
if isinstance(data, list):
|
||||||
|
return [_clean_note(item) for item in data]
|
||||||
|
if isinstance(data, dict):
|
||||||
|
# Handle search_feeds wrapper: {"items": [...]} or {"data": {"items": [...]}}
|
||||||
|
items = None
|
||||||
|
if "items" in data:
|
||||||
|
items = data["items"]
|
||||||
|
elif "data" in data and isinstance(data.get("data"), dict):
|
||||||
|
items = data["data"].get("items") or data["data"].get("notes")
|
||||||
|
if items and isinstance(items, list):
|
||||||
|
return [_clean_note(item) for item in items]
|
||||||
|
# Single note
|
||||||
|
return _clean_note(data)
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_note(note):
|
||||||
|
"""Extract useful fields from a single XHS note/feed item."""
|
||||||
|
if not isinstance(note, dict):
|
||||||
|
return note
|
||||||
|
|
||||||
|
# Some responses nest the note under "note_card" or "note"
|
||||||
|
inner = note.get("note_card") or note.get("note") or note
|
||||||
|
|
||||||
|
result = {}
|
||||||
|
|
||||||
|
# Basic info
|
||||||
|
for key in ("id", "note_id", "xsec_token", "title", "desc", "type", "time"):
|
||||||
|
if key in inner:
|
||||||
|
result[key] = inner[key]
|
||||||
|
|
||||||
|
# Content (may be in desc or content)
|
||||||
|
if "content" in inner and "desc" not in result:
|
||||||
|
result["content"] = inner["content"]
|
||||||
|
|
||||||
|
# Author
|
||||||
|
user = inner.get("user") or inner.get("author")
|
||||||
|
if isinstance(user, dict):
|
||||||
|
result["user"] = {
|
||||||
|
k: user[k] for k in ("nickname", "user_id", "nick_name") if k in user
|
||||||
|
}
|
||||||
|
|
||||||
|
# Engagement metrics
|
||||||
|
interact = inner.get("interact_info") or inner.get("note_interact_info") or {}
|
||||||
|
if isinstance(interact, dict):
|
||||||
|
for key in ("liked_count", "collected_count", "comment_count", "share_count"):
|
||||||
|
if key in interact:
|
||||||
|
result[key] = interact[key]
|
||||||
|
# Also check top-level (some API formats)
|
||||||
|
for key in ("liked_count", "collected_count", "comment_count", "share_count"):
|
||||||
|
if key in inner and key not in result:
|
||||||
|
result[key] = inner[key]
|
||||||
|
|
||||||
|
# Images — just URLs
|
||||||
|
images = inner.get("image_list") or inner.get("images_list") or []
|
||||||
|
if isinstance(images, list):
|
||||||
|
urls = []
|
||||||
|
for img in images:
|
||||||
|
if isinstance(img, dict):
|
||||||
|
url = img.get("url") or img.get("url_default") or img.get("original")
|
||||||
|
if url:
|
||||||
|
urls.append(url)
|
||||||
|
elif isinstance(img, str):
|
||||||
|
urls.append(img)
|
||||||
|
if urls:
|
||||||
|
result["images"] = urls
|
||||||
|
|
||||||
|
# Tags
|
||||||
|
tags = inner.get("tag_list") or inner.get("tags") or []
|
||||||
|
if isinstance(tags, list):
|
||||||
|
tag_names = []
|
||||||
|
for t in tags:
|
||||||
|
if isinstance(t, dict) and "name" in t:
|
||||||
|
tag_names.append(t["name"])
|
||||||
|
elif isinstance(t, str):
|
||||||
|
tag_names.append(t)
|
||||||
|
if tag_names:
|
||||||
|
result["tags"] = tag_names
|
||||||
|
|
||||||
|
# Comments (if present, e.g. from get_feed_detail with comments)
|
||||||
|
comments = inner.get("comments") or []
|
||||||
|
if isinstance(comments, list) and comments:
|
||||||
|
result["comments"] = [_clean_comment(c) for c in comments]
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _clean_comment(comment):
|
||||||
|
"""Extract useful fields from a comment."""
|
||||||
|
if not isinstance(comment, dict):
|
||||||
|
return comment
|
||||||
|
result = {}
|
||||||
|
if "content" in comment:
|
||||||
|
result["content"] = comment["content"]
|
||||||
|
user = comment.get("user_info") or comment.get("user")
|
||||||
|
if isinstance(user, dict):
|
||||||
|
result["user"] = user.get("nickname") or user.get("nick_name", "")
|
||||||
|
for key in ("like_count", "sub_comment_count"):
|
||||||
|
if key in comment:
|
||||||
|
result[key] = comment[key]
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
class XiaoHongShuChannel(Channel):
|
class XiaoHongShuChannel(Channel):
|
||||||
name = "xiaohongshu"
|
name = "xiaohongshu"
|
||||||
description = "小红书笔记"
|
description = "小红书笔记"
|
||||||
backends = ["xiaohongshu-mcp"]
|
backends = ["xhs-cli (xiaohongshu-cli)"]
|
||||||
tier = 2
|
tier = 1
|
||||||
|
|
||||||
def _mcporter_ok(self) -> bool:
|
|
||||||
"""Check if mcporter + xiaohongshu MCP is available."""
|
|
||||||
if not shutil.which("mcporter"):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
return "xiaohongshu" in r.stdout
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
def _call(self, expr: str, timeout: int = 30) -> str:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call", expr],
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
# ── Channel interface ──
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
d = urlparse(url).netloc.lower()
|
d = urlparse(url).netloc.lower()
|
||||||
return "xiaohongshu.com" in d or "xhslink.com" in d
|
return "xiaohongshu.com" in d or "xhslink.com" in d
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if not shutil.which("mcporter"):
|
xhs = shutil.which("xhs")
|
||||||
|
if not xhs:
|
||||||
return "off", (
|
return "off", (
|
||||||
"需要 mcporter + xiaohongshu-mcp。安装步骤:\n"
|
"需要安装 xhs-cli:\n"
|
||||||
" 1. npm install -g mcporter\n"
|
" pipx install xiaohongshu-cli\n"
|
||||||
" 2. docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
|
"或:\n"
|
||||||
" 3. mcporter config add xiaohongshu http://localhost:18060/mcp\n"
|
" uv tool install xiaohongshu-cli\n"
|
||||||
" 详见 https://github.com/xpzouying/xiaohongshu-mcp"
|
"安装后运行 `xhs login` 登录"
|
||||||
)
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
return "off", (
|
|
||||||
"mcporter 已装但小红书 MCP 未配置。运行:\n"
|
|
||||||
" docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
|
|
||||||
" mcporter config add xiaohongshu http://localhost:18060/mcp"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
out = self._call("xiaohongshu.check_login_status()", timeout=10)
|
r = subprocess.run(
|
||||||
if "已登录" in out or "logged" in out.lower():
|
[xhs, "status"], capture_output=True,
|
||||||
return "ok", "完整可用(阅读、搜索、发帖、评论、点赞)"
|
encoding="utf-8", errors="replace", timeout=10,
|
||||||
return "warn", "MCP 已连接但未登录,需扫码登录"
|
)
|
||||||
|
output = (r.stdout or "") + (r.stderr or "")
|
||||||
|
if r.returncode == 0 and "ok: true" in output:
|
||||||
|
return "ok", (
|
||||||
|
"完整可用(搜索、阅读、评论、发帖、热门、"
|
||||||
|
"收藏、关注、用户查询)"
|
||||||
|
)
|
||||||
|
if "not_authenticated" in output or "expired" in output:
|
||||||
|
return "warn", (
|
||||||
|
"xhs-cli 已安装但未登录。运行:\n"
|
||||||
|
" xhs login\n"
|
||||||
|
"(自动从浏览器提取 Cookie,或扫码登录)"
|
||||||
|
)
|
||||||
|
return "warn", (
|
||||||
|
"xhs-cli 已安装但状态异常。运行:\n"
|
||||||
|
" xhs -v status 查看详细信息"
|
||||||
|
)
|
||||||
except Exception:
|
except Exception:
|
||||||
return "warn", "MCP 连接异常,检查 xiaohongshu-mcp 服务是否在运行"
|
return "warn", "xhs-cli 已安装但连接失败"
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
return ReadResult(
|
|
||||||
title="XiaoHongShu",
|
|
||||||
content=(
|
|
||||||
"⚠️ 小红书需要 mcporter + xiaohongshu-mcp 才能使用。\n\n"
|
|
||||||
"安装步骤:\n"
|
|
||||||
"1. npm install -g mcporter\n"
|
|
||||||
"2. docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
|
|
||||||
"3. mcporter config add xiaohongshu http://localhost:18060/mcp\n"
|
|
||||||
"4. 运行 agent-reach doctor 检查状态\n\n"
|
|
||||||
"详见 https://github.com/xpzouying/xiaohongshu-mcp"
|
|
||||||
),
|
|
||||||
url=url, platform="xiaohongshu",
|
|
||||||
)
|
|
||||||
|
|
||||||
note_id = self._extract_note_id(url)
|
|
||||||
if not note_id:
|
|
||||||
return ReadResult(
|
|
||||||
title="XiaoHongShu",
|
|
||||||
content=f"⚠️ 无法从 URL 提取笔记 ID: {url}",
|
|
||||||
url=url, platform="xiaohongshu",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Step 1: get xsec_token from feeds
|
|
||||||
xsec_token = self._find_token(note_id)
|
|
||||||
|
|
||||||
if not xsec_token:
|
|
||||||
return ReadResult(
|
|
||||||
title="XiaoHongShu",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法获取笔记 {note_id} 的访问令牌。\n"
|
|
||||||
"小红书需要 xsec_token 才能读取笔记详情。\n"
|
|
||||||
"请先通过搜索找到这篇笔记,或直接使用搜索功能。"
|
|
||||||
),
|
|
||||||
url=url, platform="xiaohongshu",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Step 2: get detail
|
|
||||||
out = self._call(
|
|
||||||
f'xiaohongshu.get_feed_detail(feed_id: "{note_id}", xsec_token: "{xsec_token}")',
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=self._extract_title(out) or f"XHS {note_id}",
|
|
||||||
content=out.strip(),
|
|
||||||
url=url, platform="xiaohongshu",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
raise ValueError(
|
|
||||||
"小红书搜索需要 mcporter + xiaohongshu-mcp。\n"
|
|
||||||
"安装: npm install -g mcporter && mcporter config add xiaohongshu http://localhost:18060/mcp"
|
|
||||||
)
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
safe_q = query.replace('"', '\\"')
|
|
||||||
out = self._call(f'xiaohongshu.search_feeds(keyword: "{safe_q}")', timeout=30)
|
|
||||||
|
|
||||||
results = []
|
|
||||||
try:
|
|
||||||
data = json.loads(out)
|
|
||||||
for item in data.get("feeds", [])[:limit]:
|
|
||||||
card = item.get("noteCard", {})
|
|
||||||
user = card.get("user", {})
|
|
||||||
interact = card.get("interactInfo", {})
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=card.get("displayTitle", ""),
|
|
||||||
url=f"https://www.xiaohongshu.com/explore/{item.get('id', '')}",
|
|
||||||
snippet=f"👤 {user.get('nickname', '')} · ❤ {interact.get('likedCount', '0')}",
|
|
||||||
score=0,
|
|
||||||
))
|
|
||||||
except (json.JSONDecodeError, KeyError):
|
|
||||||
pass
|
|
||||||
return results
|
|
||||||
|
|
||||||
# ── Helpers ──
|
|
||||||
|
|
||||||
def _extract_note_id(self, url: str) -> str:
|
|
||||||
parts = urlparse(url).path.strip("/").split("/")
|
|
||||||
return parts[-1] if parts else ""
|
|
||||||
|
|
||||||
def _find_token(self, note_id: str) -> Optional[str]:
|
|
||||||
"""Try to find xsec_token for a note from feeds."""
|
|
||||||
try:
|
|
||||||
out = self._call("xiaohongshu.list_feeds()", timeout=15)
|
|
||||||
data = json.loads(out)
|
|
||||||
for feed in data.get("feeds", []):
|
|
||||||
if feed.get("id") == note_id:
|
|
||||||
return feed.get("xsecToken", "")
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return None
|
|
||||||
|
|
||||||
def _extract_title(self, text: str) -> str:
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line and not line.startswith(("{", "[", "#", "http")):
|
|
||||||
return line[:80]
|
|
||||||
return ""
|
|
||||||
|
|||||||
@@ -0,0 +1,54 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Xiaoyuzhou Podcast (小宇宙播客) — transcribe podcasts via Groq Whisper API."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
from agent_reach.config import Config
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
|
||||||
|
class XiaoyuzhouChannel(Channel):
|
||||||
|
name = "xiaoyuzhou"
|
||||||
|
description = "小宇宙播客转文字"
|
||||||
|
backends = ["groq-whisper", "ffmpeg"]
|
||||||
|
tier = 1
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "xiaoyuzhoufm.com" in d
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
# Check ffmpeg
|
||||||
|
if not shutil.which("ffmpeg"):
|
||||||
|
return "off", (
|
||||||
|
"需要 ffmpeg(音频转码和切片)。安装:\n"
|
||||||
|
" Ubuntu/Debian: apt install -y ffmpeg\n"
|
||||||
|
" macOS: brew install ffmpeg"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check script exists
|
||||||
|
script = os.path.expanduser("~/.agent-reach/tools/xiaoyuzhou/transcribe.sh")
|
||||||
|
if not os.path.isfile(script):
|
||||||
|
return "off", (
|
||||||
|
"转录脚本未安装。运行:\n"
|
||||||
|
" agent-reach install --env=auto\n"
|
||||||
|
" 或手动复制 transcribe.sh 到 ~/.agent-reach/tools/xiaoyuzhou/"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check GROQ_API_KEY — prefer env var, fall back to Agent Reach config
|
||||||
|
has_key = bool(os.environ.get("GROQ_API_KEY"))
|
||||||
|
if not has_key:
|
||||||
|
try:
|
||||||
|
cfg = config if config is not None else Config()
|
||||||
|
has_key = bool(cfg.get("groq_api_key"))
|
||||||
|
except Exception:
|
||||||
|
has_key = False
|
||||||
|
if not has_key:
|
||||||
|
return "warn", (
|
||||||
|
"需要配置 Groq API Key(免费)。步骤:\n"
|
||||||
|
" 1. 注册 https://console.groq.com\n"
|
||||||
|
" 2. 运行: agent-reach configure groq-key gsk_xxxxx"
|
||||||
|
)
|
||||||
|
|
||||||
|
return "ok", "完整可用(播客下载 + Whisper 转录)"
|
||||||
@@ -0,0 +1,314 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Xueqiu (雪球) — stock quotes, search, trending posts & hot stocks."""
|
||||||
|
|
||||||
|
import http.cookiejar
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
_UA = (
|
||||||
|
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
|
||||||
|
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||||
|
"Chrome/120.0.0.0 Safari/537.36"
|
||||||
|
)
|
||||||
|
_REFERER = "https://xueqiu.com/"
|
||||||
|
_TIMEOUT = 10
|
||||||
|
_XUEQIU_HOME = "https://xueqiu.com"
|
||||||
|
|
||||||
|
# --------------- cookie-aware HTTP helpers --------------- #
|
||||||
|
|
||||||
|
_cookie_jar = http.cookiejar.CookieJar()
|
||||||
|
_opener = urllib.request.build_opener(
|
||||||
|
urllib.request.HTTPCookieProcessor(_cookie_jar),
|
||||||
|
)
|
||||||
|
_cookies_initialized = False
|
||||||
|
|
||||||
|
|
||||||
|
def _inject_cookie_string(cookie_str: str) -> None:
|
||||||
|
"""Parse a 'name=value; name2=value2' string and inject into the cookie jar."""
|
||||||
|
for pair in cookie_str.split(";"):
|
||||||
|
pair = pair.strip()
|
||||||
|
if "=" not in pair:
|
||||||
|
continue
|
||||||
|
name, _, value = pair.partition("=")
|
||||||
|
cookie = http.cookiejar.Cookie(
|
||||||
|
version=0,
|
||||||
|
name=name.strip(),
|
||||||
|
value=value.strip(),
|
||||||
|
port=None,
|
||||||
|
port_specified=False,
|
||||||
|
domain=".xueqiu.com",
|
||||||
|
domain_specified=True,
|
||||||
|
domain_initial_dot=True,
|
||||||
|
path="/",
|
||||||
|
path_specified=True,
|
||||||
|
secure=True,
|
||||||
|
expires=None,
|
||||||
|
discard=True,
|
||||||
|
comment=None,
|
||||||
|
comment_url=None,
|
||||||
|
rest={},
|
||||||
|
)
|
||||||
|
_cookie_jar.set_cookie(cookie)
|
||||||
|
|
||||||
|
|
||||||
|
def _load_cookies_from_config() -> bool:
|
||||||
|
"""Try to load Xueqiu cookies from agent-reach config file (xueqiu_cookie key)."""
|
||||||
|
try:
|
||||||
|
from ..config import Config
|
||||||
|
|
||||||
|
cfg = Config()
|
||||||
|
cookie_str = cfg.get("xueqiu_cookie")
|
||||||
|
if not cookie_str:
|
||||||
|
return False
|
||||||
|
_inject_cookie_string(cookie_str)
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _load_cookies_from_browser() -> bool:
|
||||||
|
"""Try to silently load Xueqiu cookies from the local Chrome browser.
|
||||||
|
|
||||||
|
Only succeeds when browser_cookie3 is installed AND the user is logged in
|
||||||
|
(xq_a_token present). Failures are silently ignored so that agents without
|
||||||
|
a local browser keep working.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
try:
|
||||||
|
import rookiepy
|
||||||
|
cookies = rookiepy.chrome([".xueqiu.com"])
|
||||||
|
if not any(c.get("name") == "xq_a_token" for c in cookies):
|
||||||
|
return False
|
||||||
|
for c in cookies:
|
||||||
|
_cookie_jar.set(c["name"], c["value"], domain=c.get("domain", ".xueqiu.com"))
|
||||||
|
return True
|
||||||
|
except ImportError:
|
||||||
|
import browser_cookie3
|
||||||
|
cookies = list(browser_cookie3.chrome(domain_name=".xueqiu.com"))
|
||||||
|
if not any(c.name == "xq_a_token" for c in cookies):
|
||||||
|
return False
|
||||||
|
for c in cookies:
|
||||||
|
_cookie_jar.set_cookie(c)
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_cookies() -> None:
|
||||||
|
"""Populate session cookies using the best available source.
|
||||||
|
|
||||||
|
Priority order:
|
||||||
|
1. Saved cookie string in ~/.agent-reach/config.yaml (set by configure --from-browser)
|
||||||
|
2. Live Chrome browser cookies via rookiepy/browser_cookie3 (if installed + logged in)
|
||||||
|
3. Homepage visit fallback (only yields anti-DDoS acw_tc,
|
||||||
|
not enough for stock APIs)
|
||||||
|
"""
|
||||||
|
global _cookies_initialized
|
||||||
|
if _cookies_initialized:
|
||||||
|
return
|
||||||
|
if _load_cookies_from_config():
|
||||||
|
_cookies_initialized = True
|
||||||
|
return
|
||||||
|
if _load_cookies_from_browser():
|
||||||
|
_cookies_initialized = True
|
||||||
|
return
|
||||||
|
# Fallback: visit homepage to pick up acw_tc anti-DDoS cookie.
|
||||||
|
# This is not sufficient for authenticated APIs but avoids hard failures
|
||||||
|
# on public endpoints that only need the session cookie.
|
||||||
|
req = urllib.request.Request(_XUEQIU_HOME, headers={"User-Agent": _UA})
|
||||||
|
_opener.open(req, timeout=_TIMEOUT)
|
||||||
|
_cookies_initialized = True
|
||||||
|
|
||||||
|
|
||||||
|
def _get_json(url: str) -> Any:
|
||||||
|
"""Fetch *url* with Xueqiu session cookies and return parsed JSON."""
|
||||||
|
_ensure_cookies()
|
||||||
|
req = urllib.request.Request(
|
||||||
|
url, headers={"User-Agent": _UA, "Referer": _REFERER}
|
||||||
|
)
|
||||||
|
with _opener.open(req, timeout=_TIMEOUT) as resp:
|
||||||
|
return json.loads(resp.read().decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_html(text: str) -> str:
|
||||||
|
"""Remove HTML tags and decode common entities."""
|
||||||
|
text = re.sub(r"<[^>]+>", "", text)
|
||||||
|
for entity, char in ((" ", " "), ("&", "&"), ("<", "<"), (">", ">")):
|
||||||
|
text = text.replace(entity, char)
|
||||||
|
return text.strip()
|
||||||
|
|
||||||
|
|
||||||
|
class XueqiuChannel(Channel):
|
||||||
|
name = "xueqiu"
|
||||||
|
description = "雪球股票行情与社区动态"
|
||||||
|
backends = ["Xueqiu API (需要登录 Cookie)"]
|
||||||
|
tier = 1
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# URL routing
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
d = urllib.parse.urlparse(url).netloc.lower()
|
||||||
|
return "xueqiu.com" in d
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Health check
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
try:
|
||||||
|
data = _get_json(
|
||||||
|
"https://stock.xueqiu.com/v5/stock/batch/quote.json?symbol=SH000001"
|
||||||
|
)
|
||||||
|
items = (data.get("data") or {}).get("items") or []
|
||||||
|
if items:
|
||||||
|
return "ok", "公开 API 可用(行情、搜索、热帖、热股)"
|
||||||
|
return "warn", "API 响应异常(返回数据为空)"
|
||||||
|
except Exception as e:
|
||||||
|
return "warn", (
|
||||||
|
f"Xueqiu API 连接失败:{e}。"
|
||||||
|
"请先登录雪球后运行:agent-reach configure --from-browser chrome"
|
||||||
|
)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Data-fetching methods
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def get_stock_quote(self, symbol: str) -> dict:
|
||||||
|
"""获取实时股票行情。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
symbol: 股票代码,如 SH600519(沪)、SZ000858(深)、AAPL(美)、00700(港)
|
||||||
|
|
||||||
|
Returns a dict with keys:
|
||||||
|
symbol, name, current, percent, chg, high, low, open, last_close,
|
||||||
|
volume, amount, market_capital, turnover_rate, pe_ttm, timestamp
|
||||||
|
"""
|
||||||
|
data = _get_json(
|
||||||
|
f"https://stock.xueqiu.com/v5/stock/batch/quote.json?symbol={symbol}"
|
||||||
|
)
|
||||||
|
items = (data.get("data") or {}).get("items") or []
|
||||||
|
q = (items[0].get("quote") or {}) if items else {}
|
||||||
|
return {
|
||||||
|
"symbol": q.get("symbol", symbol),
|
||||||
|
"name": q.get("name", ""),
|
||||||
|
"current": q.get("current"),
|
||||||
|
"percent": q.get("percent"),
|
||||||
|
"chg": q.get("chg"),
|
||||||
|
"high": q.get("high"),
|
||||||
|
"low": q.get("low"),
|
||||||
|
"open": q.get("open"),
|
||||||
|
"last_close": q.get("last_close"),
|
||||||
|
"volume": q.get("volume"),
|
||||||
|
"amount": q.get("amount"),
|
||||||
|
"market_capital": q.get("market_capital"),
|
||||||
|
"turnover_rate": q.get("turnover_rate"),
|
||||||
|
"pe_ttm": q.get("pe_ttm"),
|
||||||
|
"timestamp": q.get("timestamp"),
|
||||||
|
}
|
||||||
|
|
||||||
|
def search_stock(self, query: str, limit: int = 10) -> list:
|
||||||
|
"""搜索股票。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
query: 股票代码或中文名称,如 "茅台"、"600519"
|
||||||
|
limit: 最多返回条数
|
||||||
|
|
||||||
|
Returns a list of dicts with keys:
|
||||||
|
symbol, name, exchange
|
||||||
|
"""
|
||||||
|
data = _get_json(
|
||||||
|
f"https://xueqiu.com/stock/search.json"
|
||||||
|
f"?code={urllib.parse.quote(query)}&size={limit}"
|
||||||
|
)
|
||||||
|
stocks = data.get("stocks") or []
|
||||||
|
results = []
|
||||||
|
for s in stocks[:limit]:
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"symbol": s.get("code", ""),
|
||||||
|
"name": s.get("name", ""),
|
||||||
|
"exchange": s.get("exchange", ""),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return results
|
||||||
|
|
||||||
|
def get_hot_posts(self, limit: int = 20) -> list:
|
||||||
|
"""获取雪球热门帖子。
|
||||||
|
|
||||||
|
Uses the v4 public timeline endpoint which returns posts in a `list`
|
||||||
|
array. Each item carries a JSON-encoded `data` field containing the
|
||||||
|
actual post payload (title, description, user, like_count, target).
|
||||||
|
|
||||||
|
Args:
|
||||||
|
limit: 最多返回条数(上限 50)
|
||||||
|
|
||||||
|
Returns a list of dicts with keys:
|
||||||
|
id, title, text, author, likes, url
|
||||||
|
"""
|
||||||
|
data = _get_json(
|
||||||
|
"https://xueqiu.com/v4/statuses/public_timeline_by_category.json"
|
||||||
|
"?since_id=-1&max_id=-1&count=20&category=-1"
|
||||||
|
)
|
||||||
|
items = data.get("list") or []
|
||||||
|
results = []
|
||||||
|
for item in items[:limit]:
|
||||||
|
# Each item.data is a JSON string containing the real post payload
|
||||||
|
try:
|
||||||
|
post = (
|
||||||
|
json.loads(item["data"])
|
||||||
|
if isinstance(item.get("data"), str)
|
||||||
|
else {}
|
||||||
|
)
|
||||||
|
except (json.JSONDecodeError, KeyError):
|
||||||
|
post = {}
|
||||||
|
user = post.get("user") or {}
|
||||||
|
text = _strip_html(
|
||||||
|
post.get("text") or post.get("description") or ""
|
||||||
|
)
|
||||||
|
target = post.get("target", "")
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"id": post.get("id", 0),
|
||||||
|
"title": post.get("title") or "",
|
||||||
|
"text": text[:200],
|
||||||
|
"author": user.get("screen_name", ""),
|
||||||
|
"likes": post.get("like_count", 0),
|
||||||
|
"url": f"https://xueqiu.com{target}" if target else "",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return results
|
||||||
|
|
||||||
|
def get_hot_stocks(self, limit: int = 10, stock_type: int = 10) -> list:
|
||||||
|
"""获取热门股票排行。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
limit: 最多返回条数(上限 50)
|
||||||
|
stock_type: 10=人气榜(默认),12=关注榜
|
||||||
|
|
||||||
|
Returns a list of dicts with keys:
|
||||||
|
symbol, name, current, percent, rank
|
||||||
|
"""
|
||||||
|
data = _get_json(
|
||||||
|
f"https://stock.xueqiu.com/v5/stock/hot_stock/list.json"
|
||||||
|
f"?size={limit}&type={stock_type}"
|
||||||
|
)
|
||||||
|
items = (data.get("data") or {}).get("items") or []
|
||||||
|
results = []
|
||||||
|
for idx, item in enumerate(items[:limit], 1):
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"symbol": item.get("code") or item.get("symbol", ""),
|
||||||
|
"name": item.get("name", ""),
|
||||||
|
"current": item.get("current"),
|
||||||
|
"percent": item.get("percent"),
|
||||||
|
"rank": idx,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return results
|
||||||
+30
-109
@@ -1,125 +1,46 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""YouTube — via yt-dlp (video info, subtitles, and search).
|
"""YouTube — check if yt-dlp is available with JS runtime."""
|
||||||
|
|
||||||
Backend: yt-dlp (https://github.com/yt-dlp/yt-dlp)
|
|
||||||
Supports: read (info + subtitles), search (ytsearch)
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
|
||||||
import tempfile
|
from agent_reach.utils.paths import get_ytdlp_config_path, render_ytdlp_fix_command
|
||||||
from pathlib import Path
|
from agent_reach.utils.text import read_utf8_text
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult, SearchResult
|
from .base import Channel
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class YouTubeChannel(Channel):
|
class YouTubeChannel(Channel):
|
||||||
name = "youtube"
|
name = "youtube"
|
||||||
description = "YouTube 视频字幕"
|
description = "YouTube 视频和字幕"
|
||||||
backends = ["yt-dlp"]
|
backends = ["yt-dlp"]
|
||||||
requires_tools = ["yt-dlp"]
|
|
||||||
tier = 0
|
tier = 0
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
d = urlparse(url).netloc.lower()
|
d = urlparse(url).netloc.lower()
|
||||||
return "youtube.com" in d or "youtu.be" in d
|
return "youtube.com" in d or "youtu.be" in d
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
def check(self, config=None):
|
||||||
if not shutil.which("yt-dlp"):
|
if not shutil.which("yt-dlp"):
|
||||||
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
|
return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
|
||||||
|
# Check JS runtime
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
has_js = shutil.which("deno") or shutil.which("node")
|
||||||
info = self._get_info(url)
|
if not has_js:
|
||||||
title = info.get("title", url)
|
return "warn", (
|
||||||
author = info.get("uploader", "")
|
"yt-dlp 已安装但缺少 JS runtime(YouTube 必须)。\n"
|
||||||
|
" 安装 Node.js 或 deno,然后运行:agent-reach install"
|
||||||
transcript = self._get_subtitles(url, tmpdir)
|
|
||||||
if not transcript:
|
|
||||||
transcript = f"[Video: {title}]\n[No subtitles available.]"
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=title, content=transcript, url=url,
|
|
||||||
author=author, platform="youtube",
|
|
||||||
extra={
|
|
||||||
"duration": info.get("duration_string"),
|
|
||||||
"view_count": info.get("view_count"),
|
|
||||||
"upload_date": info.get("upload_date"),
|
|
||||||
},
|
|
||||||
)
|
)
|
||||||
|
# Check yt-dlp config for --js-runtimes
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
# Deno works out of the box; Node.js requires explicit config
|
||||||
"""Search YouTube via yt-dlp's ytsearch."""
|
has_deno = shutil.which("deno")
|
||||||
if not shutil.which("yt-dlp"):
|
if not has_deno:
|
||||||
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
|
ytdlp_config = get_ytdlp_config_path()
|
||||||
|
has_js_config = False
|
||||||
limit = kwargs.get("limit", 10)
|
if ytdlp_config.exists():
|
||||||
|
has_js_config = "--js-runtimes" in read_utf8_text(ytdlp_config)
|
||||||
try:
|
if not has_js_config:
|
||||||
r = subprocess.run(
|
return "warn", (
|
||||||
["yt-dlp", "--dump-json", "--flat-playlist",
|
"yt-dlp 已安装但未配置 JS runtime。运行:\n"
|
||||||
f"ytsearch{limit}:{query}"],
|
f" {render_ytdlp_fix_command()}"
|
||||||
capture_output=True, text=True, timeout=30,
|
)
|
||||||
)
|
return "ok", "可提取视频信息和字幕"
|
||||||
results = []
|
|
||||||
for line in r.stdout.strip().split("\n"):
|
|
||||||
if not line.strip():
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
vid = d.get("id", "")
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=d.get("title", ""),
|
|
||||||
url=f"https://youtube.com/watch?v={vid}" if vid else "",
|
|
||||||
snippet=(
|
|
||||||
f"👤 {d.get('channel', '?')} · "
|
|
||||||
f"⏱ {d.get('duration_string', '?')} · "
|
|
||||||
f"👁 {d.get('view_count', '?')}"
|
|
||||||
),
|
|
||||||
extra={
|
|
||||||
"channel": d.get("channel"),
|
|
||||||
"duration": d.get("duration_string"),
|
|
||||||
"view_count": d.get("view_count"),
|
|
||||||
},
|
|
||||||
))
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
continue
|
|
||||||
return results
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _get_info(self, url: str) -> dict:
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["yt-dlp", "--dump-json", "--no-download", url],
|
|
||||||
capture_output=True, text=True, timeout=30,
|
|
||||||
)
|
|
||||||
if r.returncode == 0:
|
|
||||||
return json.loads(r.stdout)
|
|
||||||
except (subprocess.TimeoutExpired, json.JSONDecodeError):
|
|
||||||
pass
|
|
||||||
return {}
|
|
||||||
|
|
||||||
def _get_subtitles(self, url: str, tmpdir: str) -> str:
|
|
||||||
try:
|
|
||||||
subprocess.run(
|
|
||||||
["yt-dlp", "--write-auto-sub", "--write-sub",
|
|
||||||
"--sub-lang", "en,zh-Hans,zh",
|
|
||||||
"--skip-download", "--sub-format", "vtt",
|
|
||||||
"-o", f"{tmpdir}/%(id)s.%(ext)s", url],
|
|
||||||
capture_output=True, text=True, timeout=30,
|
|
||||||
)
|
|
||||||
for f in Path(tmpdir).glob("*.vtt"):
|
|
||||||
text = f.read_text(errors="replace")
|
|
||||||
lines = []
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if not line or line.startswith("WEBVTT") or "-->" in line or line.isdigit():
|
|
||||||
continue
|
|
||||||
if line not in lines[-1:]:
|
|
||||||
lines.append(line)
|
|
||||||
return "\n".join(lines)
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
pass
|
|
||||||
return ""
|
|
||||||
|
|||||||
+1119
-514
File diff suppressed because it is too large
Load Diff
+15
-8
@@ -21,8 +21,7 @@ class Config:
|
|||||||
# Feature → required config keys
|
# Feature → required config keys
|
||||||
FEATURE_REQUIREMENTS = {
|
FEATURE_REQUIREMENTS = {
|
||||||
"exa_search": ["exa_api_key"],
|
"exa_search": ["exa_api_key"],
|
||||||
"reddit_proxy": ["reddit_proxy"],
|
"twitter_xreach": ["twitter_auth_token", "twitter_ct0"], # legacy key name; used by bird CLI
|
||||||
"twitter_bird": ["twitter_auth_token", "twitter_ct0"],
|
|
||||||
"groq_whisper": ["groq_api_key"],
|
"groq_whisper": ["groq_api_key"],
|
||||||
"github_token": ["github_token"],
|
"github_token": ["github_token"],
|
||||||
}
|
}
|
||||||
@@ -41,7 +40,7 @@ class Config:
|
|||||||
def load(self):
|
def load(self):
|
||||||
"""Load config from YAML file."""
|
"""Load config from YAML file."""
|
||||||
if self.config_path.exists():
|
if self.config_path.exists():
|
||||||
with open(self.config_path, "r") as f:
|
with open(self.config_path, "r", encoding="utf-8") as f:
|
||||||
self.data = yaml.safe_load(f) or {}
|
self.data = yaml.safe_load(f) or {}
|
||||||
else:
|
else:
|
||||||
self.data = {}
|
self.data = {}
|
||||||
@@ -49,14 +48,22 @@ class Config:
|
|||||||
def save(self):
|
def save(self):
|
||||||
"""Save config to YAML file."""
|
"""Save config to YAML file."""
|
||||||
self._ensure_dir()
|
self._ensure_dir()
|
||||||
with open(self.config_path, "w") as f:
|
# Create file with restricted permissions from the start to avoid
|
||||||
yaml.dump(self.data, f, default_flow_style=False, allow_unicode=True)
|
# a race window where credentials are briefly world-readable.
|
||||||
# Restrict permissions — config may contain credentials
|
|
||||||
try:
|
try:
|
||||||
import stat
|
import stat
|
||||||
self.config_path.chmod(stat.S_IRUSR | stat.S_IWUSR) # 0o600
|
fd = os.open(
|
||||||
|
str(self.config_path),
|
||||||
|
os.O_WRONLY | os.O_CREAT | os.O_TRUNC,
|
||||||
|
stat.S_IRUSR | stat.S_IWUSR, # 0o600
|
||||||
|
)
|
||||||
|
with os.fdopen(fd, "w", encoding="utf-8") as f:
|
||||||
|
yaml.dump(self.data, f, default_flow_style=False, allow_unicode=True)
|
||||||
except OSError:
|
except OSError:
|
||||||
pass # Windows or permission edge cases
|
# Fallback for Windows or other edge cases where os.open flags
|
||||||
|
# are not fully supported.
|
||||||
|
with open(self.config_path, "w", encoding="utf-8") as f:
|
||||||
|
yaml.dump(self.data, f, default_flow_style=False, allow_unicode=True)
|
||||||
|
|
||||||
def get(self, key: str, default: Any = None) -> Any:
|
def get(self, key: str, default: Any = None) -> Any:
|
||||||
"""Get a config value. Also checks environment variables (uppercase)."""
|
"""Get a config value. Also checks environment variables (uppercase)."""
|
||||||
|
|||||||
+124
-23
@@ -32,6 +32,12 @@ PLATFORM_SPECS = [
|
|||||||
"cookies": ["SESSDATA", "bili_jct"],
|
"cookies": ["SESSDATA", "bili_jct"],
|
||||||
"config_key": "bilibili",
|
"config_key": "bilibili",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"name": "Xueqiu",
|
||||||
|
"domains": [".xueqiu.com", "xueqiu.com"],
|
||||||
|
"cookies": None, # grab all — xq_a_token + session cookies required
|
||||||
|
"config_key": "xueqiu",
|
||||||
|
},
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
@@ -46,36 +52,66 @@ def extract_all(browser: str = "chrome") -> Dict[str, dict]:
|
|||||||
"bilibili": {"SESSDATA": "xxx", "bili_jct": "yyy"},
|
"bilibili": {"SESSDATA": "xxx", "bili_jct": "yyy"},
|
||||||
}
|
}
|
||||||
"""
|
"""
|
||||||
|
# Try rookiepy first (Rust-based, more stable), fallback to browser_cookie3
|
||||||
|
use_rookiepy = False
|
||||||
try:
|
try:
|
||||||
import browser_cookie3
|
import rookiepy
|
||||||
|
use_rookiepy = True
|
||||||
except ImportError:
|
except ImportError:
|
||||||
raise RuntimeError(
|
try:
|
||||||
"browser_cookie3 not installed. Run: pip install browser-cookie3"
|
import browser_cookie3
|
||||||
)
|
except ImportError:
|
||||||
|
raise RuntimeError(
|
||||||
# Get browser cookie jar
|
"Cookie extraction requires rookiepy or browser_cookie3.\n"
|
||||||
browser_funcs = {
|
"Install: pip install rookiepy (recommended)\n"
|
||||||
"chrome": browser_cookie3.chrome,
|
" or: pip install browser-cookie3"
|
||||||
"firefox": browser_cookie3.firefox,
|
)
|
||||||
"edge": browser_cookie3.edge,
|
|
||||||
"brave": browser_cookie3.brave,
|
|
||||||
"opera": browser_cookie3.opera,
|
|
||||||
}
|
|
||||||
|
|
||||||
browser = browser.lower()
|
browser = browser.lower()
|
||||||
if browser not in browser_funcs:
|
supported = ["chrome", "firefox", "edge", "brave", "opera"]
|
||||||
|
if browser not in supported:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
f"Unsupported browser: {browser}. "
|
f"Unsupported browser: {browser}. Supported: {', '.join(supported)}"
|
||||||
f"Supported: {', '.join(browser_funcs.keys())}"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
try:
|
if use_rookiepy:
|
||||||
cookie_jar = browser_funcs[browser]()
|
# rookiepy returns list of dicts with name/value/domain/path keys
|
||||||
except Exception as e:
|
try:
|
||||||
raise RuntimeError(
|
browser_funcs = {
|
||||||
f"Could not read {browser} cookies: {e}\n"
|
"chrome": rookiepy.chrome,
|
||||||
f"Make sure {browser} is closed and you have permission to read its data."
|
"firefox": rookiepy.firefox,
|
||||||
)
|
"edge": rookiepy.edge,
|
||||||
|
"brave": rookiepy.brave,
|
||||||
|
"opera": rookiepy.opera,
|
||||||
|
}
|
||||||
|
raw_cookies = browser_funcs[browser]()
|
||||||
|
# Wrap into objects with .name, .value, .domain for compatibility
|
||||||
|
class _Cookie:
|
||||||
|
def __init__(self, d):
|
||||||
|
self.name = d.get("name", "")
|
||||||
|
self.value = d.get("value", "")
|
||||||
|
self.domain = d.get("domain", "")
|
||||||
|
cookie_jar = [_Cookie(c) for c in raw_cookies]
|
||||||
|
except Exception as e:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Could not read {browser} cookies via rookiepy: {e}\n"
|
||||||
|
f"Make sure {browser} is closed and you have permission."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
browser_funcs = {
|
||||||
|
"chrome": browser_cookie3.chrome,
|
||||||
|
"firefox": browser_cookie3.firefox,
|
||||||
|
"edge": browser_cookie3.edge,
|
||||||
|
"brave": browser_cookie3.brave,
|
||||||
|
"opera": browser_cookie3.opera,
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
cookie_jar = browser_funcs[browser]()
|
||||||
|
except Exception as e:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Could not read {browser} cookies: {e}\n"
|
||||||
|
f"Make sure {browser} is closed and you have permission."
|
||||||
|
)
|
||||||
|
|
||||||
results = {}
|
results = {}
|
||||||
|
|
||||||
@@ -112,6 +148,57 @@ def extract_all(browser: str = "chrome") -> Dict[str, dict]:
|
|||||||
return results
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def _sync_xfetch_session(auth_token: str, ct0: str) -> None:
|
||||||
|
"""Sync Twitter credentials to ~/.config/xfetch/session.json (legacy xreach compat)."""
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
|
||||||
|
try:
|
||||||
|
xfetch_dir = os.path.join(os.path.expanduser("~"), ".config", "xfetch")
|
||||||
|
os.makedirs(xfetch_dir, exist_ok=True)
|
||||||
|
session_path = os.path.join(xfetch_dir, "session.json")
|
||||||
|
session_data: dict = {}
|
||||||
|
if os.path.exists(session_path):
|
||||||
|
try:
|
||||||
|
with open(session_path, "r", encoding="utf-8") as sf:
|
||||||
|
session_data = json.load(sf)
|
||||||
|
except (json.JSONDecodeError, OSError):
|
||||||
|
session_data = {}
|
||||||
|
session_data["authToken"] = auth_token
|
||||||
|
session_data["ct0"] = ct0
|
||||||
|
with open(session_path, "w", encoding="utf-8") as sf:
|
||||||
|
json.dump(session_data, sf, indent=2)
|
||||||
|
os.chmod(session_path, 0o600)
|
||||||
|
except Exception:
|
||||||
|
# Non-fatal: agent-reach config is the source of truth, xfetch sync is best-effort
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _sync_bird_env(auth_token: str, ct0: str) -> None:
|
||||||
|
"""Write Twitter credentials to ~/.config/bird/credentials.env for bird CLI.
|
||||||
|
|
||||||
|
bird reads AUTH_TOKEN and CT0 from environment variables. This writes a
|
||||||
|
shell-sourceable file so users can `source ~/.config/bird/credentials.env`.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
|
||||||
|
try:
|
||||||
|
bird_dir = os.path.join(os.path.expanduser("~"), ".config", "bird")
|
||||||
|
os.makedirs(bird_dir, exist_ok=True)
|
||||||
|
env_path = os.path.join(bird_dir, "credentials.env")
|
||||||
|
with open(env_path, "w", encoding="utf-8") as f:
|
||||||
|
f.write(f'AUTH_TOKEN="{auth_token}"\n')
|
||||||
|
f.write(f'CT0="{ct0}"\n')
|
||||||
|
os.chmod(env_path, 0o600)
|
||||||
|
except Exception:
|
||||||
|
# Non-fatal: agent-reach config is the source of truth, bird env sync is best-effort
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# Alias for callers expecting the name _sync_bird_credentials
|
||||||
|
_sync_bird_credentials = _sync_bird_env
|
||||||
|
|
||||||
|
|
||||||
def configure_from_browser(browser: str, config) -> List[Tuple[str, bool, str]]:
|
def configure_from_browser(browser: str, config) -> List[Tuple[str, bool, str]]:
|
||||||
"""
|
"""
|
||||||
Extract cookies and configure all found platforms.
|
Extract cookies and configure all found platforms.
|
||||||
@@ -136,6 +223,8 @@ def configure_from_browser(browser: str, config) -> List[Tuple[str, bool, str]]:
|
|||||||
if "auth_token" in tc and "ct0" in tc:
|
if "auth_token" in tc and "ct0" in tc:
|
||||||
config.set("twitter_auth_token", tc["auth_token"])
|
config.set("twitter_auth_token", tc["auth_token"])
|
||||||
config.set("twitter_ct0", tc["ct0"])
|
config.set("twitter_ct0", tc["ct0"])
|
||||||
|
# Legacy sync (best-effort)
|
||||||
|
_sync_xfetch_session(tc["auth_token"], tc["ct0"])
|
||||||
results_list.append(("Twitter/X", True, "auth_token + ct0"))
|
results_list.append(("Twitter/X", True, "auth_token + ct0"))
|
||||||
else:
|
else:
|
||||||
found = ", ".join(tc.keys())
|
found = ", ".join(tc.keys())
|
||||||
@@ -163,4 +252,16 @@ def configure_from_browser(browser: str, config) -> List[Tuple[str, bool, str]]:
|
|||||||
results_list.append(("Bilibili", False,
|
results_list.append(("Bilibili", False,
|
||||||
f"No SESSDATA found. Make sure you're logged into bilibili.com in {browser}."))
|
f"No SESSDATA found. Make sure you're logged into bilibili.com in {browser}."))
|
||||||
|
|
||||||
|
if "xueqiu" in extracted:
|
||||||
|
cookie_str = extracted["xueqiu"].get("cookie_string", "")
|
||||||
|
# Only save if xq_a_token is present — anonymous cookies are useless
|
||||||
|
if cookie_str and "xq_a_token" in cookie_str:
|
||||||
|
config.set("xueqiu_cookie", cookie_str)
|
||||||
|
n_cookies = len(cookie_str.split(";"))
|
||||||
|
results_list.append(("Xueqiu", True, f"{n_cookies} cookies (含 xq_a_token)"))
|
||||||
|
elif cookie_str:
|
||||||
|
results_list.append(("Xueqiu", False,
|
||||||
|
f"找到 {len(cookie_str.split(';'))} 个 Cookie 但缺少 xq_a_token,"
|
||||||
|
f"请先在 {browser} 中登录 xueqiu.com"))
|
||||||
|
|
||||||
return results_list
|
return results_list
|
||||||
|
|||||||
+16
-116
@@ -1,126 +1,36 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
AgentReach — the unified entry point.
|
AgentReach — installer, doctor, and configuration tool.
|
||||||
|
|
||||||
Pure glue: routes URLs to the right channel, routes searches to the right engine.
|
Agent Reach helps AI agents install and configure upstream platform tools
|
||||||
Every channel is a thin wrapper around an external tool. Swap any backend anytime.
|
(bird CLI, yt-dlp, mcporter, gh CLI, etc.). After installation, agents
|
||||||
|
call the upstream tools directly — no wrapper layer needed.
|
||||||
|
|
||||||
Usage:
|
Usage:
|
||||||
from agent_reach import AgentReach
|
from agent_reach.doctor import check_all, format_report
|
||||||
|
from agent_reach.config import Config
|
||||||
|
|
||||||
eyes = AgentReach()
|
config = Config()
|
||||||
content = await eyes.read("https://github.com/openai/gpt-4")
|
results = check_all(config)
|
||||||
results = await eyes.search("AI agent framework")
|
print(format_report(results))
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
from typing import Dict, Optional
|
||||||
from typing import Any, Dict, List, Optional
|
|
||||||
|
|
||||||
from agent_reach.config import Config
|
from agent_reach.config import Config
|
||||||
from agent_reach.channels import get_channel_for_url, get_channel, get_all_channels
|
|
||||||
|
|
||||||
|
|
||||||
class AgentReach:
|
class AgentReach:
|
||||||
"""Give your AI Agent eyes to see the entire internet."""
|
"""Give your AI Agent eyes to see the entire internet.
|
||||||
|
|
||||||
|
This class provides health-check functionality.
|
||||||
|
For reading/searching, use the upstream tools directly
|
||||||
|
(see SKILL.md for commands).
|
||||||
|
"""
|
||||||
|
|
||||||
def __init__(self, config: Optional[Config] = None):
|
def __init__(self, config: Optional[Config] = None):
|
||||||
self.config = config or Config()
|
self.config = config or Config()
|
||||||
|
|
||||||
# ── Reading ─────────────────────────────────────────
|
|
||||||
|
|
||||||
async def read(self, url: str) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Read content from any URL. Auto-detects platform.
|
|
||||||
|
|
||||||
Supported: Web, GitHub, Reddit, Twitter, YouTube,
|
|
||||||
Bilibili, RSS, and more.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict with title, content, url, author, platform, etc.
|
|
||||||
"""
|
|
||||||
if not url.startswith(("http://", "https://")):
|
|
||||||
url = f"https://{url}"
|
|
||||||
|
|
||||||
channel = get_channel_for_url(url)
|
|
||||||
result = await channel.read(url, config=self.config)
|
|
||||||
return result.to_dict()
|
|
||||||
|
|
||||||
async def read_batch(self, urls: List[str]) -> List[Dict[str, Any]]:
|
|
||||||
"""Read multiple URLs concurrently."""
|
|
||||||
tasks = [self.read(url) for url in urls]
|
|
||||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
|
||||||
return [r for r in results if not isinstance(r, Exception)]
|
|
||||||
|
|
||||||
def detect_platform(self, url: str) -> str:
|
|
||||||
"""Detect what platform a URL belongs to."""
|
|
||||||
channel = get_channel_for_url(url)
|
|
||||||
return channel.name
|
|
||||||
|
|
||||||
# ── Searching ───────────────────────────────────────
|
|
||||||
|
|
||||||
async def search(self, query: str, num_results: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Semantic web search via Exa."""
|
|
||||||
ch = get_channel("exa_search")
|
|
||||||
results = await ch.search(query, config=self.config, limit=num_results)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_reddit(self, query: str, subreddit: Optional[str] = None, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Reddit via Exa (bypasses IP blocks)."""
|
|
||||||
ch = get_channel("exa_search")
|
|
||||||
q = f"site:reddit.com/r/{subreddit} {query}" if subreddit else f"site:reddit.com {query}"
|
|
||||||
results = await ch.search(q, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_github(self, query: str, language: Optional[str] = None, limit: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Search GitHub repositories."""
|
|
||||||
ch = get_channel("github")
|
|
||||||
results = await ch.search(query, config=self.config, language=language, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_twitter(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Twitter. Uses bird CLI if available, else Exa."""
|
|
||||||
ch = get_channel("twitter")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_youtube(self, query: str, limit: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Search YouTube via yt-dlp."""
|
|
||||||
ch = get_channel("youtube")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_bilibili(self, query: str, limit: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Bilibili. Tries yt-dlp first, falls back to Exa."""
|
|
||||||
ch = get_channel("bilibili")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_xhs(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search XiaoHongShu via mcporter."""
|
|
||||||
ch = get_channel("xiaohongshu")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_instagram(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Instagram via Exa."""
|
|
||||||
ch = get_channel("instagram")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_linkedin(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search LinkedIn via MCP or Exa."""
|
|
||||||
ch = get_channel("linkedin")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_bosszhipin(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Boss直聘 via MCP or Exa."""
|
|
||||||
ch = get_channel("bosszhipin")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
# ── Health ──────────────────────────────────────────
|
|
||||||
|
|
||||||
def doctor(self) -> Dict[str, dict]:
|
def doctor(self) -> Dict[str, dict]:
|
||||||
"""Check all channel availability."""
|
"""Check all channel availability."""
|
||||||
from agent_reach.doctor import check_all
|
from agent_reach.doctor import check_all
|
||||||
@@ -130,13 +40,3 @@ class AgentReach:
|
|||||||
"""Get formatted health report."""
|
"""Get formatted health report."""
|
||||||
from agent_reach.doctor import check_all, format_report
|
from agent_reach.doctor import check_all, format_report
|
||||||
return format_report(check_all(self.config))
|
return format_report(check_all(self.config))
|
||||||
|
|
||||||
# ── Sync wrappers ───────────────────────────────────
|
|
||||||
|
|
||||||
def read_sync(self, url: str) -> Dict[str, Any]:
|
|
||||||
"""Synchronous version of read()."""
|
|
||||||
return asyncio.run(self.read(url))
|
|
||||||
|
|
||||||
def search_sync(self, query: str, num_results: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Synchronous version of search()."""
|
|
||||||
return asyncio.run(self.search(query, num_results))
|
|
||||||
|
|||||||
+60
-29
@@ -25,53 +25,84 @@ def check_all(config: Config) -> Dict[str, dict]:
|
|||||||
|
|
||||||
|
|
||||||
def format_report(results: Dict[str, dict]) -> str:
|
def format_report(results: Dict[str, dict]) -> str:
|
||||||
"""Format results as a readable text report."""
|
"""Format results as a readable text report (with Rich markup)."""
|
||||||
|
try:
|
||||||
|
from rich.markup import escape
|
||||||
|
except ImportError:
|
||||||
|
escape = lambda x: x
|
||||||
|
|
||||||
lines = []
|
lines = []
|
||||||
lines.append("👁️ Agent Reach 状态")
|
lines.append("[bold cyan]Agent Reach 状态[/bold cyan]")
|
||||||
lines.append("=" * 40)
|
lines.append("[cyan]" + "=" * 40 + "[/cyan]")
|
||||||
|
|
||||||
ok_count = sum(1 for r in results.values() if r["status"] == "ok")
|
ok_count = sum(1 for r in results.values() if r["status"] == "ok")
|
||||||
total = len(results)
|
total = len(results)
|
||||||
|
|
||||||
# Tier 0 — zero config
|
# Tier 0 — zero config
|
||||||
lines.append("")
|
lines.append("")
|
||||||
lines.append("✅ 装好即用:")
|
lines.append("[bold]✅ 装好即用:[/bold]")
|
||||||
for key, r in results.items():
|
for key, r in results.items():
|
||||||
if r["tier"] == 0:
|
if r["tier"] == 0:
|
||||||
|
name_msg = f"[bold]{escape(r['name'])}[/bold] — {escape(r['message'])}"
|
||||||
if r["status"] == "ok":
|
if r["status"] == "ok":
|
||||||
lines.append(f" ✅ {r['name']} — {r['message']}")
|
lines.append(f" [green]✅[/green] {name_msg}")
|
||||||
elif r["status"] == "warn":
|
elif r["status"] == "warn":
|
||||||
lines.append(f" ⚠️ {r['name']} — {r['message']}")
|
lines.append(f" [yellow][!][/yellow] {name_msg}")
|
||||||
elif r["status"] in ("off", "error"):
|
elif r["status"] in ("off", "error"):
|
||||||
lines.append(f" ❌ {r['name']} — {r['message']}")
|
lines.append(f" [red][X][/red] {name_msg}")
|
||||||
|
|
||||||
# Tier 1 — needs free key
|
# Tier 1 — needs free key / login
|
||||||
tier1 = {k: r for k, r in results.items() if r["tier"] == 1}
|
tier1 = {k: r for k, r in results.items() if r["tier"] == 1}
|
||||||
if tier1:
|
tier1_active = {k: r for k, r in tier1.items() if r["status"] == "ok"}
|
||||||
|
tier1_inactive = {k: r for k, r in tier1.items() if r["status"] != "ok"}
|
||||||
|
if tier1_active:
|
||||||
lines.append("")
|
lines.append("")
|
||||||
lines.append("🔍 搜索(mcporter 即可解锁):")
|
lines.append("[bold]可选渠道(已安装):[/bold]")
|
||||||
for key, r in tier1.items():
|
for key, r in tier1_active.items():
|
||||||
if r["status"] == "ok":
|
name_msg = f"[bold]{escape(r['name'])}[/bold] — {escape(r['message'])}"
|
||||||
lines.append(f" ✅ {r['name']} — {r['message']}")
|
lines.append(f" [green]✅[/green] {name_msg}")
|
||||||
else:
|
|
||||||
lines.append(f" ⬜ {r['name']} — {r['message']}")
|
|
||||||
|
|
||||||
# Tier 2 — optional setup
|
# Tier 2 — optional complex setup
|
||||||
tier2 = {k: r for k, r in results.items() if r["tier"] == 2}
|
tier2 = {k: r for k, r in results.items() if r["tier"] == 2}
|
||||||
if tier2:
|
tier2_active = {k: r for k, r in tier2.items() if r["status"] == "ok"}
|
||||||
lines.append("")
|
tier2_inactive = {k: r for k, r in tier2.items() if r["status"] != "ok"}
|
||||||
lines.append("🔧 配置后可用:")
|
if tier2_active:
|
||||||
for key, r in tier2.items():
|
if not tier1_active:
|
||||||
if r["status"] == "ok":
|
lines.append("")
|
||||||
lines.append(f" ✅ {r['name']} — {r['message']}")
|
lines.append("[bold]可选渠道(已安装):[/bold]")
|
||||||
elif r["status"] == "warn":
|
for key, r in tier2_active.items():
|
||||||
lines.append(f" ⚠️ {r['name']} — {r['message']}")
|
name_msg = f"[bold]{escape(r['name'])}[/bold] — {escape(r['message'])}"
|
||||||
else:
|
lines.append(f" [green]✅[/green] {name_msg}")
|
||||||
lines.append(f" ⬜ {r['name']} — {r['message']}")
|
|
||||||
|
|
||||||
lines.append("")
|
lines.append("")
|
||||||
lines.append(f"状态:{ok_count}/{total} 个渠道可用")
|
status_color = "green" if ok_count == total else ("yellow" if ok_count > 0 else "red")
|
||||||
if ok_count < total:
|
lines.append(f"状态:[{status_color}]{ok_count}/{total}[/{status_color}] 个渠道可用")
|
||||||
lines.append("运行 `agent-reach setup` 解锁更多渠道")
|
|
||||||
|
# Summarize inactive optional channels in one line instead of listing each
|
||||||
|
all_inactive = list(tier1_inactive.values()) + list(tier2_inactive.values())
|
||||||
|
if all_inactive:
|
||||||
|
names = [r["name"] for r in all_inactive]
|
||||||
|
lines.append(
|
||||||
|
f"还有 {len(names)} 个可选渠道可以解锁({'、'.join(names)}),"
|
||||||
|
"告诉你的 Agent「帮我装 XXX」即可"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Security check: config file permissions (Unix only)
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
import sys
|
||||||
|
|
||||||
|
config_path = Config.CONFIG_DIR / "config.yaml"
|
||||||
|
if config_path.exists() and sys.platform != "win32":
|
||||||
|
try:
|
||||||
|
mode = config_path.stat().st_mode
|
||||||
|
if mode & (stat.S_IRGRP | stat.S_IROTH):
|
||||||
|
lines.append("")
|
||||||
|
lines.append(
|
||||||
|
"[bold red][!] 安全提示:config.yaml 权限过宽(其他用户可读)[/bold red]"
|
||||||
|
)
|
||||||
|
lines.append(" 修复:chmod 600 ~/.agent-reach/config.yaml")
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|||||||
@@ -22,8 +22,8 @@ mcporter config add exa https://mcp.exa.ai/mcp
|
|||||||
|
|
||||||
### 3. 验证
|
### 3. 验证
|
||||||
```bash
|
```bash
|
||||||
agent-reach doctor | grep "Web Search"
|
agent-reach doctor | grep "Search"
|
||||||
agent-reach search "test" -n 1
|
mcporter call 'exa.web_search_exa(query: "test", numResults: 1)'
|
||||||
```
|
```
|
||||||
|
|
||||||
## 需要用户手动做的步骤
|
## 需要用户手动做的步骤
|
||||||
|
|||||||
@@ -1,60 +1,45 @@
|
|||||||
# Reddit 代理配置指南
|
# Reddit 配置指南
|
||||||
|
|
||||||
## 功能说明
|
## 功能说明
|
||||||
Reddit 封锁了大量服务器 IP,直接访问会返回 403。需要 ISP 代理(Residential/ISP Proxy)才能读取完整的 Reddit 帖子和评论。
|
|
||||||
|
|
||||||
**注意**:即使没有代理,也可以通过 Exa 搜索找到 Reddit 内容(只需要配 Exa API key)。代理只是用于读取完整帖子+评论。
|
Reddit 封锁了几乎所有非浏览器的直接访问(包括数据中心和 ISP 代理 IP),JSON API 返回 403。
|
||||||
|
|
||||||
|
Agent Reach 通过 **Exa** 实现 Reddit 的搜索和阅读功能:
|
||||||
|
- **搜索**:`web_search_exa` + `includeDomains: ["reddit.com"]`
|
||||||
|
- **阅读完整帖子+评论**:`crawling_exa` 读取 Reddit URL 的缓存内容
|
||||||
|
|
||||||
|
免费,无需代理,无需 API Key。
|
||||||
|
|
||||||
## Agent 可自动完成的步骤
|
## Agent 可自动完成的步骤
|
||||||
|
|
||||||
1. 检查当前状态:
|
1. 检查 Exa 是否可用:
|
||||||
```bash
|
```bash
|
||||||
agent-reach doctor | grep "Reddit"
|
agent-reach doctor | grep -E "Reddit|Exa"
|
||||||
```
|
```
|
||||||
|
|
||||||
2. 如果用户提供了代理,测试连通性:
|
2. 如果 Exa 未安装,自动安装:
|
||||||
```bash
|
```bash
|
||||||
curl -s --proxy "用户提供的代理" \
|
npm install -g mcporter
|
||||||
-H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" \
|
mcporter config add exa https://mcp.exa.ai/mcp
|
||||||
"https://www.reddit.com/r/test.json?limit=1" \
|
|
||||||
-o /dev/null -w "%{http_code}"
|
|
||||||
```
|
```
|
||||||
返回 200 = 可用,403 = 代理被封,其他 = 配置错误
|
|
||||||
|
|
||||||
3. 写入配置:
|
或一键安装:
|
||||||
```python
|
```bash
|
||||||
from agent_reach.config import Config
|
agent-reach install --env=auto
|
||||||
c = Config()
|
```
|
||||||
c.set("reddit_proxy", "http://用户名:密码@IP:端口")
|
|
||||||
|
## 使用示例
|
||||||
|
|
||||||
|
搜索 Reddit 内容:
|
||||||
|
```bash
|
||||||
|
mcporter call 'exa.web_search_exa(query: "python best practices", numResults: 5, includeDomains: ["reddit.com"])'
|
||||||
|
```
|
||||||
|
|
||||||
|
阅读完整帖子和评论:
|
||||||
|
```bash
|
||||||
|
mcporter call 'exa.crawling_exa(urls: ["https://www.reddit.com/r/python/comments/xxx/"], maxCharacters: 10000)'
|
||||||
```
|
```
|
||||||
|
|
||||||
## 需要用户手动做的步骤
|
## 需要用户手动做的步骤
|
||||||
|
|
||||||
请告诉用户:
|
无。Exa 通过 `agent-reach install --env=auto` 自动配置。
|
||||||
|
|
||||||
> 要完整阅读 Reddit 帖子和评论,需要一个 ISP 代理(约 $3-10/月)。
|
|
||||||
>
|
|
||||||
> 推荐代理服务商(选一个就行):
|
|
||||||
> 1. **Smartproxy** (https://smartproxy.com) — ISP 代理,按流量计费
|
|
||||||
> 2. **Bright Data** (https://brightdata.com) — 大厂,ISP 代理
|
|
||||||
> 3. **IPRoyal** (https://iproyal.com) — 便宜,适合入门
|
|
||||||
> 4. **ProxyEmpire** (https://proxyempire.io) — 有 Reddit 专用代理
|
|
||||||
>
|
|
||||||
> 购买时选择:
|
|
||||||
> - 类型:**ISP Proxy**(不要选 Datacenter,会被封)
|
|
||||||
> - 地区:**美国**
|
|
||||||
> - 协议:**HTTP**
|
|
||||||
>
|
|
||||||
> 购买后你会得到一个代理地址,格式类似:
|
|
||||||
> `http://用户名:密码@IP地址:端口号`
|
|
||||||
>
|
|
||||||
> 把这个地址发给我就行。
|
|
||||||
>
|
|
||||||
> ⚠️ 如果不想花钱,可以跳过。我仍然可以通过搜索引擎找到 Reddit 上的内容,只是不能读完整的帖子和评论。
|
|
||||||
|
|
||||||
## Agent 收到代理后的操作
|
|
||||||
|
|
||||||
1. 测试代理:用 curl 测试 reddit.com 是否返回 200
|
|
||||||
2. 如果成功,写入配置:`config.set("reddit_proxy", proxy_url)`
|
|
||||||
3. 反馈:"✅ Reddit 完整阅读已开启!现在我可以读取 Reddit 帖子和所有评论了。"
|
|
||||||
4. 如果失败,告诉用户:"❌ 这个代理无法访问 Reddit,请检查代理是否有效,或换一个试试。"
|
|
||||||
|
|||||||
@@ -1,67 +1,81 @@
|
|||||||
# Twitter 高级功能配置指南(bird CLI)
|
# Twitter 高级功能配置指南(bird CLI)
|
||||||
|
|
||||||
## 功能说明
|
Twitter 基础阅读通过 Jina Reader 免费可用,无需配置。
|
||||||
基础 Twitter 功能(搜索+读单条推文)无需配置,开箱即用。
|
|
||||||
|
|
||||||
高级功能需要 bird CLI:
|
高级功能需要 bird CLI(@steipete/bird):
|
||||||
- 查看用户时间线
|
|
||||||
- 深度搜索(更精确、更多结果)
|
- 搜索推文(`bird search`)
|
||||||
- 读取完整线程(thread)
|
- 读取完整推文和对话链(`bird read`、`bird thread`)
|
||||||
- 查看关注列表推文
|
- 用户时间线(`bird user-tweets`)
|
||||||
|
|
||||||
bird 是免费开源工具(npm 包 @steipete/bird),但需要你的 Twitter 账号 cookie。
|
bird 是免费开源工具(npm 包 @steipete/bird),但需要你的 Twitter 账号 cookie。
|
||||||
|
|
||||||
## Agent 可自动完成的步骤
|
## 快速配置
|
||||||
|
|
||||||
1. 检查 bird 是否安装:
|
1. 检查 bird 是否安装:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
which bird && echo "installed" || echo "not installed"
|
which bird && echo "installed" || echo "not installed"
|
||||||
```
|
```
|
||||||
|
|
||||||
2. 安装 bird:
|
2. 安装 bird:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm install -g @steipete/bird
|
npm install -g @steipete/bird
|
||||||
```
|
```
|
||||||
|
|
||||||
3. 测试(需要先配置 cookie):
|
> 备选包:`npm install -g @connormartin/bird`
|
||||||
|
|
||||||
|
3. 测试是否配置好:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
AUTH_TOKEN="xxx" CT0="yyy" bird search "test" -n 1
|
AUTH_TOKEN="xxx" CT0="yyy" bird search "test" -n 1
|
||||||
```
|
```
|
||||||
|
|
||||||
4. 用 agent-reach 配置 cookie(推荐):
|
## 获取 Cookie(Cookie-Editor 方式,推荐)
|
||||||
|
|
||||||
|
1. 安装 [Cookie-Editor](https://cookie-editor.com/) 浏览器扩展
|
||||||
|
2. 登录 x.com
|
||||||
|
3. 点击 Cookie-Editor 图标 → Export → 复制全部
|
||||||
|
4. 运行配置命令:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
|
agent-reach configure twitter-cookies "粘贴的 cookie JSON"
|
||||||
```
|
```
|
||||||
|
|
||||||
## 需要用户手动做的步骤
|
这会自动提取 `auth_token` 和 `ct0`,并写入环境变量。
|
||||||
|
|
||||||
请告诉用户:
|
## 手动设置 Cookie
|
||||||
|
|
||||||
> Twitter 高级功能需要你的 Twitter 账号 cookie(完全免费)。
|
如果你已经知道 `auth_token` 和 `ct0`:
|
||||||
>
|
|
||||||
> **最简单的方式:**
|
|
||||||
> 1. 安装 Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm)
|
|
||||||
> 2. 打开 https://x.com 并确保已登录
|
|
||||||
> 3. 点击 Cookie-Editor 插件图标 → Export → Header String
|
|
||||||
> 4. 把导出的内容发给我
|
|
||||||
>
|
|
||||||
> **手动方式:**
|
|
||||||
> 1. 用 Chrome 打开 https://x.com 并确保你已登录
|
|
||||||
> 2. 按 **F12** 打开开发者工具(Mac 按 Cmd+Option+I)
|
|
||||||
> 3. 点击顶部的 **Application**(应用)标签
|
|
||||||
> 4. 左侧找到 **Cookies** → **https://x.com**
|
|
||||||
> 5. 在列表中找到以下两个值,双击复制:
|
|
||||||
> - **auth_token** — 一串字母数字
|
|
||||||
> - **ct0** — 一串字母数字
|
|
||||||
> 6. 把这两个值发给我
|
|
||||||
>
|
|
||||||
> ⚠️ 这些 cookie 让我能以你的身份读取推文(只读)。我不会发推、点赞或做任何操作。
|
|
||||||
> ⚠️ cookie 大约 1-3 个月会过期,届时需要重新导出。
|
|
||||||
|
|
||||||
## Agent 收到 cookie 后的操作
|
|
||||||
|
|
||||||
1. 安装 bird(如果没装):`npm install -g @steipete/bird`
|
1. 安装 bird(如果没装):`npm install -g @steipete/bird`
|
||||||
2. 配置 cookie:`agent-reach configure twitter-cookies "粘贴的内容"`
|
|
||||||
3. 测试:运行 `agent-reach doctor` 确认 Twitter 状态
|
2. 设置环境变量:
|
||||||
4. 反馈:"✅ Twitter 高级功能已开启!现在可以搜索推文、查看时间线了。"
|
|
||||||
5. 如果失败:"❌ Cookie 无效或已过期,请重新导出。"
|
```bash
|
||||||
|
export AUTH_TOKEN="你的auth_token"
|
||||||
|
export CT0="你的ct0"
|
||||||
|
```
|
||||||
|
|
||||||
|
3. 测试:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bird search "test" -n 1
|
||||||
|
```
|
||||||
|
|
||||||
|
## 代理配置
|
||||||
|
|
||||||
|
> bird CLI 支持通过环境变量设置代理:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export HTTP_PROXY="http://user:pass@host:port"
|
||||||
|
export HTTPS_PROXY="http://user:pass@host:port"
|
||||||
|
bird search "test" -n 1
|
||||||
|
```
|
||||||
|
|
||||||
|
也可以使用全局代理工具:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
proxychains bird search "test" -n 1
|
||||||
|
```
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ playwright install chromium
|
|||||||
|
|
||||||
3. 安装完成后测试:
|
3. 安装完成后测试:
|
||||||
```bash
|
```bash
|
||||||
agent-reach read "https://mp.weixin.qq.com/s/一个测试链接"
|
curl -s "https://r.jina.ai/https://mp.weixin.qq.com/s/一个测试链接" -H "Accept: text/markdown"
|
||||||
```
|
```
|
||||||
|
|
||||||
## 需要用户手动做的步骤
|
## 需要用户手动做的步骤
|
||||||
|
|||||||
@@ -45,12 +45,17 @@ agent-reach doctor
|
|||||||
|
|
||||||
## 需要用户手动做的步骤
|
## 需要用户手动做的步骤
|
||||||
|
|
||||||
如果 doctor 显示"MCP 已连接但未登录":
|
如果 doctor 显示"MCP 已连接但未登录",需要导入 cookies:
|
||||||
|
|
||||||
> 小红书需要登录一次(之后会记住你的登录状态)。
|
> **推荐方式:Cookie-Editor 浏览器导出(最可靠)**
|
||||||
>
|
>
|
||||||
> 打开 http://localhost:18060 ,用手机小红书 App 扫描二维码登录。
|
> 1. 在 Chrome 中安装 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 扩展
|
||||||
> 登录后 cookie 会自动保存在 Docker 容器内,大约 1-3 个月有效。
|
> 2. 浏览器登录 xiaohongshu.com
|
||||||
|
> 3. 点击 Cookie-Editor 图标 → Export → Header String
|
||||||
|
> 4. 把导出的字符串发给 Agent,运行:`agent-reach configure xhs-cookies "导出的cookie字符串"`
|
||||||
|
>
|
||||||
|
> **注意**:`http://localhost:18060` 根路径可能返回 404,这是正常的——MCP 服务在 `/mcp` 路径。
|
||||||
|
> 不要依赖 QR 扫码登录,Docker 容器内的 QR 登录页面不一定可用,且 cookies 不会自动共享到 MCP 服务。
|
||||||
|
|
||||||
## 常见问题
|
## 常见问题
|
||||||
|
|
||||||
@@ -67,5 +72,25 @@ docker run -d \
|
|||||||
**Q: 服务器上小红书提示 IP 风险?**
|
**Q: 服务器上小红书提示 IP 风险?**
|
||||||
A: 加代理参数 `-e XHS_PROXY=http://user:pass@ip:port`,推荐住宅代理。
|
A: 加代理参数 `-e XHS_PROXY=http://user:pass@ip:port`,推荐住宅代理。
|
||||||
|
|
||||||
|
**Q: Docker 镜像不支持 ARM64 / Apple Silicon?**
|
||||||
|
A: 上游镜像暂无 ARM64 版本,两种解决办法:
|
||||||
|
|
||||||
|
方法一:使用 Rosetta 模拟运行(推荐,最简单)
|
||||||
|
```bash
|
||||||
|
docker run -d \
|
||||||
|
--name xiaohongshu-mcp \
|
||||||
|
-p 18060:18060 \
|
||||||
|
--platform linux/amd64 \
|
||||||
|
xpzouying/xiaohongshu-mcp
|
||||||
|
```
|
||||||
|
|
||||||
|
方法二:从源码编译原生 ARM64 版本
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/xpzouying/xiaohongshu-mcp
|
||||||
|
cd xiaohongshu-mcp
|
||||||
|
docker build -t xiaohongshu-mcp .
|
||||||
|
docker run -d --name xiaohongshu-mcp -p 18060:18060 xiaohongshu-mcp
|
||||||
|
```
|
||||||
|
|
||||||
**Q: 我不想用 Docker?**
|
**Q: 我不想用 Docker?**
|
||||||
A: 可以从源码编译:https://github.com/xpzouying/xiaohongshu-mcp
|
A: 可以从源码编译:https://github.com/xpzouying/xiaohongshu-mcp
|
||||||
|
|||||||
@@ -1,10 +1,11 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Agent Reach MCP Server — expose all capabilities as MCP tools.
|
Agent Reach MCP Server — expose doctor/status as MCP tool.
|
||||||
|
|
||||||
Run: python -m agent_reach.integrations.mcp_server
|
Run: python -m agent_reach.integrations.mcp_server
|
||||||
|
|
||||||
8 tools for any MCP-compatible AI Agent.
|
Agent Reach is an installer + doctor tool. For actual reading/searching,
|
||||||
|
agents should call upstream tools directly (bird, yt-dlp, mcporter, etc.).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
@@ -35,50 +36,15 @@ def create_server():
|
|||||||
@server.list_tools()
|
@server.list_tools()
|
||||||
async def list_tools():
|
async def list_tools():
|
||||||
return [
|
return [
|
||||||
Tool(name="read_url",
|
|
||||||
description="Read content from any URL. Supports: web, GitHub, Reddit, Twitter, YouTube, Bilibili, RSS.",
|
|
||||||
inputSchema={"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}),
|
|
||||||
Tool(name="read_batch",
|
|
||||||
description="Read multiple URLs concurrently.",
|
|
||||||
inputSchema={"type": "object", "properties": {"urls": {"type": "array", "items": {"type": "string"}}}, "required": ["urls"]}),
|
|
||||||
Tool(name="detect_platform",
|
|
||||||
description="Detect what platform a URL belongs to.",
|
|
||||||
inputSchema={"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}),
|
|
||||||
Tool(name="search",
|
|
||||||
description="Semantic web search via Exa.",
|
|
||||||
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "num_results": {"type": "integer", "default": 5}}, "required": ["query"]}),
|
|
||||||
Tool(name="search_reddit",
|
|
||||||
description="Search Reddit posts.",
|
|
||||||
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "subreddit": {"type": "string"}, "limit": {"type": "integer", "default": 10}}, "required": ["query"]}),
|
|
||||||
Tool(name="search_github",
|
|
||||||
description="Search GitHub repositories.",
|
|
||||||
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "language": {"type": "string"}, "limit": {"type": "integer", "default": 5}}, "required": ["query"]}),
|
|
||||||
Tool(name="search_twitter",
|
|
||||||
description="Search Twitter/X posts.",
|
|
||||||
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "limit": {"type": "integer", "default": 10}}, "required": ["query"]}),
|
|
||||||
Tool(name="get_status",
|
Tool(name="get_status",
|
||||||
description="Get Agent Reach status: which channels are active.",
|
description="Get Agent Reach status: which channels are installed and active.",
|
||||||
inputSchema={"type": "object", "properties": {}}),
|
inputSchema={"type": "object", "properties": {}}),
|
||||||
]
|
]
|
||||||
|
|
||||||
@server.call_tool()
|
@server.call_tool()
|
||||||
async def call_tool(name: str, arguments: dict):
|
async def call_tool(name: str, arguments: dict):
|
||||||
try:
|
try:
|
||||||
if name == "read_url":
|
if name == "get_status":
|
||||||
result = await eyes.read(arguments["url"])
|
|
||||||
elif name == "read_batch":
|
|
||||||
result = await eyes.read_batch(arguments["urls"])
|
|
||||||
elif name == "detect_platform":
|
|
||||||
result = eyes.detect_platform(arguments["url"])
|
|
||||||
elif name == "search":
|
|
||||||
result = await eyes.search(arguments["query"], arguments.get("num_results", 5))
|
|
||||||
elif name == "search_reddit":
|
|
||||||
result = await eyes.search_reddit(arguments["query"], arguments.get("subreddit"), arguments.get("limit", 10))
|
|
||||||
elif name == "search_github":
|
|
||||||
result = await eyes.search_github(arguments["query"], arguments.get("language"), arguments.get("limit", 5))
|
|
||||||
elif name == "search_twitter":
|
|
||||||
result = await eyes.search_twitter(arguments["query"], arguments.get("limit", 10))
|
|
||||||
elif name == "get_status":
|
|
||||||
result = eyes.doctor_report()
|
result = eyes.doctor_report()
|
||||||
else:
|
else:
|
||||||
result = f"Unknown tool: {name}"
|
result = f"Unknown tool: {name}"
|
||||||
|
|||||||
@@ -1,65 +0,0 @@
|
|||||||
---
|
|
||||||
name: agent-reach
|
|
||||||
description: "Give your AI Agent eyes to see the entire internet. Search + read 10+ platforms with one install."
|
|
||||||
install: pip install git+https://github.com/Panniantong/agent-reach.git
|
|
||||||
---
|
|
||||||
|
|
||||||
# Agent Reach — 给 Agent 装上眼睛 👁️
|
|
||||||
|
|
||||||
安装后你就能搜索和阅读全网内容。
|
|
||||||
|
|
||||||
## 安装
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install git+https://github.com/Panniantong/agent-reach.git
|
|
||||||
```
|
|
||||||
|
|
||||||
## 装完直接能用的(零配置)
|
|
||||||
|
|
||||||
- `agent-reach read <URL>` — 读取任意网页、GitHub、B站、YouTube、RSS
|
|
||||||
- `agent-reach search-github "query"` — 搜索 GitHub 仓库
|
|
||||||
- `agent-reach doctor` — 查看所有平台状态
|
|
||||||
|
|
||||||
## 解锁更多功能
|
|
||||||
|
|
||||||
运行 `agent-reach configure` 进行交互式配置,或者读取以下指南:
|
|
||||||
|
|
||||||
| 功能 | 指南文件 |
|
|
||||||
|------|---------|
|
|
||||||
| 全网搜索 + Reddit搜索 + Twitter搜索 | `agent_reach/guides/setup-exa.md` |
|
|
||||||
| Reddit 完整阅读 | `agent_reach/guides/setup-reddit.md` |
|
|
||||||
| Twitter 高级(时间线/线程) | `agent_reach/guides/setup-twitter.md` |
|
|
||||||
| 小红书 | `agent_reach/guides/setup-xiaohongshu.md` |
|
|
||||||
| 微信公众号 | `agent_reach/guides/setup-wechat.md` |
|
|
||||||
| 视频语音转文字 | `agent_reach/guides/setup-groq.md` |
|
|
||||||
|
|
||||||
每个指南里有详细的步骤说明,你(Agent)可以照着做,只有需要人类操作的部分(登录、复制 key)才需要问用户。
|
|
||||||
|
|
||||||
## MCP Server
|
|
||||||
|
|
||||||
如果你的 Agent 平台支持 MCP:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install agent-reach[mcp]
|
|
||||||
python -m agent_reach.integrations.mcp_server
|
|
||||||
```
|
|
||||||
|
|
||||||
提供 8 个工具:read_url, read_batch, detect_platform, search, search_reddit, search_github, search_twitter, get_status
|
|
||||||
|
|
||||||
## Python API
|
|
||||||
|
|
||||||
```python
|
|
||||||
from agent_reach import AgentReach
|
|
||||||
import asyncio
|
|
||||||
|
|
||||||
eyes = AgentReach()
|
|
||||||
|
|
||||||
# 读取
|
|
||||||
result = asyncio.run(eyes.read("https://github.com/openai/gpt-4"))
|
|
||||||
|
|
||||||
# 搜索
|
|
||||||
results = asyncio.run(eyes.search("AI agent framework"))
|
|
||||||
|
|
||||||
# 健康检查
|
|
||||||
print(eyes.doctor_report())
|
|
||||||
```
|
|
||||||
Executable
+167
@@ -0,0 +1,167 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# 小宇宙播客转文字脚本
|
||||||
|
# 用法: bash transcribe.sh <小宇宙链接> [输出文件路径]
|
||||||
|
# 环境变量: GROQ_API_KEY (必须)
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
URL="${1:?用法: bash transcribe.sh <小宇宙链接> [输出文件路径]}"
|
||||||
|
OUTPUT="${2:-/tmp/podcast_transcript.txt}"
|
||||||
|
TMPDIR="/tmp/xiaoyuzhou_$$"
|
||||||
|
|
||||||
|
# Try env var first, then agent-reach config.yaml
|
||||||
|
if [ -z "$GROQ_API_KEY" ]; then
|
||||||
|
CONFIG_FILE="$HOME/.agent-reach/config.yaml"
|
||||||
|
if [ -f "$CONFIG_FILE" ]; then
|
||||||
|
GROQ_API_KEY=$(python3 -c "import yaml; print((yaml.safe_load(open('$CONFIG_FILE')) or {}).get('groq_api_key',''))" 2>/dev/null || true)
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
GROQ_API_KEY="${GROQ_API_KEY:?请设置 GROQ_API_KEY 环境变量或运行 agent-reach configure groq-key}"
|
||||||
|
|
||||||
|
# Groq API 限制: 25MB per file
|
||||||
|
MAX_CHUNK_SIZE_MB=20
|
||||||
|
AUDIO_BITRATE="64k"
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
rm -rf "$TMPDIR"
|
||||||
|
}
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
mkdir -p "$TMPDIR"
|
||||||
|
|
||||||
|
echo "📻 小宇宙播客转文字"
|
||||||
|
echo "===================="
|
||||||
|
|
||||||
|
# Step 1: 提取音频 URL 和标题
|
||||||
|
echo "🔍 正在解析页面..."
|
||||||
|
PAGE=$(curl -s "$URL")
|
||||||
|
AUDIO_URL=$(echo "$PAGE" | grep -oP 'https://media\.xyzcdn\.net/[^"]*\.(m4a|mp3)' | head -1)
|
||||||
|
TITLE=$(echo "$PAGE" | grep -oP '"title":"[^"]*"' | head -1 | sed 's/"title":"//;s/"//')
|
||||||
|
|
||||||
|
if [ -z "$AUDIO_URL" ]; then
|
||||||
|
echo "❌ 无法从页面提取音频链接"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "📝 标题: $TITLE"
|
||||||
|
echo "🔗 音频: $AUDIO_URL"
|
||||||
|
|
||||||
|
# Step 2: 下载音频
|
||||||
|
echo "⬇️ 正在下载音频..."
|
||||||
|
EXT="${AUDIO_URL##*.}"
|
||||||
|
curl -sL -o "$TMPDIR/original.$EXT" "$AUDIO_URL"
|
||||||
|
FILE_SIZE=$(ls -lh "$TMPDIR/original.$EXT" | awk '{print $5}')
|
||||||
|
echo "📦 文件大小: $FILE_SIZE"
|
||||||
|
|
||||||
|
# Step 3: 获取时长
|
||||||
|
DURATION=$(ffprobe -v quiet -show_entries format=duration -of csv=p=0 "$TMPDIR/original.$EXT" 2>/dev/null | cut -d. -f1)
|
||||||
|
DURATION_MIN=$((DURATION / 60))
|
||||||
|
DURATION_SEC=$((DURATION % 60))
|
||||||
|
echo "⏱️ 时长: ${DURATION_MIN}分${DURATION_SEC}秒"
|
||||||
|
|
||||||
|
# Step 4: 转为低码率单声道 MP3
|
||||||
|
echo "🔄 正在转码..."
|
||||||
|
ffmpeg -y -i "$TMPDIR/original.$EXT" -b:a "$AUDIO_BITRATE" -ac 1 "$TMPDIR/mono.mp3" 2>/dev/null
|
||||||
|
MONO_SIZE=$(stat -c%s "$TMPDIR/mono.mp3" 2>/dev/null || stat -f%z "$TMPDIR/mono.mp3")
|
||||||
|
echo "📦 转码后: $(echo "$MONO_SIZE / 1024 / 1024" | bc)MB"
|
||||||
|
|
||||||
|
# Step 5: 按大小切片
|
||||||
|
MAX_BYTES=$((MAX_CHUNK_SIZE_MB * 1024 * 1024))
|
||||||
|
|
||||||
|
if [ "$MONO_SIZE" -le "$MAX_BYTES" ]; then
|
||||||
|
# 不需要切片
|
||||||
|
cp "$TMPDIR/mono.mp3" "$TMPDIR/chunk_0.mp3"
|
||||||
|
NUM_CHUNKS=1
|
||||||
|
echo "📎 无需切片"
|
||||||
|
else
|
||||||
|
# 计算需要几个 chunk
|
||||||
|
NUM_CHUNKS=$(( (MONO_SIZE / MAX_BYTES) + 1 ))
|
||||||
|
CHUNK_DURATION=$(( DURATION / NUM_CHUNKS + 10 )) # 加 10 秒缓冲
|
||||||
|
echo "✂️ 切分为 $NUM_CHUNKS 段 (每段约 $((CHUNK_DURATION / 60)) 分钟)..."
|
||||||
|
|
||||||
|
for i in $(seq 0 $((NUM_CHUNKS - 1))); do
|
||||||
|
START=$((i * CHUNK_DURATION))
|
||||||
|
ffmpeg -y -i "$TMPDIR/mono.mp3" -ss "$START" -t "$CHUNK_DURATION" -c copy "$TMPDIR/chunk_${i}.mp3" 2>/dev/null
|
||||||
|
CHUNK_SIZE=$(ls -lh "$TMPDIR/chunk_${i}.mp3" | awk '{print $5}')
|
||||||
|
echo " 段 $((i+1))/$NUM_CHUNKS: $CHUNK_SIZE"
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Step 6: 调用 Groq Whisper API 转录
|
||||||
|
echo "🎙️ 正在转录 (Groq Whisper large-v3)..."
|
||||||
|
|
||||||
|
for i in $(seq 0 $((NUM_CHUNKS - 1))); do
|
||||||
|
echo -n " 段 $((i+1))/$NUM_CHUNKS... "
|
||||||
|
|
||||||
|
RESPONSE=$(curl -s -w "\n%{http_code}" \
|
||||||
|
https://api.groq.com/openai/v1/audio/transcriptions \
|
||||||
|
-H "Authorization: Bearer $GROQ_API_KEY" \
|
||||||
|
-F file="@$TMPDIR/chunk_${i}.mp3" \
|
||||||
|
-F model="whisper-large-v3" \
|
||||||
|
-F language="zh" \
|
||||||
|
-F response_format="text")
|
||||||
|
|
||||||
|
HTTP_CODE=$(echo "$RESPONSE" | tail -1)
|
||||||
|
BODY=$(echo "$RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$HTTP_CODE" != "200" ]; then
|
||||||
|
echo "❌ API 错误 (HTTP $HTTP_CODE)"
|
||||||
|
echo "$BODY"
|
||||||
|
|
||||||
|
# 如果是速率限制,等待后重试
|
||||||
|
if [ "$HTTP_CODE" = "429" ]; then
|
||||||
|
# 从错误信息中提取等待时间,默认 120 秒
|
||||||
|
WAIT_SEC=$(echo "$BODY" | grep -oP 'in \K[0-9]+m' | sed 's/m//' | head -1)
|
||||||
|
WAIT_SEC=${WAIT_SEC:-2}
|
||||||
|
WAIT_SEC=$((WAIT_SEC * 60 + 30))
|
||||||
|
echo " ⏳ 速率限制,等待 ${WAIT_SEC} 秒后重试..."
|
||||||
|
sleep "$WAIT_SEC"
|
||||||
|
RESPONSE=$(curl -s -w "\n%{http_code}" \
|
||||||
|
https://api.groq.com/openai/v1/audio/transcriptions \
|
||||||
|
-H "Authorization: Bearer $GROQ_API_KEY" \
|
||||||
|
-F file="@$TMPDIR/chunk_${i}.mp3" \
|
||||||
|
-F model="whisper-large-v3" \
|
||||||
|
-F language="zh" \
|
||||||
|
-F response_format="text")
|
||||||
|
HTTP_CODE=$(echo "$RESPONSE" | tail -1)
|
||||||
|
BODY=$(echo "$RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$HTTP_CODE" != "200" ]; then
|
||||||
|
echo " ❌ 重试失败"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$BODY" > "$TMPDIR/transcript_${i}.txt"
|
||||||
|
CHARS=$(wc -m < "$TMPDIR/transcript_${i}.txt")
|
||||||
|
echo "✅ ($CHARS 字)"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Step 7: 合并输出
|
||||||
|
echo "📄 正在合并文字稿..."
|
||||||
|
|
||||||
|
{
|
||||||
|
echo "# $TITLE"
|
||||||
|
echo ""
|
||||||
|
echo "来源: $URL"
|
||||||
|
echo "时长: ${DURATION_MIN}分${DURATION_SEC}秒"
|
||||||
|
echo "转录时间: $(date '+%Y-%m-%d %H:%M')"
|
||||||
|
echo ""
|
||||||
|
echo "---"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
for i in $(seq 0 $((NUM_CHUNKS - 1))); do
|
||||||
|
cat "$TMPDIR/transcript_${i}.txt"
|
||||||
|
echo ""
|
||||||
|
done
|
||||||
|
} > "$OUTPUT"
|
||||||
|
|
||||||
|
TOTAL_CHARS=$(wc -m < "$OUTPUT")
|
||||||
|
echo ""
|
||||||
|
echo "✅ 完成!"
|
||||||
|
echo "📄 输出: $OUTPUT"
|
||||||
|
echo "📊 总字数: $TOTAL_CHARS"
|
||||||
|
echo "===================="
|
||||||
+90
-71
@@ -1,87 +1,106 @@
|
|||||||
---
|
---
|
||||||
name: agent-reach
|
name: agent-reach
|
||||||
description: >
|
description: >
|
||||||
Give your AI agent eyes to see the entire internet. Read and search across
|
Give your AI agent eyes to see the entire internet.
|
||||||
Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, RSS, and any web page
|
17 platforms via CLI, MCP, curl, and Python scripts.
|
||||||
— all from a single CLI. Use when: (1) reading content from URLs (tweets, Reddit posts,
|
Zero config for 8 channels.
|
||||||
articles, videos), (2) searching across platforms (web, Twitter, Reddit, GitHub, YouTube,
|
|
||||||
Bilibili, XiaoHongShu), (3) checking channel health or updating Agent Reach.
|
【路由方式】SKILL.md 包含路由表和常用命令,复杂场景需按需阅读对应分类的 references/*.md。
|
||||||
Triggers: "search Twitter/Reddit/YouTube", "read this URL", "find posts about",
|
分类:search / social (小红书/抖音/微博/推特/B站/V2EX/Reddit) / career(LinkedIn) / dev(github) / web(网页/文章/公众号/RSS) / video(YouTube/B站/播客).
|
||||||
"搜索", "读取", "查一下", "看看这个链接".
|
|
||||||
|
Use when user asks to search, read, or interact on any supported platform,
|
||||||
|
shares a URL, or asks to search the web.
|
||||||
|
triggers:
|
||||||
|
- search: 搜/查/找/search/搜索/查一下/帮我搜
|
||||||
|
- social:
|
||||||
|
- 小红书: xiaohongshu/xhs/小红书/红书
|
||||||
|
- 抖音: douyin/抖音
|
||||||
|
- Twitter: twitter/推特/x.com/推文
|
||||||
|
- 微博: weibo/微博
|
||||||
|
- B站: bilibili/b站/哔哩哔哩
|
||||||
|
- V2EX: v2ex
|
||||||
|
- Reddit: reddit
|
||||||
|
- career: 招聘/职位/求职/linkedin/领英/找工作
|
||||||
|
- dev: github/代码/仓库/gh/issue/pr/分支/commit
|
||||||
|
- web: 网页/链接/文章/公众号/微信文章/rss/读一下/打开这个
|
||||||
|
- video: youtube/视频/播客/字幕/小宇宙/转录/yt
|
||||||
|
- finance: 雪球/股票/stock/xueqiu/行情/基金
|
||||||
|
metadata:
|
||||||
|
openclaw:
|
||||||
|
homepage: https://github.com/Panniantong/Agent-Reach
|
||||||
---
|
---
|
||||||
|
|
||||||
# Agent Reach
|
# Agent Reach — 路由器
|
||||||
|
|
||||||
Read and search the internet across 9+ platforms via unified CLI.
|
17 平台工具集合。根据用户意图选择对应分类。
|
||||||
|
|
||||||
## Setup
|
## 路由表
|
||||||
|
|
||||||
|
| 用户意图 | 分类 | 详细文档 |
|
||||||
|
|---------|------|---------|
|
||||||
|
| 网页搜索/代码搜索 | search | [references/search.md](references/search.md) |
|
||||||
|
| 小红书/抖音/微博/推特/B站/V2EX/Reddit | social | [references/social.md](references/social.md) |
|
||||||
|
| 招聘/职位/LinkedIn | career | [references/career.md](references/career.md) |
|
||||||
|
| GitHub/代码 | dev | [references/dev.md](references/dev.md) |
|
||||||
|
| 网页/文章/公众号/RSS | web | [references/web.md](references/web.md) |
|
||||||
|
| YouTube/B站/播客字幕 | video | [references/video.md](references/video.md) |
|
||||||
|
|
||||||
|
## 零配置快速命令
|
||||||
|
|
||||||
First check if agent-reach is installed:
|
|
||||||
```bash
|
```bash
|
||||||
|
# Exa 网页搜索
|
||||||
|
mcporter call 'exa.web_search_exa(query: "query", numResults: 5)'
|
||||||
|
|
||||||
|
# 通用网页阅读
|
||||||
|
curl -s "https://r.jina.ai/URL"
|
||||||
|
|
||||||
|
# GitHub 搜索
|
||||||
|
gh search repos "query" --sort stars --limit 10
|
||||||
|
|
||||||
|
# Twitter 搜索
|
||||||
|
twitter search "query" --limit 10
|
||||||
|
|
||||||
|
# YouTube/B站字幕
|
||||||
|
yt-dlp --write-sub --skip-download -o "/tmp/%(id)s" "URL"
|
||||||
|
|
||||||
|
# Reddit 搜索
|
||||||
|
rdt search "query" --limit 10
|
||||||
|
|
||||||
|
# Reddit 读帖 + 评论
|
||||||
|
rdt read POST_ID
|
||||||
|
|
||||||
|
# V2EX 热门
|
||||||
|
curl -s "https://www.v2ex.com/api/topics/hot.json" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
## 环境检查
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 检查可用 channel
|
||||||
agent-reach doctor
|
agent-reach doctor
|
||||||
|
|
||||||
|
# 查看所有 MCP 服务
|
||||||
|
mcporter_list_servers()
|
||||||
```
|
```
|
||||||
|
|
||||||
If command not found, install it:
|
## 工作区规则
|
||||||
```bash
|
|
||||||
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
|
||||||
agent-reach install --env=auto
|
|
||||||
```
|
|
||||||
|
|
||||||
`install` auto-detects your environment and installs all dependencies (Node.js, mcporter, bird CLI, gh CLI). Read the output and run `agent-reach doctor` to see what's active.
|
**不要在 agent workspace 创建文件。** 使用 `/tmp/` 存放临时输出,`~/.agent-reach/` 存放持久数据。
|
||||||
|
|
||||||
For channels that need user input, ask the user. See the full setup guide:
|
## 详细文档
|
||||||
|
|
||||||
|
根据用户需求,阅读对应的详细文档:
|
||||||
|
|
||||||
|
- [搜索工具](references/search.md) — Exa AI 搜索
|
||||||
|
- [社交媒体](references/social.md) — 小红书, 抖音, Twitter, B站, V2EX, Reddit
|
||||||
|
- [职场招聘](references/career.md) — LinkedIn
|
||||||
|
- [开发工具](references/dev.md) — GitHub CLI
|
||||||
|
- [网页阅读](references/web.md) — Jina Reader, 微信公众号, RSS
|
||||||
|
- [视频播客](references/video.md) — YouTube, B站, 小宇宙
|
||||||
|
|
||||||
|
## 配置渠道
|
||||||
|
|
||||||
|
如果某个 channel 需要配置,获取安装指南:
|
||||||
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
||||||
|
|
||||||
## Commands
|
用户只需提供 cookies,其他配置由 agent 完成。
|
||||||
|
|
||||||
### Read any URL
|
|
||||||
```bash
|
|
||||||
agent-reach read <url>
|
|
||||||
agent-reach read <url> --json # structured output
|
|
||||||
```
|
|
||||||
Handles: tweets, Reddit posts, articles, YouTube (transcripts), GitHub repos, etc.
|
|
||||||
|
|
||||||
### Search
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach search "query" # web search (Exa)
|
|
||||||
agent-reach search-twitter "query" # Twitter/X
|
|
||||||
agent-reach search-reddit "query" # Reddit (--sub <subreddit>)
|
|
||||||
agent-reach search-github "query" # GitHub (--lang <language>)
|
|
||||||
agent-reach search-youtube "query" # YouTube
|
|
||||||
agent-reach search-bilibili "query" # Bilibili (B站)
|
|
||||||
agent-reach search-xhs "query" # XiaoHongShu (小红书)
|
|
||||||
```
|
|
||||||
|
|
||||||
All search commands support `-n <count>` for number of results.
|
|
||||||
|
|
||||||
### Management
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach doctor # channel status overview
|
|
||||||
agent-reach watch # quick health + update check (for scheduled tasks)
|
|
||||||
agent-reach check-update # check for new versions
|
|
||||||
```
|
|
||||||
|
|
||||||
### Configure channels
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
|
|
||||||
agent-reach configure proxy http://user:pass@ip:port
|
|
||||||
agent-reach configure --from-browser chrome # auto-extract cookies
|
|
||||||
```
|
|
||||||
|
|
||||||
## Channel Status Tiers
|
|
||||||
|
|
||||||
- **Tier 0 (zero config):** Web, YouTube, RSS, Twitter (read-only via Jina)
|
|
||||||
- **Tier 1 (free setup):** Exa web search (mcporter required)
|
|
||||||
- **Tier 2 (user config):** Twitter search (cookie), Reddit full (proxy), GitHub (token), Bilibili (proxy), XiaoHongShu (MCP)
|
|
||||||
|
|
||||||
Run `agent-reach doctor` to see which channels are active.
|
|
||||||
|
|
||||||
## Tips
|
|
||||||
|
|
||||||
- Always try `agent-reach read <url>` first for any URL — it auto-detects the platform
|
|
||||||
- For Twitter cookies, recommend the user install [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome extension
|
|
||||||
- Reddit and Bilibili block server IPs — suggest a residential proxy (~$1/month) if on a server
|
|
||||||
- If a channel breaks, run `agent-reach doctor` to diagnose
|
|
||||||
|
|||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# 职场招聘
|
||||||
|
|
||||||
|
LinkedIn。
|
||||||
|
|
||||||
|
## LinkedIn
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 获取个人资料
|
||||||
|
mcporter call 'linkedin-scraper.get_person_profile(linkedin_url: "https://linkedin.com/in/username")'
|
||||||
|
|
||||||
|
# 搜索人才
|
||||||
|
mcporter call 'linkedin-scraper.search_people(keyword: "AI engineer", limit: 10)'
|
||||||
|
|
||||||
|
# 获取公司资料
|
||||||
|
mcporter call 'linkedin-scraper.get_company_profile(linkedin_url: "https://linkedin.com/company/xxx")'
|
||||||
|
|
||||||
|
# 搜索职位
|
||||||
|
mcporter call 'linkedin-scraper.search_jobs(keyword: "software engineer", limit: 10)'
|
||||||
|
```
|
||||||
|
|
||||||
|
> **需要登录**: LinkedIn scraper 需要有效的登录态。
|
||||||
|
|
||||||
|
### Fallback 方案
|
||||||
|
|
||||||
|
如果 MCP 不可用,可以用 Jina Reader:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "https://r.jina.ai/https://linkedin.com/in/username"
|
||||||
|
```
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
# 开发工具
|
||||||
|
|
||||||
|
GitHub CLI
|
||||||
|
|
||||||
|
## GitHub (gh CLI)
|
||||||
|
|
||||||
|
GitHub 官方命令行工具,用于仓库、Issue、PR、Actions、Release 以及 API 访问。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 认证
|
||||||
|
gh auth login
|
||||||
|
gh auth status
|
||||||
|
|
||||||
|
# 搜索
|
||||||
|
gh search repos "query" --sort stars --limit 10
|
||||||
|
gh search code "query" --language python
|
||||||
|
|
||||||
|
# 仓库
|
||||||
|
gh repo view owner/repo
|
||||||
|
gh repo clone owner/repo
|
||||||
|
gh repo create my-repo --private
|
||||||
|
gh repo fork owner/repo
|
||||||
|
gh repo fork owner/repo --clone
|
||||||
|
gh repo sync owner/repo
|
||||||
|
|
||||||
|
# Issues
|
||||||
|
gh issue list -R owner/repo --state open
|
||||||
|
gh issue view 123 -R owner/repo
|
||||||
|
gh issue create -R owner/repo --title "Title" --body "Body"
|
||||||
|
|
||||||
|
# Pull Requests
|
||||||
|
gh pr list -R owner/repo --state open
|
||||||
|
gh pr view 123 -R owner/repo
|
||||||
|
gh pr create -R owner/repo --title "Title" --body "Body"
|
||||||
|
gh pr checks 123 --repo owner/repo
|
||||||
|
|
||||||
|
# Actions / CI
|
||||||
|
gh run list --repo owner/repo --limit 10
|
||||||
|
gh run view <run-id> --repo owner/repo
|
||||||
|
gh run view <run-id> --repo owner/repo --log-failed
|
||||||
|
gh workflow list --repo owner/repo
|
||||||
|
|
||||||
|
# Releases
|
||||||
|
gh release list -R owner/repo
|
||||||
|
gh release create v1.0.0
|
||||||
|
|
||||||
|
# API
|
||||||
|
gh api /user
|
||||||
|
gh api repos/owner/repo
|
||||||
|
|
||||||
|
# JSON 输出
|
||||||
|
gh issue list --repo owner/repo --json number,title --jq '.[] | "\(.number): \(.title)"'
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## 选择指南
|
||||||
|
|
||||||
|
| 工具 | 来源 | 用途 |
|
||||||
|
|-----|------|------|
|
||||||
|
| gh CLI | agent-reach | Git 操作 |
|
||||||
|
| zread | my-mcp-tools | 读仓库内容 |
|
||||||
|
| context7 | my-mcp-tools | 查技术文档 |
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# 搜索工具
|
||||||
|
|
||||||
|
Exa AI 搜索引擎。
|
||||||
|
|
||||||
|
## Exa AI 搜索
|
||||||
|
|
||||||
|
高质量 AI 搜索引擎,擅长技术和代码搜索。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mcporter call 'exa.web_search_exa(query: "query", numResults: 5)'
|
||||||
|
mcporter call 'exa.get_code_context_exa(query: "code question", tokensNum: 3000)'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 使用场景
|
||||||
|
|
||||||
|
| 场景 | 参数 |
|
||||||
|
|-----|------|
|
||||||
|
| 网页搜索 | `web_search_exa(query: "...", numResults: 5)` |
|
||||||
|
| 代码搜索 | `get_code_context_exa(query: "...", tokensNum: 3000)` |
|
||||||
|
|
||||||
|
### 特点
|
||||||
|
|
||||||
|
- 擅长英文内容和技术文档
|
||||||
|
- 支持代码上下文搜索
|
||||||
|
- 结果质量高
|
||||||
|
|
||||||
|
## 与其他搜索工具对比
|
||||||
|
|
||||||
|
| 工具 | 来源 | 适用场景 |
|
||||||
|
|-----|------|---------|
|
||||||
|
| Exa | agent-reach | 英文/技术/代码搜索 |
|
||||||
|
| 智谱搜索 | my-mcp-tools | 中文搜索 |
|
||||||
|
| GitHub 搜索 | agent-reach (dev.md) | 仓库/代码搜索 |
|
||||||
@@ -0,0 +1,176 @@
|
|||||||
|
# 社交媒体 & 社区
|
||||||
|
|
||||||
|
小红书、抖音、Twitter/X、微博、B站、V2EX、Reddit。
|
||||||
|
|
||||||
|
## 小红书 / XiaoHongShu (xhs-cli)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 搜索笔记
|
||||||
|
xhs search "query"
|
||||||
|
|
||||||
|
# 阅读笔记详情
|
||||||
|
xhs read NOTE_ID_OR_URL
|
||||||
|
|
||||||
|
# 查看评论
|
||||||
|
xhs comments NOTE_ID_OR_URL
|
||||||
|
|
||||||
|
# 浏览热门
|
||||||
|
xhs hot
|
||||||
|
|
||||||
|
# 推荐 feed
|
||||||
|
xhs feed
|
||||||
|
|
||||||
|
# 用户主页
|
||||||
|
xhs user USER_ID
|
||||||
|
xhs user-posts USER_ID
|
||||||
|
|
||||||
|
# 发帖/互动
|
||||||
|
xhs post --title "标题" --content "正文" --images img1.jpg img2.jpg
|
||||||
|
xhs like NOTE_ID
|
||||||
|
xhs comment NOTE_ID "评论内容"
|
||||||
|
```
|
||||||
|
|
||||||
|
> **安装**: `pipx install xiaohongshu-cli`,然后 `xhs login`(自动从浏览器提取 Cookie)。
|
||||||
|
|
||||||
|
## 抖音 / Douyin
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 解析视频信息
|
||||||
|
mcporter call 'douyin.parse_douyin_video_info(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
|
||||||
|
# 获取无水印下载链接
|
||||||
|
mcporter call 'douyin.get_douyin_download_link(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
|
||||||
|
# 提取视频文案
|
||||||
|
mcporter call 'douyin.extract_douyin_text(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
```
|
||||||
|
|
||||||
|
> **无需登录**
|
||||||
|
|
||||||
|
## Twitter/X (twitter-cli)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 搜索推文
|
||||||
|
twitter search "query" --limit 10
|
||||||
|
|
||||||
|
# 读取单条推文(含回复)
|
||||||
|
twitter tweet URL_OR_ID
|
||||||
|
|
||||||
|
# 读取长文 / X Article
|
||||||
|
twitter article URL_OR_ID
|
||||||
|
|
||||||
|
# 用户时间线
|
||||||
|
twitter user-posts @username --limit 20
|
||||||
|
|
||||||
|
# 用户资料
|
||||||
|
twitter user @username
|
||||||
|
|
||||||
|
# 首页时间线
|
||||||
|
twitter feed --limit 20
|
||||||
|
```
|
||||||
|
|
||||||
|
> **安装**: `pipx install twitter-cli` 或 `uv tool install twitter-cli`
|
||||||
|
> **认证**: 设置 `TWITTER_AUTH_TOKEN` + `TWITTER_CT0` 环境变量,或确保浏览器已登录 x.com。
|
||||||
|
|
||||||
|
## 微博 / Weibo
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 使用 Jina Reader 读取
|
||||||
|
curl -s "https://r.jina.ai/https://weibo.com/USER_ID/POST_ID"
|
||||||
|
```
|
||||||
|
|
||||||
|
> 微博主要通过网页抓取,推荐使用通用网页读取方式。
|
||||||
|
|
||||||
|
## B站 / Bilibili
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 获取视频元数据
|
||||||
|
yt-dlp --dump-json "https://www.bilibili.com/video/BVxxx"
|
||||||
|
|
||||||
|
# 下载字幕
|
||||||
|
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --convert-subs vtt --skip-download -o "/tmp/%(id)s" "URL"
|
||||||
|
```
|
||||||
|
|
||||||
|
> **注意**: 服务器 IP 可能遇到 412 错误。使用 `--cookies-from-browser chrome` 或配置代理。
|
||||||
|
|
||||||
|
## V2EX (公开 API)
|
||||||
|
|
||||||
|
无需认证,直接调用公开 API。
|
||||||
|
|
||||||
|
### 热门主题
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "https://www.v2ex.com/api/topics/hot.json" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 节点主题
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# node_name 如: python, tech, jobs, qna, programmers
|
||||||
|
curl -s "https://www.v2ex.com/api/topics/show.json?node_name=python&page=1" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 主题详情
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# topic_id 从 URL 获取,如 https://www.v2ex.com/t/1234567
|
||||||
|
curl -s "https://www.v2ex.com/api/topics/show.json?id=TOPIC_ID" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 主题回复
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "https://www.v2ex.com/api/replies/show.json?topic_id=TOPIC_ID&page=1" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 用户信息
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "https://www.v2ex.com/api/members/show.json?username=USERNAME" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Python 调用示例
|
||||||
|
|
||||||
|
```python
|
||||||
|
from agent_reach.channels.v2ex import V2EXChannel
|
||||||
|
|
||||||
|
ch = V2EXChannel()
|
||||||
|
|
||||||
|
# 获取热门帖子
|
||||||
|
topics = ch.get_hot_topics(limit=10)
|
||||||
|
for t in topics:
|
||||||
|
print(f"[{t['node_title']}] {t['title']} ({t['replies']} 回复)")
|
||||||
|
|
||||||
|
# 获取节点帖子
|
||||||
|
node_topics = ch.get_node_topics("python", limit=5)
|
||||||
|
|
||||||
|
# 获取帖子详情 + 回复
|
||||||
|
topic = ch.get_topic(1234567)
|
||||||
|
print(topic["title"], "—", topic["author"])
|
||||||
|
|
||||||
|
# 获取用户信息
|
||||||
|
user = ch.get_user("Livid")
|
||||||
|
```
|
||||||
|
|
||||||
|
> **节点列表**: https://www.v2ex.com/planes
|
||||||
|
|
||||||
|
## Reddit (rdt-cli)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 搜索帖子
|
||||||
|
rdt search "query" --limit 10
|
||||||
|
|
||||||
|
# 读帖子全文 + 评论
|
||||||
|
rdt read POST_ID
|
||||||
|
|
||||||
|
# 浏览 subreddit
|
||||||
|
rdt sub python --limit 20
|
||||||
|
|
||||||
|
# 浏览热门
|
||||||
|
rdt popular --limit 10
|
||||||
|
|
||||||
|
# 浏览 /r/all
|
||||||
|
rdt all --limit 10
|
||||||
|
```
|
||||||
|
|
||||||
|
> **安装**: `pipx install rdt-cli`。无需登录即可搜索和阅读。
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# 视频/播客
|
||||||
|
|
||||||
|
YouTube、B站、小宇宙播客的字幕和转录。
|
||||||
|
|
||||||
|
## YouTube (yt-dlp)
|
||||||
|
|
||||||
|
### 获取视频元数据
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yt-dlp --dump-json "URL"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 下载字幕
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 下载字幕 (不下载视频)
|
||||||
|
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --skip-download -o "/tmp/%(id)s" "URL"
|
||||||
|
|
||||||
|
# 然后读取 .vtt 文件
|
||||||
|
cat /tmp/VIDEO_ID.*.vtt
|
||||||
|
```
|
||||||
|
|
||||||
|
### 搜索视频
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yt-dlp --dump-json "ytsearch5:query"
|
||||||
|
```
|
||||||
|
|
||||||
|
## B站 / Bilibili (yt-dlp)
|
||||||
|
|
||||||
|
### 获取视频元数据
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yt-dlp --dump-json "https://www.bilibili.com/video/BVxxx"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 下载字幕
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --convert-subs vtt --skip-download -o "/tmp/%(id)s" "URL"
|
||||||
|
```
|
||||||
|
|
||||||
|
> **注意**: 服务器 IP 可能遇到 412 错误。使用 `--cookies-from-browser chrome` 或配置代理。
|
||||||
|
|
||||||
|
## 小宇宙播客 / Xiaoyuzhou Podcast
|
||||||
|
|
||||||
|
### 转录单集播客
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 输出 Markdown 文件到 /tmp/
|
||||||
|
~/.agent-reach/tools/xiaoyuzhou/transcribe.sh "https://www.xiaoyuzhoufm.com/episode/EPISODE_ID"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 前置要求
|
||||||
|
|
||||||
|
1. **ffmpeg**: `brew install ffmpeg`
|
||||||
|
2. **Groq API Key** (免费): https://console.groq.com/keys
|
||||||
|
3. **配置 Key**: `agent-reach configure groq-key YOUR_KEY`
|
||||||
|
4. **首次运行**: `agent-reach install --env=auto` 安装工具
|
||||||
|
|
||||||
|
### 检查状态
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach doctor
|
||||||
|
```
|
||||||
|
|
||||||
|
> 输出 Markdown 文件默认保存到 `/tmp/`。
|
||||||
|
|
||||||
|
## 抖音视频解析
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 解析视频信息
|
||||||
|
mcporter call 'douyin.parse_douyin_video_info(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
|
||||||
|
# 获取无水印下载链接
|
||||||
|
mcporter call 'douyin.get_douyin_download_link(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
```
|
||||||
|
|
||||||
|
> 详见 [social.md](social.md#抖音--douyin)
|
||||||
|
|
||||||
|
## 选择指南
|
||||||
|
|
||||||
|
| 场景 | 推荐工具 |
|
||||||
|
|-----|---------|
|
||||||
|
| YouTube 字幕 | yt-dlp |
|
||||||
|
| B站字幕 | yt-dlp |
|
||||||
|
| 播客转录 | 小宇宙 transcribe.sh |
|
||||||
|
| 抖音视频解析 | douyin MCP |
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
# 网页阅读
|
||||||
|
|
||||||
|
通用网页、微信公众号、RSS。
|
||||||
|
|
||||||
|
## 通用网页 (Jina Reader)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 读取任意网页内容
|
||||||
|
curl -s "https://r.jina.ai/URL"
|
||||||
|
|
||||||
|
# 示例
|
||||||
|
curl -s "https://r.jina.ai/https://example.com/article"
|
||||||
|
```
|
||||||
|
|
||||||
|
**适用场景**: 大多数网页可以直接用 Jina Reader 读取。
|
||||||
|
|
||||||
|
## Web Reader (MCP)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 读取网页内容 (Markdown 格式)
|
||||||
|
mcporter call 'web-reader.webReader(url: "https://example.com")'
|
||||||
|
|
||||||
|
# 保留图片
|
||||||
|
mcporter call 'web-reader.webReader(url: "https://example.com", retain_images: true)'
|
||||||
|
|
||||||
|
# 纯文本格式
|
||||||
|
mcporter call 'web-reader.webReader(url: "https://example.com", return_format: "text")'
|
||||||
|
```
|
||||||
|
|
||||||
|
**适用场景**: 需要更精确控制输出格式时使用。
|
||||||
|
|
||||||
|
## 微信公众号 / WeChat Articles
|
||||||
|
|
||||||
|
### 搜索公众号文章(通过 Exa)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 搜索微信公众号文章
|
||||||
|
mcporter call 'exa.web_search_exa(query: "搜索关键词", numResults: 5, includeDomains: ["mp.weixin.qq.com"])'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 阅读公众号文章全文(通过 Exa)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 抓取文章全文
|
||||||
|
mcporter call 'exa.crawling_exa(urls: ["https://mp.weixin.qq.com/s/ARTICLE_ID"], maxCharacters: 10000)'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 可选:Camoufox 阅读(反爬更强)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/.agent-reach/tools/wechat-article-for-ai && python3 main.py "https://mp.weixin.qq.com/s/ARTICLE_ID"
|
||||||
|
```
|
||||||
|
|
||||||
|
> **注意**: Jina Reader 无法读取微信文章(被 CAPTCHA 拦截),推荐用 Exa。
|
||||||
|
|
||||||
|
## RSS (feedparser)
|
||||||
|
|
||||||
|
```python
|
||||||
|
python3 -c "
|
||||||
|
import feedparser
|
||||||
|
for e in feedparser.parse('FEED_URL').entries[:5]:
|
||||||
|
print(f'{e.title} — {e.link}')
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
**适用场景**: 订阅博客、新闻源、播客等 RSS feed。
|
||||||
|
|
||||||
|
## 选择指南
|
||||||
|
|
||||||
|
| 场景 | 推荐工具 |
|
||||||
|
|-----|---------|
|
||||||
|
| 通用网页 | Jina Reader (`curl r.jina.ai`) |
|
||||||
|
| 需要图片/格式控制 | web-reader MCP |
|
||||||
|
| 微信公众号 | Exa (搜索+阅读) / Camoufox (可选阅读) |
|
||||||
|
| RSS 订阅 | feedparser |
|
||||||
|
| 微博/知乎等 | Jina Reader |
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
"""Cross-platform path and remediation helpers for yt-dlp."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def get_ytdlp_config_dir() -> Path:
|
||||||
|
"""Return the recommended yt-dlp user config directory for this OS."""
|
||||||
|
|
||||||
|
if sys.platform == "win32":
|
||||||
|
appdata = os.environ.get("APPDATA")
|
||||||
|
if appdata:
|
||||||
|
return Path(appdata) / "yt-dlp"
|
||||||
|
return Path.home() / "AppData" / "Roaming" / "yt-dlp"
|
||||||
|
if sys.platform == "darwin":
|
||||||
|
return Path.home() / "Library" / "Application Support" / "yt-dlp"
|
||||||
|
return Path.home() / ".config" / "yt-dlp"
|
||||||
|
|
||||||
|
|
||||||
|
def get_ytdlp_config_path() -> Path:
|
||||||
|
"""Return the yt-dlp user config file path for this OS."""
|
||||||
|
|
||||||
|
return get_ytdlp_config_dir() / "config"
|
||||||
|
|
||||||
|
|
||||||
|
def render_ytdlp_fix_command() -> str:
|
||||||
|
"""Return an OS-appropriate command to enable Node.js as yt-dlp JS runtime."""
|
||||||
|
|
||||||
|
config_path = get_ytdlp_config_path()
|
||||||
|
if sys.platform == "win32":
|
||||||
|
return (
|
||||||
|
f"$cfg = '{config_path}'\n"
|
||||||
|
"New-Item -ItemType Directory -Force -Path (Split-Path $cfg) | Out-Null\n"
|
||||||
|
"if (-not (Test-Path $cfg) -or -not (Select-String -Path $cfg -Pattern '--js-runtimes' -Quiet)) {\n"
|
||||||
|
" Add-Content -Path $cfg -Value '--js-runtimes node'\n"
|
||||||
|
"}"
|
||||||
|
)
|
||||||
|
return (
|
||||||
|
f"mkdir -p '{config_path.parent}' && "
|
||||||
|
f"grep -qxF -- '--js-runtimes node' '{config_path}' 2>/dev/null || "
|
||||||
|
f"printf '%s\n' '--js-runtimes node' >> '{config_path}'"
|
||||||
|
)
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
"""UTF-8-safe text helpers for cross-platform file operations."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def read_utf8_text(path: str | Path, default: str = "") -> str:
|
||||||
|
"""Read text as UTF-8 with replacement semantics."""
|
||||||
|
|
||||||
|
target = Path(path)
|
||||||
|
if not target.exists():
|
||||||
|
return default
|
||||||
|
return target.read_text(encoding="utf-8", errors="replace")
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# Agent Reach tested dependency set
|
||||||
|
# Usage:
|
||||||
|
# pip install -c constraints.txt -e .[dev]
|
||||||
|
|
||||||
|
requests==2.32.5
|
||||||
|
feedparser==6.0.12
|
||||||
|
python-dotenv==1.2.1
|
||||||
|
loguru==0.7.3
|
||||||
|
PyYAML==6.0.3
|
||||||
|
rich==14.3.2
|
||||||
|
yt-dlp==2025.5.22
|
||||||
|
|
||||||
|
pytest==8.0.0
|
||||||
|
ruff==0.15.1
|
||||||
|
mypy==1.19.1
|
||||||
|
types-requests==2.32.4.20260107
|
||||||
|
types-PyYAML==6.0.12.20250915
|
||||||
+155
-29
@@ -11,7 +11,7 @@
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<a href="#quick-start">Quick Start</a> · <a href="../README.md">中文</a> · <a href="#supported-platforms">Platforms</a> · <a href="#design-philosophy">Philosophy</a>
|
<a href="#quick-start">Quick Start</a> · <a href="../README.md">中文</a> · <a href="README_ja.md">日本語</a> · <a href="#supported-platforms">Platforms</a> · <a href="#design-philosophy">Philosophy</a>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -39,6 +39,12 @@ Install Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/m
|
|||||||
|
|
||||||
Copy that to your Agent. A few minutes later, it can read tweets, search Reddit, and watch Bilibili.
|
Copy that to your Agent. A few minutes later, it can read tweets, search Reddit, and watch Bilibili.
|
||||||
|
|
||||||
|
**Already installed? Update in one command:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
```
|
||||||
|
|
||||||
### ✅ Before you start, you might want to know
|
### ✅ Before you start, you might want to know
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
@@ -56,17 +62,21 @@ Copy that to your Agent. A few minutes later, it can read tweets, search Reddit,
|
|||||||
| Platform | Capabilities | Setup | Notes |
|
| Platform | Capabilities | Setup | Notes |
|
||||||
|----------|-------------|:-----:|-------|
|
|----------|-------------|:-----:|-------|
|
||||||
| 🌐 **Web** | Read | Zero config | Any URL → clean Markdown ([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) |
|
| 🌐 **Web** | Read | Zero config | Any URL → clean Markdown ([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) |
|
||||||
| 🐦 **Twitter/X** | Read · Search | Zero config / Cookie | Single tweets readable out of the box. Cookie unlocks search, timeline, posting ([bird](https://github.com/steipete/bird)) |
|
| 🐦 **Twitter/X** | Read · Search | Zero config / Cookie | Single tweets readable out of the box. Cookie unlocks search, timeline, posting ([bird](https://www.npmjs.com/package/@steipete/bird)) |
|
||||||
| 📕 **XiaoHongShu** | Read · Search · **Post · Comment · Like** | mcporter | Via [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) internal API, install and go |
|
| 📕 **XiaoHongShu** | Read · Search · **Post · Comment · Like** | mcporter | Via [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) internal API, install and go |
|
||||||
| 📷 **Instagram** | Read · Search | instaloader | Posts, profiles, hashtags ([instaloader](https://github.com/instaloader/instaloader) ⭐9.8K) |
|
| 🎵 **Douyin** | Video parsing · Watermark-free download | mcporter | Via [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server), no login needed |
|
||||||
| 💼 **LinkedIn** | Read · Search | mcporter / Jina | Profiles, companies, job search ([linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) ⭐900+) |
|
| 💼 **LinkedIn** | Jina Reader (public pages) | Full profiles, companies, job search | Tell your Agent "help me set up LinkedIn" |
|
||||||
| 🏢 **Boss直聘** | Read · Search | mcporter / Jina | Job search, greet recruiters ([mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)) |
|
| 💬 **WeChat Articles** | Search + Read | Zero config | Search + read WeChat Official Account articles (full Markdown) ([wechat-article-for-ai](https://github.com/Panniantong/wechat-article-for-ai) + [miku_ai](https://github.com/GobinFan/Miku_Spider)) |
|
||||||
|
| 📰 **Weibo** | Trending · Search · Feeds · Comments | Zero config | Hot search, content/user/topic search, feeds, comments ([mcp-server-weibo](https://github.com/Panniantong/mcp-server-weibo)) |
|
||||||
|
| 💻 **V2EX** | Hot topics · Node topics · Topic detail + replies · User profile | Zero config | Public JSON API, no auth required. Great for tech community content |
|
||||||
|
| 📈 **Xueqiu (雪球)** | Stock quotes · Search · Hot posts · Hot stocks | Browser cookie | Tell your Agent "help me set up Xueqiu" |
|
||||||
|
| 🎙️ **Xiaoyuzhou Podcast** | Transcription | Free API key | Podcast audio → full text transcript via Groq Whisper (free) |
|
||||||
| 🔍 **Web Search** | Search | Auto-configured | Auto-configured during install, free, no API key ([Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter)) |
|
| 🔍 **Web Search** | Search | Auto-configured | Auto-configured during install, free, no API key ([Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter)) |
|
||||||
| 📦 **GitHub** | Read · Search | Zero config | [gh CLI](https://cli.github.com) powered. Public repos work immediately. `gh auth login` unlocks Fork, Issue, PR |
|
| 📦 **GitHub** | Read · Search | Zero config | [gh CLI](https://cli.github.com) powered. Public repos work immediately. `gh auth login` unlocks Fork, Issue, PR |
|
||||||
| 📺 **YouTube** | Read · **Search** | Zero config | Subtitles + search across 1800+ video sites ([yt-dlp](https://github.com/yt-dlp/yt-dlp) ⭐148K) |
|
| 📺 **YouTube** | Read · **Search** | Zero config | Subtitles + search across 1800+ video sites ([yt-dlp](https://github.com/yt-dlp/yt-dlp) ⭐148K) |
|
||||||
| 📺 **Bilibili** | Read · **Search** | Zero config / Proxy | Video info + subtitles + search. Local works directly, servers need a proxy ([yt-dlp](https://github.com/yt-dlp/yt-dlp)) |
|
| 📺 **Bilibili** | Read · **Search** | Zero config / Proxy | Video info + subtitles + search. Local works directly, servers need a proxy ([yt-dlp](https://github.com/yt-dlp/yt-dlp)) |
|
||||||
| 📡 **RSS** | Read | Zero config | Any RSS/Atom feed ([feedparser](https://github.com/kurtmckee/feedparser) ⭐2.3K) |
|
| 📡 **RSS** | Read | Zero config | Any RSS/Atom feed ([feedparser](https://github.com/kurtmckee/feedparser) ⭐2.3K) |
|
||||||
| 📖 **Reddit** | Search · Read | Free / Proxy | Search via Exa (free). Reading posts needs a proxy on servers |
|
| 📖 **Reddit** | Search · Read | Zero config | Search and read via Exa (free, no proxy needed) |
|
||||||
|
|
||||||
> **Setup levels:** Zero config = install and go · Auto-configured = handled during install · mcporter = needs MCP service · Cookie = export from browser · Proxy = $1/month
|
> **Setup levels:** Zero config = install and go · Auto-configured = handled during install · mcporter = needs MCP service · Cookie = export from browser · Proxy = $1/month
|
||||||
|
|
||||||
@@ -82,6 +92,11 @@ Install Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/m
|
|||||||
|
|
||||||
The Agent auto-installs, detects your environment, and tells you what's ready.
|
The Agent auto-installs, detects your environment, and tells you what's ready.
|
||||||
|
|
||||||
|
> 🔄 **Already installed?** Update in one command:
|
||||||
|
> ```
|
||||||
|
> Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
> ```
|
||||||
|
|
||||||
<details>
|
<details>
|
||||||
<summary>Manual install</summary>
|
<summary>Manual install</summary>
|
||||||
|
|
||||||
@@ -109,14 +124,14 @@ After the Skill is installed, the Agent will auto-detect whether `agent-reach` C
|
|||||||
|
|
||||||
No configuration needed — just tell your Agent:
|
No configuration needed — just tell your Agent:
|
||||||
|
|
||||||
- "Read this link" → any web page
|
- "Read this link" → `curl https://r.jina.ai/URL` for any web page
|
||||||
- "What's this GitHub repo about?" → repos, issues, code
|
- "What's this GitHub repo about?" → `gh repo view owner/repo`
|
||||||
- "What does this video cover?" → YouTube / Bilibili subtitles
|
- "What does this video cover?" → `yt-dlp --dump-json URL` for subtitles
|
||||||
- "Read this tweet" → Twitter posts
|
- "Read this tweet" → `bird read URL`
|
||||||
- "Subscribe to this RSS" → RSS / Atom feeds
|
- "Subscribe to this RSS" → `feedparser` to parse feeds
|
||||||
- "Search GitHub for LLM frameworks" → GitHub search
|
- "Search GitHub for LLM frameworks" → `gh search repos "LLM framework"`
|
||||||
|
|
||||||
**No commands to remember.** The Agent knows what to call.
|
**No commands to remember.** The Agent reads SKILL.md and knows what to call.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -130,9 +145,9 @@ Tell your Agent "help me configure Twitter cookies" — it'll guide you through
|
|||||||
|
|
||||||
### 🌐 Proxy — $1/month, servers only
|
### 🌐 Proxy — $1/month, servers only
|
||||||
|
|
||||||
Reddit and Bilibili block server IPs. Get a proxy ([Webshare](https://webshare.io) recommended, $1/month) and send the address to your Agent.
|
Bilibili blocks server IPs. Get a proxy ([Webshare](https://webshare.io) recommended, $1/month) and send the address to your Agent.
|
||||||
|
|
||||||
> Local computers don't need a proxy. Reddit search works free via Exa even without one.
|
> Reddit now works free via Exa without any proxy. Local computers don't need a proxy for Bilibili either.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -156,7 +171,7 @@ $ agent-reach doctor
|
|||||||
⬜ Web semantic search — sign up at exa.ai for free key
|
⬜ Web semantic search — sign up at exa.ai for free key
|
||||||
|
|
||||||
🔧 Configurable:
|
🔧 Configurable:
|
||||||
⬜ Reddit posts and comments — search via Exa (free). Reading needs proxy
|
✅ Reddit posts and comments — search and read via Exa (free, no proxy)
|
||||||
⬜ XiaoHongShu notes — needs cookie. Export from browser
|
⬜ XiaoHongShu notes — needs cookie. Export from browser
|
||||||
|
|
||||||
Status: 6/9 channels available
|
Status: 6/9 channels available
|
||||||
@@ -172,41 +187,46 @@ Every time you spin up a new Agent, you spend time finding tools, installing dep
|
|||||||
|
|
||||||
Agent Reach does one simple thing: **it makes those tool selection and configuration decisions for you.**
|
Agent Reach does one simple thing: **it makes those tool selection and configuration decisions for you.**
|
||||||
|
|
||||||
|
After installation, your Agent calls the upstream tools directly (bird CLI, yt-dlp, mcporter, gh CLI, etc.) — no wrapper layer in between.
|
||||||
|
|
||||||
### 🔌 Every Channel is Pluggable
|
### 🔌 Every Channel is Pluggable
|
||||||
|
|
||||||
Each platform is a single Python file implementing a unified interface. **Backends can be swapped anytime** — when a better tool comes along, change one file and nothing else breaks.
|
Each platform maps to an upstream tool. **Don't like one? Swap it out.**
|
||||||
|
|
||||||
```
|
```
|
||||||
channels/
|
channels/
|
||||||
├── web.py → Jina Reader ← swap to Firecrawl, Crawl4AI…
|
├── web.py → Jina Reader ← swap to Firecrawl, Crawl4AI…
|
||||||
├── twitter.py → bird ← swap to Nitter, official API…
|
├── twitter.py → bird CLI ← swap to Nitter, official API…
|
||||||
├── youtube.py → yt-dlp ← swap to YouTube API, Whisper…
|
├── youtube.py → yt-dlp ← swap to YouTube API, Whisper…
|
||||||
├── github.py → gh CLI ← swap to REST API, PyGithub…
|
├── github.py → gh CLI ← swap to REST API, PyGithub…
|
||||||
├── bilibili.py → yt-dlp ← swap to bilibili-api…
|
├── bilibili.py → yt-dlp ← swap to bilibili-api…
|
||||||
├── reddit.py → JSON API + Exa ← swap to PRAW, Pushshift…
|
├── reddit.py → Exa ← search + read, no proxy needed
|
||||||
├── xiaohongshu.py → mcporter MCP ← swap to other XHS tools…
|
├── xiaohongshu.py → mcporter MCP ← swap to other XHS tools…
|
||||||
├── instagram.py → instaloader ← swap to instagrapi, official API…
|
├── douyin.py → mcporter MCP ← swap to other Douyin tools…
|
||||||
├── linkedin.py → linkedin-mcp ← swap to LinkedIn API…
|
├── linkedin.py → linkedin-mcp ← swap to LinkedIn API…
|
||||||
├── bosszhipin.py → mcp-bosszp ← swap to other job tools…
|
├── rss.py → feedparser ← swap to atoma…
|
||||||
├── rss.py → feedparser ← swap to atoma…
|
|
||||||
├── exa_search.py → mcporter MCP ← swap to Tavily, SerpAPI…
|
├── exa_search.py → mcporter MCP ← swap to Tavily, SerpAPI…
|
||||||
└── __init__.py → Channel registry
|
└── __init__.py → Channel registry (for doctor checks)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Each channel file only checks whether its upstream tool is installed and working (`check()` method for `agent-reach doctor`). The actual reading and searching is done by calling the upstream tools directly.
|
||||||
|
|
||||||
### Current Tool Choices
|
### Current Tool Choices
|
||||||
|
|
||||||
| Scenario | Tool | Why |
|
| Scenario | Tool | Why |
|
||||||
|----------|------|-----|
|
|----------|------|-----|
|
||||||
| Read web pages | [Jina Reader](https://github.com/jina-ai/reader) | 9.8K stars, free, no API key needed |
|
| Read web pages | [Jina Reader](https://github.com/jina-ai/reader) | 9.8K stars, free, no API key needed |
|
||||||
| Read tweets | [bird](https://github.com/steipete/bird) | Cookie auth, free. Official API is pay-per-use ($0.005/post read) |
|
| Read tweets | [bird](https://www.npmjs.com/package/@steipete/bird) | Cookie auth, free. Official API is pay-per-use ($0.005/post read) |
|
||||||
| Video subtitles + search | [yt-dlp](https://github.com/yt-dlp/yt-dlp) | 148K stars, YouTube + Bilibili + 1800 sites |
|
| Video subtitles + search | [yt-dlp](https://github.com/yt-dlp/yt-dlp) | 148K stars, YouTube + Bilibili + 1800 sites |
|
||||||
| Search the web | [Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter) | AI semantic search, MCP integration, no API key |
|
| Search the web | [Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter) | AI semantic search, MCP integration, no API key |
|
||||||
| GitHub | [gh CLI](https://cli.github.com) | Official tool, full API after auth |
|
| GitHub | [gh CLI](https://cli.github.com) | Official tool, full API after auth |
|
||||||
| Read RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python ecosystem standard, 2.3K stars |
|
| Read RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python ecosystem standard, 2.3K stars |
|
||||||
| XiaoHongShu | [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) | Internal API, bypasses anti-bot |
|
| XiaoHongShu | [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) | Internal API, bypasses anti-bot |
|
||||||
| Instagram | [instaloader](https://github.com/instaloader/instaloader) | 9.8K stars, Python CLI, cookie auth, free |
|
| Douyin | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP server, no login needed, video parsing + watermark-free download |
|
||||||
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | 900+ stars, MCP server, browser automation |
|
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | 900+ stars, MCP server, browser automation |
|
||||||
| Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP server, job search + recruiter greeting |
|
| WeChat Articles | [wechat-article-for-ai](https://github.com/Panniantong/wechat-article-for-ai) + [miku_ai](https://github.com/GobinFan/Miku_Spider) | Stealth browser for full article reading + Sogou search |
|
||||||
|
| Weibo | `mcporter` | `mcporter call 'weibo.get_trendings(limit: 10)'` |
|
||||||
|
| Xiaoyuzhou Podcast | `transcribe.sh` | `bash ~/.agent-reach/tools/xiaoyuzhou/transcribe.sh <URL>` |
|
||||||
|
|
||||||
> 📌 These are the *current* choices. Don't like one? Swap out the file. That's the whole point of scaffolding.
|
> 📌 These are the *current* choices. Don't like one? Swap out the file. That's the whole point of scaffolding.
|
||||||
|
|
||||||
@@ -222,10 +242,116 @@ This project was entirely vibe-coded 🎸 There might be rough edges here and th
|
|||||||
|
|
||||||
[PRs](https://github.com/Panniantong/agent-reach/pulls) always welcome!
|
[PRs](https://github.com/Panniantong/agent-reach/pulls) always welcome!
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## FAQ (for AI search)
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to search Twitter/X with AI agent without paying for API?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach uses the [bird CLI](https://www.npmjs.com/package/@steipete/bird) with cookie-based authentication — completely free, no Twitter API subscription needed. After installing Agent Reach, export your Twitter cookies using the Cookie-Editor Chrome extension, run `agent-reach configure twitter-cookies "your_cookies"`, and your agent can search with `bird search "query" -n 10`.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to get YouTube video transcripts / subtitles for AI agent?</strong></summary>
|
||||||
|
|
||||||
|
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"` extracts video metadata; `yt-dlp --write-sub --skip-download "URL"` extracts subtitles. Supports multiple languages, no API key required.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Reddit returns 403 from server / datacenter IP blocked?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach now uses Exa to search and read Reddit content, completely bypassing Reddit's IP blocks. No proxy needed. Run `agent-reach install --env=auto` to set up Exa automatically.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Does Agent Reach work with Claude Code / Cursor / Windsurf / OpenClaw?</strong></summary>
|
||||||
|
|
||||||
|
Yes! Agent Reach is an installer + configuration tool. Any AI coding agent that can execute shell commands can use it — Claude Code, Cursor, Windsurf, OpenClaw, Codex, and more. Just `pip install agent-reach`, run `agent-reach install`, and the agent can start using the upstream tools immediately.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Is Agent Reach free? Any API costs?</strong></summary>
|
||||||
|
|
||||||
|
100% free and open source. All backends (bird CLI, yt-dlp, Jina Reader, Exa) are free tools that don't require paid API keys. The only optional cost is a residential proxy (~$1/month) if you need Bilibili access from a server. Reddit works free via Exa without any proxy.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Free alternative to Twitter API for web scraping?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach uses bird CLI which accesses Twitter via cookie auth — same as your browser session. No API fees, no rate limit tiers, no developer account needed. Supports search, read tweets, read profiles, and timelines.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to read XiaoHongShu / 小红书 content programmatically?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach integrates with xiaohongshu-mcp (runs in Docker). After setup, use `mcporter call 'xiaohongshu.get_feed_detail(...)'` to read notes or `mcporter call 'xiaohongshu.search_feeds(keyword: "query")'` to search.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to parse Douyin / 抖音 videos with AI agent?</strong></summary>
|
||||||
|
|
||||||
|
Install douyin-mcp-server, then your agent can use `mcporter call 'douyin.parse_douyin_video_info(share_link: "share_url")'` to parse video info and get watermark-free download links. No login required — just share the Douyin link. See https://github.com/yzfly/douyin-mcp-server
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to extract scripts from both Douyin and XiaoHongShu with one MCP?</strong></summary>
|
||||||
|
|
||||||
|
If you want one MCP server that can handle:
|
||||||
|
|
||||||
|
- Douyin videos
|
||||||
|
- XiaoHongShu video notes
|
||||||
|
- XiaoHongShu image notes
|
||||||
|
|
||||||
|
and directly write `script.md` + `info.json`, you can point the existing `douyin` mcporter alias at:
|
||||||
|
|
||||||
|
- https://github.com/JNHFlow21/social-post-extractor-mcp
|
||||||
|
|
||||||
|
It keeps backward compatibility with:
|
||||||
|
|
||||||
|
- `parse_douyin_video_info`
|
||||||
|
- `get_douyin_download_link`
|
||||||
|
- `extract_douyin_text`
|
||||||
|
|
||||||
|
and adds unified tools:
|
||||||
|
|
||||||
|
- `parse_social_post_info`
|
||||||
|
- `extract_social_post_script`
|
||||||
|
|
||||||
|
This is useful when your agent workflow is “paste a link, get a script file”.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Credits
|
## Credits
|
||||||
|
|
||||||
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://github.com/steipete/bird) · [Exa](https://exa.ai) · [feedparser](https://github.com/kurtmckee/feedparser) · [instaloader](https://github.com/instaloader/instaloader) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
|
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://www.npmjs.com/package/@steipete/bird) · [Exa](https://exa.ai) · [feedparser](https://github.com/kurtmckee/feedparser) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||||
|
|
||||||
|
## Contact
|
||||||
|
|
||||||
|
- 📧 **Email:** pnt01@foxmail.com
|
||||||
|
- 🐦 **Twitter/X:** [@Neo_Reidlab](https://x.com/Neo_Reidlab)
|
||||||
|
|
||||||
|
For collaboration or questions, add me on WeChat — I'll invite you to the community group:
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="wechat-group-qr.jpg" width="280" alt="WeChat QR">
|
||||||
|
</p>
|
||||||
|
|
||||||
|
> For bug reports and feature requests, please use [GitHub Issues](https://github.com/Panniantong/Agent-Reach/issues) — easier to track.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
[MIT](../LICENSE)
|
[MIT](../LICENSE)
|
||||||
|
|
||||||
|
## Friends
|
||||||
|
|
||||||
|
[FluxNode](https://fluxnode.org) — Low-cost AI API gateway, 90% off official pricing, pay-as-you-go or subscription. Works with OpenClaw, Claude Code, and any Agent.
|
||||||
|
|
||||||
|
[OpenClaw for Enterprise](https://github.com/littleben/openclaw-for-enterprise) — Enterprise-grade multi-user OpenClaw deployment, use AI directly in Feishu/Lark, container isolation, one-command management.
|
||||||
|
|
||||||
|
[OpenClaw on Tencent Cloud](https://www.tencentcloud.com/act/pro/intl-openclaw?referral_code=G76Y819A&lang=en&pg=) — One-click OpenClaw on Tencent Cloud: chat to connect Agent Reach & unlock internet power.
|
||||||
|
|
||||||
|
## Star History
|
||||||
|
|
||||||
|
[](https://star-history.com/#Panniantong/Agent-Reach&Date)
|
||||||
|
|||||||
@@ -0,0 +1,322 @@
|
|||||||
|
<h1 align="center">👁️ Agent Reach</h1>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<strong>AIエージェントにワンクリックでインターネット全体へのアクセスを</strong>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<a href="../LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="MIT License"></a>
|
||||||
|
<a href="https://www.python.org/"><img src="https://img.shields.io/badge/Python-3.10+-green.svg?style=for-the-badge&logo=python&logoColor=white" alt="Python 3.10+"></a>
|
||||||
|
<a href="https://github.com/Panniantong/agent-reach/stargazers"><img src="https://img.shields.io/github/stars/Panniantong/agent-reach?style=for-the-badge" alt="GitHub Stars"></a>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<a href="#クイックスタート">クイックスタート</a> · <a href="../README.md">中文</a> · <a href="README_en.md">English</a> · <a href="#対応プラットフォーム">プラットフォーム</a> · <a href="#設計思想">設計思想</a>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## なぜ Agent Reach?
|
||||||
|
|
||||||
|
AIエージェントはすでにインターネットにアクセスできます。しかし「ネットに繋がる」はほんの始まりに過ぎません。
|
||||||
|
|
||||||
|
最も価値のある情報は、さまざまなSNSやニッチなプラットフォームに散らばっています:Twitterの議論、Redditのフィードバック、YouTubeのチュートリアル、小紅書のレビュー、Bilibiliの動画、GitHubのアクティビティ… **これらこそ情報密度が最も高い場所です**。しかし、各プラットフォームにはそれぞれ障壁があります:
|
||||||
|
|
||||||
|
| 課題 | 現実 |
|
||||||
|
|------|------|
|
||||||
|
| Twitter API | 従量課金、中程度の利用で月額約$215 |
|
||||||
|
| Reddit | サーバーIPが403でブロックされる |
|
||||||
|
| 小紅書 | 閲覧にログインが必要 |
|
||||||
|
| Bilibili | 海外/サーバーIPをブロック |
|
||||||
|
|
||||||
|
エージェントをこれらのプラットフォームに接続するには、ツールを探し、依存関係をインストールし、設定をデバッグする必要があります — ひとつずつ。
|
||||||
|
|
||||||
|
**Agent Reach はこれを1つのコマンドにまとめます:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Install Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
||||||
|
```
|
||||||
|
|
||||||
|
これをエージェントにコピーするだけ。数分後には、ツイートの閲覧、Redditの検索、Bilibiliの視聴が可能になります。
|
||||||
|
|
||||||
|
**すでにインストール済み?1コマンドでアップデート:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
```
|
||||||
|
|
||||||
|
### ✅ 始める前に知っておきたいこと
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| 💰 **完全無料** | すべてのツールはオープンソース、すべてのAPIは無料。唯一のコストはサーバープロキシ(月額$1)の可能性のみ — ローカルPCでは不要 |
|
||||||
|
| 🔒 **プライバシー安全** | Cookieはローカルに保存。アップロードされることはありません。完全オープンソース — いつでも監査可能 |
|
||||||
|
| 🔄 **常に最新** | 上流ツール(yt-dlp、bird、Jina Reader等)を定期的に追跡・更新 |
|
||||||
|
| 🤖 **あらゆるエージェントに対応** | Claude Code、OpenClaw、Cursor、Windsurf… コマンドを実行できるすべてのエージェント |
|
||||||
|
| 🩺 **組み込み診断** | `agent-reach doctor` — 1コマンドで何が動き、何が動かないか、どう修正するかを表示 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 対応プラットフォーム
|
||||||
|
|
||||||
|
| プラットフォーム | 機能 | セットアップ | 備考 |
|
||||||
|
|-----------------|------|:----------:|------|
|
||||||
|
| 🌐 **Web** | 閲覧 | 設定不要 | 任意のURL → クリーンなMarkdown([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) |
|
||||||
|
| 🐦 **Twitter/X** | 閲覧・検索 | 設定不要 / Cookie | 単一ツイートはすぐに閲覧可能。Cookieで検索、タイムライン、投稿が解放([bird](https://www.npmjs.com/package/@steipete/bird)) |
|
||||||
|
| 📕 **小紅書** | 閲覧・検索・**投稿・コメント・いいね** | mcporter | [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) 内部APIで、インストールしてすぐ使える |
|
||||||
|
| 🎵 **抖音** | 動画解析・ウォーターマークなしダウンロード | mcporter | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server)、ログイン不要 |
|
||||||
|
| 💼 **LinkedIn** | Jina Reader(公開ページ) | プロフィール、企業、求人検索 | エージェントに「LinkedInの設定を手伝って」と伝えてください |
|
||||||
|
| 💬 **WeChat記事** | 検索 + 閲覧 | 設定不要 | WeChat公式アカウント記事の検索+閲覧(完全Markdown)([wechat-article-for-ai](https://github.com/Panniantong/wechat-article-for-ai) + [miku_ai](https://github.com/GobinFan/Miku_Spider)) |
|
||||||
|
| 📰 **Weibo** | トレンド・検索・フィード・コメント | 設定不要 | ホット検索、コンテンツ/ユーザー/トピック検索、フィード、コメント([mcp-server-weibo](https://github.com/Panniantong/mcp-server-weibo)) |
|
||||||
|
| 💻 **V2EX** | 人気トピック・ノードトピック・トピック詳細+返信・ユーザープロフィール | 設定不要 | 公開JSON API、認証不要。技術コミュニティのコンテンツに最適 |
|
||||||
|
| 📈 **雪球(Xueqiu)** | 株価・検索・人気投稿・人気銘柄 | 設定不要 | 公開APIで自動セッションCookie、ログイン不要 |
|
||||||
|
| 🎙️ **小宇宙Podcast** | 文字起こし | 無料APIキー | Podcast音声 → Groq Whisper(無料)による完全テキスト文字起こし |
|
||||||
|
| 🔍 **Web検索** | 検索 | 自動設定 | インストール時に自動設定、無料、APIキー不要([Exa](https://exa.ai)、[mcporter](https://github.com/nicepkg/mcporter)経由) |
|
||||||
|
| 📦 **GitHub** | 閲覧・検索 | 設定不要 | [gh CLI](https://cli.github.com) 搭載。公開リポジトリはすぐ使える。`gh auth login`でFork、Issue、PRが解放 |
|
||||||
|
| 📺 **YouTube** | 閲覧・**検索** | 設定不要 | 字幕 + 1800以上の動画サイトでの検索([yt-dlp](https://github.com/yt-dlp/yt-dlp) ⭐148K) |
|
||||||
|
| 📺 **Bilibili** | 閲覧・**検索** | 設定不要 / プロキシ | 動画情報 + 字幕 + 検索。ローカルはそのまま動作、サーバーはプロキシが必要([yt-dlp](https://github.com/yt-dlp/yt-dlp)) |
|
||||||
|
| 📡 **RSS** | 閲覧 | 設定不要 | 任意のRSS/Atomフィード([feedparser](https://github.com/kurtmckee/feedparser) ⭐2.3K) |
|
||||||
|
| 📖 **Reddit** | 検索・閲覧 | 無料 / プロキシ | 検索はExa(無料)経由。投稿の閲覧はサーバーでプロキシが必要 |
|
||||||
|
|
||||||
|
> **セットアップレベル:** 設定不要 = インストールしてすぐ使える · 自動設定 = インストール時に処理 · mcporter = MCPサービスが必要 · Cookie = ブラウザからエクスポート · プロキシ = 月額$1
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## クイックスタート
|
||||||
|
|
||||||
|
以下をAIエージェント(Claude Code、OpenClaw、Cursor等)にコピーしてください:
|
||||||
|
|
||||||
|
```
|
||||||
|
Install Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
||||||
|
```
|
||||||
|
|
||||||
|
エージェントが自動でインストールし、環境を検出し、何が使えるかを教えてくれます。
|
||||||
|
|
||||||
|
> 🔄 **すでにインストール済み?** 1コマンドでアップデート:
|
||||||
|
> ```
|
||||||
|
> Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
> ```
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>手動インストール</summary>
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
||||||
|
agent-reach install --env=auto
|
||||||
|
```
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>Skillとしてインストール(Claude Code / OpenClaw / Skills対応の任意のエージェント)</summary>
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npx skills add Panniantong/Agent-Reach@agent-reach
|
||||||
|
```
|
||||||
|
|
||||||
|
Skillインストール後、エージェントは`agent-reach` CLIが利用可能かを自動検出し、必要に応じてインストールします。
|
||||||
|
|
||||||
|
> `agent-reach install` でインストールした場合、Skillは自動的に登録されます — 追加の手順は不要です。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## すぐに使える機能
|
||||||
|
|
||||||
|
設定不要 — エージェントに伝えるだけ:
|
||||||
|
|
||||||
|
- 「このリンクを読んで」→ `curl https://r.jina.ai/URL` で任意のWebページ
|
||||||
|
- 「このGitHubリポジトリは何?」→ `gh repo view owner/repo`
|
||||||
|
- 「この動画の内容は?」→ `yt-dlp --dump-json URL` で字幕取得
|
||||||
|
- 「このツイートを読んで」→ `bird read URL`
|
||||||
|
- 「このRSSを購読して」→ `feedparser` でフィード解析
|
||||||
|
- 「GitHubでLLMフレームワークを検索して」→ `gh search repos "LLM framework"`
|
||||||
|
|
||||||
|
**コマンドを覚える必要はありません。** エージェントがSKILL.mdを読み、何を呼び出すべきか理解します。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 必要に応じてアンロック
|
||||||
|
|
||||||
|
使わない?設定しなくてOK。すべてのステップはオプションです。
|
||||||
|
|
||||||
|
### 🍪 Cookie — 無料、2分
|
||||||
|
|
||||||
|
エージェントに「Twitterのクッキーの設定を手伝って」と伝えてください — ブラウザからのエクスポート手順を案内してくれます。ローカルPCなら自動インポートも可能です。
|
||||||
|
|
||||||
|
### 🌐 プロキシ — 月額$1、サーバーのみ
|
||||||
|
|
||||||
|
RedditとBilibiliはサーバーIPをブロックします。プロキシを取得し([Webshare](https://webshare.io) 推奨、月額$1)、アドレスをエージェントに伝えてください。
|
||||||
|
|
||||||
|
> ローカルPCではプロキシは不要です。Reddit検索はプロキシなしでもExaで無料で動作します。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 一目でわかるステータス
|
||||||
|
|
||||||
|
```
|
||||||
|
$ agent-reach doctor
|
||||||
|
|
||||||
|
👁️ Agent Reach ステータス
|
||||||
|
========================================
|
||||||
|
|
||||||
|
✅ 利用可能:
|
||||||
|
✅ GitHubリポジトリとコード — 公開リポジトリの閲覧・検索可能
|
||||||
|
✅ Twitter/Xツイート — 閲覧可能。Cookieで検索・投稿が解放
|
||||||
|
✅ YouTube動画字幕 — yt-dlp
|
||||||
|
⚠️ Bilibili動画情報 — サーバーIPがブロックされる可能性あり、プロキシを設定してください
|
||||||
|
✅ RSS/Atomフィード — feedparser
|
||||||
|
✅ Webページ(任意のURL) — Jina Reader API
|
||||||
|
|
||||||
|
🔍 検索(無料Exaキーで解放):
|
||||||
|
⬜ Webセマンティック検索 — exa.aiで無料キーを取得
|
||||||
|
|
||||||
|
🔧 設定可能:
|
||||||
|
⬜ Reddit投稿とコメント — 検索はExa(無料)経由。閲覧にはプロキシが必要
|
||||||
|
⬜ 小紅書ノート — Cookieが必要。ブラウザからエクスポート
|
||||||
|
|
||||||
|
ステータス: 9チャンネル中6チャンネルが利用可能
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 設計思想
|
||||||
|
|
||||||
|
**Agent Reach はスキャフォールディングツールであり、フレームワークではありません。**
|
||||||
|
|
||||||
|
新しいエージェントを立ち上げるたびに、ツールを探し、依存関係をインストールし、設定をデバッグする時間がかかります — Twitterを読むには何を使う?Redditのブロックをどう回避する?YouTubeの字幕をどう抽出する?毎回、同じ作業を繰り返すことになります。
|
||||||
|
|
||||||
|
Agent Reach はシンプルなことを1つだけ行います:**ツールの選定と設定の判断をあなたの代わりに行います。**
|
||||||
|
|
||||||
|
インストール後、エージェントは上流ツール(bird CLI、yt-dlp、mcporter、gh CLI等)を直接呼び出します — 間にラッパーレイヤーはありません。
|
||||||
|
|
||||||
|
### 🔌 すべてのチャンネルはプラグ可能
|
||||||
|
|
||||||
|
各プラットフォームは上流ツールに対応しています。**気に入らなければ差し替えるだけ。**
|
||||||
|
|
||||||
|
```
|
||||||
|
channels/
|
||||||
|
├── web.py → Jina Reader ← Firecrawl、Crawl4AIなどに差し替え可能…
|
||||||
|
├── twitter.py → bird CLI ← Nitter、公式APIなどに差し替え可能…
|
||||||
|
├── youtube.py → yt-dlp ← YouTube API、Whisperなどに差し替え可能…
|
||||||
|
├── github.py → gh CLI ← REST API、PyGithubなどに差し替え可能…
|
||||||
|
├── bilibili.py → yt-dlp ← bilibili-apiなどに差し替え可能…
|
||||||
|
├── reddit.py → JSON API + Exa ← PRAW、Pushshiftなどに差し替え可能…
|
||||||
|
├── xiaohongshu.py → mcporter MCP ← 他のXHSツールに差し替え可能…
|
||||||
|
├── douyin.py → mcporter MCP ← 他の抖音ツールに差し替え可能…
|
||||||
|
├── linkedin.py → linkedin-mcp ← LinkedIn APIに差し替え可能…
|
||||||
|
├── rss.py → feedparser ← atomaなどに差し替え可能…
|
||||||
|
├── exa_search.py → mcporter MCP ← Tavily、SerpAPIなどに差し替え可能…
|
||||||
|
└── __init__.py → チャンネルレジストリ(doctor チェック用)
|
||||||
|
```
|
||||||
|
|
||||||
|
各チャンネルファイルは、上流ツールがインストールされ動作しているかをチェックするだけです(`agent-reach doctor` 用の `check()` メソッド)。実際の閲覧や検索は上流ツールを直接呼び出して行います。
|
||||||
|
|
||||||
|
### 現在のツール選定
|
||||||
|
|
||||||
|
| シナリオ | ツール | 理由 |
|
||||||
|
|----------|--------|------|
|
||||||
|
| Webページ閲覧 | [Jina Reader](https://github.com/jina-ai/reader) | ⭐9.8K、無料、APIキー不要 |
|
||||||
|
| ツイート閲覧 | [bird](https://www.npmjs.com/package/@steipete/bird) | Cookie認証、無料。公式APIは従量課金(投稿閲覧1件$0.005) |
|
||||||
|
| 動画字幕 + 検索 | [yt-dlp](https://github.com/yt-dlp/yt-dlp) | ⭐148K、YouTube + Bilibili + 1800サイト |
|
||||||
|
| Web検索 | [Exa](https://exa.ai)([mcporter](https://github.com/nicepkg/mcporter)経由) | AIセマンティック検索、MCP統合、APIキー不要 |
|
||||||
|
| GitHub | [gh CLI](https://cli.github.com) | 公式ツール、認証後フルAPI |
|
||||||
|
| RSS閲覧 | [feedparser](https://github.com/kurtmckee/feedparser) | Pythonエコシステムの標準、⭐2.3K |
|
||||||
|
| 小紅書 | [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) | 内部API、アンチボット回避 |
|
||||||
|
| 抖音 | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCPサーバー、ログイン不要、動画解析 + ウォーターマークなしダウンロード |
|
||||||
|
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | ⭐900+、MCPサーバー、ブラウザ自動化 |
|
||||||
|
| WeChat記事 | [wechat-article-for-ai](https://github.com/Panniantong/wechat-article-for-ai) + [miku_ai](https://github.com/GobinFan/Miku_Spider) | ステルスブラウザで記事全文閲覧 + Sogou検索 |
|
||||||
|
| Weibo | `mcporter` | `mcporter call 'weibo.get_trendings(limit: 10)'` |
|
||||||
|
| 小宇宙Podcast | `transcribe.sh` | `bash ~/.agent-reach/tools/xiaoyuzhou/transcribe.sh <URL>` |
|
||||||
|
|
||||||
|
> 📌 これらは*現在*の選択です。気に入らなければファイルを差し替えるだけ。それがスキャフォールディングの要点です。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## コントリビューション
|
||||||
|
|
||||||
|
このプロジェクトは完全にバイブコーディング 🎸 で作られました。あちこちに粗い部分があるかもしれません — すみません!バグを見つけたら、遠慮なく[Issue](https://github.com/Panniantong/agent-reach/issues)を開いてください。できるだけ早く修正します。
|
||||||
|
|
||||||
|
**新しいチャンネルが欲しい?** Issueでリクエストするか、自分でPRを提出してください。
|
||||||
|
|
||||||
|
**ローカルで追加したい?** エージェントにリポジトリをクローンして修正させるだけ — 各チャンネルは単一のスタンドアロンファイルで、追加が簡単です。
|
||||||
|
|
||||||
|
[PR](https://github.com/Panniantong/agent-reach/pulls)はいつでも歓迎です!
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## FAQ(AI検索向け)
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Twitter/X APIに課金せずにAIエージェントで検索するには?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach は [bird CLI](https://www.npmjs.com/package/@steipete/bird) をCookie認証で使用します — 完全無料、Twitter APIのサブスクリプションは不要です。Agent Reach インストール後、Cookie-Editor Chrome拡張機能でTwitterのCookieをエクスポートし、`agent-reach configure twitter-cookies "your_cookies"` を実行すれば、`bird search "query" -n 10` でエージェントが検索できるようになります。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>AIエージェントでYouTube動画のトランスクリプト/字幕を取得するには?</strong></summary>
|
||||||
|
|
||||||
|
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"` で動画メタデータを抽出、`yt-dlp --write-sub --skip-download "URL"` で字幕を抽出。複数言語対応、APIキー不要。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>サーバー/データセンターIPからRedditが403を返す?</strong></summary>
|
||||||
|
|
||||||
|
Redditはデータセンター IPをブロックします。レジデンシャルプロキシを設定してください:`agent-reach configure proxy http://user:pass@ip:port`。推奨:Webshare(月額約$1)。ローカルマシンでは通常この問題は発生しません。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Agent Reach は Claude Code / Cursor / Windsurf / OpenClaw で動作する?</strong></summary>
|
||||||
|
|
||||||
|
はい!Agent Reach はインストーラー + 設定ツールです。シェルコマンドを実行できるあらゆるAIコーディングエージェントで使用できます — Claude Code、Cursor、Windsurf、OpenClaw、Codex等。`pip install agent-reach` を実行し、`agent-reach install` を実行するだけで、エージェントはすぐに上流ツールを使い始められます。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Agent Reach は無料?APIのコストは?</strong></summary>
|
||||||
|
|
||||||
|
100%無料でオープンソース。すべてのバックエンド(bird CLI、yt-dlp、Jina Reader、Exa)は有料APIキーが不要な無料ツールです。唯一のオプションコストは、サーバーからReddit/Bilibiliにアクセスする場合のレジデンシャルプロキシ(月額約$1)です。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Twitter APIの無料代替 — Webスクレイピング用</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach はbird CLIを使用し、Cookie認証でTwitterにアクセスします — ブラウザセッションと同じです。API料金なし、レート制限のティアなし、開発者アカウント不要。検索、ツイート閲覧、プロフィール閲覧、タイムラインに対応。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>小紅書のコンテンツをプログラムで読むには?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach は xiaohongshu-mcp(Dockerで実行)と統合されています。セットアップ後、`mcporter call 'xiaohongshu.get_feed_detail(...)'` でノートを閲覧、`mcporter call 'xiaohongshu.search_feeds(keyword: "query")'` で検索できます。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>AIエージェントで抖音の動画を解析するには?</strong></summary>
|
||||||
|
|
||||||
|
douyin-mcp-serverをインストールすれば、`mcporter call 'douyin.parse_douyin_video_info(share_link: "share_url")'` で動画情報を解析し、ウォーターマークなしのダウンロードリンクを取得できます。ログイン不要 — 抖音のリンクを共有するだけ。詳細は https://github.com/yzfly/douyin-mcp-server を参照。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## クレジット
|
||||||
|
|
||||||
|
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://www.npmjs.com/package/@steipete/bird) · [Exa](https://exa.ai) · [feedparser](https://github.com/kurtmckee/feedparser) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||||
|
|
||||||
|
## お問い合わせ
|
||||||
|
|
||||||
|
- 📧 **メール:** pnt01@foxmail.com
|
||||||
|
- 🐦 **Twitter/X:** [@Neo_Reidlab](https://x.com/Neo_Reidlab)
|
||||||
|
|
||||||
|
コラボレーションやご質問は、WeChatで追加してください — コミュニティグループにご招待します:
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="wechat-group-qr.jpg" width="280" alt="WeChat QR">
|
||||||
|
</p>
|
||||||
|
|
||||||
|
> バグ報告や機能リクエストは [GitHub Issues](https://github.com/Panniantong/Agent-Reach/issues) をご利用ください — 追跡が容易です。
|
||||||
|
|
||||||
|
## ライセンス
|
||||||
|
|
||||||
|
[MIT](../LICENSE)
|
||||||
|
|
||||||
|
## Star History
|
||||||
|
|
||||||
|
[](https://star-history.com/#Panniantong/Agent-Reach&Date)
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Dependency Locking Guide
|
||||||
|
|
||||||
|
Agent Reach uses `constraints.txt` as a reproducible dependency baseline.
|
||||||
|
|
||||||
|
## Why
|
||||||
|
|
||||||
|
- Keep local/CI dependency graph stable
|
||||||
|
- Reduce "works on my machine" drift
|
||||||
|
- Make regression results easier to compare
|
||||||
|
|
||||||
|
## Install with constraints
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -c constraints.txt -e .[dev]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Update workflow
|
||||||
|
|
||||||
|
1. Update `pyproject.toml` dependency ranges as needed.
|
||||||
|
2. Validate against latest compatible versions locally.
|
||||||
|
3. Update pinned versions in `constraints.txt`.
|
||||||
|
4. Run validation:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest -q
|
||||||
|
ruff check agent_reach tests
|
||||||
|
mypy agent_reach
|
||||||
|
```
|
||||||
|
|
||||||
|
5. Open PR with dependency and validation notes.
|
||||||
+234
-76
@@ -20,7 +20,7 @@ Copy this to your AI Agent:
|
|||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Install Agent Reach and make sure **all channels are working**. Your user expects you to have full internet access capabilities when this is done — reading tweets, searching Reddit, watching YouTube, browsing GitHub, and more.
|
Install Agent Reach and all upstream tools so your user's agent has full internet access. After installation, you'll use the upstream tools directly (bird CLI, yt-dlp, mcporter, gh CLI, etc.) — Agent Reach is just the installer and health checker.
|
||||||
|
|
||||||
### ⚠️ Boundaries
|
### ⚠️ Boundaries
|
||||||
|
|
||||||
@@ -30,47 +30,84 @@ When fixing issues, stay within these limits:
|
|||||||
- **DO NOT** modify system files outside `~/.agent-reach/`
|
- **DO NOT** modify system files outside `~/.agent-reach/`
|
||||||
- **DO NOT** install packages not listed in this guide
|
- **DO NOT** install packages not listed in this guide
|
||||||
- **DO NOT** disable firewalls, security settings, or system protections
|
- **DO NOT** disable firewalls, security settings, or system protections
|
||||||
|
- **DO NOT** clone repos, create files, or run commands inside the agent workspace / working directory
|
||||||
- If something requires elevated permissions, **tell the user** what's needed and let them decide
|
- If something requires elevated permissions, **tell the user** what's needed and let them decide
|
||||||
|
|
||||||
### Step 1: Install
|
### 📁 Directory Rules
|
||||||
|
|
||||||
**Default (fully automatic):**
|
All Agent Reach files go in dedicated directories — **never in the agent workspace**:
|
||||||
|
|
||||||
|
| Purpose | Directory | Example |
|
||||||
|
|---------|-----------|---------|
|
||||||
|
| Config & tokens | `~/.agent-reach/` | `~/.agent-reach/config.json` |
|
||||||
|
| Upstream tool repos | `~/.agent-reach/tools/` | `~/.agent-reach/tools/douyin-mcp-server/` |
|
||||||
|
| Temporary files | `/tmp/` | `/tmp/yt-dlp-output/` |
|
||||||
|
| Skills | `~/.openclaw/skills/agent-reach/` | SKILL.md |
|
||||||
|
|
||||||
|
**Why?** If you clone repos or create files in the workspace, it pollutes the user's project directory and can break their agent over time. Keep the workspace clean.
|
||||||
|
|
||||||
|
### Step 1: Install the basics
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# 推荐:pipx(最省心)
|
||||||
|
pipx install https://github.com/Panniantong/agent-reach/archive/main.zip
|
||||||
|
agent-reach install --env=auto
|
||||||
|
|
||||||
|
# 如果你的 Python 来自 Homebrew / 遇到 PEP 668(externally-managed-environment)
|
||||||
|
# 用虚拟环境安装:
|
||||||
|
python3 -m venv ~/.agent-reach-venv
|
||||||
|
source ~/.agent-reach-venv/bin/activate
|
||||||
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
||||||
agent-reach install --env=auto
|
agent-reach install --env=auto
|
||||||
```
|
```
|
||||||
|
|
||||||
This auto-installs system dependencies (gh CLI, Node.js, mcporter, bird), configures Exa search, detects environment, and tests all channels.
|
This installs core infrastructure (gh CLI, Node.js, mcporter, Exa search, yt-dlp config) and activates these zero-config channels:
|
||||||
|
|
||||||
**Safe mode (no automatic system changes):**
|
- Web (Jina Reader), YouTube, GitHub, RSS, Exa Search, V2EX, Bilibili (basic)
|
||||||
|
|
||||||
If the user asked for safe mode, or you're on a production machine:
|
> 💡 **macOS / Homebrew Python 提示 `externally-managed-environment`?**
|
||||||
|
> 这是 PEP 668 保护,不是 Agent Reach 本身的问题。优先用 `pipx install ...`,或先创建 `venv` 再安装。
|
||||||
|
|
||||||
|
**Safe mode / Dry run:**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
agent-reach install --env=auto --safe # Check only, no auto-install
|
||||||
agent-reach install --env=auto --safe
|
agent-reach install --env=auto --dry-run # Preview what would be done
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe mode checks what's installed and tells you what's missing, but doesn't auto-install system packages. You can then install each dependency manually.
|
### Step 2: Ask the user which optional channels they want
|
||||||
|
|
||||||
**Dry run (preview only):**
|
After installing the basics, **ask the user** which additional channels they need. Present this list:
|
||||||
|
|
||||||
|
> 基础渠道装好了!你现在可以让我搜网页、看 YouTube、读 GitHub 等。
|
||||||
|
>
|
||||||
|
> 还有这些可选渠道,你需要哪些?
|
||||||
|
>
|
||||||
|
> - 🐦 **Twitter/X** — 搜推文、看时间线(需要登录 Cookie)
|
||||||
|
> - 📰 **微博** — 热搜、搜索、用户动态
|
||||||
|
> - 💬 **微信公众号** — 搜索和阅读公众号文章
|
||||||
|
> - 📈 **雪球** — 股票行情、热门帖子(需要登录 Cookie)
|
||||||
|
> - 🎙️ **小宇宙播客** — 音频转文字(需要免费 Groq Key)
|
||||||
|
> - 📕 **小红书** — 阅读、搜索、发帖(需要登录)
|
||||||
|
> - 📖 **Reddit** — 搜索和阅读帖子
|
||||||
|
> - 📺 **B站完整版** — 热门、排行、搜索
|
||||||
|
> - 🎵 **抖音** — 视频解析
|
||||||
|
> - 💼 **LinkedIn** — Profile、职位搜索
|
||||||
|
>
|
||||||
|
> 告诉我你要哪些,比如"帮我装微博和 Twitter"。或者说"全部装"。
|
||||||
|
|
||||||
|
Based on the user's choice, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
agent-reach install --env=auto --dry-run
|
agent-reach install --env=auto --channels=twitter,weibo # Example: user chose Twitter + Weibo
|
||||||
|
agent-reach install --env=auto --channels=all # User wants everything
|
||||||
```
|
```
|
||||||
|
|
||||||
Shows what would be done without making any changes.
|
Supported channel names: `twitter`, `weibo`, `wechat`, `xiaoyuzhou`, `xueqiu`, `xiaohongshu`, `reddit`, `bilibili`, `douyin`, `linkedin`, `all`
|
||||||
|
|
||||||
**Read the output carefully**, then run:
|
### Step 3: Fix what's broken
|
||||||
|
|
||||||
```bash
|
Run `agent-reach doctor` and check the output.
|
||||||
agent-reach doctor
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 2: Fix what's broken
|
|
||||||
|
|
||||||
`agent-reach doctor` shows you which channels are active and which aren't.
|
|
||||||
|
|
||||||
Try to get as many channels to ✅ as possible. If something failed during install or shows as ❌/⚠️ in doctor, diagnose the problem and try to fix it — but stay within the boundaries above. If a fix requires elevated permissions or system changes, ask the user first.
|
Try to get as many channels to ✅ as possible. If something failed during install or shows as ❌/⚠️ in doctor, diagnose the problem and try to fix it — but stay within the boundaries above. If a fix requires elevated permissions or system changes, ask the user first.
|
||||||
|
|
||||||
@@ -80,16 +117,39 @@ Only ask the user when you genuinely need their input (credentials, permissions,
|
|||||||
|
|
||||||
Some channels need credentials only the user can provide. Based on the doctor output, ask for what's missing:
|
Some channels need credentials only the user can provide. Based on the doctor output, ask for what's missing:
|
||||||
|
|
||||||
> 🔒 **Security tip:** For platforms that need cookies (Twitter, XiaoHongShu, Instagram), we recommend using a **dedicated/secondary account** rather than your main account. Cookie-based auth grants full account access — using a separate account limits the blast radius if credentials are ever compromised.
|
> 🔒 **Security tip:** For platforms that need cookies (Twitter, XiaoHongShu), we recommend using a **dedicated/secondary account** rather than your main account. Cookie-based auth carries two risks:
|
||||||
|
> 1. **Account ban** — platforms may detect non-browser API calls and restrict or ban the account
|
||||||
|
> 2. **Credential exposure** — cookies grant full account access; using a secondary account limits the blast radius if credentials are ever compromised
|
||||||
|
|
||||||
**Twitter search & posting (server users):**
|
> 🍪 **Cookie 导入(所有需要登录的平台通用):**
|
||||||
|
>
|
||||||
|
> 所有需要 Cookie 的平台(Twitter、小红书、雪球等),**优先使用 Cookie-Editor 导入**,这是最简单最可靠的方式:
|
||||||
|
> 1. 用户在自己的浏览器上登录对应平台
|
||||||
|
> 2. 安装 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome 插件
|
||||||
|
> 3. 点击插件 → Export → Header String
|
||||||
|
> 4. 把导出的字符串发给 Agent
|
||||||
|
>
|
||||||
|
> **本地电脑用户**也可以用 `agent-reach configure --from-browser chrome` 一键自动提取(支持 Twitter + 小红书 + 雪球)。
|
||||||
|
|
||||||
|
**Twitter search & posting:**
|
||||||
> "To unlock Twitter search, I need your Twitter cookies. Install the Cookie-Editor Chrome extension, go to x.com/twitter.com, click the extension → Export → Header String, and paste it to me."
|
> "To unlock Twitter search, I need your Twitter cookies. Install the Cookie-Editor Chrome extension, go to x.com/twitter.com, click the extension → Export → Header String, and paste it to me."
|
||||||
> Extension: https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
agent-reach configure twitter-cookies "PASTED_STRING"
|
agent-reach configure twitter-cookies "PASTED_STRING"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **代理说明(中国大陆等需要翻墙的网络环境):**
|
||||||
|
>
|
||||||
|
> bird CLI 使用 Node.js,在需要代理的网络环境下可通过环境变量配置代理。
|
||||||
|
> Agent Reach 安装时会自动安装 `undici`(`npm install -g undici`)以提供代理支持。
|
||||||
|
>
|
||||||
|
> **你(Agent)需要做的:**
|
||||||
|
> 1. 确认用户配了代理:`agent-reach configure proxy http://user:pass@ip:port`
|
||||||
|
> 2. 设置环境变量:`export HTTP_PROXY="..." HTTPS_PROXY="..."`
|
||||||
|
> 3. Agent Reach 会自动处理剩下的,不需要用户做额外操作
|
||||||
|
>
|
||||||
|
> 如果用户报告 "fetch failed",参考 [troubleshooting.md](troubleshooting.md)
|
||||||
|
|
||||||
**Reddit & Bilibili full access (server users):**
|
**Reddit & Bilibili full access (server users):**
|
||||||
> "Reddit and Bilibili block server IPs. To unlock full access, I need a residential proxy. You can get one at https://webshare.io ($1/month). Send me the proxy address."
|
> "Reddit and Bilibili block server IPs. To unlock full access, I need a residential proxy. You can get one at https://webshare.io ($1/month). Send me the proxy address."
|
||||||
|
|
||||||
@@ -108,23 +168,140 @@ mcporter config add xiaohongshu http://localhost:18060/mcp
|
|||||||
> 如果在服务器上,建议加代理避免 IP 风控:
|
> 如果在服务器上,建议加代理避免 IP 风控:
|
||||||
> `docker run -d --name xiaohongshu-mcp -p 18060:18060 -e XHS_PROXY=http://user:pass@ip:port xpzouying/xiaohongshu-mcp`
|
> `docker run -d --name xiaohongshu-mcp -p 18060:18060 -e XHS_PROXY=http://user:pass@ip:port xpzouying/xiaohongshu-mcp`
|
||||||
>
|
>
|
||||||
> **登录方式:**
|
> **登录方式(优先用 Cookie-Editor,最简单):**
|
||||||
> - **本地电脑(有浏览器):** 打开 http://localhost:18060 扫码登录即可。
|
> 1. 用户在自己的浏览器登录小红书 (xiaohongshu.com)
|
||||||
> - **服务器(无 UI 界面):** 服务器上通常没有浏览器,无法直接扫码。最方便的方式是在自己的电脑上用浏览器登录小红书,然后用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 插件导出 Cookie(Header String 格式),发给 Agent 即可完成配置。详见 [Cookie 导出指南](cookie-export.md)。
|
> 2. 用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 插件导出 Cookie(JSON 或 Header String 格式均可)
|
||||||
|
> 3. 把 Cookie 字符串发给 Agent
|
||||||
|
> 4. Agent 运行命令完成登录:
|
||||||
|
>
|
||||||
|
> ```bash
|
||||||
|
> # JSON 格式(Cookie-Editor → Export → JSON)
|
||||||
|
> agent-reach configure xhs-cookies '[{"name":"web_session","value":"xxx","domain":".xiaohongshu.com",...}]'
|
||||||
|
>
|
||||||
|
> # 或 Header String 格式(Cookie-Editor → Export → Header String)
|
||||||
|
> agent-reach configure xhs-cookies "key1=val1; key2=val2; ..."
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> **注意:** `http://localhost:18060` 根路径可能返回 404,MCP 服务在 `/mcp` 路径。推荐使用 Cookie-Editor 导出方式,不要依赖 Docker 容器内的 QR 扫码登录。
|
||||||
|
|
||||||
**Instagram (需要 instaloader):**
|
**微博 / Weibo (mcp-server-weibo):**
|
||||||
> "Instagram 需要 instaloader。我来帮你安装。"
|
> "微博已默认安装,装好即用。可搜索微博内容、查看热搜、获取用户动态和评论。"
|
||||||
|
|
||||||
|
如果自动安装失败,手动安装:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install instaloader
|
pip install git+https://github.com/Panniantong/mcp-server-weibo.git
|
||||||
|
mcporter config add weibo --command 'mcp-server-weibo'
|
||||||
```
|
```
|
||||||
|
|
||||||
> **登录方式(解锁私密内容):**
|
> 无需登录、无需 Cookie、无需代理。海外服务器也可以直接访问。
|
||||||
> - **方法 1(推荐):Cookie-Editor 导入:** 在浏览器登录 Instagram → 用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 导出 Header String → 粘贴:
|
|
||||||
> ```bash
|
**雪球 / Xueqiu (股票行情 + 热门帖子):**
|
||||||
> agent-reach configure instagram-cookies "sessionid=xxx; csrftoken=yyy; ..."
|
> "雪球需要登录后的 Cookie。请先在 Chrome 里登录 xueqiu.com,然后运行:"
|
||||||
> ```
|
|
||||||
> - **方法 2:instaloader 命令行登录:** `instaloader --login YOUR_USERNAME`(需要输密码,有 2FA 的话还要输验证码)
|
```bash
|
||||||
|
agent-reach configure --from-browser chrome
|
||||||
|
```
|
||||||
|
|
||||||
|
> Cookie 会随其他平台一起自动提取。
|
||||||
|
|
||||||
|
**小宇宙播客 / Xiaoyuzhou Podcast (Groq Whisper):**
|
||||||
|
> "小宇宙播客转文字已默认安装,只需要一个免费的 Groq API Key。"
|
||||||
|
|
||||||
|
脚本已随 Agent Reach 自动安装,用户只需提供 Key:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach configure groq-key gsk_xxxxx
|
||||||
|
```
|
||||||
|
|
||||||
|
> **获取 Groq API Key(免费、无需信用卡、30 秒搞定):**
|
||||||
|
> 1. 打开 https://console.groq.com
|
||||||
|
> 2. 用 Google/GitHub 账号登录(或注册)
|
||||||
|
> 3. 左侧菜单 → API Keys → Create API Key
|
||||||
|
> 4. 复制 Key(以 `gsk_` 开头),发给 Agent 即可
|
||||||
|
>
|
||||||
|
> **使用方式:**
|
||||||
|
> 用户发一个小宇宙链接给 Agent,Agent 自动调用:
|
||||||
|
> ```bash
|
||||||
|
> bash ~/.agent-reach/tools/xiaoyuzhou/transcribe.sh https://www.xiaoyuzhoufm.com/episode/xxxxx
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> 自动下载音频 → 转码切片 → Groq Whisper 转录 → 输出完整中文文字稿。
|
||||||
|
>
|
||||||
|
> **免费额度和限制:**
|
||||||
|
> - 每小时约 2 小时音频(7200 秒),超出后等 15 分钟自动恢复
|
||||||
|
> - 日常听几期播客完全够用
|
||||||
|
> - 转录质量高(Whisper large-v3),但不区分说话人
|
||||||
|
> - 2 小时以上的播客建议分批处理
|
||||||
|
|
||||||
|
**抖音 / Douyin (douyin-mcp-server):**
|
||||||
|
> "抖音视频解析需要一个 MCP 服务。安装 douyin-mcp-server 后即可解析视频、获取无水印下载链接。"
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. 安装
|
||||||
|
pip install douyin-mcp-server
|
||||||
|
|
||||||
|
# 2. 启动 HTTP 服务(端口 18070)
|
||||||
|
# 方式一:用 uv(推荐)
|
||||||
|
mkdir -p ~/.agent-reach/tools && cd ~/.agent-reach/tools
|
||||||
|
git clone https://github.com/yzfly/douyin-mcp-server.git && cd douyin-mcp-server
|
||||||
|
uv sync && uv run python run_http.py
|
||||||
|
|
||||||
|
# 方式二:直接用 Python 启动
|
||||||
|
python -c "
|
||||||
|
from douyin_mcp_server.server import mcp
|
||||||
|
mcp.settings.host = '127.0.0.1'
|
||||||
|
mcp.settings.port = 18070
|
||||||
|
mcp.run(transport='streamable-http')
|
||||||
|
"
|
||||||
|
|
||||||
|
# 3. 注册到 mcporter
|
||||||
|
mcporter config add douyin http://localhost:18070/mcp
|
||||||
|
```
|
||||||
|
|
||||||
|
> 无需认证即可解析视频信息和获取下载链接。
|
||||||
|
> 如需 AI 语音识别提取文案功能,需要配置硅基流动 API Key(`export API_KEY="sk-xxx"`)。
|
||||||
|
>
|
||||||
|
> 详见 https://github.com/yzfly/douyin-mcp-server
|
||||||
|
|
||||||
|
**可选实现:Douyin + XiaoHongShu unified extractor**
|
||||||
|
> "如果你想把抖音和小红书统一成一个 MCP,并直接输出 `script.md` 和 `info.json`,可以改用 social-post-extractor-mcp。"
|
||||||
|
|
||||||
|
适用场景:
|
||||||
|
|
||||||
|
- 抖音视频转文字稿
|
||||||
|
- 小红书视频笔记转文字稿
|
||||||
|
- 小红书图文笔记正文 + 图片文字提取
|
||||||
|
|
||||||
|
兼容性:
|
||||||
|
|
||||||
|
- 仍然可以注册成 `douyin` 这个 mcporter server 名称
|
||||||
|
- 兼容旧工具名 `parse_douyin_video_info` / `get_douyin_download_link` / `extract_douyin_text`
|
||||||
|
- 同时新增 `parse_social_post_info` / `extract_social_post_script`
|
||||||
|
|
||||||
|
示例配置:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/JNHFlow21/social-post-extractor-mcp.git
|
||||||
|
cd social-post-extractor-mcp
|
||||||
|
uv sync
|
||||||
|
|
||||||
|
mcporter config add douyin \
|
||||||
|
--command /bin/zsh \
|
||||||
|
--arg -lc \
|
||||||
|
--arg "cd '$PWD' && exec '.venv/bin/python' -m social_post_extractor_mcp" \
|
||||||
|
--env ASR_PROVIDER=bailian \
|
||||||
|
--env ASR_MODEL=paraformer-v2 \
|
||||||
|
--env VISION_PROVIDER=bailian \
|
||||||
|
--env VISION_MODEL=qwen3-vl-flash \
|
||||||
|
--env CLEAN_PROVIDER=bailian \
|
||||||
|
--env CLEAN_MODEL=qwen-flash \
|
||||||
|
--env BAILIAN_API_KEY=YOUR_BAILIAN_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
> 这个实现更适合“把链接直接交给 Agent,然后拿到脚本文件”的工作流。
|
||||||
|
>
|
||||||
|
> 详见 https://github.com/JNHFlow21/social-post-extractor-mcp
|
||||||
|
|
||||||
**LinkedIn (可选 — linkedin-scraper-mcp):**
|
**LinkedIn (可选 — linkedin-scraper-mcp):**
|
||||||
> "LinkedIn 基本内容可通过 Jina Reader 读取。完整功能(Profile 详情、职位搜索)需要 linkedin-scraper-mcp。"
|
> "LinkedIn 基本内容可通过 Jina Reader 读取。完整功能(Profile 详情、职位搜索)需要 linkedin-scraper-mcp。"
|
||||||
@@ -165,35 +342,6 @@ pip install linkedin-scraper-mcp
|
|||||||
>
|
>
|
||||||
> 详见 https://github.com/stickerdaniel/linkedin-mcp-server
|
> 详见 https://github.com/stickerdaniel/linkedin-mcp-server
|
||||||
|
|
||||||
**Boss直聘 (可选 — mcp-bosszp):**
|
|
||||||
> "Boss直聘职位页面可直接读取。完整搜索和打招呼功能需要 mcp-bosszp。"
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git clone https://github.com/mucsbr/mcp-bosszp.git && cd mcp-bosszp
|
|
||||||
pip install -r requirements.txt && playwright install chromium
|
|
||||||
```
|
|
||||||
|
|
||||||
> **登录方式(手机 App 扫码):**
|
|
||||||
>
|
|
||||||
> mcp-bosszp 使用二维码登录,需要 Boss直聘 App 扫码。
|
|
||||||
>
|
|
||||||
> ```bash
|
|
||||||
> # 1. 启动 MCP 服务
|
|
||||||
> python boss_zhipin_fastmcp_v2.py
|
|
||||||
>
|
|
||||||
> # 2. 通过 MCP 触发登录(会生成二维码图片)
|
|
||||||
> mcporter call 'bosszhipin.start_login()'
|
|
||||||
>
|
|
||||||
> # 3. 浏览器打开二维码图片(服务器上用 http://服务器IP:8000/static/qrcode_xxx.png)
|
|
||||||
> # 4. 用 Boss直聘 App 扫码确认
|
|
||||||
> ```
|
|
||||||
>
|
|
||||||
> 扫码成功后 session 自动保存,之后可以搜索职位和自动打招呼。
|
|
||||||
>
|
|
||||||
> 或用 Docker:`docker-compose up -d`
|
|
||||||
>
|
|
||||||
> 详见 https://github.com/mucsbr/mcp-bosszp
|
|
||||||
|
|
||||||
### Step 4: Final check
|
### Step 4: Final check
|
||||||
|
|
||||||
Run `agent-reach doctor` one final time and report the results to your user.
|
Run `agent-reach doctor` one final time and report the results to your user.
|
||||||
@@ -221,7 +369,9 @@ If the user wants a different agent to handle it, let them choose.
|
|||||||
|
|
||||||
| Command | What it does |
|
| Command | What it does |
|
||||||
|---------|-------------|
|
|---------|-------------|
|
||||||
| `agent-reach install --env=auto` | Full auto-setup (installs deps + configures) |
|
| `agent-reach install --env=auto` | Install core channels (lightweight, zero-config) |
|
||||||
|
| `agent-reach install --env=auto --channels=twitter,weibo` | Install core + optional channels |
|
||||||
|
| `agent-reach install --env=auto --channels=all` | Install everything |
|
||||||
| `agent-reach install --env=auto --safe` | Safe setup (no auto system changes) |
|
| `agent-reach install --env=auto --safe` | Safe setup (no auto system changes) |
|
||||||
| `agent-reach install --env=auto --dry-run` | Preview what would be done |
|
| `agent-reach install --env=auto --dry-run` | Preview what would be done |
|
||||||
| `agent-reach doctor` | Show channel status |
|
| `agent-reach doctor` | Show channel status |
|
||||||
@@ -229,14 +379,22 @@ If the user wants a different agent to handle it, let them choose.
|
|||||||
| `agent-reach check-update` | Check for new versions |
|
| `agent-reach check-update` | Check for new versions |
|
||||||
| `agent-reach configure twitter-cookies "..."` | Unlock Twitter search + posting |
|
| `agent-reach configure twitter-cookies "..."` | Unlock Twitter search + posting |
|
||||||
| `agent-reach configure proxy URL` | Unlock Reddit + Bilibili on servers |
|
| `agent-reach configure proxy URL` | Unlock Reddit + Bilibili on servers |
|
||||||
| `agent-reach read <url>` | Read any URL |
|
| `agent-reach configure groq-key gsk_xxx` | Unlock Xiaoyuzhou podcast transcription |
|
||||||
| `agent-reach search "query"` | Search the web |
|
|
||||||
| `agent-reach search-reddit "query"` | Search Reddit |
|
After installation, use upstream tools directly. See SKILL.md for the full command reference:
|
||||||
| `agent-reach search-twitter "query"` | Search Twitter |
|
|
||||||
| `agent-reach search-github "query"` | Search GitHub |
|
| Platform | Upstream Tool | Example |
|
||||||
| `agent-reach search-youtube "query"` | Search YouTube |
|
|----------|--------------|---------|
|
||||||
| `agent-reach search-bilibili "query"` | Search Bilibili |
|
| Twitter/X | `bird` | `bird search "query" -n 10` |
|
||||||
| `agent-reach search-xhs "query"` | Search XiaoHongShu |
|
| YouTube | `yt-dlp` | `yt-dlp --dump-json URL` |
|
||||||
| `agent-reach search-instagram "query"` | Search Instagram |
|
| Bilibili | `yt-dlp` | `yt-dlp --dump-json URL` |
|
||||||
| `agent-reach search-linkedin "query"` | Search LinkedIn |
|
| Reddit | `mcporter` (Exa) | `mcporter call 'exa.web_search_exa(query: "...", includeDomains: ["reddit.com"])'` |
|
||||||
| `agent-reach search-bosszhipin "query"` | Search Boss直聘 |
|
| GitHub | `gh` | `gh search repos "query"` |
|
||||||
|
| Web | `curl` + Jina | `curl -s "https://r.jina.ai/URL"` |
|
||||||
|
| Exa Search | `mcporter` | `mcporter call 'exa.web_search_exa(...)'` |
|
||||||
|
| 小红书 | `mcporter` | `mcporter call 'xiaohongshu.search_feeds(...)'` |
|
||||||
|
| 微博 | `mcporter` | `mcporter call 'weibo.get_trendings(limit: 10)'` |
|
||||||
|
| 小宇宙播客 | `transcribe.sh` | `bash ~/.agent-reach/tools/xiaoyuzhou/transcribe.sh <URL>` |
|
||||||
|
| 抖音 | `mcporter` | `mcporter call 'douyin.parse_douyin_video_info(...)'` |
|
||||||
|
| LinkedIn | `mcporter` | `mcporter call 'linkedin.get_person_profile(...)'` |
|
||||||
|
| RSS | `feedparser` | `python3 -c "import feedparser; ..."` |
|
||||||
|
|||||||
@@ -0,0 +1,59 @@
|
|||||||
|
# 常见问题排查
|
||||||
|
|
||||||
|
## 雪球 / Xueqiu: API 返回 400
|
||||||
|
|
||||||
|
**症状:** `agent-reach doctor` 显示雪球 ⚠️,报 `HTTP Error 400`
|
||||||
|
|
||||||
|
**原因:** 雪球 API 需要登录 Cookie,无法通过匿名访问获取。
|
||||||
|
|
||||||
|
**解决方案:** 在 Chrome 里登录 xueqiu.com,然后运行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach configure --from-browser chrome
|
||||||
|
```
|
||||||
|
|
||||||
|
再次运行 `agent-reach doctor` 确认恢复 ✅。Cookie 过期后重新运行即可。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Twitter/X: bird CLI 连接失败
|
||||||
|
|
||||||
|
**症状:** `bird search` 或其他命令返回错误
|
||||||
|
|
||||||
|
**原因:** bird CLI 需要 AUTH_TOKEN 和 CT0 环境变量才能访问 Twitter API。如果你的网络环境需要代理才能访问 x.com,需要配置代理。
|
||||||
|
|
||||||
|
**解决方案:**
|
||||||
|
|
||||||
|
### 方案 1:设置环境变量代理
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export HTTP_PROXY="http://user:pass@host:port"
|
||||||
|
export HTTPS_PROXY="http://user:pass@host:port"
|
||||||
|
bird search "test" -n 1
|
||||||
|
```
|
||||||
|
|
||||||
|
### 方案 2:使用全局代理工具
|
||||||
|
|
||||||
|
让代理工具接管所有网络流量,这样 bird 的请求也会走代理:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# macOS — ClashX / Surge 开启"增强模式"
|
||||||
|
# Linux — proxychains 或 tun2socks
|
||||||
|
proxychains bird search "test" -n 1
|
||||||
|
```
|
||||||
|
|
||||||
|
### 方案 3:不用 bird,用 Exa 搜索替代
|
||||||
|
|
||||||
|
bird 不可用时,可以直接用 Exa 搜索 Twitter 内容:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mcporter call 'exa.web_search_exa(query: "site:x.com 搜索词", numResults: 5)'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 方案 4:检查认证
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bird check
|
||||||
|
```
|
||||||
|
|
||||||
|
> 如果返回 "Missing credentials",需要设置 AUTH_TOKEN 和 CT0 环境变量。
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
# Agent Reach — Update Guide
|
||||||
|
|
||||||
|
## For Humans
|
||||||
|
|
||||||
|
Copy this to your AI Agent:
|
||||||
|
|
||||||
|
```
|
||||||
|
帮我更新 Agent Reach:https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Or in English:
|
||||||
|
|
||||||
|
```
|
||||||
|
Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## For AI Agents
|
||||||
|
|
||||||
|
### ⚠️ Workspace Rules
|
||||||
|
|
||||||
|
**Never create files, clone repos, or run commands in the agent workspace.** Use `/tmp/` for temporary work and `~/.agent-reach/` for persistent data.
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
Update Agent Reach to the latest version, then verify everything still works.
|
||||||
|
|
||||||
|
### Step 1: Check current version
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach check-update
|
||||||
|
```
|
||||||
|
|
||||||
|
If it says "已是最新版本", you're done — tell the user.
|
||||||
|
|
||||||
|
### Step 2: Update
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install --upgrade https://github.com/Panniantong/agent-reach/archive/main.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Verify
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach version
|
||||||
|
agent-reach doctor
|
||||||
|
```
|
||||||
|
|
||||||
|
Report the new version and channel status to the user. If any channel changed status (was ✅ now ❌), investigate and fix.
|
||||||
|
|
||||||
|
### Step 4: Update SKILL.md (if installed)
|
||||||
|
|
||||||
|
If the user has the Agent Reach skill installed, update it too:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
agent-reach install --skill-only
|
||||||
|
```
|
||||||
|
|
||||||
|
If the `--skill-only` flag is not available, manually copy the latest SKILL.md:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -c "
|
||||||
|
import importlib.resources, shutil, os
|
||||||
|
skill = importlib.resources.files('agent_reach').joinpath('skill', 'SKILL.md').read_text()
|
||||||
|
for d in [os.path.expanduser('~/.openclaw/skills/agent-reach'),
|
||||||
|
os.path.expanduser('~/.claude/skills/agent-reach')]:
|
||||||
|
if os.path.isdir(d):
|
||||||
|
with open(os.path.join(d, 'SKILL.md'), 'w') as f:
|
||||||
|
f.write(skill)
|
||||||
|
print(f'✅ Updated: {d}')
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
Done. Tell the user what version they're now on and how many channels are available.
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 145 KiB |
@@ -0,0 +1,32 @@
|
|||||||
|
# Agent Reach
|
||||||
|
|
||||||
|
> Give your AI agent eyes to see the entire internet. A unified CLI tool that lets AI agents read and search across 12+ platforms — Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, Instagram, LinkedIn, Boss直聘, RSS, and any web page. One install, zero API fees.
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
- [Installation Guide](https://github.com/Panniantong/Agent-Reach/blob/main/docs/install.md): Step-by-step setup instructions for AI agents
|
||||||
|
- [README (中文)](https://github.com/Panniantong/Agent-Reach/blob/main/README.md): Full documentation in Chinese
|
||||||
|
- [README (English)](https://github.com/Panniantong/Agent-Reach/blob/main/docs/README_en.md): Full documentation in English
|
||||||
|
|
||||||
|
## Core Commands
|
||||||
|
|
||||||
|
- [CLI Usage](https://github.com/Panniantong/Agent-Reach/blob/main/docs/install.md): `agent-reach read <url>` reads any URL, `agent-reach search-twitter "query"` searches Twitter, etc.
|
||||||
|
|
||||||
|
## Key Features
|
||||||
|
|
||||||
|
- Read any URL: tweets, Reddit posts, YouTube videos (transcripts), GitHub repos, articles, XiaoHongShu notes, Bilibili videos, RSS feeds
|
||||||
|
- Search across platforms: Twitter/X, Reddit, GitHub, YouTube, Bilibili, XiaoHongShu, Instagram, LinkedIn, Boss直聘, Web (via Exa)
|
||||||
|
- Self-diagnosis: `agent-reach doctor` checks what works and what needs setup
|
||||||
|
- Auto-installs dependencies: `agent-reach install --env=auto`
|
||||||
|
- Cookie-based auth for platforms that require login (Twitter, XiaoHongShu, Instagram)
|
||||||
|
- Proxy support for platforms that block server IPs (Reddit, Bilibili)
|
||||||
|
- Zero API fees: all backends are free and open-source (bird CLI, yt-dlp, Jina Reader, instaloader, etc.)
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- [Troubleshooting Guide](https://github.com/Panniantong/Agent-Reach/blob/main/docs/troubleshooting.md): Common issues and solutions
|
||||||
|
|
||||||
|
## Optional
|
||||||
|
|
||||||
|
- [SKILL.md](https://github.com/Panniantong/Agent-Reach/blob/main/agent_reach/skill/SKILL.md): Integration guide for AI agent frameworks (OpenClaw, Claude Code, etc.)
|
||||||
|
- [PyPI Package](https://pypi.org/project/agent-reach/): `pip install agent-reach`
|
||||||
+35
-2
@@ -1,12 +1,20 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "agent-reach"
|
name = "agent-reach"
|
||||||
version = "1.1.0"
|
version = "1.4.0"
|
||||||
description = "Give your AI Agent eyes to see the entire internet. Search + Read 10+ platforms."
|
description = "Give your AI Agent eyes to see the entire internet. Search + Read 10+ platforms."
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
license = {text = "MIT"}
|
license = {text = "MIT"}
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
authors = [{name = "Neo Reid"}]
|
authors = [{name = "Neo Reid"}]
|
||||||
keywords = ["ai-agent", "web-reader", "search", "mcp", "agent-infrastructure", "llm-tools"]
|
keywords = [
|
||||||
|
"ai-agent", "llm-tools", "agent-infrastructure", "mcp",
|
||||||
|
"web-reader", "web-scraper", "search",
|
||||||
|
"twitter-scraper", "reddit-scraper", "youtube-transcript",
|
||||||
|
"bilibili", "xiaohongshu",
|
||||||
|
"ai-search", "cli", "automation",
|
||||||
|
"claude-code", "cursor", "openai",
|
||||||
|
"free-api", "no-api-key",
|
||||||
|
]
|
||||||
classifiers = [
|
classifiers = [
|
||||||
"Development Status :: 4 - Beta",
|
"Development Status :: 4 - Beta",
|
||||||
"Intended Audience :: Developers",
|
"Intended Audience :: Developers",
|
||||||
@@ -33,6 +41,13 @@ dependencies = [
|
|||||||
browser = ["playwright>=1.40"]
|
browser = ["playwright>=1.40"]
|
||||||
cookies = ["browser-cookie3>=0.19"]
|
cookies = ["browser-cookie3>=0.19"]
|
||||||
all = ["playwright>=1.40", "mcp[cli]>=1.0", "browser-cookie3>=0.19"]
|
all = ["playwright>=1.40", "mcp[cli]>=1.0", "browser-cookie3>=0.19"]
|
||||||
|
dev = [
|
||||||
|
"pytest>=8.0",
|
||||||
|
"ruff>=0.8",
|
||||||
|
"mypy>=1.12",
|
||||||
|
"types-requests>=2.32",
|
||||||
|
"types-PyYAML>=6.0",
|
||||||
|
]
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
agent-reach = "agent_reach.cli:main"
|
agent-reach = "agent_reach.cli:main"
|
||||||
@@ -52,3 +67,21 @@ packages = ["agent_reach"]
|
|||||||
[tool.hatch.build.targets.wheel.force-include]
|
[tool.hatch.build.targets.wheel.force-include]
|
||||||
"agent_reach/guides" = "agent_reach/guides"
|
"agent_reach/guides" = "agent_reach/guides"
|
||||||
"agent_reach/skill" = "agent_reach/skill"
|
"agent_reach/skill" = "agent_reach/skill"
|
||||||
|
"agent_reach/scripts" = "agent_reach/scripts"
|
||||||
|
|
||||||
|
[tool.ruff]
|
||||||
|
target-version = "py310"
|
||||||
|
line-length = 100
|
||||||
|
|
||||||
|
[tool.ruff.lint]
|
||||||
|
select = ["E", "F", "I"]
|
||||||
|
ignore = ["E501"]
|
||||||
|
|
||||||
|
[tool.mypy]
|
||||||
|
python_version = "3.10"
|
||||||
|
warn_unused_configs = true
|
||||||
|
warn_redundant_casts = true
|
||||||
|
warn_unused_ignores = true
|
||||||
|
check_untyped_defs = true
|
||||||
|
ignore_missing_imports = true
|
||||||
|
exclude = ["^tests/"]
|
||||||
|
|||||||
@@ -0,0 +1,155 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Contract tests for channel adapters."""
|
||||||
|
|
||||||
|
from agent_reach.channels import get_all_channels
|
||||||
|
from agent_reach.config import Config
|
||||||
|
|
||||||
|
|
||||||
|
def test_channel_registry_contract():
|
||||||
|
channels = get_all_channels()
|
||||||
|
assert channels, "channel registry must not be empty"
|
||||||
|
names = [ch.name for ch in channels]
|
||||||
|
assert len(names) == len(set(names)), "channel names must be unique"
|
||||||
|
|
||||||
|
for ch in channels:
|
||||||
|
assert isinstance(ch.name, str) and ch.name
|
||||||
|
assert isinstance(ch.description, str) and ch.description
|
||||||
|
assert isinstance(ch.backends, list)
|
||||||
|
assert ch.tier in {0, 1, 2}
|
||||||
|
|
||||||
|
|
||||||
|
def test_channel_check_contract_with_minimal_runtime(monkeypatch, tmp_path):
|
||||||
|
# Keep contract tests deterministic by simulating "deps mostly absent".
|
||||||
|
monkeypatch.setattr("shutil.which", lambda _cmd: None)
|
||||||
|
config = Config(config_path=tmp_path / "config.yaml")
|
||||||
|
|
||||||
|
for ch in get_all_channels():
|
||||||
|
status, message = ch.check(config)
|
||||||
|
assert status in {"ok", "warn", "off", "error"}
|
||||||
|
assert isinstance(message, str) and message.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def test_youtube_warns_when_node_only_and_no_config(monkeypatch, tmp_path):
|
||||||
|
"""YouTube should warn when only Node.js is installed but no yt-dlp config exists."""
|
||||||
|
from agent_reach.channels.youtube import YouTubeChannel
|
||||||
|
|
||||||
|
def fake_which(cmd):
|
||||||
|
if cmd == "yt-dlp":
|
||||||
|
return "/usr/bin/yt-dlp"
|
||||||
|
if cmd == "node":
|
||||||
|
return "/usr/bin/node"
|
||||||
|
return None # deno not installed
|
||||||
|
|
||||||
|
monkeypatch.setattr("shutil.which", fake_which)
|
||||||
|
# Point to a non-existent config file
|
||||||
|
monkeypatch.setattr("os.path.expanduser", lambda p: str(tmp_path / ".config/yt-dlp/config"))
|
||||||
|
|
||||||
|
ch = YouTubeChannel()
|
||||||
|
status, message = ch.check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "--js-runtimes" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_youtube_warns_with_windows_specific_fix_command(monkeypatch, tmp_path):
|
||||||
|
"""Windows guidance should use a PowerShell-style yt-dlp config command."""
|
||||||
|
from agent_reach.channels.youtube import YouTubeChannel
|
||||||
|
|
||||||
|
def fake_which(cmd):
|
||||||
|
if cmd == "yt-dlp":
|
||||||
|
return "C:/yt-dlp.exe"
|
||||||
|
if cmd == "node":
|
||||||
|
return "C:/node.exe"
|
||||||
|
return None
|
||||||
|
|
||||||
|
monkeypatch.setattr("shutil.which", fake_which)
|
||||||
|
monkeypatch.setattr("agent_reach.utils.paths.sys.platform", "win32")
|
||||||
|
monkeypatch.setenv("APPDATA", str(tmp_path / "AppData" / "Roaming"))
|
||||||
|
|
||||||
|
ch = YouTubeChannel()
|
||||||
|
status, message = ch.check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "Select-String" in message
|
||||||
|
assert "--js-runtimes node" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_youtube_ok_when_deno_installed(monkeypatch):
|
||||||
|
"""YouTube should return ok when Deno is installed (no config needed)."""
|
||||||
|
from agent_reach.channels.youtube import YouTubeChannel
|
||||||
|
|
||||||
|
def fake_which(cmd):
|
||||||
|
if cmd == "yt-dlp":
|
||||||
|
return "/usr/bin/yt-dlp"
|
||||||
|
if cmd == "deno":
|
||||||
|
return "/usr/bin/deno"
|
||||||
|
return None
|
||||||
|
|
||||||
|
monkeypatch.setattr("shutil.which", fake_which)
|
||||||
|
|
||||||
|
ch = YouTubeChannel()
|
||||||
|
status, _msg = ch.check()
|
||||||
|
assert status == "ok"
|
||||||
|
|
||||||
|
|
||||||
|
def test_douyin_check_does_not_call_with_invalid_url(monkeypatch, tmp_path):
|
||||||
|
"""Douyin check should use 'mcporter list' instead of calling with a hardcoded URL."""
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
from agent_reach.channels.douyin import DouyinChannel
|
||||||
|
|
||||||
|
calls = []
|
||||||
|
original_run = subprocess.run
|
||||||
|
|
||||||
|
def tracking_run(cmd, **kwargs):
|
||||||
|
calls.append(cmd)
|
||||||
|
# Simulate mcporter config list returning douyin
|
||||||
|
if "config" in cmd and "list" in cmd:
|
||||||
|
|
||||||
|
class R:
|
||||||
|
stdout = "douyin http://localhost:18070/mcp"
|
||||||
|
returncode = 0
|
||||||
|
|
||||||
|
return R()
|
||||||
|
# Simulate mcporter list douyin returning tools
|
||||||
|
if "list" in cmd and "douyin" in cmd:
|
||||||
|
|
||||||
|
class R:
|
||||||
|
stdout = "parse_douyin_video_info"
|
||||||
|
returncode = 0
|
||||||
|
|
||||||
|
return R()
|
||||||
|
return original_run(cmd, **kwargs)
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"shutil.which", lambda cmd: "/usr/bin/mcporter" if cmd == "mcporter" else None
|
||||||
|
)
|
||||||
|
monkeypatch.setattr("subprocess.run", tracking_run)
|
||||||
|
|
||||||
|
ch = DouyinChannel()
|
||||||
|
status, _msg = ch.check()
|
||||||
|
|
||||||
|
# Should NOT contain any hardcoded douyin.com URL in subprocess calls
|
||||||
|
for call in calls:
|
||||||
|
call_str = " ".join(call) if isinstance(call, list) else str(call)
|
||||||
|
assert "https://www.douyin.com" not in call_str
|
||||||
|
|
||||||
|
|
||||||
|
def test_channel_can_handle_contract():
|
||||||
|
url_samples = {
|
||||||
|
"github": "https://github.com/panniantong/agent-reach",
|
||||||
|
"twitter": "https://x.com/user/status/1",
|
||||||
|
"youtube": "https://youtube.com/watch?v=abc",
|
||||||
|
"reddit": "https://reddit.com/r/python",
|
||||||
|
"bilibili": "https://www.bilibili.com/video/BV1xx411",
|
||||||
|
"xiaohongshu": "https://www.xiaohongshu.com/explore/123",
|
||||||
|
"douyin": "https://www.douyin.com/video/123",
|
||||||
|
"linkedin": "https://www.linkedin.com/in/test",
|
||||||
|
"weibo": "https://weibo.com/u/1749127163",
|
||||||
|
"rss": "https://example.com/feed.xml",
|
||||||
|
"xueqiu": "https://xueqiu.com/S/SH600519",
|
||||||
|
"exa_search": "https://example.com",
|
||||||
|
"web": "https://example.com",
|
||||||
|
}
|
||||||
|
for ch in get_all_channels():
|
||||||
|
sample = url_samples.get(ch.name, "https://example.com")
|
||||||
|
result = ch.can_handle(sample)
|
||||||
|
assert isinstance(result, bool)
|
||||||
+650
-84
@@ -1,114 +1,680 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Tests for the channel system."""
|
"""Tests for channel registry basics and health checks."""
|
||||||
|
|
||||||
import pytest
|
import json
|
||||||
from unittest.mock import patch, MagicMock
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
from urllib.error import URLError
|
||||||
|
|
||||||
from agent_reach.channels import get_channel_for_url, get_channel, get_all_channels
|
from agent_reach.channels import get_all_channels, get_channel
|
||||||
from agent_reach.channels.base import ReadResult, SearchResult
|
from agent_reach.channels.xiaohongshu import XiaoHongShuChannel
|
||||||
|
from agent_reach.channels.xueqiu import XueqiuChannel
|
||||||
|
from agent_reach.channels.v2ex import V2EXChannel
|
||||||
|
|
||||||
|
|
||||||
class TestChannelRouting:
|
class TestChannelRegistry:
|
||||||
def test_github_url(self):
|
|
||||||
ch = get_channel_for_url("https://github.com/openai/gpt-4")
|
|
||||||
assert ch.name == "github"
|
|
||||||
|
|
||||||
def test_twitter_url(self):
|
|
||||||
ch = get_channel_for_url("https://x.com/elonmusk/status/123")
|
|
||||||
assert ch.name == "twitter"
|
|
||||||
|
|
||||||
def test_youtube_url(self):
|
|
||||||
ch = get_channel_for_url("https://youtube.com/watch?v=abc")
|
|
||||||
assert ch.name == "youtube"
|
|
||||||
|
|
||||||
def test_reddit_url(self):
|
|
||||||
ch = get_channel_for_url("https://reddit.com/r/test")
|
|
||||||
assert ch.name == "reddit"
|
|
||||||
|
|
||||||
def test_bilibili_url(self):
|
|
||||||
ch = get_channel_for_url("https://bilibili.com/video/BV1xx")
|
|
||||||
assert ch.name == "bilibili"
|
|
||||||
|
|
||||||
def test_rss_url(self):
|
|
||||||
ch = get_channel_for_url("https://example.com/feed.xml")
|
|
||||||
assert ch.name == "rss"
|
|
||||||
|
|
||||||
def test_generic_url_fallback(self):
|
|
||||||
ch = get_channel_for_url("https://example.com")
|
|
||||||
assert ch.name == "web"
|
|
||||||
|
|
||||||
def test_get_channel_by_name(self):
|
def test_get_channel_by_name(self):
|
||||||
ch = get_channel("github")
|
ch = get_channel("github")
|
||||||
assert ch is not None
|
assert ch is not None
|
||||||
assert ch.name == "github"
|
assert ch.name == "github"
|
||||||
|
|
||||||
|
def test_get_unknown_channel_returns_none(self):
|
||||||
|
assert get_channel("not-exists") is None
|
||||||
|
|
||||||
def test_all_channels_registered(self):
|
def test_all_channels_registered(self):
|
||||||
channels = get_all_channels()
|
channels = get_all_channels()
|
||||||
names = [ch.name for ch in channels]
|
names = [ch.name for ch in channels]
|
||||||
assert "web" in names
|
assert "web" in names
|
||||||
assert "github" in names
|
assert "github" in names
|
||||||
assert "twitter" in names
|
assert "twitter" in names
|
||||||
|
assert "v2ex" in names
|
||||||
|
|
||||||
|
|
||||||
class TestReadResult:
|
class TestV2EXChannel:
|
||||||
def test_to_dict(self):
|
def test_can_handle_v2ex_urls(self):
|
||||||
r = ReadResult(title="Test", content="Body", url="https://example.com", platform="web")
|
ch = V2EXChannel()
|
||||||
d = r.to_dict()
|
assert ch.can_handle("https://www.v2ex.com/t/1234567")
|
||||||
assert d["title"] == "Test"
|
assert ch.can_handle("https://v2ex.com/go/python")
|
||||||
assert d["content"] == "Body"
|
assert not ch.can_handle("https://github.com/user/repo")
|
||||||
assert d["platform"] == "web"
|
assert not ch.can_handle("https://reddit.com/r/Python")
|
||||||
|
|
||||||
def test_to_dict_optional_fields(self):
|
def test_check_ok_when_api_reachable(self, monkeypatch):
|
||||||
r = ReadResult(title="T", content="C", url="u", author="A", date="2025-01-01")
|
import urllib.request
|
||||||
d = r.to_dict()
|
|
||||||
assert d["author"] == "A"
|
|
||||||
assert d["date"] == "2025-01-01"
|
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
status = 200
|
||||||
|
|
||||||
class TestSearchResult:
|
def __enter__(self):
|
||||||
def test_to_dict(self):
|
return self
|
||||||
r = SearchResult(title="Test", url="https://example.com", snippet="A snippet")
|
|
||||||
d = r.to_dict()
|
|
||||||
assert d["title"] == "Test"
|
|
||||||
assert d["snippet"] == "A snippet"
|
|
||||||
|
|
||||||
|
def __exit__(self, *args):
|
||||||
|
pass
|
||||||
|
|
||||||
class TestGitHubChannel:
|
def read(self):
|
||||||
@patch("agent_reach.channels.github.requests.get")
|
return b"[]"
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_search(self, mock_get):
|
monkeypatch.setattr(
|
||||||
mock_resp = MagicMock()
|
urllib.request,
|
||||||
mock_resp.json.return_value = {
|
"urlopen",
|
||||||
"items": [{"full_name": "test/repo", "html_url": "https://github.com/test/repo",
|
lambda req, timeout=None: FakeResponse(),
|
||||||
"description": "A test", "stargazers_count": 100, "forks_count": 10,
|
)
|
||||||
"language": "Python", "updated_at": "2025-01-01"}]
|
status, msg = V2EXChannel().check()
|
||||||
|
assert status == "ok"
|
||||||
|
assert "公开 API 可用" in msg
|
||||||
|
|
||||||
|
def test_check_warn_when_api_unreachable(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
def raise_error(req, timeout=None):
|
||||||
|
raise URLError("connection refused")
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", raise_error)
|
||||||
|
status, msg = V2EXChannel().check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "失败" in msg
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_hot_topics
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_hot_topics_returns_list(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
fake_data = [
|
||||||
|
{
|
||||||
|
"id": 111,
|
||||||
|
"title": "Python 3.13 发布了",
|
||||||
|
"url": "https://www.v2ex.com/t/111",
|
||||||
|
"replies": 42,
|
||||||
|
"content": "发布公告内容",
|
||||||
|
"created": 1700000000,
|
||||||
|
"node": {"name": "python", "title": "Python"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 222,
|
||||||
|
"title": "Rust 好学吗",
|
||||||
|
"url": "https://www.v2ex.com/t/222",
|
||||||
|
"replies": 10,
|
||||||
|
"content": "",
|
||||||
|
"created": 1700000001,
|
||||||
|
"node": {"name": "rust", "title": "Rust"},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
status = 200
|
||||||
|
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", lambda req, timeout=None: FakeResponse())
|
||||||
|
topics = V2EXChannel().get_hot_topics(limit=5)
|
||||||
|
assert len(topics) == 2
|
||||||
|
assert topics[0]["id"] == 111
|
||||||
|
assert topics[0]["title"] == "Python 3.13 发布了"
|
||||||
|
assert topics[0]["replies"] == 42
|
||||||
|
assert topics[0]["node_name"] == "python"
|
||||||
|
assert topics[0]["node_title"] == "Python"
|
||||||
|
assert topics[0]["created"] == 1700000000
|
||||||
|
|
||||||
|
def test_get_hot_topics_respects_limit(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
fake_data = [
|
||||||
|
{"id": i, "title": f"Topic {i}", "url": f"https://v2ex.com/t/{i}", "replies": i,
|
||||||
|
"content": "", "created": 1700000000 + i, "node": {"name": "tech", "title": "Tech"}}
|
||||||
|
for i in range(10)
|
||||||
|
]
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", lambda req, timeout=None: FakeResponse())
|
||||||
|
topics = V2EXChannel().get_hot_topics(limit=3)
|
||||||
|
assert len(topics) == 3
|
||||||
|
|
||||||
|
def test_get_hot_topics_truncates_content(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
long_content = "A" * 300
|
||||||
|
fake_data = [
|
||||||
|
{"id": 1, "title": "Long post", "url": "https://v2ex.com/t/1", "replies": 0,
|
||||||
|
"content": long_content, "created": 1700000000, "node": {"name": "tech", "title": "Tech"}}
|
||||||
|
]
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", lambda req, timeout=None: FakeResponse())
|
||||||
|
topics = V2EXChannel().get_hot_topics(limit=1)
|
||||||
|
assert len(topics[0]["content"]) == 200
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_node_topics
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_node_topics(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
fake_data = [
|
||||||
|
{
|
||||||
|
"id": 333,
|
||||||
|
"title": "Flask 部署问题",
|
||||||
|
"url": "https://www.v2ex.com/t/333",
|
||||||
|
"replies": 5,
|
||||||
|
"content": "求帮助",
|
||||||
|
"created": 1710000000,
|
||||||
|
"node": {"name": "python", "title": "Python"},
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", lambda req, timeout=None: FakeResponse())
|
||||||
|
topics = V2EXChannel().get_node_topics("python")
|
||||||
|
assert len(topics) == 1
|
||||||
|
assert topics[0]["id"] == 333
|
||||||
|
assert topics[0]["node_name"] == "python"
|
||||||
|
assert topics[0]["title"] == "Flask 部署问题"
|
||||||
|
assert topics[0]["created"] == 1710000000
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_topic
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_topic_returns_detail_and_replies(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
topic_data = [
|
||||||
|
{
|
||||||
|
"id": 999,
|
||||||
|
"title": "测试帖子",
|
||||||
|
"url": "https://www.v2ex.com/t/999",
|
||||||
|
"content": "帖子正文",
|
||||||
|
"replies": 2,
|
||||||
|
"node": {"name": "qna", "title": "问与答"},
|
||||||
|
"member": {"username": "alice"},
|
||||||
|
"created": 1700000000,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
replies_data = [
|
||||||
|
{
|
||||||
|
"member": {"username": "bob"},
|
||||||
|
"content": "第一条回复",
|
||||||
|
"created": 1700000100,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"member": {"username": "carol"},
|
||||||
|
"content": "第二条回复",
|
||||||
|
"created": 1700000200,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
call_count = {"n": 0}
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __init__(self, payload):
|
||||||
|
self._payload = payload
|
||||||
|
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return json.dumps(self._payload).encode()
|
||||||
|
|
||||||
|
def fake_urlopen(req, timeout=None):
|
||||||
|
url = req.full_url
|
||||||
|
if "replies" in url:
|
||||||
|
return FakeResponse(replies_data)
|
||||||
|
return FakeResponse(topic_data)
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
|
||||||
|
result = V2EXChannel().get_topic(999)
|
||||||
|
|
||||||
|
assert result["id"] == 999
|
||||||
|
assert result["title"] == "测试帖子"
|
||||||
|
assert result["author"] == "alice"
|
||||||
|
assert result["node_name"] == "qna"
|
||||||
|
assert len(result["replies"]) == 2
|
||||||
|
assert result["replies"][0]["author"] == "bob"
|
||||||
|
assert result["replies"][1]["content"] == "第二条回复"
|
||||||
|
|
||||||
|
def test_get_topic_handles_empty_replies(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
topic_data = [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"title": "孤独帖子",
|
||||||
|
"url": "https://www.v2ex.com/t/1",
|
||||||
|
"content": "",
|
||||||
|
"replies": 0,
|
||||||
|
"node": {"name": "offtopic", "title": "水"},
|
||||||
|
"member": {"username": "dave"},
|
||||||
|
"created": 0,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __init__(self, payload): self._payload = payload
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return json.dumps(self._payload).encode()
|
||||||
|
|
||||||
|
def fake_urlopen(req, timeout=None):
|
||||||
|
if "replies" in req.full_url:
|
||||||
|
return FakeResponse([])
|
||||||
|
return FakeResponse(topic_data)
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", fake_urlopen)
|
||||||
|
result = V2EXChannel().get_topic(1)
|
||||||
|
assert result["replies"] == []
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_user
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_user_returns_profile(self, monkeypatch):
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
fake_user = {
|
||||||
|
"id": 42,
|
||||||
|
"username": "alice",
|
||||||
|
"url": "https://www.v2ex.com/member/alice",
|
||||||
|
"website": "https://alice.dev",
|
||||||
|
"twitter": "alice_tw",
|
||||||
|
"psn": "",
|
||||||
|
"github": "alice",
|
||||||
|
"btc": "",
|
||||||
|
"location": "Shanghai",
|
||||||
|
"bio": "Python dev",
|
||||||
|
"avatar_large": "https://cdn.v2ex.com/avatars/alice_large.png",
|
||||||
|
"created": 1500000000,
|
||||||
}
|
}
|
||||||
mock_resp.raise_for_status = MagicMock()
|
|
||||||
mock_get.return_value = mock_resp
|
|
||||||
|
|
||||||
ch = get_channel("github")
|
class FakeResponse:
|
||||||
results = await ch.search("test query")
|
def __enter__(self): return self
|
||||||
assert len(results) == 1
|
def __exit__(self, *_): pass
|
||||||
assert results[0].title == "test/repo"
|
def read(self): return json.dumps(fake_user).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(urllib.request, "urlopen", lambda req, timeout=None: FakeResponse())
|
||||||
|
user = V2EXChannel().get_user("alice")
|
||||||
|
|
||||||
|
assert user["id"] == 42
|
||||||
|
assert user["username"] == "alice"
|
||||||
|
assert user["github"] == "alice"
|
||||||
|
assert user["location"] == "Shanghai"
|
||||||
|
assert "alice_large.png" in user["avatar"]
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# search
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_search_returns_unavailable_notice(self):
|
||||||
|
result = V2EXChannel().search("python asyncio")
|
||||||
|
assert len(result) == 1
|
||||||
|
assert "error" in result[0]
|
||||||
|
assert "V2EX" in result[0]["error"]
|
||||||
|
|
||||||
|
|
||||||
class TestExaSearch:
|
class TestXueqiuChannel:
|
||||||
@patch("agent_reach.channels.exa_search.requests.post")
|
def test_can_handle_xueqiu_urls(self):
|
||||||
@pytest.mark.asyncio
|
ch = XueqiuChannel()
|
||||||
async def test_search(self, mock_post):
|
assert ch.can_handle("https://xueqiu.com/S/SH600519")
|
||||||
from agent_reach.config import Config
|
assert ch.can_handle("https://stock.xueqiu.com/v5/stock/batch/quote.json")
|
||||||
config = Config(config_path="/tmp/test-exa-config.yaml")
|
assert ch.can_handle("https://www.xueqiu.com/1234567890/12345")
|
||||||
config.set("exa_api_key", "test-key")
|
assert not ch.can_handle("https://github.com/user/repo")
|
||||||
|
assert not ch.can_handle("https://v2ex.com/t/123")
|
||||||
|
|
||||||
mock_resp = MagicMock()
|
def test_check_ok_when_api_reachable(self, monkeypatch):
|
||||||
mock_resp.json.return_value = {
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
"results": [{"title": "Result", "url": "https://example.com",
|
|
||||||
"text": "snippet", "publishedDate": "", "score": 0.9}]
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
fake_response_data = {
|
||||||
|
"data": {
|
||||||
|
"items": [
|
||||||
|
{"quote": {"symbol": "SH000001", "name": "上证指数", "current": 3200.0}}
|
||||||
|
]
|
||||||
|
}
|
||||||
}
|
}
|
||||||
mock_resp.raise_for_status = MagicMock()
|
|
||||||
mock_post.return_value = mock_resp
|
|
||||||
|
|
||||||
ch = get_channel("exa_search")
|
class FakeResponse:
|
||||||
results = await ch.search("test", config=config)
|
def __enter__(self):
|
||||||
assert len(results) == 1
|
return self
|
||||||
assert results[0].title == "Result"
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_response_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", lambda req, timeout=None: FakeResponse())
|
||||||
|
status, msg = XueqiuChannel().check()
|
||||||
|
assert status == "ok"
|
||||||
|
assert "公开 API 可用" in msg
|
||||||
|
|
||||||
|
def test_check_warn_when_api_unreachable(self, monkeypatch):
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
def raise_error(req, timeout=None):
|
||||||
|
raise URLError("connection refused")
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", raise_error)
|
||||||
|
status, msg = XueqiuChannel().check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "失败" in msg
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_stock_quote
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_stock_quote(self, monkeypatch):
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
fake_data = {
|
||||||
|
"data": {
|
||||||
|
"items": [
|
||||||
|
{
|
||||||
|
"quote": {
|
||||||
|
"symbol": "SH600519",
|
||||||
|
"name": "贵州茅台",
|
||||||
|
"current": 1800.0,
|
||||||
|
"percent": 1.5,
|
||||||
|
"chg": 26.6,
|
||||||
|
"high": 1810.0,
|
||||||
|
"low": 1770.0,
|
||||||
|
"open": 1775.0,
|
||||||
|
"last_close": 1773.4,
|
||||||
|
"volume": 12345678,
|
||||||
|
"amount": 22000000000,
|
||||||
|
"market_capital": 2260000000000,
|
||||||
|
"turnover_rate": 0.098,
|
||||||
|
"pe_ttm": 30.5,
|
||||||
|
"timestamp": 1700000000000,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", lambda req, timeout=None: FakeResponse())
|
||||||
|
quote = XueqiuChannel().get_stock_quote("SH600519")
|
||||||
|
assert quote["symbol"] == "SH600519"
|
||||||
|
assert quote["name"] == "贵州茅台"
|
||||||
|
assert quote["current"] == 1800.0
|
||||||
|
assert quote["percent"] == 1.5
|
||||||
|
assert quote["volume"] == 12345678
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# search_stock
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_search_stock(self, monkeypatch):
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
fake_data = {
|
||||||
|
"stocks": [
|
||||||
|
{"code": "SH600519", "name": "贵州茅台", "exchange": "SHA"},
|
||||||
|
{"code": "SZ000858", "name": "五粮液", "exchange": "SZA"},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", lambda req, timeout=None: FakeResponse())
|
||||||
|
results = XueqiuChannel().search_stock("茅台", limit=5)
|
||||||
|
assert len(results) == 2
|
||||||
|
assert results[0]["symbol"] == "SH600519"
|
||||||
|
assert results[0]["name"] == "贵州茅台"
|
||||||
|
assert results[1]["exchange"] == "SZA"
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_hot_posts
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_hot_posts_returns_list(self, monkeypatch):
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
# v4 timeline: each item has a JSON-encoded `data` field
|
||||||
|
def make_item(id_, title, text, author, likes, target):
|
||||||
|
post = {
|
||||||
|
"id": id_,
|
||||||
|
"title": title,
|
||||||
|
"text": text,
|
||||||
|
"user": {"screen_name": author},
|
||||||
|
"like_count": likes,
|
||||||
|
"target": target,
|
||||||
|
}
|
||||||
|
return {"data": json.dumps(post), "original_status": None}
|
||||||
|
|
||||||
|
fake_data = {
|
||||||
|
"list": [
|
||||||
|
make_item(111, "市场分析", "<p>今天大盘走势&分析</p>", "投资者A", 42, "/1234567890/111"),
|
||||||
|
make_item(222, "", "短评", "投资者B", 10, "/9876543210/222"),
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", lambda req, timeout=None: FakeResponse())
|
||||||
|
posts = XueqiuChannel().get_hot_posts(limit=10)
|
||||||
|
assert len(posts) == 2
|
||||||
|
assert posts[0]["id"] == 111
|
||||||
|
assert posts[0]["author"] == "投资者A"
|
||||||
|
assert posts[0]["likes"] == 42
|
||||||
|
assert "今天大盘走势&分析" in posts[0]["text"] # HTML stripped
|
||||||
|
assert "<p>" not in posts[0]["text"]
|
||||||
|
assert posts[0]["url"] == "https://xueqiu.com/1234567890/111"
|
||||||
|
|
||||||
|
def test_get_hot_posts_respects_limit(self, monkeypatch):
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
fake_data = {
|
||||||
|
"list": [
|
||||||
|
{
|
||||||
|
"data": json.dumps({
|
||||||
|
"id": i,
|
||||||
|
"title": f"Post {i}",
|
||||||
|
"text": f"Content {i}",
|
||||||
|
"user": {"screen_name": f"User {i}"},
|
||||||
|
"like_count": i,
|
||||||
|
"target": f"/user/{i}",
|
||||||
|
}),
|
||||||
|
"original_status": None,
|
||||||
|
}
|
||||||
|
for i in range(10)
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", lambda req, timeout=None: FakeResponse())
|
||||||
|
posts = XueqiuChannel().get_hot_posts(limit=3)
|
||||||
|
assert len(posts) == 3
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# get_hot_stocks
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_get_hot_stocks(self, monkeypatch):
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
|
||||||
|
fake_data = {
|
||||||
|
"data": {
|
||||||
|
"items": [
|
||||||
|
{"code": "SH600519", "name": "贵州茅台", "current": 1800.0, "percent": 1.5},
|
||||||
|
{"code": "SZ000858", "name": "五粮液", "current": 160.0, "percent": -0.8},
|
||||||
|
{"code": "SH601318", "name": "中国平安", "current": 45.0, "percent": 0.3},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
class FakeResponse:
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def read(self):
|
||||||
|
return json.dumps(fake_data).encode()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", lambda req, timeout=None: FakeResponse())
|
||||||
|
stocks = XueqiuChannel().get_hot_stocks(limit=10, stock_type=10)
|
||||||
|
assert len(stocks) == 3
|
||||||
|
assert stocks[0]["symbol"] == "SH600519"
|
||||||
|
assert stocks[0]["rank"] == 1
|
||||||
|
assert stocks[1]["percent"] == -0.8
|
||||||
|
assert stocks[2]["rank"] == 3
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# Cookie loading
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def test_ensure_cookies_loads_from_config(self, monkeypatch, tmp_path):
|
||||||
|
"""_ensure_cookies() should inject cookies from the config file."""
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", False)
|
||||||
|
|
||||||
|
# Provide a fake Config that returns a cookie string with xq_a_token
|
||||||
|
class FakeConfig:
|
||||||
|
def get(self, key, default=None):
|
||||||
|
if key == "xueqiu_cookie":
|
||||||
|
return "xq_a_token=TESTTOKEN; xq_is_login=1"
|
||||||
|
return default
|
||||||
|
|
||||||
|
import agent_reach.channels.xueqiu as xq_mod
|
||||||
|
monkeypatch.setattr(
|
||||||
|
xq_mod,
|
||||||
|
"_load_cookies_from_config",
|
||||||
|
lambda: (xq_mod._inject_cookie_string("xq_a_token=TESTTOKEN; xq_is_login=1") or True),
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(xq_mod, "_load_cookies_from_browser", lambda: False)
|
||||||
|
|
||||||
|
# Patch opener so no real HTTP call is made
|
||||||
|
class FakeResp:
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return b'{"data":{"items":[]}}'
|
||||||
|
|
||||||
|
monkeypatch.setattr(xq_mod._opener, "open", lambda req, timeout=None: FakeResp())
|
||||||
|
|
||||||
|
xq_mod._ensure_cookies()
|
||||||
|
assert xq_mod._cookies_initialized is True
|
||||||
|
cookie_names = {c.name for c in xq_mod._cookie_jar}
|
||||||
|
assert "xq_a_token" in cookie_names
|
||||||
|
|
||||||
|
def test_get_json_sends_referer_and_browser_ua(self, monkeypatch):
|
||||||
|
"""_get_json() must send Referer and a browser-like User-Agent."""
|
||||||
|
import agent_reach.channels.xueqiu as xueqiu_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod, "_cookies_initialized", True)
|
||||||
|
captured = {}
|
||||||
|
|
||||||
|
class FakeResp:
|
||||||
|
def __enter__(self): return self
|
||||||
|
def __exit__(self, *_): pass
|
||||||
|
def read(self): return b'{"data":{"items":[]}}'
|
||||||
|
|
||||||
|
def fake_open(req, timeout=None):
|
||||||
|
captured["ua"] = req.get_header("User-agent")
|
||||||
|
captured["referer"] = req.get_header("Referer")
|
||||||
|
return FakeResp()
|
||||||
|
|
||||||
|
monkeypatch.setattr(xueqiu_mod._opener, "open", fake_open)
|
||||||
|
xueqiu_mod._get_json("https://stock.xueqiu.com/v5/stock/batch/quote.json?symbol=SH000001")
|
||||||
|
|
||||||
|
assert captured["referer"] == "https://xueqiu.com/"
|
||||||
|
assert "Mozilla" in captured["ua"]
|
||||||
|
assert "agent-reach" not in captured["ua"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestXiaoHongShuChannel:
|
||||||
|
def test_reports_ok_when_cli_authenticated(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(shutil, "which", lambda _: "/usr/local/bin/xhs")
|
||||||
|
|
||||||
|
def fake_run(cmd, **kwargs):
|
||||||
|
return subprocess.CompletedProcess(cmd, 0, "ok: true\nusername: testuser\n", "")
|
||||||
|
|
||||||
|
monkeypatch.setattr(subprocess, "run", fake_run)
|
||||||
|
|
||||||
|
status, msg = XiaoHongShuChannel().check()
|
||||||
|
assert status == "ok"
|
||||||
|
assert "完整可用" in msg
|
||||||
|
|
||||||
|
def test_reports_warn_when_not_authenticated(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(shutil, "which", lambda _: "/usr/local/bin/xhs")
|
||||||
|
|
||||||
|
def fake_run(cmd, **kwargs):
|
||||||
|
return subprocess.CompletedProcess(cmd, 1, "", "ok: false\nerror:\n code: not_authenticated\n")
|
||||||
|
|
||||||
|
monkeypatch.setattr(subprocess, "run", fake_run)
|
||||||
|
|
||||||
|
status, msg = XiaoHongShuChannel().check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "xhs login" in msg
|
||||||
|
|
||||||
|
def test_reports_off_when_not_installed(self, monkeypatch):
|
||||||
|
monkeypatch.setattr(shutil, "which", lambda _: None)
|
||||||
|
status, msg = XiaoHongShuChannel().check()
|
||||||
|
assert status == "off"
|
||||||
|
assert "xiaohongshu-cli" in msg
|
||||||
|
|||||||
@@ -2,7 +2,9 @@
|
|||||||
"""Tests for Agent Reach CLI."""
|
"""Tests for Agent Reach CLI."""
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
import requests
|
||||||
from unittest.mock import patch
|
from unittest.mock import patch
|
||||||
|
import agent_reach.cli as cli
|
||||||
from agent_reach.cli import main
|
from agent_reach.cli import main
|
||||||
|
|
||||||
|
|
||||||
@@ -27,3 +29,98 @@ class TestCLI:
|
|||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "Agent Reach" in captured.out
|
assert "Agent Reach" in captured.out
|
||||||
assert "✅" in captured.out
|
assert "✅" in captured.out
|
||||||
|
|
||||||
|
def test_parse_twitter_cookie_input_separate_values(self):
|
||||||
|
auth_token, ct0 = cli._parse_twitter_cookie_input("token123 ct0abc")
|
||||||
|
assert auth_token == "token123"
|
||||||
|
assert ct0 == "ct0abc"
|
||||||
|
|
||||||
|
def test_parse_twitter_cookie_input_cookie_header(self):
|
||||||
|
auth_token, ct0 = cli._parse_twitter_cookie_input(
|
||||||
|
"auth_token=token123; ct0=ct0abc; other=value"
|
||||||
|
)
|
||||||
|
assert auth_token == "token123"
|
||||||
|
assert ct0 == "ct0abc"
|
||||||
|
|
||||||
|
|
||||||
|
class TestCheckUpdateRetry:
|
||||||
|
def test_retry_timeout_classification(self):
|
||||||
|
sleeps = []
|
||||||
|
|
||||||
|
def fake_sleep(seconds):
|
||||||
|
sleeps.append(seconds)
|
||||||
|
|
||||||
|
with patch("requests.get", side_effect=requests.exceptions.Timeout("timed out")):
|
||||||
|
resp, err, attempts = cli._github_get_with_retry(
|
||||||
|
"https://api.github.com/test",
|
||||||
|
timeout=1,
|
||||||
|
retries=3,
|
||||||
|
sleeper=fake_sleep,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert resp is None
|
||||||
|
assert err == "timeout"
|
||||||
|
assert attempts == 3
|
||||||
|
assert sleeps == [1, 2]
|
||||||
|
|
||||||
|
def test_retry_dns_classification(self):
|
||||||
|
error = requests.exceptions.ConnectionError("getaddrinfo failed for api.github.com")
|
||||||
|
with patch("requests.get", side_effect=error):
|
||||||
|
resp, err, attempts = cli._github_get_with_retry(
|
||||||
|
"https://api.github.com/test",
|
||||||
|
retries=1,
|
||||||
|
sleeper=lambda _x: None,
|
||||||
|
)
|
||||||
|
assert resp is None
|
||||||
|
assert err == "dns"
|
||||||
|
assert attempts == 1
|
||||||
|
|
||||||
|
def test_retry_rate_limit_then_success(self):
|
||||||
|
sleeps = []
|
||||||
|
|
||||||
|
class R:
|
||||||
|
def __init__(self, code, payload=None, headers=None):
|
||||||
|
self.status_code = code
|
||||||
|
self._payload = payload or {}
|
||||||
|
self.headers = headers or {}
|
||||||
|
|
||||||
|
def json(self):
|
||||||
|
return self._payload
|
||||||
|
|
||||||
|
sequence = [
|
||||||
|
R(429, headers={"Retry-After": "3"}),
|
||||||
|
R(200, payload={"tag_name": "v1.4.0"}),
|
||||||
|
]
|
||||||
|
|
||||||
|
with patch("requests.get", side_effect=sequence):
|
||||||
|
resp, err, attempts = cli._github_get_with_retry(
|
||||||
|
"https://api.github.com/test",
|
||||||
|
retries=3,
|
||||||
|
sleeper=lambda s: sleeps.append(s),
|
||||||
|
)
|
||||||
|
|
||||||
|
assert err is None
|
||||||
|
assert resp is not None
|
||||||
|
assert resp.status_code == 200
|
||||||
|
assert attempts == 2
|
||||||
|
assert sleeps == [3.0]
|
||||||
|
|
||||||
|
def test_classify_rate_limit_from_403(self):
|
||||||
|
class R:
|
||||||
|
status_code = 403
|
||||||
|
headers = {"X-RateLimit-Remaining": "0"}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def json():
|
||||||
|
return {"message": "API rate limit exceeded"}
|
||||||
|
|
||||||
|
assert cli._classify_github_response_error(R()) == "rate_limit"
|
||||||
|
|
||||||
|
def test_check_update_reports_classified_error(self, capsys):
|
||||||
|
with patch("agent_reach.cli._github_get_with_retry", return_value=(None, "timeout", 3)):
|
||||||
|
result = cli._cmd_check_update()
|
||||||
|
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert result == "error"
|
||||||
|
assert "网络超时" in captured.out
|
||||||
|
assert "已重试 3 次" in captured.out
|
||||||
|
|||||||
+13
-5
@@ -61,11 +61,6 @@ class TestConfig:
|
|||||||
tmp_config.set("exa_api_key", "test-key")
|
tmp_config.set("exa_api_key", "test-key")
|
||||||
assert tmp_config.is_configured("exa_search")
|
assert tmp_config.is_configured("exa_search")
|
||||||
|
|
||||||
def test_is_configured_reddit(self, tmp_config):
|
|
||||||
assert not tmp_config.is_configured("reddit_proxy")
|
|
||||||
tmp_config.set("reddit_proxy", "http://user:pass@ip:port")
|
|
||||||
assert tmp_config.is_configured("reddit_proxy")
|
|
||||||
|
|
||||||
def test_get_configured_features(self, tmp_config):
|
def test_get_configured_features(self, tmp_config):
|
||||||
features = tmp_config.get_configured_features()
|
features = tmp_config.get_configured_features()
|
||||||
assert isinstance(features, dict)
|
assert isinstance(features, dict)
|
||||||
@@ -78,3 +73,16 @@ class TestConfig:
|
|||||||
masked = tmp_config.to_dict()
|
masked = tmp_config.to_dict()
|
||||||
assert masked["exa_api_key"] == "super-se..."
|
assert masked["exa_api_key"] == "super-se..."
|
||||||
assert masked["normal_setting"] == "visible"
|
assert masked["normal_setting"] == "visible"
|
||||||
|
|
||||||
|
def test_save_creates_file_with_restricted_permissions(self, tmp_path):
|
||||||
|
import stat
|
||||||
|
import sys
|
||||||
|
config_file = tmp_path / "secure_config.yaml"
|
||||||
|
config = Config(config_path=config_file)
|
||||||
|
config.set("secret_key", "my-secret")
|
||||||
|
|
||||||
|
if sys.platform != "win32":
|
||||||
|
mode = config_file.stat().st_mode
|
||||||
|
# File should be owner-only read/write (0o600)
|
||||||
|
assert not (mode & stat.S_IRGRP), "group read should not be set"
|
||||||
|
assert not (mode & stat.S_IROTH), "other read should not be set"
|
||||||
|
|||||||
+1
-8
@@ -2,6 +2,7 @@
|
|||||||
"""Tests for AgentReach core class."""
|
"""Tests for AgentReach core class."""
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from agent_reach.config import Config
|
from agent_reach.config import Config
|
||||||
from agent_reach.core import AgentReach
|
from agent_reach.core import AgentReach
|
||||||
|
|
||||||
@@ -16,14 +17,6 @@ class TestAgentReach:
|
|||||||
def test_init(self, eyes):
|
def test_init(self, eyes):
|
||||||
assert eyes.config is not None
|
assert eyes.config is not None
|
||||||
|
|
||||||
def test_detect_platform(self, eyes):
|
|
||||||
assert eyes.detect_platform("https://github.com/test/repo") == "github"
|
|
||||||
assert eyes.detect_platform("https://reddit.com/r/test") == "reddit"
|
|
||||||
assert eyes.detect_platform("https://x.com/user/status/123") == "twitter"
|
|
||||||
assert eyes.detect_platform("https://youtube.com/watch?v=abc") == "youtube"
|
|
||||||
assert eyes.detect_platform("https://bilibili.com/video/BV1xx") == "bilibili"
|
|
||||||
assert eyes.detect_platform("https://example.com") == "web"
|
|
||||||
|
|
||||||
def test_doctor(self, eyes):
|
def test_doctor(self, eyes):
|
||||||
results = eyes.doctor()
|
results = eyes.doctor()
|
||||||
assert isinstance(results, dict)
|
assert isinstance(results, dict)
|
||||||
|
|||||||
+84
-20
@@ -2,8 +2,22 @@
|
|||||||
"""Tests for doctor module."""
|
"""Tests for doctor module."""
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
|
import agent_reach.doctor as doctor
|
||||||
from agent_reach.config import Config
|
from agent_reach.config import Config
|
||||||
from agent_reach.doctor import check_all, format_report
|
|
||||||
|
|
||||||
|
class _StubChannel:
|
||||||
|
def __init__(self, name, description, tier, status, message, backends=None):
|
||||||
|
self.name = name
|
||||||
|
self.description = description
|
||||||
|
self.tier = tier
|
||||||
|
self._status = status
|
||||||
|
self._message = message
|
||||||
|
self.backends = backends or []
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
return self._status, self._message
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
@@ -12,25 +26,75 @@ def tmp_config(tmp_path):
|
|||||||
|
|
||||||
|
|
||||||
class TestDoctor:
|
class TestDoctor:
|
||||||
def test_zero_config_channels_ok(self, tmp_config):
|
def test_check_all_collects_channel_results(self, tmp_config, monkeypatch):
|
||||||
results = check_all(tmp_config)
|
monkeypatch.setattr(
|
||||||
assert results["web"]["status"] == "ok"
|
doctor,
|
||||||
assert results["github"]["status"] == "ok"
|
"get_all_channels",
|
||||||
assert results["bilibili"]["status"] in ("ok", "warn") # warn on servers
|
lambda: [
|
||||||
assert results["rss"]["status"] == "ok"
|
_StubChannel("web", "网页", 0, "ok", "可抓取网页", ["requests"]),
|
||||||
|
_StubChannel("github", "GitHub", 0, "warn", "gh 未安装", ["gh"]),
|
||||||
|
_StubChannel("exa_search", "全网语义搜索", 1, "off", "mcporter 未配置", ["Exa"]),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
def test_exa_off_without_key(self, tmp_config):
|
results = doctor.check_all(tmp_config)
|
||||||
results = check_all(tmp_config)
|
|
||||||
assert results["exa_search"]["status"] == "off"
|
|
||||||
|
|
||||||
def test_exa_on_with_key(self, tmp_config):
|
assert results == {
|
||||||
tmp_config.set("exa_api_key", "test-key")
|
"web": {
|
||||||
results = check_all(tmp_config)
|
"status": "ok",
|
||||||
assert results["exa_search"]["status"] == "ok"
|
"name": "网页",
|
||||||
|
"message": "可抓取网页",
|
||||||
|
"tier": 0,
|
||||||
|
"backends": ["requests"],
|
||||||
|
},
|
||||||
|
"github": {
|
||||||
|
"status": "warn",
|
||||||
|
"name": "GitHub",
|
||||||
|
"message": "gh 未安装",
|
||||||
|
"tier": 0,
|
||||||
|
"backends": ["gh"],
|
||||||
|
},
|
||||||
|
"exa_search": {
|
||||||
|
"status": "off",
|
||||||
|
"name": "全网语义搜索",
|
||||||
|
"message": "mcporter 未配置",
|
||||||
|
"tier": 1,
|
||||||
|
"backends": ["Exa"],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
def test_format_report(self, tmp_config):
|
def test_format_report(self):
|
||||||
results = check_all(tmp_config)
|
report = doctor.format_report(
|
||||||
report = format_report(results)
|
{
|
||||||
assert "Agent Reach" in report
|
"web": {
|
||||||
assert "✅" in report
|
"status": "ok",
|
||||||
assert "渠道可用" in report
|
"name": "网页",
|
||||||
|
"message": "可抓取网页",
|
||||||
|
"tier": 0,
|
||||||
|
"backends": ["requests"],
|
||||||
|
},
|
||||||
|
"exa_search": {
|
||||||
|
"status": "off",
|
||||||
|
"name": "全网语义搜索",
|
||||||
|
"message": "mcporter 未配置",
|
||||||
|
"tier": 1,
|
||||||
|
"backends": ["Exa"],
|
||||||
|
},
|
||||||
|
"xiaohongshu": {
|
||||||
|
"status": "warn",
|
||||||
|
"name": "小红书",
|
||||||
|
"message": "MCP 已配置,但健康检查超时",
|
||||||
|
"tier": 2,
|
||||||
|
"backends": ["mcporter"],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Strip Rich markup tags for assertion (PR #170 added [bold], [yellow] etc.)
|
||||||
|
import re
|
||||||
|
plain = re.sub(r"\[[^\]]*\]", "", report)
|
||||||
|
assert "Agent Reach" in plain
|
||||||
|
assert "装好即用:" in plain
|
||||||
|
assert "1/3 个渠道可用" in plain
|
||||||
|
# Inactive optional channels should be summarized in one line
|
||||||
|
assert "可选渠道可以解锁" in plain
|
||||||
|
|||||||
@@ -0,0 +1,90 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Tests for 'agent-reach skill' command and _install_skill / _uninstall_skill."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
from agent_reach.cli import _install_skill, _uninstall_skill
|
||||||
|
|
||||||
|
|
||||||
|
class TestSkillCommand(unittest.TestCase):
|
||||||
|
"""Test skill install and uninstall via CLI helpers."""
|
||||||
|
|
||||||
|
def test_install_skill_creates_skill_md(self):
|
||||||
|
"""_install_skill should create SKILL.md in the first available skill dir."""
|
||||||
|
with tempfile.TemporaryDirectory() as tmpdir:
|
||||||
|
skill_dir = os.path.join(tmpdir, "skills")
|
||||||
|
os.makedirs(skill_dir)
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"agent_reach.cli.os.path.expanduser",
|
||||||
|
side_effect=lambda p: p.replace("~", tmpdir),
|
||||||
|
), patch.dict(os.environ, {}, clear=False):
|
||||||
|
# Remove OPENCLAW_HOME to avoid interference
|
||||||
|
env = os.environ.copy()
|
||||||
|
env.pop("OPENCLAW_HOME", None)
|
||||||
|
with patch.dict(os.environ, env, clear=True):
|
||||||
|
_install_skill()
|
||||||
|
|
||||||
|
target = os.path.join(skill_dir, "agent-reach", "SKILL.md")
|
||||||
|
# Check at least one known skill dir pattern
|
||||||
|
found = False
|
||||||
|
for dirpath, _, filenames in os.walk(tmpdir):
|
||||||
|
if "SKILL.md" in filenames:
|
||||||
|
found = True
|
||||||
|
# Verify content is non-empty
|
||||||
|
with open(os.path.join(dirpath, "SKILL.md")) as f:
|
||||||
|
content = f.read()
|
||||||
|
self.assertIn("Agent Reach", content)
|
||||||
|
# _install_skill may or may not find dirs depending on mock; just ensure no crash
|
||||||
|
# The important test is that the function runs without error
|
||||||
|
|
||||||
|
def test_uninstall_skill_removes_dir(self):
|
||||||
|
"""_uninstall_skill should remove skill directories."""
|
||||||
|
with tempfile.TemporaryDirectory() as tmpdir:
|
||||||
|
# Create a fake skill installation
|
||||||
|
skill_path = os.path.join(tmpdir, ".openclaw", "skills", "agent-reach")
|
||||||
|
os.makedirs(skill_path)
|
||||||
|
with open(os.path.join(skill_path, "SKILL.md"), "w") as f:
|
||||||
|
f.write("test")
|
||||||
|
|
||||||
|
self.assertTrue(os.path.exists(skill_path))
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"agent_reach.cli.os.path.expanduser",
|
||||||
|
side_effect=lambda p: p.replace("~", tmpdir),
|
||||||
|
), patch.dict(os.environ, {}, clear=False):
|
||||||
|
env = os.environ.copy()
|
||||||
|
env.pop("OPENCLAW_HOME", None)
|
||||||
|
with patch.dict(os.environ, env, clear=True):
|
||||||
|
_uninstall_skill()
|
||||||
|
|
||||||
|
self.assertFalse(os.path.exists(skill_path))
|
||||||
|
|
||||||
|
def test_install_creates_dir_if_parent_exists(self):
|
||||||
|
"""_install_skill should create agent-reach dir inside existing skill dir."""
|
||||||
|
with tempfile.TemporaryDirectory() as tmpdir:
|
||||||
|
# Create the .openclaw/skills parent but not agent-reach subdir
|
||||||
|
skill_parent = os.path.join(tmpdir, ".openclaw", "skills")
|
||||||
|
os.makedirs(skill_parent)
|
||||||
|
|
||||||
|
with patch(
|
||||||
|
"agent_reach.cli.os.path.expanduser",
|
||||||
|
side_effect=lambda p: p.replace("~", tmpdir),
|
||||||
|
), patch.dict(os.environ, {}, clear=False):
|
||||||
|
env = os.environ.copy()
|
||||||
|
env.pop("OPENCLAW_HOME", None)
|
||||||
|
with patch.dict(os.environ, env, clear=True):
|
||||||
|
_install_skill()
|
||||||
|
|
||||||
|
target = os.path.join(skill_parent, "agent-reach", "SKILL.md")
|
||||||
|
self.assertTrue(os.path.exists(target))
|
||||||
|
with open(target) as f:
|
||||||
|
content = f.read()
|
||||||
|
self.assertIn("Agent Reach", content)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
from unittest.mock import patch, Mock
|
||||||
|
|
||||||
|
from agent_reach.channels.twitter import TwitterChannel
|
||||||
|
|
||||||
|
|
||||||
|
def _cp(stdout="", stderr="", returncode=0):
|
||||||
|
m = Mock()
|
||||||
|
m.stdout = stdout
|
||||||
|
m.stderr = stderr
|
||||||
|
m.returncode = returncode
|
||||||
|
return m
|
||||||
|
|
||||||
|
|
||||||
|
def test_check_twitter_cli_found_and_auth_ok():
|
||||||
|
"""twitter-cli found + twitter status ok → ok."""
|
||||||
|
channel = TwitterChannel()
|
||||||
|
with patch("shutil.which", return_value="/usr/local/bin/twitter"), patch(
|
||||||
|
"subprocess.run",
|
||||||
|
return_value=_cp(stdout="ok: true\nusername: testuser\n", returncode=0),
|
||||||
|
):
|
||||||
|
status, message = channel.check()
|
||||||
|
assert status == "ok"
|
||||||
|
assert "完整可用" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_check_twitter_cli_found_auth_missing():
|
||||||
|
"""twitter-cli found + not_authenticated → warn about auth."""
|
||||||
|
channel = TwitterChannel()
|
||||||
|
with patch("shutil.which", return_value="/usr/local/bin/twitter"), patch(
|
||||||
|
"subprocess.run",
|
||||||
|
return_value=_cp(
|
||||||
|
stderr="ok: false\nerror:\n code: not_authenticated\n",
|
||||||
|
returncode=1,
|
||||||
|
),
|
||||||
|
):
|
||||||
|
status, message = channel.check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "未认证" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_check_twitter_cli_not_found():
|
||||||
|
"""twitter-cli not found → warn with install hint."""
|
||||||
|
channel = TwitterChannel()
|
||||||
|
with patch("shutil.which", return_value=None):
|
||||||
|
status, message = channel.check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "twitter-cli" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_check_twitter_cli_generic_failure():
|
||||||
|
"""twitter status returns 1 without not_authenticated → generic warn."""
|
||||||
|
channel = TwitterChannel()
|
||||||
|
with patch("shutil.which", return_value="/usr/local/bin/twitter"), patch(
|
||||||
|
"subprocess.run",
|
||||||
|
return_value=_cp(stderr="some error\n", returncode=1),
|
||||||
|
):
|
||||||
|
status, message = channel.check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "认证检查失败" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_check_twitter_cli_exception():
|
||||||
|
"""twitter status throws exception → warn."""
|
||||||
|
channel = TwitterChannel()
|
||||||
|
with patch("shutil.which", return_value="/usr/local/bin/twitter"), patch(
|
||||||
|
"subprocess.run", side_effect=Exception("timeout"),
|
||||||
|
):
|
||||||
|
status, message = channel.check()
|
||||||
|
assert status == "warn"
|
||||||
|
assert "连接失败" in message
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Tests for XiaoHongShu output formatter (issue #134)."""
|
||||||
|
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
from agent_reach.channels.xiaohongshu import format_xhs_result
|
||||||
|
|
||||||
|
|
||||||
|
class TestFormatXhsResult(unittest.TestCase):
|
||||||
|
"""Test format_xhs_result strips redundant fields."""
|
||||||
|
|
||||||
|
SAMPLE_NOTE = {
|
||||||
|
"id": "abc123",
|
||||||
|
"title": "测试笔记",
|
||||||
|
"desc": "这是正文内容",
|
||||||
|
"type": "normal",
|
||||||
|
"xsec_token": "tok_xxx",
|
||||||
|
"user": {
|
||||||
|
"nickname": "小红",
|
||||||
|
"user_id": "u123",
|
||||||
|
"avatar": "https://example.com/avatar.jpg",
|
||||||
|
"extra_field": "should be dropped",
|
||||||
|
},
|
||||||
|
"interact_info": {
|
||||||
|
"liked_count": "100",
|
||||||
|
"collected_count": "50",
|
||||||
|
"comment_count": "20",
|
||||||
|
"share_count": "10",
|
||||||
|
"sticky_count": "0",
|
||||||
|
"relation": "none",
|
||||||
|
},
|
||||||
|
"image_list": [
|
||||||
|
{
|
||||||
|
"url": "https://img.example.com/1.jpg",
|
||||||
|
"info_list": [{"url": "https://img.example.com/1_small.jpg", "image_scene": "WB_DFT"}],
|
||||||
|
"width": 1080,
|
||||||
|
"height": 1440,
|
||||||
|
"trace_id": "tr_123",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"url": "https://img.example.com/2.jpg",
|
||||||
|
"info_list": [{"url": "https://img.example.com/2_small.jpg"}],
|
||||||
|
"width": 1080,
|
||||||
|
"height": 1080,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"tag_list": [
|
||||||
|
{"id": "t1", "name": "旅行", "type": "topic"},
|
||||||
|
{"id": "t2", "name": "美食", "type": "topic"},
|
||||||
|
],
|
||||||
|
"at_user_list": [],
|
||||||
|
"geo_info": {"latitude": 0, "longitude": 0},
|
||||||
|
"audit_info": {"audit_status": 0},
|
||||||
|
"model_type": None,
|
||||||
|
"note_flow_source": "search",
|
||||||
|
}
|
||||||
|
|
||||||
|
def test_single_note_keeps_useful_fields(self):
|
||||||
|
result = format_xhs_result(self.SAMPLE_NOTE)
|
||||||
|
self.assertEqual(result["id"], "abc123")
|
||||||
|
self.assertEqual(result["title"], "测试笔记")
|
||||||
|
self.assertEqual(result["desc"], "这是正文内容")
|
||||||
|
self.assertEqual(result["type"], "normal")
|
||||||
|
self.assertEqual(result["user"]["nickname"], "小红")
|
||||||
|
self.assertEqual(result["liked_count"], "100")
|
||||||
|
self.assertEqual(result["collected_count"], "50")
|
||||||
|
self.assertEqual(result["images"], [
|
||||||
|
"https://img.example.com/1.jpg",
|
||||||
|
"https://img.example.com/2.jpg",
|
||||||
|
])
|
||||||
|
self.assertEqual(result["tags"], ["旅行", "美食"])
|
||||||
|
|
||||||
|
def test_single_note_drops_useless_fields(self):
|
||||||
|
result = format_xhs_result(self.SAMPLE_NOTE)
|
||||||
|
self.assertNotIn("at_user_list", result)
|
||||||
|
self.assertNotIn("geo_info", result)
|
||||||
|
self.assertNotIn("audit_info", result)
|
||||||
|
self.assertNotIn("model_type", result)
|
||||||
|
self.assertNotIn("note_flow_source", result)
|
||||||
|
# User should not have extra fields
|
||||||
|
self.assertNotIn("avatar", result.get("user", {}))
|
||||||
|
self.assertNotIn("extra_field", result.get("user", {}))
|
||||||
|
|
||||||
|
def test_search_results_wrapper(self):
|
||||||
|
"""Handle {"items": [...]} wrapper from search_feeds."""
|
||||||
|
wrapped = {"items": [self.SAMPLE_NOTE, self.SAMPLE_NOTE]}
|
||||||
|
result = format_xhs_result(wrapped)
|
||||||
|
self.assertIsInstance(result, list)
|
||||||
|
self.assertEqual(len(result), 2)
|
||||||
|
self.assertEqual(result[0]["title"], "测试笔记")
|
||||||
|
|
||||||
|
def test_list_input(self):
|
||||||
|
result = format_xhs_result([self.SAMPLE_NOTE])
|
||||||
|
self.assertIsInstance(result, list)
|
||||||
|
self.assertEqual(len(result), 1)
|
||||||
|
self.assertEqual(result[0]["title"], "测试笔记")
|
||||||
|
|
||||||
|
def test_note_card_wrapper(self):
|
||||||
|
"""Handle notes nested under 'note_card'."""
|
||||||
|
wrapped = {"note_card": self.SAMPLE_NOTE}
|
||||||
|
result = format_xhs_result(wrapped)
|
||||||
|
self.assertEqual(result["title"], "测试笔记")
|
||||||
|
|
||||||
|
def test_with_comments(self):
|
||||||
|
note = dict(self.SAMPLE_NOTE)
|
||||||
|
note["comments"] = [
|
||||||
|
{
|
||||||
|
"content": "写得好!",
|
||||||
|
"user_info": {"nickname": "路人甲", "user_id": "u456"},
|
||||||
|
"like_count": 5,
|
||||||
|
"sub_comment_count": 1,
|
||||||
|
"ip_location": "上海",
|
||||||
|
"status": 0,
|
||||||
|
}
|
||||||
|
]
|
||||||
|
result = format_xhs_result(note)
|
||||||
|
self.assertEqual(len(result["comments"]), 1)
|
||||||
|
self.assertEqual(result["comments"][0]["content"], "写得好!")
|
||||||
|
self.assertEqual(result["comments"][0]["user"], "路人甲")
|
||||||
|
self.assertEqual(result["comments"][0]["like_count"], 5)
|
||||||
|
self.assertNotIn("ip_location", result["comments"][0])
|
||||||
|
|
||||||
|
def test_empty_input(self):
|
||||||
|
self.assertEqual(format_xhs_result({}), {})
|
||||||
|
self.assertEqual(format_xhs_result([]), [])
|
||||||
|
|
||||||
|
def test_non_dict_passthrough(self):
|
||||||
|
self.assertEqual(format_xhs_result("hello"), "hello")
|
||||||
|
self.assertIsNone(format_xhs_result(None))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import agent_reach.cli as cli
|
||||||
|
|
||||||
|
|
||||||
|
class _DummyConfig:
|
||||||
|
def get(self, _key):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def test_install_xiaoyuzhou_deps_does_not_raise_when_no_groq_key(capsys):
|
||||||
|
with patch("agent_reach.config.Config", return_value=_DummyConfig()), \
|
||||||
|
patch("os.path.isfile", side_effect=lambda p: True if str(p).endswith("transcribe.sh") else False), \
|
||||||
|
patch("shutil.which", return_value=None):
|
||||||
|
cli._install_xiaoyuzhou_deps()
|
||||||
|
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "Xiaoyuzhou" in out
|
||||||
|
assert "Groq API key not set" in out
|
||||||
Reference in New Issue
Block a user