54 Commits

Author SHA1 Message Date
Pnant ca29c4fee5 release: v1.3.0 (#70)
ci / test (3.10) (push) Has been cancelled
ci / test (3.11) (push) Has been cancelled
ci / test (3.12) (push) Has been cancelled
New:
- WeChat Official Account channel (search + read)
  - Search via miku_ai (Sogou WeChat search)
  - Read via Camoufox (stealth Firefox, bypasses anti-bot)
- 13 channels total

Fixed:
- Windows encoding: text=True → encoding='utf-8' in all subprocess calls (#64)
- Windows .cmd resolution: use shutil.which() paths (#64)

Docs:
- Clarified agent-reach has no read/search commands (#58, #59)
- Fixed xiaohongshu API function names in SKILL.md (#65)
- Added CONTRIBUTING.md (#62)
- Updated README (zh + en) with WeChat channel

Version bumps: pyproject.toml, __init__.py, test_cli.py

Co-authored-by: Panniantong <panniantong@users.noreply.github.com>
2026-03-04 18:18:18 +08:00
Pnant 49d05b7023 docs: add WeChat channel to README (zh + en) (#69)
Co-authored-by: Panniantong <panniantong@users.noreply.github.com>
2026-03-04 18:06:47 +08:00
Pnant 476935be92 feat: add WeChat search via miku_ai (sogou) (#68)
- wechat channel now supports both search and read
- Search: miku_ai (sogou weixin search) → article list with URLs
- Read: camoufox (stealth Firefox) → full Markdown content
- Doctor shows granular status (search only / read only / both)
- SKILL.md updated with search + read workflow examples

Co-authored-by: Panniantong <panniantong@users.noreply.github.com>
2026-03-04 18:02:05 +08:00
Pnant 4f4ad99d49 feat: add WeChat Official Account channel (#54) (#67)
New channel: wechat (微信公众号文章)
- Backend: wechat-article-for-ai (Camoufox stealth browser)
- Can read mp.weixin.qq.com article URLs → clean Markdown
- Bypasses WeChat anti-bot detection via stealth Firefox
- Doctor checks for camoufox package availability
- SKILL.md updated with usage examples

Co-authored-by: Panniantong <panniantong@users.noreply.github.com>
2026-03-04 17:53:39 +08:00
Sense_wang 561b248a2b docs: add CONTRIBUTING.md with development guidelines (#62)
Add contribution guidelines including:
- Development setup instructions
- Code style guidelines (ruff, mypy, pytest)
- Channel development guide
- PR guidelines
- Issue reporting template
- Bilingual (EN/CN) documentation

Co-authored-by: haosenwang1018 <haosenwang1018@users.noreply.github.com>
2026-03-04 17:30:15 +08:00
Pnant a91a74d880 fix: Windows encoding + .cmd subprocess resolution (#64) (#66)
- Replace text=True with encoding='utf-8', errors='replace' in all
  subprocess.run calls (channels + cli.py) to fix GBK decode errors
  on Chinese Windows systems
- Use shutil.which() resolved paths in channel subprocess calls to
  handle Windows .cmd wrappers (npm global installs)

Co-authored-by: Panniantong <panniantong@users.noreply.github.com>
2026-03-04 17:30:11 +08:00
Yves 7434c3cb9f docs: fix xiaohongshu API in SKILL.md — replace non-existent functions with correct ones (#65) 2026-03-04 17:18:13 +08:00
Pnant 1a61a2cc81 docs: clarify agent-reach has no read/search commands (#59)
- Remove dead '# ── read ──' placeholder comment in cli.py
- Reword SKILL.md to make it explicit that agent-reach is an
  installer/config tool, not a content fetcher
- Addresses confusion from issue #58

Co-authored-by: Panniantong <panniantong@users.noreply.github.com>
2026-03-04 14:33:38 +08:00
Pnant 932837ff9a Merge pull request #52 from Panniantong/fix/twitter-cookie-sync-issue50
fix: sync Twitter cookies to xreach session.json on configure
2026-02-28 20:29:38 +08:00
Panniantong 4f52942260 improve: add chmod 0o600 for session.json (from PR #51 suggestion) 2026-02-28 13:28:14 +01:00
Panniantong 3c1a04a587 fix: sync Twitter cookies to xreach session.json on configure
When running 'agent-reach configure twitter-cookies', credentials are now
automatically written to ~/.config/xfetch/session.json so that 'xreach auth
check' works without manual intervention.

Preserves existing session.json fields if the file already exists.
Falls back gracefully if the sync fails (still saves to agent-reach config).

Fixes #50
2026-02-28 11:11:55 +01:00
Pnant 6a28b215af Merge pull request #49 from Panniantong/fix/ci-test-github-check
fix: CI 测试修复 — gh CLI 在 CI 环境不存在
2026-02-27 20:41:30 +08:00
Panniantong a881db4261 fix: CI test — allow github channel 'warn' when gh CLI not installed
GitHub Actions runners don't have gh CLI pre-installed, so the doctor
check returns 'warn' instead of 'ok'. Allow both in the test assertion.
2026-02-27 13:39:34 +01:00
Pnant 875f0987a6 Merge pull request #48 from Panniantong/fix/yt-bilibili-cookie-docs
fix: YouTube 需要 JS runtime — install 自动配置 Node.js
2026-02-27 18:46:32 +08:00
Panniantong 0d1a33236e fix: YouTube requires JS runtime — auto-configure Node.js in install
yt-dlp now requires an external JS runtime (deno or Node.js) for YouTube.
Since agent-reach already installs Node.js, we configure yt-dlp to use it:

1. cli.py: install writes '--js-runtimes node' to ~/.config/yt-dlp/config
2. youtube.py: doctor checks JS runtime availability and config
3. SKILL.md: updated YouTube section with accurate troubleshooting

Bilibili note: 412 errors are IP-based (needs proxy), not cookie-related.

Fixes #47
2026-02-27 11:45:39 +01:00
Panniantong e620a1db8f docs: add cookie/proxy warnings for YouTube and Bilibili in SKILL.md
YouTube may trigger 'Sign in to confirm you're not a bot' on server IPs.
Bilibili may return 412 on non-residential IPs.

Added --cookies-from-browser fallback examples for both channels.

Fixes #47
2026-02-27 11:25:29 +01:00
Pnant 15a2961dfa Merge pull request #46 from Panniantong/migrate-bird-to-xreach
feat: 迁移 Twitter 后端 bird → xreach CLI
2026-02-27 15:21:25 +08:00
Pnant 4bc860c74d Merge pull request #45 from Panniantong/fix/xiaohongshu-arm64-support
fix: ARM64/Apple Silicon support for xiaohongshu-mcp Docker setup
2026-02-27 15:21:21 +08:00
Panniantong 62aacf38b5 feat: migrate Twitter backend from bird CLI to xreach CLI
bird CLI (@steipete/bird) is deprecated and no longer maintained.
xreach CLI (xreach-cli on npm) is our maintained fork with:
- Fixed SearchTimeline (POST + updated query ID)
- Built-in proxy rotation support
- Additional features (DMs, notifications, lists)

Changes across 11 files:
- channels/twitter.py: detect xreach instead of bird/birdx
- cli.py: install/doctor/uninstall all reference xreach-cli
- SKILL.md: updated command examples (bird read → xreach tweet)
- guides/setup-twitter.md: rewritten for xreach
- docs/troubleshooting.md: updated proxy guidance
- README.md + README_en.md: all references updated
- config.py: twitter_bird → twitter_xreach
- core.py, mcp_server.py: comment updates

npm package: https://www.npmjs.com/package/xreach-cli
Source: https://github.com/Panniantong/xfetch
2026-02-27 08:17:51 +01:00
Panniantong d4e437bf2c fix: add ARM64/Apple Silicon support for xiaohongshu-mcp Docker setup
- Detect ARM64 architecture and auto-add --platform linux/amd64 to
  docker run hints in doctor/check output
- Add ARM64 FAQ section to setup guide with two options:
  Rosetta emulation (--platform linux/amd64) or build from source
- Closes #44
2026-02-27 08:06:06 +01:00
Pnant dbcc668f47 Merge pull request #40 from peter941221/fix/security-hardening-split-v2
fix: split security hardening and setup mcporter config checks
2026-02-27 13:30:52 +08:00
Pnant 40f833d0d1 Merge pull request #41 from peter941221/chore/quality-infra-split-v3
chore: split quality infrastructure into focused PR
2026-02-27 13:26:49 +08:00
Pnant dc4d7cd7a5 Merge pull request #24 from Daiyimo/main
feat: add uninstall command; fix utf-8 encoding and minor cleanups
2026-02-27 13:24:21 +08:00
Pnant 60f5ddf4f0 Merge pull request #35 from Panniantong/fix/mcporter-list-hang
fix: replace 'mcporter list' with 'mcporter config list' to prevent hangs
2026-02-27 12:56:40 +08:00
Your Name a726aa7fe1 chore: add quality infra matrix, constraints, and test baseline fixes 2026-02-27 12:27:07 +08:00
Your Name 4b2e6f2ffb fix: split security hardening and setup mcporter config checks 2026-02-27 12:16:00 +08:00
Panniantong 39f76ee4dc fix: replace 'mcporter list' with 'mcporter config list' to prevent hangs
mcporter list probes all configured servers sequentially, which can hang
indefinitely when servers are slow/offline. mcporter config list reads
config without probing — sufficient for checking if a server is configured.

Affected: doctor checks (4 channels) and install command (2 calls).

Fixes #34
2026-02-27 03:06:27 +01:00
Pnant a5682716ec Merge pull request #31 from Panniantong/docs/cookie-ban-warning
docs: 添加 Cookie 登录封号风险提醒
2026-02-26 22:41:53 +08:00
Panniantong f269667a34 docs: 添加 Cookie 登录封号风险提醒,强调使用小号
- README: Cookie 安全建议新增封号风险说明
- install.md: Security tip 补充 account ban 风险
- SKILL.md: Cookie 导入段落加封号提醒
2026-02-26 15:13:25 +01:00
Pnant 090b01c909 Merge pull request #30 from Panniantong/fix/sync-version-string
fix: sync __version__ with pyproject.toml (1.0.0 → 1.2.0)
2026-02-26 22:08:35 +08:00
Panniantong 58b3a4dcb3 fix: sync __version__ with pyproject.toml (1.0.0 → 1.2.0)
Fixes #29 — __version__ in __init__.py was still '1.0.0' while pyproject.toml
had already been bumped to '1.2.0'. This caused misleading output in
'agent-reach --version' and 'agent-reach doctor'.
2026-02-26 15:07:24 +01:00
Panniantong 37b4cdeddc bump: v1.2.0 2026-02-26 13:34:41 +01:00
Pnant bf6dc7084b Merge pull request #28 from Panniantong/feat/douyin-channel
feat: 新增抖音(Douyin)渠道
2026-02-26 20:34:17 +08:00
Panniantong 50f2181fcc feat: 新增抖音(Douyin)渠道 - 基于 douyin-mcp-server
- 新增 agent_reach/channels/douyin.py
- 更新 doctor 检测逻辑
- 更新 install.md/SKILL.md/README 文档
- 通过 mcporter 调用 douyin-mcp-server
- 支持视频解析、下载链接获取
- 无需登录即可使用
2026-02-26 13:25:57 +01:00
Pnant b387a67b51 Merge pull request #27 from Panniantong/docs/cookie-editor-first
docs: Cookie-Editor 作为所有 cookie 平台的首选登录方式
2026-02-26 19:17:35 +08:00
Panniantong a96c5cda14 docs: Cookie-Editor 作为首选登录方式,统一所有 cookie 平台
- install.md: 新增通用 Cookie 导入说明,所有平台统一流程
- install.md: 小红书登录方式改为 Cookie-Editor 优先,扫码降级为备选
- SKILL.md: Cookie 导入独立成段,明确优先级
- README.md: 更新 Cookie 说明,强调 Cookie-Editor 比扫码更简单可靠
2026-02-26 12:16:34 +01:00
Daiyimo 9c6c04c800 feat: add uninstall command; fix utf-8 encoding and minor cleanups 2026-02-26 16:01:22 +08:00
Pnant cf70302e92 Merge pull request #23 from Panniantong/refactor/simplify-to-installer
refactor: 回归安装器+文档定位,砍掉 read/search 包装层
2026-02-26 15:29:44 +08:00
Panniantong 80a08c8c10 docs: update all remaining references to use upstream tools directly
- docs/README_en.md: update FAQ, Works Out of the Box, Design Philosophy
- docs/troubleshooting.md: Exa fallback uses mcporter directly
- agent_reach/guides/setup-exa.md: test with mcporter call
- agent_reach/guides/setup-wechat.md: test with Jina Reader
2026-02-26 08:25:06 +01:00
Panniantong 606e8900ce docs: update install.md — remove read/search references, add upstream tools table 2026-02-26 08:20:20 +01:00
Panniantong a37e9aa190 refactor: strip to installer + doctor + docs, remove read/search wrapper layer
BREAKING CHANGE: Remove all `agent-reach read` and `agent-reach search-*` commands.

Agent Reach is now an installer, configuration tool, and doctor —
not a wrapper layer. After installation, agents call upstream tools
directly (bird CLI, yt-dlp, mcporter, gh CLI, Jina Reader, etc.).

What's kept:
- agent-reach install: one-shot installer
- agent-reach doctor: channel status overview
- agent-reach configure: cookies, proxy, credentials
- agent-reach setup: interactive wizard
- SKILL.md: complete guide for agents to use upstream tools directly

What's removed:
- agent-reach read URL (and all channel read() methods)
- agent-reach search-* commands (and all channel search() methods)
- ReadResult / SearchResult data classes
- URL routing system (get_channel_for_url)
- All parsing/conversion logic (VTT, Reddit JSON, bird text parser, etc.)
- MCP server read/search tools (kept only get_status)

Net change: -1790 lines. Less code = fewer bugs.
2026-02-26 08:15:56 +01:00
Pnant 1cbf6a7b9c fix(xiaohongshu): 修复 xsec_token 丢失导致无法读取笔记详情 (#17)
问题:搜索结果包含 xsecToken 但被丢弃,read() 只从首页 feed
查找 token,几乎不可能匹配到目标笔记。

修复:
1. search() 返回的 URL 携带 xsec_token 查询参数
2. read() 三级 token 查找:URL 参数 → 首页 feed → 搜索反查
3. 全部失败时回退到 Jina Reader

Fixes #15

Co-authored-by: Panniantong <panniantong@users.noreply.github.com>
2026-02-26 14:36:36 +08:00
Pnant 68319ec683 Merge pull request #16 from Panniantong/remove/instagram
remove(instagram): 移除 Instagram 渠道(上游反爬封杀)
2026-02-26 14:23:26 +08:00
Panniantong f70711e75e remove(instagram): 移除 Instagram 渠道
Instagram 反爬封杀导致所有开源工具(instaloader 等)失效,
无论有无 cookies 都无法正常使用。

- 删除 instagram.py 渠道文件
- 移除 CLI 中 search-instagram、configure instagram-cookies 等命令
- 移除 setup/doctor 中 instaloader 依赖检查
- 更新 README、docs、SKILL.md、pyproject.toml

上游 issue: instaloader#2585, instaloader#2648
Relates to: #13
2026-02-26 07:20:13 +01:00
Panniantong c3a9813b1c seo: GitHub SEO + GEO 优化,提升项目可发现性
1. 新增 llms.txt — 让 AI 搜索引擎(ChatGPT/Claude/Perplexity)理解项目
2. README 中英文加 FAQ 区块 — AI 搜索友好的常见问题解答
3. pyproject.toml keywords 扩展到 21 个 — 覆盖更多搜索词
2026-02-25 22:32:22 +01:00
Panniantong d291d27664 docs: install.md 和 SKILL.md 补充 Twitter 代理方案说明
让 Agent 在配置 Twitter 时知道:
- undici 需要安装(npm install -g undici)
- 代理注入是自动的,不需要用户额外操作
- fetch failed 的排查步骤
2026-02-25 21:45:40 +01:00
Pnant 0be4f097d0 Merge pull request #10 from Panniantong/fix/twitter-fetch-failed
fix: Twitter bird CLI fetch failed — 自动 fallback + 连通性检测
2026-02-26 04:44:39 +08:00
Panniantong 12bd727e30 fix: Twitter bird CLI fetch failed — 自动代理注入 + 连通性检测 + Exa fallback
问题:bird CLI 用 Node.js 原生 fetch(),不走 HTTP_PROXY,
导致需要代理的用户(如 Windows + 本地代理)完全无法使用。

修复:
1. 自动代理注入:检测到 HTTP_PROXY/HTTPS_PROXY 时,通过 undici 的
   EnvHttpProxyAgent 让 Node.js fetch 自动走代理,用户无需任何额外配置
2. doctor 检测:实际跑 bird whoami 测连通性,失败时给出具体原因
3. search fallback:bird 搜索失败时自动 fallback 到 Exa
4. install:自动安装 undici(npm install -g undici)
5. 新增 troubleshooting.md:常见问题解决方案

Fixes #9
2026-02-25 21:35:21 +01:00
Panniantong b6d135c977 docs: 小红书配置方式统一为「告诉 Agent 帮我配」 2026-02-25 16:03:50 +01:00
Pnant 291d073223 Merge pull request #8 from Panniantong/refactor/skill-consolidate
refactor: 合并 SKILL.md 为单一来源,更新为 12 渠道
2026-02-25 22:32:31 +08:00
Pnant ef0889dd69 Merge pull request #7 from Panniantong/docs/channel-limitations
docs: 补充新渠道使用限制说明
2026-02-25 22:32:27 +08:00
Panniantong 545a6c5761 docs: 新渠道配置方式统一为「告诉 Agent 帮我配」
跟其他渠道风格一致,不单独列限制说明。
具体配置步骤都在 install.md 里,Agent 会自动引导。
2026-02-25 15:27:51 +01:00
Panniantong 00f3edd9cc refactor: 合并 SKILL.md 为单一来源,更新为 12 个渠道
- 删除 agent-reach/SKILL.md(重复)
- 删除 agent_reach/integrations/skill/SKILL.md(重复)
- 保留并更新 agent_reach/skill/SKILL.md 为唯一来源
- 新增 Instagram/LinkedIn/Boss直聘 搜索命令
- 新增渠道配置引导流程说明
- 描述从 9+ 更新为 12+
2026-02-25 15:11:51 +01:00
Panniantong 2f674f1e45 security: doctor warns if config.yaml permissions are too open
Adds a check in 'agent-reach doctor' that warns when ~/.agent-reach/config.yaml
is readable by other users (group/world). Suggests chmod 600 to fix.

Ref: #6
2026-02-25 14:16:08 +01:00
45 changed files with 1893 additions and 2863 deletions
+30
View File
@@ -0,0 +1,30 @@
name: ci
on:
push:
pull_request:
jobs:
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12"]
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install package and test deps
run: |
python -m pip install --upgrade pip
pip install -c constraints.txt -e .[dev]
- name: Run tests
run: |
pytest -q
+9 -12
View File
@@ -10,13 +10,10 @@ All notable changes to this project will be documented in this file.
### 🆕 New Channels / 新增渠道
#### 📷 Instagram
- Read public posts and profiles via [instaloader](https://github.com/instaloader/instaloader)
- Search via Exa (free, no API key)
- Optional cookie login for private content
- 通过 instaloader 读取公开帖子和 Profile
- 搜索通过 Exa(免费,无需 API Key
- 可选 Cookie 登录解锁私密内容
#### ~~📷 Instagram~~ (removed — upstream blocked)
- ~~Read public posts and profiles via [instaloader](https://github.com/instaloader/instaloader)~~
- **Removed:** Instagram's aggressive anti-scraping measures broke all available open-source tools (instaloader, etc.). See [instaloader#2585](https://github.com/instaloader/instaloader/issues/2585). Will re-add when upstream recovers.
- **已移除:** Instagram 反爬封杀导致所有开源工具(instaloader 等)失效。上游恢复后会重新加回。
#### 💼 LinkedIn
- Read person profiles, company pages, and job details via [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
@@ -38,12 +35,12 @@ All notable changes to this project will be documented in this file.
- Channel count: 9 → 12
- `agent-reach doctor` now detects all 12 channels
- CLI: added `search-instagram`, `search-linkedin`, `search-bosszhipin` subcommands
- CLI: added `search-linkedin`, `search-bosszhipin` subcommands
- Updated install guide with setup instructions for new channels
- 渠道数量:9 → 12
- `agent-reach doctor` 现在检测全部 12 个渠道
- CLI:新增 `search-instagram``search-linkedin``search-bosszhipin` 子命令
- 安装指南新增三个渠道配置说明
- 渠道数量:9 → 11
- `agent-reach doctor` 现在检测全部 11 个渠道
- CLI:新增 `search-linkedin``search-bosszhipin` 子命令
- 安装指南新增渠道配置说明
---
+107
View File
@@ -0,0 +1,107 @@
# Contributing to Agent Reach
Thank you for your interest in contributing to Agent Reach! This document provides guidelines and instructions for contributing.
## Getting Started
1. Fork the repository on GitHub
2. Clone your fork locally
3. Create a new branch for your contribution
4. Make your changes
5. Run tests and linting
6. Submit a pull request
## Development Setup
```bash
# Clone your fork
git clone https://github.com/YOUR_USERNAME/Agent-Reach.git
cd Agent-Reach
# Install in development mode
pip install -e ".[dev]"
# Install pre-commit hooks (optional but recommended)
pre-commit install
```
## Code Style
We use the following tools to maintain code quality:
- **ruff**: Linting and import sorting
- **mypy**: Type checking
- **pytest**: Testing
Run all checks before submitting a PR:
```bash
# Linting
ruff check agent_reach tests
ruff format agent_reach tests
# Type checking
mypy agent_reach
# Tests
pytest
```
## Adding New Channels
Agent Reach uses a unified channel interface. To add a new platform:
1. Create a new file in `agent_reach/channels/`
2. Implement the channel contract (see existing channels for examples)
3. Add tests in `tests/test_channels.py`
4. Update `agent_reach/doctor.py` to include the new channel
5. Update documentation
## Pull Request Guidelines
- **Small, focused changes** are preferred over large refactors
- Include tests for new functionality
- Update documentation if needed
- Follow existing code style
- Reference any related issues
## Reporting Issues
When reporting bugs, please include:
- Python version
- Operating system
- Steps to reproduce
- Expected vs actual behavior
- Any error messages
## Questions?
Feel free to open an issue for questions or join discussions.
---
感谢您对 Agent Reach 的贡献!本文档提供了贡献指南。
## 快速开始
1. 在 GitHub 上 fork 仓库
2. 本地 clone 您的 fork
3. 创建新分支
4. 提交更改
5. 运行测试和 lint
6. 提交 pull request
## 代码规范
- 使用 **ruff** 进行代码检查
- 使用 **mypy** 进行类型检查
- 使用 **pytest** 运行测试
## 添加新渠道
1.`agent_reach/channels/` 创建新文件
2. 实现渠道接口
3. 添加测试
4. 更新 doctor 检测
5. 更新文档
+110 -27
View File
@@ -11,7 +11,7 @@
</p>
<p align="center">
<a href="#30-秒上手">快速开始</a> · <a href="docs/README_en.md">English</a> · <a href="#支持的平台">支持平台</a> · <a href="#设计理念">设计理念</a>
<a href="#快速上手">快速开始</a> · <a href="docs/README_en.md">English</a> · <a href="#支持的平台">支持平台</a> · <a href="#设计理念">设计理念</a>
</p>
---
@@ -50,7 +50,7 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|---|---|
| 💰 **完全免费** | 所有工具开源、所有 API 免费。唯一可能花钱的是服务器代理($1/月),本地电脑不需要 |
| 🔒 **隐私安全** | Cookie 只存在你本地,不上传不外传。代码完全开源,随时可审查 |
| 🔄 **持续更新** | 底层工具(yt-dlp、bird、Jina Reader 等)定期追踪更新到最新版,你不用自己盯 |
| 🔄 **持续更新** | 底层工具(yt-dlp、xreach、Jina Reader 等)定期追踪更新到最新版,你不用自己盯 |
| 🤖 **兼容所有 Agent** | Claude Code、OpenClaw、Cursor、Windsurf……任何能跑命令行的 Agent 都能用 |
| 🩺 **自带诊断** | `agent-reach doctor` 一条命令告诉你哪个通、哪个不通、怎么修 |
@@ -68,14 +68,15 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
| 🐦 **Twitter/X** | 读单条推文 | 搜索推文、浏览时间线、发推 | 告诉 Agent「帮我配 Twitter」 |
| 📺 **B站** | 本地:字幕提取 + 搜索 | 服务器也能用 | 告诉 Agent「帮我配代理」 |
| 📖 **Reddit** | 搜索(通过 Exa 免费) | 读帖子和评论 | 告诉 Agent「帮我配代理」 |
| 📕 **小红书** | — | 阅读、搜索、发帖、评论、点赞 | `docker run -d -p 18060:18060 xpzouying/xiaohongshu-mcp` 然后告诉 Agent「帮我配小红书」 |
| 📷 **Instagram** | — | 读取帖子和 Profile | Cookie-Editor 导出 Cookie → `agent-reach configure instagram-cookies "..."` |
| 💼 **LinkedIn** | Jina Reader 读公开页面 | Profile 详情、公司、职位搜索 | `linkedin-scraper-mcp --login`(需浏览器,服务器用 VNC |
| 🏢 **Boss直聘** | Jina Reader 读职位页 | 搜索职位、向 HR 打招呼 | 启动 mcp-bosszp → Boss直聘 App 扫码 |
| 📕 **小红书** | — | 阅读、搜索、发帖、评论、点赞 | 告诉 Agent「帮我配小红书」 |
| 🎵 **抖音** | — | 视频解析、无水印下载链接获取 | 告诉 Agent「帮我配抖音」 |
| 💼 **LinkedIn** | Jina Reader 读公开页面 | Profile 详情、公司页面、职位搜索 | 告诉 Agent「帮我配 LinkedIn」 |
| 🏢 **Boss直聘** | Jina Reader 读职位页 | 搜索职位、向 HR 打招呼 | 告诉 Agent「帮我配 Boss直聘」 |
| 💬 **微信公众号** | — | 搜索 + 阅读公众号文章(全文 Markdown) | `pip install miku_ai camoufox[geoip]` |
> **不知道怎么配?不用查文档。** 直接告诉 Agent「帮我配 XXX」,它知道需要什么、会一步一步引导你。
>
> 🍪 需要 Cookie 的平台(Twitter、小红书等),建议使用 Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 一键导出。**服务器用户**没有浏览器界面,请在自己的电脑上登录对应网站后导出 Cookie,再发给 Agent 配置
> 🍪 需要 Cookie 的平台(Twitter、小红书等),**优先使用** Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 导出 Cookie,发给 Agent 即可配置。流程统一:浏览器登录 → Cookie-Editor 导出 → 发给 Agent。比扫码更简单可靠
>
> 🔒 Cookie 只存在你本地,不上传不外传。代码完全开源,随时可审查。
> 💻 本地电脑不需要代理。代理只有部署在服务器上才需要(~$1/月)。
@@ -102,10 +103,10 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
<summary>它会做什么?(点击展开)</summary>
1. **安装 CLI 工具**`pip install` 装好 `agent-reach` 命令行
2. **安装系统依赖** — 自动检测并安装 Node.js、gh CLI、mcporter、bird
2. **安装系统依赖** — 自动检测并安装 Node.js、gh CLI、mcporter、xreach
3. **配置搜索引擎** — 通过 MCP 接入 Exa(免费,无需 API Key
4. **检测环境** — 判断是本地电脑还是服务器,给出对应的配置建议
5. **注册 Skill** — 在 Agent 的 skills 目录安装 SKILL.md,以后 Agent 遇到"搜推特"、"看视频"这类需求,会自动知道调用 Agent Reach
5. **注册 SKILL.md** — 在 Agent 的 skills 目录安装使用指南,以后 Agent 遇到"搜推特"、"看视频"这类需求,会自动知道该调哪个上游工具
安装完之后,`agent-reach doctor` 一条命令告诉你每个渠道的状态。
</details>
@@ -116,14 +117,14 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
不需要任何配置,告诉 Agent 就行:
- "帮我看看这个链接" → 任意网页
- "这个 GitHub 仓库是做什么的" → GitHub 仓库、Issue、代码
- "这个视频讲了什么" → YouTube / B站字幕提取
- "帮我看看这条推文" → Twitter 推文
- "订阅这个 RSS" → RSS / Atom 源
- "搜一下 GitHub 上有什么 LLM 框架" → GitHub 搜索
- "帮我看看这个链接" → `curl https://r.jina.ai/URL`任意网页
- "这个 GitHub 仓库是做什么的" → `gh repo view owner/repo`
- "这个视频讲了什么" → `yt-dlp --dump-json URL` 提取字幕
- "帮我看看这条推文" → `xreach tweet URL --json`
- "订阅这个 RSS" → `feedparser` 解析
- "搜一下 GitHub 上有什么 LLM 框架" → `gh search repos "LLM framework"`
**不需要记命令。** Agent 自己知道该调什么。
**不需要记命令。** Agent 读了 SKILL.md 之后自己知道该调什么。
---
@@ -135,41 +136,47 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
Agent Reach 做的事情很简单:**帮你把这些选型和配置的活儿做完了。**
安装完成后,Agent 直接调用上游工具(xreach CLI、yt-dlp、mcporter、gh CLI 等),不需要经过 Agent Reach 的包装层。
### 🔌 每个渠道都是可插拔的
每个平台对应一个独立的 Python 文件,实现统一接口。**后端工具随时可以换**——哪天出了更好的工具,改一个文件就行,其他不用动。
每个平台背后是一个独立的上游工具。**不满意?换掉就行。**
```
channels/
├── web.py → Jina Reader ← 可以换成 Firecrawl、Crawl4AI……
├── twitter.py → bird ← 可以换成 Nitter、官方 API……
├── youtube.py → yt-dlp ← 可以换成 YouTube API、Whisper……
├── twitter.py → xreach ← 可以换成 Nitter、官方 API……
├── youtube.py → yt-dlp ← 可以换成 YouTube API、Whisper……
├── github.py → gh CLI ← 可以换成 REST API、PyGithub……
├── bilibili.py → yt-dlp ← 可以换成 bilibili-api……
├── bilibili.py → yt-dlp ← 可以换成 bilibili-api……
├── reddit.py → JSON API + Exa ← 可以换成 PRAW、Pushshift……
├── xiaohongshu.py → mcporter MCP ← 可以换成其他 XHS 工具……
├── instagram.py → instaloader ← 可以换成 instagrapi、官方 API……
├── douyin.py → mcporter MCP ← 可以换成其他抖音工具……
├── linkedin.py → linkedin-mcp ← 可以换成 LinkedIn API……
├── bosszhipin.py → mcp-bosszp ← 可以换成其他招聘工具……
├── rss.py → feedparser ← 可以换成 atoma……
├── wechat.py → camoufox+miku ← 搜索+阅读微信公众号文章
├── rss.py → feedparser ← 可以换成 atoma……
├── exa_search.py → mcporter MCP ← 可以换成 Tavily、SerpAPI……
└── __init__.py → 渠道注册
└── __init__.py → 渠道注册doctor 检测用)
```
每个渠道文件只负责检测对应上游工具是否可用(`check()` 方法),给 `agent-reach doctor` 提供状态信息。实际的读取和搜索由 Agent 直接调用上游工具完成。
### 当前选型
| 场景 | 选型 | 为什么选它 |
|------|------|-----------|
| 读网页 | [Jina Reader](https://github.com/jina-ai/reader) | 9.8K Star,免费,不需要 API Key |
| 读推特 | [bird](https://www.npmjs.com/package/@steipete/bird) | Cookie 登录,免费。官方 API 按量付费(读一条 $0.005 |
| 读推特 | [xreach](https://www.npmjs.com/package/xreach-cli) | Cookie 登录,免费。官方 API 按量付费(读一条 $0.005 |
| 视频字幕 + 搜索 | [yt-dlp](https://github.com/yt-dlp/yt-dlp) | 148K StarYouTube + B站 + 1800 站通吃 |
| 搜全网 | [Exa](https://exa.ai) via [mcporter](https://github.com/steipete/mcporter) | AI 语义搜索,MCP 接入免 Key |
| GitHub | [gh CLI](https://cli.github.com) | 官方工具,认证后完整 API 能力 |
| 读 RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python 生态标准选择,2.3K Star |
| 小红书 | [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) | ⭐9K+Go 语言,Docker 一键部署 |
| Instagram | [instaloader](https://github.com/instaloader/instaloader) | ⭐9.8KPython CLICookie 登录,免费 |
| 抖音 | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP 服务,无需登录,视频解析 + 无水印下载 |
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | ⭐900+,MCP 服务,浏览器自动化 |
| Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP 服务,支持职位搜索和打招呼 |
| 微信公众号 | [wechat-article-for-ai](https://github.com/bzd6661/wechat-article-for-ai) + [miku_ai](https://github.com/GobinFan/Miku_Spider) | Camoufox 隐身浏览器读全文 + 搜狗搜索 |
> 📌 这些都是「当前选型」。不满意?换掉对应文件就行。这正是脚手架的意义。
@@ -189,7 +196,11 @@ Agent Reach 在设计上重视安全:
### 🍪 Cookie 安全建议
需要 Cookie 的平台(Twitter、小红书、Instagram)建议使用**专用小号**,不要用主账号。Cookie 等同于完整登录权限,用小号可以在凭据泄露时限制影响范围。
> ⚠️ **封号风险提醒:** 使用 Cookie 登录的平台(Twitter、小红书等),通过脚本/API 调用**存在被平台检测并封号的风险**。请务必使用**专用小号**,不要用你的主账号。
需要 Cookie 的平台(Twitter、小红书)建议使用**专用小号**,不要用主账号。原因有二:
1. **封号风险** — 平台可能检测到非正常浏览器的 API 调用行为,导致账号被限制或封禁
2. **安全风险** — Cookie 等同于完整登录权限,用小号可以在凭据泄露时限制影响范围
### 📦 安装方式
@@ -199,6 +210,24 @@ Agent Reach 在设计上重视安全:
| 安全模式 | `agent-reach install --env=auto --safe` | 生产服务器、多人共用机器 |
| 仅预览 | `agent-reach install --env=auto --dry-run` | 先看看会做什么 |
### 🗑️ 卸载
```bash
agent-reach uninstall
```
会清除:`~/.agent-reach/`(含所有 token/cookie)、各 Agent 的 skill 文件、mcporter 中的 MCP 配置。
```bash
# 只预览,不实际删除
agent-reach uninstall --dry-run
# 只删 skill 文件,保留 token 配置(重装时用)
agent-reach uninstall --keep-config
```
卸载 Python 包本身:`pip uninstall agent-reach`
---
## 贡献
@@ -225,9 +254,63 @@ Agent Reach 在设计上重视安全:
Star 一下,下次需要的时候能找到。⭐
---
## 常见问题 / FAQ
<details>
<summary><strong>AI Agent 怎么搜索 Twitter / X?不想付 API 费用</strong></summary>
Agent Reach 使用 [xreach CLI](https://www.npmjs.com/package/xreach-cli) 通过 Cookie 认证访问 Twitter,完全免费。安装 Agent Reach 后,用 Cookie-Editor 导出你的 Twitter Cookie,运行 `agent-reach configure twitter-cookies "your_cookies"` 即可。之后 Agent 就可以用 `xreach search "关键词" --json` 搜索推文了。
</details>
<details>
<summary><strong>How to search Twitter/X with AI agent for free (no API)?</strong></summary>
Agent Reach uses the xreach CLI with cookie auth — zero API fees. After installing, export your Twitter cookies with the Cookie-Editor extension, run `agent-reach configure twitter-cookies "your_cookies"`, then your agent can search with `xreach search "query" --json`.
</details>
<details>
<summary><strong>Reddit 返回 403 / 服务器 IP 被封怎么办?</strong></summary>
Reddit 封锁数据中心 IP。配置一个住宅代理即可解决:`agent-reach configure proxy http://user:pass@ip:port`。推荐 Webshare ($1/月)。本地电脑一般不会遇到这个问题。
</details>
<details>
<summary><strong>How to get YouTube video transcripts for AI?</strong></summary>
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"` extracts video metadata; `yt-dlp --write-sub --skip-download "URL"` extracts subtitles. Uses yt-dlp under the hood, supports multiple languages. No API key needed.
</details>
<details>
<summary><strong>怎么让 AI Agent 读小红书?</strong></summary>
小红书需要通过 Docker 运行一个 MCP 服务。安装 Docker 后,运行 `agent-reach install` 会自动配置。之后 Agent 就能用 `mcporter call 'xiaohongshu.get_feed_detail(...)'` 读取笔记或 `mcporter call 'xiaohongshu.search_feeds(keyword: "关键词")'` 搜索了。
</details>
<details>
<summary><strong>怎么让 AI Agent 解析抖音视频?</strong></summary>
安装 douyin-mcp-server 后,Agent 就能用 `mcporter call 'douyin.parse_douyin_video_info(share_link: "分享链接")'` 解析视频信息、获取无水印下载链接。不需要登录,把抖音分享链接发给 Agent 就行。详见 https://github.com/yzfly/douyin-mcp-server
</details>
<details>
<summary><strong>Compatible with Claude Code / Cursor / OpenClaw / Windsurf?</strong></summary>
Yes! Agent Reach is an installer + configuration tool — any AI coding agent that can run shell commands can use it. Works with Claude Code, Cursor, OpenClaw, Windsurf, Codex, and more. Just `pip install agent-reach`, run `agent-reach install`, and the agent can start using the upstream tools immediately.
</details>
<details>
<summary><strong>Is this free? Any API costs?</strong></summary>
100% free. All backends are open-source tools (xreach CLI, yt-dlp, Jina Reader, Exa, etc.) that don't require paid API keys. The only optional cost is a residential proxy (~$1/month) if you need Reddit/Bilibili access from a server.
</details>
---
## 致谢
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://www.npmjs.com/package/@steipete/bird) · [Exa](https://exa.ai) · [mcporter](https://github.com/steipete/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) · [instaloader](https://github.com/instaloader/instaloader) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [xreach](https://www.npmjs.com/package/xreach-cli) · [Exa](https://exa.ai) · [mcporter](https://github.com/steipete/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
## License
-87
View File
@@ -1,87 +0,0 @@
---
name: agent-reach
description: >
Give your AI agent eyes to see the entire internet. Read and search across
Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, RSS, and any web page
— all from a single CLI. Use when: (1) reading content from URLs (tweets, Reddit posts,
articles, videos), (2) searching across platforms (web, Twitter, Reddit, GitHub, YouTube,
Bilibili, XiaoHongShu), (3) checking channel health or updating Agent Reach.
Triggers: "search Twitter/Reddit/YouTube", "read this URL", "find posts about",
"搜索", "读取", "查一下", "看看这个链接".
---
# Agent Reach
Read and search the internet across 9+ platforms via unified CLI.
## Setup
First check if agent-reach is installed:
```bash
agent-reach doctor
```
If command not found, install it:
```bash
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
agent-reach install --env=auto
```
`install` auto-detects your environment and installs all dependencies (Node.js, mcporter, bird CLI, gh CLI). Read the output and run `agent-reach doctor` to see what's active.
For channels that need user input, ask the user. See the full setup guide:
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
## Commands
### Read any URL
```bash
agent-reach read <url>
agent-reach read <url> --json # structured output
```
Handles: tweets, Reddit posts, articles, YouTube (transcripts), GitHub repos, etc.
### Search
```bash
agent-reach search "query" # web search (Exa)
agent-reach search-twitter "query" # Twitter/X
agent-reach search-reddit "query" # Reddit (--sub <subreddit>)
agent-reach search-github "query" # GitHub (--lang <language>)
agent-reach search-youtube "query" # YouTube
agent-reach search-bilibili "query" # Bilibili (B站)
agent-reach search-xhs "query" # XiaoHongShu (小红书)
```
All search commands support `-n <count>` for number of results.
### Management
```bash
agent-reach doctor # channel status overview
agent-reach watch # quick health + update check (for scheduled tasks)
agent-reach check-update # check for new versions
```
### Configure channels
```bash
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
agent-reach configure proxy http://user:pass@ip:port
agent-reach configure --from-browser chrome # auto-extract cookies
```
## Channel Status Tiers
- **Tier 0 (zero config):** Web, YouTube, RSS, Twitter (read-only via Jina)
- **Tier 1 (free setup):** Exa web search (mcporter required)
- **Tier 2 (user config):** Twitter search (cookie), Reddit full (proxy), GitHub (token), Bilibili (proxy), XiaoHongShu (MCP)
Run `agent-reach doctor` to see which channels are active.
## Tips
- Always try `agent-reach read <url>` first for any URL — it auto-detects the platform
- For Twitter cookies, recommend the user install [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome extension
- Reddit and Bilibili block server IPs — suggest a residential proxy (~$1/month) if on a server
- If a channel breaks, run `agent-reach doctor` to diagnose
+1 -1
View File
@@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-
"""Agent Reach — Give your AI Agent eyes to see the entire internet."""
__version__ = "1.0.0"
__version__ = "1.3.0"
__author__ = "Neo Reid"
from agent_reach.core import AgentReach
+12 -27
View File
@@ -1,14 +1,10 @@
# -*- coding: utf-8 -*-
"""
Channel registry — routes URLs to the right channel.
This is the core of Agent Reach' pluggable architecture.
Add a new channel: just create a file and register it here.
Swap a backend: just change the implementation inside the channel file.
Channel registry — lists all supported platforms for doctor checks.
"""
from typing import Dict, List, Optional
from .base import Channel, ReadResult, SearchResult
from typing import List, Optional
from .base import Channel
# Import all channels
from .web import WebChannel
@@ -20,12 +16,13 @@ from .rss import RSSChannel
from .bilibili import BilibiliChannel
from .exa_search import ExaSearchChannel
from .xiaohongshu import XiaoHongShuChannel
from .instagram import InstagramChannel
from .douyin import DouyinChannel
from .linkedin import LinkedInChannel
from .bosszhipin import BossZhipinChannel
from .wechat import WeChatChannel
# Channel registry — order matters (first match wins, web is last as fallback)
# Channel registry
ALL_CHANNELS: List[Channel] = [
GitHubChannel(),
TwitterChannel(),
@@ -33,27 +30,15 @@ ALL_CHANNELS: List[Channel] = [
RedditChannel(),
BilibiliChannel(),
XiaoHongShuChannel(),
InstagramChannel(),
DouyinChannel(),
LinkedInChannel(),
BossZhipinChannel(),
WeChatChannel(),
RSSChannel(),
ExaSearchChannel(),
WebChannel(), # Fallback — handles any URL
WebChannel(),
]
# Search-capable channels
SEARCH_CHANNELS: Dict[str, Channel] = {
ch.name: ch for ch in ALL_CHANNELS if ch.can_search()
}
def get_channel_for_url(url: str) -> Channel:
"""Find the right channel for a URL."""
for channel in ALL_CHANNELS:
if channel.can_handle(url):
return channel
return WebChannel() # Should never reach here, but just in case
def get_channel(name: str) -> Optional[Channel]:
"""Get a channel by name."""
@@ -69,7 +54,7 @@ def get_all_channels() -> List[Channel]:
__all__ = [
"Channel", "ReadResult", "SearchResult",
"ALL_CHANNELS", "SEARCH_CHANNELS",
"get_channel_for_url", "get_channel", "get_all_channels",
"Channel",
"ALL_CHANNELS",
"get_channel", "get_all_channels",
]
+11 -115
View File
@@ -1,110 +1,28 @@
# -*- coding: utf-8 -*-
"""
Channel base class — the universal interface for all platforms.
Channel base class — platform availability checking.
Every channel (YouTube, Twitter, GitHub, etc.) implements this interface.
The backend tool can be swapped anytime without changing anything else.
Each channel represents a platform (YouTube, Twitter, GitHub, etc.)
and provides:
- can_handle(url) → does this URL belong to this platform?
- check(config) → is the upstream tool installed and configured?
Example:
class YouTubeChannel(Channel):
name = "youtube"
backends = ["yt-dlp"] # current backend, can be swapped
async def read(self, url, config):
# Just call yt-dlp, return standardized dict
...
After installation, agents call upstream tools directly.
"""
import shutil
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Any, Dict, List, Optional, Tuple
@dataclass
class ReadResult:
"""Standardized read result. Every channel returns this."""
title: str
content: str
url: str
author: str = ""
date: str = ""
platform: str = ""
extra: dict = None
def __post_init__(self):
self.extra = self.extra or {}
def to_dict(self) -> dict:
d = {
"title": self.title,
"content": self.content,
"url": self.url,
"platform": self.platform,
}
if self.author:
d["author"] = self.author
if self.date:
d["date"] = self.date
if self.extra:
d["extra"] = self.extra
return d
@dataclass
class SearchResult:
"""Standardized search result."""
title: str
url: str
snippet: str = ""
author: str = ""
date: str = ""
score: float = 0
extra: dict = None
def __post_init__(self):
self.extra = self.extra or {}
def to_dict(self) -> dict:
d = {
"title": self.title,
"url": self.url,
"snippet": self.snippet,
}
if self.author:
d["author"] = self.author
if self.date:
d["date"] = self.date
if self.extra:
d["extra"] = self.extra
return d
from typing import List, Tuple
class Channel(ABC):
"""
Base class for all channels.
Subclasses just need to implement:
- read(url, config) → ReadResult
- can_handle(url) → bool
- check(config) → (status, message)
Optionally:
- search(query, config, **kwargs) → list[SearchResult]
"""
"""Base class for all channels."""
name: str = "" # e.g. "youtube"
description: str = "" # e.g. "YouTube video transcripts"
backends: List[str] = [] # e.g. ["yt-dlp"] — what external tool is used
requires_config: List[str] = [] # e.g. ["reddit_proxy"]
requires_tools: List[str] = [] # e.g. ["yt-dlp"]
description: str = "" # e.g. "YouTube 视频和字幕"
backends: List[str] = [] # e.g. ["yt-dlp"] — what upstream tool is used
tier: int = 0 # 0=zero-config, 1=needs free key, 2=needs setup
@abstractmethod
async def read(self, url: str, config=None) -> ReadResult:
"""Read content from a URL. Must return ReadResult."""
...
@abstractmethod
def can_handle(self, url: str) -> bool:
"""Check if this channel can handle this URL."""
@@ -112,29 +30,7 @@ class Channel(ABC):
def check(self, config=None) -> Tuple[str, str]:
"""
Check if this channel is available.
Check if this channel's upstream tool is available.
Returns (status, message) where status is 'ok'/'warn'/'off'/'error'.
"""
# Check required tools
for tool in self.requires_tools:
if not shutil.which(tool):
return "off", f"需要安装:pip install {tool}"
# Check required config
for key in self.requires_config:
if config and not config.get(key):
return "off", f"需要配置 {key},运行 agent-reach setup"
return "ok", f"{''.join(self.backends) if self.backends else '内置'}"
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
"""Search this platform. Override if supported."""
raise NotImplementedError(f"{self.name} does not support search")
def can_search(self) -> bool:
"""Whether this channel supports search."""
try:
# Check if search is overridden
return type(self).search is not Channel.search
except:
return False
+9 -190
View File
@@ -1,207 +1,26 @@
# -*- coding: utf-8 -*-
"""Bilibili — via yt-dlp (same backend as YouTube).
"""Bilibili — check if yt-dlp is available."""
Backend: yt-dlp (https://github.com/yt-dlp/yt-dlp)
yt-dlp natively supports Bilibili — video info, subtitles, and search.
"""
import json
import os
import shutil
import subprocess
from urllib.parse import urlparse
from .base import Channel, ReadResult, SearchResult
from typing import List
from .base import Channel
class BilibiliChannel(Channel):
name = "bilibili"
description = "B站视频信息和字幕"
description = "B站视频和字幕"
backends = ["yt-dlp"]
requires_tools = ["yt-dlp"]
tier = 0
tier = 1
def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
d = urlparse(url).netloc.lower()
return "bilibili.com" in d or "b23.tv" in d
def check(self, config=None):
if not shutil.which("yt-dlp"):
return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
proxy = config.get("bilibili_proxy") if config else None
proxy = (config.get("bilibili_proxy") if config else None) or os.environ.get("BILIBILI_PROXY")
if proxy:
return "ok", "已配置代理,完整可用"
import os
is_server = bool(os.environ.get("SSH_CONNECTION") or os.path.exists("/etc/cloud"))
if is_server:
return "warn", "服务器 IP 可能被封,配置代理即可解决:agent-reach configure proxy URL"
return "ok", "本地直连可用"
async def read(self, url: str, config=None) -> ReadResult:
if not shutil.which("yt-dlp"):
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
proxy = config.get("bilibili_proxy") if config else None
# Get video info via yt-dlp
info = self._get_info(url, proxy)
if not info:
return ReadResult(
title="Bilibili",
content=f"⚠️ 无法获取视频信息: {url}\n服务器 IP 可能被封,配个代理:agent-reach configure proxy URL",
url=url, platform="bilibili",
)
title = info.get("title", url)
author = info.get("uploader", "")
desc = info.get("description", "")
# Try subtitles
subtitle = self._get_subtitles(url, proxy)
content = desc
if subtitle:
content += f"\n\n## 字幕\n{subtitle}"
return ReadResult(
title=title, content=content, url=url,
author=author, platform="bilibili",
extra={
"view_count": info.get("view_count"),
"like_count": info.get("like_count"),
"duration": info.get("duration_string"),
},
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
"""Search Bilibili.
Strategy:
1. Try yt-dlp bilisearch (works on local machines)
2. Fallback to Exa site:bilibili.com (works on servers)
"""
if not shutil.which("yt-dlp"):
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
limit = kwargs.get("limit", 5)
proxy = config.get("bilibili_proxy") if config else None
# Strategy 1: yt-dlp bilisearch
results = self._search_ytdlp(query, limit, proxy)
if results:
return results
# Strategy 2: Exa fallback (server-friendly)
results = self._search_exa(query, limit)
if results:
return results
return []
def _search_ytdlp(self, query: str, limit: int, proxy: str = None) -> List[SearchResult]:
"""Search via yt-dlp bilisearch (needs local/Chinese IP)."""
cmd = [
"yt-dlp", "--dump-json", "--no-download",
f"bilisearch{limit}:{query}",
]
if proxy:
cmd += ["--proxy", proxy]
try:
r = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
if r.returncode != 0:
return []
results = []
for line in r.stdout.strip().split("\n"):
if not line.strip():
continue
try:
d = json.loads(line)
vid = d.get("id", "")
url = d.get("webpage_url", f"https://www.bilibili.com/video/av{vid}")
results.append(SearchResult(
title=d.get("title", f"av{vid}"),
url=url,
snippet=f"👤 {d.get('uploader', '?')} · 👁 {d.get('view_count', '?')}",
extra={
"view_count": d.get("view_count"),
"uploader": d.get("uploader"),
"duration": d.get("duration_string"),
},
))
except json.JSONDecodeError:
continue
return results
except subprocess.TimeoutExpired:
return []
def _search_exa(self, query: str, limit: int) -> List[SearchResult]:
"""Fallback: search via Exa (site:bilibili.com). Works on any IP."""
try:
r = subprocess.run(
["mcporter", "call",
f'exa.web_search_exa(query: "site:bilibili.com {query}", numResults: {limit})'],
capture_output=True, text=True, timeout=30,
)
if r.returncode != 0:
return []
results = []
# Parse mcporter output: Title: / Author: / URL: / Text: blocks
title, author, url = "", "", ""
for line in r.stdout.split("\n"):
if line.startswith("Title: "):
title = line[7:].strip()
elif line.startswith("Author: "):
author = line[8:].strip()
elif line.startswith("URL: "):
url = line[5:].strip()
if url and "bilibili.com" in url:
results.append(SearchResult(
title=title or url,
url=url,
snippet=f"👤 {author}" if author else "(via Exa search)",
))
title, author, url = "", "", ""
return results
except Exception:
return []
def _get_info(self, url: str, proxy: str = None) -> dict:
cmd = ["yt-dlp", "--dump-json", "--no-download", url]
if proxy:
cmd += ["--proxy", proxy]
try:
r = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
if r.returncode == 0:
return json.loads(r.stdout)
except (subprocess.TimeoutExpired, json.JSONDecodeError):
pass
return {}
def _get_subtitles(self, url: str, proxy: str = None) -> str:
import tempfile
from pathlib import Path
with tempfile.TemporaryDirectory() as tmpdir:
cmd = [
"yt-dlp", "--write-sub", "--write-auto-sub",
"--sub-lang", "zh-Hans,zh,en",
"--skip-download", "--sub-format", "vtt",
"-o", f"{tmpdir}/%(id)s.%(ext)s", url,
]
if proxy:
cmd += ["--proxy", proxy]
try:
subprocess.run(cmd, capture_output=True, text=True, timeout=30)
for f in Path(tmpdir).glob("*.vtt"):
text = f.read_text(errors="replace")
lines = []
for line in text.split("\n"):
line = line.strip()
if not line or line.startswith("WEBVTT") or "-->" in line or line.isdigit():
continue
if line not in lines[-1:]:
lines.append(line)
return "\n".join(lines)
except subprocess.TimeoutExpired:
pass
return ""
return "ok", "可提取视频信息和字幕(代理已配置)"
return "ok", "可提取视频信息和字幕(本地环境)。服务器可能需要代理"
+23 -163
View File
@@ -1,62 +1,9 @@
# -*- coding: utf-8 -*-
"""Boss直聘 (BOSS Zhipin) — via mcp-bosszp (MCP) or Jina Reader fallback.
"""Boss直聘 — check if mcp-bosszp is available."""
Backend: mcp-bosszp (161 stars, FastMCP + Playwright)
Swap to: any Boss直聘 access tool
"""
import json
import shutil
import subprocess
from urllib.parse import urlparse
from .base import Channel, ReadResult, SearchResult
from typing import List
import requests
def _mcporter_has_bosszhipin() -> bool:
"""Check if mcporter has Boss直聘 MCP configured."""
if not shutil.which("mcporter"):
return False
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
# Check for various possible config names
out = r.stdout.lower()
return "boss" in out or "zhipin" in out or "bosszhipin" in out
except Exception:
return False
def _mcporter_call(expr: str, timeout: int = 30) -> str:
"""Call a Boss直聘 MCP tool via mcporter."""
r = subprocess.run(
["mcporter", "call", expr],
capture_output=True, text=True, timeout=timeout,
)
if r.returncode != 0:
raise RuntimeError(r.stderr or r.stdout)
return r.stdout
def _get_mcp_name() -> str:
"""Get the actual MCP server name configured in mcporter."""
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
for line in r.stdout.split("\n"):
line_lower = line.strip().lower()
for name in ["bosszhipin", "boss-zp", "bosszp", "boss"]:
if name in line_lower:
# Extract the actual server name
parts = line.strip().split()
if parts:
return parts[0]
return "bosszhipin"
except Exception:
return "bosszhipin"
from .base import Channel
class BossZhipinChannel(Channel):
@@ -66,118 +13,31 @@ class BossZhipinChannel(Channel):
tier = 2
def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
domain = urlparse(url).netloc.lower()
return "zhipin.com" in domain or "boss.com" in domain
def check(self, config=None):
if _mcporter_has_bosszhipin():
return "ok", "可搜索职位、向 HR 打招呼"
mcporter = shutil.which("mcporter")
if not mcporter:
return "off", (
"可通过 Jina Reader 读取职位页面。完整功能需要:\n"
" 1. git clone https://github.com/mucsbr/mcp-bosszp.git\n"
" 2. cd mcp-bosszp && pip install -r requirements.txt && playwright install chromium\n"
" 3. python boss_zhipin_fastmcp_v2.py(启动后扫码登录)\n"
" 4. mcporter config add bosszhipin http://localhost:8000/mcp"
)
try:
r = subprocess.run(
[mcporter, "list"], capture_output=True,
encoding="utf-8", errors="replace", timeout=10
)
out = r.stdout.lower()
if "boss" in out or "zhipin" in out:
return "ok", "可搜索职位、向 HR 打招呼"
except Exception:
pass
return "off", (
"可通过 Jina Reader 读取职位页面。完整功能需要:\n"
" 1. git clone https://github.com/mucsbr/mcp-bosszp.git\n"
" 2. cd mcp-bosszp && pip install -r requirements.txt && playwright install chromium\n"
" 3. python boss_zhipin_fastmcp_v2.py(启动后扫码登录)\n"
" 4. mcporter config add bosszhipin http://localhost:8000/mcp\n"
" 或用 Dockerdocker-compose up -d\n"
"mcporter 已装但 Boss直聘 MCP 未配置。\n"
" 详见 https://github.com/mucsbr/mcp-bosszp"
)
async def read(self, url: str, config=None) -> ReadResult:
# Boss直聘 pages mostly work with Jina Reader
return await self._read_jina(url)
async def _read_jina(self, url: str) -> ReadResult:
"""Read Boss直聘 page via Jina Reader."""
try:
resp = requests.get(
f"https://r.jina.ai/{url}",
headers={"Accept": "text/markdown"},
timeout=15,
)
resp.raise_for_status()
text = resp.text
if len(text.strip()) < 50:
return ReadResult(
title="Boss直聘",
content=(
f"⚠️ 无法读取此页面内容: {url}\n\n"
"提示:\n"
"- 安装 mcp-bosszp 可解锁职位搜索和自动打招呼\n"
"- 详见 https://github.com/mucsbr/mcp-bosszp"
),
url=url,
platform="bosszhipin",
)
return ReadResult(
title=text[:100] if text else url,
content=text,
url=url,
platform="bosszhipin",
)
except Exception:
return ReadResult(
title="Boss直聘",
content=(
f"⚠️ 无法读取此 Boss直聘页面: {url}\n\n"
"提示:\n"
"- Boss直聘部分页面需要登录\n"
"- 安装 mcp-bosszp 可解锁完整功能\n"
"- 详见 https://github.com/mucsbr/mcp-bosszp"
),
url=url,
platform="bosszhipin",
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
limit = kwargs.get("limit", 10)
# Try MCP search first
if _mcporter_has_bosszhipin():
try:
return await self._search_mcp(query, limit, config)
except Exception:
pass
# Fallback to Exa
from agent_reach.channels.exa_search import ExaSearchChannel
exa = ExaSearchChannel()
return await exa.search(f"site:zhipin.com {query}", config=config, limit=limit)
async def _search_mcp(self, query: str, limit: int, config=None) -> List[SearchResult]:
"""Search Boss直聘 via MCP."""
server = _get_mcp_name()
try:
out = _mcporter_call(
f'{server}.get_recommend_jobs_tool(page: 1)',
timeout=30,
)
return self._parse_jobs(out, limit)
except Exception:
return []
def _parse_jobs(self, text: str, limit: int) -> List[SearchResult]:
"""Parse MCP job search output into SearchResults."""
results = []
try:
data = json.loads(text)
jobs = data if isinstance(data, list) else data.get("jobs", data.get("results", []))
for job in jobs[:limit]:
if isinstance(job, dict):
title = job.get("title") or job.get("jobName", "")
company = job.get("company") or job.get("brandName", "")
salary = job.get("salary") or job.get("salaryDesc", "")
url = job.get("url", "")
snippet = f"🏢 {company}" if company else ""
if salary:
snippet += f" · 💰 {salary}"
results.append(SearchResult(
title=title,
url=url,
snippet=snippet,
))
except (json.JSONDecodeError, KeyError):
pass
return results
+54
View File
@@ -0,0 +1,54 @@
# -*- coding: utf-8 -*-
"""Douyin (抖音) — check if mcporter + douyin-mcp-server is available."""
import shutil
import subprocess
from .base import Channel
class DouyinChannel(Channel):
name = "douyin"
description = "抖音短视频"
backends = ["douyin-mcp-server"]
tier = 2
def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
d = urlparse(url).netloc.lower()
return "douyin.com" in d or "iesdouyin.com" in d
def check(self, config=None):
mcporter = shutil.which("mcporter")
if not mcporter:
return "off", (
"需要 mcporter + douyin-mcp-server。安装步骤:\n"
" 1. npm install -g mcporter\n"
" 2. pip install douyin-mcp-server\n"
" 3. 启动服务(见下方说明)\n"
" 4. mcporter config add douyin http://localhost:18070/mcp\n"
" 详见 https://github.com/yzfly/douyin-mcp-server"
)
try:
r = subprocess.run(
[mcporter, "config", "list"], capture_output=True,
encoding="utf-8", errors="replace", timeout=5
)
if "douyin" not in r.stdout:
return "off", (
"mcporter 已装但抖音 MCP 未配置。运行:\n"
" pip install douyin-mcp-server\n"
" # 启动服务后:\n"
" mcporter config add douyin http://localhost:18070/mcp"
)
except Exception:
return "off", "mcporter 连接异常"
try:
r = subprocess.run(
[mcporter, "call", "douyin.parse_douyin_video_info(share_link: \"https://www.douyin.com\")"],
capture_output=True, encoding="utf-8", errors="replace", timeout=15
)
if r.returncode == 0:
return "ok", "完整可用(视频解析、下载链接获取)"
return "warn", "MCP 已连接但调用异常,检查 douyin-mcp-server 服务是否在运行"
except Exception:
return "warn", "MCP 连接异常,检查 douyin-mcp-server 服务是否在运行"
+22 -94
View File
@@ -1,110 +1,38 @@
# -*- coding: utf-8 -*-
"""Exa semantic search — via mcporter + Exa MCP server.
"""Exa Search — check if mcporter + Exa MCP is available."""
Backend: Exa MCP at mcp.exa.ai (OAuth, no API key needed)
Requires: mcporter CLI
"""
import json
import shutil
import subprocess
from .base import Channel, SearchResult
from typing import List
from .base import Channel
class ExaSearchChannel(Channel):
name = "exa_search"
description = "全网语义搜索(同时支持 Reddit/Twitter 搜索)"
backends = ["exa-mcp"]
tier = 1
def _mcporter_ok(self) -> bool:
if not shutil.which("mcporter"):
return False
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
return "exa" in r.stdout
except Exception:
return False
def _call(self, expr: str, timeout: int = 30) -> str:
r = subprocess.run(
["mcporter", "call", expr],
capture_output=True, text=True, timeout=timeout,
)
if r.returncode != 0:
raise RuntimeError(r.stderr or r.stdout)
return r.stdout
# ── Channel interface ──
description = "全网语义搜索"
backends = ["Exa via mcporter"]
tier = 0
def can_handle(self, url: str) -> bool:
return False # search-only
async def read(self, url: str, config=None):
raise NotImplementedError("Exa is a search engine, not a reader")
return False # Search-only channel
def check(self, config=None):
if not shutil.which("mcporter"):
mcporter = shutil.which("mcporter")
if not mcporter:
return "off", (
"需要 mcporter。安装:npm install -g mcporter && "
"mcporter config add exa https://mcp.exa.ai/mcp"
)
if not self._mcporter_ok():
return "off", "mcporter 已装但 Exa 未配置。运行:mcporter config add exa https://mcp.exa.ai/mcp"
return "ok", "MCP 已连接,免 Key 直接可用(全网搜索 + Reddit + Twitter"
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
if not self._mcporter_ok():
raise ValueError(
"Exa 搜索需要 mcporter。安装:\n"
"需要 mcporter + Exa MCP。安装:\n"
" npm install -g mcporter\n"
" mcporter config add exa https://mcp.exa.ai/mcp"
)
limit = kwargs.get("limit", 5)
safe_q = query.replace('"', '\\"')
out = self._call(
f'exa.web_search_exa(query: "{safe_q}", numResults: {min(limit, 10)})',
timeout=30,
)
return self._parse_output(out, limit)
# ── Parse mcporter text output ──
def _parse_output(self, text: str, limit: int) -> List[SearchResult]:
"""Parse mcporter's Title/URL/Text block format."""
results = []
cur = {}
for line in text.split("\n"):
line = line.strip()
if line.startswith("Title: "):
if cur.get("title"):
results.append(self._make_result(cur))
cur = {"title": line[7:]}
elif line.startswith("URL: "):
cur["url"] = line[5:]
elif line.startswith("Published Date: "):
cur["date"] = line[16:]
elif line.startswith("Text: "):
cur["text"] = line[6:]
elif "text" in cur and line:
cur["text"] += " " + line
if cur.get("title"):
results.append(self._make_result(cur))
return results[:limit]
@staticmethod
def _make_result(d: dict) -> SearchResult:
return SearchResult(
title=d.get("title", ""),
url=d.get("url", ""),
snippet=d.get("text", "")[:500],
date=d.get("date", ""),
score=0,
)
try:
r = subprocess.run(
[mcporter, "config", "list"], capture_output=True,
encoding="utf-8", errors="replace", timeout=5
)
if "exa" in r.stdout.lower():
return "ok", "全网语义搜索可用(免费,无需 API Key)"
return "off", (
"mcporter 已装但 Exa 未配置。运行:\n"
" mcporter config add exa https://mcp.exa.ai/mcp"
)
except Exception:
return "off", "mcporter 连接异常"
+13 -120
View File
@@ -1,16 +1,9 @@
# -*- coding: utf-8 -*-
"""GitHub — via gh CLI.
"""GitHub — check if gh CLI is available."""
Backend: gh CLI (https://cli.github.com)
Swap to: GitHub REST API
"""
import json
import shutil
import subprocess
from urllib.parse import urlparse
from .base import Channel, ReadResult, SearchResult
from typing import List
from .base import Channel
class GitHubChannel(Channel):
@@ -19,121 +12,21 @@ class GitHubChannel(Channel):
backends = ["gh CLI"]
tier = 0
def _gh(self, args: list, timeout: int = 15) -> str:
r = subprocess.run(
["gh"] + args,
capture_output=True, text=True, timeout=timeout,
)
if r.returncode != 0:
raise RuntimeError(r.stderr or r.stdout)
return r.stdout
def _gh_json(self, args: list, timeout: int = 15) -> dict:
return json.loads(self._gh(args + ["--json"], timeout))
def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
return "github.com" in urlparse(url).netloc.lower()
def check(self, config=None):
if not shutil.which("gh"):
return "warn", "gh CLI 未安装。安装:https://cli.github.com 。公开仓库仍可通过 Jina Reader 读取"
gh = shutil.which("gh")
if not gh:
return "warn", "gh CLI 未安装。安装:https://cli.github.com"
try:
self._gh(["auth", "status"], timeout=5)
return "ok", "完整可用(读取、搜索、Fork、Issue、PR 等)"
except Exception:
return "ok", "gh CLI 已装但未认证。运行 gh auth login 可解锁完整功能"
async def read(self, url: str, config=None) -> ReadResult:
if not shutil.which("gh"):
# Fallback to Jina Reader for public repos
from agent_reach.channels.web import WebChannel
return await WebChannel().read(url, config)
path = urlparse(url).path.strip("/").split("/")
if len(path) < 2:
from agent_reach.channels.web import WebChannel
return await WebChannel().read(url, config)
owner, repo = path[0], path[1]
# Issues / PRs
if len(path) >= 4 and path[2] in ("issues", "pull"):
return await self._read_issue(owner, repo, path[3], url)
# Repo
return await self._read_repo(owner, repo, url)
async def _read_repo(self, owner: str, repo: str, url: str) -> ReadResult:
slug = f"{owner}/{repo}"
try:
# Get repo info
info = self._gh(["repo", "view", slug])
# Get README
try:
readme = self._gh(
["api", f"repos/{slug}/readme", "--jq", ".content"],
timeout=10,
)
import base64
readme_text = base64.b64decode(readme).decode("utf-8", errors="replace")
except Exception:
readme_text = ""
content = readme_text or info
return ReadResult(
title=slug, content=content, url=url,
author=owner, platform="github",
r = subprocess.run(
[gh, "auth", "status"],
capture_output=True, encoding="utf-8", errors="replace", timeout=5
)
if r.returncode == 0:
return "ok", "完整可用(读取、搜索、Fork、Issue、PR 等)"
return "warn", "gh CLI 已安装但未认证。运行 gh auth login 可解锁完整功能"
except Exception:
from agent_reach.channels.web import WebChannel
return await WebChannel().read(url)
async def _read_issue(self, owner: str, repo: str, num: str, url: str) -> ReadResult:
slug = f"{owner}/{repo}"
try:
out = self._gh(["issue", "view", num, "-R", slug])
return ReadResult(
title=f"{slug}#{num}", content=out, url=url,
platform="github",
)
except Exception:
# Might be a PR
try:
out = self._gh(["pr", "view", num, "-R", slug])
return ReadResult(
title=f"{slug}#{num}", content=out, url=url,
platform="github",
)
except Exception:
from agent_reach.channels.web import WebChannel
return await WebChannel().read(url)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
if not shutil.which("gh"):
raise ValueError("GitHub search requires gh CLI. Install: https://cli.github.com")
language = kwargs.get("language")
limit = kwargs.get("limit", 5)
args = ["search", "repos", query, "--sort", "stars", f"--limit={limit}"]
if language:
args += [f"--language={language}"]
out = self._gh(args, timeout=15)
results = []
for line in out.strip().split("\n"):
if not line.strip():
continue
parts = line.split("\t")
if len(parts) >= 1:
slug = parts[0].strip()
desc = parts[1].strip() if len(parts) > 1 else ""
stars = parts[3].strip() if len(parts) > 3 else ""
lang = parts[5].strip() if len(parts) > 5 else ""
results.append(SearchResult(
title=slug,
url=f"https://github.com/{slug}",
snippet=desc,
extra={"stars": stars, "language": lang},
))
return results
return "warn", "gh CLI 状态检查失败,运行 gh auth status 查看详情"
-248
View File
@@ -1,248 +0,0 @@
# -*- coding: utf-8 -*-
"""Instagram — via instaloader (free, open source).
Backend: instaloader (9.8K stars, Python CLI + library)
Swap to: any Instagram access tool
"""
import re
import shutil
import subprocess
from pathlib import Path
from urllib.parse import urlparse
from .base import Channel, ReadResult, SearchResult
from typing import List
class InstagramChannel(Channel):
name = "instagram"
description = "Instagram 帖子和 Profile"
backends = ["instaloader"]
tier = 2 # Needs login for full access
def can_handle(self, url: str) -> bool:
domain = urlparse(url).netloc.lower()
return "instagram.com" in domain or "instagr.am" in domain
def check(self, config=None):
# Check both CLI and Python module
has_cli = shutil.which("instaloader")
has_module = False
try:
import instaloader
has_module = True
except ImportError:
pass
if not has_cli and not has_module:
return "off", (
"需要安装 instaloaderpip install instaloader\n"
" 安装后可读取 Instagram 帖子和 Profile\n"
" 登录: agent-reach configure instagram-cookies \"sessionid=xxx; csrftoken=yyy; ...\""
)
# Check if cookies are configured
cookie_file = Path.home() / ".agent-reach" / "instagram-cookies.txt"
if cookie_file.exists():
return "ok", "已登录,可读取 Instagram 帖子和 Profile"
return "ok", "可读取公开帖子和 Profile。登录可访问更多内容:\n agent-reach configure instagram-cookies \"sessionid=xxx; csrftoken=yyy; ...\""
async def read(self, url: str, config=None) -> ReadResult:
# Try instaloader (module or CLI)
try:
import instaloader
return await self._read_instaloader(url, config)
except ImportError:
pass
# Fallback: Jina Reader
return await self._read_jina(url)
async def _read_instaloader(self, url: str, config=None) -> ReadResult:
"""Read Instagram content using instaloader Python API."""
import asyncio
import concurrent.futures
def _sync_read():
import instaloader
L = instaloader.Instaloader(
download_pictures=False,
download_videos=False,
download_video_thumbnails=False,
download_geotags=False,
download_comments=False,
save_metadata=False,
compress_json=False,
max_connection_attempts=1, # Don't retry on rate limit
)
# Try to load session: cookie file > saved session
cookie_file = Path.home() / ".agent-reach" / "instagram-cookies.txt"
if cookie_file.exists():
try:
cookie_str = cookie_file.read_text().strip()
cookies = {}
for part in cookie_str.split(";"):
part = part.strip()
if "=" in part:
k, v = part.split("=", 1)
cookies[k.strip()] = v.strip()
if "sessionid" in cookies and "csrftoken" in cookies:
# Extract username from ds_user_id or use generic
username = cookies.get("ds_user_id", "user")
L.context.load_session(username, cookies)
except Exception:
pass
elif config and config.get("instagram_username"):
try:
L.load_session_from_file(config.get("instagram_username"))
except Exception:
pass
path = urlparse(url).path.strip("/")
if "/p/" in url or "/reel/" in url:
return self._read_post_sync(L, url, path)
else:
return self._read_profile_sync(L, url, path)
try:
# Run with 15s timeout to avoid instaloader's 30-min retry
loop = asyncio.get_event_loop()
with concurrent.futures.ThreadPoolExecutor() as pool:
result = await asyncio.wait_for(
loop.run_in_executor(pool, _sync_read),
timeout=15,
)
return result
except (asyncio.TimeoutError, Exception):
# Any error or timeout → Jina fallback
return await self._read_jina(url)
def _read_post_sync(self, L, url: str, path: str) -> ReadResult:
"""Read a single Instagram post (sync, runs in executor)."""
import instaloader
# Extract shortcode from URL
match = re.search(r"/(?:p|reel)/([A-Za-z0-9_-]+)", url)
if not match:
raise ValueError("Cannot extract shortcode from URL")
shortcode = match.group(1)
try:
post = instaloader.Post.from_shortcode(L.context, shortcode)
lines = []
if post.caption:
lines.append(post.caption)
lines.append("")
lines.append(f"👤 @{post.owner_username}")
lines.append(f"❤️ {post.likes} likes")
if post.comments:
lines.append(f"💬 {post.comments} comments")
lines.append(f"📅 {post.date_utc.strftime('%Y-%m-%d %H:%M')}")
if post.location:
lines.append(f"📍 {post.location}")
if post.hashtags:
lines.append(f"#️⃣ {' '.join('#' + h for h in post.hashtags)}")
return ReadResult(
title=f"@{post.owner_username}: {(post.caption or '')[:80]}",
content="\n".join(lines),
url=url,
author=f"@{post.owner_username}",
date=post.date_utc.strftime("%Y-%m-%d"),
platform="instagram",
extra={"likes": post.likes, "comments": post.comments},
)
except Exception:
raise # Let executor timeout handle fallback
def _read_profile_sync(self, L, url: str, path: str) -> ReadResult:
"""Read an Instagram profile (sync, runs in executor)."""
import instaloader
# Extract username from path
username = path.split("/")[0] if path else ""
if not username or username in ("p", "reel", "stories", "explore"):
raise ValueError("Cannot extract username from URL")
try:
profile = instaloader.Profile.from_username(L.context, username)
lines = []
lines.append(f"👤 {profile.full_name} (@{profile.username})")
if profile.biography:
lines.append(f"📝 {profile.biography}")
if profile.external_url:
lines.append(f"🔗 {profile.external_url}")
lines.append("")
lines.append(f"📊 {profile.mediacount} posts · "
f"{profile.followers} followers · "
f"{profile.followees} following")
if profile.is_verified:
lines.append("✅ Verified")
if profile.is_business_account and profile.business_category_name:
lines.append(f"🏢 {profile.business_category_name}")
# Get recent posts (up to 5)
lines.append("")
lines.append("📸 Recent posts:")
count = 0
for post in profile.get_posts():
if count >= 5:
break
caption = (post.caption or "")[:100].replace("\n", " ")
lines.append(f" • ❤️{post.likes} | {post.date_utc.strftime('%m-%d')} | {caption}")
count += 1
return ReadResult(
title=f"{profile.full_name} (@{profile.username}) - Instagram",
content="\n".join(lines),
url=url,
author=f"@{profile.username}",
platform="instagram",
extra={
"followers": profile.followers,
"posts": profile.mediacount,
},
)
except Exception:
raise # Let executor timeout handle fallback
async def _read_jina(self, url: str) -> ReadResult:
"""Fallback: use Jina Reader."""
import requests
try:
resp = requests.get(
f"https://r.jina.ai/{url}",
headers={"Accept": "text/markdown"},
timeout=15,
)
resp.raise_for_status()
text = resp.text
return ReadResult(
title=text[:100] if text else url,
content=text,
url=url,
platform="instagram",
)
except Exception:
return ReadResult(
title="Instagram",
content=(
f"⚠️ 无法读取此 Instagram 内容: {url}\n\n"
"提示:\n"
"- 确保 URL 正确\n"
"- 安装 instaloader: pip install instaloader\n"
"- 登录以访问更多内容: instaloader --login YOUR_USERNAME"
),
url=url,
platform="instagram",
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
"""Search Instagram via Exa."""
limit = kwargs.get("limit", 10)
from agent_reach.channels.exa_search import ExaSearchChannel
exa = ExaSearchChannel()
return await exa.search(f"site:instagram.com {query}", config=config, limit=limit)
+24 -251
View File
@@ -1,268 +1,41 @@
# -*- coding: utf-8 -*-
"""LinkedIn — via linkedin-scraper-mcp (MCP) or Jina Reader fallback.
Backend: linkedin-scraper-mcp (916 stars, Patchright browser automation)
Swap to: any LinkedIn access tool
"""
"""LinkedIn — check if linkedin-scraper-mcp is available."""
import shutil
import subprocess
from urllib.parse import urlparse
from .base import Channel, ReadResult, SearchResult
from typing import List
import requests
def _mcporter_has_linkedin() -> bool:
"""Check if mcporter has linkedin MCP configured."""
if not shutil.which("mcporter"):
return False
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
return "linkedin" in r.stdout.lower()
except Exception:
return False
def _mcporter_call(expr: str, timeout: int = 30) -> str:
"""Call a LinkedIn MCP tool via mcporter."""
r = subprocess.run(
["mcporter", "call", expr],
capture_output=True, text=True, timeout=timeout,
)
if r.returncode != 0:
raise RuntimeError(r.stderr or r.stdout)
return r.stdout
from .base import Channel
class LinkedInChannel(Channel):
name = "linkedin"
description = "LinkedIn 个人/公司 Profile 和职位"
description = "LinkedIn 职业社交"
backends = ["linkedin-scraper-mcp", "Jina Reader"]
tier = 2
def can_handle(self, url: str) -> bool:
domain = urlparse(url).netloc.lower()
return "linkedin.com" in domain
from urllib.parse import urlparse
return "linkedin.com" in urlparse(url).netloc.lower()
def check(self, config=None):
if _mcporter_has_linkedin():
return "ok", "完整可用(Profile、公司、职位搜索)"
# Check if linkedin-scraper-mcp is installed as CLI
if shutil.which("linkedin-scraper-mcp"):
return "warn", (
"linkedin-scraper-mcp 已安装但未接入 mcporter。运行:\n"
" 1. linkedin-scraper-mcp --login(在有浏览器的机器上登录)\n"
" 2. linkedin-scraper-mcp --transport streamable-http --port 8001\n"
" 3. mcporter config add linkedin http://localhost:8001/mcp"
mcporter = shutil.which("mcporter")
if not mcporter:
return "off", (
"基本内容可通过 Jina Reader 读取。完整功能需要:\n"
" pip install linkedin-scraper-mcp\n"
" mcporter config add linkedin http://localhost:3000/mcp\n"
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
)
try:
r = subprocess.run(
[mcporter, "config", "list"], capture_output=True,
encoding="utf-8", errors="replace", timeout=5
)
if "linkedin" in r.stdout.lower():
return "ok", "完整可用(Profile、公司、职位搜索)"
except Exception:
pass
return "off", (
"可通过 Jina Reader 读取部分内容。完整功能需要\n"
" 1. pip install linkedin-scraper-mcp\n"
" 2. linkedin-scraper-mcp --login(在有浏览器的机器上登录)\n"
" 3. linkedin-scraper-mcp --transport streamable-http --port 8001\n"
" 4. mcporter config add linkedin http://localhost:8001/mcp\n"
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
"mcporter 已装但 LinkedIn MCP 未配置。运行\n"
" pip install linkedin-scraper-mcp\n"
" mcporter config add linkedin http://localhost:3000/mcp"
)
async def read(self, url: str, config=None) -> ReadResult:
path = urlparse(url).path.strip("/")
# Try MCP first
if _mcporter_has_linkedin():
try:
if "/in/" in url:
return await self._read_profile_mcp(url)
elif "/company/" in url:
return await self._read_company_mcp(url)
elif "/jobs/view/" in url:
return await self._read_job_mcp(url)
except Exception:
pass # Fall through to Jina
# Fallback: Jina Reader
return await self._read_jina(url)
async def _read_profile_mcp(self, url: str) -> ReadResult:
"""Read a LinkedIn profile via MCP."""
import re
# Extract username from URL: /in/username/
match = re.search(r"/in/([^/]+)", url)
if not match:
return await self._read_jina(url)
username = match.group(1)
safe_username = username.replace('"', '\\"')
out = _mcporter_call(
f'linkedin.get_person_profile(linkedin_username: "{safe_username}")',
timeout=60,
)
return ReadResult(
title=self._extract_title(out) or f"LinkedIn Profile - {username}",
content=out.strip(),
url=url,
platform="linkedin",
)
async def _read_company_mcp(self, url: str) -> ReadResult:
"""Read a LinkedIn company page via MCP."""
import re
# Extract company name from URL: /company/name/
match = re.search(r"/company/([^/]+)", url)
if not match:
return await self._read_jina(url)
company = match.group(1)
safe_company = company.replace('"', '\\"')
out = _mcporter_call(
f'linkedin.get_company_profile(company_name: "{safe_company}")',
timeout=60,
)
return ReadResult(
title=self._extract_title(out) or "LinkedIn Company",
content=out.strip(),
url=url,
platform="linkedin",
)
async def _read_job_mcp(self, url: str) -> ReadResult:
"""Read a LinkedIn job posting via MCP."""
import re
match = re.search(r"/jobs/view/(\d+)", url)
if not match:
return await self._read_jina(url)
job_id = match.group(1)
out = _mcporter_call(
f'linkedin.get_job_details(job_id: "{job_id}")',
timeout=30,
)
return ReadResult(
title=self._extract_title(out) or f"LinkedIn Job {job_id}",
content=out.strip(),
url=url,
platform="linkedin",
)
async def _read_jina(self, url: str) -> ReadResult:
"""Fallback: use Jina Reader."""
try:
resp = requests.get(
f"https://r.jina.ai/{url}",
headers={"Accept": "text/markdown"},
timeout=15,
)
resp.raise_for_status()
text = resp.text
# Check if content is usable
if len(text.strip()) < 100 or "Sign in" in text[:200]:
return ReadResult(
title="LinkedIn",
content=(
f"⚠️ LinkedIn 页面需要登录才能完整查看。\n\n"
f"URL: {url}\n\n"
"完整功能需安装 linkedin-scraper-mcp\n"
" pip install linkedin-scraper-mcp\n"
" uvx linkedin-scraper-mcp --login\n"
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
),
url=url,
platform="linkedin",
)
return ReadResult(
title=text[:100] if text else url,
content=text,
url=url,
platform="linkedin",
)
except Exception:
return ReadResult(
title="LinkedIn",
content=(
f"⚠️ 无法读取此 LinkedIn 页面: {url}\n\n"
"提示:\n"
"- LinkedIn 需要登录才能查看大部分内容\n"
"- 安装 linkedin-scraper-mcp 解锁完整功能\n"
"- 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
),
url=url,
platform="linkedin",
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
limit = kwargs.get("limit", 10)
# Try MCP search first
if _mcporter_has_linkedin():
try:
return await self._search_mcp(query, limit)
except Exception:
pass
# Fallback to Exa
from agent_reach.channels.exa_search import ExaSearchChannel
exa = ExaSearchChannel()
return await exa.search(f"site:linkedin.com {query}", config=config, limit=limit)
async def _search_mcp(self, query: str, limit: int) -> List[SearchResult]:
"""Search LinkedIn via MCP."""
safe_q = query.replace('"', '\\"')
# Try job search first (most common use case)
try:
out = _mcporter_call(
f'linkedin.search_jobs(keywords: "{safe_q}")',
timeout=60,
)
results = self._parse_search_results(out, "job")
if results:
return results[:limit]
except Exception:
pass
# Try people search
try:
out = _mcporter_call(
f'linkedin.search_people(keywords: "{safe_q}")',
timeout=60,
)
results = self._parse_search_results(out, "people")
if results:
return results
except Exception:
pass
return []
def _parse_search_results(self, text: str, result_type: str) -> List[SearchResult]:
"""Parse MCP search output into SearchResults."""
import json
results = []
try:
data = json.loads(text)
items = data if isinstance(data, list) else data.get("results", data.get("jobs", []))
for item in items:
if isinstance(item, dict):
title = item.get("title") or item.get("name") or item.get("headline", "")
url = item.get("url") or item.get("link", "")
snippet = item.get("description") or item.get("company", "")
results.append(SearchResult(
title=title,
url=url,
snippet=snippet[:200] if snippet else "",
))
except (json.JSONDecodeError, KeyError):
# Try line-by-line parsing
pass
return results
def _extract_title(self, text: str) -> str:
"""Extract a title from MCP output."""
for line in text.split("\n"):
line = line.strip()
if line and not line.startswith(("{", "[", "#", "http")):
return line[:80]
return ""
+13 -165
View File
@@ -1,178 +1,26 @@
# -*- coding: utf-8 -*-
"""Reddit — via Reddit JSON API + optional proxy.
Backend: Reddit public JSON API (append .json to any URL)
Swap to: any Reddit access method
"""
"""Reddit — check if proxy and credentials are configured."""
import os
import requests
from urllib.parse import urlparse
from .base import Channel, ReadResult
from .base import Channel
class RedditChannel(Channel):
name = "reddit"
description = "Reddit 帖子和评论"
backends = ["Reddit JSON API"]
tier = 2
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
backends = ["JSON API", "Exa"]
tier = 1
def can_handle(self, url: str) -> bool:
domain = urlparse(url).netloc.lower()
return "reddit.com" in domain or "redd.it" in domain
from urllib.parse import urlparse
d = urlparse(url).netloc.lower()
return "reddit.com" in d or "redd.it" in d
def check(self, config=None):
proxy = config.get("reddit_proxy") if config else None
has_bot = bool(os.environ.get("REDDIT_CLIENT_ID"))
if proxy and has_bot:
return "ok", "完整可用(代理 + OAuth Bot"
elif proxy:
return "ok", "代理已配置,可读取帖子。配置 REDDIT_CLIENT_ID/SECRET 可解锁高级搜索和发帖"
elif has_bot:
return "warn", "OAuth Bot 已配置,但服务器直连可能被封。配个代理更稳定:agent-reach configure proxy URL"
else:
return "off", "搜索用 Exa 免费可用。读帖子需配个代理:agent-reach configure proxy URL"
async def read(self, url: str, config=None) -> ReadResult:
proxy = config.get("reddit_proxy") if config else None
proxies = {"http": proxy, "https": proxy} if proxy else None
# Clean URL: remove query params, trailing slash, then add .json
parsed = urlparse(url)
clean_path = parsed.path.rstrip("/")
# Remove trailing .json if already present (avoid double .json)
if clean_path.endswith(".json"):
clean_path = clean_path[:-5]
json_url = f"https://www.reddit.com{clean_path}.json"
try:
resp = requests.get(
json_url,
headers={"User-Agent": self.USER_AGENT},
proxies=proxies,
params={"limit": 50},
timeout=15,
)
resp.raise_for_status()
except requests.exceptions.HTTPError as e:
status = e.response.status_code if e.response is not None else 0
if status in (403, 429):
return ReadResult(
title="Reddit",
content="⚠️ Reddit blocked this request (403 Forbidden). "
"Reddit blocks most server IPs.\n"
"Fix: agent-reach configure proxy http://user:pass@ip:port\n"
"Cheap option: https://www.webshare.io ($1/month)\n\n"
"Alternatively, search Reddit via Exa (free, no proxy needed): "
"agent-reach search-reddit \"your query\"",
url=url,
platform="reddit",
)
raise
data = resp.json()
# Subreddit listing page: /r/sub/, /r/sub/hot, /r/sub/new, /r/sub/top
if isinstance(data, dict) and data.get("kind") == "Listing":
return self._parse_listing(data, url)
if isinstance(data, list) and len(data) >= 1:
# Post page: [post_listing, comments_listing]
post = data[0]["data"]["children"][0]["data"]
title = post.get("title", "")
author = post.get("author", "")
selftext = post.get("selftext", "")
score = post.get("score", 0)
subreddit = post.get("subreddit", "")
# Extract comments
comments_text = ""
if len(data) >= 2:
comments_text = self._extract_comments(data[1])
content = selftext
if comments_text:
content += f"\n\n---\n## Comments\n{comments_text}"
return ReadResult(
title=title,
content=content,
url=url,
author=f"u/{author}",
platform="reddit",
extra={"subreddit": subreddit, "score": score},
)
raise ValueError(f"Could not parse Reddit response for: {url}")
def _parse_listing(self, data: dict, url: str) -> ReadResult:
"""Parse a subreddit listing (hot/new/top/rising)."""
children = data.get("data", {}).get("children", [])
# Extract subreddit name and sort from URL
parsed = urlparse(url)
path_parts = [p for p in parsed.path.strip("/").split("/") if p]
subreddit = path_parts[1] if len(path_parts) >= 2 else "reddit"
sort_type = path_parts[2] if len(path_parts) >= 3 else "hot"
lines = []
for i, child in enumerate(children, 1):
if child.get("kind") != "t3":
continue
post = child.get("data", {})
title = post.get("title", "")
author = post.get("author", "")
score = post.get("score", 0)
num_comments = post.get("num_comments", 0)
permalink = post.get("permalink", "")
post_url = post.get("url", "")
is_self = post.get("is_self", False)
lines.append(f"### {i}. {title}")
lines.append(f"👤 u/{author} · ⬆ {score} · 💬 {num_comments}")
if not is_self and post_url:
lines.append(f"🔗 {post_url}")
lines.append(f"📎 https://www.reddit.com{permalink}")
# Add selftext preview (first 200 chars)
selftext = post.get("selftext", "")
if selftext:
preview = selftext[:200].replace("\n", " ")
if len(selftext) > 200:
preview += "..."
lines.append(f"> {preview}")
lines.append("")
content = "\n".join(lines) if lines else "No posts found."
return ReadResult(
title=f"r/{subreddit}{sort_type}",
content=content,
url=url,
platform="reddit",
extra={"subreddit": subreddit, "sort": sort_type, "count": len(children)},
proxy = (config.get("reddit_proxy") if config else None) or os.environ.get("REDDIT_PROXY")
if proxy:
return "ok", "代理已配置,可读取帖子。搜索走 Exa"
return "warn", (
"无代理。服务器 IP 可能被 Reddit 封锁。配置代理:\n"
" agent-reach configure proxy http://user:pass@ip:port"
)
def _extract_comments(self, comments_data: dict, depth: int = 0, max_depth: int = 3) -> str:
"""Recursively extract comments."""
lines = []
children = comments_data.get("data", {}).get("children", [])
for child in children:
if child.get("kind") != "t1":
continue
data = child.get("data", {})
author = data.get("author", "[deleted]")
body = data.get("body", "")
score = data.get("score", 0)
indent = " " * depth
lines.append(f"{indent}**u/{author}** ({score} points):")
lines.append(f"{indent}{body}")
lines.append("")
# Recurse into replies
if depth < max_depth and data.get("replies") and isinstance(data["replies"], dict):
lines.append(self._extract_comments(data["replies"], depth + 1, max_depth))
return "\n".join(lines)
+9 -45
View File
@@ -1,13 +1,7 @@
# -*- coding: utf-8 -*-
"""RSS feeds — via feedparser (free, pip dependency).
"""RSS — check if feedparser is available."""
Backend: feedparser (https://github.com/kurtmckee/feedparser)
Swap to: any RSS parser
"""
import feedparser
from urllib.parse import urlparse
from .base import Channel, ReadResult
from .base import Channel
class RSSChannel(Channel):
@@ -17,41 +11,11 @@ class RSSChannel(Channel):
tier = 0
def can_handle(self, url: str) -> bool:
lower = url.lower()
domain = urlparse(url).netloc.lower()
return (lower.endswith(".xml") or "/rss" in lower or "/feed" in lower
or "/atom" in lower or "rss" in domain)
return any(x in url.lower() for x in ["/feed", "/rss", ".xml", "atom"])
async def read(self, url: str, config=None) -> ReadResult:
feed = feedparser.parse(url)
if feed.bozo and not feed.entries:
raise ValueError(f"Failed to parse RSS feed: {url}")
if not feed.entries:
raise ValueError(f"No entries in RSS feed: {url}")
# Return latest entry
entry = feed.entries[0]
content = entry.get("summary", "") or entry.get("description", "")
# If multiple entries, summarize all
if len(feed.entries) > 1:
lines = [f"# {feed.feed.get('title', 'RSS Feed')}\n"]
for i, e in enumerate(feed.entries[:20], 1):
title = e.get("title", "Untitled")
link = e.get("link", "")
summary = e.get("summary", "")[:200]
lines.append(f"## {i}. {title}")
lines.append(f"🔗 {link}")
if summary:
lines.append(summary)
lines.append("")
content = "\n".join(lines)
return ReadResult(
title=feed.feed.get("title", entry.get("title", url)),
content=content,
url=url,
platform="rss",
)
def check(self, config=None):
try:
import feedparser
return "ok", "可读取 RSS/Atom 源"
except ImportError:
return "off", "feedparser 未安装。安装:pip install feedparser"
+23 -193
View File
@@ -1,209 +1,39 @@
# -*- coding: utf-8 -*-
"""Twitter/X — via bird CLI (free) or Jina Reader fallback.
Backend: bird (@steipete/bird npm package) for search/timeline
Jina Reader for single tweets
Swap to: any Twitter access tool
"""
"""Twitter/X — check if xreach CLI is available."""
import shutil
import subprocess
from urllib.parse import urlparse
from .base import Channel, ReadResult, SearchResult
from typing import List
import requests
def _bird_cmd():
"""Find bird CLI binary."""
return shutil.which("bird") or shutil.which("birdx")
def _bird_env(config=None):
"""Build env dict with Twitter cookies for bird CLI."""
import os
env = os.environ.copy()
if config:
auth_token = config.get("twitter_auth_token")
ct0 = config.get("twitter_ct0")
if auth_token:
env["AUTH_TOKEN"] = auth_token
if ct0:
env["CT0"] = ct0
return env
from .base import Channel
class TwitterChannel(Channel):
name = "twitter"
description = "Twitter/X 推文"
backends = ["bird", "Jina Reader"]
tier = 0 # Single tweet reading is zero-config
backends = ["xreach CLI"]
tier = 1
def can_handle(self, url: str) -> bool:
domain = urlparse(url).netloc.lower()
return "x.com" in domain or "twitter.com" in domain
from urllib.parse import urlparse
d = urlparse(url).netloc.lower()
return "x.com" in d or "twitter.com" in d
def check(self, config=None):
# Basic reading always works (Jina fallback)
if _bird_cmd():
return "ok", "搜索、时间线、发推全部可用"
return "ok", "可读取推文。安装 bird + 配置 Cookie 可解锁搜索和发推"
async def read(self, url: str, config=None) -> ReadResult:
# Try bird first
bird = _bird_cmd()
if bird:
return await self._read_bird(url, bird, config)
# Fallback: Jina Reader
return await self._read_jina(url)
async def _read_bird(self, url: str, bird: str, config=None) -> ReadResult:
result = subprocess.run(
[bird, "read", url],
capture_output=True, timeout=30,
encoding='utf-8', errors='replace',
env=_bird_env(config),
)
if result.returncode != 0:
return await self._read_jina(url)
text = result.stdout.strip()
# Extract author from first line
author = ""
lines = text.split("\n")
if lines and lines[0].startswith("@"):
author = lines[0].split()[0]
return ReadResult(
title=text[:100],
content=text,
url=url,
author=author,
platform="twitter",
)
async def _read_jina(self, url: str) -> ReadResult:
try:
resp = requests.get(
f"https://r.jina.ai/{url}",
headers={"Accept": "text/markdown"},
timeout=15,
xreach = shutil.which("xreach")
if not xreach:
return "warn", (
"xreach CLI 未安装。搜索可通过 Exa 替代。安装:\n"
" npm install -g xreach-cli"
)
resp.raise_for_status()
text = resp.text
# Detect unusable Jina responses for X/Twitter (JS-required pages)
unusable_indicators = [
"page doesn", # "this page doesn't exist" (handles both ' and ')
"miss what", # "Don't miss what's happening"
"Something went wrong. Try reloading",
"Log in](", # Markdown link: [Log in](...)
]
if any(indicator in text for indicator in unusable_indicators):
return ReadResult(
title="Twitter/X",
content="⚠️ Could not read this tweet.\n"
"The tweet may have been deleted, or the account is private.\n\n"
"Tips:\n"
"- Make sure the URL is correct\n"
"- Try: bird read <url> (if bird CLI is installed)\n"
"- For protected tweets, configure Twitter cookies: "
"agent-reach configure twitter-cookies AUTH_TOKEN CT0",
url=url,
platform="twitter",
)
title = text[:100] if text else url
return ReadResult(
title=title,
content=text,
url=url,
platform="twitter",
try:
r = subprocess.run(
[xreach, "auth", "check"], capture_output=True,
encoding="utf-8", errors="replace", timeout=10
)
if r.returncode == 0:
return "ok", "完整可用(读取、搜索推文)"
return "warn", (
"xreach CLI 已安装但未配置 Cookie。运行:\n"
" agent-reach configure twitter-cookies \"auth_token=xxx; ct0=yyy\""
)
except Exception:
return ReadResult(
title="Twitter/X",
content="⚠️ Could not read this tweet.\n"
"The tweet may have been deleted, or the account is private.\n\n"
"Tips:\n"
"- Make sure the URL is correct\n"
"- Try: bird read <url> (if bird CLI is installed)\n"
"- For protected tweets, configure Twitter cookies: "
"agent-reach configure twitter-cookies AUTH_TOKEN CT0",
url=url,
platform="twitter",
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
limit = kwargs.get("limit", 10)
bird = _bird_cmd()
if bird:
return await self._search_bird(query, limit, bird, config)
# Fallback to Exa
return await self._search_exa(query, limit, config)
async def _search_bird(self, query: str, limit: int, bird: str, config=None) -> List[SearchResult]:
try:
result = subprocess.run(
[bird, "search", query, "-n", str(limit)],
capture_output=True, timeout=30,
encoding='utf-8', errors='replace',
env=_bird_env(config),
)
if result.returncode != 0:
return []
return self._parse_bird_output(result.stdout)
except (subprocess.TimeoutExpired, FileNotFoundError):
return []
def _parse_bird_output(self, text: str) -> List[SearchResult]:
"""Parse bird text output into SearchResults."""
results = []
current = {}
text_lines = []
for line in text.strip().split("\n"):
line = line.strip()
if line.startswith(""):
if current:
current["text"] = "\n".join(text_lines).strip()
results.append(SearchResult(
title=current.get("text", "")[:80],
url=current.get("url", ""),
snippet=current.get("text", ""),
author=current.get("author", ""),
date=current.get("date", ""),
))
current = {}
text_lines = []
continue
if line.startswith("@") and line.endswith(":") and "(" in line:
current["author"] = line.split()[0]
continue
if line.startswith("date:"):
current["date"] = line[5:].strip()
continue
if line.startswith("url:"):
current["url"] = line[4:].strip()
continue
if current is not None:
text_lines.append(line)
if current and text_lines:
current["text"] = "\n".join(text_lines).strip()
results.append(SearchResult(
title=current.get("text", "")[:80],
url=current.get("url", ""),
snippet=current.get("text", ""),
author=current.get("author", ""),
date=current.get("date", ""),
))
return results
async def _search_exa(self, query: str, limit: int, config=None) -> List[SearchResult]:
from agent_reach.channels.exa_search import ExaSearchChannel
exa = ExaSearchChannel()
return await exa.search(f"site:x.com {query}", config=config, limit=limit)
return "warn", "xreach CLI 已安装但连接失败"
+7 -39
View File
@@ -1,49 +1,17 @@
# -*- coding: utf-8 -*-
"""Web pages — via Jina Reader API (free, no config needed).
"""Web — any URL via Jina Reader. Always available."""
Backend: Jina Reader (https://r.jina.ai)
Swap to: Firecrawl, Trafilatura, or any other reader API
"""
import requests
from .base import Channel, ReadResult
from .base import Channel
class WebChannel(Channel):
name = "web"
description = "网页(任意 URL"
backends = ["Jina Reader API"]
description = "任意网页"
backends = ["Jina Reader"]
tier = 0
JINA_URL = "https://r.jina.ai/"
def can_handle(self, url: str) -> bool:
# Fallback — handles any URL not matched by other channels
return True
return True # Fallback — handles any URL
async def read(self, url: str, config=None) -> ReadResult:
resp = requests.get(
f"{self.JINA_URL}{url}",
headers={"Accept": "text/markdown"},
timeout=15,
)
resp.raise_for_status()
text = resp.text
# Extract title from first markdown heading
title = url
for line in text.split("\n"):
line = line.strip()
if line.startswith("# "):
title = line[2:].strip()
break
if line.startswith("Title:"):
title = line[6:].strip()
break
return ReadResult(
title=title,
content=text,
url=url,
platform="web",
)
def check(self, config=None):
return "ok", "通过 Jina Reader 读取任意网页(curl https://r.jina.ai/URL"
+57
View File
@@ -0,0 +1,57 @@
# -*- coding: utf-8 -*-
"""WeChat Official Account articles — read and search.
Read: wechat-article-for-ai (Camoufox stealth browser)
Search: miku_ai (Sogou WeChat search)
"""
import shutil
import subprocess
from .base import Channel
class WeChatChannel(Channel):
name = "wechat"
description = "微信公众号文章"
backends = ["wechat-article-for-ai (Camoufox)", "miku_ai (搜狗搜索)"]
tier = 2
def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
d = urlparse(url).netloc.lower()
return "mp.weixin.qq.com" in d or "weixin.qq.com" in d
def check(self, config=None):
has_read = False
has_search = False
try:
import camoufox # noqa: F401
has_read = True
except ImportError:
pass
try:
import miku_ai # noqa: F401
has_search = True
except ImportError:
pass
if has_read and has_search:
return "ok", "完整可用(搜索 + 阅读公众号文章)"
elif has_read:
return "ok", "可阅读公众号文章(URL → Markdown)。安装 miku_ai 可解锁搜索:pip install miku_ai"
elif has_search:
return "warn", (
"可搜索公众号文章但无法阅读全文。安装阅读工具:\n"
" pip install camoufox[geoip] markdownify beautifulsoup4 httpx mcp"
)
else:
return "off", (
"需要安装微信公众号工具:\n"
" # 阅读(URL → Markdown):\n"
" pip install camoufox[geoip] markdownify beautifulsoup4 httpx mcp\n"
" # 搜索(关键词 → 文章列表):\n"
" pip install miku_ai\n"
" 详见 https://github.com/bzd6661/wechat-article-for-ai"
)
+46 -144
View File
@@ -1,16 +1,31 @@
# -*- coding: utf-8 -*-
"""XiaoHongShu (小红书) — via mcporter + xiaohongshu MCP server.
"""XiaoHongShu -- check if mcporter + xiaohongshu MCP is available."""
Backend: xiaohongshu-mcp server (internal API, reliable)
Requires: mcporter CLI + xiaohongshu MCP server running
"""
import json
import platform
import shutil
import subprocess
from urllib.parse import urlparse
from .base import Channel, ReadResult, SearchResult
from typing import List, Optional
from .base import Channel
def _is_arm64() -> bool:
"""Detect ARM64 architecture (e.g. Apple Silicon)."""
machine = platform.machine().lower()
return machine in ("arm64", "aarch64")
def _docker_run_hint() -> str:
"""Return the docker run command, with --platform flag for ARM64."""
if _is_arm64():
return (
" docker run -d --name xiaohongshu-mcp -p 18060:18060 "
"--platform linux/amd64 xpzouying/xiaohongshu-mcp\n"
" # ARM64 also: build from source: "
"https://github.com/xpzouying/xiaohongshu-mcp"
)
return (
" docker run -d --name xiaohongshu-mcp -p 18060:18060 "
"xpzouying/xiaohongshu-mcp"
)
class XiaoHongShuChannel(Channel):
@@ -19,154 +34,41 @@ class XiaoHongShuChannel(Channel):
backends = ["xiaohongshu-mcp"]
tier = 2
def _mcporter_ok(self) -> bool:
"""Check if mcporter + xiaohongshu MCP is available."""
if not shutil.which("mcporter"):
return False
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
return "xiaohongshu" in r.stdout
except Exception:
return False
def _call(self, expr: str, timeout: int = 30) -> str:
r = subprocess.run(
["mcporter", "call", expr],
capture_output=True, text=True, timeout=timeout,
)
if r.returncode != 0:
raise RuntimeError(r.stderr or r.stdout)
return r.stdout
# ── Channel interface ──
def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
d = urlparse(url).netloc.lower()
return "xiaohongshu.com" in d or "xhslink.com" in d
def check(self, config=None):
if not shutil.which("mcporter"):
mcporter = shutil.which("mcporter")
if not mcporter:
return "off", (
"需要 mcporter + xiaohongshu-mcp。安装步骤:\n"
" 1. npm install -g mcporter\n"
" 2. docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
" 2. " + _docker_run_hint().strip() + "\n"
" 3. mcporter config add xiaohongshu http://localhost:18060/mcp\n"
" 详见 https://github.com/xpzouying/xiaohongshu-mcp"
)
if not self._mcporter_ok():
return "off", (
"mcporter 已装但小红书 MCP 未配置。运行:\n"
" docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
" mcporter config add xiaohongshu http://localhost:18060/mcp"
)
try:
out = self._call("xiaohongshu.check_login_status()", timeout=10)
if "已登录" in out or "logged" in out.lower():
r = subprocess.run(
[mcporter, "config", "list"], capture_output=True,
encoding="utf-8", errors="replace", timeout=5
)
if "xiaohongshu" not in r.stdout:
return "off", (
"mcporter 已装但小红书 MCP 未配置。运行:\n"
+ _docker_run_hint() + "\n"
" mcporter config add xiaohongshu http://localhost:18060/mcp"
)
except Exception:
return "off", "mcporter 连接异常"
try:
r = subprocess.run(
[mcporter, "call", "xiaohongshu.check_login_status()"],
capture_output=True, encoding="utf-8", errors="replace", timeout=10
)
if "已登录" in r.stdout or "logged" in r.stdout.lower():
return "ok", "完整可用(阅读、搜索、发帖、评论、点赞)"
return "warn", "MCP 已连接但未登录,需扫码登录"
except Exception:
return "warn", "MCP 连接异常,检查 xiaohongshu-mcp 服务是否在运行"
async def read(self, url: str, config=None) -> ReadResult:
if not self._mcporter_ok():
return ReadResult(
title="XiaoHongShu",
content=(
"⚠️ 小红书需要 mcporter + xiaohongshu-mcp 才能使用。\n\n"
"安装步骤:\n"
"1. npm install -g mcporter\n"
"2. docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
"3. mcporter config add xiaohongshu http://localhost:18060/mcp\n"
"4. 运行 agent-reach doctor 检查状态\n\n"
"详见 https://github.com/xpzouying/xiaohongshu-mcp"
),
url=url, platform="xiaohongshu",
)
note_id = self._extract_note_id(url)
if not note_id:
return ReadResult(
title="XiaoHongShu",
content=f"⚠️ 无法从 URL 提取笔记 ID: {url}",
url=url, platform="xiaohongshu",
)
# Step 1: get xsec_token from feeds
xsec_token = self._find_token(note_id)
if not xsec_token:
return ReadResult(
title="XiaoHongShu",
content=(
f"⚠️ 无法获取笔记 {note_id} 的访问令牌。\n"
"小红书需要 xsec_token 才能读取笔记详情。\n"
"请先通过搜索找到这篇笔记,或直接使用搜索功能。"
),
url=url, platform="xiaohongshu",
)
# Step 2: get detail
out = self._call(
f'xiaohongshu.get_feed_detail(feed_id: "{note_id}", xsec_token: "{xsec_token}")',
timeout=15,
)
return ReadResult(
title=self._extract_title(out) or f"XHS {note_id}",
content=out.strip(),
url=url, platform="xiaohongshu",
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
if not self._mcporter_ok():
raise ValueError(
"小红书搜索需要 mcporter + xiaohongshu-mcp。\n"
"安装: npm install -g mcporter && mcporter config add xiaohongshu http://localhost:18060/mcp"
)
limit = kwargs.get("limit", 10)
safe_q = query.replace('"', '\\"')
out = self._call(f'xiaohongshu.search_feeds(keyword: "{safe_q}")', timeout=30)
results = []
try:
data = json.loads(out)
for item in data.get("feeds", [])[:limit]:
card = item.get("noteCard", {})
user = card.get("user", {})
interact = card.get("interactInfo", {})
results.append(SearchResult(
title=card.get("displayTitle", ""),
url=f"https://www.xiaohongshu.com/explore/{item.get('id', '')}",
snippet=f"👤 {user.get('nickname', '')} · ❤ {interact.get('likedCount', '0')}",
score=0,
))
except (json.JSONDecodeError, KeyError):
pass
return results
# ── Helpers ──
def _extract_note_id(self, url: str) -> str:
parts = urlparse(url).path.strip("/").split("/")
return parts[-1] if parts else ""
def _find_token(self, note_id: str) -> Optional[str]:
"""Try to find xsec_token for a note from feeds."""
try:
out = self._call("xiaohongshu.list_feeds()", timeout=15)
data = json.loads(out)
for feed in data.get("feeds", []):
if feed.get("id") == note_id:
return feed.get("xsecToken", "")
except Exception:
pass
return None
def _extract_title(self, text: str) -> str:
for line in text.split("\n"):
line = line.strip()
if line and not line.startswith(("{", "[", "#", "http")):
return line[:80]
return ""
+24 -109
View File
@@ -1,125 +1,40 @@
# -*- coding: utf-8 -*-
"""YouTube — via yt-dlp (video info, subtitles, and search).
"""YouTube — check if yt-dlp is available with JS runtime."""
Backend: yt-dlp (https://github.com/yt-dlp/yt-dlp)
Supports: read (info + subtitles), search (ytsearch)
"""
import json
import os
import shutil
import subprocess
import tempfile
from pathlib import Path
from urllib.parse import urlparse
from .base import Channel, ReadResult, SearchResult
from typing import List
from .base import Channel
class YouTubeChannel(Channel):
name = "youtube"
description = "YouTube 视频字幕"
description = "YouTube 视频字幕"
backends = ["yt-dlp"]
requires_tools = ["yt-dlp"]
tier = 0
def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
d = urlparse(url).netloc.lower()
return "youtube.com" in d or "youtu.be" in d
async def read(self, url: str, config=None) -> ReadResult:
def check(self, config=None):
if not shutil.which("yt-dlp"):
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
with tempfile.TemporaryDirectory() as tmpdir:
info = self._get_info(url)
title = info.get("title", url)
author = info.get("uploader", "")
transcript = self._get_subtitles(url, tmpdir)
if not transcript:
transcript = f"[Video: {title}]\n[No subtitles available.]"
return ReadResult(
title=title, content=transcript, url=url,
author=author, platform="youtube",
extra={
"duration": info.get("duration_string"),
"view_count": info.get("view_count"),
"upload_date": info.get("upload_date"),
},
return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
# Check JS runtime
has_js = shutil.which("deno") or shutil.which("node")
if not has_js:
return "warn", (
"yt-dlp 已安装但缺少 JS runtimeYouTube 必须)。\n"
" 安装 Node.js 或 deno,然后运行:agent-reach install"
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
"""Search YouTube via yt-dlp's ytsearch."""
if not shutil.which("yt-dlp"):
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
limit = kwargs.get("limit", 10)
try:
r = subprocess.run(
["yt-dlp", "--dump-json", "--flat-playlist",
f"ytsearch{limit}:{query}"],
capture_output=True, text=True, timeout=30,
)
results = []
for line in r.stdout.strip().split("\n"):
if not line.strip():
continue
try:
d = json.loads(line)
vid = d.get("id", "")
results.append(SearchResult(
title=d.get("title", ""),
url=f"https://youtube.com/watch?v={vid}" if vid else "",
snippet=(
f"👤 {d.get('channel', '?')} · "
f"{d.get('duration_string', '?')} · "
f"👁 {d.get('view_count', '?')}"
),
extra={
"channel": d.get("channel"),
"duration": d.get("duration_string"),
"view_count": d.get("view_count"),
},
))
except json.JSONDecodeError:
continue
return results
except subprocess.TimeoutExpired:
return []
def _get_info(self, url: str) -> dict:
try:
r = subprocess.run(
["yt-dlp", "--dump-json", "--no-download", url],
capture_output=True, text=True, timeout=30,
)
if r.returncode == 0:
return json.loads(r.stdout)
except (subprocess.TimeoutExpired, json.JSONDecodeError):
pass
return {}
def _get_subtitles(self, url: str, tmpdir: str) -> str:
try:
subprocess.run(
["yt-dlp", "--write-auto-sub", "--write-sub",
"--sub-lang", "en,zh-Hans,zh",
"--skip-download", "--sub-format", "vtt",
"-o", f"{tmpdir}/%(id)s.%(ext)s", url],
capture_output=True, text=True, timeout=30,
)
for f in Path(tmpdir).glob("*.vtt"):
text = f.read_text(errors="replace")
lines = []
for line in text.split("\n"):
line = line.strip()
if not line or line.startswith("WEBVTT") or "-->" in line or line.isdigit():
continue
if line not in lines[-1:]:
lines.append(line)
return "\n".join(lines)
except subprocess.TimeoutExpired:
pass
return ""
# Check yt-dlp config for --js-runtimes
ytdlp_config = os.path.expanduser("~/.config/yt-dlp/config")
has_deno = shutil.which("deno")
if not has_deno and os.path.exists(ytdlp_config):
with open(ytdlp_config, "r") as f:
if "--js-runtimes" not in f.read():
return "warn", (
"yt-dlp 已安装但未配置 JS runtime。运行:\n"
" mkdir -p ~/.config/yt-dlp && echo '--js-runtimes node' >> ~/.config/yt-dlp/config"
)
return "ok", "可提取视频信息和字幕"
+429 -344
View File
@@ -1,35 +1,43 @@
# -*- coding: utf-8 -*-
"""
Agent Reach CLI — command-line interface.
Agent Reach CLI — installer, doctor, and configuration tool.
Usage:
agent-reach read <url>
agent-reach search <query>
agent-reach search-reddit <query> [--sub <subreddit>]
agent-reach search-github <query> [--lang <language>]
agent-reach search-twitter <query>
agent-reach setup
agent-reach install --env=auto
agent-reach doctor
agent-reach version
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
agent-reach setup
"""
import sys
import asyncio
import argparse
import json
import os
# Fix Windows console encoding — emoji/CJK characters crash on cp936/cp1252
if sys.platform == 'win32':
import io
if hasattr(sys.stdout, 'buffer'):
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', errors='replace')
if hasattr(sys.stderr, 'buffer'):
sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8', errors='replace')
import time
from agent_reach import __version__
def _ensure_utf8_console():
"""Best-effort Windows console UTF-8 setup for CLI runtime only."""
if sys.platform != "win32":
return
# Avoid interfering with pytest/captured streams.
if os.environ.get("PYTEST_CURRENT_TEST"):
return
if not getattr(sys.stdout, "isatty", lambda: False)():
return
try:
import io
if hasattr(sys.stdout, "buffer"):
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")
if hasattr(sys.stderr, "buffer"):
sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding="utf-8", errors="replace")
except Exception:
# Do not crash CLI just because encoding patch failed.
pass
def _configure_logging(verbose: bool = False):
"""Suppress loguru output unless --verbose is set."""
from loguru import logger
@@ -39,6 +47,8 @@ def _configure_logging(verbose: bool = False):
def main():
_ensure_utf8_console()
parser = argparse.ArgumentParser(
prog="agent-reach",
description="👁️ Give your AI Agent eyes to see the entire internet",
@@ -47,63 +57,6 @@ def main():
parser.add_argument("--version", action="version", version=f"Agent Reach v{__version__}")
sub = parser.add_subparsers(dest="command", help="Available commands")
# ── read ──
p_read = sub.add_parser("read", help="Read content from a URL")
p_read.add_argument("url", help="URL to read")
p_read.add_argument("--json", dest="as_json", action="store_true", help="Output as JSON")
# ── search ──
p_search = sub.add_parser("search", help="Search the web (Exa)")
p_search.add_argument("query", nargs="+", help="Search query")
p_search.add_argument("-n", "--num", type=int, default=5, help="Number of results")
# ── search-reddit ──
p_sr = sub.add_parser("search-reddit", help="Search Reddit")
p_sr.add_argument("query", nargs="+", help="Search query")
p_sr.add_argument("--sub", help="Subreddit filter")
p_sr.add_argument("-n", "--num", type=int, default=10, help="Number of results")
# ── search-github ──
p_sg = sub.add_parser("search-github", help="Search GitHub")
p_sg.add_argument("query", nargs="+", help="Search query")
p_sg.add_argument("--lang", help="Language filter")
p_sg.add_argument("-n", "--num", type=int, default=5, help="Number of results")
# ── search-twitter ──
p_st = sub.add_parser("search-twitter", help="Search Twitter")
p_st.add_argument("query", nargs="+", help="Search query")
p_st.add_argument("-n", "--num", type=int, default=10, help="Number of results")
# ── search-youtube ──
p_sy = sub.add_parser("search-youtube", help="Search YouTube")
p_sy.add_argument("query", nargs="+", help="Search query")
p_sy.add_argument("-n", "--num", type=int, default=5, help="Number of results")
# ── search-bilibili ──
p_sb = sub.add_parser("search-bilibili", help="Search Bilibili")
p_sb.add_argument("query", nargs="+", help="Search query")
p_sb.add_argument("-n", "--num", type=int, default=5, help="Number of results")
# ── search-xhs ──
p_sx = sub.add_parser("search-xhs", help="Search XiaoHongShu")
p_sx.add_argument("query", nargs="+", help="Search query")
p_sx.add_argument("-n", "--num", type=int, default=10, help="Number of results")
# ── search-instagram ──
p_si = sub.add_parser("search-instagram", help="Search Instagram")
p_si.add_argument("query", nargs="+", help="Search query")
p_si.add_argument("-n", "--num", type=int, default=10, help="Number of results")
# ── search-linkedin ──
p_sl = sub.add_parser("search-linkedin", help="Search LinkedIn")
p_sl.add_argument("query", nargs="+", help="Search query")
p_sl.add_argument("-n", "--num", type=int, default=10, help="Number of results")
# ── search-bosszhipin ──
p_sbz = sub.add_parser("search-bosszhipin", help="Search Boss直聘")
p_sbz.add_argument("query", nargs="+", help="Search query")
p_sbz.add_argument("-n", "--num", type=int, default=10, help="Number of results")
# ── setup ──
sub.add_parser("setup", help="Interactive configuration wizard")
@@ -122,8 +75,7 @@ def main():
p_conf = sub.add_parser("configure", help="Set a config value or auto-extract from browser")
p_conf.add_argument("key", nargs="?", default=None,
choices=["proxy", "github-token", "groq-key",
"twitter-cookies", "youtube-cookies",
"instagram-cookies"],
"twitter-cookies", "youtube-cookies"],
help="What to configure (omit if using --from-browser)")
p_conf.add_argument("value", nargs="*", help="The value(s) to set")
p_conf.add_argument("--from-browser", metavar="BROWSER",
@@ -133,6 +85,13 @@ def main():
# ── doctor ──
sub.add_parser("doctor", help="Check platform availability")
# ── uninstall ──
p_uninstall = sub.add_parser("uninstall", help="Remove all Agent Reach config, tokens, and skill files")
p_uninstall.add_argument("--dry-run", action="store_true",
help="Show what would be removed without making any changes")
p_uninstall.add_argument("--keep-config", action="store_true",
help="Remove skill files only, keep ~/.agent-reach/ config and tokens")
# ── check-update ──
sub.add_parser("check-update", help="Check for new versions and changes")
@@ -167,10 +126,8 @@ def main():
_cmd_install(args)
elif args.command == "configure":
_cmd_configure(args)
elif args.command == "read":
asyncio.run(_cmd_read(args))
elif args.command.startswith("search"):
asyncio.run(_cmd_search(args))
elif args.command == "uninstall":
_cmd_uninstall(args)
# ── Command handlers ────────────────────────────────
@@ -345,6 +302,7 @@ def _install_system_deps():
import shutil
import subprocess
import platform
import tempfile
print("🔧 Checking system dependencies...")
@@ -356,15 +314,25 @@ def _install_system_deps():
os_type = platform.system().lower()
if os_type == "linux":
try:
# Official GitHub method for Linux
cmds = [
"curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg 2>/dev/null",
'echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null',
"apt-get update -qq 2>/dev/null",
"apt-get install -y -qq gh 2>/dev/null",
]
for cmd in cmds:
subprocess.run(cmd, shell=True, capture_output=True, timeout=60)
# Official GitHub apt source setup without invoking a shell.
keyring_path = "/usr/share/keyrings/githubcli-archive-keyring.gpg"
list_path = "/etc/apt/sources.list.d/github-cli.list"
arch = subprocess.run(
["dpkg", "--print-architecture"],
capture_output=True, encoding="utf-8", errors="replace", timeout=10,
).stdout.strip() or "amd64"
subprocess.run(
["curl", "-fsSL", "https://cli.github.com/packages/githubcli-archive-keyring.gpg", "-o", keyring_path],
capture_output=True, timeout=60,
)
repo_line = (
f"deb [arch={arch} signed-by={keyring_path}] "
"https://cli.github.com/packages stable main\n"
)
with open(list_path, "w", encoding="utf-8") as f:
f.write(repo_line)
subprocess.run(["apt-get", "update", "-qq"], capture_output=True, timeout=60)
subprocess.run(["apt-get", "install", "-y", "-qq", "gh"], capture_output=True, timeout=60)
if shutil.which("gh"):
print(" ✅ gh CLI installed")
else:
@@ -392,10 +360,24 @@ def _install_system_deps():
else:
print(" 📥 Installing Node.js...")
try:
# Use NodeSource for quick install
# Use NodeSource setup script without invoking a shell pipeline.
with tempfile.NamedTemporaryFile(delete=False, suffix=".sh") as tf:
script_path = tf.name
subprocess.run(
"curl -fsSL https://deb.nodesource.com/setup_22.x | bash - 2>/dev/null && apt-get install -y -qq nodejs 2>/dev/null",
shell=True, capture_output=True, timeout=120,
["curl", "-fsSL", "https://deb.nodesource.com/setup_22.x", "-o", script_path],
capture_output=True, timeout=60,
)
subprocess.run(
["bash", script_path],
capture_output=True, timeout=120,
)
try:
os.unlink(script_path)
except Exception:
pass
subprocess.run(
["apt-get", "install", "-y", "-qq", "nodejs"],
capture_output=True, timeout=120,
)
if shutil.which("node"):
print(" ✅ Node.js installed")
@@ -404,41 +386,56 @@ def _install_system_deps():
except Exception:
print(" ⚠️ Node.js install failed. Try: apt install nodejs npm, or nvm install 22, or download from https://nodejs.org")
# ── bird CLI (for Twitter search) ──
if shutil.which("bird") or shutil.which("birdx"):
print("bird CLI already installed")
# ── xreach CLI (for Twitter search) ──
if shutil.which("xreach"):
print("xreach CLI already installed")
else:
if shutil.which("npm"):
try:
subprocess.run(
["npm", "install", "-g", "@steipete/bird"],
capture_output=True, text=True, timeout=120,
["npm", "install", "-g", "xreach-cli"],
capture_output=True, encoding="utf-8", errors="replace", timeout=120,
)
if shutil.which("bird"):
print("bird CLI installed (Twitter search + timeline)")
if shutil.which("xreach"):
print("xreach CLI installed (Twitter search + timeline)")
else:
print("bird CLI install failed (optional — Twitter reading still works via Jina)")
print("xreach CLI install failed (optional — Twitter reading still works via Jina)")
except Exception:
print("bird CLI install failed (optional — Twitter reading still works via Jina)")
print("xreach CLI install failed (optional — Twitter reading still works via Jina)")
else:
print("bird CLI requires Node.js (optional — Twitter reading still works via Jina)")
print("xreach CLI requires Node.js (optional — Twitter reading still works via Jina)")
# ── instaloader (for Instagram) ──
if shutil.which("instaloader"):
print(" ✅ instaloader already installed")
else:
print(" 📥 Installing instaloader...")
try:
subprocess.run(
[sys.executable, "-m", "pip", "install", "instaloader"],
capture_output=True, text=True, timeout=120,
)
if shutil.which("instaloader"):
print(" ✅ instaloader installed (Instagram reading)")
else:
print(" ⬜ instaloader install failed (optional — try: pip install instaloader)")
except Exception:
print(" ⬜ instaloader install failed (optional — try: pip install instaloader)")
# ── undici (proxy support for Node.js fetch) ──
if shutil.which("npm"):
npm_root = subprocess.run(["npm", "root", "-g"], capture_output=True, encoding="utf-8", errors="replace", timeout=5).stdout.strip()
undici_path = os.path.join(npm_root, "undici", "index.js") if npm_root else ""
if os.path.exists(undici_path):
print(" ✅ undici already installed (Node.js proxy support)")
else:
try:
subprocess.run(["npm", "install", "-g", "undici"], capture_output=True, encoding="utf-8", errors="replace", timeout=60)
print(" ✅ undici installed (Node.js proxy support)")
except Exception:
print(" ⬜ undici install failed (optional — xreach may not work behind proxies)")
# ── yt-dlp JS runtime config (YouTube requires external JS runtime) ──
if shutil.which("node"):
ytdlp_config_dir = os.path.expanduser("~/.config/yt-dlp")
ytdlp_config = os.path.join(ytdlp_config_dir, "config")
needs_config = True
if os.path.exists(ytdlp_config):
with open(ytdlp_config, "r") as f:
if "--js-runtimes" in f.read():
needs_config = False
print(" ✅ yt-dlp JS runtime already configured")
if needs_config:
try:
os.makedirs(ytdlp_config_dir, exist_ok=True)
with open(ytdlp_config, "a") as f:
f.write("--js-runtimes node\n")
print(" ✅ yt-dlp configured to use Node.js as JS runtime (YouTube)")
except Exception:
print(" ⬜ Could not configure yt-dlp JS runtime (YouTube may not work)")
def _install_system_deps_safe():
@@ -450,8 +447,7 @@ def _install_system_deps_safe():
deps = [
("gh", ["gh"], "GitHub CLI", "https://cli.github.com — or: apt install gh / brew install gh"),
("node", ["node", "npm"], "Node.js", "https://nodejs.org — or: apt install nodejs npm"),
("bird", ["bird", "birdx"], "bird CLI (Twitter)", "npm install -g @steipete/bird"),
("instaloader", ["instaloader"], "instaloader (Instagram)", "pip install instaloader"),
("xreach", ["xreach"], "xreach CLI (Twitter)", "npm install -g xreach-cli"),
]
missing = []
@@ -481,8 +477,7 @@ def _install_system_deps_dryrun():
checks = [
("gh CLI", ["gh"], "apt install gh / brew install gh"),
("Node.js", ["node"], "curl NodeSource setup | bash + apt install nodejs"),
("bird CLI", ["bird", "birdx"], "npm install -g @steipete/bird"),
("instaloader", ["instaloader"], "pip install instaloader"),
("xreach CLI", ["xreach"], "npm install -g xreach-cli"),
]
for label, binaries, method in checks:
@@ -511,7 +506,7 @@ def _install_mcporter():
try:
subprocess.run(
["npm", "install", "-g", "mcporter"],
capture_output=True, text=True, timeout=120,
capture_output=True, encoding="utf-8", errors="replace", timeout=120,
)
if shutil.which("mcporter"):
print(" ✅ mcporter installed")
@@ -525,12 +520,12 @@ def _install_mcporter():
# Configure Exa MCP (free, no key needed)
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
["mcporter", "config", "list"], capture_output=True, encoding="utf-8", errors="replace", timeout=5
)
if "exa" not in r.stdout:
subprocess.run(
["mcporter", "config", "add", "exa", "https://mcp.exa.ai/mcp"],
capture_output=True, text=True, timeout=10,
capture_output=True, encoding="utf-8", errors="replace", timeout=10,
)
print(" ✅ Exa search configured (free, no API key needed)")
else:
@@ -541,7 +536,7 @@ def _install_mcporter():
# Check XiaoHongShu MCP (only if server is running)
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
["mcporter", "config", "list"], capture_output=True, encoding="utf-8", errors="replace", timeout=5
)
if "xiaohongshu" in r.stdout:
print(" ✅ XiaoHongShu MCP already configured")
@@ -552,7 +547,7 @@ def _install_mcporter():
requests.get("http://localhost:18060/", timeout=3)
subprocess.run(
["mcporter", "config", "add", "xiaohongshu", "http://localhost:18060/mcp"],
capture_output=True, text=True, timeout=10,
capture_output=True, encoding="utf-8", errors="replace", timeout=10,
)
print(" ✅ XiaoHongShu MCP auto-detected and configured")
except Exception:
@@ -611,7 +606,7 @@ def _detect_environment():
# systemd-detect-virt
try:
import subprocess
result = subprocess.run(["systemd-detect-virt"], capture_output=True, text=True, timeout=3)
result = subprocess.run(["systemd-detect-virt"], capture_output=True, encoding="utf-8", errors="replace", timeout=3)
if result.returncode == 0 and result.stdout.strip() != "none":
indicators += 1
except:
@@ -708,22 +703,41 @@ def _cmd_configure(args):
if auth_token and ct0:
config.set("twitter_auth_token", auth_token)
config.set("twitter_ct0", ct0)
print(f"✅ Twitter cookies configured!")
# Sync credentials to xreach's session.json so xreach auth check works
try:
import json
xfetch_dir = os.path.join(os.path.expanduser("~"), ".config", "xfetch")
os.makedirs(xfetch_dir, exist_ok=True)
session_path = os.path.join(xfetch_dir, "session.json")
session_data = {}
if os.path.exists(session_path):
with open(session_path, "r", encoding="utf-8") as sf:
session_data = json.load(sf)
session_data["authToken"] = auth_token
session_data["ct0"] = ct0
with open(session_path, "w", encoding="utf-8") as sf:
json.dump(session_data, sf, indent=2)
os.chmod(session_path, 0o600)
print("✅ Twitter cookies configured (synced to xreach)!")
except Exception as e:
print("✅ Twitter cookies configured!")
print(f"⚠️ Could not sync to xreach session.json: {e}")
print("Testing Twitter access...", end=" ")
try:
import subprocess
bird = shutil.which("bird") or shutil.which("birdx")
if not bird:
print("⚠️ bird CLI not installed. Run: npm install -g @steipete/bird")
xreach = shutil.which("xreach")
if not xreach:
print("⚠️ xreach CLI not installed. Run: npm install -g xreach-cli")
else:
import os
env = os.environ.copy()
env["AUTH_TOKEN"] = auth_token
env["CT0"] = ct0
result = subprocess.run(
[bird, "search", "test", "-n", "1"],
capture_output=True, text=True, timeout=15,
[xreach, "search", "test", "-n", "1"],
capture_output=True, encoding="utf-8", errors="replace", timeout=15,
env=env,
)
if result.returncode == 0 and result.stdout.strip():
@@ -751,8 +765,103 @@ def _cmd_configure(args):
config.set("groq_api_key", value)
print(f"✅ Groq key configured!")
elif args.key == "instagram-cookies":
_configure_instagram_cookies(value)
def _cmd_uninstall(args):
"""Remove all Agent Reach config, tokens, and skill files."""
import shutil
import subprocess
dry_run = args.dry_run
keep_config = args.keep_config
print()
print("Agent Reach Uninstaller")
print("=" * 40)
if dry_run:
print("DRY RUN — showing what would be removed (no changes)")
print()
removed_any = False
# ── 1. Config directory (~/.agent-reach/) ──
config_dir = os.path.expanduser("~/.agent-reach")
if not keep_config:
if os.path.isdir(config_dir):
if dry_run:
print(f"[dry-run] Would remove config directory: {config_dir}")
print(" (contains config.yaml with all tokens/cookies/API keys)")
else:
try:
shutil.rmtree(config_dir)
print(f" Removed config directory: {config_dir}")
removed_any = True
except Exception as e:
print(f" Could not remove {config_dir}: {e}")
else:
print(f" Config directory not found (already clean): {config_dir}")
else:
print(f" Skipping config directory (--keep-config): {config_dir}")
# ── 2. Skill files ──
skill_dirs = [
("~/.openclaw/skills/agent-reach", "OpenClaw"),
("~/.claude/skills/agent-reach", "Claude Code"),
("~/.agents/skills/agent-reach", "Agent"),
]
for skill_path_template, platform_name in skill_dirs:
skill_path = os.path.expanduser(skill_path_template)
if os.path.isdir(skill_path):
if dry_run:
print(f"[dry-run] Would remove {platform_name} skill: {skill_path}")
else:
try:
shutil.rmtree(skill_path)
print(f" Removed {platform_name} skill: {skill_path}")
removed_any = True
except Exception as e:
print(f" Could not remove {skill_path}: {e}")
# ── 3. mcporter MCP entries ──
if shutil.which("mcporter"):
for mcp_name in ("exa", "xiaohongshu"):
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, encoding="utf-8", errors="replace", timeout=10
)
if mcp_name in r.stdout:
if dry_run:
print(f"[dry-run] Would remove mcporter entry: {mcp_name}")
else:
subprocess.run(
["mcporter", "config", "remove", mcp_name],
capture_output=True, encoding="utf-8", errors="replace", timeout=10,
)
print(f" Removed mcporter entry: {mcp_name}")
removed_any = True
except Exception:
pass
# ── 4. Summary and optional steps ──
print()
if dry_run:
print("Dry run complete. No changes were made.")
print("Run without --dry-run to actually remove the above.")
else:
if removed_any:
print("Agent Reach data removed.")
else:
print("Nothing to remove — already clean.")
print()
print("Optional: remove the Agent Reach Python package itself:")
print(" pip uninstall agent-reach")
print()
print("Optional: remove tools installed by Agent Reach:")
print(" npm uninstall -g mcporter")
print(" npm uninstall -g xreach-cli")
print(" npm uninstall -g undici")
def _cmd_doctor():
@@ -763,43 +872,6 @@ def _cmd_doctor():
print(format_report(results))
def _parse_cookie_header(cookie_str: str) -> dict:
"""Parse Cookie-Editor 'Header String' format into a dict."""
cookies = {}
for part in cookie_str.split(";"):
part = part.strip()
if "=" in part:
k, v = part.split("=", 1)
cookies[k.strip()] = v.strip()
return cookies
def _configure_instagram_cookies(value: str):
"""Save Instagram cookies from Cookie-Editor Header String."""
from pathlib import Path
cookies = _parse_cookie_header(value)
if "sessionid" not in cookies:
print("❌ Cookie 里缺少 sessionid。")
print(" 确保你已登录 Instagram,然后用 Cookie-Editor 导出 Header String。")
print(' 格式: agent-reach configure instagram-cookies "sessionid=xxx; csrftoken=yyy; ..."')
return
cookie_dir = Path.home() / ".agent-reach"
cookie_dir.mkdir(parents=True, exist_ok=True)
cookie_file = cookie_dir / "instagram-cookies.txt"
cookie_file.write_text(value.strip())
cookie_file.chmod(0o600)
print(f"✅ Instagram cookies 已保存!")
print(f" sessionid: {cookies['sessionid'][:8]}...")
if "csrftoken" in cookies:
print(f" csrftoken: ✅")
if "ds_user_id" in cookies:
print(f" ds_user_id: {cookies['ds_user_id']}")
print(f" 文件: {cookie_file}")
def _cmd_setup():
from agent_reach.config import Config
@@ -809,29 +881,41 @@ def _cmd_setup():
print("=" * 40)
print()
# Step 1: Exa
print("【推荐】全网搜索 — Exa Search API")
print(" 免费 1000 次/月,注册地址: https://exa.ai")
current = config.get("exa_api_key")
if current:
print(f" 当前状态: ✅ 已配置 ({current[:8]}...)")
change = input(" 要更换吗?[y/N]: ").strip().lower()
if change != "y":
print()
else:
key = input(" EXA_API_KEY: ").strip()
if key:
config.set("exa_api_key", key)
print(" ✅ 已更新!")
print()
# Step 1: Exa (via mcporter, no API key required)
import shutil
import subprocess
print("【推荐】全网搜索 — Exa(通过 mcporter")
print(" 免费,无需 API Key")
if not shutil.which("mcporter"):
print(" 当前状态: ⬜ mcporter 未安装")
print(" 安装:npm install -g mcporter")
print(" 然后:mcporter config add exa https://mcp.exa.ai/mcp")
print()
else:
print(" 当前状态: ⬜ 未配置")
key = input(" EXA_API_KEY (回车跳过): ").strip()
if key:
config.set("exa_api_key", key)
print(" ✅ 全网搜索 + Reddit搜索 + Twitter搜索 已开启!")
else:
print(" ℹ️ 跳过。稍后可运行 agent-reach setup 配置")
try:
r = subprocess.run(
["mcporter", "config", "list"], capture_output=True, encoding="utf-8", errors="replace", timeout=10
)
if "exa" in r.stdout.lower():
print(" 当前状态: ✅ 已配置")
else:
print(" 当前状态: ⬜ 未配置")
setup_now = input(" 现在自动配置 Exa 吗?[Y/n]: ").strip().lower()
if setup_now in ("", "y", "yes"):
add_r = subprocess.run(
["mcporter", "config", "add", "exa", "https://mcp.exa.ai/mcp"],
capture_output=True, encoding="utf-8", errors="replace", timeout=10,
)
if add_r.returncode == 0:
print(" ✅ Exa 已配置")
else:
print(" ⚠️ 自动配置失败,请手动执行:")
print(" mcporter config add exa https://mcp.exa.ai/mcp")
except Exception:
print(" ⚠️ 无法检查 Exa 配置,请手动执行:")
print(" mcporter config add exa https://mcp.exa.ai/mcp")
print()
# Step 2: GitHub token
@@ -888,156 +972,160 @@ def _cmd_setup():
print()
async def _cmd_read(args):
from agent_reach.core import AgentReach
eyes = AgentReach()
try:
result = await eyes.read(args.url)
if args.as_json:
print(json.dumps(result, ensure_ascii=False, indent=2))
else:
print(f"\n📖 {result.get('title', 'Untitled')}")
print(f"🔗 {result.get('url', '')}")
if result.get("author"):
print(f"👤 {result['author']}")
print(f"\n{result.get('content', '')}")
except Exception as e:
error_str = str(e)
if "400" in error_str and "Bad Request" in error_str:
print(f"❌ Invalid URL: {args.url}", file=sys.stderr)
print(" Please provide a valid URL (e.g., https://example.com)", file=sys.stderr)
elif "ConnectionError" in type(e).__name__ or "Timeout" in type(e).__name__:
print(f"❌ Could not connect to: {args.url}", file=sys.stderr)
print(" Check your internet connection or the URL.", file=sys.stderr)
else:
print(f"❌ Error: {e}", file=sys.stderr)
sys.exit(1)
def _classify_update_error(exc):
"""Classify update-check errors for user-friendly diagnostics."""
import requests
if isinstance(exc, requests.exceptions.Timeout):
return "timeout"
if isinstance(exc, requests.exceptions.ConnectionError):
msg = str(exc).lower()
dns_markers = [
"name or service not known",
"temporary failure in name resolution",
"nodename nor servname",
"getaddrinfo failed",
"name resolution",
"dns",
]
if any(marker in msg for marker in dns_markers):
return "dns"
return "connection"
if isinstance(exc, requests.exceptions.HTTPError):
return "http"
return "unknown"
async def _cmd_search(args):
from agent_reach.core import AgentReach
eyes = AgentReach()
query = " ".join(args.query).strip()
num = args.num
def _update_error_text(kind):
"""Map internal error kinds to user-facing text."""
mapping = {
"timeout": "网络超时",
"dns": "DNS 解析失败",
"rate_limit": "GitHub API 速率限制",
"connection": "网络连接失败",
"server_error": "GitHub 服务暂时不可用",
"http": "HTTP 请求失败",
"unknown": "未知网络错误",
}
return mapping.get(kind, "请求失败")
if not query:
print("Please provide a search query.", file=sys.stderr)
sys.exit(1)
try:
if args.command == "search":
results = await eyes.search(query, num_results=num)
elif args.command == "search-reddit":
results = await eyes.search_reddit(query, subreddit=getattr(args, "sub", None), limit=num)
elif args.command == "search-github":
results = await eyes.search_github(query, language=getattr(args, "lang", None), limit=num)
elif args.command == "search-twitter":
results = await eyes.search_twitter(query, limit=num)
elif args.command == "search-youtube":
results = await eyes.search_youtube(query, limit=num)
elif args.command == "search-bilibili":
results = await eyes.search_bilibili(query, limit=num)
elif args.command == "search-xhs":
results = await eyes.search_xhs(query, limit=num)
elif args.command == "search-instagram":
results = await eyes.search_instagram(query, limit=num)
elif args.command == "search-linkedin":
results = await eyes.search_linkedin(query, limit=num)
elif args.command == "search-bosszhipin":
results = await eyes.search_bosszhipin(query, limit=num)
else:
print(f"Unknown command: {args.command}", file=sys.stderr)
sys.exit(1)
except Exception as e:
error_str = str(e)
if "401" in error_str or "Unauthorized" in error_str:
print("⚠️ Exa API key not configured or invalid.")
print("Get a free key at https://exa.ai (1000 searches/month free)")
print("Then run: agent-reach configure exa-key YOUR_KEY")
sys.exit(1)
elif "exa" in error_str.lower() or "api_key" in error_str.lower():
print("⚠️ Exa API key not configured.")
print("Get a free key at https://exa.ai")
print("Then run: agent-reach configure exa-key YOUR_KEY")
sys.exit(1)
else:
print(f"❌ Error: {e}", file=sys.stderr)
sys.exit(1)
def _classify_github_response_error(resp):
"""Classify non-200 GitHub responses that merit special handling."""
if resp is None:
return "unknown"
if resp.status_code == 429:
return "rate_limit"
if resp.status_code == 403:
remaining = resp.headers.get("X-RateLimit-Remaining", "")
if remaining == "0":
return "rate_limit"
try:
message = resp.json().get("message", "").lower()
if "rate limit" in message:
return "rate_limit"
except Exception:
pass
if 500 <= resp.status_code < 600:
return "server_error"
return None
if not results:
print("No results found.")
return
for i, r in enumerate(results, 1):
title = r.get("title") or r.get("name") or r.get("text", "")[:60]
url = r.get("url", "")
snippet = r.get("snippet") or r.get("description") or r.get("text", "")
print(f"\n{i}. {title}")
print(f" 🔗 {url}")
if snippet:
print(f" {snippet[:200]}")
# Extra info for GitHub
extra = r.get("extra", {})
if extra.get("stars"):
print(f"{extra['stars']} 🍴 {extra.get('forks', 0)} 📝 {extra.get('language', '')}")
def _github_get_with_retry(url, timeout=10, retries=3, sleeper=time.sleep):
"""GET GitHub API with retry/backoff and basic error classification."""
import requests
for attempt in range(1, retries + 1):
try:
resp = requests.get(url, timeout=timeout)
except requests.exceptions.RequestException as exc:
if attempt >= retries:
return None, _classify_update_error(exc), attempt
sleeper(2 ** (attempt - 1))
continue
err_kind = _classify_github_response_error(resp)
if err_kind in ("rate_limit", "server_error"):
if attempt >= retries:
return None, err_kind, attempt
delay = 2 ** (attempt - 1)
retry_after = resp.headers.get("Retry-After")
if err_kind == "rate_limit" and retry_after:
try:
delay = max(delay, float(retry_after))
except Exception:
pass
sleeper(delay)
continue
return resp, None, attempt
return None, "unknown", retries
def _cmd_check_update():
"""Check for newer versions on GitHub."""
import requests
from agent_reach import __version__
print(f"📦 当前版本: v{__version__}")
release_url = "https://api.github.com/repos/Panniantong/Agent-Reach/releases/latest"
commit_url = "https://api.github.com/repos/Panniantong/Agent-Reach/commits/main"
try:
# Fetch latest version from GitHub
resp = requests.get(
"https://api.github.com/repos/Panniantong/Agent-Reach/releases/latest",
timeout=10,
)
if resp.status_code == 200:
data = resp.json()
latest = data.get("tag_name", "").lstrip("v")
body = data.get("body", "")
if latest and latest != __version__:
print(f"🆕 最新版本: v{latest} ← 有更新!")
if body:
print()
print("更新内容:")
# Show first 20 lines of release notes
for line in body.strip().split("\n")[:20]:
print(f" {line}")
print()
print("更新命令:")
print(" pip install --upgrade https://github.com/Panniantong/agent-reach/archive/main.zip")
return "update_available"
else:
print(f"✅ 已是最新版本")
return "up_to_date"
else:
# No releases yet, fall back to comparing commit
resp2 = requests.get(
"https://api.github.com/repos/Panniantong/Agent-Reach/commits/main",
timeout=10,
)
if resp2.status_code == 200:
commit = resp2.json()
sha = commit.get("sha", "")[:7]
msg = commit.get("commit", {}).get("message", "").split("\n")[0]
date = commit.get("commit", {}).get("committer", {}).get("date", "")[:10]
print(f"🔍 最新提交: {sha} ({date}) {msg}")
print()
print("更新命令:")
print(" pip install --upgrade https://github.com/Panniantong/agent-reach/archive/main.zip")
return "unknown"
else:
print("⚠️ 无法检查更新(网络问题)")
return "error"
except Exception as e:
print(f"⚠️ 无法检查更新: {e}")
# Fetch latest release with retry/backoff.
resp, err, attempts = _github_get_with_retry(release_url, timeout=10, retries=3)
if err:
print(f"⚠️ 无法检查更新({_update_error_text(err)},已重试 {attempts} 次)")
return "error"
if resp.status_code == 200:
data = resp.json()
latest = data.get("tag_name", "").lstrip("v")
body = data.get("body", "")
if latest and latest != __version__:
print(f"🆕 最新版本: v{latest} ← 有更新!")
if body:
print()
print("更新内容:")
# Show first 20 lines of release notes
for line in body.strip().split("\n")[:20]:
print(f" {line}")
print()
print("更新命令:")
print(" pip install --upgrade https://github.com/Panniantong/agent-reach/archive/main.zip")
return "update_available"
print(f"✅ 已是最新版本")
return "up_to_date"
release_err = _classify_github_response_error(resp)
if release_err == "rate_limit":
print("⚠️ 无法检查更新(GitHub API 速率限制,请稍后重试)")
return "error"
# No releases yet, fall back to latest main commit.
resp2, err2, attempts2 = _github_get_with_retry(commit_url, timeout=10, retries=2)
if err2:
print(f"⚠️ 无法检查更新({_update_error_text(err2)},已重试 {attempts + attempts2} 次)")
return "error"
if resp2.status_code == 200:
commit = resp2.json()
sha = commit.get("sha", "")[:7]
msg = commit.get("commit", {}).get("message", "").split("\n")[0]
date = commit.get("commit", {}).get("committer", {}).get("date", "")[:10]
print(f"🔍 最新提交: {sha} ({date}) {msg}")
print()
print("更新命令:")
print(" pip install --upgrade https://github.com/Panniantong/agent-reach/archive/main.zip")
return "unknown"
commit_err = _classify_github_response_error(resp2)
if commit_err == "rate_limit":
print("⚠️ 无法检查更新(GitHub API 速率限制,请稍后重试)")
return "error"
print(f"⚠️ 无法检查更新(GitHub 返回 {resp2.status_code}")
return "error"
def _cmd_watch():
"""Quick health check + update check, designed for scheduled tasks.
@@ -1046,7 +1134,6 @@ def _cmd_watch():
"""
from agent_reach.config import Config
from agent_reach.doctor import check_all
import requests
from agent_reach import __version__
config = Config()
@@ -1068,20 +1155,18 @@ def _cmd_watch():
update_available = False
new_version = ""
release_body = ""
try:
resp = requests.get(
"https://api.github.com/repos/Panniantong/Agent-Reach/releases/latest",
timeout=10,
)
if resp.status_code == 200:
data = resp.json()
latest = data.get("tag_name", "").lstrip("v")
if latest and latest != __version__:
update_available = True
new_version = latest
release_body = data.get("body", "")
except Exception:
pass
resp, err, _attempts = _github_get_with_retry(
"https://api.github.com/repos/Panniantong/Agent-Reach/releases/latest",
timeout=10,
retries=2,
)
if not err and resp and resp.status_code == 200:
data = resp.json()
latest = data.get("tag_name", "").lstrip("v")
if latest and latest != __version__:
update_available = True
new_version = latest
release_body = data.get("body", "")
# Output
if not issues and not update_available:
+3 -3
View File
@@ -22,7 +22,7 @@ class Config:
FEATURE_REQUIREMENTS = {
"exa_search": ["exa_api_key"],
"reddit_proxy": ["reddit_proxy"],
"twitter_bird": ["twitter_auth_token", "twitter_ct0"],
"twitter_xreach": ["twitter_auth_token", "twitter_ct0"],
"groq_whisper": ["groq_api_key"],
"github_token": ["github_token"],
}
@@ -41,7 +41,7 @@ class Config:
def load(self):
"""Load config from YAML file."""
if self.config_path.exists():
with open(self.config_path, "r") as f:
with open(self.config_path, "r", encoding="utf-8") as f:
self.data = yaml.safe_load(f) or {}
else:
self.data = {}
@@ -49,7 +49,7 @@ class Config:
def save(self):
"""Save config to YAML file."""
self._ensure_dir()
with open(self.config_path, "w") as f:
with open(self.config_path, "w", encoding="utf-8") as f:
yaml.dump(self.data, f, default_flow_style=False, allow_unicode=True)
# Restrict permissions — config may contain credentials
try:
+16 -116
View File
@@ -1,126 +1,36 @@
# -*- coding: utf-8 -*-
"""
AgentReach the unified entry point.
AgentReach installer, doctor, and configuration tool.
Pure glue: routes URLs to the right channel, routes searches to the right engine.
Every channel is a thin wrapper around an external tool. Swap any backend anytime.
Agent Reach helps AI agents install and configure upstream platform tools
(xreach CLI, yt-dlp, mcporter, gh CLI, etc.). After installation, agents
call the upstream tools directly no wrapper layer needed.
Usage:
from agent_reach import AgentReach
from agent_reach.doctor import check_all, format_report
from agent_reach.config import Config
eyes = AgentReach()
content = await eyes.read("https://github.com/openai/gpt-4")
results = await eyes.search("AI agent framework")
config = Config()
results = check_all(config)
print(format_report(results))
"""
import asyncio
from typing import Any, Dict, List, Optional
from typing import Dict, Optional
from agent_reach.config import Config
from agent_reach.channels import get_channel_for_url, get_channel, get_all_channels
class AgentReach:
"""Give your AI Agent eyes to see the entire internet."""
"""Give your AI Agent eyes to see the entire internet.
This class provides health-check functionality.
For reading/searching, use the upstream tools directly
(see SKILL.md for commands).
"""
def __init__(self, config: Optional[Config] = None):
self.config = config or Config()
# ── Reading ─────────────────────────────────────────
async def read(self, url: str) -> Dict[str, Any]:
"""
Read content from any URL. Auto-detects platform.
Supported: Web, GitHub, Reddit, Twitter, YouTube,
Bilibili, RSS, and more.
Returns:
Dict with title, content, url, author, platform, etc.
"""
if not url.startswith(("http://", "https://")):
url = f"https://{url}"
channel = get_channel_for_url(url)
result = await channel.read(url, config=self.config)
return result.to_dict()
async def read_batch(self, urls: List[str]) -> List[Dict[str, Any]]:
"""Read multiple URLs concurrently."""
tasks = [self.read(url) for url in urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
return [r for r in results if not isinstance(r, Exception)]
def detect_platform(self, url: str) -> str:
"""Detect what platform a URL belongs to."""
channel = get_channel_for_url(url)
return channel.name
# ── Searching ───────────────────────────────────────
async def search(self, query: str, num_results: int = 5) -> List[Dict[str, Any]]:
"""Semantic web search via Exa."""
ch = get_channel("exa_search")
results = await ch.search(query, config=self.config, limit=num_results)
return [r.to_dict() for r in results]
async def search_reddit(self, query: str, subreddit: Optional[str] = None, limit: int = 10) -> List[Dict[str, Any]]:
"""Search Reddit via Exa (bypasses IP blocks)."""
ch = get_channel("exa_search")
q = f"site:reddit.com/r/{subreddit} {query}" if subreddit else f"site:reddit.com {query}"
results = await ch.search(q, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_github(self, query: str, language: Optional[str] = None, limit: int = 5) -> List[Dict[str, Any]]:
"""Search GitHub repositories."""
ch = get_channel("github")
results = await ch.search(query, config=self.config, language=language, limit=limit)
return [r.to_dict() for r in results]
async def search_twitter(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
"""Search Twitter. Uses bird CLI if available, else Exa."""
ch = get_channel("twitter")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_youtube(self, query: str, limit: int = 5) -> List[Dict[str, Any]]:
"""Search YouTube via yt-dlp."""
ch = get_channel("youtube")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_bilibili(self, query: str, limit: int = 5) -> List[Dict[str, Any]]:
"""Search Bilibili. Tries yt-dlp first, falls back to Exa."""
ch = get_channel("bilibili")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_xhs(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
"""Search XiaoHongShu via mcporter."""
ch = get_channel("xiaohongshu")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_instagram(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
"""Search Instagram via Exa."""
ch = get_channel("instagram")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_linkedin(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
"""Search LinkedIn via MCP or Exa."""
ch = get_channel("linkedin")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_bosszhipin(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
"""Search Boss直聘 via MCP or Exa."""
ch = get_channel("bosszhipin")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
# ── Health ──────────────────────────────────────────
def doctor(self) -> Dict[str, dict]:
"""Check all channel availability."""
from agent_reach.doctor import check_all
@@ -130,13 +40,3 @@ class AgentReach:
"""Get formatted health report."""
from agent_reach.doctor import check_all, format_report
return format_report(check_all(self.config))
# ── Sync wrappers ───────────────────────────────────
def read_sync(self, url: str) -> Dict[str, Any]:
"""Synchronous version of read()."""
return asyncio.run(self.read(url))
def search_sync(self, query: str, num_results: int = 5) -> List[Dict[str, Any]]:
"""Synchronous version of search()."""
return asyncio.run(self.search(query, num_results))
+15
View File
@@ -74,4 +74,19 @@ def format_report(results: Dict[str, dict]) -> str:
if ok_count < total:
lines.append("运行 `agent-reach setup` 解锁更多渠道")
# Security check: config file permissions (Unix only)
import os
import stat
import sys
config_path = Config.CONFIG_DIR / "config.yaml"
if config_path.exists() and sys.platform != "win32":
try:
mode = config_path.stat().st_mode
if mode & (stat.S_IRGRP | stat.S_IROTH):
lines.append("")
lines.append("⚠️ 安全提示:config.yaml 权限过宽(其他用户可读)")
lines.append(" 修复:chmod 600 ~/.agent-reach/config.yaml")
except OSError:
pass
return "\n".join(lines)
+2 -2
View File
@@ -22,8 +22,8 @@ mcporter config add exa https://mcp.exa.ai/mcp
### 3. 验证
```bash
agent-reach doctor | grep "Web Search"
agent-reach search "test" -n 1
agent-reach doctor | grep "Search"
mcporter call 'exa.web_search_exa(query: "test", numResults: 1)'
```
## 需要用户手动做的步骤
+56 -46
View File
@@ -1,67 +1,77 @@
# Twitter 高级功能配置指南(bird CLI
# Twitter 高级功能配置指南(xreach CLI
## 功能说明
基础 Twitter 功能(搜索+读单条推文)无需配置,开箱即用。
Twitter 基础阅读通过 Jina Reader 免费可用,无需配置。
高级功能需要 bird CLI
- 查看用户时间线
- 深度搜索(更精确、更多结果)
- 读取完整线程(thread
- 查看关注列表推文
高级功能需要 xreach CLI
bird 是免费开源工具(npm 包 @steipete/bird),但需要你的 Twitter 账号 cookie。
- 搜索推文(`xreach search`
- 读取完整推文和对话链(`xreach tweet``xreach thread`
- 用户时间线(`xreach tweets`
## Agent 可自动完成的步骤
xreach 是免费开源工具(npm 包 xreach-cli),但需要你的 Twitter 账号 cookie。
## 快速配置
1. 检查 xreach 是否安装:
1. 检查 bird 是否安装:
```bash
which bird && echo "installed" || echo "not installed"
which xreach && echo "installed" || echo "not installed"
```
2. 安装 bird
2. 安装 xreach
```bash
npm install -g @steipete/bird
npm install -g xreach-cli
```
3. 测试(需要先配置 cookie
3. 测试是否配置好
```bash
AUTH_TOKEN="xxx" CT0="yyy" bird search "test" -n 1
AUTH_TOKEN="xxx" CT0="yyy" xreach search "test" -n 1
```
4. 用 agent-reach 配置 cookie推荐)
## 获取 CookieCookie-Editor 方式,推荐)
1. 安装 [Cookie-Editor](https://cookie-editor.com/) 浏览器扩展
2. 登录 x.com
3. 点击 Cookie-Editor 图标 → Export → 复制全部
4. 运行配置命令:
```bash
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
agent-reach configure twitter-cookies "粘贴的 cookie JSON"
```
## 需要用户手动做的步骤
这会自动提取 `auth_token``ct0`,并写入环境变量。
请告诉用户:
## 手动设置 Cookie
> Twitter 高级功能需要你的 Twitter 账号 cookie(完全免费)。
>
> **最简单的方式:**
> 1. 安装 Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm)
> 2. 打开 https://x.com 并确保已登录
> 3. 点击 Cookie-Editor 插件图标 → Export → Header String
> 4. 把导出的内容发给我
>
> **手动方式:**
> 1. 用 Chrome 打开 https://x.com 并确保你已登录
> 2. 按 **F12** 打开开发者工具(Mac 按 Cmd+Option+I
> 3. 点击顶部的 **Application**(应用)标签
> 4. 左侧找到 **Cookies** → **https://x.com**
> 5. 在列表中找到以下两个值,双击复制:
> - **auth_token** — 一串字母数字
> - **ct0** — 一串字母数字
> 6. 把这两个值发给我
>
> ⚠️ 这些 cookie 让我能以你的身份读取推文(只读)。我不会发推、点赞或做任何操作。
> ⚠️ cookie 大约 1-3 个月会过期,届时需要重新导出。
如果你已经知道 `auth_token``ct0`
## Agent 收到 cookie 后的操作
1. 安装 xreach(如果没装):`npm install -g xreach-cli`
1. 安装 bird(如果没装):`npm install -g @steipete/bird`
2. 配置 cookie`agent-reach configure twitter-cookies "粘贴的内容"`
3. 测试:运行 `agent-reach doctor` 确认 Twitter 状态
4. 反馈:"✅ Twitter 高级功能已开启!现在可以搜索推文、查看时间线了。"
5. 如果失败:"❌ Cookie 无效或已过期,请重新导出。"
2. 设置环境变量:
```bash
export AUTH_TOKEN="你的auth_token"
export CT0="你的ct0"
```
3. 测试:
```bash
xreach search "test" --auth-token "$AUTH_TOKEN" --ct0 "$CT0" -n 1
```
## 代理配置
> xreach CLI 内置代理支持,通过 `--proxy` 参数传入:
```bash
xreach search "test" --auth-token "$AUTH_TOKEN" --ct0 "$CT0" --proxy "http://user:pass@host:port"
```
也支持代理轮换文件:
```bash
xreach search "test" --auth-token "$AUTH_TOKEN" --ct0 "$CT0" --proxy-file proxies.txt
```
+1 -1
View File
@@ -18,7 +18,7 @@ playwright install chromium
3. 安装完成后测试:
```bash
agent-reach read "https://mp.weixin.qq.com/s/一个测试链接"
curl -s "https://r.jina.ai/https://mp.weixin.qq.com/s/一个测试链接" -H "Accept: text/markdown"
```
## 需要用户手动做的步骤
+20
View File
@@ -67,5 +67,25 @@ docker run -d \
**Q: 服务器上小红书提示 IP 风险?**
A: 加代理参数 `-e XHS_PROXY=http://user:pass@ip:port`,推荐住宅代理。
**Q: Docker 镜像不支持 ARM64 / Apple Silicon**
A: 上游镜像暂无 ARM64 版本,两种解决办法:
方法一:使用 Rosetta 模拟运行(推荐,最简单)
```bash
docker run -d \
--name xiaohongshu-mcp \
-p 18060:18060 \
--platform linux/amd64 \
xpzouying/xiaohongshu-mcp
```
方法二:从源码编译原生 ARM64 版本
```bash
git clone https://github.com/xpzouying/xiaohongshu-mcp
cd xiaohongshu-mcp
docker build -t xiaohongshu-mcp .
docker run -d --name xiaohongshu-mcp -p 18060:18060 xiaohongshu-mcp
```
**Q: 我不想用 Docker**
A: 可以从源码编译:https://github.com/xpzouying/xiaohongshu-mcp
+5 -39
View File
@@ -1,10 +1,11 @@
# -*- coding: utf-8 -*-
"""
Agent Reach MCP Server expose all capabilities as MCP tools.
Agent Reach MCP Server expose doctor/status as MCP tool.
Run: python -m agent_reach.integrations.mcp_server
8 tools for any MCP-compatible AI Agent.
Agent Reach is an installer + doctor tool. For actual reading/searching,
agents should call upstream tools directly (xreach, yt-dlp, mcporter, etc.).
"""
import asyncio
@@ -35,50 +36,15 @@ def create_server():
@server.list_tools()
async def list_tools():
return [
Tool(name="read_url",
description="Read content from any URL. Supports: web, GitHub, Reddit, Twitter, YouTube, Bilibili, RSS.",
inputSchema={"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}),
Tool(name="read_batch",
description="Read multiple URLs concurrently.",
inputSchema={"type": "object", "properties": {"urls": {"type": "array", "items": {"type": "string"}}}, "required": ["urls"]}),
Tool(name="detect_platform",
description="Detect what platform a URL belongs to.",
inputSchema={"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}),
Tool(name="search",
description="Semantic web search via Exa.",
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "num_results": {"type": "integer", "default": 5}}, "required": ["query"]}),
Tool(name="search_reddit",
description="Search Reddit posts.",
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "subreddit": {"type": "string"}, "limit": {"type": "integer", "default": 10}}, "required": ["query"]}),
Tool(name="search_github",
description="Search GitHub repositories.",
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "language": {"type": "string"}, "limit": {"type": "integer", "default": 5}}, "required": ["query"]}),
Tool(name="search_twitter",
description="Search Twitter/X posts.",
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "limit": {"type": "integer", "default": 10}}, "required": ["query"]}),
Tool(name="get_status",
description="Get Agent Reach status: which channels are active.",
description="Get Agent Reach status: which channels are installed and active.",
inputSchema={"type": "object", "properties": {}}),
]
@server.call_tool()
async def call_tool(name: str, arguments: dict):
try:
if name == "read_url":
result = await eyes.read(arguments["url"])
elif name == "read_batch":
result = await eyes.read_batch(arguments["urls"])
elif name == "detect_platform":
result = eyes.detect_platform(arguments["url"])
elif name == "search":
result = await eyes.search(arguments["query"], arguments.get("num_results", 5))
elif name == "search_reddit":
result = await eyes.search_reddit(arguments["query"], arguments.get("subreddit"), arguments.get("limit", 10))
elif name == "search_github":
result = await eyes.search_github(arguments["query"], arguments.get("language"), arguments.get("limit", 5))
elif name == "search_twitter":
result = await eyes.search_twitter(arguments["query"], arguments.get("limit", 10))
elif name == "get_status":
if name == "get_status":
result = eyes.doctor_report()
else:
result = f"Unknown tool: {name}"
-65
View File
@@ -1,65 +0,0 @@
---
name: agent-reach
description: "Give your AI Agent eyes to see the entire internet. Search + read 10+ platforms with one install."
install: pip install git+https://github.com/Panniantong/agent-reach.git
---
# Agent Reach — 给 Agent 装上眼睛 👁️
安装后你就能搜索和阅读全网内容。
## 安装
```bash
pip install git+https://github.com/Panniantong/agent-reach.git
```
## 装完直接能用的(零配置)
- `agent-reach read <URL>` — 读取任意网页、GitHub、B站、YouTube、RSS
- `agent-reach search-github "query"` — 搜索 GitHub 仓库
- `agent-reach doctor` — 查看所有平台状态
## 解锁更多功能
运行 `agent-reach configure` 进行交互式配置,或者读取以下指南:
| 功能 | 指南文件 |
|------|---------|
| 全网搜索 + Reddit搜索 + Twitter搜索 | `agent_reach/guides/setup-exa.md` |
| Reddit 完整阅读 | `agent_reach/guides/setup-reddit.md` |
| Twitter 高级(时间线/线程) | `agent_reach/guides/setup-twitter.md` |
| 小红书 | `agent_reach/guides/setup-xiaohongshu.md` |
| 微信公众号 | `agent_reach/guides/setup-wechat.md` |
| 视频语音转文字 | `agent_reach/guides/setup-groq.md` |
每个指南里有详细的步骤说明,你(Agent)可以照着做,只有需要人类操作的部分(登录、复制 key)才需要问用户。
## MCP Server
如果你的 Agent 平台支持 MCP:
```bash
pip install agent-reach[mcp]
python -m agent_reach.integrations.mcp_server
```
提供 8 个工具:read_url, read_batch, detect_platform, search, search_reddit, search_github, search_twitter, get_status
## Python API
```python
from agent_reach import AgentReach
import asyncio
eyes = AgentReach()
# 读取
result = asyncio.run(eyes.read("https://github.com/openai/gpt-4"))
# 搜索
results = asyncio.run(eyes.search("AI agent framework"))
# 健康检查
print(eyes.doctor_report())
```
+280 -55
View File
@@ -1,87 +1,312 @@
---
name: agent-reach
description: >
Give your AI agent eyes to see the entire internet. Read and search across
Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, RSS, and any web page
— all from a single CLI. Use when: (1) reading content from URLs (tweets, Reddit posts,
articles, videos), (2) searching across platforms (web, Twitter, Reddit, GitHub, YouTube,
Bilibili, XiaoHongShu), (3) checking channel health or updating Agent Reach.
Triggers: "search Twitter/Reddit/YouTube", "read this URL", "find posts about",
"搜索", "读取", "查一下", "看看这个链接".
Give your AI agent eyes to see the entire internet. Install and configure
upstream tools for Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu,
Douyin, LinkedIn, Boss直聘, WeChat (微信公众号), RSS, and any web page — then call them directly.
Use when: (1) setting up platform access tools for the first time,
(2) checking which platforms are available,
(3) user asks to configure/enable a platform channel.
Triggers: "帮我配", "帮我添加", "帮我安装", "agent reach", "install channels",
"configure twitter", "enable reddit".
---
# Agent Reach
Read and search the internet across 9+ platforms via unified CLI.
Install and configure upstream tools for 12+ platforms. After setup, call them directly — no wrapper layer.
## Setup
First check if agent-reach is installed:
```bash
agent-reach doctor
```
If command not found, install it:
```bash
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
agent-reach install --env=auto
agent-reach doctor
```
`install` auto-detects your environment and installs all dependencies (Node.js, mcporter, bird CLI, gh CLI). Read the output and run `agent-reach doctor` to see what's active.
`install` auto-detects your environment and installs core dependencies (Node.js, mcporter, xreach CLI, gh CLI, yt-dlp, feedparser). Run `doctor` to see what's active.
For channels that need user input, ask the user. See the full setup guide:
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
## Commands
### Read any URL
```bash
agent-reach read <url>
agent-reach read <url> --json # structured output
```
Handles: tweets, Reddit posts, articles, YouTube (transcripts), GitHub repos, etc.
### Search
```bash
agent-reach search "query" # web search (Exa)
agent-reach search-twitter "query" # Twitter/X
agent-reach search-reddit "query" # Reddit (--sub <subreddit>)
agent-reach search-github "query" # GitHub (--lang <language>)
agent-reach search-youtube "query" # YouTube
agent-reach search-bilibili "query" # Bilibili (B站)
agent-reach search-xhs "query" # XiaoHongShu (小红书)
```
All search commands support `-n <count>` for number of results.
### Management
## Management
```bash
agent-reach doctor # channel status overview
agent-reach watch # quick health + update check (for scheduled tasks)
agent-reach watch # quick health + update check
agent-reach check-update # check for new versions
```
### Configure channels
## Configure channels
```bash
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
agent-reach configure proxy http://user:pass@ip:port
agent-reach configure --from-browser chrome # auto-extract cookies
agent-reach configure --from-browser chrome # auto-extract cookies from local browser
```
## Channel Status Tiers
## Configuring a channel ("帮我配 XXX")
- **Tier 0 (zero config):** Web, YouTube, RSS, Twitter (read-only via Jina)
- **Tier 1 (free setup):** Exa web search (mcporter required)
- **Tier 2 (user config):** Twitter search (cookie), Reddit full (proxy), GitHub (token), Bilibili (proxy), XiaoHongShu (MCP)
When a user asks to configure/enable any channel:
Run `agent-reach doctor` to see which channels are active.
1. Run `agent-reach doctor`
2. Find the channel — it shows status (✅/⚠️/⬜) and **what to do next**
3. Execute what you can automatically (install packages, start services)
4. For human-required steps (paste cookies), tell the user what to do
5. Run `agent-reach doctor` again to verify
## Tips
**Do NOT memorize per-channel steps.** Always rely on `doctor` output.
- Always try `agent-reach read <url>` first for any URL — it auto-detects the platform
- For Twitter cookies, recommend the user install [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome extension
- Reddit and Bilibili block server IPs — suggest a residential proxy (~$1/month) if on a server
- If a channel breaks, run `agent-reach doctor` to diagnose
### Cookie 导入(所有需要登录的平台通用)
> ⚠️ **重要提醒:** 使用 Cookie 登录的平台存在封号风险,请提醒用户务必使用**专用小号**。
所有需要 Cookie 的平台(Twitter、小红书等),**优先使用 Cookie-Editor 导入**
1. 用户在自己的浏览器登录对应平台
2. 安装 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome 插件
3. 点击插件 → Export → Header String
4. 把导出的字符串发给 Agent
本地电脑用户也可以用 `agent-reach configure --from-browser chrome` 一键自动提取。
扫码登录是备选方案(仅限本地有浏览器的情况),Cookie-Editor 更简单可靠。
### Other human actions
- **Proxy:** Reddit/Bilibili/XiaoHongShu may block server IPs — suggest a residential proxy if on a server
---
## Using Upstream Tools Directly
After `agent-reach install`, call the upstream tools directly.
> **Note:** `agent-reach` is an installer and config tool — it does NOT have `read`, `search`, or content-fetching commands. Use the upstream tools below instead.
### Twitter/X (xreach CLI)
```bash
# Search tweets
xreach search "query" --json -n 10
# Read a specific tweet
xreach tweet https://x.com/user/status/123 --json
# Read a user's timeline
xreach tweets @username --json -n 20
```
### YouTube (yt-dlp)
> ⚠️ yt-dlp 需要 JS runtime 才能下载 YouTube。`agent-reach install` 会自动配置 Node.js 作为 runtime。
> 如果遇到 "Sign in to confirm you're not a bot",是 IP 被 YouTube 反爬,换代理或加 cookies。
```bash
# Get video metadata
yt-dlp --dump-json "https://www.youtube.com/watch?v=xxx"
# Download subtitles only
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --skip-download -o "/tmp/%(id)s" "URL"
# Then read the .vtt file
# Search (yt-dlp ytsearch)
yt-dlp --dump-json "ytsearch5:query"
# If "no JS runtime" warning: ensure Node.js is installed, then run:
# mkdir -p ~/.config/yt-dlp && echo "--js-runtimes node" >> ~/.config/yt-dlp/config
```
### Bilibili (yt-dlp)
> ⚠️ 服务器 IP 可能被 Bilibili 拦截(412 错误)。建议通过代理访问,或加 `--cookies-from-browser chrome`
```bash
# Get video metadata
yt-dlp --dump-json "https://www.bilibili.com/video/BVxxx"
# Download subtitles
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --convert-subs vtt --skip-download -o "/tmp/%(id)s" "URL"
# If blocked (412 / login required):
yt-dlp --cookies-from-browser chrome --dump-json "URL"
```
### Reddit (JSON API)
```bash
# Read a subreddit
curl -s "https://www.reddit.com/r/python/hot.json?limit=10" -H "User-Agent: agent-reach/1.0"
# Read a post with comments
curl -s "https://www.reddit.com/r/python/comments/POST_ID.json" -H "User-Agent: agent-reach/1.0"
# Search
curl -s "https://www.reddit.com/search.json?q=query&limit=10" -H "User-Agent: agent-reach/1.0"
```
Note: On servers, Reddit may block your IP. Use proxy or search via Exa instead.
### 小红书 / XiaoHongShu (mcporter + xiaohongshu-mcp)
> ⚠️ 需要登录。使用 Cookie-Editor 导入 cookies 或扫码登录。
```bash
# 搜索笔记
mcporter call 'xiaohongshu.search_feeds(keyword: "query")'
# 获取笔记详情(含评论)
mcporter call 'xiaohongshu.get_feed_detail(feed_id: "xxx", xsec_token: "yyy")'
# 获取全部评论
mcporter call 'xiaohongshu.get_feed_detail(feed_id: "xxx", xsec_token: "yyy", load_all_comments: true)'
# 发布图文笔记
mcporter call 'xiaohongshu.publish_content(title: "标题", content: "正文", images: ["/path/to/img.jpg"], tags: ["美食"])'
# 发布视频笔记
mcporter call 'xiaohongshu.publish_with_video(title: "标题", content: "正文", video: "/path/to/video.mp4", tags: ["vlog"])'
```
其他功能(点赞、收藏、评论、用户主页等):`npx mcporter list xiaohongshu`
### 抖音 / Douyin (mcporter + douyin-mcp-server)
```bash
# 解析抖音视频信息(分享链接 → 标题、作者、无水印视频URL等)
mcporter call 'douyin.parse_douyin_video_info(share_link: "https://v.douyin.com/xxx/")'
# 获取无水印视频下载链接
mcporter call 'douyin.get_douyin_download_link(share_link: "https://v.douyin.com/xxx/")'
# AI 提取视频语音文案(需要配置硅基流动 API Key)
mcporter call 'douyin.extract_douyin_text(share_link: "https://v.douyin.com/xxx/")'
```
> 无需登录即可解析视频。支持抖音分享链接和直接链接。
### GitHub (gh CLI)
```bash
# Search repos
gh search repos "query" --sort stars --limit 10
# View a repo
gh repo view owner/repo
# Search code
gh search code "query" --language python
# List issues
gh issue list -R owner/repo --state open
# View a specific issue/PR
gh issue view 123 -R owner/repo
```
### Web — Any URL (Jina Reader)
```bash
# Read any webpage as markdown
curl -s "https://r.jina.ai/URL" -H "Accept: text/markdown"
# Search the web
curl -s "https://s.jina.ai/query" -H "Accept: text/markdown"
```
### Exa Search (mcporter + exa MCP)
```bash
# Web search
mcporter call 'exa.web_search_exa(query: "query", numResults: 5)'
# Code search (GitHub, StackOverflow, docs)
mcporter call 'exa.get_code_context_exa(query: "how to parse JSON in Python", tokensNum: 3000)'
# Company research
mcporter call 'exa.company_research_exa(companyName: "OpenAI")'
```
### LinkedIn (mcporter + linkedin-scraper-mcp)
```bash
# View a profile
mcporter call 'linkedin.get_person_profile(linkedin_url: "https://linkedin.com/in/username")'
# Search people
mcporter call 'linkedin.search_people(keyword: "AI engineer", limit: 10)'
# View company
mcporter call 'linkedin.get_company_profile(linkedin_url: "https://linkedin.com/company/xxx")'
```
Fallback: `curl -s "https://r.jina.ai/https://linkedin.com/in/username"`
### Boss直聘 (mcporter + mcp-bosszp)
```bash
# Browse recommended jobs
mcporter call 'bosszhipin.get_recommend_jobs_tool(page: 1)'
# Search jobs
mcporter call 'bosszhipin.search_jobs_tool(keyword: "Python", city: "北京", page: 1)'
# View job details
mcporter call 'bosszhipin.get_job_detail_tool(job_url: "https://www.zhipin.com/job_detail/xxx")'
```
Fallback: `curl -s "https://r.jina.ai/https://www.zhipin.com/job_detail/xxx"`
### 微信公众号 (wechat-article-for-ai + miku_ai)
**Search** (miku_ai — Sogou WeChat search):
```python
# Search WeChat articles by keyword
python3 -c "
import asyncio
from miku_ai import get_wexin_article
async def search():
articles = await get_wexin_article('AI Agent', 5)
for a in articles:
print(f'{a[\"title\"]} | {a[\"source\"]} | {a[\"date\"]}')
print(f' {a[\"url\"]}')
asyncio.run(search())
"
```
**Read** (Camoufox — stealth Firefox, bypasses WeChat anti-bot):
```bash
# Read a WeChat article (returns Markdown with images)
cd /path/to/wechat-article-for-ai && python3 main.py "https://mp.weixin.qq.com/s/ARTICLE_ID"
# Run as MCP server (for AI agent integration)
python3 mcp_server.py
```
Typical agent workflow: search → get URLs → immediately read full content.
Note: WeChat articles require a real browser to render. Jina Reader and curl cannot read them.
### RSS (feedparser)
```python
python3 -c "
import feedparser
d = feedparser.parse('https://example.com/feed')
for e in d.entries[:5]:
print(f'{e.title} — {e.link}')
"
```
## Troubleshooting
### Twitter "fetch failed"
xreach CLI uses Node.js `undici`, which doesn't respect `HTTP_PROXY`. Solutions:
1. Ensure `undici` is installed: `npm install -g undici`
2. Configure proxy: `agent-reach configure proxy http://user:pass@ip:port`
3. If still failing, use transparent proxy (Clash TUN, Proxifier)
### Channel broken?
Run `agent-reach doctor` — it shows what's wrong and how to fix it.
+17
View File
@@ -0,0 +1,17 @@
# Agent Reach tested dependency set
# Usage:
# pip install -c constraints.txt -e .[dev]
requests==2.32.5
feedparser==6.0.12
python-dotenv==1.2.1
loguru==0.7.3
PyYAML==6.0.3
rich==14.3.2
yt-dlp==2025.5.22
pytest==8.0.0
ruff==0.15.1
mypy==1.19.1
types-requests==2.32.4.20260107
types-PyYAML==6.0.12.20250915
+82 -22
View File
@@ -45,7 +45,7 @@ Copy that to your Agent. A few minutes later, it can read tweets, search Reddit,
|---|---|
| 💰 **Completely free** | All tools are open source, all APIs are free. The only possible cost is a server proxy ($1/month) — local computers don't need one |
| 🔒 **Privacy safe** | Cookies stay local. Never uploaded. Fully open source — audit anytime |
| 🔄 **Kept up to date** | Upstream tools (yt-dlp, bird, Jina Reader, etc.) are tracked and updated regularly |
| 🔄 **Kept up to date** | Upstream tools (yt-dlp, xreach, Jina Reader, etc.) are tracked and updated regularly |
| 🤖 **Works with any Agent** | Claude Code, OpenClaw, Cursor, Windsurf… any Agent that can run commands |
| 🩺 **Built-in diagnostics** | `agent-reach doctor` — one command shows what works, what doesn't, and how to fix it |
@@ -56,11 +56,12 @@ Copy that to your Agent. A few minutes later, it can read tweets, search Reddit,
| Platform | Capabilities | Setup | Notes |
|----------|-------------|:-----:|-------|
| 🌐 **Web** | Read | Zero config | Any URL → clean Markdown ([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) |
| 🐦 **Twitter/X** | Read · Search | Zero config / Cookie | Single tweets readable out of the box. Cookie unlocks search, timeline, posting ([bird](https://github.com/steipete/bird)) |
| 🐦 **Twitter/X** | Read · Search | Zero config / Cookie | Single tweets readable out of the box. Cookie unlocks search, timeline, posting ([xreach](https://www.npmjs.com/package/xreach-cli)) |
| 📕 **XiaoHongShu** | Read · Search · **Post · Comment · Like** | mcporter | Via [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) internal API, install and go |
| 📷 **Instagram** | Read · Search | instaloader | Posts, profiles, hashtags ([instaloader](https://github.com/instaloader/instaloader) ⭐9.8K) |
| 💼 **LinkedIn** | Read · Search | mcporter / Jina | Profiles, companies, job search ([linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) ⭐900+) |
| 🏢 **Boss直聘** | Read · Search | mcporter / Jina | Job search, greet recruiters ([mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)) |
| 🎵 **Douyin** | Video parsing · Watermark-free download | mcporter | Via [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server), no login needed |
| 💼 **LinkedIn** | Jina Reader (public pages) | Full profiles, companies, job search | Tell your Agent "help me set up LinkedIn" |
| 🏢 **Boss直聘** | Jina Reader (job pages) | Job search, greet recruiters | Tell your Agent "help me set up Boss直聘" |
| 💬 **WeChat Articles** | — | Search + read WeChat Official Account articles (full Markdown) | `pip install miku_ai camoufox[geoip]` |
| 🔍 **Web Search** | Search | Auto-configured | Auto-configured during install, free, no API key ([Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter)) |
| 📦 **GitHub** | Read · Search | Zero config | [gh CLI](https://cli.github.com) powered. Public repos work immediately. `gh auth login` unlocks Fork, Issue, PR |
| 📺 **YouTube** | Read · **Search** | Zero config | Subtitles + search across 1800+ video sites ([yt-dlp](https://github.com/yt-dlp/yt-dlp) ⭐148K) |
@@ -109,14 +110,14 @@ After the Skill is installed, the Agent will auto-detect whether `agent-reach` C
No configuration needed — just tell your Agent:
- "Read this link" → any web page
- "What's this GitHub repo about?" → repos, issues, code
- "What does this video cover?" → YouTube / Bilibili subtitles
- "Read this tweet" → Twitter posts
- "Subscribe to this RSS" → RSS / Atom feeds
- "Search GitHub for LLM frameworks" → GitHub search
- "Read this link" → `curl https://r.jina.ai/URL` for any web page
- "What's this GitHub repo about?" → `gh repo view owner/repo`
- "What does this video cover?" → `yt-dlp --dump-json URL` for subtitles
- "Read this tweet" → `xreach tweet URL --json`
- "Subscribe to this RSS" → `feedparser` to parse feeds
- "Search GitHub for LLM frameworks" → `gh search repos "LLM framework"`
**No commands to remember.** The Agent knows what to call.
**No commands to remember.** The Agent reads SKILL.md and knows what to call.
---
@@ -172,41 +173,46 @@ Every time you spin up a new Agent, you spend time finding tools, installing dep
Agent Reach does one simple thing: **it makes those tool selection and configuration decisions for you.**
After installation, your Agent calls the upstream tools directly (xreach CLI, yt-dlp, mcporter, gh CLI, etc.) — no wrapper layer in between.
### 🔌 Every Channel is Pluggable
Each platform is a single Python file implementing a unified interface. **Backends can be swapped anytime** — when a better tool comes along, change one file and nothing else breaks.
Each platform maps to an upstream tool. **Don't like one? Swap it out.**
```
channels/
├── web.py → Jina Reader ← swap to Firecrawl, Crawl4AI…
├── twitter.py → bird ← swap to Nitter, official API…
├── youtube.py → yt-dlp ← swap to YouTube API, Whisper…
├── twitter.py → xreach ← swap to Nitter, official API…
├── youtube.py → yt-dlp ← swap to YouTube API, Whisper…
├── github.py → gh CLI ← swap to REST API, PyGithub…
├── bilibili.py → yt-dlp ← swap to bilibili-api…
├── bilibili.py → yt-dlp ← swap to bilibili-api…
├── reddit.py → JSON API + Exa ← swap to PRAW, Pushshift…
├── xiaohongshu.py → mcporter MCP ← swap to other XHS tools…
├── instagram.py → instaloader ← swap to instagrapi, official API
├── douyin.py → mcporter MCP ← swap to other Douyin tools
├── linkedin.py → linkedin-mcp ← swap to LinkedIn API…
├── bosszhipin.py → mcp-bosszp ← swap to other job tools…
├── rss.py → feedparser ← swap to atoma…
├── rss.py → feedparser ← swap to atoma…
├── exa_search.py → mcporter MCP ← swap to Tavily, SerpAPI…
└── __init__.py → Channel registry
└── __init__.py → Channel registry (for doctor checks)
```
Each channel file only checks whether its upstream tool is installed and working (`check()` method for `agent-reach doctor`). The actual reading and searching is done by calling the upstream tools directly.
### Current Tool Choices
| Scenario | Tool | Why |
|----------|------|-----|
| Read web pages | [Jina Reader](https://github.com/jina-ai/reader) | 9.8K stars, free, no API key needed |
| Read tweets | [bird](https://github.com/steipete/bird) | Cookie auth, free. Official API is pay-per-use ($0.005/post read) |
| Read tweets | [xreach](https://www.npmjs.com/package/xreach-cli) | Cookie auth, free. Official API is pay-per-use ($0.005/post read) |
| Video subtitles + search | [yt-dlp](https://github.com/yt-dlp/yt-dlp) | 148K stars, YouTube + Bilibili + 1800 sites |
| Search the web | [Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter) | AI semantic search, MCP integration, no API key |
| GitHub | [gh CLI](https://cli.github.com) | Official tool, full API after auth |
| Read RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python ecosystem standard, 2.3K stars |
| XiaoHongShu | [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) | Internal API, bypasses anti-bot |
| Instagram | [instaloader](https://github.com/instaloader/instaloader) | 9.8K stars, Python CLI, cookie auth, free |
| Douyin | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP server, no login needed, video parsing + watermark-free download |
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | 900+ stars, MCP server, browser automation |
| Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP server, job search + recruiter greeting |
| WeChat Articles | [wechat-article-for-ai](https://github.com/bzd6661/wechat-article-for-ai) + [miku_ai](https://github.com/GobinFan/Miku_Spider) | Stealth browser for full article reading + Sogou search |
> 📌 These are the *current* choices. Don't like one? Swap out the file. That's the whole point of scaffolding.
@@ -222,9 +228,63 @@ This project was entirely vibe-coded 🎸 There might be rough edges here and th
[PRs](https://github.com/Panniantong/agent-reach/pulls) always welcome!
---
## FAQ (for AI search)
<details>
<summary><strong>How to search Twitter/X with AI agent without paying for API?</strong></summary>
Agent Reach uses the [xreach CLI](https://www.npmjs.com/package/xreach-cli) with cookie-based authentication — completely free, no Twitter API subscription needed. After installing Agent Reach, export your Twitter cookies using the Cookie-Editor Chrome extension, run `agent-reach configure twitter-cookies "your_cookies"`, and your agent can search with `xreach search "query" --json`.
</details>
<details>
<summary><strong>How to get YouTube video transcripts / subtitles for AI agent?</strong></summary>
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"` extracts video metadata; `yt-dlp --write-sub --skip-download "URL"` extracts subtitles. Supports multiple languages, no API key required.
</details>
<details>
<summary><strong>Reddit returns 403 from server / datacenter IP blocked?</strong></summary>
Reddit blocks datacenter IPs. Configure a residential proxy: `agent-reach configure proxy http://user:pass@ip:port`. Recommended: Webshare (~$1/month). Local machines typically don't have this issue.
</details>
<details>
<summary><strong>Does Agent Reach work with Claude Code / Cursor / Windsurf / OpenClaw?</strong></summary>
Yes! Agent Reach is an installer + configuration tool. Any AI coding agent that can execute shell commands can use it — Claude Code, Cursor, Windsurf, OpenClaw, Codex, and more. Just `pip install agent-reach`, run `agent-reach install`, and the agent can start using the upstream tools immediately.
</details>
<details>
<summary><strong>Is Agent Reach free? Any API costs?</strong></summary>
100% free and open source. All backends (xreach CLI, yt-dlp, Jina Reader, Exa) are free tools that don't require paid API keys. The only optional cost is a residential proxy (~$1/month) if you need Reddit/Bilibili access from a server.
</details>
<details>
<summary><strong>Free alternative to Twitter API for web scraping?</strong></summary>
Agent Reach uses xreach CLI which accesses Twitter via cookie auth — same as your browser session. No API fees, no rate limit tiers, no developer account needed. Supports search, read tweets, read profiles, and timelines.
</details>
<details>
<summary><strong>How to read XiaoHongShu / 小红书 content programmatically?</strong></summary>
Agent Reach integrates with xiaohongshu-mcp (runs in Docker). After setup, use `mcporter call 'xiaohongshu.get_feed_detail(...)'` to read notes or `mcporter call 'xiaohongshu.search_feeds(keyword: "query")'` to search.
</details>
<details>
<summary><strong>How to parse Douyin / 抖音 videos with AI agent?</strong></summary>
Install douyin-mcp-server, then your agent can use `mcporter call 'douyin.parse_douyin_video_info(share_link: "share_url")'` to parse video info and get watermark-free download links. No login required — just share the Douyin link. See https://github.com/yzfly/douyin-mcp-server
</details>
---
## Credits
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://github.com/steipete/bird) · [Exa](https://exa.ai) · [feedparser](https://github.com/kurtmckee/feedparser) · [instaloader](https://github.com/instaloader/instaloader) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [xreach](https://www.npmjs.com/package/xreach-cli) · [Exa](https://exa.ai) · [feedparser](https://github.com/kurtmckee/feedparser) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
## License
+30
View File
@@ -0,0 +1,30 @@
# Dependency Locking Guide
Agent Reach uses `constraints.txt` as a reproducible dependency baseline.
## Why
- Keep local/CI dependency graph stable
- Reduce "works on my machine" drift
- Make regression results easier to compare
## Install with constraints
```bash
pip install -c constraints.txt -e .[dev]
```
## Update workflow
1. Update `pyproject.toml` dependency ranges as needed.
2. Validate against latest compatible versions locally.
3. Update pinned versions in `constraints.txt`.
4. Run validation:
```bash
pytest -q
ruff check agent_reach tests
mypy agent_reach
```
5. Open PR with dependency and validation notes.
+77 -28
View File
@@ -20,7 +20,7 @@ Copy this to your AI Agent:
### Goal
Install Agent Reach and make sure **all channels are working**. Your user expects you to have full internet access capabilities when this is done — reading tweets, searching Reddit, watching YouTube, browsing GitHub, and more.
Install Agent Reach and all upstream tools so your user's agent has full internet access. After installation, you'll use the upstream tools directly (xreach CLI, yt-dlp, mcporter, gh CLI, etc.) — Agent Reach is just the installer and health checker.
### ⚠️ Boundaries
@@ -41,7 +41,7 @@ pip install https://github.com/Panniantong/agent-reach/archive/main.zip
agent-reach install --env=auto
```
This auto-installs system dependencies (gh CLI, Node.js, mcporter, bird), configures Exa search, detects environment, and tests all channels.
This auto-installs system dependencies (gh CLI, Node.js, mcporter, xreach), configures Exa search, detects environment, and tests all channels.
**Safe mode (no automatic system changes):**
@@ -80,16 +80,40 @@ Only ask the user when you genuinely need their input (credentials, permissions,
Some channels need credentials only the user can provide. Based on the doctor output, ask for what's missing:
> 🔒 **Security tip:** For platforms that need cookies (Twitter, XiaoHongShu, Instagram), we recommend using a **dedicated/secondary account** rather than your main account. Cookie-based auth grants full account access — using a separate account limits the blast radius if credentials are ever compromised.
> 🔒 **Security tip:** For platforms that need cookies (Twitter, XiaoHongShu), we recommend using a **dedicated/secondary account** rather than your main account. Cookie-based auth carries two risks:
> 1. **Account ban** — platforms may detect non-browser API calls and restrict or ban the account
> 2. **Credential exposure** — cookies grant full account access; using a secondary account limits the blast radius if credentials are ever compromised
**Twitter search & posting (server users):**
> 🍪 **Cookie 导入(所有需要登录的平台通用):**
>
> 所有需要 Cookie 的平台(Twitter、小红书等),**优先使用 Cookie-Editor 导入**,这是最简单最可靠的方式:
> 1. 用户在自己的浏览器上登录对应平台
> 2. 安装 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome 插件
> 3. 点击插件 → Export → Header String
> 4. 把导出的字符串发给 Agent
>
> **本地电脑用户**也可以用 `agent-reach configure --from-browser chrome` 一键自动提取(支持 Twitter + 小红书)。
**Twitter search & posting:**
> "To unlock Twitter search, I need your Twitter cookies. Install the Cookie-Editor Chrome extension, go to x.com/twitter.com, click the extension → Export → Header String, and paste it to me."
> Extension: https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm
```bash
agent-reach configure twitter-cookies "PASTED_STRING"
```
> **代理说明(中国大陆等需要翻墙的网络环境):**
>
> xreach CLI 使用 Node.js 原生 `fetch()`,默认不走系统代理(`HTTP_PROXY`/`HTTPS_PROXY`)。
> Agent Reach 已内置解决方案:安装时会自动安装 `undici``npm install -g undici`),
> 运行 xreach 时自动注入 `EnvHttpProxyAgent`,让 fetch 走用户配置的代理。
>
> **你(Agent)需要做的:**
> 1. 确认 `undici` 已全局安装:`npm list -g undici`,没有就装:`npm install -g undici`
> 2. 确认用户配了代理:`agent-reach configure proxy http://user:pass@ip:port`
> 3. Agent Reach 会自动处理剩下的,不需要用户做额外操作
>
> 如果用户报告 "fetch failed",参考 [troubleshooting.md](troubleshooting.md)
**Reddit & Bilibili full access (server users):**
> "Reddit and Bilibili block server IPs. To unlock full access, I need a residential proxy. You can get one at https://webshare.io ($1/month). Send me the proxy address."
@@ -108,23 +132,42 @@ mcporter config add xiaohongshu http://localhost:18060/mcp
> 如果在服务器上,建议加代理避免 IP 风控:
> `docker run -d --name xiaohongshu-mcp -p 18060:18060 -e XHS_PROXY=http://user:pass@ip:port xpzouying/xiaohongshu-mcp`
>
> **登录方式:**
> - **本地电脑(有浏览器):** 打开 http://localhost:18060 扫码登录即可。
> - **服务器(无 UI 界面):** 服务器上通常没有浏览器,无法直接扫码。最方便的方式是在自己的电脑上用浏览器登录小红书,然后用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 插件导出 CookieHeader String 格式),发给 Agent 即可完成配置。详见 [Cookie 导出指南](cookie-export.md)。
> **登录方式(优先用 Cookie-Editor,最简单)**
> 1. 用户在自己的浏览器登录小红书 (xiaohongshu.com)
> 2. 用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 插件导出 CookieHeader String 格式)
> 3. 把 Cookie 字符串发给 Agent
> 4. Agent 将 Cookie 写入 MCP 服务的 cookie 文件完成登录
>
> **备选:** 本地电脑如果有浏览器,也可以打开 http://localhost:18060 扫码登录。
**Instagram (需要 instaloader):**
> "Instagram 需要 instaloader。我来帮你安装。"
**抖音 / Douyin (douyin-mcp-server):**
> "抖音视频解析需要一个 MCP 服务。安装 douyin-mcp-server 后即可解析视频、获取无水印下载链接。"
```bash
pip install instaloader
# 1. 安装
pip install douyin-mcp-server
# 2. 启动 HTTP 服务(端口 18070
# 方式一:用 uv(推荐)
git clone https://github.com/yzfly/douyin-mcp-server.git && cd douyin-mcp-server
uv sync && uv run python run_http.py
# 方式二:直接用 Python 启动
python -c "
from douyin_mcp_server.server import mcp
mcp.settings.host = '127.0.0.1'
mcp.settings.port = 18070
mcp.run(transport='streamable-http')
"
# 3. 注册到 mcporter
mcporter config add douyin http://localhost:18070/mcp
```
> **登录方式(解锁私密内容):**
> - **方法 1(推荐):Cookie-Editor 导入:** 在浏览器登录 Instagram → 用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 导出 Header String → 粘贴:
> ```bash
> agent-reach configure instagram-cookies "sessionid=xxx; csrftoken=yyy; ..."
> ```
> - **方法 2instaloader 命令行登录:** `instaloader --login YOUR_USERNAME`(需要输密码,有 2FA 的话还要输验证码)
> 无需认证即可解析视频信息和获取下载链接。
> 如需 AI 语音识别提取文案功能,需要配置硅基流动 API Key(`export API_KEY="sk-xxx"`)。
>
> 详见 https://github.com/yzfly/douyin-mcp-server
**LinkedIn (可选 — linkedin-scraper-mcp):**
> "LinkedIn 基本内容可通过 Jina Reader 读取。完整功能(Profile 详情、职位搜索)需要 linkedin-scraper-mcp。"
@@ -229,14 +272,20 @@ If the user wants a different agent to handle it, let them choose.
| `agent-reach check-update` | Check for new versions |
| `agent-reach configure twitter-cookies "..."` | Unlock Twitter search + posting |
| `agent-reach configure proxy URL` | Unlock Reddit + Bilibili on servers |
| `agent-reach read <url>` | Read any URL |
| `agent-reach search "query"` | Search the web |
| `agent-reach search-reddit "query"` | Search Reddit |
| `agent-reach search-twitter "query"` | Search Twitter |
| `agent-reach search-github "query"` | Search GitHub |
| `agent-reach search-youtube "query"` | Search YouTube |
| `agent-reach search-bilibili "query"` | Search Bilibili |
| `agent-reach search-xhs "query"` | Search XiaoHongShu |
| `agent-reach search-instagram "query"` | Search Instagram |
| `agent-reach search-linkedin "query"` | Search LinkedIn |
| `agent-reach search-bosszhipin "query"` | Search Boss直聘 |
After installation, use upstream tools directly. See SKILL.md for the full command reference:
| Platform | Upstream Tool | Example |
|----------|--------------|---------|
| Twitter/X | `xreach` | `xreach search "query" --json` |
| YouTube | `yt-dlp` | `yt-dlp --dump-json URL` |
| Bilibili | `yt-dlp` | `yt-dlp --dump-json URL` |
| Reddit | `curl` | `curl -s "https://reddit.com/r/xxx.json"` |
| GitHub | `gh` | `gh search repos "query"` |
| Web | `curl` + Jina | `curl -s "https://r.jina.ai/URL"` |
| Exa Search | `mcporter` | `mcporter call 'exa.web_search_exa(...)'` |
| 小红书 | `mcporter` | `mcporter call 'xiaohongshu.search_feeds(...)'` |
| 抖音 | `mcporter` | `mcporter call 'douyin.parse_douyin_video_info(...)'` |
| LinkedIn | `mcporter` | `mcporter call 'linkedin.get_person_profile(...)'` |
| Boss直聘 | `mcporter` | `mcporter call 'bosszhipin.search_jobs_tool(...)'` |
| RSS | `feedparser` | `python3 -c "import feedparser; ..."` |
+44
View File
@@ -0,0 +1,44 @@
# 常见问题排查
## Twitter/X: xreach CLI "fetch failed"
**症状:** `xreach search` 或其他命令返回 "fetch failed"
**原因:** xreach CLI 使用 Node.js 的 `undici` 库发请求。如果你的网络环境需要代理才能访问 x.com,需要明确传入代理参数。
**解决方案:**
### 方案 1:使用 --proxy 参数
```bash
xreach search "test" --auth-token "$AUTH_TOKEN" --ct0 "$CT0" --proxy "http://user:pass@host:port"
```
### 方案 2:使用全局代理工具
让代理工具接管所有网络流量,这样 xreach 的请求也会走代理:
```bash
# macOS — ClashX / Surge 开启"增强模式"
# Linux — proxychains 或 tun2socks
proxychains xreach search "test" -n 1
```
### 方案 3:不用 xreach,用 Exa 搜索替代
xreach 不可用时,可以直接用 Exa 搜索 Twitter 内容:
```bash
mcporter call 'exa.web_search_exa(query: "site:x.com 搜索词", numResults: 5)'
```
### 方案 4:设置 HTTP_PROXY 环境变量
```bash
export HTTP_PROXY="http://127.0.0.1:7890"
export HTTPS_PROXY="http://127.0.0.1:7890"
xreach search "test"
```
> ⚠️ 注意:Node.js 原生 fetch 不一定读取这些环境变量,推荐用方案 1 的 --proxy 参数。
+32
View File
@@ -0,0 +1,32 @@
# Agent Reach
> Give your AI agent eyes to see the entire internet. A unified CLI tool that lets AI agents read and search across 12+ platforms — Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, Instagram, LinkedIn, Boss直聘, RSS, and any web page. One install, zero API fees.
## Quick Start
- [Installation Guide](https://github.com/Panniantong/Agent-Reach/blob/main/docs/install.md): Step-by-step setup instructions for AI agents
- [README (中文)](https://github.com/Panniantong/Agent-Reach/blob/main/README.md): Full documentation in Chinese
- [README (English)](https://github.com/Panniantong/Agent-Reach/blob/main/docs/README_en.md): Full documentation in English
## Core Commands
- [CLI Usage](https://github.com/Panniantong/Agent-Reach/blob/main/docs/install.md): `agent-reach read <url>` reads any URL, `agent-reach search-twitter "query"` searches Twitter, etc.
## Key Features
- Read any URL: tweets, Reddit posts, YouTube videos (transcripts), GitHub repos, articles, XiaoHongShu notes, Bilibili videos, RSS feeds
- Search across platforms: Twitter/X, Reddit, GitHub, YouTube, Bilibili, XiaoHongShu, Instagram, LinkedIn, Boss直聘, Web (via Exa)
- Self-diagnosis: `agent-reach doctor` checks what works and what needs setup
- Auto-installs dependencies: `agent-reach install --env=auto`
- Cookie-based auth for platforms that require login (Twitter, XiaoHongShu, Instagram)
- Proxy support for platforms that block server IPs (Reddit, Bilibili)
- Zero API fees: all backends are free and open-source (bird CLI, yt-dlp, Jina Reader, instaloader, etc.)
## Troubleshooting
- [Troubleshooting Guide](https://github.com/Panniantong/Agent-Reach/blob/main/docs/troubleshooting.md): Common issues and solutions
## Optional
- [SKILL.md](https://github.com/Panniantong/Agent-Reach/blob/main/agent_reach/skill/SKILL.md): Integration guide for AI agent frameworks (OpenClaw, Claude Code, etc.)
- [PyPI Package](https://pypi.org/project/agent-reach/): `pip install agent-reach`
+34 -2
View File
@@ -1,12 +1,20 @@
[project]
name = "agent-reach"
version = "1.1.0"
version = "1.3.0"
description = "Give your AI Agent eyes to see the entire internet. Search + Read 10+ platforms."
readme = "README.md"
license = {text = "MIT"}
requires-python = ">=3.10"
authors = [{name = "Neo Reid"}]
keywords = ["ai-agent", "web-reader", "search", "mcp", "agent-infrastructure", "llm-tools"]
keywords = [
"ai-agent", "llm-tools", "agent-infrastructure", "mcp",
"web-reader", "web-scraper", "search",
"twitter-scraper", "reddit-scraper", "youtube-transcript",
"bilibili", "xiaohongshu",
"ai-search", "cli", "automation",
"claude-code", "cursor", "openai",
"free-api", "no-api-key",
]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
@@ -33,6 +41,13 @@ dependencies = [
browser = ["playwright>=1.40"]
cookies = ["browser-cookie3>=0.19"]
all = ["playwright>=1.40", "mcp[cli]>=1.0", "browser-cookie3>=0.19"]
dev = [
"pytest>=8.0",
"ruff>=0.8",
"mypy>=1.12",
"types-requests>=2.32",
"types-PyYAML>=6.0",
]
[project.scripts]
agent-reach = "agent_reach.cli:main"
@@ -52,3 +67,20 @@ packages = ["agent_reach"]
[tool.hatch.build.targets.wheel.force-include]
"agent_reach/guides" = "agent_reach/guides"
"agent_reach/skill" = "agent_reach/skill"
[tool.ruff]
target-version = "py310"
line-length = 100
[tool.ruff.lint]
select = ["E", "F", "I"]
ignore = ["E501"]
[tool.mypy]
python_version = "3.10"
warn_unused_configs = true
warn_redundant_casts = true
warn_unused_ignores = true
check_untyped_defs = true
ignore_missing_imports = true
exclude = ["^tests/"]
+50
View File
@@ -0,0 +1,50 @@
# -*- coding: utf-8 -*-
"""Contract tests for channel adapters."""
from agent_reach.channels import get_all_channels
from agent_reach.config import Config
def test_channel_registry_contract():
channels = get_all_channels()
assert channels, "channel registry must not be empty"
names = [ch.name for ch in channels]
assert len(names) == len(set(names)), "channel names must be unique"
for ch in channels:
assert isinstance(ch.name, str) and ch.name
assert isinstance(ch.description, str) and ch.description
assert isinstance(ch.backends, list)
assert ch.tier in {0, 1, 2}
def test_channel_check_contract_with_minimal_runtime(monkeypatch, tmp_path):
# Keep contract tests deterministic by simulating "deps mostly absent".
monkeypatch.setattr("shutil.which", lambda _cmd: None)
config = Config(config_path=tmp_path / "config.yaml")
for ch in get_all_channels():
status, message = ch.check(config)
assert status in {"ok", "warn", "off", "error"}
assert isinstance(message, str) and message.strip()
def test_channel_can_handle_contract():
url_samples = {
"github": "https://github.com/panniantong/agent-reach",
"twitter": "https://x.com/user/status/1",
"youtube": "https://youtube.com/watch?v=abc",
"reddit": "https://reddit.com/r/python",
"bilibili": "https://www.bilibili.com/video/BV1xx411",
"xiaohongshu": "https://www.xiaohongshu.com/explore/123",
"douyin": "https://www.douyin.com/video/123",
"linkedin": "https://www.linkedin.com/in/test",
"bosszhipin": "https://www.zhipin.com/web/geek/job?query=python",
"rss": "https://example.com/feed.xml",
"exa_search": "https://example.com",
"web": "https://example.com",
}
for ch in get_all_channels():
sample = url_samples.get(ch.name, "https://example.com")
result = ch.can_handle(sample)
assert isinstance(result, bool)
+6 -99
View File
@@ -1,114 +1,21 @@
# -*- coding: utf-8 -*-
"""Tests for the channel system."""
"""Tests for channel registry basics."""
import pytest
from unittest.mock import patch, MagicMock
from agent_reach.channels import get_channel_for_url, get_channel, get_all_channels
from agent_reach.channels.base import ReadResult, SearchResult
from agent_reach.channels import get_all_channels, get_channel
class TestChannelRouting:
def test_github_url(self):
ch = get_channel_for_url("https://github.com/openai/gpt-4")
assert ch.name == "github"
def test_twitter_url(self):
ch = get_channel_for_url("https://x.com/elonmusk/status/123")
assert ch.name == "twitter"
def test_youtube_url(self):
ch = get_channel_for_url("https://youtube.com/watch?v=abc")
assert ch.name == "youtube"
def test_reddit_url(self):
ch = get_channel_for_url("https://reddit.com/r/test")
assert ch.name == "reddit"
def test_bilibili_url(self):
ch = get_channel_for_url("https://bilibili.com/video/BV1xx")
assert ch.name == "bilibili"
def test_rss_url(self):
ch = get_channel_for_url("https://example.com/feed.xml")
assert ch.name == "rss"
def test_generic_url_fallback(self):
ch = get_channel_for_url("https://example.com")
assert ch.name == "web"
class TestChannelRegistry:
def test_get_channel_by_name(self):
ch = get_channel("github")
assert ch is not None
assert ch.name == "github"
def test_get_unknown_channel_returns_none(self):
assert get_channel("not-exists") is None
def test_all_channels_registered(self):
channels = get_all_channels()
names = [ch.name for ch in channels]
assert "web" in names
assert "github" in names
assert "twitter" in names
class TestReadResult:
def test_to_dict(self):
r = ReadResult(title="Test", content="Body", url="https://example.com", platform="web")
d = r.to_dict()
assert d["title"] == "Test"
assert d["content"] == "Body"
assert d["platform"] == "web"
def test_to_dict_optional_fields(self):
r = ReadResult(title="T", content="C", url="u", author="A", date="2025-01-01")
d = r.to_dict()
assert d["author"] == "A"
assert d["date"] == "2025-01-01"
class TestSearchResult:
def test_to_dict(self):
r = SearchResult(title="Test", url="https://example.com", snippet="A snippet")
d = r.to_dict()
assert d["title"] == "Test"
assert d["snippet"] == "A snippet"
class TestGitHubChannel:
@patch("agent_reach.channels.github.requests.get")
@pytest.mark.asyncio
async def test_search(self, mock_get):
mock_resp = MagicMock()
mock_resp.json.return_value = {
"items": [{"full_name": "test/repo", "html_url": "https://github.com/test/repo",
"description": "A test", "stargazers_count": 100, "forks_count": 10,
"language": "Python", "updated_at": "2025-01-01"}]
}
mock_resp.raise_for_status = MagicMock()
mock_get.return_value = mock_resp
ch = get_channel("github")
results = await ch.search("test query")
assert len(results) == 1
assert results[0].title == "test/repo"
class TestExaSearch:
@patch("agent_reach.channels.exa_search.requests.post")
@pytest.mark.asyncio
async def test_search(self, mock_post):
from agent_reach.config import Config
config = Config(config_path="/tmp/test-exa-config.yaml")
config.set("exa_api_key", "test-key")
mock_resp = MagicMock()
mock_resp.json.return_value = {
"results": [{"title": "Result", "url": "https://example.com",
"text": "snippet", "publishedDate": "", "score": 0.9}]
}
mock_resp.raise_for_status = MagicMock()
mock_post.return_value = mock_resp
ch = get_channel("exa_search")
results = await ch.search("test", config=config)
assert len(results) == 1
assert results[0].title == "Result"
+85
View File
@@ -2,7 +2,9 @@
"""Tests for Agent Reach CLI."""
import pytest
import requests
from unittest.mock import patch
import agent_reach.cli as cli
from agent_reach.cli import main
@@ -27,3 +29,86 @@ class TestCLI:
captured = capsys.readouterr()
assert "Agent Reach" in captured.out
assert "" in captured.out
class TestCheckUpdateRetry:
def test_retry_timeout_classification(self):
sleeps = []
def fake_sleep(seconds):
sleeps.append(seconds)
with patch("requests.get", side_effect=requests.exceptions.Timeout("timed out")):
resp, err, attempts = cli._github_get_with_retry(
"https://api.github.com/test",
timeout=1,
retries=3,
sleeper=fake_sleep,
)
assert resp is None
assert err == "timeout"
assert attempts == 3
assert sleeps == [1, 2]
def test_retry_dns_classification(self):
error = requests.exceptions.ConnectionError("getaddrinfo failed for api.github.com")
with patch("requests.get", side_effect=error):
resp, err, attempts = cli._github_get_with_retry(
"https://api.github.com/test",
retries=1,
sleeper=lambda _x: None,
)
assert resp is None
assert err == "dns"
assert attempts == 1
def test_retry_rate_limit_then_success(self):
sleeps = []
class R:
def __init__(self, code, payload=None, headers=None):
self.status_code = code
self._payload = payload or {}
self.headers = headers or {}
def json(self):
return self._payload
sequence = [
R(429, headers={"Retry-After": "3"}),
R(200, payload={"tag_name": "v1.3.0"}),
]
with patch("requests.get", side_effect=sequence):
resp, err, attempts = cli._github_get_with_retry(
"https://api.github.com/test",
retries=3,
sleeper=lambda s: sleeps.append(s),
)
assert err is None
assert resp is not None
assert resp.status_code == 200
assert attempts == 2
assert sleeps == [3.0]
def test_classify_rate_limit_from_403(self):
class R:
status_code = 403
headers = {"X-RateLimit-Remaining": "0"}
@staticmethod
def json():
return {"message": "API rate limit exceeded"}
assert cli._classify_github_response_error(R()) == "rate_limit"
def test_check_update_reports_classified_error(self, capsys):
with patch("agent_reach.cli._github_get_with_retry", return_value=(None, "timeout", 3)):
result = cli._cmd_check_update()
captured = capsys.readouterr()
assert result == "error"
assert "网络超时" in captured.out
assert "已重试 3 次" in captured.out
+1 -8
View File
@@ -2,6 +2,7 @@
"""Tests for AgentReach core class."""
import pytest
from agent_reach.config import Config
from agent_reach.core import AgentReach
@@ -16,14 +17,6 @@ class TestAgentReach:
def test_init(self, eyes):
assert eyes.config is not None
def test_detect_platform(self, eyes):
assert eyes.detect_platform("https://github.com/test/repo") == "github"
assert eyes.detect_platform("https://reddit.com/r/test") == "reddit"
assert eyes.detect_platform("https://x.com/user/status/123") == "twitter"
assert eyes.detect_platform("https://youtube.com/watch?v=abc") == "youtube"
assert eyes.detect_platform("https://bilibili.com/video/BV1xx") == "bilibili"
assert eyes.detect_platform("https://example.com") == "web"
def test_doctor(self, eyes):
results = eyes.doctor()
assert isinstance(results, dict)
+4 -3
View File
@@ -15,7 +15,7 @@ class TestDoctor:
def test_zero_config_channels_ok(self, tmp_config):
results = check_all(tmp_config)
assert results["web"]["status"] == "ok"
assert results["github"]["status"] == "ok"
assert results["github"]["status"] in ("ok", "warn") # warn if gh CLI not installed
assert results["bilibili"]["status"] in ("ok", "warn") # warn on servers
assert results["rss"]["status"] == "ok"
@@ -23,10 +23,11 @@ class TestDoctor:
results = check_all(tmp_config)
assert results["exa_search"]["status"] == "off"
def test_exa_on_with_key(self, tmp_config):
def test_exa_key_does_not_force_enabled(self, tmp_config):
# Exa availability is determined by mcporter runtime/config state.
tmp_config.set("exa_api_key", "test-key")
results = check_all(tmp_config)
assert results["exa_search"]["status"] == "ok"
assert results["exa_search"]["status"] in ("off", "ok")
def test_format_report(self, tmp_config):
results = check_all(tmp_config)