22 Commits

Author SHA1 Message Date
Panniantong 37b4cdeddc bump: v1.2.0 2026-02-26 13:34:41 +01:00
Pnant bf6dc7084b Merge pull request #28 from Panniantong/feat/douyin-channel
feat: 新增抖音(Douyin)渠道
2026-02-26 20:34:17 +08:00
Panniantong 50f2181fcc feat: 新增抖音(Douyin)渠道 - 基于 douyin-mcp-server
- 新增 agent_reach/channels/douyin.py
- 更新 doctor 检测逻辑
- 更新 install.md/SKILL.md/README 文档
- 通过 mcporter 调用 douyin-mcp-server
- 支持视频解析、下载链接获取
- 无需登录即可使用
2026-02-26 13:25:57 +01:00
Pnant b387a67b51 Merge pull request #27 from Panniantong/docs/cookie-editor-first
docs: Cookie-Editor 作为所有 cookie 平台的首选登录方式
2026-02-26 19:17:35 +08:00
Panniantong a96c5cda14 docs: Cookie-Editor 作为首选登录方式,统一所有 cookie 平台
- install.md: 新增通用 Cookie 导入说明,所有平台统一流程
- install.md: 小红书登录方式改为 Cookie-Editor 优先,扫码降级为备选
- SKILL.md: Cookie 导入独立成段,明确优先级
- README.md: 更新 Cookie 说明,强调 Cookie-Editor 比扫码更简单可靠
2026-02-26 12:16:34 +01:00
Pnant cf70302e92 Merge pull request #23 from Panniantong/refactor/simplify-to-installer
refactor: 回归安装器+文档定位,砍掉 read/search 包装层
2026-02-26 15:29:44 +08:00
Panniantong 80a08c8c10 docs: update all remaining references to use upstream tools directly
- docs/README_en.md: update FAQ, Works Out of the Box, Design Philosophy
- docs/troubleshooting.md: Exa fallback uses mcporter directly
- agent_reach/guides/setup-exa.md: test with mcporter call
- agent_reach/guides/setup-wechat.md: test with Jina Reader
2026-02-26 08:25:06 +01:00
Panniantong 606e8900ce docs: update install.md — remove read/search references, add upstream tools table 2026-02-26 08:20:20 +01:00
Panniantong a37e9aa190 refactor: strip to installer + doctor + docs, remove read/search wrapper layer
BREAKING CHANGE: Remove all `agent-reach read` and `agent-reach search-*` commands.

Agent Reach is now an installer, configuration tool, and doctor —
not a wrapper layer. After installation, agents call upstream tools
directly (bird CLI, yt-dlp, mcporter, gh CLI, Jina Reader, etc.).

What's kept:
- agent-reach install: one-shot installer
- agent-reach doctor: channel status overview
- agent-reach configure: cookies, proxy, credentials
- agent-reach setup: interactive wizard
- SKILL.md: complete guide for agents to use upstream tools directly

What's removed:
- agent-reach read URL (and all channel read() methods)
- agent-reach search-* commands (and all channel search() methods)
- ReadResult / SearchResult data classes
- URL routing system (get_channel_for_url)
- All parsing/conversion logic (VTT, Reddit JSON, bird text parser, etc.)
- MCP server read/search tools (kept only get_status)

Net change: -1790 lines. Less code = fewer bugs.
2026-02-26 08:15:56 +01:00
Pnant 1cbf6a7b9c fix(xiaohongshu): 修复 xsec_token 丢失导致无法读取笔记详情 (#17)
问题:搜索结果包含 xsecToken 但被丢弃,read() 只从首页 feed
查找 token,几乎不可能匹配到目标笔记。

修复:
1. search() 返回的 URL 携带 xsec_token 查询参数
2. read() 三级 token 查找:URL 参数 → 首页 feed → 搜索反查
3. 全部失败时回退到 Jina Reader

Fixes #15

Co-authored-by: Panniantong <panniantong@users.noreply.github.com>
2026-02-26 14:36:36 +08:00
Pnant 68319ec683 Merge pull request #16 from Panniantong/remove/instagram
remove(instagram): 移除 Instagram 渠道(上游反爬封杀)
2026-02-26 14:23:26 +08:00
Panniantong f70711e75e remove(instagram): 移除 Instagram 渠道
Instagram 反爬封杀导致所有开源工具(instaloader 等)失效,
无论有无 cookies 都无法正常使用。

- 删除 instagram.py 渠道文件
- 移除 CLI 中 search-instagram、configure instagram-cookies 等命令
- 移除 setup/doctor 中 instaloader 依赖检查
- 更新 README、docs、SKILL.md、pyproject.toml

上游 issue: instaloader#2585, instaloader#2648
Relates to: #13
2026-02-26 07:20:13 +01:00
Panniantong c3a9813b1c seo: GitHub SEO + GEO 优化,提升项目可发现性
1. 新增 llms.txt — 让 AI 搜索引擎(ChatGPT/Claude/Perplexity)理解项目
2. README 中英文加 FAQ 区块 — AI 搜索友好的常见问题解答
3. pyproject.toml keywords 扩展到 21 个 — 覆盖更多搜索词
2026-02-25 22:32:22 +01:00
Panniantong d291d27664 docs: install.md 和 SKILL.md 补充 Twitter 代理方案说明
让 Agent 在配置 Twitter 时知道:
- undici 需要安装(npm install -g undici)
- 代理注入是自动的,不需要用户额外操作
- fetch failed 的排查步骤
2026-02-25 21:45:40 +01:00
Pnant 0be4f097d0 Merge pull request #10 from Panniantong/fix/twitter-fetch-failed
fix: Twitter bird CLI fetch failed — 自动 fallback + 连通性检测
2026-02-26 04:44:39 +08:00
Panniantong 12bd727e30 fix: Twitter bird CLI fetch failed — 自动代理注入 + 连通性检测 + Exa fallback
问题:bird CLI 用 Node.js 原生 fetch(),不走 HTTP_PROXY,
导致需要代理的用户(如 Windows + 本地代理)完全无法使用。

修复:
1. 自动代理注入:检测到 HTTP_PROXY/HTTPS_PROXY 时,通过 undici 的
   EnvHttpProxyAgent 让 Node.js fetch 自动走代理,用户无需任何额外配置
2. doctor 检测:实际跑 bird whoami 测连通性,失败时给出具体原因
3. search fallback:bird 搜索失败时自动 fallback 到 Exa
4. install:自动安装 undici(npm install -g undici)
5. 新增 troubleshooting.md:常见问题解决方案

Fixes #9
2026-02-25 21:35:21 +01:00
Panniantong b6d135c977 docs: 小红书配置方式统一为「告诉 Agent 帮我配」 2026-02-25 16:03:50 +01:00
Pnant 291d073223 Merge pull request #8 from Panniantong/refactor/skill-consolidate
refactor: 合并 SKILL.md 为单一来源,更新为 12 渠道
2026-02-25 22:32:31 +08:00
Pnant ef0889dd69 Merge pull request #7 from Panniantong/docs/channel-limitations
docs: 补充新渠道使用限制说明
2026-02-25 22:32:27 +08:00
Panniantong 545a6c5761 docs: 新渠道配置方式统一为「告诉 Agent 帮我配」
跟其他渠道风格一致,不单独列限制说明。
具体配置步骤都在 install.md 里,Agent 会自动引导。
2026-02-25 15:27:51 +01:00
Panniantong 00f3edd9cc refactor: 合并 SKILL.md 为单一来源,更新为 12 个渠道
- 删除 agent-reach/SKILL.md(重复)
- 删除 agent_reach/integrations/skill/SKILL.md(重复)
- 保留并更新 agent_reach/skill/SKILL.md 为唯一来源
- 新增 Instagram/LinkedIn/Boss直聘 搜索命令
- 新增渠道配置引导流程说明
- 描述从 9+ 更新为 12+
2026-02-25 15:11:51 +01:00
Panniantong 2f674f1e45 security: doctor warns if config.yaml permissions are too open
Adds a check in 'agent-reach doctor' that warns when ~/.agent-reach/config.yaml
is readable by other users (group/world). Suggests chmod 600 to fix.

Ref: #6
2026-02-25 14:16:08 +01:00
31 changed files with 863 additions and 2559 deletions
+9 -12
View File
@@ -10,13 +10,10 @@ All notable changes to this project will be documented in this file.
### 🆕 New Channels / 新增渠道 ### 🆕 New Channels / 新增渠道
#### 📷 Instagram #### ~~📷 Instagram~~ (removed — upstream blocked)
- Read public posts and profiles via [instaloader](https://github.com/instaloader/instaloader) - ~~Read public posts and profiles via [instaloader](https://github.com/instaloader/instaloader)~~
- Search via Exa (free, no API key) - **Removed:** Instagram's aggressive anti-scraping measures broke all available open-source tools (instaloader, etc.). See [instaloader#2585](https://github.com/instaloader/instaloader/issues/2585). Will re-add when upstream recovers.
- Optional cookie login for private content - **已移除:** Instagram 反爬封杀导致所有开源工具(instaloader 等)失效。上游恢复后会重新加回。
- 通过 instaloader 读取公开帖子和 Profile
- 搜索通过 Exa(免费,无需 API Key
- 可选 Cookie 登录解锁私密内容
#### 💼 LinkedIn #### 💼 LinkedIn
- Read person profiles, company pages, and job details via [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) - Read person profiles, company pages, and job details via [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
@@ -38,12 +35,12 @@ All notable changes to this project will be documented in this file.
- Channel count: 9 → 12 - Channel count: 9 → 12
- `agent-reach doctor` now detects all 12 channels - `agent-reach doctor` now detects all 12 channels
- CLI: added `search-instagram`, `search-linkedin`, `search-bosszhipin` subcommands - CLI: added `search-linkedin`, `search-bosszhipin` subcommands
- Updated install guide with setup instructions for new channels - Updated install guide with setup instructions for new channels
- 渠道数量:9 → 12 - 渠道数量:9 → 11
- `agent-reach doctor` 现在检测全部 12 个渠道 - `agent-reach doctor` 现在检测全部 11 个渠道
- CLI:新增 `search-instagram``search-linkedin``search-bosszhipin` 子命令 - CLI:新增 `search-linkedin``search-bosszhipin` 子命令
- 安装指南新增三个渠道配置说明 - 安装指南新增渠道配置说明
--- ---
+82 -24
View File
@@ -11,7 +11,7 @@
</p> </p>
<p align="center"> <p align="center">
<a href="#30-秒上手">快速开始</a> · <a href="docs/README_en.md">English</a> · <a href="#支持的平台">支持平台</a> · <a href="#设计理念">设计理念</a> <a href="#快速上手">快速开始</a> · <a href="docs/README_en.md">English</a> · <a href="#支持的平台">支持平台</a> · <a href="#设计理念">设计理念</a>
</p> </p>
--- ---
@@ -68,14 +68,14 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
| 🐦 **Twitter/X** | 读单条推文 | 搜索推文、浏览时间线、发推 | 告诉 Agent「帮我配 Twitter」 | | 🐦 **Twitter/X** | 读单条推文 | 搜索推文、浏览时间线、发推 | 告诉 Agent「帮我配 Twitter」 |
| 📺 **B站** | 本地:字幕提取 + 搜索 | 服务器也能用 | 告诉 Agent「帮我配代理」 | | 📺 **B站** | 本地:字幕提取 + 搜索 | 服务器也能用 | 告诉 Agent「帮我配代理」 |
| 📖 **Reddit** | 搜索(通过 Exa 免费) | 读帖子和评论 | 告诉 Agent「帮我配代理」 | | 📖 **Reddit** | 搜索(通过 Exa 免费) | 读帖子和评论 | 告诉 Agent「帮我配代理」 |
| 📕 **小红书** | — | 阅读、搜索、发帖、评论、点赞 | `docker run -d -p 18060:18060 xpzouying/xiaohongshu-mcp` 然后告诉 Agent「帮我配小红书」 | | 📕 **小红书** | — | 阅读、搜索、发帖、评论、点赞 | 告诉 Agent「帮我配小红书」 |
| 📷 **Instagram** | — | 读取帖子和 Profile | Cookie-Editor 导出 Cookie → `agent-reach configure instagram-cookies "..."` | | 🎵 **抖音** | — | 视频解析、无水印下载链接获取 | 告诉 Agent「帮我配抖音」 |
| 💼 **LinkedIn** | Jina Reader 读公开页面 | Profile 详情、公司、职位搜索 | `linkedin-scraper-mcp --login`(需浏览器,服务器用 VNC | | 💼 **LinkedIn** | Jina Reader 读公开页面 | Profile 详情、公司页面、职位搜索 | 告诉 Agent「帮我配 LinkedIn」 |
| 🏢 **Boss直聘** | Jina Reader 读职位页 | 搜索职位、向 HR 打招呼 | 启动 mcp-bosszp → Boss直聘 App 扫码 | | 🏢 **Boss直聘** | Jina Reader 读职位页 | 搜索职位、向 HR 打招呼 | 告诉 Agent「帮我配 Boss直聘」 |
> **不知道怎么配?不用查文档。** 直接告诉 Agent「帮我配 XXX」,它知道需要什么、会一步一步引导你。 > **不知道怎么配?不用查文档。** 直接告诉 Agent「帮我配 XXX」,它知道需要什么、会一步一步引导你。
> >
> 🍪 需要 Cookie 的平台(Twitter、小红书等),建议使用 Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 一键导出。**服务器用户**没有浏览器界面,请在自己的电脑上登录对应网站后导出 Cookie,再发给 Agent 配置 > 🍪 需要 Cookie 的平台(Twitter、小红书等),**优先使用** Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 导出 Cookie,发给 Agent 即可配置。流程统一:浏览器登录 → Cookie-Editor 导出 → 发给 Agent。比扫码更简单可靠
> >
> 🔒 Cookie 只存在你本地,不上传不外传。代码完全开源,随时可审查。 > 🔒 Cookie 只存在你本地,不上传不外传。代码完全开源,随时可审查。
> 💻 本地电脑不需要代理。代理只有部署在服务器上才需要(~$1/月)。 > 💻 本地电脑不需要代理。代理只有部署在服务器上才需要(~$1/月)。
@@ -105,7 +105,7 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
2. **安装系统依赖** — 自动检测并安装 Node.js、gh CLI、mcporter、bird 等 2. **安装系统依赖** — 自动检测并安装 Node.js、gh CLI、mcporter、bird 等
3. **配置搜索引擎** — 通过 MCP 接入 Exa(免费,无需 API Key 3. **配置搜索引擎** — 通过 MCP 接入 Exa(免费,无需 API Key
4. **检测环境** — 判断是本地电脑还是服务器,给出对应的配置建议 4. **检测环境** — 判断是本地电脑还是服务器,给出对应的配置建议
5. **注册 Skill** — 在 Agent 的 skills 目录安装 SKILL.md,以后 Agent 遇到"搜推特"、"看视频"这类需求,会自动知道调用 Agent Reach 5. **注册 SKILL.md** — 在 Agent 的 skills 目录安装使用指南,以后 Agent 遇到"搜推特"、"看视频"这类需求,会自动知道该调哪个上游工具
安装完之后,`agent-reach doctor` 一条命令告诉你每个渠道的状态。 安装完之后,`agent-reach doctor` 一条命令告诉你每个渠道的状态。
</details> </details>
@@ -116,14 +116,14 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
不需要任何配置,告诉 Agent 就行: 不需要任何配置,告诉 Agent 就行:
- "帮我看看这个链接" → 任意网页 - "帮我看看这个链接" → `curl https://r.jina.ai/URL`任意网页
- "这个 GitHub 仓库是做什么的" → GitHub 仓库、Issue、代码 - "这个 GitHub 仓库是做什么的" → `gh repo view owner/repo`
- "这个视频讲了什么" → YouTube / B站字幕提取 - "这个视频讲了什么" → `yt-dlp --dump-json URL` 提取字幕
- "帮我看看这条推文" → Twitter 推文 - "帮我看看这条推文" → `bird read URL --json`
- "订阅这个 RSS" → RSS / Atom 源 - "订阅这个 RSS" → `feedparser` 解析
- "搜一下 GitHub 上有什么 LLM 框架" → GitHub 搜索 - "搜一下 GitHub 上有什么 LLM 框架" → `gh search repos "LLM framework"`
**不需要记命令。** Agent 自己知道该调什么。 **不需要记命令。** Agent 读了 SKILL.md 之后自己知道该调什么。
--- ---
@@ -135,27 +135,31 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
Agent Reach 做的事情很简单:**帮你把这些选型和配置的活儿做完了。** Agent Reach 做的事情很简单:**帮你把这些选型和配置的活儿做完了。**
安装完成后,Agent 直接调用上游工具(bird CLI、yt-dlp、mcporter、gh CLI 等),不需要经过 Agent Reach 的包装层。
### 🔌 每个渠道都是可插拔的 ### 🔌 每个渠道都是可插拔的
每个平台对应一个独立的 Python 文件,实现统一接口。**后端工具随时可以换**——哪天出了更好的工具,改一个文件就行,其他不用动。 每个平台背后是一个独立的上游工具。**不满意?换掉就行。**
``` ```
channels/ channels/
├── web.py → Jina Reader ← 可以换成 Firecrawl、Crawl4AI…… ├── web.py → Jina Reader ← 可以换成 Firecrawl、Crawl4AI……
├── twitter.py → bird ← 可以换成 Nitter、官方 API…… ├── twitter.py → bird ← 可以换成 Nitter、官方 API……
├── youtube.py → yt-dlp ← 可以换成 YouTube API、Whisper…… ├── youtube.py → yt-dlp ← 可以换成 YouTube API、Whisper……
├── github.py → gh CLI ← 可以换成 REST API、PyGithub…… ├── github.py → gh CLI ← 可以换成 REST API、PyGithub……
├── bilibili.py → yt-dlp ← 可以换成 bilibili-api…… ├── bilibili.py → yt-dlp ← 可以换成 bilibili-api……
├── reddit.py → JSON API + Exa ← 可以换成 PRAW、Pushshift…… ├── reddit.py → JSON API + Exa ← 可以换成 PRAW、Pushshift……
├── xiaohongshu.py → mcporter MCP ← 可以换成其他 XHS 工具…… ├── xiaohongshu.py → mcporter MCP ← 可以换成其他 XHS 工具……
├── instagram.py → instaloader ← 可以换成 instagrapi、官方 API…… ├── douyin.py → mcporter MCP ← 可以换成其他抖音工具……
├── linkedin.py → linkedin-mcp ← 可以换成 LinkedIn API…… ├── linkedin.py → linkedin-mcp ← 可以换成 LinkedIn API……
├── bosszhipin.py → mcp-bosszp ← 可以换成其他招聘工具…… ├── bosszhipin.py → mcp-bosszp ← 可以换成其他招聘工具……
├── rss.py → feedparser ← 可以换成 atoma…… ├── rss.py → feedparser ← 可以换成 atoma……
├── exa_search.py → mcporter MCP ← 可以换成 Tavily、SerpAPI…… ├── exa_search.py → mcporter MCP ← 可以换成 Tavily、SerpAPI……
└── __init__.py → 渠道注册 └── __init__.py → 渠道注册doctor 检测用)
``` ```
每个渠道文件只负责检测对应上游工具是否可用(`check()` 方法),给 `agent-reach doctor` 提供状态信息。实际的读取和搜索由 Agent 直接调用上游工具完成。
### 当前选型 ### 当前选型
| 场景 | 选型 | 为什么选它 | | 场景 | 选型 | 为什么选它 |
@@ -167,7 +171,7 @@ channels/
| GitHub | [gh CLI](https://cli.github.com) | 官方工具,认证后完整 API 能力 | | GitHub | [gh CLI](https://cli.github.com) | 官方工具,认证后完整 API 能力 |
| 读 RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python 生态标准选择,2.3K Star | | 读 RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python 生态标准选择,2.3K Star |
| 小红书 | [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) | ⭐9K+Go 语言,Docker 一键部署 | | 小红书 | [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) | ⭐9K+Go 语言,Docker 一键部署 |
| Instagram | [instaloader](https://github.com/instaloader/instaloader) | ⭐9.8KPython CLICookie 登录,免费 | | 抖音 | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP 服务,无需登录,视频解析 + 无水印下载 |
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | ⭐900+,MCP 服务,浏览器自动化 | | LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | ⭐900+,MCP 服务,浏览器自动化 |
| Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP 服务,支持职位搜索和打招呼 | | Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP 服务,支持职位搜索和打招呼 |
@@ -189,7 +193,7 @@ Agent Reach 在设计上重视安全:
### 🍪 Cookie 安全建议 ### 🍪 Cookie 安全建议
需要 Cookie 的平台(Twitter、小红书、Instagram)建议使用**专用小号**,不要用主账号。Cookie 等同于完整登录权限,用小号可以在凭据泄露时限制影响范围。 需要 Cookie 的平台(Twitter、小红书)建议使用**专用小号**,不要用主账号。Cookie 等同于完整登录权限,用小号可以在凭据泄露时限制影响范围。
### 📦 安装方式 ### 📦 安装方式
@@ -225,9 +229,63 @@ Agent Reach 在设计上重视安全:
Star 一下,下次需要的时候能找到。⭐ Star 一下,下次需要的时候能找到。⭐
---
## 常见问题 / FAQ
<details>
<summary><strong>AI Agent 怎么搜索 Twitter / X?不想付 API 费用</strong></summary>
Agent Reach 使用 [bird CLI](https://www.npmjs.com/package/@steipete/bird) 通过 Cookie 认证访问 Twitter,完全免费。安装 Agent Reach 后,用 Cookie-Editor 导出你的 Twitter Cookie,运行 `agent-reach configure twitter-cookies "your_cookies"` 即可。之后 Agent 就可以用 `bird search "关键词" --json` 搜索推文了。
</details>
<details>
<summary><strong>How to search Twitter/X with AI agent for free (no API)?</strong></summary>
Agent Reach uses the bird CLI with cookie auth — zero API fees. After installing, export your Twitter cookies with the Cookie-Editor extension, run `agent-reach configure twitter-cookies "your_cookies"`, then your agent can search with `bird search "query" --json`.
</details>
<details>
<summary><strong>Reddit 返回 403 / 服务器 IP 被封怎么办?</strong></summary>
Reddit 封锁数据中心 IP。配置一个住宅代理即可解决:`agent-reach configure proxy http://user:pass@ip:port`。推荐 Webshare ($1/月)。本地电脑一般不会遇到这个问题。
</details>
<details>
<summary><strong>How to get YouTube video transcripts for AI?</strong></summary>
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"` extracts video metadata; `yt-dlp --write-sub --skip-download "URL"` extracts subtitles. Uses yt-dlp under the hood, supports multiple languages. No API key needed.
</details>
<details>
<summary><strong>怎么让 AI Agent 读小红书?</strong></summary>
小红书需要通过 Docker 运行一个 MCP 服务。安装 Docker 后,运行 `agent-reach install` 会自动配置。之后 Agent 就能用 `mcporter call 'xiaohongshu.get_feed_detail(...)'` 读取笔记或 `mcporter call 'xiaohongshu.search_feeds(keyword: "关键词")'` 搜索了。
</details>
<details>
<summary><strong>怎么让 AI Agent 解析抖音视频?</strong></summary>
安装 douyin-mcp-server 后,Agent 就能用 `mcporter call 'douyin.parse_douyin_video_info(share_link: "分享链接")'` 解析视频信息、获取无水印下载链接。不需要登录,把抖音分享链接发给 Agent 就行。详见 https://github.com/yzfly/douyin-mcp-server
</details>
<details>
<summary><strong>Compatible with Claude Code / Cursor / OpenClaw / Windsurf?</strong></summary>
Yes! Agent Reach is an installer + configuration tool — any AI coding agent that can run shell commands can use it. Works with Claude Code, Cursor, OpenClaw, Windsurf, Codex, and more. Just `pip install agent-reach`, run `agent-reach install`, and the agent can start using the upstream tools immediately.
</details>
<details>
<summary><strong>Is this free? Any API costs?</strong></summary>
100% free. All backends are open-source tools (bird CLI, yt-dlp, Jina Reader, Exa, etc.) that don't require paid API keys. The only optional cost is a residential proxy (~$1/month) if you need Reddit/Bilibili access from a server.
</details>
---
## 致谢 ## 致谢
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://www.npmjs.com/package/@steipete/bird) · [Exa](https://exa.ai) · [mcporter](https://github.com/steipete/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) · [instaloader](https://github.com/instaloader/instaloader) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) [Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://www.npmjs.com/package/@steipete/bird) · [Exa](https://exa.ai) · [mcporter](https://github.com/steipete/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
## License ## License
-87
View File
@@ -1,87 +0,0 @@
---
name: agent-reach
description: >
Give your AI agent eyes to see the entire internet. Read and search across
Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, RSS, and any web page
— all from a single CLI. Use when: (1) reading content from URLs (tweets, Reddit posts,
articles, videos), (2) searching across platforms (web, Twitter, Reddit, GitHub, YouTube,
Bilibili, XiaoHongShu), (3) checking channel health or updating Agent Reach.
Triggers: "search Twitter/Reddit/YouTube", "read this URL", "find posts about",
"搜索", "读取", "查一下", "看看这个链接".
---
# Agent Reach
Read and search the internet across 9+ platforms via unified CLI.
## Setup
First check if agent-reach is installed:
```bash
agent-reach doctor
```
If command not found, install it:
```bash
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
agent-reach install --env=auto
```
`install` auto-detects your environment and installs all dependencies (Node.js, mcporter, bird CLI, gh CLI). Read the output and run `agent-reach doctor` to see what's active.
For channels that need user input, ask the user. See the full setup guide:
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
## Commands
### Read any URL
```bash
agent-reach read <url>
agent-reach read <url> --json # structured output
```
Handles: tweets, Reddit posts, articles, YouTube (transcripts), GitHub repos, etc.
### Search
```bash
agent-reach search "query" # web search (Exa)
agent-reach search-twitter "query" # Twitter/X
agent-reach search-reddit "query" # Reddit (--sub <subreddit>)
agent-reach search-github "query" # GitHub (--lang <language>)
agent-reach search-youtube "query" # YouTube
agent-reach search-bilibili "query" # Bilibili (B站)
agent-reach search-xhs "query" # XiaoHongShu (小红书)
```
All search commands support `-n <count>` for number of results.
### Management
```bash
agent-reach doctor # channel status overview
agent-reach watch # quick health + update check (for scheduled tasks)
agent-reach check-update # check for new versions
```
### Configure channels
```bash
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
agent-reach configure proxy http://user:pass@ip:port
agent-reach configure --from-browser chrome # auto-extract cookies
```
## Channel Status Tiers
- **Tier 0 (zero config):** Web, YouTube, RSS, Twitter (read-only via Jina)
- **Tier 1 (free setup):** Exa web search (mcporter required)
- **Tier 2 (user config):** Twitter search (cookie), Reddit full (proxy), GitHub (token), Bilibili (proxy), XiaoHongShu (MCP)
Run `agent-reach doctor` to see which channels are active.
## Tips
- Always try `agent-reach read <url>` first for any URL — it auto-detects the platform
- For Twitter cookies, recommend the user install [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome extension
- Reddit and Bilibili block server IPs — suggest a residential proxy (~$1/month) if on a server
- If a channel breaks, run `agent-reach doctor` to diagnose
+10 -27
View File
@@ -1,14 +1,10 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
""" """
Channel registry — routes URLs to the right channel. Channel registry — lists all supported platforms for doctor checks.
This is the core of Agent Reach' pluggable architecture.
Add a new channel: just create a file and register it here.
Swap a backend: just change the implementation inside the channel file.
""" """
from typing import Dict, List, Optional from typing import List, Optional
from .base import Channel, ReadResult, SearchResult from .base import Channel
# Import all channels # Import all channels
from .web import WebChannel from .web import WebChannel
@@ -20,12 +16,12 @@ from .rss import RSSChannel
from .bilibili import BilibiliChannel from .bilibili import BilibiliChannel
from .exa_search import ExaSearchChannel from .exa_search import ExaSearchChannel
from .xiaohongshu import XiaoHongShuChannel from .xiaohongshu import XiaoHongShuChannel
from .instagram import InstagramChannel from .douyin import DouyinChannel
from .linkedin import LinkedInChannel from .linkedin import LinkedInChannel
from .bosszhipin import BossZhipinChannel from .bosszhipin import BossZhipinChannel
# Channel registry — order matters (first match wins, web is last as fallback) # Channel registry
ALL_CHANNELS: List[Channel] = [ ALL_CHANNELS: List[Channel] = [
GitHubChannel(), GitHubChannel(),
TwitterChannel(), TwitterChannel(),
@@ -33,27 +29,14 @@ ALL_CHANNELS: List[Channel] = [
RedditChannel(), RedditChannel(),
BilibiliChannel(), BilibiliChannel(),
XiaoHongShuChannel(), XiaoHongShuChannel(),
InstagramChannel(), DouyinChannel(),
LinkedInChannel(), LinkedInChannel(),
BossZhipinChannel(), BossZhipinChannel(),
RSSChannel(), RSSChannel(),
ExaSearchChannel(), ExaSearchChannel(),
WebChannel(), # Fallback — handles any URL WebChannel(),
] ]
# Search-capable channels
SEARCH_CHANNELS: Dict[str, Channel] = {
ch.name: ch for ch in ALL_CHANNELS if ch.can_search()
}
def get_channel_for_url(url: str) -> Channel:
"""Find the right channel for a URL."""
for channel in ALL_CHANNELS:
if channel.can_handle(url):
return channel
return WebChannel() # Should never reach here, but just in case
def get_channel(name: str) -> Optional[Channel]: def get_channel(name: str) -> Optional[Channel]:
"""Get a channel by name.""" """Get a channel by name."""
@@ -69,7 +52,7 @@ def get_all_channels() -> List[Channel]:
__all__ = [ __all__ = [
"Channel", "ReadResult", "SearchResult", "Channel",
"ALL_CHANNELS", "SEARCH_CHANNELS", "ALL_CHANNELS",
"get_channel_for_url", "get_channel", "get_all_channels", "get_channel", "get_all_channels",
] ]
+11 -115
View File
@@ -1,110 +1,28 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
""" """
Channel base class — the universal interface for all platforms. Channel base class — platform availability checking.
Every channel (YouTube, Twitter, GitHub, etc.) implements this interface. Each channel represents a platform (YouTube, Twitter, GitHub, etc.)
The backend tool can be swapped anytime without changing anything else. and provides:
- can_handle(url) → does this URL belong to this platform?
- check(config) → is the upstream tool installed and configured?
Example: After installation, agents call upstream tools directly.
class YouTubeChannel(Channel):
name = "youtube"
backends = ["yt-dlp"] # current backend, can be swapped
async def read(self, url, config):
# Just call yt-dlp, return standardized dict
...
""" """
import shutil import shutil
from abc import ABC, abstractmethod from abc import ABC, abstractmethod
from dataclasses import dataclass from typing import List, Tuple
from typing import Any, Dict, List, Optional, Tuple
@dataclass
class ReadResult:
"""Standardized read result. Every channel returns this."""
title: str
content: str
url: str
author: str = ""
date: str = ""
platform: str = ""
extra: dict = None
def __post_init__(self):
self.extra = self.extra or {}
def to_dict(self) -> dict:
d = {
"title": self.title,
"content": self.content,
"url": self.url,
"platform": self.platform,
}
if self.author:
d["author"] = self.author
if self.date:
d["date"] = self.date
if self.extra:
d["extra"] = self.extra
return d
@dataclass
class SearchResult:
"""Standardized search result."""
title: str
url: str
snippet: str = ""
author: str = ""
date: str = ""
score: float = 0
extra: dict = None
def __post_init__(self):
self.extra = self.extra or {}
def to_dict(self) -> dict:
d = {
"title": self.title,
"url": self.url,
"snippet": self.snippet,
}
if self.author:
d["author"] = self.author
if self.date:
d["date"] = self.date
if self.extra:
d["extra"] = self.extra
return d
class Channel(ABC): class Channel(ABC):
""" """Base class for all channels."""
Base class for all channels.
Subclasses just need to implement:
- read(url, config) → ReadResult
- can_handle(url) → bool
- check(config) → (status, message)
Optionally:
- search(query, config, **kwargs) → list[SearchResult]
"""
name: str = "" # e.g. "youtube" name: str = "" # e.g. "youtube"
description: str = "" # e.g. "YouTube video transcripts" description: str = "" # e.g. "YouTube 视频和字幕"
backends: List[str] = [] # e.g. ["yt-dlp"] — what external tool is used backends: List[str] = [] # e.g. ["yt-dlp"] — what upstream tool is used
requires_config: List[str] = [] # e.g. ["reddit_proxy"]
requires_tools: List[str] = [] # e.g. ["yt-dlp"]
tier: int = 0 # 0=zero-config, 1=needs free key, 2=needs setup tier: int = 0 # 0=zero-config, 1=needs free key, 2=needs setup
@abstractmethod
async def read(self, url: str, config=None) -> ReadResult:
"""Read content from a URL. Must return ReadResult."""
...
@abstractmethod @abstractmethod
def can_handle(self, url: str) -> bool: def can_handle(self, url: str) -> bool:
"""Check if this channel can handle this URL.""" """Check if this channel can handle this URL."""
@@ -112,29 +30,7 @@ class Channel(ABC):
def check(self, config=None) -> Tuple[str, str]: def check(self, config=None) -> Tuple[str, str]:
""" """
Check if this channel is available. Check if this channel's upstream tool is available.
Returns (status, message) where status is 'ok'/'warn'/'off'/'error'. Returns (status, message) where status is 'ok'/'warn'/'off'/'error'.
""" """
# Check required tools
for tool in self.requires_tools:
if not shutil.which(tool):
return "off", f"需要安装:pip install {tool}"
# Check required config
for key in self.requires_config:
if config and not config.get(key):
return "off", f"需要配置 {key},运行 agent-reach setup"
return "ok", f"{''.join(self.backends) if self.backends else '内置'}" return "ok", f"{''.join(self.backends) if self.backends else '内置'}"
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
"""Search this platform. Override if supported."""
raise NotImplementedError(f"{self.name} does not support search")
def can_search(self) -> bool:
"""Whether this channel supports search."""
try:
# Check if search is overridden
return type(self).search is not Channel.search
except:
return False
+9 -190
View File
@@ -1,207 +1,26 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""Bilibili — via yt-dlp (same backend as YouTube). """Bilibili — check if yt-dlp is available."""
Backend: yt-dlp (https://github.com/yt-dlp/yt-dlp) import os
yt-dlp natively supports Bilibili — video info, subtitles, and search.
"""
import json
import shutil import shutil
import subprocess from .base import Channel
from urllib.parse import urlparse
from .base import Channel, ReadResult, SearchResult
from typing import List
class BilibiliChannel(Channel): class BilibiliChannel(Channel):
name = "bilibili" name = "bilibili"
description = "B站视频信息和字幕" description = "B站视频和字幕"
backends = ["yt-dlp"] backends = ["yt-dlp"]
requires_tools = ["yt-dlp"] tier = 1
tier = 0
def can_handle(self, url: str) -> bool: def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
d = urlparse(url).netloc.lower() d = urlparse(url).netloc.lower()
return "bilibili.com" in d or "b23.tv" in d return "bilibili.com" in d or "b23.tv" in d
def check(self, config=None): def check(self, config=None):
if not shutil.which("yt-dlp"): if not shutil.which("yt-dlp"):
return "off", "yt-dlp 未安装。安装:pip install yt-dlp" return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
proxy = config.get("bilibili_proxy") if config else None proxy = (config.get("bilibili_proxy") if config else None) or os.environ.get("BILIBILI_PROXY")
if proxy: if proxy:
return "ok", "已配置代理,完整可用" return "ok", "可提取视频信息和字幕(代理已配置)"
import os return "ok", "可提取视频信息和字幕(本地环境)。服务器可能需要代理"
is_server = bool(os.environ.get("SSH_CONNECTION") or os.path.exists("/etc/cloud"))
if is_server:
return "warn", "服务器 IP 可能被封,配置代理即可解决:agent-reach configure proxy URL"
return "ok", "本地直连可用"
async def read(self, url: str, config=None) -> ReadResult:
if not shutil.which("yt-dlp"):
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
proxy = config.get("bilibili_proxy") if config else None
# Get video info via yt-dlp
info = self._get_info(url, proxy)
if not info:
return ReadResult(
title="Bilibili",
content=f"⚠️ 无法获取视频信息: {url}\n服务器 IP 可能被封,配个代理:agent-reach configure proxy URL",
url=url, platform="bilibili",
)
title = info.get("title", url)
author = info.get("uploader", "")
desc = info.get("description", "")
# Try subtitles
subtitle = self._get_subtitles(url, proxy)
content = desc
if subtitle:
content += f"\n\n## 字幕\n{subtitle}"
return ReadResult(
title=title, content=content, url=url,
author=author, platform="bilibili",
extra={
"view_count": info.get("view_count"),
"like_count": info.get("like_count"),
"duration": info.get("duration_string"),
},
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
"""Search Bilibili.
Strategy:
1. Try yt-dlp bilisearch (works on local machines)
2. Fallback to Exa site:bilibili.com (works on servers)
"""
if not shutil.which("yt-dlp"):
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
limit = kwargs.get("limit", 5)
proxy = config.get("bilibili_proxy") if config else None
# Strategy 1: yt-dlp bilisearch
results = self._search_ytdlp(query, limit, proxy)
if results:
return results
# Strategy 2: Exa fallback (server-friendly)
results = self._search_exa(query, limit)
if results:
return results
return []
def _search_ytdlp(self, query: str, limit: int, proxy: str = None) -> List[SearchResult]:
"""Search via yt-dlp bilisearch (needs local/Chinese IP)."""
cmd = [
"yt-dlp", "--dump-json", "--no-download",
f"bilisearch{limit}:{query}",
]
if proxy:
cmd += ["--proxy", proxy]
try:
r = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
if r.returncode != 0:
return []
results = []
for line in r.stdout.strip().split("\n"):
if not line.strip():
continue
try:
d = json.loads(line)
vid = d.get("id", "")
url = d.get("webpage_url", f"https://www.bilibili.com/video/av{vid}")
results.append(SearchResult(
title=d.get("title", f"av{vid}"),
url=url,
snippet=f"👤 {d.get('uploader', '?')} · 👁 {d.get('view_count', '?')}",
extra={
"view_count": d.get("view_count"),
"uploader": d.get("uploader"),
"duration": d.get("duration_string"),
},
))
except json.JSONDecodeError:
continue
return results
except subprocess.TimeoutExpired:
return []
def _search_exa(self, query: str, limit: int) -> List[SearchResult]:
"""Fallback: search via Exa (site:bilibili.com). Works on any IP."""
try:
r = subprocess.run(
["mcporter", "call",
f'exa.web_search_exa(query: "site:bilibili.com {query}", numResults: {limit})'],
capture_output=True, text=True, timeout=30,
)
if r.returncode != 0:
return []
results = []
# Parse mcporter output: Title: / Author: / URL: / Text: blocks
title, author, url = "", "", ""
for line in r.stdout.split("\n"):
if line.startswith("Title: "):
title = line[7:].strip()
elif line.startswith("Author: "):
author = line[8:].strip()
elif line.startswith("URL: "):
url = line[5:].strip()
if url and "bilibili.com" in url:
results.append(SearchResult(
title=title or url,
url=url,
snippet=f"👤 {author}" if author else "(via Exa search)",
))
title, author, url = "", "", ""
return results
except Exception:
return []
def _get_info(self, url: str, proxy: str = None) -> dict:
cmd = ["yt-dlp", "--dump-json", "--no-download", url]
if proxy:
cmd += ["--proxy", proxy]
try:
r = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
if r.returncode == 0:
return json.loads(r.stdout)
except (subprocess.TimeoutExpired, json.JSONDecodeError):
pass
return {}
def _get_subtitles(self, url: str, proxy: str = None) -> str:
import tempfile
from pathlib import Path
with tempfile.TemporaryDirectory() as tmpdir:
cmd = [
"yt-dlp", "--write-sub", "--write-auto-sub",
"--sub-lang", "zh-Hans,zh,en",
"--skip-download", "--sub-format", "vtt",
"-o", f"{tmpdir}/%(id)s.%(ext)s", url,
]
if proxy:
cmd += ["--proxy", proxy]
try:
subprocess.run(cmd, capture_output=True, text=True, timeout=30)
for f in Path(tmpdir).glob("*.vtt"):
text = f.read_text(errors="replace")
lines = []
for line in text.split("\n"):
line = line.strip()
if not line or line.startswith("WEBVTT") or "-->" in line or line.isdigit():
continue
if line not in lines[-1:]:
lines.append(line)
return "\n".join(lines)
except subprocess.TimeoutExpired:
pass
return ""
+21 -163
View File
@@ -1,62 +1,9 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""Boss直聘 (BOSS Zhipin) — via mcp-bosszp (MCP) or Jina Reader fallback. """Boss直聘 — check if mcp-bosszp is available."""
Backend: mcp-bosszp (161 stars, FastMCP + Playwright)
Swap to: any Boss直聘 access tool
"""
import json
import shutil import shutil
import subprocess import subprocess
from urllib.parse import urlparse from .base import Channel
from .base import Channel, ReadResult, SearchResult
from typing import List
import requests
def _mcporter_has_bosszhipin() -> bool:
"""Check if mcporter has Boss直聘 MCP configured."""
if not shutil.which("mcporter"):
return False
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
# Check for various possible config names
out = r.stdout.lower()
return "boss" in out or "zhipin" in out or "bosszhipin" in out
except Exception:
return False
def _mcporter_call(expr: str, timeout: int = 30) -> str:
"""Call a Boss直聘 MCP tool via mcporter."""
r = subprocess.run(
["mcporter", "call", expr],
capture_output=True, text=True, timeout=timeout,
)
if r.returncode != 0:
raise RuntimeError(r.stderr or r.stdout)
return r.stdout
def _get_mcp_name() -> str:
"""Get the actual MCP server name configured in mcporter."""
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
for line in r.stdout.split("\n"):
line_lower = line.strip().lower()
for name in ["bosszhipin", "boss-zp", "bosszp", "boss"]:
if name in line_lower:
# Extract the actual server name
parts = line.strip().split()
if parts:
return parts[0]
return "bosszhipin"
except Exception:
return "bosszhipin"
class BossZhipinChannel(Channel): class BossZhipinChannel(Channel):
@@ -66,118 +13,29 @@ class BossZhipinChannel(Channel):
tier = 2 tier = 2
def can_handle(self, url: str) -> bool: def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
domain = urlparse(url).netloc.lower() domain = urlparse(url).netloc.lower()
return "zhipin.com" in domain or "boss.com" in domain return "zhipin.com" in domain or "boss.com" in domain
def check(self, config=None): def check(self, config=None):
if _mcporter_has_bosszhipin(): if not shutil.which("mcporter"):
return "ok", "可搜索职位、向 HR 打招呼" return "off", (
"可通过 Jina Reader 读取职位页面。完整功能需要:\n"
" 1. git clone https://github.com/mucsbr/mcp-bosszp.git\n"
" 2. cd mcp-bosszp && pip install -r requirements.txt && playwright install chromium\n"
" 3. python boss_zhipin_fastmcp_v2.py(启动后扫码登录)\n"
" 4. mcporter config add bosszhipin http://localhost:8000/mcp"
)
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
out = r.stdout.lower()
if "boss" in out or "zhipin" in out:
return "ok", "可搜索职位、向 HR 打招呼"
except Exception:
pass
return "off", ( return "off", (
"可通过 Jina Reader 读取职位页面。完整功能需要:\n" "mcporter 已装但 Boss直聘 MCP 未配置。\n"
" 1. git clone https://github.com/mucsbr/mcp-bosszp.git\n"
" 2. cd mcp-bosszp && pip install -r requirements.txt && playwright install chromium\n"
" 3. python boss_zhipin_fastmcp_v2.py(启动后扫码登录)\n"
" 4. mcporter config add bosszhipin http://localhost:8000/mcp\n"
" 或用 Dockerdocker-compose up -d\n"
" 详见 https://github.com/mucsbr/mcp-bosszp" " 详见 https://github.com/mucsbr/mcp-bosszp"
) )
async def read(self, url: str, config=None) -> ReadResult:
# Boss直聘 pages mostly work with Jina Reader
return await self._read_jina(url)
async def _read_jina(self, url: str) -> ReadResult:
"""Read Boss直聘 page via Jina Reader."""
try:
resp = requests.get(
f"https://r.jina.ai/{url}",
headers={"Accept": "text/markdown"},
timeout=15,
)
resp.raise_for_status()
text = resp.text
if len(text.strip()) < 50:
return ReadResult(
title="Boss直聘",
content=(
f"⚠️ 无法读取此页面内容: {url}\n\n"
"提示:\n"
"- 安装 mcp-bosszp 可解锁职位搜索和自动打招呼\n"
"- 详见 https://github.com/mucsbr/mcp-bosszp"
),
url=url,
platform="bosszhipin",
)
return ReadResult(
title=text[:100] if text else url,
content=text,
url=url,
platform="bosszhipin",
)
except Exception:
return ReadResult(
title="Boss直聘",
content=(
f"⚠️ 无法读取此 Boss直聘页面: {url}\n\n"
"提示:\n"
"- Boss直聘部分页面需要登录\n"
"- 安装 mcp-bosszp 可解锁完整功能\n"
"- 详见 https://github.com/mucsbr/mcp-bosszp"
),
url=url,
platform="bosszhipin",
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
limit = kwargs.get("limit", 10)
# Try MCP search first
if _mcporter_has_bosszhipin():
try:
return await self._search_mcp(query, limit, config)
except Exception:
pass
# Fallback to Exa
from agent_reach.channels.exa_search import ExaSearchChannel
exa = ExaSearchChannel()
return await exa.search(f"site:zhipin.com {query}", config=config, limit=limit)
async def _search_mcp(self, query: str, limit: int, config=None) -> List[SearchResult]:
"""Search Boss直聘 via MCP."""
server = _get_mcp_name()
try:
out = _mcporter_call(
f'{server}.get_recommend_jobs_tool(page: 1)',
timeout=30,
)
return self._parse_jobs(out, limit)
except Exception:
return []
def _parse_jobs(self, text: str, limit: int) -> List[SearchResult]:
"""Parse MCP job search output into SearchResults."""
results = []
try:
data = json.loads(text)
jobs = data if isinstance(data, list) else data.get("jobs", data.get("results", []))
for job in jobs[:limit]:
if isinstance(job, dict):
title = job.get("title") or job.get("jobName", "")
company = job.get("company") or job.get("brandName", "")
salary = job.get("salary") or job.get("salaryDesc", "")
url = job.get("url", "")
snippet = f"🏢 {company}" if company else ""
if salary:
snippet += f" · 💰 {salary}"
results.append(SearchResult(
title=title,
url=url,
snippet=snippet,
))
except (json.JSONDecodeError, KeyError):
pass
return results
+52
View File
@@ -0,0 +1,52 @@
# -*- coding: utf-8 -*-
"""Douyin (抖音) — check if mcporter + douyin-mcp-server is available."""
import shutil
import subprocess
from .base import Channel
class DouyinChannel(Channel):
name = "douyin"
description = "抖音短视频"
backends = ["douyin-mcp-server"]
tier = 2
def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
d = urlparse(url).netloc.lower()
return "douyin.com" in d or "iesdouyin.com" in d
def check(self, config=None):
if not shutil.which("mcporter"):
return "off", (
"需要 mcporter + douyin-mcp-server。安装步骤:\n"
" 1. npm install -g mcporter\n"
" 2. pip install douyin-mcp-server\n"
" 3. 启动服务(见下方说明)\n"
" 4. mcporter config add douyin http://localhost:18070/mcp\n"
" 详见 https://github.com/yzfly/douyin-mcp-server"
)
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
if "douyin" not in r.stdout:
return "off", (
"mcporter 已装但抖音 MCP 未配置。运行:\n"
" pip install douyin-mcp-server\n"
" # 启动服务后:\n"
" mcporter config add douyin http://localhost:18070/mcp"
)
except Exception:
return "off", "mcporter 连接异常"
try:
r = subprocess.run(
["mcporter", "call", "douyin.parse_douyin_video_info(share_link: \"https://www.douyin.com\")"],
capture_output=True, text=True, timeout=15
)
if r.returncode == 0:
return "ok", "完整可用(视频解析、下载链接获取)"
return "warn", "MCP 已连接但调用异常,检查 douyin-mcp-server 服务是否在运行"
except Exception:
return "warn", "MCP 连接异常,检查 douyin-mcp-server 服务是否在运行"
+19 -93
View File
@@ -1,110 +1,36 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""Exa semantic search — via mcporter + Exa MCP server. """Exa Search — check if mcporter + Exa MCP is available."""
Backend: Exa MCP at mcp.exa.ai (OAuth, no API key needed)
Requires: mcporter CLI
"""
import json
import shutil import shutil
import subprocess import subprocess
from .base import Channel, SearchResult from .base import Channel
from typing import List
class ExaSearchChannel(Channel): class ExaSearchChannel(Channel):
name = "exa_search" name = "exa_search"
description = "全网语义搜索(同时支持 Reddit/Twitter 搜索)" description = "全网语义搜索"
backends = ["exa-mcp"] backends = ["Exa via mcporter"]
tier = 1 tier = 0
def _mcporter_ok(self) -> bool:
if not shutil.which("mcporter"):
return False
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
return "exa" in r.stdout
except Exception:
return False
def _call(self, expr: str, timeout: int = 30) -> str:
r = subprocess.run(
["mcporter", "call", expr],
capture_output=True, text=True, timeout=timeout,
)
if r.returncode != 0:
raise RuntimeError(r.stderr or r.stdout)
return r.stdout
# ── Channel interface ──
def can_handle(self, url: str) -> bool: def can_handle(self, url: str) -> bool:
return False # search-only return False # Search-only channel
async def read(self, url: str, config=None):
raise NotImplementedError("Exa is a search engine, not a reader")
def check(self, config=None): def check(self, config=None):
if not shutil.which("mcporter"): if not shutil.which("mcporter"):
return "off", ( return "off", (
"需要 mcporter。安装:npm install -g mcporter && " "需要 mcporter + Exa MCP。安装:\n"
"mcporter config add exa https://mcp.exa.ai/mcp"
)
if not self._mcporter_ok():
return "off", "mcporter 已装但 Exa 未配置。运行:mcporter config add exa https://mcp.exa.ai/mcp"
return "ok", "MCP 已连接,免 Key 直接可用(全网搜索 + Reddit + Twitter"
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
if not self._mcporter_ok():
raise ValueError(
"Exa 搜索需要 mcporter。安装:\n"
" npm install -g mcporter\n" " npm install -g mcporter\n"
" mcporter config add exa https://mcp.exa.ai/mcp" " mcporter config add exa https://mcp.exa.ai/mcp"
) )
try:
limit = kwargs.get("limit", 5) r = subprocess.run(
safe_q = query.replace('"', '\\"') ["mcporter", "list"], capture_output=True, text=True, timeout=10
out = self._call( )
f'exa.web_search_exa(query: "{safe_q}", numResults: {min(limit, 10)})', if "exa" in r.stdout.lower():
timeout=30, return "ok", "全网语义搜索可用(免费,无需 API Key)"
) return "off", (
return self._parse_output(out, limit) "mcporter 已装但 Exa 未配置。运行:\n"
" mcporter config add exa https://mcp.exa.ai/mcp"
# ── Parse mcporter text output ── )
except Exception:
def _parse_output(self, text: str, limit: int) -> List[SearchResult]: return "off", "mcporter 连接异常"
"""Parse mcporter's Title/URL/Text block format."""
results = []
cur = {}
for line in text.split("\n"):
line = line.strip()
if line.startswith("Title: "):
if cur.get("title"):
results.append(self._make_result(cur))
cur = {"title": line[7:]}
elif line.startswith("URL: "):
cur["url"] = line[5:]
elif line.startswith("Published Date: "):
cur["date"] = line[16:]
elif line.startswith("Text: "):
cur["text"] = line[6:]
elif "text" in cur and line:
cur["text"] += " " + line
if cur.get("title"):
results.append(self._make_result(cur))
return results[:limit]
@staticmethod
def _make_result(d: dict) -> SearchResult:
return SearchResult(
title=d.get("title", ""),
url=d.get("url", ""),
snippet=d.get("text", "")[:500],
date=d.get("date", ""),
score=0,
)
+8 -118
View File
@@ -1,16 +1,9 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""GitHub — via gh CLI. """GitHub — check if gh CLI is available."""
Backend: gh CLI (https://cli.github.com)
Swap to: GitHub REST API
"""
import json
import shutil import shutil
import subprocess import subprocess
from urllib.parse import urlparse from .base import Channel
from .base import Channel, ReadResult, SearchResult
from typing import List
class GitHubChannel(Channel): class GitHubChannel(Channel):
@@ -19,121 +12,18 @@ class GitHubChannel(Channel):
backends = ["gh CLI"] backends = ["gh CLI"]
tier = 0 tier = 0
def _gh(self, args: list, timeout: int = 15) -> str:
r = subprocess.run(
["gh"] + args,
capture_output=True, text=True, timeout=timeout,
)
if r.returncode != 0:
raise RuntimeError(r.stderr or r.stdout)
return r.stdout
def _gh_json(self, args: list, timeout: int = 15) -> dict:
return json.loads(self._gh(args + ["--json"], timeout))
def can_handle(self, url: str) -> bool: def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
return "github.com" in urlparse(url).netloc.lower() return "github.com" in urlparse(url).netloc.lower()
def check(self, config=None): def check(self, config=None):
if not shutil.which("gh"): if not shutil.which("gh"):
return "warn", "gh CLI 未安装。安装:https://cli.github.com 。公开仓库仍可通过 Jina Reader 读取" return "warn", "gh CLI 未安装。安装:https://cli.github.com"
try: try:
self._gh(["auth", "status"], timeout=5) subprocess.run(
["gh", "auth", "status"],
capture_output=True, text=True, timeout=5
)
return "ok", "完整可用(读取、搜索、Fork、Issue、PR 等)" return "ok", "完整可用(读取、搜索、Fork、Issue、PR 等)"
except Exception: except Exception:
return "ok", "gh CLI 已装但未认证。运行 gh auth login 可解锁完整功能" return "ok", "gh CLI 已装但未认证。运行 gh auth login 可解锁完整功能"
async def read(self, url: str, config=None) -> ReadResult:
if not shutil.which("gh"):
# Fallback to Jina Reader for public repos
from agent_reach.channels.web import WebChannel
return await WebChannel().read(url, config)
path = urlparse(url).path.strip("/").split("/")
if len(path) < 2:
from agent_reach.channels.web import WebChannel
return await WebChannel().read(url, config)
owner, repo = path[0], path[1]
# Issues / PRs
if len(path) >= 4 and path[2] in ("issues", "pull"):
return await self._read_issue(owner, repo, path[3], url)
# Repo
return await self._read_repo(owner, repo, url)
async def _read_repo(self, owner: str, repo: str, url: str) -> ReadResult:
slug = f"{owner}/{repo}"
try:
# Get repo info
info = self._gh(["repo", "view", slug])
# Get README
try:
readme = self._gh(
["api", f"repos/{slug}/readme", "--jq", ".content"],
timeout=10,
)
import base64
readme_text = base64.b64decode(readme).decode("utf-8", errors="replace")
except Exception:
readme_text = ""
content = readme_text or info
return ReadResult(
title=slug, content=content, url=url,
author=owner, platform="github",
)
except Exception:
from agent_reach.channels.web import WebChannel
return await WebChannel().read(url)
async def _read_issue(self, owner: str, repo: str, num: str, url: str) -> ReadResult:
slug = f"{owner}/{repo}"
try:
out = self._gh(["issue", "view", num, "-R", slug])
return ReadResult(
title=f"{slug}#{num}", content=out, url=url,
platform="github",
)
except Exception:
# Might be a PR
try:
out = self._gh(["pr", "view", num, "-R", slug])
return ReadResult(
title=f"{slug}#{num}", content=out, url=url,
platform="github",
)
except Exception:
from agent_reach.channels.web import WebChannel
return await WebChannel().read(url)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
if not shutil.which("gh"):
raise ValueError("GitHub search requires gh CLI. Install: https://cli.github.com")
language = kwargs.get("language")
limit = kwargs.get("limit", 5)
args = ["search", "repos", query, "--sort", "stars", f"--limit={limit}"]
if language:
args += [f"--language={language}"]
out = self._gh(args, timeout=15)
results = []
for line in out.strip().split("\n"):
if not line.strip():
continue
parts = line.split("\t")
if len(parts) >= 1:
slug = parts[0].strip()
desc = parts[1].strip() if len(parts) > 1 else ""
stars = parts[3].strip() if len(parts) > 3 else ""
lang = parts[5].strip() if len(parts) > 5 else ""
results.append(SearchResult(
title=slug,
url=f"https://github.com/{slug}",
snippet=desc,
extra={"stars": stars, "language": lang},
))
return results
-248
View File
@@ -1,248 +0,0 @@
# -*- coding: utf-8 -*-
"""Instagram — via instaloader (free, open source).
Backend: instaloader (9.8K stars, Python CLI + library)
Swap to: any Instagram access tool
"""
import re
import shutil
import subprocess
from pathlib import Path
from urllib.parse import urlparse
from .base import Channel, ReadResult, SearchResult
from typing import List
class InstagramChannel(Channel):
name = "instagram"
description = "Instagram 帖子和 Profile"
backends = ["instaloader"]
tier = 2 # Needs login for full access
def can_handle(self, url: str) -> bool:
domain = urlparse(url).netloc.lower()
return "instagram.com" in domain or "instagr.am" in domain
def check(self, config=None):
# Check both CLI and Python module
has_cli = shutil.which("instaloader")
has_module = False
try:
import instaloader
has_module = True
except ImportError:
pass
if not has_cli and not has_module:
return "off", (
"需要安装 instaloaderpip install instaloader\n"
" 安装后可读取 Instagram 帖子和 Profile\n"
" 登录: agent-reach configure instagram-cookies \"sessionid=xxx; csrftoken=yyy; ...\""
)
# Check if cookies are configured
cookie_file = Path.home() / ".agent-reach" / "instagram-cookies.txt"
if cookie_file.exists():
return "ok", "已登录,可读取 Instagram 帖子和 Profile"
return "ok", "可读取公开帖子和 Profile。登录可访问更多内容:\n agent-reach configure instagram-cookies \"sessionid=xxx; csrftoken=yyy; ...\""
async def read(self, url: str, config=None) -> ReadResult:
# Try instaloader (module or CLI)
try:
import instaloader
return await self._read_instaloader(url, config)
except ImportError:
pass
# Fallback: Jina Reader
return await self._read_jina(url)
async def _read_instaloader(self, url: str, config=None) -> ReadResult:
"""Read Instagram content using instaloader Python API."""
import asyncio
import concurrent.futures
def _sync_read():
import instaloader
L = instaloader.Instaloader(
download_pictures=False,
download_videos=False,
download_video_thumbnails=False,
download_geotags=False,
download_comments=False,
save_metadata=False,
compress_json=False,
max_connection_attempts=1, # Don't retry on rate limit
)
# Try to load session: cookie file > saved session
cookie_file = Path.home() / ".agent-reach" / "instagram-cookies.txt"
if cookie_file.exists():
try:
cookie_str = cookie_file.read_text().strip()
cookies = {}
for part in cookie_str.split(";"):
part = part.strip()
if "=" in part:
k, v = part.split("=", 1)
cookies[k.strip()] = v.strip()
if "sessionid" in cookies and "csrftoken" in cookies:
# Extract username from ds_user_id or use generic
username = cookies.get("ds_user_id", "user")
L.context.load_session(username, cookies)
except Exception:
pass
elif config and config.get("instagram_username"):
try:
L.load_session_from_file(config.get("instagram_username"))
except Exception:
pass
path = urlparse(url).path.strip("/")
if "/p/" in url or "/reel/" in url:
return self._read_post_sync(L, url, path)
else:
return self._read_profile_sync(L, url, path)
try:
# Run with 15s timeout to avoid instaloader's 30-min retry
loop = asyncio.get_event_loop()
with concurrent.futures.ThreadPoolExecutor() as pool:
result = await asyncio.wait_for(
loop.run_in_executor(pool, _sync_read),
timeout=15,
)
return result
except (asyncio.TimeoutError, Exception):
# Any error or timeout → Jina fallback
return await self._read_jina(url)
def _read_post_sync(self, L, url: str, path: str) -> ReadResult:
"""Read a single Instagram post (sync, runs in executor)."""
import instaloader
# Extract shortcode from URL
match = re.search(r"/(?:p|reel)/([A-Za-z0-9_-]+)", url)
if not match:
raise ValueError("Cannot extract shortcode from URL")
shortcode = match.group(1)
try:
post = instaloader.Post.from_shortcode(L.context, shortcode)
lines = []
if post.caption:
lines.append(post.caption)
lines.append("")
lines.append(f"👤 @{post.owner_username}")
lines.append(f"❤️ {post.likes} likes")
if post.comments:
lines.append(f"💬 {post.comments} comments")
lines.append(f"📅 {post.date_utc.strftime('%Y-%m-%d %H:%M')}")
if post.location:
lines.append(f"📍 {post.location}")
if post.hashtags:
lines.append(f"#️⃣ {' '.join('#' + h for h in post.hashtags)}")
return ReadResult(
title=f"@{post.owner_username}: {(post.caption or '')[:80]}",
content="\n".join(lines),
url=url,
author=f"@{post.owner_username}",
date=post.date_utc.strftime("%Y-%m-%d"),
platform="instagram",
extra={"likes": post.likes, "comments": post.comments},
)
except Exception:
raise # Let executor timeout handle fallback
def _read_profile_sync(self, L, url: str, path: str) -> ReadResult:
"""Read an Instagram profile (sync, runs in executor)."""
import instaloader
# Extract username from path
username = path.split("/")[0] if path else ""
if not username or username in ("p", "reel", "stories", "explore"):
raise ValueError("Cannot extract username from URL")
try:
profile = instaloader.Profile.from_username(L.context, username)
lines = []
lines.append(f"👤 {profile.full_name} (@{profile.username})")
if profile.biography:
lines.append(f"📝 {profile.biography}")
if profile.external_url:
lines.append(f"🔗 {profile.external_url}")
lines.append("")
lines.append(f"📊 {profile.mediacount} posts · "
f"{profile.followers} followers · "
f"{profile.followees} following")
if profile.is_verified:
lines.append("✅ Verified")
if profile.is_business_account and profile.business_category_name:
lines.append(f"🏢 {profile.business_category_name}")
# Get recent posts (up to 5)
lines.append("")
lines.append("📸 Recent posts:")
count = 0
for post in profile.get_posts():
if count >= 5:
break
caption = (post.caption or "")[:100].replace("\n", " ")
lines.append(f" • ❤️{post.likes} | {post.date_utc.strftime('%m-%d')} | {caption}")
count += 1
return ReadResult(
title=f"{profile.full_name} (@{profile.username}) - Instagram",
content="\n".join(lines),
url=url,
author=f"@{profile.username}",
platform="instagram",
extra={
"followers": profile.followers,
"posts": profile.mediacount,
},
)
except Exception:
raise # Let executor timeout handle fallback
async def _read_jina(self, url: str) -> ReadResult:
"""Fallback: use Jina Reader."""
import requests
try:
resp = requests.get(
f"https://r.jina.ai/{url}",
headers={"Accept": "text/markdown"},
timeout=15,
)
resp.raise_for_status()
text = resp.text
return ReadResult(
title=text[:100] if text else url,
content=text,
url=url,
platform="instagram",
)
except Exception:
return ReadResult(
title="Instagram",
content=(
f"⚠️ 无法读取此 Instagram 内容: {url}\n\n"
"提示:\n"
"- 确保 URL 正确\n"
"- 安装 instaloader: pip install instaloader\n"
"- 登录以访问更多内容: instaloader --login YOUR_USERNAME"
),
url=url,
platform="instagram",
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
"""Search Instagram via Exa."""
limit = kwargs.get("limit", 10)
from agent_reach.channels.exa_search import ExaSearchChannel
exa = ExaSearchChannel()
return await exa.search(f"site:instagram.com {query}", config=config, limit=limit)
+22 -251
View File
@@ -1,268 +1,39 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""LinkedIn — via linkedin-scraper-mcp (MCP) or Jina Reader fallback. """LinkedIn — check if linkedin-scraper-mcp is available."""
Backend: linkedin-scraper-mcp (916 stars, Patchright browser automation)
Swap to: any LinkedIn access tool
"""
import shutil import shutil
import subprocess import subprocess
from urllib.parse import urlparse from .base import Channel
from .base import Channel, ReadResult, SearchResult
from typing import List
import requests
def _mcporter_has_linkedin() -> bool:
"""Check if mcporter has linkedin MCP configured."""
if not shutil.which("mcporter"):
return False
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
return "linkedin" in r.stdout.lower()
except Exception:
return False
def _mcporter_call(expr: str, timeout: int = 30) -> str:
"""Call a LinkedIn MCP tool via mcporter."""
r = subprocess.run(
["mcporter", "call", expr],
capture_output=True, text=True, timeout=timeout,
)
if r.returncode != 0:
raise RuntimeError(r.stderr or r.stdout)
return r.stdout
class LinkedInChannel(Channel): class LinkedInChannel(Channel):
name = "linkedin" name = "linkedin"
description = "LinkedIn 个人/公司 Profile 和职位" description = "LinkedIn 职业社交"
backends = ["linkedin-scraper-mcp", "Jina Reader"] backends = ["linkedin-scraper-mcp", "Jina Reader"]
tier = 2 tier = 2
def can_handle(self, url: str) -> bool: def can_handle(self, url: str) -> bool:
domain = urlparse(url).netloc.lower() from urllib.parse import urlparse
return "linkedin.com" in domain return "linkedin.com" in urlparse(url).netloc.lower()
def check(self, config=None): def check(self, config=None):
if _mcporter_has_linkedin(): if not shutil.which("mcporter"):
return "ok", "完整可用(Profile、公司、职位搜索)" return "off", (
"基本内容可通过 Jina Reader 读取。完整功能需要:\n"
# Check if linkedin-scraper-mcp is installed as CLI " pip install linkedin-scraper-mcp\n"
if shutil.which("linkedin-scraper-mcp"): " mcporter config add linkedin http://localhost:3000/mcp\n"
return "warn", ( " 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
"linkedin-scraper-mcp 已安装但未接入 mcporter。运行:\n"
" 1. linkedin-scraper-mcp --login(在有浏览器的机器上登录)\n"
" 2. linkedin-scraper-mcp --transport streamable-http --port 8001\n"
" 3. mcporter config add linkedin http://localhost:8001/mcp"
) )
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
if "linkedin" in r.stdout.lower():
return "ok", "完整可用(Profile、公司、职位搜索)"
except Exception:
pass
return "off", ( return "off", (
"可通过 Jina Reader 读取部分内容。完整功能需要\n" "mcporter 已装但 LinkedIn MCP 未配置。运行\n"
" 1. pip install linkedin-scraper-mcp\n" " pip install linkedin-scraper-mcp\n"
" 2. linkedin-scraper-mcp --login(在有浏览器的机器上登录)\n" " mcporter config add linkedin http://localhost:3000/mcp"
" 3. linkedin-scraper-mcp --transport streamable-http --port 8001\n"
" 4. mcporter config add linkedin http://localhost:8001/mcp\n"
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
) )
async def read(self, url: str, config=None) -> ReadResult:
path = urlparse(url).path.strip("/")
# Try MCP first
if _mcporter_has_linkedin():
try:
if "/in/" in url:
return await self._read_profile_mcp(url)
elif "/company/" in url:
return await self._read_company_mcp(url)
elif "/jobs/view/" in url:
return await self._read_job_mcp(url)
except Exception:
pass # Fall through to Jina
# Fallback: Jina Reader
return await self._read_jina(url)
async def _read_profile_mcp(self, url: str) -> ReadResult:
"""Read a LinkedIn profile via MCP."""
import re
# Extract username from URL: /in/username/
match = re.search(r"/in/([^/]+)", url)
if not match:
return await self._read_jina(url)
username = match.group(1)
safe_username = username.replace('"', '\\"')
out = _mcporter_call(
f'linkedin.get_person_profile(linkedin_username: "{safe_username}")',
timeout=60,
)
return ReadResult(
title=self._extract_title(out) or f"LinkedIn Profile - {username}",
content=out.strip(),
url=url,
platform="linkedin",
)
async def _read_company_mcp(self, url: str) -> ReadResult:
"""Read a LinkedIn company page via MCP."""
import re
# Extract company name from URL: /company/name/
match = re.search(r"/company/([^/]+)", url)
if not match:
return await self._read_jina(url)
company = match.group(1)
safe_company = company.replace('"', '\\"')
out = _mcporter_call(
f'linkedin.get_company_profile(company_name: "{safe_company}")',
timeout=60,
)
return ReadResult(
title=self._extract_title(out) or "LinkedIn Company",
content=out.strip(),
url=url,
platform="linkedin",
)
async def _read_job_mcp(self, url: str) -> ReadResult:
"""Read a LinkedIn job posting via MCP."""
import re
match = re.search(r"/jobs/view/(\d+)", url)
if not match:
return await self._read_jina(url)
job_id = match.group(1)
out = _mcporter_call(
f'linkedin.get_job_details(job_id: "{job_id}")',
timeout=30,
)
return ReadResult(
title=self._extract_title(out) or f"LinkedIn Job {job_id}",
content=out.strip(),
url=url,
platform="linkedin",
)
async def _read_jina(self, url: str) -> ReadResult:
"""Fallback: use Jina Reader."""
try:
resp = requests.get(
f"https://r.jina.ai/{url}",
headers={"Accept": "text/markdown"},
timeout=15,
)
resp.raise_for_status()
text = resp.text
# Check if content is usable
if len(text.strip()) < 100 or "Sign in" in text[:200]:
return ReadResult(
title="LinkedIn",
content=(
f"⚠️ LinkedIn 页面需要登录才能完整查看。\n\n"
f"URL: {url}\n\n"
"完整功能需安装 linkedin-scraper-mcp\n"
" pip install linkedin-scraper-mcp\n"
" uvx linkedin-scraper-mcp --login\n"
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
),
url=url,
platform="linkedin",
)
return ReadResult(
title=text[:100] if text else url,
content=text,
url=url,
platform="linkedin",
)
except Exception:
return ReadResult(
title="LinkedIn",
content=(
f"⚠️ 无法读取此 LinkedIn 页面: {url}\n\n"
"提示:\n"
"- LinkedIn 需要登录才能查看大部分内容\n"
"- 安装 linkedin-scraper-mcp 解锁完整功能\n"
"- 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
),
url=url,
platform="linkedin",
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
limit = kwargs.get("limit", 10)
# Try MCP search first
if _mcporter_has_linkedin():
try:
return await self._search_mcp(query, limit)
except Exception:
pass
# Fallback to Exa
from agent_reach.channels.exa_search import ExaSearchChannel
exa = ExaSearchChannel()
return await exa.search(f"site:linkedin.com {query}", config=config, limit=limit)
async def _search_mcp(self, query: str, limit: int) -> List[SearchResult]:
"""Search LinkedIn via MCP."""
safe_q = query.replace('"', '\\"')
# Try job search first (most common use case)
try:
out = _mcporter_call(
f'linkedin.search_jobs(keywords: "{safe_q}")',
timeout=60,
)
results = self._parse_search_results(out, "job")
if results:
return results[:limit]
except Exception:
pass
# Try people search
try:
out = _mcporter_call(
f'linkedin.search_people(keywords: "{safe_q}")',
timeout=60,
)
results = self._parse_search_results(out, "people")
if results:
return results
except Exception:
pass
return []
def _parse_search_results(self, text: str, result_type: str) -> List[SearchResult]:
"""Parse MCP search output into SearchResults."""
import json
results = []
try:
data = json.loads(text)
items = data if isinstance(data, list) else data.get("results", data.get("jobs", []))
for item in items:
if isinstance(item, dict):
title = item.get("title") or item.get("name") or item.get("headline", "")
url = item.get("url") or item.get("link", "")
snippet = item.get("description") or item.get("company", "")
results.append(SearchResult(
title=title,
url=url,
snippet=snippet[:200] if snippet else "",
))
except (json.JSONDecodeError, KeyError):
# Try line-by-line parsing
pass
return results
def _extract_title(self, text: str) -> str:
"""Extract a title from MCP output."""
for line in text.split("\n"):
line = line.strip()
if line and not line.startswith(("{", "[", "#", "http")):
return line[:80]
return ""
+13 -165
View File
@@ -1,178 +1,26 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""Reddit — via Reddit JSON API + optional proxy. """Reddit — check if proxy and credentials are configured."""
Backend: Reddit public JSON API (append .json to any URL)
Swap to: any Reddit access method
"""
import os import os
import requests from .base import Channel
from urllib.parse import urlparse
from .base import Channel, ReadResult
class RedditChannel(Channel): class RedditChannel(Channel):
name = "reddit" name = "reddit"
description = "Reddit 帖子和评论" description = "Reddit 帖子和评论"
backends = ["Reddit JSON API"] backends = ["JSON API", "Exa"]
tier = 2 tier = 1
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
def can_handle(self, url: str) -> bool: def can_handle(self, url: str) -> bool:
domain = urlparse(url).netloc.lower() from urllib.parse import urlparse
return "reddit.com" in domain or "redd.it" in domain d = urlparse(url).netloc.lower()
return "reddit.com" in d or "redd.it" in d
def check(self, config=None): def check(self, config=None):
proxy = config.get("reddit_proxy") if config else None proxy = (config.get("reddit_proxy") if config else None) or os.environ.get("REDDIT_PROXY")
has_bot = bool(os.environ.get("REDDIT_CLIENT_ID")) if proxy:
if proxy and has_bot: return "ok", "代理已配置,可读取帖子。搜索走 Exa"
return "ok", "完整可用(代理 + OAuth Bot" return "warn", (
elif proxy: "无代理。服务器 IP 可能被 Reddit 封锁。配置代理:\n"
return "ok", "代理已配置,可读取帖子。配置 REDDIT_CLIENT_ID/SECRET 可解锁高级搜索和发帖" " agent-reach configure proxy http://user:pass@ip:port"
elif has_bot:
return "warn", "OAuth Bot 已配置,但服务器直连可能被封。配个代理更稳定:agent-reach configure proxy URL"
else:
return "off", "搜索用 Exa 免费可用。读帖子需配个代理:agent-reach configure proxy URL"
async def read(self, url: str, config=None) -> ReadResult:
proxy = config.get("reddit_proxy") if config else None
proxies = {"http": proxy, "https": proxy} if proxy else None
# Clean URL: remove query params, trailing slash, then add .json
parsed = urlparse(url)
clean_path = parsed.path.rstrip("/")
# Remove trailing .json if already present (avoid double .json)
if clean_path.endswith(".json"):
clean_path = clean_path[:-5]
json_url = f"https://www.reddit.com{clean_path}.json"
try:
resp = requests.get(
json_url,
headers={"User-Agent": self.USER_AGENT},
proxies=proxies,
params={"limit": 50},
timeout=15,
)
resp.raise_for_status()
except requests.exceptions.HTTPError as e:
status = e.response.status_code if e.response is not None else 0
if status in (403, 429):
return ReadResult(
title="Reddit",
content="⚠️ Reddit blocked this request (403 Forbidden). "
"Reddit blocks most server IPs.\n"
"Fix: agent-reach configure proxy http://user:pass@ip:port\n"
"Cheap option: https://www.webshare.io ($1/month)\n\n"
"Alternatively, search Reddit via Exa (free, no proxy needed): "
"agent-reach search-reddit \"your query\"",
url=url,
platform="reddit",
)
raise
data = resp.json()
# Subreddit listing page: /r/sub/, /r/sub/hot, /r/sub/new, /r/sub/top
if isinstance(data, dict) and data.get("kind") == "Listing":
return self._parse_listing(data, url)
if isinstance(data, list) and len(data) >= 1:
# Post page: [post_listing, comments_listing]
post = data[0]["data"]["children"][0]["data"]
title = post.get("title", "")
author = post.get("author", "")
selftext = post.get("selftext", "")
score = post.get("score", 0)
subreddit = post.get("subreddit", "")
# Extract comments
comments_text = ""
if len(data) >= 2:
comments_text = self._extract_comments(data[1])
content = selftext
if comments_text:
content += f"\n\n---\n## Comments\n{comments_text}"
return ReadResult(
title=title,
content=content,
url=url,
author=f"u/{author}",
platform="reddit",
extra={"subreddit": subreddit, "score": score},
)
raise ValueError(f"Could not parse Reddit response for: {url}")
def _parse_listing(self, data: dict, url: str) -> ReadResult:
"""Parse a subreddit listing (hot/new/top/rising)."""
children = data.get("data", {}).get("children", [])
# Extract subreddit name and sort from URL
parsed = urlparse(url)
path_parts = [p for p in parsed.path.strip("/").split("/") if p]
subreddit = path_parts[1] if len(path_parts) >= 2 else "reddit"
sort_type = path_parts[2] if len(path_parts) >= 3 else "hot"
lines = []
for i, child in enumerate(children, 1):
if child.get("kind") != "t3":
continue
post = child.get("data", {})
title = post.get("title", "")
author = post.get("author", "")
score = post.get("score", 0)
num_comments = post.get("num_comments", 0)
permalink = post.get("permalink", "")
post_url = post.get("url", "")
is_self = post.get("is_self", False)
lines.append(f"### {i}. {title}")
lines.append(f"👤 u/{author} · ⬆ {score} · 💬 {num_comments}")
if not is_self and post_url:
lines.append(f"🔗 {post_url}")
lines.append(f"📎 https://www.reddit.com{permalink}")
# Add selftext preview (first 200 chars)
selftext = post.get("selftext", "")
if selftext:
preview = selftext[:200].replace("\n", " ")
if len(selftext) > 200:
preview += "..."
lines.append(f"> {preview}")
lines.append("")
content = "\n".join(lines) if lines else "No posts found."
return ReadResult(
title=f"r/{subreddit}{sort_type}",
content=content,
url=url,
platform="reddit",
extra={"subreddit": subreddit, "sort": sort_type, "count": len(children)},
) )
def _extract_comments(self, comments_data: dict, depth: int = 0, max_depth: int = 3) -> str:
"""Recursively extract comments."""
lines = []
children = comments_data.get("data", {}).get("children", [])
for child in children:
if child.get("kind") != "t1":
continue
data = child.get("data", {})
author = data.get("author", "[deleted]")
body = data.get("body", "")
score = data.get("score", 0)
indent = " " * depth
lines.append(f"{indent}**u/{author}** ({score} points):")
lines.append(f"{indent}{body}")
lines.append("")
# Recurse into replies
if depth < max_depth and data.get("replies") and isinstance(data["replies"], dict):
lines.append(self._extract_comments(data["replies"], depth + 1, max_depth))
return "\n".join(lines)
+9 -45
View File
@@ -1,13 +1,7 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""RSS feeds — via feedparser (free, pip dependency). """RSS — check if feedparser is available."""
Backend: feedparser (https://github.com/kurtmckee/feedparser) from .base import Channel
Swap to: any RSS parser
"""
import feedparser
from urllib.parse import urlparse
from .base import Channel, ReadResult
class RSSChannel(Channel): class RSSChannel(Channel):
@@ -17,41 +11,11 @@ class RSSChannel(Channel):
tier = 0 tier = 0
def can_handle(self, url: str) -> bool: def can_handle(self, url: str) -> bool:
lower = url.lower() return any(x in url.lower() for x in ["/feed", "/rss", ".xml", "atom"])
domain = urlparse(url).netloc.lower()
return (lower.endswith(".xml") or "/rss" in lower or "/feed" in lower
or "/atom" in lower or "rss" in domain)
async def read(self, url: str, config=None) -> ReadResult: def check(self, config=None):
feed = feedparser.parse(url) try:
import feedparser
if feed.bozo and not feed.entries: return "ok", "可读取 RSS/Atom 源"
raise ValueError(f"Failed to parse RSS feed: {url}") except ImportError:
return "off", "feedparser 未安装。安装:pip install feedparser"
if not feed.entries:
raise ValueError(f"No entries in RSS feed: {url}")
# Return latest entry
entry = feed.entries[0]
content = entry.get("summary", "") or entry.get("description", "")
# If multiple entries, summarize all
if len(feed.entries) > 1:
lines = [f"# {feed.feed.get('title', 'RSS Feed')}\n"]
for i, e in enumerate(feed.entries[:20], 1):
title = e.get("title", "Untitled")
link = e.get("link", "")
summary = e.get("summary", "")[:200]
lines.append(f"## {i}. {title}")
lines.append(f"🔗 {link}")
if summary:
lines.append(summary)
lines.append("")
content = "\n".join(lines)
return ReadResult(
title=feed.feed.get("title", entry.get("title", url)),
content=content,
url=url,
platform="rss",
)
+22 -193
View File
@@ -1,209 +1,38 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""Twitter/X — via bird CLI (free) or Jina Reader fallback. """Twitter/X — check if bird CLI is available."""
Backend: bird (@steipete/bird npm package) for search/timeline
Jina Reader for single tweets
Swap to: any Twitter access tool
"""
import shutil import shutil
import subprocess import subprocess
from urllib.parse import urlparse from .base import Channel
from .base import Channel, ReadResult, SearchResult
from typing import List
import requests
def _bird_cmd():
"""Find bird CLI binary."""
return shutil.which("bird") or shutil.which("birdx")
def _bird_env(config=None):
"""Build env dict with Twitter cookies for bird CLI."""
import os
env = os.environ.copy()
if config:
auth_token = config.get("twitter_auth_token")
ct0 = config.get("twitter_ct0")
if auth_token:
env["AUTH_TOKEN"] = auth_token
if ct0:
env["CT0"] = ct0
return env
class TwitterChannel(Channel): class TwitterChannel(Channel):
name = "twitter" name = "twitter"
description = "Twitter/X 推文" description = "Twitter/X 推文"
backends = ["bird", "Jina Reader"] backends = ["bird CLI"]
tier = 0 # Single tweet reading is zero-config tier = 1
def can_handle(self, url: str) -> bool: def can_handle(self, url: str) -> bool:
domain = urlparse(url).netloc.lower() from urllib.parse import urlparse
return "x.com" in domain or "twitter.com" in domain d = urlparse(url).netloc.lower()
return "x.com" in d or "twitter.com" in d
def check(self, config=None): def check(self, config=None):
# Basic reading always works (Jina fallback) bird = shutil.which("bird") or shutil.which("birdx")
if _bird_cmd(): if not bird:
return "ok", "搜索、时间线、发推全部可用" return "warn", (
return "ok", "可读取推文。安装 bird + 配置 Cookie 可解锁搜索和发推" "bird CLI 未安装。搜索可通过 Exa 替代。安装:\n"
" npm install -g @steipete/bird"
async def read(self, url: str, config=None) -> ReadResult:
# Try bird first
bird = _bird_cmd()
if bird:
return await self._read_bird(url, bird, config)
# Fallback: Jina Reader
return await self._read_jina(url)
async def _read_bird(self, url: str, bird: str, config=None) -> ReadResult:
result = subprocess.run(
[bird, "read", url],
capture_output=True, timeout=30,
encoding='utf-8', errors='replace',
env=_bird_env(config),
)
if result.returncode != 0:
return await self._read_jina(url)
text = result.stdout.strip()
# Extract author from first line
author = ""
lines = text.split("\n")
if lines and lines[0].startswith("@"):
author = lines[0].split()[0]
return ReadResult(
title=text[:100],
content=text,
url=url,
author=author,
platform="twitter",
)
async def _read_jina(self, url: str) -> ReadResult:
try:
resp = requests.get(
f"https://r.jina.ai/{url}",
headers={"Accept": "text/markdown"},
timeout=15,
) )
resp.raise_for_status() try:
text = resp.text r = subprocess.run(
[bird, "whoami"], capture_output=True, text=True, timeout=10
# Detect unusable Jina responses for X/Twitter (JS-required pages) )
unusable_indicators = [ if r.returncode == 0:
"page doesn", # "this page doesn't exist" (handles both ' and ') return "ok", "完整可用(读取、搜索推文)"
"miss what", # "Don't miss what's happening" return "warn", (
"Something went wrong. Try reloading", "bird CLI 已安装但未配置 Cookie。运行:\n"
"Log in](", # Markdown link: [Log in](...) " agent-reach configure twitter-cookies \"auth_token=xxx; ct0=yyy\""
]
if any(indicator in text for indicator in unusable_indicators):
return ReadResult(
title="Twitter/X",
content="⚠️ Could not read this tweet.\n"
"The tweet may have been deleted, or the account is private.\n\n"
"Tips:\n"
"- Make sure the URL is correct\n"
"- Try: bird read <url> (if bird CLI is installed)\n"
"- For protected tweets, configure Twitter cookies: "
"agent-reach configure twitter-cookies AUTH_TOKEN CT0",
url=url,
platform="twitter",
)
title = text[:100] if text else url
return ReadResult(
title=title,
content=text,
url=url,
platform="twitter",
) )
except Exception: except Exception:
return ReadResult( return "warn", "bird CLI 已安装但连接失败"
title="Twitter/X",
content="⚠️ Could not read this tweet.\n"
"The tweet may have been deleted, or the account is private.\n\n"
"Tips:\n"
"- Make sure the URL is correct\n"
"- Try: bird read <url> (if bird CLI is installed)\n"
"- For protected tweets, configure Twitter cookies: "
"agent-reach configure twitter-cookies AUTH_TOKEN CT0",
url=url,
platform="twitter",
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
limit = kwargs.get("limit", 10)
bird = _bird_cmd()
if bird:
return await self._search_bird(query, limit, bird, config)
# Fallback to Exa
return await self._search_exa(query, limit, config)
async def _search_bird(self, query: str, limit: int, bird: str, config=None) -> List[SearchResult]:
try:
result = subprocess.run(
[bird, "search", query, "-n", str(limit)],
capture_output=True, timeout=30,
encoding='utf-8', errors='replace',
env=_bird_env(config),
)
if result.returncode != 0:
return []
return self._parse_bird_output(result.stdout)
except (subprocess.TimeoutExpired, FileNotFoundError):
return []
def _parse_bird_output(self, text: str) -> List[SearchResult]:
"""Parse bird text output into SearchResults."""
results = []
current = {}
text_lines = []
for line in text.strip().split("\n"):
line = line.strip()
if line.startswith(""):
if current:
current["text"] = "\n".join(text_lines).strip()
results.append(SearchResult(
title=current.get("text", "")[:80],
url=current.get("url", ""),
snippet=current.get("text", ""),
author=current.get("author", ""),
date=current.get("date", ""),
))
current = {}
text_lines = []
continue
if line.startswith("@") and line.endswith(":") and "(" in line:
current["author"] = line.split()[0]
continue
if line.startswith("date:"):
current["date"] = line[5:].strip()
continue
if line.startswith("url:"):
current["url"] = line[4:].strip()
continue
if current is not None:
text_lines.append(line)
if current and text_lines:
current["text"] = "\n".join(text_lines).strip()
results.append(SearchResult(
title=current.get("text", "")[:80],
url=current.get("url", ""),
snippet=current.get("text", ""),
author=current.get("author", ""),
date=current.get("date", ""),
))
return results
async def _search_exa(self, query: str, limit: int, config=None) -> List[SearchResult]:
from agent_reach.channels.exa_search import ExaSearchChannel
exa = ExaSearchChannel()
return await exa.search(f"site:x.com {query}", config=config, limit=limit)
+7 -39
View File
@@ -1,49 +1,17 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""Web pages — via Jina Reader API (free, no config needed). """Web — any URL via Jina Reader. Always available."""
Backend: Jina Reader (https://r.jina.ai) from .base import Channel
Swap to: Firecrawl, Trafilatura, or any other reader API
"""
import requests
from .base import Channel, ReadResult
class WebChannel(Channel): class WebChannel(Channel):
name = "web" name = "web"
description = "网页(任意 URL" description = "任意网页"
backends = ["Jina Reader API"] backends = ["Jina Reader"]
tier = 0 tier = 0
JINA_URL = "https://r.jina.ai/"
def can_handle(self, url: str) -> bool: def can_handle(self, url: str) -> bool:
# Fallback — handles any URL not matched by other channels return True # Fallback — handles any URL
return True
async def read(self, url: str, config=None) -> ReadResult: def check(self, config=None):
resp = requests.get( return "ok", "通过 Jina Reader 读取任意网页(curl https://r.jina.ai/URL"
f"{self.JINA_URL}{url}",
headers={"Accept": "text/markdown"},
timeout=15,
)
resp.raise_for_status()
text = resp.text
# Extract title from first markdown heading
title = url
for line in text.split("\n"):
line = line.strip()
if line.startswith("# "):
title = line[2:].strip()
break
if line.startswith("Title:"):
title = line[6:].strip()
break
return ReadResult(
title=title,
content=text,
url=url,
platform="web",
)
+20 -142
View File
@@ -1,16 +1,9 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""XiaoHongShu (小红书) — via mcporter + xiaohongshu MCP server. """XiaoHongShu — check if mcporter + xiaohongshu MCP is available."""
Backend: xiaohongshu-mcp server (internal API, reliable)
Requires: mcporter CLI + xiaohongshu MCP server running
"""
import json
import shutil import shutil
import subprocess import subprocess
from urllib.parse import urlparse from .base import Channel
from .base import Channel, ReadResult, SearchResult
from typing import List, Optional
class XiaoHongShuChannel(Channel): class XiaoHongShuChannel(Channel):
@@ -19,30 +12,8 @@ class XiaoHongShuChannel(Channel):
backends = ["xiaohongshu-mcp"] backends = ["xiaohongshu-mcp"]
tier = 2 tier = 2
def _mcporter_ok(self) -> bool:
"""Check if mcporter + xiaohongshu MCP is available."""
if not shutil.which("mcporter"):
return False
try:
r = subprocess.run(
["mcporter", "list"], capture_output=True, text=True, timeout=10
)
return "xiaohongshu" in r.stdout
except Exception:
return False
def _call(self, expr: str, timeout: int = 30) -> str:
r = subprocess.run(
["mcporter", "call", expr],
capture_output=True, text=True, timeout=timeout,
)
if r.returncode != 0:
raise RuntimeError(r.stderr or r.stdout)
return r.stdout
# ── Channel interface ──
def can_handle(self, url: str) -> bool: def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
d = urlparse(url).netloc.lower() d = urlparse(url).netloc.lower()
return "xiaohongshu.com" in d or "xhslink.com" in d return "xiaohongshu.com" in d or "xhslink.com" in d
@@ -55,118 +26,25 @@ class XiaoHongShuChannel(Channel):
" 3. mcporter config add xiaohongshu http://localhost:18060/mcp\n" " 3. mcporter config add xiaohongshu http://localhost:18060/mcp\n"
" 详见 https://github.com/xpzouying/xiaohongshu-mcp" " 详见 https://github.com/xpzouying/xiaohongshu-mcp"
) )
if not self._mcporter_ok():
return "off", (
"mcporter 已装但小红书 MCP 未配置。运行:\n"
" docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
" mcporter config add xiaohongshu http://localhost:18060/mcp"
)
try: try:
out = self._call("xiaohongshu.check_login_status()", timeout=10) r = subprocess.run(
if "已登录" in out or "logged" in out.lower(): ["mcporter", "list"], capture_output=True, text=True, timeout=10
)
if "xiaohongshu" not in r.stdout:
return "off", (
"mcporter 已装但小红书 MCP 未配置。运行:\n"
" docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
" mcporter config add xiaohongshu http://localhost:18060/mcp"
)
except Exception:
return "off", "mcporter 连接异常"
try:
r = subprocess.run(
["mcporter", "call", "xiaohongshu.check_login_status()"],
capture_output=True, text=True, timeout=10
)
if "已登录" in r.stdout or "logged" in r.stdout.lower():
return "ok", "完整可用(阅读、搜索、发帖、评论、点赞)" return "ok", "完整可用(阅读、搜索、发帖、评论、点赞)"
return "warn", "MCP 已连接但未登录,需扫码登录" return "warn", "MCP 已连接但未登录,需扫码登录"
except Exception: except Exception:
return "warn", "MCP 连接异常,检查 xiaohongshu-mcp 服务是否在运行" return "warn", "MCP 连接异常,检查 xiaohongshu-mcp 服务是否在运行"
async def read(self, url: str, config=None) -> ReadResult:
if not self._mcporter_ok():
return ReadResult(
title="XiaoHongShu",
content=(
"⚠️ 小红书需要 mcporter + xiaohongshu-mcp 才能使用。\n\n"
"安装步骤:\n"
"1. npm install -g mcporter\n"
"2. docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
"3. mcporter config add xiaohongshu http://localhost:18060/mcp\n"
"4. 运行 agent-reach doctor 检查状态\n\n"
"详见 https://github.com/xpzouying/xiaohongshu-mcp"
),
url=url, platform="xiaohongshu",
)
note_id = self._extract_note_id(url)
if not note_id:
return ReadResult(
title="XiaoHongShu",
content=f"⚠️ 无法从 URL 提取笔记 ID: {url}",
url=url, platform="xiaohongshu",
)
# Step 1: get xsec_token from feeds
xsec_token = self._find_token(note_id)
if not xsec_token:
return ReadResult(
title="XiaoHongShu",
content=(
f"⚠️ 无法获取笔记 {note_id} 的访问令牌。\n"
"小红书需要 xsec_token 才能读取笔记详情。\n"
"请先通过搜索找到这篇笔记,或直接使用搜索功能。"
),
url=url, platform="xiaohongshu",
)
# Step 2: get detail
out = self._call(
f'xiaohongshu.get_feed_detail(feed_id: "{note_id}", xsec_token: "{xsec_token}")',
timeout=15,
)
return ReadResult(
title=self._extract_title(out) or f"XHS {note_id}",
content=out.strip(),
url=url, platform="xiaohongshu",
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
if not self._mcporter_ok():
raise ValueError(
"小红书搜索需要 mcporter + xiaohongshu-mcp。\n"
"安装: npm install -g mcporter && mcporter config add xiaohongshu http://localhost:18060/mcp"
)
limit = kwargs.get("limit", 10)
safe_q = query.replace('"', '\\"')
out = self._call(f'xiaohongshu.search_feeds(keyword: "{safe_q}")', timeout=30)
results = []
try:
data = json.loads(out)
for item in data.get("feeds", [])[:limit]:
card = item.get("noteCard", {})
user = card.get("user", {})
interact = card.get("interactInfo", {})
results.append(SearchResult(
title=card.get("displayTitle", ""),
url=f"https://www.xiaohongshu.com/explore/{item.get('id', '')}",
snippet=f"👤 {user.get('nickname', '')} · ❤ {interact.get('likedCount', '0')}",
score=0,
))
except (json.JSONDecodeError, KeyError):
pass
return results
# ── Helpers ──
def _extract_note_id(self, url: str) -> str:
parts = urlparse(url).path.strip("/").split("/")
return parts[-1] if parts else ""
def _find_token(self, note_id: str) -> Optional[str]:
"""Try to find xsec_token for a note from feeds."""
try:
out = self._call("xiaohongshu.list_feeds()", timeout=15)
data = json.loads(out)
for feed in data.get("feeds", []):
if feed.get("id") == note_id:
return feed.get("xsecToken", "")
except Exception:
pass
return None
def _extract_title(self, text: str) -> str:
for line in text.split("\n"):
line = line.strip()
if line and not line.startswith(("{", "[", "#", "http")):
return line[:80]
return ""
+8 -111
View File
@@ -1,125 +1,22 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""YouTube — via yt-dlp (video info, subtitles, and search). """YouTube — check if yt-dlp is available."""
Backend: yt-dlp (https://github.com/yt-dlp/yt-dlp)
Supports: read (info + subtitles), search (ytsearch)
"""
import json
import shutil import shutil
import subprocess from .base import Channel
import tempfile
from pathlib import Path
from urllib.parse import urlparse
from .base import Channel, ReadResult, SearchResult
from typing import List
class YouTubeChannel(Channel): class YouTubeChannel(Channel):
name = "youtube" name = "youtube"
description = "YouTube 视频字幕" description = "YouTube 视频字幕"
backends = ["yt-dlp"] backends = ["yt-dlp"]
requires_tools = ["yt-dlp"]
tier = 0 tier = 0
def can_handle(self, url: str) -> bool: def can_handle(self, url: str) -> bool:
from urllib.parse import urlparse
d = urlparse(url).netloc.lower() d = urlparse(url).netloc.lower()
return "youtube.com" in d or "youtu.be" in d return "youtube.com" in d or "youtu.be" in d
async def read(self, url: str, config=None) -> ReadResult: def check(self, config=None):
if not shutil.which("yt-dlp"): if shutil.which("yt-dlp"):
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp") return "ok", "可提取视频信息和字幕"
return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
with tempfile.TemporaryDirectory() as tmpdir:
info = self._get_info(url)
title = info.get("title", url)
author = info.get("uploader", "")
transcript = self._get_subtitles(url, tmpdir)
if not transcript:
transcript = f"[Video: {title}]\n[No subtitles available.]"
return ReadResult(
title=title, content=transcript, url=url,
author=author, platform="youtube",
extra={
"duration": info.get("duration_string"),
"view_count": info.get("view_count"),
"upload_date": info.get("upload_date"),
},
)
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
"""Search YouTube via yt-dlp's ytsearch."""
if not shutil.which("yt-dlp"):
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
limit = kwargs.get("limit", 10)
try:
r = subprocess.run(
["yt-dlp", "--dump-json", "--flat-playlist",
f"ytsearch{limit}:{query}"],
capture_output=True, text=True, timeout=30,
)
results = []
for line in r.stdout.strip().split("\n"):
if not line.strip():
continue
try:
d = json.loads(line)
vid = d.get("id", "")
results.append(SearchResult(
title=d.get("title", ""),
url=f"https://youtube.com/watch?v={vid}" if vid else "",
snippet=(
f"👤 {d.get('channel', '?')} · "
f"{d.get('duration_string', '?')} · "
f"👁 {d.get('view_count', '?')}"
),
extra={
"channel": d.get("channel"),
"duration": d.get("duration_string"),
"view_count": d.get("view_count"),
},
))
except json.JSONDecodeError:
continue
return results
except subprocess.TimeoutExpired:
return []
def _get_info(self, url: str) -> dict:
try:
r = subprocess.run(
["yt-dlp", "--dump-json", "--no-download", url],
capture_output=True, text=True, timeout=30,
)
if r.returncode == 0:
return json.loads(r.stdout)
except (subprocess.TimeoutExpired, json.JSONDecodeError):
pass
return {}
def _get_subtitles(self, url: str, tmpdir: str) -> str:
try:
subprocess.run(
["yt-dlp", "--write-auto-sub", "--write-sub",
"--sub-lang", "en,zh-Hans,zh",
"--skip-download", "--sub-format", "vtt",
"-o", f"{tmpdir}/%(id)s.%(ext)s", url],
capture_output=True, text=True, timeout=30,
)
for f in Path(tmpdir).glob("*.vtt"):
text = f.read_text(errors="replace")
lines = []
for line in text.split("\n"):
line = line.strip()
if not line or line.startswith("WEBVTT") or "-->" in line or line.isdigit():
continue
if line not in lines[-1:]:
lines.append(line)
return "\n".join(lines)
except subprocess.TimeoutExpired:
pass
return ""
+17 -210
View File
@@ -1,20 +1,15 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
""" """
Agent Reach CLI — command-line interface. Agent Reach CLI — installer, doctor, and configuration tool.
Usage: Usage:
agent-reach read <url> agent-reach install --env=auto
agent-reach search <query>
agent-reach search-reddit <query> [--sub <subreddit>]
agent-reach search-github <query> [--lang <language>]
agent-reach search-twitter <query>
agent-reach setup
agent-reach doctor agent-reach doctor
agent-reach version agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
agent-reach setup
""" """
import sys import sys
import asyncio
import argparse import argparse
import json import json
import os import os
@@ -48,62 +43,6 @@ def main():
sub = parser.add_subparsers(dest="command", help="Available commands") sub = parser.add_subparsers(dest="command", help="Available commands")
# ── read ── # ── read ──
p_read = sub.add_parser("read", help="Read content from a URL")
p_read.add_argument("url", help="URL to read")
p_read.add_argument("--json", dest="as_json", action="store_true", help="Output as JSON")
# ── search ──
p_search = sub.add_parser("search", help="Search the web (Exa)")
p_search.add_argument("query", nargs="+", help="Search query")
p_search.add_argument("-n", "--num", type=int, default=5, help="Number of results")
# ── search-reddit ──
p_sr = sub.add_parser("search-reddit", help="Search Reddit")
p_sr.add_argument("query", nargs="+", help="Search query")
p_sr.add_argument("--sub", help="Subreddit filter")
p_sr.add_argument("-n", "--num", type=int, default=10, help="Number of results")
# ── search-github ──
p_sg = sub.add_parser("search-github", help="Search GitHub")
p_sg.add_argument("query", nargs="+", help="Search query")
p_sg.add_argument("--lang", help="Language filter")
p_sg.add_argument("-n", "--num", type=int, default=5, help="Number of results")
# ── search-twitter ──
p_st = sub.add_parser("search-twitter", help="Search Twitter")
p_st.add_argument("query", nargs="+", help="Search query")
p_st.add_argument("-n", "--num", type=int, default=10, help="Number of results")
# ── search-youtube ──
p_sy = sub.add_parser("search-youtube", help="Search YouTube")
p_sy.add_argument("query", nargs="+", help="Search query")
p_sy.add_argument("-n", "--num", type=int, default=5, help="Number of results")
# ── search-bilibili ──
p_sb = sub.add_parser("search-bilibili", help="Search Bilibili")
p_sb.add_argument("query", nargs="+", help="Search query")
p_sb.add_argument("-n", "--num", type=int, default=5, help="Number of results")
# ── search-xhs ──
p_sx = sub.add_parser("search-xhs", help="Search XiaoHongShu")
p_sx.add_argument("query", nargs="+", help="Search query")
p_sx.add_argument("-n", "--num", type=int, default=10, help="Number of results")
# ── search-instagram ──
p_si = sub.add_parser("search-instagram", help="Search Instagram")
p_si.add_argument("query", nargs="+", help="Search query")
p_si.add_argument("-n", "--num", type=int, default=10, help="Number of results")
# ── search-linkedin ──
p_sl = sub.add_parser("search-linkedin", help="Search LinkedIn")
p_sl.add_argument("query", nargs="+", help="Search query")
p_sl.add_argument("-n", "--num", type=int, default=10, help="Number of results")
# ── search-bosszhipin ──
p_sbz = sub.add_parser("search-bosszhipin", help="Search Boss直聘")
p_sbz.add_argument("query", nargs="+", help="Search query")
p_sbz.add_argument("-n", "--num", type=int, default=10, help="Number of results")
# ── setup ── # ── setup ──
sub.add_parser("setup", help="Interactive configuration wizard") sub.add_parser("setup", help="Interactive configuration wizard")
@@ -122,8 +61,7 @@ def main():
p_conf = sub.add_parser("configure", help="Set a config value or auto-extract from browser") p_conf = sub.add_parser("configure", help="Set a config value or auto-extract from browser")
p_conf.add_argument("key", nargs="?", default=None, p_conf.add_argument("key", nargs="?", default=None,
choices=["proxy", "github-token", "groq-key", choices=["proxy", "github-token", "groq-key",
"twitter-cookies", "youtube-cookies", "twitter-cookies", "youtube-cookies"],
"instagram-cookies"],
help="What to configure (omit if using --from-browser)") help="What to configure (omit if using --from-browser)")
p_conf.add_argument("value", nargs="*", help="The value(s) to set") p_conf.add_argument("value", nargs="*", help="The value(s) to set")
p_conf.add_argument("--from-browser", metavar="BROWSER", p_conf.add_argument("--from-browser", metavar="BROWSER",
@@ -167,10 +105,6 @@ def main():
_cmd_install(args) _cmd_install(args)
elif args.command == "configure": elif args.command == "configure":
_cmd_configure(args) _cmd_configure(args)
elif args.command == "read":
asyncio.run(_cmd_read(args))
elif args.command.startswith("search"):
asyncio.run(_cmd_search(args))
# ── Command handlers ──────────────────────────────── # ── Command handlers ────────────────────────────────
@@ -423,22 +357,18 @@ def _install_system_deps():
else: else:
print(" ⬜ bird CLI requires Node.js (optional — Twitter reading still works via Jina)") print(" ⬜ bird CLI requires Node.js (optional — Twitter reading still works via Jina)")
# ── instaloader (for Instagram) ── # ── undici (proxy support for Node.js fetch) ──
if shutil.which("instaloader"): if shutil.which("npm"):
print(" ✅ instaloader already installed") npm_root = subprocess.run(["npm", "root", "-g"], capture_output=True, text=True, timeout=5).stdout.strip()
else: undici_path = os.path.join(npm_root, "undici", "index.js") if npm_root else ""
print(" 📥 Installing instaloader...") if os.path.exists(undici_path):
try: print(" ✅ undici already installed (Node.js proxy support)")
subprocess.run( else:
[sys.executable, "-m", "pip", "install", "instaloader"], try:
capture_output=True, text=True, timeout=120, subprocess.run(["npm", "install", "-g", "undici"], capture_output=True, text=True, timeout=60)
) print(" ✅ undici installed (Node.js proxy support)")
if shutil.which("instaloader"): except Exception:
print(" ✅ instaloader installed (Instagram reading)") print(" ⬜ undici install failed (optional — bird may not work behind proxies)")
else:
print(" ⬜ instaloader install failed (optional — try: pip install instaloader)")
except Exception:
print(" ⬜ instaloader install failed (optional — try: pip install instaloader)")
def _install_system_deps_safe(): def _install_system_deps_safe():
@@ -451,7 +381,6 @@ def _install_system_deps_safe():
("gh", ["gh"], "GitHub CLI", "https://cli.github.com — or: apt install gh / brew install gh"), ("gh", ["gh"], "GitHub CLI", "https://cli.github.com — or: apt install gh / brew install gh"),
("node", ["node", "npm"], "Node.js", "https://nodejs.org — or: apt install nodejs npm"), ("node", ["node", "npm"], "Node.js", "https://nodejs.org — or: apt install nodejs npm"),
("bird", ["bird", "birdx"], "bird CLI (Twitter)", "npm install -g @steipete/bird"), ("bird", ["bird", "birdx"], "bird CLI (Twitter)", "npm install -g @steipete/bird"),
("instaloader", ["instaloader"], "instaloader (Instagram)", "pip install instaloader"),
] ]
missing = [] missing = []
@@ -482,7 +411,6 @@ def _install_system_deps_dryrun():
("gh CLI", ["gh"], "apt install gh / brew install gh"), ("gh CLI", ["gh"], "apt install gh / brew install gh"),
("Node.js", ["node"], "curl NodeSource setup | bash + apt install nodejs"), ("Node.js", ["node"], "curl NodeSource setup | bash + apt install nodejs"),
("bird CLI", ["bird", "birdx"], "npm install -g @steipete/bird"), ("bird CLI", ["bird", "birdx"], "npm install -g @steipete/bird"),
("instaloader", ["instaloader"], "pip install instaloader"),
] ]
for label, binaries, method in checks: for label, binaries, method in checks:
@@ -751,9 +679,6 @@ def _cmd_configure(args):
config.set("groq_api_key", value) config.set("groq_api_key", value)
print(f"✅ Groq key configured!") print(f"✅ Groq key configured!")
elif args.key == "instagram-cookies":
_configure_instagram_cookies(value)
def _cmd_doctor(): def _cmd_doctor():
from agent_reach.config import Config from agent_reach.config import Config
@@ -774,30 +699,6 @@ def _parse_cookie_header(cookie_str: str) -> dict:
return cookies return cookies
def _configure_instagram_cookies(value: str):
"""Save Instagram cookies from Cookie-Editor Header String."""
from pathlib import Path
cookies = _parse_cookie_header(value)
if "sessionid" not in cookies:
print("❌ Cookie 里缺少 sessionid。")
print(" 确保你已登录 Instagram,然后用 Cookie-Editor 导出 Header String。")
print(' 格式: agent-reach configure instagram-cookies "sessionid=xxx; csrftoken=yyy; ..."')
return
cookie_dir = Path.home() / ".agent-reach"
cookie_dir.mkdir(parents=True, exist_ok=True)
cookie_file = cookie_dir / "instagram-cookies.txt"
cookie_file.write_text(value.strip())
cookie_file.chmod(0o600)
print(f"✅ Instagram cookies 已保存!")
print(f" sessionid: {cookies['sessionid'][:8]}...")
if "csrftoken" in cookies:
print(f" csrftoken: ✅")
if "ds_user_id" in cookies:
print(f" ds_user_id: {cookies['ds_user_id']}")
print(f" 文件: {cookie_file}")
def _cmd_setup(): def _cmd_setup():
@@ -888,100 +789,6 @@ def _cmd_setup():
print() print()
async def _cmd_read(args):
from agent_reach.core import AgentReach
eyes = AgentReach()
try:
result = await eyes.read(args.url)
if args.as_json:
print(json.dumps(result, ensure_ascii=False, indent=2))
else:
print(f"\n📖 {result.get('title', 'Untitled')}")
print(f"🔗 {result.get('url', '')}")
if result.get("author"):
print(f"👤 {result['author']}")
print(f"\n{result.get('content', '')}")
except Exception as e:
error_str = str(e)
if "400" in error_str and "Bad Request" in error_str:
print(f"❌ Invalid URL: {args.url}", file=sys.stderr)
print(" Please provide a valid URL (e.g., https://example.com)", file=sys.stderr)
elif "ConnectionError" in type(e).__name__ or "Timeout" in type(e).__name__:
print(f"❌ Could not connect to: {args.url}", file=sys.stderr)
print(" Check your internet connection or the URL.", file=sys.stderr)
else:
print(f"❌ Error: {e}", file=sys.stderr)
sys.exit(1)
async def _cmd_search(args):
from agent_reach.core import AgentReach
eyes = AgentReach()
query = " ".join(args.query).strip()
num = args.num
if not query:
print("Please provide a search query.", file=sys.stderr)
sys.exit(1)
try:
if args.command == "search":
results = await eyes.search(query, num_results=num)
elif args.command == "search-reddit":
results = await eyes.search_reddit(query, subreddit=getattr(args, "sub", None), limit=num)
elif args.command == "search-github":
results = await eyes.search_github(query, language=getattr(args, "lang", None), limit=num)
elif args.command == "search-twitter":
results = await eyes.search_twitter(query, limit=num)
elif args.command == "search-youtube":
results = await eyes.search_youtube(query, limit=num)
elif args.command == "search-bilibili":
results = await eyes.search_bilibili(query, limit=num)
elif args.command == "search-xhs":
results = await eyes.search_xhs(query, limit=num)
elif args.command == "search-instagram":
results = await eyes.search_instagram(query, limit=num)
elif args.command == "search-linkedin":
results = await eyes.search_linkedin(query, limit=num)
elif args.command == "search-bosszhipin":
results = await eyes.search_bosszhipin(query, limit=num)
else:
print(f"Unknown command: {args.command}", file=sys.stderr)
sys.exit(1)
except Exception as e:
error_str = str(e)
if "401" in error_str or "Unauthorized" in error_str:
print("⚠️ Exa API key not configured or invalid.")
print("Get a free key at https://exa.ai (1000 searches/month free)")
print("Then run: agent-reach configure exa-key YOUR_KEY")
sys.exit(1)
elif "exa" in error_str.lower() or "api_key" in error_str.lower():
print("⚠️ Exa API key not configured.")
print("Get a free key at https://exa.ai")
print("Then run: agent-reach configure exa-key YOUR_KEY")
sys.exit(1)
else:
print(f"❌ Error: {e}", file=sys.stderr)
sys.exit(1)
if not results:
print("No results found.")
return
for i, r in enumerate(results, 1):
title = r.get("title") or r.get("name") or r.get("text", "")[:60]
url = r.get("url", "")
snippet = r.get("snippet") or r.get("description") or r.get("text", "")
print(f"\n{i}. {title}")
print(f" 🔗 {url}")
if snippet:
print(f" {snippet[:200]}")
# Extra info for GitHub
extra = r.get("extra", {})
if extra.get("stars"):
print(f"{extra['stars']} 🍴 {extra.get('forks', 0)} 📝 {extra.get('language', '')}")
def _cmd_check_update(): def _cmd_check_update():
"""Check for newer versions on GitHub.""" """Check for newer versions on GitHub."""
import requests import requests
+16 -116
View File
@@ -1,126 +1,36 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
""" """
AgentReach — the unified entry point. AgentReach — installer, doctor, and configuration tool.
Pure glue: routes URLs to the right channel, routes searches to the right engine. Agent Reach helps AI agents install and configure upstream platform tools
Every channel is a thin wrapper around an external tool. Swap any backend anytime. (bird CLI, yt-dlp, mcporter, gh CLI, etc.). After installation, agents
call the upstream tools directly — no wrapper layer needed.
Usage: Usage:
from agent_reach import AgentReach from agent_reach.doctor import check_all, format_report
from agent_reach.config import Config
eyes = AgentReach() config = Config()
content = await eyes.read("https://github.com/openai/gpt-4") results = check_all(config)
results = await eyes.search("AI agent framework") print(format_report(results))
""" """
import asyncio from typing import Dict, Optional
from typing import Any, Dict, List, Optional
from agent_reach.config import Config from agent_reach.config import Config
from agent_reach.channels import get_channel_for_url, get_channel, get_all_channels
class AgentReach: class AgentReach:
"""Give your AI Agent eyes to see the entire internet.""" """Give your AI Agent eyes to see the entire internet.
This class provides health-check functionality.
For reading/searching, use the upstream tools directly
(see SKILL.md for commands).
"""
def __init__(self, config: Optional[Config] = None): def __init__(self, config: Optional[Config] = None):
self.config = config or Config() self.config = config or Config()
# ── Reading ─────────────────────────────────────────
async def read(self, url: str) -> Dict[str, Any]:
"""
Read content from any URL. Auto-detects platform.
Supported: Web, GitHub, Reddit, Twitter, YouTube,
Bilibili, RSS, and more.
Returns:
Dict with title, content, url, author, platform, etc.
"""
if not url.startswith(("http://", "https://")):
url = f"https://{url}"
channel = get_channel_for_url(url)
result = await channel.read(url, config=self.config)
return result.to_dict()
async def read_batch(self, urls: List[str]) -> List[Dict[str, Any]]:
"""Read multiple URLs concurrently."""
tasks = [self.read(url) for url in urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
return [r for r in results if not isinstance(r, Exception)]
def detect_platform(self, url: str) -> str:
"""Detect what platform a URL belongs to."""
channel = get_channel_for_url(url)
return channel.name
# ── Searching ───────────────────────────────────────
async def search(self, query: str, num_results: int = 5) -> List[Dict[str, Any]]:
"""Semantic web search via Exa."""
ch = get_channel("exa_search")
results = await ch.search(query, config=self.config, limit=num_results)
return [r.to_dict() for r in results]
async def search_reddit(self, query: str, subreddit: Optional[str] = None, limit: int = 10) -> List[Dict[str, Any]]:
"""Search Reddit via Exa (bypasses IP blocks)."""
ch = get_channel("exa_search")
q = f"site:reddit.com/r/{subreddit} {query}" if subreddit else f"site:reddit.com {query}"
results = await ch.search(q, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_github(self, query: str, language: Optional[str] = None, limit: int = 5) -> List[Dict[str, Any]]:
"""Search GitHub repositories."""
ch = get_channel("github")
results = await ch.search(query, config=self.config, language=language, limit=limit)
return [r.to_dict() for r in results]
async def search_twitter(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
"""Search Twitter. Uses bird CLI if available, else Exa."""
ch = get_channel("twitter")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_youtube(self, query: str, limit: int = 5) -> List[Dict[str, Any]]:
"""Search YouTube via yt-dlp."""
ch = get_channel("youtube")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_bilibili(self, query: str, limit: int = 5) -> List[Dict[str, Any]]:
"""Search Bilibili. Tries yt-dlp first, falls back to Exa."""
ch = get_channel("bilibili")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_xhs(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
"""Search XiaoHongShu via mcporter."""
ch = get_channel("xiaohongshu")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_instagram(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
"""Search Instagram via Exa."""
ch = get_channel("instagram")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_linkedin(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
"""Search LinkedIn via MCP or Exa."""
ch = get_channel("linkedin")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
async def search_bosszhipin(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
"""Search Boss直聘 via MCP or Exa."""
ch = get_channel("bosszhipin")
results = await ch.search(query, config=self.config, limit=limit)
return [r.to_dict() for r in results]
# ── Health ──────────────────────────────────────────
def doctor(self) -> Dict[str, dict]: def doctor(self) -> Dict[str, dict]:
"""Check all channel availability.""" """Check all channel availability."""
from agent_reach.doctor import check_all from agent_reach.doctor import check_all
@@ -130,13 +40,3 @@ class AgentReach:
"""Get formatted health report.""" """Get formatted health report."""
from agent_reach.doctor import check_all, format_report from agent_reach.doctor import check_all, format_report
return format_report(check_all(self.config)) return format_report(check_all(self.config))
# ── Sync wrappers ───────────────────────────────────
def read_sync(self, url: str) -> Dict[str, Any]:
"""Synchronous version of read()."""
return asyncio.run(self.read(url))
def search_sync(self, query: str, num_results: int = 5) -> List[Dict[str, Any]]:
"""Synchronous version of search()."""
return asyncio.run(self.search(query, num_results))
+14
View File
@@ -74,4 +74,18 @@ def format_report(results: Dict[str, dict]) -> str:
if ok_count < total: if ok_count < total:
lines.append("运行 `agent-reach setup` 解锁更多渠道") lines.append("运行 `agent-reach setup` 解锁更多渠道")
# Security check: config file permissions
import os
import stat
config_path = Config.CONFIG_DIR / "config.yaml"
if config_path.exists():
try:
mode = config_path.stat().st_mode
if mode & (stat.S_IRGRP | stat.S_IROTH):
lines.append("")
lines.append("⚠️ 安全提示:config.yaml 权限过宽(其他用户可读)")
lines.append(" 修复:chmod 600 ~/.agent-reach/config.yaml")
except OSError:
pass
return "\n".join(lines) return "\n".join(lines)
+2 -2
View File
@@ -22,8 +22,8 @@ mcporter config add exa https://mcp.exa.ai/mcp
### 3. 验证 ### 3. 验证
```bash ```bash
agent-reach doctor | grep "Web Search" agent-reach doctor | grep "Search"
agent-reach search "test" -n 1 mcporter call 'exa.web_search_exa(query: "test", numResults: 1)'
``` ```
## 需要用户手动做的步骤 ## 需要用户手动做的步骤
+1 -1
View File
@@ -18,7 +18,7 @@ playwright install chromium
3. 安装完成后测试: 3. 安装完成后测试:
```bash ```bash
agent-reach read "https://mp.weixin.qq.com/s/一个测试链接" curl -s "https://r.jina.ai/https://mp.weixin.qq.com/s/一个测试链接" -H "Accept: text/markdown"
``` ```
## 需要用户手动做的步骤 ## 需要用户手动做的步骤
+5 -39
View File
@@ -1,10 +1,11 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
""" """
Agent Reach MCP Server — expose all capabilities as MCP tools. Agent Reach MCP Server — expose doctor/status as MCP tool.
Run: python -m agent_reach.integrations.mcp_server Run: python -m agent_reach.integrations.mcp_server
8 tools for any MCP-compatible AI Agent. Agent Reach is an installer + doctor tool. For actual reading/searching,
agents should call upstream tools directly (bird, yt-dlp, mcporter, etc.).
""" """
import asyncio import asyncio
@@ -35,50 +36,15 @@ def create_server():
@server.list_tools() @server.list_tools()
async def list_tools(): async def list_tools():
return [ return [
Tool(name="read_url",
description="Read content from any URL. Supports: web, GitHub, Reddit, Twitter, YouTube, Bilibili, RSS.",
inputSchema={"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}),
Tool(name="read_batch",
description="Read multiple URLs concurrently.",
inputSchema={"type": "object", "properties": {"urls": {"type": "array", "items": {"type": "string"}}}, "required": ["urls"]}),
Tool(name="detect_platform",
description="Detect what platform a URL belongs to.",
inputSchema={"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}),
Tool(name="search",
description="Semantic web search via Exa.",
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "num_results": {"type": "integer", "default": 5}}, "required": ["query"]}),
Tool(name="search_reddit",
description="Search Reddit posts.",
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "subreddit": {"type": "string"}, "limit": {"type": "integer", "default": 10}}, "required": ["query"]}),
Tool(name="search_github",
description="Search GitHub repositories.",
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "language": {"type": "string"}, "limit": {"type": "integer", "default": 5}}, "required": ["query"]}),
Tool(name="search_twitter",
description="Search Twitter/X posts.",
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "limit": {"type": "integer", "default": 10}}, "required": ["query"]}),
Tool(name="get_status", Tool(name="get_status",
description="Get Agent Reach status: which channels are active.", description="Get Agent Reach status: which channels are installed and active.",
inputSchema={"type": "object", "properties": {}}), inputSchema={"type": "object", "properties": {}}),
] ]
@server.call_tool() @server.call_tool()
async def call_tool(name: str, arguments: dict): async def call_tool(name: str, arguments: dict):
try: try:
if name == "read_url": if name == "get_status":
result = await eyes.read(arguments["url"])
elif name == "read_batch":
result = await eyes.read_batch(arguments["urls"])
elif name == "detect_platform":
result = eyes.detect_platform(arguments["url"])
elif name == "search":
result = await eyes.search(arguments["query"], arguments.get("num_results", 5))
elif name == "search_reddit":
result = await eyes.search_reddit(arguments["query"], arguments.get("subreddit"), arguments.get("limit", 10))
elif name == "search_github":
result = await eyes.search_github(arguments["query"], arguments.get("language"), arguments.get("limit", 5))
elif name == "search_twitter":
result = await eyes.search_twitter(arguments["query"], arguments.get("limit", 10))
elif name == "get_status":
result = eyes.doctor_report() result = eyes.doctor_report()
else: else:
result = f"Unknown tool: {name}" result = f"Unknown tool: {name}"
-65
View File
@@ -1,65 +0,0 @@
---
name: agent-reach
description: "Give your AI Agent eyes to see the entire internet. Search + read 10+ platforms with one install."
install: pip install git+https://github.com/Panniantong/agent-reach.git
---
# Agent Reach — 给 Agent 装上眼睛 👁️
安装后你就能搜索和阅读全网内容。
## 安装
```bash
pip install git+https://github.com/Panniantong/agent-reach.git
```
## 装完直接能用的(零配置)
- `agent-reach read <URL>` — 读取任意网页、GitHub、B站、YouTube、RSS
- `agent-reach search-github "query"` — 搜索 GitHub 仓库
- `agent-reach doctor` — 查看所有平台状态
## 解锁更多功能
运行 `agent-reach configure` 进行交互式配置,或者读取以下指南:
| 功能 | 指南文件 |
|------|---------|
| 全网搜索 + Reddit搜索 + Twitter搜索 | `agent_reach/guides/setup-exa.md` |
| Reddit 完整阅读 | `agent_reach/guides/setup-reddit.md` |
| Twitter 高级(时间线/线程) | `agent_reach/guides/setup-twitter.md` |
| 小红书 | `agent_reach/guides/setup-xiaohongshu.md` |
| 微信公众号 | `agent_reach/guides/setup-wechat.md` |
| 视频语音转文字 | `agent_reach/guides/setup-groq.md` |
每个指南里有详细的步骤说明,你(Agent)可以照着做,只有需要人类操作的部分(登录、复制 key)才需要问用户。
## MCP Server
如果你的 Agent 平台支持 MCP:
```bash
pip install agent-reach[mcp]
python -m agent_reach.integrations.mcp_server
```
提供 8 个工具:read_url, read_batch, detect_platform, search, search_reddit, search_github, search_twitter, get_status
## Python API
```python
from agent_reach import AgentReach
import asyncio
eyes = AgentReach()
# 读取
result = asyncio.run(eyes.read("https://github.com/openai/gpt-4"))
# 搜索
results = asyncio.run(eyes.search("AI agent framework"))
# 健康检查
print(eyes.doctor_report())
```
+224 -55
View File
@@ -1,87 +1,256 @@
--- ---
name: agent-reach name: agent-reach
description: > description: >
Give your AI agent eyes to see the entire internet. Read and search across Give your AI agent eyes to see the entire internet. Install and configure
Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, RSS, and any web page upstream tools for Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu,
— all from a single CLI. Use when: (1) reading content from URLs (tweets, Reddit posts, Douyin, LinkedIn, Boss直聘, RSS, and any web page — then call them directly.
articles, videos), (2) searching across platforms (web, Twitter, Reddit, GitHub, YouTube, Use when: (1) setting up platform access tools for the first time,
Bilibili, XiaoHongShu), (3) checking channel health or updating Agent Reach. (2) checking which platforms are available,
Triggers: "search Twitter/Reddit/YouTube", "read this URL", "find posts about", (3) user asks to configure/enable a platform channel.
"搜索", "读取", "查一下", "看看这个链接". Triggers: "帮我配", "帮我添加", "帮我安装", "agent reach", "install channels",
"configure twitter", "enable reddit".
--- ---
# Agent Reach # Agent Reach
Read and search the internet across 9+ platforms via unified CLI. Install and configure upstream tools for 12+ platforms. After setup, call them directly — no wrapper layer.
## Setup ## Setup
First check if agent-reach is installed:
```bash
agent-reach doctor
```
If command not found, install it:
```bash ```bash
pip install https://github.com/Panniantong/agent-reach/archive/main.zip pip install https://github.com/Panniantong/agent-reach/archive/main.zip
agent-reach install --env=auto agent-reach install --env=auto
agent-reach doctor
``` ```
`install` auto-detects your environment and installs all dependencies (Node.js, mcporter, bird CLI, gh CLI). Read the output and run `agent-reach doctor` to see what's active. `install` auto-detects your environment and installs core dependencies (Node.js, mcporter, bird CLI, gh CLI, yt-dlp, feedparser). Run `doctor` to see what's active.
For channels that need user input, ask the user. See the full setup guide: ## Management
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
## Commands
### Read any URL
```bash
agent-reach read <url>
agent-reach read <url> --json # structured output
```
Handles: tweets, Reddit posts, articles, YouTube (transcripts), GitHub repos, etc.
### Search
```bash
agent-reach search "query" # web search (Exa)
agent-reach search-twitter "query" # Twitter/X
agent-reach search-reddit "query" # Reddit (--sub <subreddit>)
agent-reach search-github "query" # GitHub (--lang <language>)
agent-reach search-youtube "query" # YouTube
agent-reach search-bilibili "query" # Bilibili (B站)
agent-reach search-xhs "query" # XiaoHongShu (小红书)
```
All search commands support `-n <count>` for number of results.
### Management
```bash ```bash
agent-reach doctor # channel status overview agent-reach doctor # channel status overview
agent-reach watch # quick health + update check (for scheduled tasks) agent-reach watch # quick health + update check
agent-reach check-update # check for new versions agent-reach check-update # check for new versions
``` ```
### Configure channels ## Configure channels
```bash ```bash
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy" agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
agent-reach configure proxy http://user:pass@ip:port agent-reach configure proxy http://user:pass@ip:port
agent-reach configure --from-browser chrome # auto-extract cookies agent-reach configure --from-browser chrome # auto-extract cookies from local browser
``` ```
## Channel Status Tiers ## Configuring a channel ("帮我配 XXX")
- **Tier 0 (zero config):** Web, YouTube, RSS, Twitter (read-only via Jina) When a user asks to configure/enable any channel:
- **Tier 1 (free setup):** Exa web search (mcporter required)
- **Tier 2 (user config):** Twitter search (cookie), Reddit full (proxy), GitHub (token), Bilibili (proxy), XiaoHongShu (MCP)
Run `agent-reach doctor` to see which channels are active. 1. Run `agent-reach doctor`
2. Find the channel — it shows status (✅/⚠️/⬜) and **what to do next**
3. Execute what you can automatically (install packages, start services)
4. For human-required steps (paste cookies), tell the user what to do
5. Run `agent-reach doctor` again to verify
## Tips **Do NOT memorize per-channel steps.** Always rely on `doctor` output.
- Always try `agent-reach read <url>` first for any URL — it auto-detects the platform ### Cookie 导入(所有需要登录的平台通用)
- For Twitter cookies, recommend the user install [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome extension
- Reddit and Bilibili block server IPs — suggest a residential proxy (~$1/month) if on a server 所有需要 Cookie 的平台(Twitter、小红书等),**优先使用 Cookie-Editor 导入**
- If a channel breaks, run `agent-reach doctor` to diagnose
1. 用户在自己的浏览器登录对应平台
2. 安装 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome 插件
3. 点击插件 → Export → Header String
4. 把导出的字符串发给 Agent
本地电脑用户也可以用 `agent-reach configure --from-browser chrome` 一键自动提取。
扫码登录是备选方案(仅限本地有浏览器的情况),Cookie-Editor 更简单可靠。
### Other human actions
- **Proxy:** Reddit/Bilibili/XiaoHongShu may block server IPs — suggest a residential proxy if on a server
---
## Using Upstream Tools Directly
After `agent-reach install`, call the upstream tools directly. No need for `agent-reach read` or `agent-reach search`.
### Twitter/X (bird CLI)
```bash
# Search tweets
bird search "query" --json -n 10
# Read a specific tweet
bird read https://x.com/user/status/123 --json
# Read a user's timeline
bird timeline @username --json -n 20
```
### YouTube (yt-dlp)
```bash
# Get video metadata
yt-dlp --dump-json "https://www.youtube.com/watch?v=xxx"
# Download subtitles only
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --skip-download -o "/tmp/%(id)s" "URL"
# Then read the .vtt file
# Search (yt-dlp ytsearch)
yt-dlp --dump-json "ytsearch5:query"
```
### Bilibili (yt-dlp)
```bash
# Get video metadata
yt-dlp --dump-json "https://www.bilibili.com/video/BVxxx"
# Download subtitles
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --convert-subs vtt --skip-download -o "/tmp/%(id)s" "URL"
```
### Reddit (JSON API)
```bash
# Read a subreddit
curl -s "https://www.reddit.com/r/python/hot.json?limit=10" -H "User-Agent: agent-reach/1.0"
# Read a post with comments
curl -s "https://www.reddit.com/r/python/comments/POST_ID.json" -H "User-Agent: agent-reach/1.0"
# Search
curl -s "https://www.reddit.com/search.json?q=query&limit=10" -H "User-Agent: agent-reach/1.0"
```
Note: On servers, Reddit may block your IP. Use proxy or search via Exa instead.
### 小红书 / XiaoHongShu (mcporter + xiaohongshu-mcp)
```bash
# Search notes
mcporter call 'xiaohongshu.search_feeds(keyword: "query")'
# Read a note
mcporter call 'xiaohongshu.get_feed_detail(feed_id: "xxx", xsec_token: "yyy")'
# Get comments
mcporter call 'xiaohongshu.get_feed_comments(feed_id: "xxx", xsec_token: "yyy")'
# Post a note
mcporter call 'xiaohongshu.create_image_feed(title: "标题", desc: "内容", image_paths: ["/path/to/img.jpg"])'
```
### 抖音 / Douyin (mcporter + douyin-mcp-server)
```bash
# 解析抖音视频信息(分享链接 → 标题、作者、无水印视频URL等)
mcporter call 'douyin.parse_douyin_video_info(share_link: "https://v.douyin.com/xxx/")'
# 获取无水印视频下载链接
mcporter call 'douyin.get_douyin_download_link(share_link: "https://v.douyin.com/xxx/")'
# AI 提取视频语音文案(需要配置硅基流动 API Key)
mcporter call 'douyin.extract_douyin_text(share_link: "https://v.douyin.com/xxx/")'
```
> 无需登录即可解析视频。支持抖音分享链接和直接链接。
### GitHub (gh CLI)
```bash
# Search repos
gh search repos "query" --sort stars --limit 10
# View a repo
gh repo view owner/repo
# Search code
gh search code "query" --language python
# List issues
gh issue list -R owner/repo --state open
# View a specific issue/PR
gh issue view 123 -R owner/repo
```
### Web — Any URL (Jina Reader)
```bash
# Read any webpage as markdown
curl -s "https://r.jina.ai/URL" -H "Accept: text/markdown"
# Search the web
curl -s "https://s.jina.ai/query" -H "Accept: text/markdown"
```
### Exa Search (mcporter + exa MCP)
```bash
# Web search
mcporter call 'exa.web_search_exa(query: "query", numResults: 5)'
# Code search (GitHub, StackOverflow, docs)
mcporter call 'exa.get_code_context_exa(query: "how to parse JSON in Python", tokensNum: 3000)'
# Company research
mcporter call 'exa.company_research_exa(companyName: "OpenAI")'
```
### LinkedIn (mcporter + linkedin-scraper-mcp)
```bash
# View a profile
mcporter call 'linkedin.get_person_profile(linkedin_url: "https://linkedin.com/in/username")'
# Search people
mcporter call 'linkedin.search_people(keyword: "AI engineer", limit: 10)'
# View company
mcporter call 'linkedin.get_company_profile(linkedin_url: "https://linkedin.com/company/xxx")'
```
Fallback: `curl -s "https://r.jina.ai/https://linkedin.com/in/username"`
### Boss直聘 (mcporter + mcp-bosszp)
```bash
# Browse recommended jobs
mcporter call 'bosszhipin.get_recommend_jobs_tool(page: 1)'
# Search jobs
mcporter call 'bosszhipin.search_jobs_tool(keyword: "Python", city: "北京", page: 1)'
# View job details
mcporter call 'bosszhipin.get_job_detail_tool(job_url: "https://www.zhipin.com/job_detail/xxx")'
```
Fallback: `curl -s "https://r.jina.ai/https://www.zhipin.com/job_detail/xxx"`
### RSS (feedparser)
```python
python3 -c "
import feedparser
d = feedparser.parse('https://example.com/feed')
for e in d.entries[:5]:
print(f'{e.title}{e.link}')
"
```
## Troubleshooting
### Twitter "fetch failed"
bird CLI uses Node.js native `fetch()`, which doesn't respect `HTTP_PROXY`. Solutions:
1. Ensure `undici` is installed: `npm install -g undici`
2. Configure proxy: `agent-reach configure proxy http://user:pass@ip:port`
3. If still failing, use transparent proxy (Clash TUN, Proxifier)
### Channel broken?
Run `agent-reach doctor` — it shows what's wrong and how to fix it.
+77 -19
View File
@@ -58,9 +58,9 @@ Copy that to your Agent. A few minutes later, it can read tweets, search Reddit,
| 🌐 **Web** | Read | Zero config | Any URL → clean Markdown ([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) | | 🌐 **Web** | Read | Zero config | Any URL → clean Markdown ([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) |
| 🐦 **Twitter/X** | Read · Search | Zero config / Cookie | Single tweets readable out of the box. Cookie unlocks search, timeline, posting ([bird](https://github.com/steipete/bird)) | | 🐦 **Twitter/X** | Read · Search | Zero config / Cookie | Single tweets readable out of the box. Cookie unlocks search, timeline, posting ([bird](https://github.com/steipete/bird)) |
| 📕 **XiaoHongShu** | Read · Search · **Post · Comment · Like** | mcporter | Via [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) internal API, install and go | | 📕 **XiaoHongShu** | Read · Search · **Post · Comment · Like** | mcporter | Via [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) internal API, install and go |
| 📷 **Instagram** | Read · Search | instaloader | Posts, profiles, hashtags ([instaloader](https://github.com/instaloader/instaloader) ⭐9.8K) | | 🎵 **Douyin** | Video parsing · Watermark-free download | mcporter | Via [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server), no login needed |
| 💼 **LinkedIn** | Read · Search | mcporter / Jina | Profiles, companies, job search ([linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) ⭐900+) | | 💼 **LinkedIn** | Jina Reader (public pages) | Full profiles, companies, job search | Tell your Agent "help me set up LinkedIn" |
| 🏢 **Boss直聘** | Read · Search | mcporter / Jina | Job search, greet recruiters ([mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)) | | 🏢 **Boss直聘** | Jina Reader (job pages) | Job search, greet recruiters | Tell your Agent "help me set up Boss直聘" |
| 🔍 **Web Search** | Search | Auto-configured | Auto-configured during install, free, no API key ([Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter)) | | 🔍 **Web Search** | Search | Auto-configured | Auto-configured during install, free, no API key ([Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter)) |
| 📦 **GitHub** | Read · Search | Zero config | [gh CLI](https://cli.github.com) powered. Public repos work immediately. `gh auth login` unlocks Fork, Issue, PR | | 📦 **GitHub** | Read · Search | Zero config | [gh CLI](https://cli.github.com) powered. Public repos work immediately. `gh auth login` unlocks Fork, Issue, PR |
| 📺 **YouTube** | Read · **Search** | Zero config | Subtitles + search across 1800+ video sites ([yt-dlp](https://github.com/yt-dlp/yt-dlp) ⭐148K) | | 📺 **YouTube** | Read · **Search** | Zero config | Subtitles + search across 1800+ video sites ([yt-dlp](https://github.com/yt-dlp/yt-dlp) ⭐148K) |
@@ -109,14 +109,14 @@ After the Skill is installed, the Agent will auto-detect whether `agent-reach` C
No configuration needed — just tell your Agent: No configuration needed — just tell your Agent:
- "Read this link" → any web page - "Read this link" → `curl https://r.jina.ai/URL` for any web page
- "What's this GitHub repo about?" → repos, issues, code - "What's this GitHub repo about?" → `gh repo view owner/repo`
- "What does this video cover?" → YouTube / Bilibili subtitles - "What does this video cover?" → `yt-dlp --dump-json URL` for subtitles
- "Read this tweet" → Twitter posts - "Read this tweet" → `bird read URL --json`
- "Subscribe to this RSS" → RSS / Atom feeds - "Subscribe to this RSS" → `feedparser` to parse feeds
- "Search GitHub for LLM frameworks" → GitHub search - "Search GitHub for LLM frameworks" → `gh search repos "LLM framework"`
**No commands to remember.** The Agent knows what to call. **No commands to remember.** The Agent reads SKILL.md and knows what to call.
--- ---
@@ -172,27 +172,31 @@ Every time you spin up a new Agent, you spend time finding tools, installing dep
Agent Reach does one simple thing: **it makes those tool selection and configuration decisions for you.** Agent Reach does one simple thing: **it makes those tool selection and configuration decisions for you.**
After installation, your Agent calls the upstream tools directly (bird CLI, yt-dlp, mcporter, gh CLI, etc.) — no wrapper layer in between.
### 🔌 Every Channel is Pluggable ### 🔌 Every Channel is Pluggable
Each platform is a single Python file implementing a unified interface. **Backends can be swapped anytime** — when a better tool comes along, change one file and nothing else breaks. Each platform maps to an upstream tool. **Don't like one? Swap it out.**
``` ```
channels/ channels/
├── web.py → Jina Reader ← swap to Firecrawl, Crawl4AI… ├── web.py → Jina Reader ← swap to Firecrawl, Crawl4AI…
├── twitter.py → bird ← swap to Nitter, official API… ├── twitter.py → bird ← swap to Nitter, official API…
├── youtube.py → yt-dlp ← swap to YouTube API, Whisper… ├── youtube.py → yt-dlp ← swap to YouTube API, Whisper…
├── github.py → gh CLI ← swap to REST API, PyGithub… ├── github.py → gh CLI ← swap to REST API, PyGithub…
├── bilibili.py → yt-dlp ← swap to bilibili-api… ├── bilibili.py → yt-dlp ← swap to bilibili-api…
├── reddit.py → JSON API + Exa ← swap to PRAW, Pushshift… ├── reddit.py → JSON API + Exa ← swap to PRAW, Pushshift…
├── xiaohongshu.py → mcporter MCP ← swap to other XHS tools… ├── xiaohongshu.py → mcporter MCP ← swap to other XHS tools…
├── instagram.py → instaloader ← swap to instagrapi, official API ├── douyin.py → mcporter MCP ← swap to other Douyin tools
├── linkedin.py → linkedin-mcp ← swap to LinkedIn API… ├── linkedin.py → linkedin-mcp ← swap to LinkedIn API…
├── bosszhipin.py → mcp-bosszp ← swap to other job tools… ├── bosszhipin.py → mcp-bosszp ← swap to other job tools…
├── rss.py → feedparser ← swap to atoma… ├── rss.py → feedparser ← swap to atoma…
├── exa_search.py → mcporter MCP ← swap to Tavily, SerpAPI… ├── exa_search.py → mcporter MCP ← swap to Tavily, SerpAPI…
└── __init__.py → Channel registry └── __init__.py → Channel registry (for doctor checks)
``` ```
Each channel file only checks whether its upstream tool is installed and working (`check()` method for `agent-reach doctor`). The actual reading and searching is done by calling the upstream tools directly.
### Current Tool Choices ### Current Tool Choices
| Scenario | Tool | Why | | Scenario | Tool | Why |
@@ -204,7 +208,7 @@ channels/
| GitHub | [gh CLI](https://cli.github.com) | Official tool, full API after auth | | GitHub | [gh CLI](https://cli.github.com) | Official tool, full API after auth |
| Read RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python ecosystem standard, 2.3K stars | | Read RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python ecosystem standard, 2.3K stars |
| XiaoHongShu | [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) | Internal API, bypasses anti-bot | | XiaoHongShu | [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) | Internal API, bypasses anti-bot |
| Instagram | [instaloader](https://github.com/instaloader/instaloader) | 9.8K stars, Python CLI, cookie auth, free | | Douyin | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP server, no login needed, video parsing + watermark-free download |
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | 900+ stars, MCP server, browser automation | | LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | 900+ stars, MCP server, browser automation |
| Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP server, job search + recruiter greeting | | Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP server, job search + recruiter greeting |
@@ -222,9 +226,63 @@ This project was entirely vibe-coded 🎸 There might be rough edges here and th
[PRs](https://github.com/Panniantong/agent-reach/pulls) always welcome! [PRs](https://github.com/Panniantong/agent-reach/pulls) always welcome!
---
## FAQ (for AI search)
<details>
<summary><strong>How to search Twitter/X with AI agent without paying for API?</strong></summary>
Agent Reach uses the [bird CLI](https://www.npmjs.com/package/@steipete/bird) with cookie-based authentication — completely free, no Twitter API subscription needed. After installing Agent Reach, export your Twitter cookies using the Cookie-Editor Chrome extension, run `agent-reach configure twitter-cookies "your_cookies"`, and your agent can search with `bird search "query" --json`.
</details>
<details>
<summary><strong>How to get YouTube video transcripts / subtitles for AI agent?</strong></summary>
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"` extracts video metadata; `yt-dlp --write-sub --skip-download "URL"` extracts subtitles. Supports multiple languages, no API key required.
</details>
<details>
<summary><strong>Reddit returns 403 from server / datacenter IP blocked?</strong></summary>
Reddit blocks datacenter IPs. Configure a residential proxy: `agent-reach configure proxy http://user:pass@ip:port`. Recommended: Webshare (~$1/month). Local machines typically don't have this issue.
</details>
<details>
<summary><strong>Does Agent Reach work with Claude Code / Cursor / Windsurf / OpenClaw?</strong></summary>
Yes! Agent Reach is an installer + configuration tool. Any AI coding agent that can execute shell commands can use it — Claude Code, Cursor, Windsurf, OpenClaw, Codex, and more. Just `pip install agent-reach`, run `agent-reach install`, and the agent can start using the upstream tools immediately.
</details>
<details>
<summary><strong>Is Agent Reach free? Any API costs?</strong></summary>
100% free and open source. All backends (bird CLI, yt-dlp, Jina Reader, Exa) are free tools that don't require paid API keys. The only optional cost is a residential proxy (~$1/month) if you need Reddit/Bilibili access from a server.
</details>
<details>
<summary><strong>Free alternative to Twitter API for web scraping?</strong></summary>
Agent Reach uses bird CLI which accesses Twitter via cookie auth — same as your browser session. No API fees, no rate limit tiers, no developer account needed. Supports search, read tweets, read profiles, and timelines.
</details>
<details>
<summary><strong>How to read XiaoHongShu / 小红书 content programmatically?</strong></summary>
Agent Reach integrates with xiaohongshu-mcp (runs in Docker). After setup, use `mcporter call 'xiaohongshu.get_feed_detail(...)'` to read notes or `mcporter call 'xiaohongshu.search_feeds(keyword: "query")'` to search.
</details>
<details>
<summary><strong>How to parse Douyin / 抖音 videos with AI agent?</strong></summary>
Install douyin-mcp-server, then your agent can use `mcporter call 'douyin.parse_douyin_video_info(share_link: "share_url")'` to parse video info and get watermark-free download links. No login required — just share the Douyin link. See https://github.com/yzfly/douyin-mcp-server
</details>
---
## Credits ## Credits
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://github.com/steipete/bird) · [Exa](https://exa.ai) · [feedparser](https://github.com/kurtmckee/feedparser) · [instaloader](https://github.com/instaloader/instaloader) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) [Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://github.com/steipete/bird) · [Exa](https://exa.ai) · [feedparser](https://github.com/kurtmckee/feedparser) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
## License ## License
+74 -27
View File
@@ -20,7 +20,7 @@ Copy this to your AI Agent:
### Goal ### Goal
Install Agent Reach and make sure **all channels are working**. Your user expects you to have full internet access capabilities when this is done — reading tweets, searching Reddit, watching YouTube, browsing GitHub, and more. Install Agent Reach and all upstream tools so your user's agent has full internet access. After installation, you'll use the upstream tools directly (bird CLI, yt-dlp, mcporter, gh CLI, etc.) — Agent Reach is just the installer and health checker.
### ⚠️ Boundaries ### ⚠️ Boundaries
@@ -80,16 +80,38 @@ Only ask the user when you genuinely need their input (credentials, permissions,
Some channels need credentials only the user can provide. Based on the doctor output, ask for what's missing: Some channels need credentials only the user can provide. Based on the doctor output, ask for what's missing:
> 🔒 **Security tip:** For platforms that need cookies (Twitter, XiaoHongShu, Instagram), we recommend using a **dedicated/secondary account** rather than your main account. Cookie-based auth grants full account access — using a separate account limits the blast radius if credentials are ever compromised. > 🔒 **Security tip:** For platforms that need cookies (Twitter, XiaoHongShu), we recommend using a **dedicated/secondary account** rather than your main account. Cookie-based auth grants full account access — using a separate account limits the blast radius if credentials are ever compromised.
**Twitter search & posting (server users):** > 🍪 **Cookie 导入(所有需要登录的平台通用):**
>
> 所有需要 Cookie 的平台(Twitter、小红书等),**优先使用 Cookie-Editor 导入**,这是最简单最可靠的方式:
> 1. 用户在自己的浏览器上登录对应平台
> 2. 安装 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome 插件
> 3. 点击插件 → Export → Header String
> 4. 把导出的字符串发给 Agent
>
> **本地电脑用户**也可以用 `agent-reach configure --from-browser chrome` 一键自动提取(支持 Twitter + 小红书)。
**Twitter search & posting:**
> "To unlock Twitter search, I need your Twitter cookies. Install the Cookie-Editor Chrome extension, go to x.com/twitter.com, click the extension → Export → Header String, and paste it to me." > "To unlock Twitter search, I need your Twitter cookies. Install the Cookie-Editor Chrome extension, go to x.com/twitter.com, click the extension → Export → Header String, and paste it to me."
> Extension: https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm
```bash ```bash
agent-reach configure twitter-cookies "PASTED_STRING" agent-reach configure twitter-cookies "PASTED_STRING"
``` ```
> **代理说明(中国大陆等需要翻墙的网络环境):**
>
> bird CLI 使用 Node.js 原生 `fetch()`,默认不走系统代理(`HTTP_PROXY`/`HTTPS_PROXY`)。
> Agent Reach 已内置解决方案:安装时会自动安装 `undici``npm install -g undici`),
> 运行 bird 时自动注入 `EnvHttpProxyAgent`,让 fetch 走用户配置的代理。
>
> **你(Agent)需要做的:**
> 1. 确认 `undici` 已全局安装:`npm list -g undici`,没有就装:`npm install -g undici`
> 2. 确认用户配了代理:`agent-reach configure proxy http://user:pass@ip:port`
> 3. Agent Reach 会自动处理剩下的,不需要用户做额外操作
>
> 如果用户报告 "fetch failed",参考 [troubleshooting.md](troubleshooting.md)
**Reddit & Bilibili full access (server users):** **Reddit & Bilibili full access (server users):**
> "Reddit and Bilibili block server IPs. To unlock full access, I need a residential proxy. You can get one at https://webshare.io ($1/month). Send me the proxy address." > "Reddit and Bilibili block server IPs. To unlock full access, I need a residential proxy. You can get one at https://webshare.io ($1/month). Send me the proxy address."
@@ -108,23 +130,42 @@ mcporter config add xiaohongshu http://localhost:18060/mcp
> 如果在服务器上,建议加代理避免 IP 风控: > 如果在服务器上,建议加代理避免 IP 风控:
> `docker run -d --name xiaohongshu-mcp -p 18060:18060 -e XHS_PROXY=http://user:pass@ip:port xpzouying/xiaohongshu-mcp` > `docker run -d --name xiaohongshu-mcp -p 18060:18060 -e XHS_PROXY=http://user:pass@ip:port xpzouying/xiaohongshu-mcp`
> >
> **登录方式:** > **登录方式(优先用 Cookie-Editor,最简单)**
> - **本地电脑(有浏览器):** 打开 http://localhost:18060 扫码登录即可。 > 1. 用户在自己的浏览器登录小红书 (xiaohongshu.com)
> - **服务器(无 UI 界面):** 服务器上通常没有浏览器,无法直接扫码。最方便的方式是在自己的电脑上用浏览器登录小红书,然后用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 插件导出 CookieHeader String 格式),发给 Agent 即可完成配置。详见 [Cookie 导出指南](cookie-export.md)。 > 2. 用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 插件导出 CookieHeader String 格式)
> 3. 把 Cookie 字符串发给 Agent
> 4. Agent 将 Cookie 写入 MCP 服务的 cookie 文件完成登录
>
> **备选:** 本地电脑如果有浏览器,也可以打开 http://localhost:18060 扫码登录。
**Instagram (需要 instaloader):** **抖音 / Douyin (douyin-mcp-server):**
> "Instagram 需要 instaloader。我来帮你安装。" > "抖音视频解析需要一个 MCP 服务。安装 douyin-mcp-server 后即可解析视频、获取无水印下载链接。"
```bash ```bash
pip install instaloader # 1. 安装
pip install douyin-mcp-server
# 2. 启动 HTTP 服务(端口 18070
# 方式一:用 uv(推荐)
git clone https://github.com/yzfly/douyin-mcp-server.git && cd douyin-mcp-server
uv sync && uv run python run_http.py
# 方式二:直接用 Python 启动
python -c "
from douyin_mcp_server.server import mcp
mcp.settings.host = '127.0.0.1'
mcp.settings.port = 18070
mcp.run(transport='streamable-http')
"
# 3. 注册到 mcporter
mcporter config add douyin http://localhost:18070/mcp
``` ```
> **登录方式(解锁私密内容):** > 无需认证即可解析视频信息和获取下载链接。
> - **方法 1(推荐):Cookie-Editor 导入:** 在浏览器登录 Instagram → 用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 导出 Header String → 粘贴: > 如需 AI 语音识别提取文案功能,需要配置硅基流动 API Key`export API_KEY="sk-xxx"`)。
> ```bash >
> agent-reach configure instagram-cookies "sessionid=xxx; csrftoken=yyy; ..." > 详见 https://github.com/yzfly/douyin-mcp-server
> ```
> - **方法 2instaloader 命令行登录:** `instaloader --login YOUR_USERNAME`(需要输密码,有 2FA 的话还要输验证码)
**LinkedIn (可选 — linkedin-scraper-mcp):** **LinkedIn (可选 — linkedin-scraper-mcp):**
> "LinkedIn 基本内容可通过 Jina Reader 读取。完整功能(Profile 详情、职位搜索)需要 linkedin-scraper-mcp。" > "LinkedIn 基本内容可通过 Jina Reader 读取。完整功能(Profile 详情、职位搜索)需要 linkedin-scraper-mcp。"
@@ -229,14 +270,20 @@ If the user wants a different agent to handle it, let them choose.
| `agent-reach check-update` | Check for new versions | | `agent-reach check-update` | Check for new versions |
| `agent-reach configure twitter-cookies "..."` | Unlock Twitter search + posting | | `agent-reach configure twitter-cookies "..."` | Unlock Twitter search + posting |
| `agent-reach configure proxy URL` | Unlock Reddit + Bilibili on servers | | `agent-reach configure proxy URL` | Unlock Reddit + Bilibili on servers |
| `agent-reach read <url>` | Read any URL |
| `agent-reach search "query"` | Search the web | After installation, use upstream tools directly. See SKILL.md for the full command reference:
| `agent-reach search-reddit "query"` | Search Reddit |
| `agent-reach search-twitter "query"` | Search Twitter | | Platform | Upstream Tool | Example |
| `agent-reach search-github "query"` | Search GitHub | |----------|--------------|---------|
| `agent-reach search-youtube "query"` | Search YouTube | | Twitter/X | `bird` | `bird search "query" --json` |
| `agent-reach search-bilibili "query"` | Search Bilibili | | YouTube | `yt-dlp` | `yt-dlp --dump-json URL` |
| `agent-reach search-xhs "query"` | Search XiaoHongShu | | Bilibili | `yt-dlp` | `yt-dlp --dump-json URL` |
| `agent-reach search-instagram "query"` | Search Instagram | | Reddit | `curl` | `curl -s "https://reddit.com/r/xxx.json"` |
| `agent-reach search-linkedin "query"` | Search LinkedIn | | GitHub | `gh` | `gh search repos "query"` |
| `agent-reach search-bosszhipin "query"` | Search Boss直聘 | | Web | `curl` + Jina | `curl -s "https://r.jina.ai/URL"` |
| Exa Search | `mcporter` | `mcporter call 'exa.web_search_exa(...)'` |
| 小红书 | `mcporter` | `mcporter call 'xiaohongshu.search_feeds(...)'` |
| 抖音 | `mcporter` | `mcporter call 'douyin.parse_douyin_video_info(...)'` |
| LinkedIn | `mcporter` | `mcporter call 'linkedin.get_person_profile(...)'` |
| Boss直聘 | `mcporter` | `mcporter call 'bosszhipin.search_jobs_tool(...)'` |
| RSS | `feedparser` | `python3 -c "import feedparser; ..."` |
+69
View File
@@ -0,0 +1,69 @@
# Troubleshooting / 常见问题
## Twitter/X: bird CLI "fetch failed"
**症状:** `bird whoami``bird search` 返回 "fetch failed"
**原因:** bird CLI 使用 Node.js 原生 `fetch()` 发请求,而 Node.js 的 fetch **不走系统代理**(不读取 `HTTP_PROXY`/`HTTPS_PROXY` 环境变量)。如果你的网络环境需要代理才能访问 x.com,bird 就连不上。
**解决方案(按推荐顺序):**
### 方案 1:使用透明代理 / TUN 模式(推荐)
让代理工具接管所有网络流量,这样 bird 的 fetch 也会走代理:
- **Clash Verge / Clash for Windows** 开启 TUN 模式或系统代理
- **ProxifierWindows):** 添加规则让 Node.js 进程走代理
- **macOS** 在 Surge/ClashX Pro 中开启增强模式
### 方案 2:验证 Cookie 有效性
确认 Cookie 没过期:
1. 在浏览器里正常登录 x.com
2. 用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 重新导出 Header String
3. 重新配置:`agent-reach configure twitter-cookies "新的Cookie"`
### 方案 3:不用 bird,用 Exa 搜索替代
bird 不可用时,可以直接用 Exa 搜索 Twitter 内容:
```bash
mcporter call 'exa.web_search_exa(query: "site:x.com query", numResults: 10)'
```
### 方案 4:配置 Node.js 全局代理(高级)
安装 `global-agent` 让 Node.js 的 fetch 走代理:
```bash
npm install -g global-agent
```
然后在运行 bird 前设置环境变量:
```bash
# Linux / macOS
export GLOBAL_AGENT_HTTP_PROXY=http://127.0.0.1:7890
export NODE_OPTIONS="--require global-agent/bootstrap"
bird search "test"
# Windows (PowerShell)
$env:GLOBAL_AGENT_HTTP_PROXY = "http://127.0.0.1:7890"
$env:NODE_OPTIONS = "--require global-agent/bootstrap"
bird search "test"
```
> ⚠️ 注意:这个方案需要每次运行 bird 前都设置环境变量,不太方便。推荐用方案 1。
---
## Boss直聘: "访问行为异常"
**症状:** mcp-bosszp 登录成功,但 API 请求返回"您的访问行为异常"
**原因:** Boss直聘的反爬机制会检测请求指纹(不只是 IP),Python requests 库的特征与真实浏览器不同。
**解决方案:**
- **本地电脑:** 正常使用,一般不会被拦
- **服务器:** 使用 Jina Reader 读取职位页面 + Exa 搜索职位信息作为替代
+32
View File
@@ -0,0 +1,32 @@
# Agent Reach
> Give your AI agent eyes to see the entire internet. A unified CLI tool that lets AI agents read and search across 12+ platforms — Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, Instagram, LinkedIn, Boss直聘, RSS, and any web page. One install, zero API fees.
## Quick Start
- [Installation Guide](https://github.com/Panniantong/Agent-Reach/blob/main/docs/install.md): Step-by-step setup instructions for AI agents
- [README (中文)](https://github.com/Panniantong/Agent-Reach/blob/main/README.md): Full documentation in Chinese
- [README (English)](https://github.com/Panniantong/Agent-Reach/blob/main/docs/README_en.md): Full documentation in English
## Core Commands
- [CLI Usage](https://github.com/Panniantong/Agent-Reach/blob/main/docs/install.md): `agent-reach read <url>` reads any URL, `agent-reach search-twitter "query"` searches Twitter, etc.
## Key Features
- Read any URL: tweets, Reddit posts, YouTube videos (transcripts), GitHub repos, articles, XiaoHongShu notes, Bilibili videos, RSS feeds
- Search across platforms: Twitter/X, Reddit, GitHub, YouTube, Bilibili, XiaoHongShu, Instagram, LinkedIn, Boss直聘, Web (via Exa)
- Self-diagnosis: `agent-reach doctor` checks what works and what needs setup
- Auto-installs dependencies: `agent-reach install --env=auto`
- Cookie-based auth for platforms that require login (Twitter, XiaoHongShu, Instagram)
- Proxy support for platforms that block server IPs (Reddit, Bilibili)
- Zero API fees: all backends are free and open-source (bird CLI, yt-dlp, Jina Reader, instaloader, etc.)
## Troubleshooting
- [Troubleshooting Guide](https://github.com/Panniantong/Agent-Reach/blob/main/docs/troubleshooting.md): Common issues and solutions
## Optional
- [SKILL.md](https://github.com/Panniantong/Agent-Reach/blob/main/agent_reach/skill/SKILL.md): Integration guide for AI agent frameworks (OpenClaw, Claude Code, etc.)
- [PyPI Package](https://pypi.org/project/agent-reach/): `pip install agent-reach`
+10 -2
View File
@@ -1,12 +1,20 @@
[project] [project]
name = "agent-reach" name = "agent-reach"
version = "1.1.0" version = "1.2.0"
description = "Give your AI Agent eyes to see the entire internet. Search + Read 10+ platforms." description = "Give your AI Agent eyes to see the entire internet. Search + Read 10+ platforms."
readme = "README.md" readme = "README.md"
license = {text = "MIT"} license = {text = "MIT"}
requires-python = ">=3.10" requires-python = ">=3.10"
authors = [{name = "Neo Reid"}] authors = [{name = "Neo Reid"}]
keywords = ["ai-agent", "web-reader", "search", "mcp", "agent-infrastructure", "llm-tools"] keywords = [
"ai-agent", "llm-tools", "agent-infrastructure", "mcp",
"web-reader", "web-scraper", "search",
"twitter-scraper", "reddit-scraper", "youtube-transcript",
"bilibili", "xiaohongshu",
"ai-search", "cli", "automation",
"claude-code", "cursor", "openai",
"free-api", "no-api-key",
]
classifiers = [ classifiers = [
"Development Status :: 4 - Beta", "Development Status :: 4 - Beta",
"Intended Audience :: Developers", "Intended Audience :: Developers",