Compare commits
22 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 37b4cdeddc | |||
| bf6dc7084b | |||
| 50f2181fcc | |||
| b387a67b51 | |||
| a96c5cda14 | |||
| cf70302e92 | |||
| 80a08c8c10 | |||
| 606e8900ce | |||
| a37e9aa190 | |||
| 1cbf6a7b9c | |||
| 68319ec683 | |||
| f70711e75e | |||
| c3a9813b1c | |||
| d291d27664 | |||
| 0be4f097d0 | |||
| 12bd727e30 | |||
| b6d135c977 | |||
| 291d073223 | |||
| ef0889dd69 | |||
| 545a6c5761 | |||
| 00f3edd9cc | |||
| 2f674f1e45 |
+9
-12
@@ -10,13 +10,10 @@ All notable changes to this project will be documented in this file.
|
|||||||
|
|
||||||
### 🆕 New Channels / 新增渠道
|
### 🆕 New Channels / 新增渠道
|
||||||
|
|
||||||
#### 📷 Instagram
|
#### ~~📷 Instagram~~ (removed — upstream blocked)
|
||||||
- Read public posts and profiles via [instaloader](https://github.com/instaloader/instaloader)
|
- ~~Read public posts and profiles via [instaloader](https://github.com/instaloader/instaloader)~~
|
||||||
- Search via Exa (free, no API key)
|
- **Removed:** Instagram's aggressive anti-scraping measures broke all available open-source tools (instaloader, etc.). See [instaloader#2585](https://github.com/instaloader/instaloader/issues/2585). Will re-add when upstream recovers.
|
||||||
- Optional cookie login for private content
|
- **已移除:** Instagram 反爬封杀导致所有开源工具(instaloader 等)失效。上游恢复后会重新加回。
|
||||||
- 通过 instaloader 读取公开帖子和 Profile
|
|
||||||
- 搜索通过 Exa(免费,无需 API Key)
|
|
||||||
- 可选 Cookie 登录解锁私密内容
|
|
||||||
|
|
||||||
#### 💼 LinkedIn
|
#### 💼 LinkedIn
|
||||||
- Read person profiles, company pages, and job details via [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
- Read person profiles, company pages, and job details via [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||||
@@ -38,12 +35,12 @@ All notable changes to this project will be documented in this file.
|
|||||||
|
|
||||||
- Channel count: 9 → 12
|
- Channel count: 9 → 12
|
||||||
- `agent-reach doctor` now detects all 12 channels
|
- `agent-reach doctor` now detects all 12 channels
|
||||||
- CLI: added `search-instagram`, `search-linkedin`, `search-bosszhipin` subcommands
|
- CLI: added `search-linkedin`, `search-bosszhipin` subcommands
|
||||||
- Updated install guide with setup instructions for new channels
|
- Updated install guide with setup instructions for new channels
|
||||||
- 渠道数量:9 → 12
|
- 渠道数量:9 → 11
|
||||||
- `agent-reach doctor` 现在检测全部 12 个渠道
|
- `agent-reach doctor` 现在检测全部 11 个渠道
|
||||||
- CLI:新增 `search-instagram`、`search-linkedin`、`search-bosszhipin` 子命令
|
- CLI:新增 `search-linkedin`、`search-bosszhipin` 子命令
|
||||||
- 安装指南新增三个渠道的配置说明
|
- 安装指南新增渠道配置说明
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -11,7 +11,7 @@
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<a href="#30-秒上手">快速开始</a> · <a href="docs/README_en.md">English</a> · <a href="#支持的平台">支持平台</a> · <a href="#设计理念">设计理念</a>
|
<a href="#快速上手">快速开始</a> · <a href="docs/README_en.md">English</a> · <a href="#支持的平台">支持平台</a> · <a href="#设计理念">设计理念</a>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -68,14 +68,14 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
| 🐦 **Twitter/X** | 读单条推文 | 搜索推文、浏览时间线、发推 | 告诉 Agent「帮我配 Twitter」 |
|
| 🐦 **Twitter/X** | 读单条推文 | 搜索推文、浏览时间线、发推 | 告诉 Agent「帮我配 Twitter」 |
|
||||||
| 📺 **B站** | 本地:字幕提取 + 搜索 | 服务器也能用 | 告诉 Agent「帮我配代理」 |
|
| 📺 **B站** | 本地:字幕提取 + 搜索 | 服务器也能用 | 告诉 Agent「帮我配代理」 |
|
||||||
| 📖 **Reddit** | 搜索(通过 Exa 免费) | 读帖子和评论 | 告诉 Agent「帮我配代理」 |
|
| 📖 **Reddit** | 搜索(通过 Exa 免费) | 读帖子和评论 | 告诉 Agent「帮我配代理」 |
|
||||||
| 📕 **小红书** | — | 阅读、搜索、发帖、评论、点赞 | `docker run -d -p 18060:18060 xpzouying/xiaohongshu-mcp` 然后告诉 Agent「帮我配置小红书」 |
|
| 📕 **小红书** | — | 阅读、搜索、发帖、评论、点赞 | 告诉 Agent「帮我配小红书」 |
|
||||||
| 📷 **Instagram** | — | 读取帖子和 Profile | Cookie-Editor 导出 Cookie → `agent-reach configure instagram-cookies "..."` |
|
| 🎵 **抖音** | — | 视频解析、无水印下载链接获取 | 告诉 Agent「帮我配抖音」 |
|
||||||
| 💼 **LinkedIn** | Jina Reader 读公开页面 | Profile 详情、公司、职位搜索 | `linkedin-scraper-mcp --login`(需浏览器,服务器用 VNC) |
|
| 💼 **LinkedIn** | Jina Reader 读公开页面 | Profile 详情、公司页面、职位搜索 | 告诉 Agent「帮我配 LinkedIn」 |
|
||||||
| 🏢 **Boss直聘** | Jina Reader 读职位页 | 搜索职位、向 HR 打招呼 | 启动 mcp-bosszp → Boss直聘 App 扫码 |
|
| 🏢 **Boss直聘** | Jina Reader 读职位页 | 搜索职位、向 HR 打招呼 | 告诉 Agent「帮我配 Boss直聘」 |
|
||||||
|
|
||||||
> **不知道怎么配?不用查文档。** 直接告诉 Agent「帮我配 XXX」,它知道需要什么、会一步一步引导你。
|
> **不知道怎么配?不用查文档。** 直接告诉 Agent「帮我配 XXX」,它知道需要什么、会一步一步引导你。
|
||||||
>
|
>
|
||||||
> 🍪 需要 Cookie 的平台(Twitter、小红书等),建议使用 Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 一键导出。**服务器用户**没有浏览器界面,请在自己的电脑上登录对应网站后导出 Cookie,再发给 Agent 配置。
|
> 🍪 需要 Cookie 的平台(Twitter、小红书等),**优先使用** Chrome 插件 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 导出 Cookie,发给 Agent 即可配置。流程统一:浏览器登录 → Cookie-Editor 导出 → 发给 Agent。比扫码更简单可靠。
|
||||||
>
|
>
|
||||||
> 🔒 Cookie 只存在你本地,不上传不外传。代码完全开源,随时可审查。
|
> 🔒 Cookie 只存在你本地,不上传不外传。代码完全开源,随时可审查。
|
||||||
> 💻 本地电脑不需要代理。代理只有部署在服务器上才需要(~$1/月)。
|
> 💻 本地电脑不需要代理。代理只有部署在服务器上才需要(~$1/月)。
|
||||||
@@ -105,7 +105,7 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
2. **安装系统依赖** — 自动检测并安装 Node.js、gh CLI、mcporter、bird 等
|
2. **安装系统依赖** — 自动检测并安装 Node.js、gh CLI、mcporter、bird 等
|
||||||
3. **配置搜索引擎** — 通过 MCP 接入 Exa(免费,无需 API Key)
|
3. **配置搜索引擎** — 通过 MCP 接入 Exa(免费,无需 API Key)
|
||||||
4. **检测环境** — 判断是本地电脑还是服务器,给出对应的配置建议
|
4. **检测环境** — 判断是本地电脑还是服务器,给出对应的配置建议
|
||||||
5. **注册 Skill** — 在 Agent 的 skills 目录安装 SKILL.md,以后 Agent 遇到"搜推特"、"看视频"这类需求,会自动知道调用 Agent Reach
|
5. **注册 SKILL.md** — 在 Agent 的 skills 目录安装使用指南,以后 Agent 遇到"搜推特"、"看视频"这类需求,会自动知道该调哪个上游工具
|
||||||
|
|
||||||
安装完之后,`agent-reach doctor` 一条命令告诉你每个渠道的状态。
|
安装完之后,`agent-reach doctor` 一条命令告诉你每个渠道的状态。
|
||||||
</details>
|
</details>
|
||||||
@@ -116,14 +116,14 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|
|
||||||
不需要任何配置,告诉 Agent 就行:
|
不需要任何配置,告诉 Agent 就行:
|
||||||
|
|
||||||
- "帮我看看这个链接" → 任意网页
|
- "帮我看看这个链接" → `curl https://r.jina.ai/URL` 读任意网页
|
||||||
- "这个 GitHub 仓库是做什么的" → GitHub 仓库、Issue、代码
|
- "这个 GitHub 仓库是做什么的" → `gh repo view owner/repo`
|
||||||
- "这个视频讲了什么" → YouTube / B站字幕提取
|
- "这个视频讲了什么" → `yt-dlp --dump-json URL` 提取字幕
|
||||||
- "帮我看看这条推文" → Twitter 推文
|
- "帮我看看这条推文" → `bird read URL --json`
|
||||||
- "订阅这个 RSS" → RSS / Atom 源
|
- "订阅这个 RSS" → `feedparser` 解析
|
||||||
- "搜一下 GitHub 上有什么 LLM 框架" → GitHub 搜索
|
- "搜一下 GitHub 上有什么 LLM 框架" → `gh search repos "LLM framework"`
|
||||||
|
|
||||||
**不需要记命令。** Agent 自己知道该调什么。
|
**不需要记命令。** Agent 读了 SKILL.md 之后自己知道该调什么。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -135,9 +135,11 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
|||||||
|
|
||||||
Agent Reach 做的事情很简单:**帮你把这些选型和配置的活儿做完了。**
|
Agent Reach 做的事情很简单:**帮你把这些选型和配置的活儿做完了。**
|
||||||
|
|
||||||
|
安装完成后,Agent 直接调用上游工具(bird CLI、yt-dlp、mcporter、gh CLI 等),不需要经过 Agent Reach 的包装层。
|
||||||
|
|
||||||
### 🔌 每个渠道都是可插拔的
|
### 🔌 每个渠道都是可插拔的
|
||||||
|
|
||||||
每个平台对应一个独立的 Python 文件,实现统一接口。**后端工具随时可以换**——哪天出了更好的工具,改一个文件就行,其他不用动。
|
每个平台背后是一个独立的上游工具。**不满意?换掉就行。**
|
||||||
|
|
||||||
```
|
```
|
||||||
channels/
|
channels/
|
||||||
@@ -148,14 +150,16 @@ channels/
|
|||||||
├── bilibili.py → yt-dlp ← 可以换成 bilibili-api……
|
├── bilibili.py → yt-dlp ← 可以换成 bilibili-api……
|
||||||
├── reddit.py → JSON API + Exa ← 可以换成 PRAW、Pushshift……
|
├── reddit.py → JSON API + Exa ← 可以换成 PRAW、Pushshift……
|
||||||
├── xiaohongshu.py → mcporter MCP ← 可以换成其他 XHS 工具……
|
├── xiaohongshu.py → mcporter MCP ← 可以换成其他 XHS 工具……
|
||||||
├── instagram.py → instaloader ← 可以换成 instagrapi、官方 API……
|
├── douyin.py → mcporter MCP ← 可以换成其他抖音工具……
|
||||||
├── linkedin.py → linkedin-mcp ← 可以换成 LinkedIn API……
|
├── linkedin.py → linkedin-mcp ← 可以换成 LinkedIn API……
|
||||||
├── bosszhipin.py → mcp-bosszp ← 可以换成其他招聘工具……
|
├── bosszhipin.py → mcp-bosszp ← 可以换成其他招聘工具……
|
||||||
├── rss.py → feedparser ← 可以换成 atoma……
|
├── rss.py → feedparser ← 可以换成 atoma……
|
||||||
├── exa_search.py → mcporter MCP ← 可以换成 Tavily、SerpAPI……
|
├── exa_search.py → mcporter MCP ← 可以换成 Tavily、SerpAPI……
|
||||||
└── __init__.py → 渠道注册
|
└── __init__.py → 渠道注册(doctor 检测用)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
每个渠道文件只负责检测对应上游工具是否可用(`check()` 方法),给 `agent-reach doctor` 提供状态信息。实际的读取和搜索由 Agent 直接调用上游工具完成。
|
||||||
|
|
||||||
### 当前选型
|
### 当前选型
|
||||||
|
|
||||||
| 场景 | 选型 | 为什么选它 |
|
| 场景 | 选型 | 为什么选它 |
|
||||||
@@ -167,7 +171,7 @@ channels/
|
|||||||
| GitHub | [gh CLI](https://cli.github.com) | 官方工具,认证后完整 API 能力 |
|
| GitHub | [gh CLI](https://cli.github.com) | 官方工具,认证后完整 API 能力 |
|
||||||
| 读 RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python 生态标准选择,2.3K Star |
|
| 读 RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python 生态标准选择,2.3K Star |
|
||||||
| 小红书 | [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) | ⭐9K+,Go 语言,Docker 一键部署 |
|
| 小红书 | [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) | ⭐9K+,Go 语言,Docker 一键部署 |
|
||||||
| Instagram | [instaloader](https://github.com/instaloader/instaloader) | ⭐9.8K,Python CLI,Cookie 登录,免费 |
|
| 抖音 | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP 服务,无需登录,视频解析 + 无水印下载 |
|
||||||
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | ⭐900+,MCP 服务,浏览器自动化 |
|
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | ⭐900+,MCP 服务,浏览器自动化 |
|
||||||
| Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP 服务,支持职位搜索和打招呼 |
|
| Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP 服务,支持职位搜索和打招呼 |
|
||||||
|
|
||||||
@@ -189,7 +193,7 @@ Agent Reach 在设计上重视安全:
|
|||||||
|
|
||||||
### 🍪 Cookie 安全建议
|
### 🍪 Cookie 安全建议
|
||||||
|
|
||||||
需要 Cookie 的平台(Twitter、小红书、Instagram)建议使用**专用小号**,不要用主账号。Cookie 等同于完整登录权限,用小号可以在凭据泄露时限制影响范围。
|
需要 Cookie 的平台(Twitter、小红书)建议使用**专用小号**,不要用主账号。Cookie 等同于完整登录权限,用小号可以在凭据泄露时限制影响范围。
|
||||||
|
|
||||||
### 📦 安装方式
|
### 📦 安装方式
|
||||||
|
|
||||||
@@ -225,9 +229,63 @@ Agent Reach 在设计上重视安全:
|
|||||||
|
|
||||||
Star 一下,下次需要的时候能找到。⭐
|
Star 一下,下次需要的时候能找到。⭐
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 常见问题 / FAQ
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>AI Agent 怎么搜索 Twitter / X?不想付 API 费用</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach 使用 [bird CLI](https://www.npmjs.com/package/@steipete/bird) 通过 Cookie 认证访问 Twitter,完全免费。安装 Agent Reach 后,用 Cookie-Editor 导出你的 Twitter Cookie,运行 `agent-reach configure twitter-cookies "your_cookies"` 即可。之后 Agent 就可以用 `bird search "关键词" --json` 搜索推文了。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to search Twitter/X with AI agent for free (no API)?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach uses the bird CLI with cookie auth — zero API fees. After installing, export your Twitter cookies with the Cookie-Editor extension, run `agent-reach configure twitter-cookies "your_cookies"`, then your agent can search with `bird search "query" --json`.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Reddit 返回 403 / 服务器 IP 被封怎么办?</strong></summary>
|
||||||
|
|
||||||
|
Reddit 封锁数据中心 IP。配置一个住宅代理即可解决:`agent-reach configure proxy http://user:pass@ip:port`。推荐 Webshare ($1/月)。本地电脑一般不会遇到这个问题。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to get YouTube video transcripts for AI?</strong></summary>
|
||||||
|
|
||||||
|
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"` extracts video metadata; `yt-dlp --write-sub --skip-download "URL"` extracts subtitles. Uses yt-dlp under the hood, supports multiple languages. No API key needed.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>怎么让 AI Agent 读小红书?</strong></summary>
|
||||||
|
|
||||||
|
小红书需要通过 Docker 运行一个 MCP 服务。安装 Docker 后,运行 `agent-reach install` 会自动配置。之后 Agent 就能用 `mcporter call 'xiaohongshu.get_feed_detail(...)'` 读取笔记或 `mcporter call 'xiaohongshu.search_feeds(keyword: "关键词")'` 搜索了。
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>怎么让 AI Agent 解析抖音视频?</strong></summary>
|
||||||
|
|
||||||
|
安装 douyin-mcp-server 后,Agent 就能用 `mcporter call 'douyin.parse_douyin_video_info(share_link: "分享链接")'` 解析视频信息、获取无水印下载链接。不需要登录,把抖音分享链接发给 Agent 就行。详见 https://github.com/yzfly/douyin-mcp-server
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Compatible with Claude Code / Cursor / OpenClaw / Windsurf?</strong></summary>
|
||||||
|
|
||||||
|
Yes! Agent Reach is an installer + configuration tool — any AI coding agent that can run shell commands can use it. Works with Claude Code, Cursor, OpenClaw, Windsurf, Codex, and more. Just `pip install agent-reach`, run `agent-reach install`, and the agent can start using the upstream tools immediately.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Is this free? Any API costs?</strong></summary>
|
||||||
|
|
||||||
|
100% free. All backends are open-source tools (bird CLI, yt-dlp, Jina Reader, Exa, etc.) that don't require paid API keys. The only optional cost is a residential proxy (~$1/month) if you need Reddit/Bilibili access from a server.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 致谢
|
## 致谢
|
||||||
|
|
||||||
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://www.npmjs.com/package/@steipete/bird) · [Exa](https://exa.ai) · [mcporter](https://github.com/steipete/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) · [instaloader](https://github.com/instaloader/instaloader) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
|
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://www.npmjs.com/package/@steipete/bird) · [Exa](https://exa.ai) · [mcporter](https://github.com/steipete/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [xiaohongshu-mcp](https://github.com/xpzouying/xiaohongshu-mcp) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
@@ -1,87 +0,0 @@
|
|||||||
---
|
|
||||||
name: agent-reach
|
|
||||||
description: >
|
|
||||||
Give your AI agent eyes to see the entire internet. Read and search across
|
|
||||||
Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, RSS, and any web page
|
|
||||||
— all from a single CLI. Use when: (1) reading content from URLs (tweets, Reddit posts,
|
|
||||||
articles, videos), (2) searching across platforms (web, Twitter, Reddit, GitHub, YouTube,
|
|
||||||
Bilibili, XiaoHongShu), (3) checking channel health or updating Agent Reach.
|
|
||||||
Triggers: "search Twitter/Reddit/YouTube", "read this URL", "find posts about",
|
|
||||||
"搜索", "读取", "查一下", "看看这个链接".
|
|
||||||
---
|
|
||||||
|
|
||||||
# Agent Reach
|
|
||||||
|
|
||||||
Read and search the internet across 9+ platforms via unified CLI.
|
|
||||||
|
|
||||||
## Setup
|
|
||||||
|
|
||||||
First check if agent-reach is installed:
|
|
||||||
```bash
|
|
||||||
agent-reach doctor
|
|
||||||
```
|
|
||||||
|
|
||||||
If command not found, install it:
|
|
||||||
```bash
|
|
||||||
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
|
||||||
agent-reach install --env=auto
|
|
||||||
```
|
|
||||||
|
|
||||||
`install` auto-detects your environment and installs all dependencies (Node.js, mcporter, bird CLI, gh CLI). Read the output and run `agent-reach doctor` to see what's active.
|
|
||||||
|
|
||||||
For channels that need user input, ask the user. See the full setup guide:
|
|
||||||
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
|
||||||
|
|
||||||
## Commands
|
|
||||||
|
|
||||||
### Read any URL
|
|
||||||
```bash
|
|
||||||
agent-reach read <url>
|
|
||||||
agent-reach read <url> --json # structured output
|
|
||||||
```
|
|
||||||
Handles: tweets, Reddit posts, articles, YouTube (transcripts), GitHub repos, etc.
|
|
||||||
|
|
||||||
### Search
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach search "query" # web search (Exa)
|
|
||||||
agent-reach search-twitter "query" # Twitter/X
|
|
||||||
agent-reach search-reddit "query" # Reddit (--sub <subreddit>)
|
|
||||||
agent-reach search-github "query" # GitHub (--lang <language>)
|
|
||||||
agent-reach search-youtube "query" # YouTube
|
|
||||||
agent-reach search-bilibili "query" # Bilibili (B站)
|
|
||||||
agent-reach search-xhs "query" # XiaoHongShu (小红书)
|
|
||||||
```
|
|
||||||
|
|
||||||
All search commands support `-n <count>` for number of results.
|
|
||||||
|
|
||||||
### Management
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach doctor # channel status overview
|
|
||||||
agent-reach watch # quick health + update check (for scheduled tasks)
|
|
||||||
agent-reach check-update # check for new versions
|
|
||||||
```
|
|
||||||
|
|
||||||
### Configure channels
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
|
|
||||||
agent-reach configure proxy http://user:pass@ip:port
|
|
||||||
agent-reach configure --from-browser chrome # auto-extract cookies
|
|
||||||
```
|
|
||||||
|
|
||||||
## Channel Status Tiers
|
|
||||||
|
|
||||||
- **Tier 0 (zero config):** Web, YouTube, RSS, Twitter (read-only via Jina)
|
|
||||||
- **Tier 1 (free setup):** Exa web search (mcporter required)
|
|
||||||
- **Tier 2 (user config):** Twitter search (cookie), Reddit full (proxy), GitHub (token), Bilibili (proxy), XiaoHongShu (MCP)
|
|
||||||
|
|
||||||
Run `agent-reach doctor` to see which channels are active.
|
|
||||||
|
|
||||||
## Tips
|
|
||||||
|
|
||||||
- Always try `agent-reach read <url>` first for any URL — it auto-detects the platform
|
|
||||||
- For Twitter cookies, recommend the user install [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome extension
|
|
||||||
- Reddit and Bilibili block server IPs — suggest a residential proxy (~$1/month) if on a server
|
|
||||||
- If a channel breaks, run `agent-reach doctor` to diagnose
|
|
||||||
@@ -1,14 +1,10 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Channel registry — routes URLs to the right channel.
|
Channel registry — lists all supported platforms for doctor checks.
|
||||||
|
|
||||||
This is the core of Agent Reach' pluggable architecture.
|
|
||||||
Add a new channel: just create a file and register it here.
|
|
||||||
Swap a backend: just change the implementation inside the channel file.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Dict, List, Optional
|
from typing import List, Optional
|
||||||
from .base import Channel, ReadResult, SearchResult
|
from .base import Channel
|
||||||
|
|
||||||
# Import all channels
|
# Import all channels
|
||||||
from .web import WebChannel
|
from .web import WebChannel
|
||||||
@@ -20,12 +16,12 @@ from .rss import RSSChannel
|
|||||||
from .bilibili import BilibiliChannel
|
from .bilibili import BilibiliChannel
|
||||||
from .exa_search import ExaSearchChannel
|
from .exa_search import ExaSearchChannel
|
||||||
from .xiaohongshu import XiaoHongShuChannel
|
from .xiaohongshu import XiaoHongShuChannel
|
||||||
from .instagram import InstagramChannel
|
from .douyin import DouyinChannel
|
||||||
from .linkedin import LinkedInChannel
|
from .linkedin import LinkedInChannel
|
||||||
from .bosszhipin import BossZhipinChannel
|
from .bosszhipin import BossZhipinChannel
|
||||||
|
|
||||||
|
|
||||||
# Channel registry — order matters (first match wins, web is last as fallback)
|
# Channel registry
|
||||||
ALL_CHANNELS: List[Channel] = [
|
ALL_CHANNELS: List[Channel] = [
|
||||||
GitHubChannel(),
|
GitHubChannel(),
|
||||||
TwitterChannel(),
|
TwitterChannel(),
|
||||||
@@ -33,27 +29,14 @@ ALL_CHANNELS: List[Channel] = [
|
|||||||
RedditChannel(),
|
RedditChannel(),
|
||||||
BilibiliChannel(),
|
BilibiliChannel(),
|
||||||
XiaoHongShuChannel(),
|
XiaoHongShuChannel(),
|
||||||
InstagramChannel(),
|
DouyinChannel(),
|
||||||
LinkedInChannel(),
|
LinkedInChannel(),
|
||||||
BossZhipinChannel(),
|
BossZhipinChannel(),
|
||||||
RSSChannel(),
|
RSSChannel(),
|
||||||
ExaSearchChannel(),
|
ExaSearchChannel(),
|
||||||
WebChannel(), # Fallback — handles any URL
|
WebChannel(),
|
||||||
]
|
]
|
||||||
|
|
||||||
# Search-capable channels
|
|
||||||
SEARCH_CHANNELS: Dict[str, Channel] = {
|
|
||||||
ch.name: ch for ch in ALL_CHANNELS if ch.can_search()
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def get_channel_for_url(url: str) -> Channel:
|
|
||||||
"""Find the right channel for a URL."""
|
|
||||||
for channel in ALL_CHANNELS:
|
|
||||||
if channel.can_handle(url):
|
|
||||||
return channel
|
|
||||||
return WebChannel() # Should never reach here, but just in case
|
|
||||||
|
|
||||||
|
|
||||||
def get_channel(name: str) -> Optional[Channel]:
|
def get_channel(name: str) -> Optional[Channel]:
|
||||||
"""Get a channel by name."""
|
"""Get a channel by name."""
|
||||||
@@ -69,7 +52,7 @@ def get_all_channels() -> List[Channel]:
|
|||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"Channel", "ReadResult", "SearchResult",
|
"Channel",
|
||||||
"ALL_CHANNELS", "SEARCH_CHANNELS",
|
"ALL_CHANNELS",
|
||||||
"get_channel_for_url", "get_channel", "get_all_channels",
|
"get_channel", "get_all_channels",
|
||||||
]
|
]
|
||||||
|
|||||||
+11
-115
@@ -1,110 +1,28 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Channel base class — the universal interface for all platforms.
|
Channel base class — platform availability checking.
|
||||||
|
|
||||||
Every channel (YouTube, Twitter, GitHub, etc.) implements this interface.
|
Each channel represents a platform (YouTube, Twitter, GitHub, etc.)
|
||||||
The backend tool can be swapped anytime without changing anything else.
|
and provides:
|
||||||
|
- can_handle(url) → does this URL belong to this platform?
|
||||||
|
- check(config) → is the upstream tool installed and configured?
|
||||||
|
|
||||||
Example:
|
After installation, agents call upstream tools directly.
|
||||||
class YouTubeChannel(Channel):
|
|
||||||
name = "youtube"
|
|
||||||
backends = ["yt-dlp"] # current backend, can be swapped
|
|
||||||
|
|
||||||
async def read(self, url, config):
|
|
||||||
# Just call yt-dlp, return standardized dict
|
|
||||||
...
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
from abc import ABC, abstractmethod
|
from abc import ABC, abstractmethod
|
||||||
from dataclasses import dataclass
|
from typing import List, Tuple
|
||||||
from typing import Any, Dict, List, Optional, Tuple
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class ReadResult:
|
|
||||||
"""Standardized read result. Every channel returns this."""
|
|
||||||
title: str
|
|
||||||
content: str
|
|
||||||
url: str
|
|
||||||
author: str = ""
|
|
||||||
date: str = ""
|
|
||||||
platform: str = ""
|
|
||||||
extra: dict = None
|
|
||||||
|
|
||||||
def __post_init__(self):
|
|
||||||
self.extra = self.extra or {}
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
d = {
|
|
||||||
"title": self.title,
|
|
||||||
"content": self.content,
|
|
||||||
"url": self.url,
|
|
||||||
"platform": self.platform,
|
|
||||||
}
|
|
||||||
if self.author:
|
|
||||||
d["author"] = self.author
|
|
||||||
if self.date:
|
|
||||||
d["date"] = self.date
|
|
||||||
if self.extra:
|
|
||||||
d["extra"] = self.extra
|
|
||||||
return d
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class SearchResult:
|
|
||||||
"""Standardized search result."""
|
|
||||||
title: str
|
|
||||||
url: str
|
|
||||||
snippet: str = ""
|
|
||||||
author: str = ""
|
|
||||||
date: str = ""
|
|
||||||
score: float = 0
|
|
||||||
extra: dict = None
|
|
||||||
|
|
||||||
def __post_init__(self):
|
|
||||||
self.extra = self.extra or {}
|
|
||||||
|
|
||||||
def to_dict(self) -> dict:
|
|
||||||
d = {
|
|
||||||
"title": self.title,
|
|
||||||
"url": self.url,
|
|
||||||
"snippet": self.snippet,
|
|
||||||
}
|
|
||||||
if self.author:
|
|
||||||
d["author"] = self.author
|
|
||||||
if self.date:
|
|
||||||
d["date"] = self.date
|
|
||||||
if self.extra:
|
|
||||||
d["extra"] = self.extra
|
|
||||||
return d
|
|
||||||
|
|
||||||
|
|
||||||
class Channel(ABC):
|
class Channel(ABC):
|
||||||
"""
|
"""Base class for all channels."""
|
||||||
Base class for all channels.
|
|
||||||
|
|
||||||
Subclasses just need to implement:
|
|
||||||
- read(url, config) → ReadResult
|
|
||||||
- can_handle(url) → bool
|
|
||||||
- check(config) → (status, message)
|
|
||||||
|
|
||||||
Optionally:
|
|
||||||
- search(query, config, **kwargs) → list[SearchResult]
|
|
||||||
"""
|
|
||||||
|
|
||||||
name: str = "" # e.g. "youtube"
|
name: str = "" # e.g. "youtube"
|
||||||
description: str = "" # e.g. "YouTube video transcripts"
|
description: str = "" # e.g. "YouTube 视频和字幕"
|
||||||
backends: List[str] = [] # e.g. ["yt-dlp"] — what external tool is used
|
backends: List[str] = [] # e.g. ["yt-dlp"] — what upstream tool is used
|
||||||
requires_config: List[str] = [] # e.g. ["reddit_proxy"]
|
|
||||||
requires_tools: List[str] = [] # e.g. ["yt-dlp"]
|
|
||||||
tier: int = 0 # 0=zero-config, 1=needs free key, 2=needs setup
|
tier: int = 0 # 0=zero-config, 1=needs free key, 2=needs setup
|
||||||
|
|
||||||
@abstractmethod
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
"""Read content from a URL. Must return ReadResult."""
|
|
||||||
...
|
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
"""Check if this channel can handle this URL."""
|
"""Check if this channel can handle this URL."""
|
||||||
@@ -112,29 +30,7 @@ class Channel(ABC):
|
|||||||
|
|
||||||
def check(self, config=None) -> Tuple[str, str]:
|
def check(self, config=None) -> Tuple[str, str]:
|
||||||
"""
|
"""
|
||||||
Check if this channel is available.
|
Check if this channel's upstream tool is available.
|
||||||
Returns (status, message) where status is 'ok'/'warn'/'off'/'error'.
|
Returns (status, message) where status is 'ok'/'warn'/'off'/'error'.
|
||||||
"""
|
"""
|
||||||
# Check required tools
|
|
||||||
for tool in self.requires_tools:
|
|
||||||
if not shutil.which(tool):
|
|
||||||
return "off", f"需要安装:pip install {tool}"
|
|
||||||
|
|
||||||
# Check required config
|
|
||||||
for key in self.requires_config:
|
|
||||||
if config and not config.get(key):
|
|
||||||
return "off", f"需要配置 {key},运行 agent-reach setup"
|
|
||||||
|
|
||||||
return "ok", f"{'、'.join(self.backends) if self.backends else '内置'}"
|
return "ok", f"{'、'.join(self.backends) if self.backends else '内置'}"
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
"""Search this platform. Override if supported."""
|
|
||||||
raise NotImplementedError(f"{self.name} does not support search")
|
|
||||||
|
|
||||||
def can_search(self) -> bool:
|
|
||||||
"""Whether this channel supports search."""
|
|
||||||
try:
|
|
||||||
# Check if search is overridden
|
|
||||||
return type(self).search is not Channel.search
|
|
||||||
except:
|
|
||||||
return False
|
|
||||||
|
|||||||
@@ -1,207 +1,26 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Bilibili — via yt-dlp (same backend as YouTube).
|
"""Bilibili — check if yt-dlp is available."""
|
||||||
|
|
||||||
Backend: yt-dlp (https://github.com/yt-dlp/yt-dlp)
|
import os
|
||||||
yt-dlp natively supports Bilibili — video info, subtitles, and search.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
from .base import Channel
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class BilibiliChannel(Channel):
|
class BilibiliChannel(Channel):
|
||||||
name = "bilibili"
|
name = "bilibili"
|
||||||
description = "B站视频信息和字幕"
|
description = "B站视频和字幕"
|
||||||
backends = ["yt-dlp"]
|
backends = ["yt-dlp"]
|
||||||
requires_tools = ["yt-dlp"]
|
tier = 1
|
||||||
tier = 0
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
d = urlparse(url).netloc.lower()
|
d = urlparse(url).netloc.lower()
|
||||||
return "bilibili.com" in d or "b23.tv" in d
|
return "bilibili.com" in d or "b23.tv" in d
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if not shutil.which("yt-dlp"):
|
if not shutil.which("yt-dlp"):
|
||||||
return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
|
return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
|
||||||
proxy = config.get("bilibili_proxy") if config else None
|
proxy = (config.get("bilibili_proxy") if config else None) or os.environ.get("BILIBILI_PROXY")
|
||||||
if proxy:
|
if proxy:
|
||||||
return "ok", "已配置代理,完整可用"
|
return "ok", "可提取视频信息和字幕(代理已配置)"
|
||||||
import os
|
return "ok", "可提取视频信息和字幕(本地环境)。服务器可能需要代理"
|
||||||
is_server = bool(os.environ.get("SSH_CONNECTION") or os.path.exists("/etc/cloud"))
|
|
||||||
if is_server:
|
|
||||||
return "warn", "服务器 IP 可能被封,配置代理即可解决:agent-reach configure proxy URL"
|
|
||||||
return "ok", "本地直连可用"
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
if not shutil.which("yt-dlp"):
|
|
||||||
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
|
|
||||||
|
|
||||||
proxy = config.get("bilibili_proxy") if config else None
|
|
||||||
|
|
||||||
# Get video info via yt-dlp
|
|
||||||
info = self._get_info(url, proxy)
|
|
||||||
if not info:
|
|
||||||
return ReadResult(
|
|
||||||
title="Bilibili",
|
|
||||||
content=f"⚠️ 无法获取视频信息: {url}\n服务器 IP 可能被封,配个代理:agent-reach configure proxy URL",
|
|
||||||
url=url, platform="bilibili",
|
|
||||||
)
|
|
||||||
|
|
||||||
title = info.get("title", url)
|
|
||||||
author = info.get("uploader", "")
|
|
||||||
desc = info.get("description", "")
|
|
||||||
|
|
||||||
# Try subtitles
|
|
||||||
subtitle = self._get_subtitles(url, proxy)
|
|
||||||
content = desc
|
|
||||||
if subtitle:
|
|
||||||
content += f"\n\n## 字幕\n{subtitle}"
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=title, content=content, url=url,
|
|
||||||
author=author, platform="bilibili",
|
|
||||||
extra={
|
|
||||||
"view_count": info.get("view_count"),
|
|
||||||
"like_count": info.get("like_count"),
|
|
||||||
"duration": info.get("duration_string"),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
"""Search Bilibili.
|
|
||||||
|
|
||||||
Strategy:
|
|
||||||
1. Try yt-dlp bilisearch (works on local machines)
|
|
||||||
2. Fallback to Exa site:bilibili.com (works on servers)
|
|
||||||
"""
|
|
||||||
if not shutil.which("yt-dlp"):
|
|
||||||
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
|
|
||||||
|
|
||||||
limit = kwargs.get("limit", 5)
|
|
||||||
proxy = config.get("bilibili_proxy") if config else None
|
|
||||||
|
|
||||||
# Strategy 1: yt-dlp bilisearch
|
|
||||||
results = self._search_ytdlp(query, limit, proxy)
|
|
||||||
if results:
|
|
||||||
return results
|
|
||||||
|
|
||||||
# Strategy 2: Exa fallback (server-friendly)
|
|
||||||
results = self._search_exa(query, limit)
|
|
||||||
if results:
|
|
||||||
return results
|
|
||||||
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _search_ytdlp(self, query: str, limit: int, proxy: str = None) -> List[SearchResult]:
|
|
||||||
"""Search via yt-dlp bilisearch (needs local/Chinese IP)."""
|
|
||||||
cmd = [
|
|
||||||
"yt-dlp", "--dump-json", "--no-download",
|
|
||||||
f"bilisearch{limit}:{query}",
|
|
||||||
]
|
|
||||||
if proxy:
|
|
||||||
cmd += ["--proxy", proxy]
|
|
||||||
|
|
||||||
try:
|
|
||||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=60)
|
|
||||||
if r.returncode != 0:
|
|
||||||
return []
|
|
||||||
results = []
|
|
||||||
for line in r.stdout.strip().split("\n"):
|
|
||||||
if not line.strip():
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
vid = d.get("id", "")
|
|
||||||
url = d.get("webpage_url", f"https://www.bilibili.com/video/av{vid}")
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=d.get("title", f"av{vid}"),
|
|
||||||
url=url,
|
|
||||||
snippet=f"👤 {d.get('uploader', '?')} · 👁 {d.get('view_count', '?')}",
|
|
||||||
extra={
|
|
||||||
"view_count": d.get("view_count"),
|
|
||||||
"uploader": d.get("uploader"),
|
|
||||||
"duration": d.get("duration_string"),
|
|
||||||
},
|
|
||||||
))
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
continue
|
|
||||||
return results
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _search_exa(self, query: str, limit: int) -> List[SearchResult]:
|
|
||||||
"""Fallback: search via Exa (site:bilibili.com). Works on any IP."""
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call",
|
|
||||||
f'exa.web_search_exa(query: "site:bilibili.com {query}", numResults: {limit})'],
|
|
||||||
capture_output=True, text=True, timeout=30,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
return []
|
|
||||||
|
|
||||||
results = []
|
|
||||||
# Parse mcporter output: Title: / Author: / URL: / Text: blocks
|
|
||||||
title, author, url = "", "", ""
|
|
||||||
for line in r.stdout.split("\n"):
|
|
||||||
if line.startswith("Title: "):
|
|
||||||
title = line[7:].strip()
|
|
||||||
elif line.startswith("Author: "):
|
|
||||||
author = line[8:].strip()
|
|
||||||
elif line.startswith("URL: "):
|
|
||||||
url = line[5:].strip()
|
|
||||||
if url and "bilibili.com" in url:
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title or url,
|
|
||||||
url=url,
|
|
||||||
snippet=f"👤 {author}" if author else "(via Exa search)",
|
|
||||||
))
|
|
||||||
title, author, url = "", "", ""
|
|
||||||
return results
|
|
||||||
except Exception:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _get_info(self, url: str, proxy: str = None) -> dict:
|
|
||||||
cmd = ["yt-dlp", "--dump-json", "--no-download", url]
|
|
||||||
if proxy:
|
|
||||||
cmd += ["--proxy", proxy]
|
|
||||||
try:
|
|
||||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
|
||||||
if r.returncode == 0:
|
|
||||||
return json.loads(r.stdout)
|
|
||||||
except (subprocess.TimeoutExpired, json.JSONDecodeError):
|
|
||||||
pass
|
|
||||||
return {}
|
|
||||||
|
|
||||||
def _get_subtitles(self, url: str, proxy: str = None) -> str:
|
|
||||||
import tempfile
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
|
||||||
cmd = [
|
|
||||||
"yt-dlp", "--write-sub", "--write-auto-sub",
|
|
||||||
"--sub-lang", "zh-Hans,zh,en",
|
|
||||||
"--skip-download", "--sub-format", "vtt",
|
|
||||||
"-o", f"{tmpdir}/%(id)s.%(ext)s", url,
|
|
||||||
]
|
|
||||||
if proxy:
|
|
||||||
cmd += ["--proxy", proxy]
|
|
||||||
try:
|
|
||||||
subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
|
||||||
for f in Path(tmpdir).glob("*.vtt"):
|
|
||||||
text = f.read_text(errors="replace")
|
|
||||||
lines = []
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if not line or line.startswith("WEBVTT") or "-->" in line or line.isdigit():
|
|
||||||
continue
|
|
||||||
if line not in lines[-1:]:
|
|
||||||
lines.append(line)
|
|
||||||
return "\n".join(lines)
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
pass
|
|
||||||
return ""
|
|
||||||
|
|||||||
@@ -1,62 +1,9 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Boss直聘 (BOSS Zhipin) — via mcp-bosszp (MCP) or Jina Reader fallback.
|
"""Boss直聘 — check if mcp-bosszp is available."""
|
||||||
|
|
||||||
Backend: mcp-bosszp (161 stars, FastMCP + Playwright)
|
|
||||||
Swap to: any Boss直聘 access tool
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
import requests
|
|
||||||
|
|
||||||
|
|
||||||
def _mcporter_has_bosszhipin() -> bool:
|
|
||||||
"""Check if mcporter has Boss直聘 MCP configured."""
|
|
||||||
if not shutil.which("mcporter"):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
# Check for various possible config names
|
|
||||||
out = r.stdout.lower()
|
|
||||||
return "boss" in out or "zhipin" in out or "bosszhipin" in out
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _mcporter_call(expr: str, timeout: int = 30) -> str:
|
|
||||||
"""Call a Boss直聘 MCP tool via mcporter."""
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call", expr],
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
|
|
||||||
def _get_mcp_name() -> str:
|
|
||||||
"""Get the actual MCP server name configured in mcporter."""
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
for line in r.stdout.split("\n"):
|
|
||||||
line_lower = line.strip().lower()
|
|
||||||
for name in ["bosszhipin", "boss-zp", "bosszp", "boss"]:
|
|
||||||
if name in line_lower:
|
|
||||||
# Extract the actual server name
|
|
||||||
parts = line.strip().split()
|
|
||||||
if parts:
|
|
||||||
return parts[0]
|
|
||||||
return "bosszhipin"
|
|
||||||
except Exception:
|
|
||||||
return "bosszhipin"
|
|
||||||
|
|
||||||
|
|
||||||
class BossZhipinChannel(Channel):
|
class BossZhipinChannel(Channel):
|
||||||
@@ -66,118 +13,29 @@ class BossZhipinChannel(Channel):
|
|||||||
tier = 2
|
tier = 2
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
domain = urlparse(url).netloc.lower()
|
domain = urlparse(url).netloc.lower()
|
||||||
return "zhipin.com" in domain or "boss.com" in domain
|
return "zhipin.com" in domain or "boss.com" in domain
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if _mcporter_has_bosszhipin():
|
if not shutil.which("mcporter"):
|
||||||
return "ok", "可搜索职位、向 HR 打招呼"
|
|
||||||
|
|
||||||
return "off", (
|
return "off", (
|
||||||
"可通过 Jina Reader 读取职位页面。完整功能需要:\n"
|
"可通过 Jina Reader 读取职位页面。完整功能需要:\n"
|
||||||
" 1. git clone https://github.com/mucsbr/mcp-bosszp.git\n"
|
" 1. git clone https://github.com/mucsbr/mcp-bosszp.git\n"
|
||||||
" 2. cd mcp-bosszp && pip install -r requirements.txt && playwright install chromium\n"
|
" 2. cd mcp-bosszp && pip install -r requirements.txt && playwright install chromium\n"
|
||||||
" 3. python boss_zhipin_fastmcp_v2.py(启动后扫码登录)\n"
|
" 3. python boss_zhipin_fastmcp_v2.py(启动后扫码登录)\n"
|
||||||
" 4. mcporter config add bosszhipin http://localhost:8000/mcp\n"
|
" 4. mcporter config add bosszhipin http://localhost:8000/mcp"
|
||||||
" 或用 Docker:docker-compose up -d\n"
|
)
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
||||||
|
)
|
||||||
|
out = r.stdout.lower()
|
||||||
|
if "boss" in out or "zhipin" in out:
|
||||||
|
return "ok", "可搜索职位、向 HR 打招呼"
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return "off", (
|
||||||
|
"mcporter 已装但 Boss直聘 MCP 未配置。\n"
|
||||||
" 详见 https://github.com/mucsbr/mcp-bosszp"
|
" 详见 https://github.com/mucsbr/mcp-bosszp"
|
||||||
)
|
)
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
# Boss直聘 pages mostly work with Jina Reader
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
async def _read_jina(self, url: str) -> ReadResult:
|
|
||||||
"""Read Boss直聘 page via Jina Reader."""
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
f"https://r.jina.ai/{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
|
|
||||||
if len(text.strip()) < 50:
|
|
||||||
return ReadResult(
|
|
||||||
title="Boss直聘",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法读取此页面内容: {url}\n\n"
|
|
||||||
"提示:\n"
|
|
||||||
"- 安装 mcp-bosszp 可解锁职位搜索和自动打招呼\n"
|
|
||||||
"- 详见 https://github.com/mucsbr/mcp-bosszp"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="bosszhipin",
|
|
||||||
)
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=text[:100] if text else url,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="bosszhipin",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
return ReadResult(
|
|
||||||
title="Boss直聘",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法读取此 Boss直聘页面: {url}\n\n"
|
|
||||||
"提示:\n"
|
|
||||||
"- Boss直聘部分页面需要登录\n"
|
|
||||||
"- 安装 mcp-bosszp 可解锁完整功能\n"
|
|
||||||
"- 详见 https://github.com/mucsbr/mcp-bosszp"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="bosszhipin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
|
|
||||||
# Try MCP search first
|
|
||||||
if _mcporter_has_bosszhipin():
|
|
||||||
try:
|
|
||||||
return await self._search_mcp(query, limit, config)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Fallback to Exa
|
|
||||||
from agent_reach.channels.exa_search import ExaSearchChannel
|
|
||||||
exa = ExaSearchChannel()
|
|
||||||
return await exa.search(f"site:zhipin.com {query}", config=config, limit=limit)
|
|
||||||
|
|
||||||
async def _search_mcp(self, query: str, limit: int, config=None) -> List[SearchResult]:
|
|
||||||
"""Search Boss直聘 via MCP."""
|
|
||||||
server = _get_mcp_name()
|
|
||||||
try:
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'{server}.get_recommend_jobs_tool(page: 1)',
|
|
||||||
timeout=30,
|
|
||||||
)
|
|
||||||
return self._parse_jobs(out, limit)
|
|
||||||
except Exception:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_jobs(self, text: str, limit: int) -> List[SearchResult]:
|
|
||||||
"""Parse MCP job search output into SearchResults."""
|
|
||||||
results = []
|
|
||||||
try:
|
|
||||||
data = json.loads(text)
|
|
||||||
jobs = data if isinstance(data, list) else data.get("jobs", data.get("results", []))
|
|
||||||
for job in jobs[:limit]:
|
|
||||||
if isinstance(job, dict):
|
|
||||||
title = job.get("title") or job.get("jobName", "")
|
|
||||||
company = job.get("company") or job.get("brandName", "")
|
|
||||||
salary = job.get("salary") or job.get("salaryDesc", "")
|
|
||||||
url = job.get("url", "")
|
|
||||||
snippet = f"🏢 {company}" if company else ""
|
|
||||||
if salary:
|
|
||||||
snippet += f" · 💰 {salary}"
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title,
|
|
||||||
url=url,
|
|
||||||
snippet=snippet,
|
|
||||||
))
|
|
||||||
except (json.JSONDecodeError, KeyError):
|
|
||||||
pass
|
|
||||||
return results
|
|
||||||
|
|||||||
@@ -0,0 +1,52 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Douyin (抖音) — check if mcporter + douyin-mcp-server is available."""
|
||||||
|
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
from .base import Channel
|
||||||
|
|
||||||
|
|
||||||
|
class DouyinChannel(Channel):
|
||||||
|
name = "douyin"
|
||||||
|
description = "抖音短视频"
|
||||||
|
backends = ["douyin-mcp-server"]
|
||||||
|
tier = 2
|
||||||
|
|
||||||
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "douyin.com" in d or "iesdouyin.com" in d
|
||||||
|
|
||||||
|
def check(self, config=None):
|
||||||
|
if not shutil.which("mcporter"):
|
||||||
|
return "off", (
|
||||||
|
"需要 mcporter + douyin-mcp-server。安装步骤:\n"
|
||||||
|
" 1. npm install -g mcporter\n"
|
||||||
|
" 2. pip install douyin-mcp-server\n"
|
||||||
|
" 3. 启动服务(见下方说明)\n"
|
||||||
|
" 4. mcporter config add douyin http://localhost:18070/mcp\n"
|
||||||
|
" 详见 https://github.com/yzfly/douyin-mcp-server"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
||||||
|
)
|
||||||
|
if "douyin" not in r.stdout:
|
||||||
|
return "off", (
|
||||||
|
"mcporter 已装但抖音 MCP 未配置。运行:\n"
|
||||||
|
" pip install douyin-mcp-server\n"
|
||||||
|
" # 启动服务后:\n"
|
||||||
|
" mcporter config add douyin http://localhost:18070/mcp"
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return "off", "mcporter 连接异常"
|
||||||
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
["mcporter", "call", "douyin.parse_douyin_video_info(share_link: \"https://www.douyin.com\")"],
|
||||||
|
capture_output=True, text=True, timeout=15
|
||||||
|
)
|
||||||
|
if r.returncode == 0:
|
||||||
|
return "ok", "完整可用(视频解析、下载链接获取)"
|
||||||
|
return "warn", "MCP 已连接但调用异常,检查 douyin-mcp-server 服务是否在运行"
|
||||||
|
except Exception:
|
||||||
|
return "warn", "MCP 连接异常,检查 douyin-mcp-server 服务是否在运行"
|
||||||
@@ -1,110 +1,36 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Exa semantic search — via mcporter + Exa MCP server.
|
"""Exa Search — check if mcporter + Exa MCP is available."""
|
||||||
|
|
||||||
Backend: Exa MCP at mcp.exa.ai (OAuth, no API key needed)
|
|
||||||
Requires: mcporter CLI
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from .base import Channel, SearchResult
|
from .base import Channel
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class ExaSearchChannel(Channel):
|
class ExaSearchChannel(Channel):
|
||||||
name = "exa_search"
|
name = "exa_search"
|
||||||
description = "全网语义搜索(同时支持 Reddit/Twitter 搜索)"
|
description = "全网语义搜索"
|
||||||
backends = ["exa-mcp"]
|
backends = ["Exa via mcporter"]
|
||||||
tier = 1
|
tier = 0
|
||||||
|
|
||||||
def _mcporter_ok(self) -> bool:
|
|
||||||
if not shutil.which("mcporter"):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
return "exa" in r.stdout
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
def _call(self, expr: str, timeout: int = 30) -> str:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call", expr],
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
# ── Channel interface ──
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
return False # search-only
|
return False # Search-only channel
|
||||||
|
|
||||||
async def read(self, url: str, config=None):
|
|
||||||
raise NotImplementedError("Exa is a search engine, not a reader")
|
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if not shutil.which("mcporter"):
|
if not shutil.which("mcporter"):
|
||||||
return "off", (
|
return "off", (
|
||||||
"需要 mcporter。安装:npm install -g mcporter && "
|
"需要 mcporter + Exa MCP。安装:\n"
|
||||||
"mcporter config add exa https://mcp.exa.ai/mcp"
|
|
||||||
)
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
return "off", "mcporter 已装但 Exa 未配置。运行:mcporter config add exa https://mcp.exa.ai/mcp"
|
|
||||||
return "ok", "MCP 已连接,免 Key 直接可用(全网搜索 + Reddit + Twitter)"
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
raise ValueError(
|
|
||||||
"Exa 搜索需要 mcporter。安装:\n"
|
|
||||||
" npm install -g mcporter\n"
|
" npm install -g mcporter\n"
|
||||||
" mcporter config add exa https://mcp.exa.ai/mcp"
|
" mcporter config add exa https://mcp.exa.ai/mcp"
|
||||||
)
|
)
|
||||||
|
try:
|
||||||
limit = kwargs.get("limit", 5)
|
r = subprocess.run(
|
||||||
safe_q = query.replace('"', '\\"')
|
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
||||||
out = self._call(
|
|
||||||
f'exa.web_search_exa(query: "{safe_q}", numResults: {min(limit, 10)})',
|
|
||||||
timeout=30,
|
|
||||||
)
|
)
|
||||||
return self._parse_output(out, limit)
|
if "exa" in r.stdout.lower():
|
||||||
|
return "ok", "全网语义搜索可用(免费,无需 API Key)"
|
||||||
# ── Parse mcporter text output ──
|
return "off", (
|
||||||
|
"mcporter 已装但 Exa 未配置。运行:\n"
|
||||||
def _parse_output(self, text: str, limit: int) -> List[SearchResult]:
|
" mcporter config add exa https://mcp.exa.ai/mcp"
|
||||||
"""Parse mcporter's Title/URL/Text block format."""
|
|
||||||
results = []
|
|
||||||
cur = {}
|
|
||||||
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line.startswith("Title: "):
|
|
||||||
if cur.get("title"):
|
|
||||||
results.append(self._make_result(cur))
|
|
||||||
cur = {"title": line[7:]}
|
|
||||||
elif line.startswith("URL: "):
|
|
||||||
cur["url"] = line[5:]
|
|
||||||
elif line.startswith("Published Date: "):
|
|
||||||
cur["date"] = line[16:]
|
|
||||||
elif line.startswith("Text: "):
|
|
||||||
cur["text"] = line[6:]
|
|
||||||
elif "text" in cur and line:
|
|
||||||
cur["text"] += " " + line
|
|
||||||
|
|
||||||
if cur.get("title"):
|
|
||||||
results.append(self._make_result(cur))
|
|
||||||
|
|
||||||
return results[:limit]
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _make_result(d: dict) -> SearchResult:
|
|
||||||
return SearchResult(
|
|
||||||
title=d.get("title", ""),
|
|
||||||
url=d.get("url", ""),
|
|
||||||
snippet=d.get("text", "")[:500],
|
|
||||||
date=d.get("date", ""),
|
|
||||||
score=0,
|
|
||||||
)
|
)
|
||||||
|
except Exception:
|
||||||
|
return "off", "mcporter 连接异常"
|
||||||
|
|||||||
@@ -1,16 +1,9 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""GitHub — via gh CLI.
|
"""GitHub — check if gh CLI is available."""
|
||||||
|
|
||||||
Backend: gh CLI (https://cli.github.com)
|
|
||||||
Swap to: GitHub REST API
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class GitHubChannel(Channel):
|
class GitHubChannel(Channel):
|
||||||
@@ -19,121 +12,18 @@ class GitHubChannel(Channel):
|
|||||||
backends = ["gh CLI"]
|
backends = ["gh CLI"]
|
||||||
tier = 0
|
tier = 0
|
||||||
|
|
||||||
def _gh(self, args: list, timeout: int = 15) -> str:
|
|
||||||
r = subprocess.run(
|
|
||||||
["gh"] + args,
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
def _gh_json(self, args: list, timeout: int = 15) -> dict:
|
|
||||||
return json.loads(self._gh(args + ["--json"], timeout))
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
return "github.com" in urlparse(url).netloc.lower()
|
return "github.com" in urlparse(url).netloc.lower()
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if not shutil.which("gh"):
|
if not shutil.which("gh"):
|
||||||
return "warn", "gh CLI 未安装。安装:https://cli.github.com 。公开仓库仍可通过 Jina Reader 读取"
|
return "warn", "gh CLI 未安装。安装:https://cli.github.com"
|
||||||
try:
|
try:
|
||||||
self._gh(["auth", "status"], timeout=5)
|
subprocess.run(
|
||||||
|
["gh", "auth", "status"],
|
||||||
|
capture_output=True, text=True, timeout=5
|
||||||
|
)
|
||||||
return "ok", "完整可用(读取、搜索、Fork、Issue、PR 等)"
|
return "ok", "完整可用(读取、搜索、Fork、Issue、PR 等)"
|
||||||
except Exception:
|
except Exception:
|
||||||
return "ok", "gh CLI 已装但未认证。运行 gh auth login 可解锁完整功能"
|
return "ok", "gh CLI 已装但未认证。运行 gh auth login 可解锁完整功能"
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
if not shutil.which("gh"):
|
|
||||||
# Fallback to Jina Reader for public repos
|
|
||||||
from agent_reach.channels.web import WebChannel
|
|
||||||
return await WebChannel().read(url, config)
|
|
||||||
|
|
||||||
path = urlparse(url).path.strip("/").split("/")
|
|
||||||
if len(path) < 2:
|
|
||||||
from agent_reach.channels.web import WebChannel
|
|
||||||
return await WebChannel().read(url, config)
|
|
||||||
|
|
||||||
owner, repo = path[0], path[1]
|
|
||||||
|
|
||||||
# Issues / PRs
|
|
||||||
if len(path) >= 4 and path[2] in ("issues", "pull"):
|
|
||||||
return await self._read_issue(owner, repo, path[3], url)
|
|
||||||
|
|
||||||
# Repo
|
|
||||||
return await self._read_repo(owner, repo, url)
|
|
||||||
|
|
||||||
async def _read_repo(self, owner: str, repo: str, url: str) -> ReadResult:
|
|
||||||
slug = f"{owner}/{repo}"
|
|
||||||
try:
|
|
||||||
# Get repo info
|
|
||||||
info = self._gh(["repo", "view", slug])
|
|
||||||
# Get README
|
|
||||||
try:
|
|
||||||
readme = self._gh(
|
|
||||||
["api", f"repos/{slug}/readme", "--jq", ".content"],
|
|
||||||
timeout=10,
|
|
||||||
)
|
|
||||||
import base64
|
|
||||||
readme_text = base64.b64decode(readme).decode("utf-8", errors="replace")
|
|
||||||
except Exception:
|
|
||||||
readme_text = ""
|
|
||||||
|
|
||||||
content = readme_text or info
|
|
||||||
return ReadResult(
|
|
||||||
title=slug, content=content, url=url,
|
|
||||||
author=owner, platform="github",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
from agent_reach.channels.web import WebChannel
|
|
||||||
return await WebChannel().read(url)
|
|
||||||
|
|
||||||
async def _read_issue(self, owner: str, repo: str, num: str, url: str) -> ReadResult:
|
|
||||||
slug = f"{owner}/{repo}"
|
|
||||||
try:
|
|
||||||
out = self._gh(["issue", "view", num, "-R", slug])
|
|
||||||
return ReadResult(
|
|
||||||
title=f"{slug}#{num}", content=out, url=url,
|
|
||||||
platform="github",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
# Might be a PR
|
|
||||||
try:
|
|
||||||
out = self._gh(["pr", "view", num, "-R", slug])
|
|
||||||
return ReadResult(
|
|
||||||
title=f"{slug}#{num}", content=out, url=url,
|
|
||||||
platform="github",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
from agent_reach.channels.web import WebChannel
|
|
||||||
return await WebChannel().read(url)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
if not shutil.which("gh"):
|
|
||||||
raise ValueError("GitHub search requires gh CLI. Install: https://cli.github.com")
|
|
||||||
|
|
||||||
language = kwargs.get("language")
|
|
||||||
limit = kwargs.get("limit", 5)
|
|
||||||
|
|
||||||
args = ["search", "repos", query, "--sort", "stars", f"--limit={limit}"]
|
|
||||||
if language:
|
|
||||||
args += [f"--language={language}"]
|
|
||||||
|
|
||||||
out = self._gh(args, timeout=15)
|
|
||||||
results = []
|
|
||||||
for line in out.strip().split("\n"):
|
|
||||||
if not line.strip():
|
|
||||||
continue
|
|
||||||
parts = line.split("\t")
|
|
||||||
if len(parts) >= 1:
|
|
||||||
slug = parts[0].strip()
|
|
||||||
desc = parts[1].strip() if len(parts) > 1 else ""
|
|
||||||
stars = parts[3].strip() if len(parts) > 3 else ""
|
|
||||||
lang = parts[5].strip() if len(parts) > 5 else ""
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=slug,
|
|
||||||
url=f"https://github.com/{slug}",
|
|
||||||
snippet=desc,
|
|
||||||
extra={"stars": stars, "language": lang},
|
|
||||||
))
|
|
||||||
return results
|
|
||||||
|
|||||||
@@ -1,248 +0,0 @@
|
|||||||
# -*- coding: utf-8 -*-
|
|
||||||
"""Instagram — via instaloader (free, open source).
|
|
||||||
|
|
||||||
Backend: instaloader (9.8K stars, Python CLI + library)
|
|
||||||
Swap to: any Instagram access tool
|
|
||||||
"""
|
|
||||||
|
|
||||||
import re
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class InstagramChannel(Channel):
|
|
||||||
name = "instagram"
|
|
||||||
description = "Instagram 帖子和 Profile"
|
|
||||||
backends = ["instaloader"]
|
|
||||||
tier = 2 # Needs login for full access
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
|
||||||
domain = urlparse(url).netloc.lower()
|
|
||||||
return "instagram.com" in domain or "instagr.am" in domain
|
|
||||||
|
|
||||||
def check(self, config=None):
|
|
||||||
# Check both CLI and Python module
|
|
||||||
has_cli = shutil.which("instaloader")
|
|
||||||
has_module = False
|
|
||||||
try:
|
|
||||||
import instaloader
|
|
||||||
has_module = True
|
|
||||||
except ImportError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
if not has_cli and not has_module:
|
|
||||||
return "off", (
|
|
||||||
"需要安装 instaloader:pip install instaloader\n"
|
|
||||||
" 安装后可读取 Instagram 帖子和 Profile\n"
|
|
||||||
" 登录: agent-reach configure instagram-cookies \"sessionid=xxx; csrftoken=yyy; ...\""
|
|
||||||
)
|
|
||||||
|
|
||||||
# Check if cookies are configured
|
|
||||||
cookie_file = Path.home() / ".agent-reach" / "instagram-cookies.txt"
|
|
||||||
if cookie_file.exists():
|
|
||||||
return "ok", "已登录,可读取 Instagram 帖子和 Profile"
|
|
||||||
return "ok", "可读取公开帖子和 Profile。登录可访问更多内容:\n agent-reach configure instagram-cookies \"sessionid=xxx; csrftoken=yyy; ...\""
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
# Try instaloader (module or CLI)
|
|
||||||
try:
|
|
||||||
import instaloader
|
|
||||||
return await self._read_instaloader(url, config)
|
|
||||||
except ImportError:
|
|
||||||
pass
|
|
||||||
# Fallback: Jina Reader
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
async def _read_instaloader(self, url: str, config=None) -> ReadResult:
|
|
||||||
"""Read Instagram content using instaloader Python API."""
|
|
||||||
import asyncio
|
|
||||||
import concurrent.futures
|
|
||||||
|
|
||||||
def _sync_read():
|
|
||||||
import instaloader
|
|
||||||
L = instaloader.Instaloader(
|
|
||||||
download_pictures=False,
|
|
||||||
download_videos=False,
|
|
||||||
download_video_thumbnails=False,
|
|
||||||
download_geotags=False,
|
|
||||||
download_comments=False,
|
|
||||||
save_metadata=False,
|
|
||||||
compress_json=False,
|
|
||||||
max_connection_attempts=1, # Don't retry on rate limit
|
|
||||||
)
|
|
||||||
|
|
||||||
# Try to load session: cookie file > saved session
|
|
||||||
cookie_file = Path.home() / ".agent-reach" / "instagram-cookies.txt"
|
|
||||||
if cookie_file.exists():
|
|
||||||
try:
|
|
||||||
cookie_str = cookie_file.read_text().strip()
|
|
||||||
cookies = {}
|
|
||||||
for part in cookie_str.split(";"):
|
|
||||||
part = part.strip()
|
|
||||||
if "=" in part:
|
|
||||||
k, v = part.split("=", 1)
|
|
||||||
cookies[k.strip()] = v.strip()
|
|
||||||
if "sessionid" in cookies and "csrftoken" in cookies:
|
|
||||||
# Extract username from ds_user_id or use generic
|
|
||||||
username = cookies.get("ds_user_id", "user")
|
|
||||||
L.context.load_session(username, cookies)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
elif config and config.get("instagram_username"):
|
|
||||||
try:
|
|
||||||
L.load_session_from_file(config.get("instagram_username"))
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
path = urlparse(url).path.strip("/")
|
|
||||||
|
|
||||||
if "/p/" in url or "/reel/" in url:
|
|
||||||
return self._read_post_sync(L, url, path)
|
|
||||||
else:
|
|
||||||
return self._read_profile_sync(L, url, path)
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Run with 15s timeout to avoid instaloader's 30-min retry
|
|
||||||
loop = asyncio.get_event_loop()
|
|
||||||
with concurrent.futures.ThreadPoolExecutor() as pool:
|
|
||||||
result = await asyncio.wait_for(
|
|
||||||
loop.run_in_executor(pool, _sync_read),
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
return result
|
|
||||||
except (asyncio.TimeoutError, Exception):
|
|
||||||
# Any error or timeout → Jina fallback
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
def _read_post_sync(self, L, url: str, path: str) -> ReadResult:
|
|
||||||
"""Read a single Instagram post (sync, runs in executor)."""
|
|
||||||
import instaloader
|
|
||||||
|
|
||||||
# Extract shortcode from URL
|
|
||||||
match = re.search(r"/(?:p|reel)/([A-Za-z0-9_-]+)", url)
|
|
||||||
if not match:
|
|
||||||
raise ValueError("Cannot extract shortcode from URL")
|
|
||||||
|
|
||||||
shortcode = match.group(1)
|
|
||||||
try:
|
|
||||||
post = instaloader.Post.from_shortcode(L.context, shortcode)
|
|
||||||
|
|
||||||
lines = []
|
|
||||||
if post.caption:
|
|
||||||
lines.append(post.caption)
|
|
||||||
lines.append("")
|
|
||||||
lines.append(f"👤 @{post.owner_username}")
|
|
||||||
lines.append(f"❤️ {post.likes} likes")
|
|
||||||
if post.comments:
|
|
||||||
lines.append(f"💬 {post.comments} comments")
|
|
||||||
lines.append(f"📅 {post.date_utc.strftime('%Y-%m-%d %H:%M')}")
|
|
||||||
if post.location:
|
|
||||||
lines.append(f"📍 {post.location}")
|
|
||||||
if post.hashtags:
|
|
||||||
lines.append(f"#️⃣ {' '.join('#' + h for h in post.hashtags)}")
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=f"@{post.owner_username}: {(post.caption or '')[:80]}",
|
|
||||||
content="\n".join(lines),
|
|
||||||
url=url,
|
|
||||||
author=f"@{post.owner_username}",
|
|
||||||
date=post.date_utc.strftime("%Y-%m-%d"),
|
|
||||||
platform="instagram",
|
|
||||||
extra={"likes": post.likes, "comments": post.comments},
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
raise # Let executor timeout handle fallback
|
|
||||||
|
|
||||||
def _read_profile_sync(self, L, url: str, path: str) -> ReadResult:
|
|
||||||
"""Read an Instagram profile (sync, runs in executor)."""
|
|
||||||
import instaloader
|
|
||||||
|
|
||||||
# Extract username from path
|
|
||||||
username = path.split("/")[0] if path else ""
|
|
||||||
if not username or username in ("p", "reel", "stories", "explore"):
|
|
||||||
raise ValueError("Cannot extract username from URL")
|
|
||||||
|
|
||||||
try:
|
|
||||||
profile = instaloader.Profile.from_username(L.context, username)
|
|
||||||
|
|
||||||
lines = []
|
|
||||||
lines.append(f"👤 {profile.full_name} (@{profile.username})")
|
|
||||||
if profile.biography:
|
|
||||||
lines.append(f"📝 {profile.biography}")
|
|
||||||
if profile.external_url:
|
|
||||||
lines.append(f"🔗 {profile.external_url}")
|
|
||||||
lines.append("")
|
|
||||||
lines.append(f"📊 {profile.mediacount} posts · "
|
|
||||||
f"{profile.followers} followers · "
|
|
||||||
f"{profile.followees} following")
|
|
||||||
if profile.is_verified:
|
|
||||||
lines.append("✅ Verified")
|
|
||||||
if profile.is_business_account and profile.business_category_name:
|
|
||||||
lines.append(f"🏢 {profile.business_category_name}")
|
|
||||||
|
|
||||||
# Get recent posts (up to 5)
|
|
||||||
lines.append("")
|
|
||||||
lines.append("📸 Recent posts:")
|
|
||||||
count = 0
|
|
||||||
for post in profile.get_posts():
|
|
||||||
if count >= 5:
|
|
||||||
break
|
|
||||||
caption = (post.caption or "")[:100].replace("\n", " ")
|
|
||||||
lines.append(f" • ❤️{post.likes} | {post.date_utc.strftime('%m-%d')} | {caption}")
|
|
||||||
count += 1
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=f"{profile.full_name} (@{profile.username}) - Instagram",
|
|
||||||
content="\n".join(lines),
|
|
||||||
url=url,
|
|
||||||
author=f"@{profile.username}",
|
|
||||||
platform="instagram",
|
|
||||||
extra={
|
|
||||||
"followers": profile.followers,
|
|
||||||
"posts": profile.mediacount,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
raise # Let executor timeout handle fallback
|
|
||||||
|
|
||||||
async def _read_jina(self, url: str) -> ReadResult:
|
|
||||||
"""Fallback: use Jina Reader."""
|
|
||||||
import requests
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
f"https://r.jina.ai/{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
return ReadResult(
|
|
||||||
title=text[:100] if text else url,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="instagram",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
return ReadResult(
|
|
||||||
title="Instagram",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法读取此 Instagram 内容: {url}\n\n"
|
|
||||||
"提示:\n"
|
|
||||||
"- 确保 URL 正确\n"
|
|
||||||
"- 安装 instaloader: pip install instaloader\n"
|
|
||||||
"- 登录以访问更多内容: instaloader --login YOUR_USERNAME"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="instagram",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
"""Search Instagram via Exa."""
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
from agent_reach.channels.exa_search import ExaSearchChannel
|
|
||||||
exa = ExaSearchChannel()
|
|
||||||
return await exa.search(f"site:instagram.com {query}", config=config, limit=limit)
|
|
||||||
@@ -1,268 +1,39 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""LinkedIn — via linkedin-scraper-mcp (MCP) or Jina Reader fallback.
|
"""LinkedIn — check if linkedin-scraper-mcp is available."""
|
||||||
|
|
||||||
Backend: linkedin-scraper-mcp (916 stars, Patchright browser automation)
|
|
||||||
Swap to: any LinkedIn access tool
|
|
||||||
"""
|
|
||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
import requests
|
|
||||||
|
|
||||||
|
|
||||||
def _mcporter_has_linkedin() -> bool:
|
|
||||||
"""Check if mcporter has linkedin MCP configured."""
|
|
||||||
if not shutil.which("mcporter"):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
return "linkedin" in r.stdout.lower()
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _mcporter_call(expr: str, timeout: int = 30) -> str:
|
|
||||||
"""Call a LinkedIn MCP tool via mcporter."""
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call", expr],
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
|
|
||||||
class LinkedInChannel(Channel):
|
class LinkedInChannel(Channel):
|
||||||
name = "linkedin"
|
name = "linkedin"
|
||||||
description = "LinkedIn 个人/公司 Profile 和职位"
|
description = "LinkedIn 职业社交"
|
||||||
backends = ["linkedin-scraper-mcp", "Jina Reader"]
|
backends = ["linkedin-scraper-mcp", "Jina Reader"]
|
||||||
tier = 2
|
tier = 2
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
domain = urlparse(url).netloc.lower()
|
from urllib.parse import urlparse
|
||||||
return "linkedin.com" in domain
|
return "linkedin.com" in urlparse(url).netloc.lower()
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
if _mcporter_has_linkedin():
|
if not shutil.which("mcporter"):
|
||||||
return "ok", "完整可用(Profile、公司、职位搜索)"
|
|
||||||
|
|
||||||
# Check if linkedin-scraper-mcp is installed as CLI
|
|
||||||
if shutil.which("linkedin-scraper-mcp"):
|
|
||||||
return "warn", (
|
|
||||||
"linkedin-scraper-mcp 已安装但未接入 mcporter。运行:\n"
|
|
||||||
" 1. linkedin-scraper-mcp --login(在有浏览器的机器上登录)\n"
|
|
||||||
" 2. linkedin-scraper-mcp --transport streamable-http --port 8001\n"
|
|
||||||
" 3. mcporter config add linkedin http://localhost:8001/mcp"
|
|
||||||
)
|
|
||||||
|
|
||||||
return "off", (
|
return "off", (
|
||||||
"可通过 Jina Reader 读取部分内容。完整功能需要:\n"
|
"基本内容可通过 Jina Reader 读取。完整功能需要:\n"
|
||||||
" 1. pip install linkedin-scraper-mcp\n"
|
|
||||||
" 2. linkedin-scraper-mcp --login(在有浏览器的机器上登录)\n"
|
|
||||||
" 3. linkedin-scraper-mcp --transport streamable-http --port 8001\n"
|
|
||||||
" 4. mcporter config add linkedin http://localhost:8001/mcp\n"
|
|
||||||
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
|
|
||||||
)
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
path = urlparse(url).path.strip("/")
|
|
||||||
|
|
||||||
# Try MCP first
|
|
||||||
if _mcporter_has_linkedin():
|
|
||||||
try:
|
|
||||||
if "/in/" in url:
|
|
||||||
return await self._read_profile_mcp(url)
|
|
||||||
elif "/company/" in url:
|
|
||||||
return await self._read_company_mcp(url)
|
|
||||||
elif "/jobs/view/" in url:
|
|
||||||
return await self._read_job_mcp(url)
|
|
||||||
except Exception:
|
|
||||||
pass # Fall through to Jina
|
|
||||||
|
|
||||||
# Fallback: Jina Reader
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
async def _read_profile_mcp(self, url: str) -> ReadResult:
|
|
||||||
"""Read a LinkedIn profile via MCP."""
|
|
||||||
import re
|
|
||||||
# Extract username from URL: /in/username/
|
|
||||||
match = re.search(r"/in/([^/]+)", url)
|
|
||||||
if not match:
|
|
||||||
return await self._read_jina(url)
|
|
||||||
username = match.group(1)
|
|
||||||
safe_username = username.replace('"', '\\"')
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.get_person_profile(linkedin_username: "{safe_username}")',
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
return ReadResult(
|
|
||||||
title=self._extract_title(out) or f"LinkedIn Profile - {username}",
|
|
||||||
content=out.strip(),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_company_mcp(self, url: str) -> ReadResult:
|
|
||||||
"""Read a LinkedIn company page via MCP."""
|
|
||||||
import re
|
|
||||||
# Extract company name from URL: /company/name/
|
|
||||||
match = re.search(r"/company/([^/]+)", url)
|
|
||||||
if not match:
|
|
||||||
return await self._read_jina(url)
|
|
||||||
company = match.group(1)
|
|
||||||
safe_company = company.replace('"', '\\"')
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.get_company_profile(company_name: "{safe_company}")',
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
return ReadResult(
|
|
||||||
title=self._extract_title(out) or "LinkedIn Company",
|
|
||||||
content=out.strip(),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_job_mcp(self, url: str) -> ReadResult:
|
|
||||||
"""Read a LinkedIn job posting via MCP."""
|
|
||||||
import re
|
|
||||||
match = re.search(r"/jobs/view/(\d+)", url)
|
|
||||||
if not match:
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
job_id = match.group(1)
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.get_job_details(job_id: "{job_id}")',
|
|
||||||
timeout=30,
|
|
||||||
)
|
|
||||||
return ReadResult(
|
|
||||||
title=self._extract_title(out) or f"LinkedIn Job {job_id}",
|
|
||||||
content=out.strip(),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_jina(self, url: str) -> ReadResult:
|
|
||||||
"""Fallback: use Jina Reader."""
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
f"https://r.jina.ai/{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
|
|
||||||
# Check if content is usable
|
|
||||||
if len(text.strip()) < 100 or "Sign in" in text[:200]:
|
|
||||||
return ReadResult(
|
|
||||||
title="LinkedIn",
|
|
||||||
content=(
|
|
||||||
f"⚠️ LinkedIn 页面需要登录才能完整查看。\n\n"
|
|
||||||
f"URL: {url}\n\n"
|
|
||||||
"完整功能需安装 linkedin-scraper-mcp:\n"
|
|
||||||
" pip install linkedin-scraper-mcp\n"
|
" pip install linkedin-scraper-mcp\n"
|
||||||
" uvx linkedin-scraper-mcp --login\n"
|
" mcporter config add linkedin http://localhost:3000/mcp\n"
|
||||||
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
|
" 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
)
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=text[:100] if text else url,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
return ReadResult(
|
|
||||||
title="LinkedIn",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法读取此 LinkedIn 页面: {url}\n\n"
|
|
||||||
"提示:\n"
|
|
||||||
"- LinkedIn 需要登录才能查看大部分内容\n"
|
|
||||||
"- 安装 linkedin-scraper-mcp 解锁完整功能\n"
|
|
||||||
"- 详见 https://github.com/stickerdaniel/linkedin-mcp-server"
|
|
||||||
),
|
|
||||||
url=url,
|
|
||||||
platform="linkedin",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
|
|
||||||
# Try MCP search first
|
|
||||||
if _mcporter_has_linkedin():
|
|
||||||
try:
|
try:
|
||||||
return await self._search_mcp(query, limit)
|
r = subprocess.run(
|
||||||
|
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
||||||
|
)
|
||||||
|
if "linkedin" in r.stdout.lower():
|
||||||
|
return "ok", "完整可用(Profile、公司、职位搜索)"
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
return "off", (
|
||||||
# Fallback to Exa
|
"mcporter 已装但 LinkedIn MCP 未配置。运行:\n"
|
||||||
from agent_reach.channels.exa_search import ExaSearchChannel
|
" pip install linkedin-scraper-mcp\n"
|
||||||
exa = ExaSearchChannel()
|
" mcporter config add linkedin http://localhost:3000/mcp"
|
||||||
return await exa.search(f"site:linkedin.com {query}", config=config, limit=limit)
|
|
||||||
|
|
||||||
async def _search_mcp(self, query: str, limit: int) -> List[SearchResult]:
|
|
||||||
"""Search LinkedIn via MCP."""
|
|
||||||
safe_q = query.replace('"', '\\"')
|
|
||||||
# Try job search first (most common use case)
|
|
||||||
try:
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.search_jobs(keywords: "{safe_q}")',
|
|
||||||
timeout=60,
|
|
||||||
)
|
)
|
||||||
results = self._parse_search_results(out, "job")
|
|
||||||
if results:
|
|
||||||
return results[:limit]
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Try people search
|
|
||||||
try:
|
|
||||||
out = _mcporter_call(
|
|
||||||
f'linkedin.search_people(keywords: "{safe_q}")',
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
results = self._parse_search_results(out, "people")
|
|
||||||
if results:
|
|
||||||
return results
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_search_results(self, text: str, result_type: str) -> List[SearchResult]:
|
|
||||||
"""Parse MCP search output into SearchResults."""
|
|
||||||
import json
|
|
||||||
results = []
|
|
||||||
try:
|
|
||||||
data = json.loads(text)
|
|
||||||
items = data if isinstance(data, list) else data.get("results", data.get("jobs", []))
|
|
||||||
for item in items:
|
|
||||||
if isinstance(item, dict):
|
|
||||||
title = item.get("title") or item.get("name") or item.get("headline", "")
|
|
||||||
url = item.get("url") or item.get("link", "")
|
|
||||||
snippet = item.get("description") or item.get("company", "")
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title,
|
|
||||||
url=url,
|
|
||||||
snippet=snippet[:200] if snippet else "",
|
|
||||||
))
|
|
||||||
except (json.JSONDecodeError, KeyError):
|
|
||||||
# Try line-by-line parsing
|
|
||||||
pass
|
|
||||||
return results
|
|
||||||
|
|
||||||
def _extract_title(self, text: str) -> str:
|
|
||||||
"""Extract a title from MCP output."""
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line and not line.startswith(("{", "[", "#", "http")):
|
|
||||||
return line[:80]
|
|
||||||
return ""
|
|
||||||
|
|||||||
+13
-165
@@ -1,178 +1,26 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Reddit — via Reddit JSON API + optional proxy.
|
"""Reddit — check if proxy and credentials are configured."""
|
||||||
|
|
||||||
Backend: Reddit public JSON API (append .json to any URL)
|
|
||||||
Swap to: any Reddit access method
|
|
||||||
"""
|
|
||||||
|
|
||||||
import os
|
import os
|
||||||
import requests
|
from .base import Channel
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult
|
|
||||||
|
|
||||||
|
|
||||||
class RedditChannel(Channel):
|
class RedditChannel(Channel):
|
||||||
name = "reddit"
|
name = "reddit"
|
||||||
description = "Reddit 帖子和评论"
|
description = "Reddit 帖子和评论"
|
||||||
backends = ["Reddit JSON API"]
|
backends = ["JSON API", "Exa"]
|
||||||
tier = 2
|
tier = 1
|
||||||
|
|
||||||
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
domain = urlparse(url).netloc.lower()
|
from urllib.parse import urlparse
|
||||||
return "reddit.com" in domain or "redd.it" in domain
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "reddit.com" in d or "redd.it" in d
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
proxy = config.get("reddit_proxy") if config else None
|
proxy = (config.get("reddit_proxy") if config else None) or os.environ.get("REDDIT_PROXY")
|
||||||
has_bot = bool(os.environ.get("REDDIT_CLIENT_ID"))
|
if proxy:
|
||||||
if proxy and has_bot:
|
return "ok", "代理已配置,可读取帖子。搜索走 Exa"
|
||||||
return "ok", "完整可用(代理 + OAuth Bot)"
|
return "warn", (
|
||||||
elif proxy:
|
"无代理。服务器 IP 可能被 Reddit 封锁。配置代理:\n"
|
||||||
return "ok", "代理已配置,可读取帖子。配置 REDDIT_CLIENT_ID/SECRET 可解锁高级搜索和发帖"
|
" agent-reach configure proxy http://user:pass@ip:port"
|
||||||
elif has_bot:
|
|
||||||
return "warn", "OAuth Bot 已配置,但服务器直连可能被封。配个代理更稳定:agent-reach configure proxy URL"
|
|
||||||
else:
|
|
||||||
return "off", "搜索用 Exa 免费可用。读帖子需配个代理:agent-reach configure proxy URL"
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
proxy = config.get("reddit_proxy") if config else None
|
|
||||||
proxies = {"http": proxy, "https": proxy} if proxy else None
|
|
||||||
|
|
||||||
# Clean URL: remove query params, trailing slash, then add .json
|
|
||||||
parsed = urlparse(url)
|
|
||||||
clean_path = parsed.path.rstrip("/")
|
|
||||||
# Remove trailing .json if already present (avoid double .json)
|
|
||||||
if clean_path.endswith(".json"):
|
|
||||||
clean_path = clean_path[:-5]
|
|
||||||
json_url = f"https://www.reddit.com{clean_path}.json"
|
|
||||||
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
json_url,
|
|
||||||
headers={"User-Agent": self.USER_AGENT},
|
|
||||||
proxies=proxies,
|
|
||||||
params={"limit": 50},
|
|
||||||
timeout=15,
|
|
||||||
)
|
)
|
||||||
resp.raise_for_status()
|
|
||||||
except requests.exceptions.HTTPError as e:
|
|
||||||
status = e.response.status_code if e.response is not None else 0
|
|
||||||
if status in (403, 429):
|
|
||||||
return ReadResult(
|
|
||||||
title="Reddit",
|
|
||||||
content="⚠️ Reddit blocked this request (403 Forbidden). "
|
|
||||||
"Reddit blocks most server IPs.\n"
|
|
||||||
"Fix: agent-reach configure proxy http://user:pass@ip:port\n"
|
|
||||||
"Cheap option: https://www.webshare.io ($1/month)\n\n"
|
|
||||||
"Alternatively, search Reddit via Exa (free, no proxy needed): "
|
|
||||||
"agent-reach search-reddit \"your query\"",
|
|
||||||
url=url,
|
|
||||||
platform="reddit",
|
|
||||||
)
|
|
||||||
raise
|
|
||||||
|
|
||||||
data = resp.json()
|
|
||||||
|
|
||||||
# Subreddit listing page: /r/sub/, /r/sub/hot, /r/sub/new, /r/sub/top
|
|
||||||
if isinstance(data, dict) and data.get("kind") == "Listing":
|
|
||||||
return self._parse_listing(data, url)
|
|
||||||
|
|
||||||
if isinstance(data, list) and len(data) >= 1:
|
|
||||||
# Post page: [post_listing, comments_listing]
|
|
||||||
post = data[0]["data"]["children"][0]["data"]
|
|
||||||
title = post.get("title", "")
|
|
||||||
author = post.get("author", "")
|
|
||||||
selftext = post.get("selftext", "")
|
|
||||||
score = post.get("score", 0)
|
|
||||||
subreddit = post.get("subreddit", "")
|
|
||||||
|
|
||||||
# Extract comments
|
|
||||||
comments_text = ""
|
|
||||||
if len(data) >= 2:
|
|
||||||
comments_text = self._extract_comments(data[1])
|
|
||||||
|
|
||||||
content = selftext
|
|
||||||
if comments_text:
|
|
||||||
content += f"\n\n---\n## Comments\n{comments_text}"
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=title,
|
|
||||||
content=content,
|
|
||||||
url=url,
|
|
||||||
author=f"u/{author}",
|
|
||||||
platform="reddit",
|
|
||||||
extra={"subreddit": subreddit, "score": score},
|
|
||||||
)
|
|
||||||
|
|
||||||
raise ValueError(f"Could not parse Reddit response for: {url}")
|
|
||||||
|
|
||||||
def _parse_listing(self, data: dict, url: str) -> ReadResult:
|
|
||||||
"""Parse a subreddit listing (hot/new/top/rising)."""
|
|
||||||
children = data.get("data", {}).get("children", [])
|
|
||||||
|
|
||||||
# Extract subreddit name and sort from URL
|
|
||||||
parsed = urlparse(url)
|
|
||||||
path_parts = [p for p in parsed.path.strip("/").split("/") if p]
|
|
||||||
subreddit = path_parts[1] if len(path_parts) >= 2 else "reddit"
|
|
||||||
sort_type = path_parts[2] if len(path_parts) >= 3 else "hot"
|
|
||||||
|
|
||||||
lines = []
|
|
||||||
for i, child in enumerate(children, 1):
|
|
||||||
if child.get("kind") != "t3":
|
|
||||||
continue
|
|
||||||
post = child.get("data", {})
|
|
||||||
title = post.get("title", "")
|
|
||||||
author = post.get("author", "")
|
|
||||||
score = post.get("score", 0)
|
|
||||||
num_comments = post.get("num_comments", 0)
|
|
||||||
permalink = post.get("permalink", "")
|
|
||||||
post_url = post.get("url", "")
|
|
||||||
is_self = post.get("is_self", False)
|
|
||||||
|
|
||||||
lines.append(f"### {i}. {title}")
|
|
||||||
lines.append(f"👤 u/{author} · ⬆ {score} · 💬 {num_comments}")
|
|
||||||
if not is_self and post_url:
|
|
||||||
lines.append(f"🔗 {post_url}")
|
|
||||||
lines.append(f"📎 https://www.reddit.com{permalink}")
|
|
||||||
# Add selftext preview (first 200 chars)
|
|
||||||
selftext = post.get("selftext", "")
|
|
||||||
if selftext:
|
|
||||||
preview = selftext[:200].replace("\n", " ")
|
|
||||||
if len(selftext) > 200:
|
|
||||||
preview += "..."
|
|
||||||
lines.append(f"> {preview}")
|
|
||||||
lines.append("")
|
|
||||||
|
|
||||||
content = "\n".join(lines) if lines else "No posts found."
|
|
||||||
return ReadResult(
|
|
||||||
title=f"r/{subreddit} — {sort_type}",
|
|
||||||
content=content,
|
|
||||||
url=url,
|
|
||||||
platform="reddit",
|
|
||||||
extra={"subreddit": subreddit, "sort": sort_type, "count": len(children)},
|
|
||||||
)
|
|
||||||
|
|
||||||
def _extract_comments(self, comments_data: dict, depth: int = 0, max_depth: int = 3) -> str:
|
|
||||||
"""Recursively extract comments."""
|
|
||||||
lines = []
|
|
||||||
children = comments_data.get("data", {}).get("children", [])
|
|
||||||
|
|
||||||
for child in children:
|
|
||||||
if child.get("kind") != "t1":
|
|
||||||
continue
|
|
||||||
data = child.get("data", {})
|
|
||||||
author = data.get("author", "[deleted]")
|
|
||||||
body = data.get("body", "")
|
|
||||||
score = data.get("score", 0)
|
|
||||||
indent = " " * depth
|
|
||||||
|
|
||||||
lines.append(f"{indent}**u/{author}** ({score} points):")
|
|
||||||
lines.append(f"{indent}{body}")
|
|
||||||
lines.append("")
|
|
||||||
|
|
||||||
# Recurse into replies
|
|
||||||
if depth < max_depth and data.get("replies") and isinstance(data["replies"], dict):
|
|
||||||
lines.append(self._extract_comments(data["replies"], depth + 1, max_depth))
|
|
||||||
|
|
||||||
return "\n".join(lines)
|
|
||||||
|
|||||||
@@ -1,13 +1,7 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""RSS feeds — via feedparser (free, pip dependency).
|
"""RSS — check if feedparser is available."""
|
||||||
|
|
||||||
Backend: feedparser (https://github.com/kurtmckee/feedparser)
|
from .base import Channel
|
||||||
Swap to: any RSS parser
|
|
||||||
"""
|
|
||||||
|
|
||||||
import feedparser
|
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult
|
|
||||||
|
|
||||||
|
|
||||||
class RSSChannel(Channel):
|
class RSSChannel(Channel):
|
||||||
@@ -17,41 +11,11 @@ class RSSChannel(Channel):
|
|||||||
tier = 0
|
tier = 0
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
lower = url.lower()
|
return any(x in url.lower() for x in ["/feed", "/rss", ".xml", "atom"])
|
||||||
domain = urlparse(url).netloc.lower()
|
|
||||||
return (lower.endswith(".xml") or "/rss" in lower or "/feed" in lower
|
|
||||||
or "/atom" in lower or "rss" in domain)
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
def check(self, config=None):
|
||||||
feed = feedparser.parse(url)
|
try:
|
||||||
|
import feedparser
|
||||||
if feed.bozo and not feed.entries:
|
return "ok", "可读取 RSS/Atom 源"
|
||||||
raise ValueError(f"Failed to parse RSS feed: {url}")
|
except ImportError:
|
||||||
|
return "off", "feedparser 未安装。安装:pip install feedparser"
|
||||||
if not feed.entries:
|
|
||||||
raise ValueError(f"No entries in RSS feed: {url}")
|
|
||||||
|
|
||||||
# Return latest entry
|
|
||||||
entry = feed.entries[0]
|
|
||||||
content = entry.get("summary", "") or entry.get("description", "")
|
|
||||||
|
|
||||||
# If multiple entries, summarize all
|
|
||||||
if len(feed.entries) > 1:
|
|
||||||
lines = [f"# {feed.feed.get('title', 'RSS Feed')}\n"]
|
|
||||||
for i, e in enumerate(feed.entries[:20], 1):
|
|
||||||
title = e.get("title", "Untitled")
|
|
||||||
link = e.get("link", "")
|
|
||||||
summary = e.get("summary", "")[:200]
|
|
||||||
lines.append(f"## {i}. {title}")
|
|
||||||
lines.append(f"🔗 {link}")
|
|
||||||
if summary:
|
|
||||||
lines.append(summary)
|
|
||||||
lines.append("")
|
|
||||||
content = "\n".join(lines)
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=feed.feed.get("title", entry.get("title", url)),
|
|
||||||
content=content,
|
|
||||||
url=url,
|
|
||||||
platform="rss",
|
|
||||||
)
|
|
||||||
|
|||||||
+20
-191
@@ -1,209 +1,38 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Twitter/X — via bird CLI (free) or Jina Reader fallback.
|
"""Twitter/X — check if bird CLI is available."""
|
||||||
|
|
||||||
Backend: bird (@steipete/bird npm package) for search/timeline
|
|
||||||
Jina Reader for single tweets
|
|
||||||
Swap to: any Twitter access tool
|
|
||||||
"""
|
|
||||||
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
import requests
|
|
||||||
|
|
||||||
|
|
||||||
def _bird_cmd():
|
|
||||||
"""Find bird CLI binary."""
|
|
||||||
return shutil.which("bird") or shutil.which("birdx")
|
|
||||||
|
|
||||||
|
|
||||||
def _bird_env(config=None):
|
|
||||||
"""Build env dict with Twitter cookies for bird CLI."""
|
|
||||||
import os
|
|
||||||
env = os.environ.copy()
|
|
||||||
if config:
|
|
||||||
auth_token = config.get("twitter_auth_token")
|
|
||||||
ct0 = config.get("twitter_ct0")
|
|
||||||
if auth_token:
|
|
||||||
env["AUTH_TOKEN"] = auth_token
|
|
||||||
if ct0:
|
|
||||||
env["CT0"] = ct0
|
|
||||||
return env
|
|
||||||
|
|
||||||
|
|
||||||
class TwitterChannel(Channel):
|
class TwitterChannel(Channel):
|
||||||
name = "twitter"
|
name = "twitter"
|
||||||
description = "Twitter/X 推文"
|
description = "Twitter/X 推文"
|
||||||
backends = ["bird", "Jina Reader"]
|
backends = ["bird CLI"]
|
||||||
tier = 0 # Single tweet reading is zero-config
|
tier = 1
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
domain = urlparse(url).netloc.lower()
|
from urllib.parse import urlparse
|
||||||
return "x.com" in domain or "twitter.com" in domain
|
d = urlparse(url).netloc.lower()
|
||||||
|
return "x.com" in d or "twitter.com" in d
|
||||||
|
|
||||||
def check(self, config=None):
|
def check(self, config=None):
|
||||||
# Basic reading always works (Jina fallback)
|
bird = shutil.which("bird") or shutil.which("birdx")
|
||||||
if _bird_cmd():
|
if not bird:
|
||||||
return "ok", "搜索、时间线、发推全部可用"
|
return "warn", (
|
||||||
return "ok", "可读取推文。安装 bird + 配置 Cookie 可解锁搜索和发推"
|
"bird CLI 未安装。搜索可通过 Exa 替代。安装:\n"
|
||||||
|
" npm install -g @steipete/bird"
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
# Try bird first
|
|
||||||
bird = _bird_cmd()
|
|
||||||
if bird:
|
|
||||||
return await self._read_bird(url, bird, config)
|
|
||||||
# Fallback: Jina Reader
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
async def _read_bird(self, url: str, bird: str, config=None) -> ReadResult:
|
|
||||||
result = subprocess.run(
|
|
||||||
[bird, "read", url],
|
|
||||||
capture_output=True, timeout=30,
|
|
||||||
encoding='utf-8', errors='replace',
|
|
||||||
env=_bird_env(config),
|
|
||||||
)
|
)
|
||||||
if result.returncode != 0:
|
|
||||||
return await self._read_jina(url)
|
|
||||||
|
|
||||||
text = result.stdout.strip()
|
|
||||||
# Extract author from first line
|
|
||||||
author = ""
|
|
||||||
lines = text.split("\n")
|
|
||||||
if lines and lines[0].startswith("@"):
|
|
||||||
author = lines[0].split()[0]
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=text[:100],
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
author=author,
|
|
||||||
platform="twitter",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _read_jina(self, url: str) -> ReadResult:
|
|
||||||
try:
|
try:
|
||||||
resp = requests.get(
|
r = subprocess.run(
|
||||||
f"https://r.jina.ai/{url}",
|
[bird, "whoami"], capture_output=True, text=True, timeout=10
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
)
|
||||||
resp.raise_for_status()
|
if r.returncode == 0:
|
||||||
text = resp.text
|
return "ok", "完整可用(读取、搜索推文)"
|
||||||
|
return "warn", (
|
||||||
# Detect unusable Jina responses for X/Twitter (JS-required pages)
|
"bird CLI 已安装但未配置 Cookie。运行:\n"
|
||||||
unusable_indicators = [
|
" agent-reach configure twitter-cookies \"auth_token=xxx; ct0=yyy\""
|
||||||
"page doesn", # "this page doesn't exist" (handles both ' and ')
|
|
||||||
"miss what", # "Don't miss what's happening"
|
|
||||||
"Something went wrong. Try reloading",
|
|
||||||
"Log in](", # Markdown link: [Log in](...)
|
|
||||||
]
|
|
||||||
if any(indicator in text for indicator in unusable_indicators):
|
|
||||||
return ReadResult(
|
|
||||||
title="Twitter/X",
|
|
||||||
content="⚠️ Could not read this tweet.\n"
|
|
||||||
"The tweet may have been deleted, or the account is private.\n\n"
|
|
||||||
"Tips:\n"
|
|
||||||
"- Make sure the URL is correct\n"
|
|
||||||
"- Try: bird read <url> (if bird CLI is installed)\n"
|
|
||||||
"- For protected tweets, configure Twitter cookies: "
|
|
||||||
"agent-reach configure twitter-cookies AUTH_TOKEN CT0",
|
|
||||||
url=url,
|
|
||||||
platform="twitter",
|
|
||||||
)
|
|
||||||
|
|
||||||
title = text[:100] if text else url
|
|
||||||
return ReadResult(
|
|
||||||
title=title,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="twitter",
|
|
||||||
)
|
)
|
||||||
except Exception:
|
except Exception:
|
||||||
return ReadResult(
|
return "warn", "bird CLI 已安装但连接失败"
|
||||||
title="Twitter/X",
|
|
||||||
content="⚠️ Could not read this tweet.\n"
|
|
||||||
"The tweet may have been deleted, or the account is private.\n\n"
|
|
||||||
"Tips:\n"
|
|
||||||
"- Make sure the URL is correct\n"
|
|
||||||
"- Try: bird read <url> (if bird CLI is installed)\n"
|
|
||||||
"- For protected tweets, configure Twitter cookies: "
|
|
||||||
"agent-reach configure twitter-cookies AUTH_TOKEN CT0",
|
|
||||||
url=url,
|
|
||||||
platform="twitter",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
|
|
||||||
bird = _bird_cmd()
|
|
||||||
if bird:
|
|
||||||
return await self._search_bird(query, limit, bird, config)
|
|
||||||
|
|
||||||
# Fallback to Exa
|
|
||||||
return await self._search_exa(query, limit, config)
|
|
||||||
|
|
||||||
async def _search_bird(self, query: str, limit: int, bird: str, config=None) -> List[SearchResult]:
|
|
||||||
try:
|
|
||||||
result = subprocess.run(
|
|
||||||
[bird, "search", query, "-n", str(limit)],
|
|
||||||
capture_output=True, timeout=30,
|
|
||||||
encoding='utf-8', errors='replace',
|
|
||||||
env=_bird_env(config),
|
|
||||||
)
|
|
||||||
if result.returncode != 0:
|
|
||||||
return []
|
|
||||||
|
|
||||||
return self._parse_bird_output(result.stdout)
|
|
||||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_bird_output(self, text: str) -> List[SearchResult]:
|
|
||||||
"""Parse bird text output into SearchResults."""
|
|
||||||
results = []
|
|
||||||
current = {}
|
|
||||||
text_lines = []
|
|
||||||
|
|
||||||
for line in text.strip().split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line.startswith("─"):
|
|
||||||
if current:
|
|
||||||
current["text"] = "\n".join(text_lines).strip()
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=current.get("text", "")[:80],
|
|
||||||
url=current.get("url", ""),
|
|
||||||
snippet=current.get("text", ""),
|
|
||||||
author=current.get("author", ""),
|
|
||||||
date=current.get("date", ""),
|
|
||||||
))
|
|
||||||
current = {}
|
|
||||||
text_lines = []
|
|
||||||
continue
|
|
||||||
if line.startswith("@") and line.endswith(":") and "(" in line:
|
|
||||||
current["author"] = line.split()[0]
|
|
||||||
continue
|
|
||||||
if line.startswith("date:"):
|
|
||||||
current["date"] = line[5:].strip()
|
|
||||||
continue
|
|
||||||
if line.startswith("url:"):
|
|
||||||
current["url"] = line[4:].strip()
|
|
||||||
continue
|
|
||||||
if current is not None:
|
|
||||||
text_lines.append(line)
|
|
||||||
|
|
||||||
if current and text_lines:
|
|
||||||
current["text"] = "\n".join(text_lines).strip()
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=current.get("text", "")[:80],
|
|
||||||
url=current.get("url", ""),
|
|
||||||
snippet=current.get("text", ""),
|
|
||||||
author=current.get("author", ""),
|
|
||||||
date=current.get("date", ""),
|
|
||||||
))
|
|
||||||
return results
|
|
||||||
|
|
||||||
async def _search_exa(self, query: str, limit: int, config=None) -> List[SearchResult]:
|
|
||||||
from agent_reach.channels.exa_search import ExaSearchChannel
|
|
||||||
exa = ExaSearchChannel()
|
|
||||||
return await exa.search(f"site:x.com {query}", config=config, limit=limit)
|
|
||||||
|
|||||||
@@ -1,49 +1,17 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""Web pages — via Jina Reader API (free, no config needed).
|
"""Web — any URL via Jina Reader. Always available."""
|
||||||
|
|
||||||
Backend: Jina Reader (https://r.jina.ai)
|
from .base import Channel
|
||||||
Swap to: Firecrawl, Trafilatura, or any other reader API
|
|
||||||
"""
|
|
||||||
|
|
||||||
import requests
|
|
||||||
from .base import Channel, ReadResult
|
|
||||||
|
|
||||||
|
|
||||||
class WebChannel(Channel):
|
class WebChannel(Channel):
|
||||||
name = "web"
|
name = "web"
|
||||||
description = "网页(任意 URL)"
|
description = "任意网页"
|
||||||
backends = ["Jina Reader API"]
|
backends = ["Jina Reader"]
|
||||||
tier = 0
|
tier = 0
|
||||||
|
|
||||||
JINA_URL = "https://r.jina.ai/"
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
# Fallback — handles any URL not matched by other channels
|
return True # Fallback — handles any URL
|
||||||
return True
|
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
def check(self, config=None):
|
||||||
resp = requests.get(
|
return "ok", "通过 Jina Reader 读取任意网页(curl https://r.jina.ai/URL)"
|
||||||
f"{self.JINA_URL}{url}",
|
|
||||||
headers={"Accept": "text/markdown"},
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
resp.raise_for_status()
|
|
||||||
text = resp.text
|
|
||||||
|
|
||||||
# Extract title from first markdown heading
|
|
||||||
title = url
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line.startswith("# "):
|
|
||||||
title = line[2:].strip()
|
|
||||||
break
|
|
||||||
if line.startswith("Title:"):
|
|
||||||
title = line[6:].strip()
|
|
||||||
break
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=title,
|
|
||||||
content=text,
|
|
||||||
url=url,
|
|
||||||
platform="web",
|
|
||||||
)
|
|
||||||
|
|||||||
@@ -1,16 +1,9 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""XiaoHongShu (小红书) — via mcporter + xiaohongshu MCP server.
|
"""XiaoHongShu — check if mcporter + xiaohongshu MCP is available."""
|
||||||
|
|
||||||
Backend: xiaohongshu-mcp server (internal API, reliable)
|
|
||||||
Requires: mcporter CLI + xiaohongshu MCP server running
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
from urllib.parse import urlparse
|
from .base import Channel
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List, Optional
|
|
||||||
|
|
||||||
|
|
||||||
class XiaoHongShuChannel(Channel):
|
class XiaoHongShuChannel(Channel):
|
||||||
@@ -19,30 +12,8 @@ class XiaoHongShuChannel(Channel):
|
|||||||
backends = ["xiaohongshu-mcp"]
|
backends = ["xiaohongshu-mcp"]
|
||||||
tier = 2
|
tier = 2
|
||||||
|
|
||||||
def _mcporter_ok(self) -> bool:
|
|
||||||
"""Check if mcporter + xiaohongshu MCP is available."""
|
|
||||||
if not shutil.which("mcporter"):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
|
||||||
)
|
|
||||||
return "xiaohongshu" in r.stdout
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
def _call(self, expr: str, timeout: int = 30) -> str:
|
|
||||||
r = subprocess.run(
|
|
||||||
["mcporter", "call", expr],
|
|
||||||
capture_output=True, text=True, timeout=timeout,
|
|
||||||
)
|
|
||||||
if r.returncode != 0:
|
|
||||||
raise RuntimeError(r.stderr or r.stdout)
|
|
||||||
return r.stdout
|
|
||||||
|
|
||||||
# ── Channel interface ──
|
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
d = urlparse(url).netloc.lower()
|
d = urlparse(url).netloc.lower()
|
||||||
return "xiaohongshu.com" in d or "xhslink.com" in d
|
return "xiaohongshu.com" in d or "xhslink.com" in d
|
||||||
|
|
||||||
@@ -55,118 +26,25 @@ class XiaoHongShuChannel(Channel):
|
|||||||
" 3. mcporter config add xiaohongshu http://localhost:18060/mcp\n"
|
" 3. mcporter config add xiaohongshu http://localhost:18060/mcp\n"
|
||||||
" 详见 https://github.com/xpzouying/xiaohongshu-mcp"
|
" 详见 https://github.com/xpzouying/xiaohongshu-mcp"
|
||||||
)
|
)
|
||||||
if not self._mcporter_ok():
|
try:
|
||||||
|
r = subprocess.run(
|
||||||
|
["mcporter", "list"], capture_output=True, text=True, timeout=10
|
||||||
|
)
|
||||||
|
if "xiaohongshu" not in r.stdout:
|
||||||
return "off", (
|
return "off", (
|
||||||
"mcporter 已装但小红书 MCP 未配置。运行:\n"
|
"mcporter 已装但小红书 MCP 未配置。运行:\n"
|
||||||
" docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
|
" docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
|
||||||
" mcporter config add xiaohongshu http://localhost:18060/mcp"
|
" mcporter config add xiaohongshu http://localhost:18060/mcp"
|
||||||
)
|
)
|
||||||
|
except Exception:
|
||||||
|
return "off", "mcporter 连接异常"
|
||||||
try:
|
try:
|
||||||
out = self._call("xiaohongshu.check_login_status()", timeout=10)
|
r = subprocess.run(
|
||||||
if "已登录" in out or "logged" in out.lower():
|
["mcporter", "call", "xiaohongshu.check_login_status()"],
|
||||||
|
capture_output=True, text=True, timeout=10
|
||||||
|
)
|
||||||
|
if "已登录" in r.stdout or "logged" in r.stdout.lower():
|
||||||
return "ok", "完整可用(阅读、搜索、发帖、评论、点赞)"
|
return "ok", "完整可用(阅读、搜索、发帖、评论、点赞)"
|
||||||
return "warn", "MCP 已连接但未登录,需扫码登录"
|
return "warn", "MCP 已连接但未登录,需扫码登录"
|
||||||
except Exception:
|
except Exception:
|
||||||
return "warn", "MCP 连接异常,检查 xiaohongshu-mcp 服务是否在运行"
|
return "warn", "MCP 连接异常,检查 xiaohongshu-mcp 服务是否在运行"
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
return ReadResult(
|
|
||||||
title="XiaoHongShu",
|
|
||||||
content=(
|
|
||||||
"⚠️ 小红书需要 mcporter + xiaohongshu-mcp 才能使用。\n\n"
|
|
||||||
"安装步骤:\n"
|
|
||||||
"1. npm install -g mcporter\n"
|
|
||||||
"2. docker run -d --name xiaohongshu-mcp -p 18060:18060 xpzouying/xiaohongshu-mcp\n"
|
|
||||||
"3. mcporter config add xiaohongshu http://localhost:18060/mcp\n"
|
|
||||||
"4. 运行 agent-reach doctor 检查状态\n\n"
|
|
||||||
"详见 https://github.com/xpzouying/xiaohongshu-mcp"
|
|
||||||
),
|
|
||||||
url=url, platform="xiaohongshu",
|
|
||||||
)
|
|
||||||
|
|
||||||
note_id = self._extract_note_id(url)
|
|
||||||
if not note_id:
|
|
||||||
return ReadResult(
|
|
||||||
title="XiaoHongShu",
|
|
||||||
content=f"⚠️ 无法从 URL 提取笔记 ID: {url}",
|
|
||||||
url=url, platform="xiaohongshu",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Step 1: get xsec_token from feeds
|
|
||||||
xsec_token = self._find_token(note_id)
|
|
||||||
|
|
||||||
if not xsec_token:
|
|
||||||
return ReadResult(
|
|
||||||
title="XiaoHongShu",
|
|
||||||
content=(
|
|
||||||
f"⚠️ 无法获取笔记 {note_id} 的访问令牌。\n"
|
|
||||||
"小红书需要 xsec_token 才能读取笔记详情。\n"
|
|
||||||
"请先通过搜索找到这篇笔记,或直接使用搜索功能。"
|
|
||||||
),
|
|
||||||
url=url, platform="xiaohongshu",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Step 2: get detail
|
|
||||||
out = self._call(
|
|
||||||
f'xiaohongshu.get_feed_detail(feed_id: "{note_id}", xsec_token: "{xsec_token}")',
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=self._extract_title(out) or f"XHS {note_id}",
|
|
||||||
content=out.strip(),
|
|
||||||
url=url, platform="xiaohongshu",
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
if not self._mcporter_ok():
|
|
||||||
raise ValueError(
|
|
||||||
"小红书搜索需要 mcporter + xiaohongshu-mcp。\n"
|
|
||||||
"安装: npm install -g mcporter && mcporter config add xiaohongshu http://localhost:18060/mcp"
|
|
||||||
)
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
safe_q = query.replace('"', '\\"')
|
|
||||||
out = self._call(f'xiaohongshu.search_feeds(keyword: "{safe_q}")', timeout=30)
|
|
||||||
|
|
||||||
results = []
|
|
||||||
try:
|
|
||||||
data = json.loads(out)
|
|
||||||
for item in data.get("feeds", [])[:limit]:
|
|
||||||
card = item.get("noteCard", {})
|
|
||||||
user = card.get("user", {})
|
|
||||||
interact = card.get("interactInfo", {})
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=card.get("displayTitle", ""),
|
|
||||||
url=f"https://www.xiaohongshu.com/explore/{item.get('id', '')}",
|
|
||||||
snippet=f"👤 {user.get('nickname', '')} · ❤ {interact.get('likedCount', '0')}",
|
|
||||||
score=0,
|
|
||||||
))
|
|
||||||
except (json.JSONDecodeError, KeyError):
|
|
||||||
pass
|
|
||||||
return results
|
|
||||||
|
|
||||||
# ── Helpers ──
|
|
||||||
|
|
||||||
def _extract_note_id(self, url: str) -> str:
|
|
||||||
parts = urlparse(url).path.strip("/").split("/")
|
|
||||||
return parts[-1] if parts else ""
|
|
||||||
|
|
||||||
def _find_token(self, note_id: str) -> Optional[str]:
|
|
||||||
"""Try to find xsec_token for a note from feeds."""
|
|
||||||
try:
|
|
||||||
out = self._call("xiaohongshu.list_feeds()", timeout=15)
|
|
||||||
data = json.loads(out)
|
|
||||||
for feed in data.get("feeds", []):
|
|
||||||
if feed.get("id") == note_id:
|
|
||||||
return feed.get("xsecToken", "")
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return None
|
|
||||||
|
|
||||||
def _extract_title(self, text: str) -> str:
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if line and not line.startswith(("{", "[", "#", "http")):
|
|
||||||
return line[:80]
|
|
||||||
return ""
|
|
||||||
|
|||||||
@@ -1,125 +1,22 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""YouTube — via yt-dlp (video info, subtitles, and search).
|
"""YouTube — check if yt-dlp is available."""
|
||||||
|
|
||||||
Backend: yt-dlp (https://github.com/yt-dlp/yt-dlp)
|
|
||||||
Supports: read (info + subtitles), search (ytsearch)
|
|
||||||
"""
|
|
||||||
|
|
||||||
import json
|
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
from .base import Channel
|
||||||
import tempfile
|
|
||||||
from pathlib import Path
|
|
||||||
from urllib.parse import urlparse
|
|
||||||
from .base import Channel, ReadResult, SearchResult
|
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
class YouTubeChannel(Channel):
|
class YouTubeChannel(Channel):
|
||||||
name = "youtube"
|
name = "youtube"
|
||||||
description = "YouTube 视频字幕"
|
description = "YouTube 视频和字幕"
|
||||||
backends = ["yt-dlp"]
|
backends = ["yt-dlp"]
|
||||||
requires_tools = ["yt-dlp"]
|
|
||||||
tier = 0
|
tier = 0
|
||||||
|
|
||||||
def can_handle(self, url: str) -> bool:
|
def can_handle(self, url: str) -> bool:
|
||||||
|
from urllib.parse import urlparse
|
||||||
d = urlparse(url).netloc.lower()
|
d = urlparse(url).netloc.lower()
|
||||||
return "youtube.com" in d or "youtu.be" in d
|
return "youtube.com" in d or "youtu.be" in d
|
||||||
|
|
||||||
async def read(self, url: str, config=None) -> ReadResult:
|
def check(self, config=None):
|
||||||
if not shutil.which("yt-dlp"):
|
if shutil.which("yt-dlp"):
|
||||||
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
|
return "ok", "可提取视频信息和字幕"
|
||||||
|
return "off", "yt-dlp 未安装。安装:pip install yt-dlp"
|
||||||
with tempfile.TemporaryDirectory() as tmpdir:
|
|
||||||
info = self._get_info(url)
|
|
||||||
title = info.get("title", url)
|
|
||||||
author = info.get("uploader", "")
|
|
||||||
|
|
||||||
transcript = self._get_subtitles(url, tmpdir)
|
|
||||||
if not transcript:
|
|
||||||
transcript = f"[Video: {title}]\n[No subtitles available.]"
|
|
||||||
|
|
||||||
return ReadResult(
|
|
||||||
title=title, content=transcript, url=url,
|
|
||||||
author=author, platform="youtube",
|
|
||||||
extra={
|
|
||||||
"duration": info.get("duration_string"),
|
|
||||||
"view_count": info.get("view_count"),
|
|
||||||
"upload_date": info.get("upload_date"),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, config=None, **kwargs) -> List[SearchResult]:
|
|
||||||
"""Search YouTube via yt-dlp's ytsearch."""
|
|
||||||
if not shutil.which("yt-dlp"):
|
|
||||||
raise RuntimeError("yt-dlp not installed. Install: pip install yt-dlp")
|
|
||||||
|
|
||||||
limit = kwargs.get("limit", 10)
|
|
||||||
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["yt-dlp", "--dump-json", "--flat-playlist",
|
|
||||||
f"ytsearch{limit}:{query}"],
|
|
||||||
capture_output=True, text=True, timeout=30,
|
|
||||||
)
|
|
||||||
results = []
|
|
||||||
for line in r.stdout.strip().split("\n"):
|
|
||||||
if not line.strip():
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
d = json.loads(line)
|
|
||||||
vid = d.get("id", "")
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=d.get("title", ""),
|
|
||||||
url=f"https://youtube.com/watch?v={vid}" if vid else "",
|
|
||||||
snippet=(
|
|
||||||
f"👤 {d.get('channel', '?')} · "
|
|
||||||
f"⏱ {d.get('duration_string', '?')} · "
|
|
||||||
f"👁 {d.get('view_count', '?')}"
|
|
||||||
),
|
|
||||||
extra={
|
|
||||||
"channel": d.get("channel"),
|
|
||||||
"duration": d.get("duration_string"),
|
|
||||||
"view_count": d.get("view_count"),
|
|
||||||
},
|
|
||||||
))
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
continue
|
|
||||||
return results
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _get_info(self, url: str) -> dict:
|
|
||||||
try:
|
|
||||||
r = subprocess.run(
|
|
||||||
["yt-dlp", "--dump-json", "--no-download", url],
|
|
||||||
capture_output=True, text=True, timeout=30,
|
|
||||||
)
|
|
||||||
if r.returncode == 0:
|
|
||||||
return json.loads(r.stdout)
|
|
||||||
except (subprocess.TimeoutExpired, json.JSONDecodeError):
|
|
||||||
pass
|
|
||||||
return {}
|
|
||||||
|
|
||||||
def _get_subtitles(self, url: str, tmpdir: str) -> str:
|
|
||||||
try:
|
|
||||||
subprocess.run(
|
|
||||||
["yt-dlp", "--write-auto-sub", "--write-sub",
|
|
||||||
"--sub-lang", "en,zh-Hans,zh",
|
|
||||||
"--skip-download", "--sub-format", "vtt",
|
|
||||||
"-o", f"{tmpdir}/%(id)s.%(ext)s", url],
|
|
||||||
capture_output=True, text=True, timeout=30,
|
|
||||||
)
|
|
||||||
for f in Path(tmpdir).glob("*.vtt"):
|
|
||||||
text = f.read_text(errors="replace")
|
|
||||||
lines = []
|
|
||||||
for line in text.split("\n"):
|
|
||||||
line = line.strip()
|
|
||||||
if not line or line.startswith("WEBVTT") or "-->" in line or line.isdigit():
|
|
||||||
continue
|
|
||||||
if line not in lines[-1:]:
|
|
||||||
lines.append(line)
|
|
||||||
return "\n".join(lines)
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
pass
|
|
||||||
return ""
|
|
||||||
|
|||||||
+14
-207
@@ -1,20 +1,15 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Agent Reach CLI — command-line interface.
|
Agent Reach CLI — installer, doctor, and configuration tool.
|
||||||
|
|
||||||
Usage:
|
Usage:
|
||||||
agent-reach read <url>
|
agent-reach install --env=auto
|
||||||
agent-reach search <query>
|
|
||||||
agent-reach search-reddit <query> [--sub <subreddit>]
|
|
||||||
agent-reach search-github <query> [--lang <language>]
|
|
||||||
agent-reach search-twitter <query>
|
|
||||||
agent-reach setup
|
|
||||||
agent-reach doctor
|
agent-reach doctor
|
||||||
agent-reach version
|
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
|
||||||
|
agent-reach setup
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import sys
|
import sys
|
||||||
import asyncio
|
|
||||||
import argparse
|
import argparse
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
@@ -48,62 +43,6 @@ def main():
|
|||||||
sub = parser.add_subparsers(dest="command", help="Available commands")
|
sub = parser.add_subparsers(dest="command", help="Available commands")
|
||||||
|
|
||||||
# ── read ──
|
# ── read ──
|
||||||
p_read = sub.add_parser("read", help="Read content from a URL")
|
|
||||||
p_read.add_argument("url", help="URL to read")
|
|
||||||
p_read.add_argument("--json", dest="as_json", action="store_true", help="Output as JSON")
|
|
||||||
|
|
||||||
# ── search ──
|
|
||||||
p_search = sub.add_parser("search", help="Search the web (Exa)")
|
|
||||||
p_search.add_argument("query", nargs="+", help="Search query")
|
|
||||||
p_search.add_argument("-n", "--num", type=int, default=5, help="Number of results")
|
|
||||||
|
|
||||||
# ── search-reddit ──
|
|
||||||
p_sr = sub.add_parser("search-reddit", help="Search Reddit")
|
|
||||||
p_sr.add_argument("query", nargs="+", help="Search query")
|
|
||||||
p_sr.add_argument("--sub", help="Subreddit filter")
|
|
||||||
p_sr.add_argument("-n", "--num", type=int, default=10, help="Number of results")
|
|
||||||
|
|
||||||
# ── search-github ──
|
|
||||||
p_sg = sub.add_parser("search-github", help="Search GitHub")
|
|
||||||
p_sg.add_argument("query", nargs="+", help="Search query")
|
|
||||||
p_sg.add_argument("--lang", help="Language filter")
|
|
||||||
p_sg.add_argument("-n", "--num", type=int, default=5, help="Number of results")
|
|
||||||
|
|
||||||
# ── search-twitter ──
|
|
||||||
p_st = sub.add_parser("search-twitter", help="Search Twitter")
|
|
||||||
p_st.add_argument("query", nargs="+", help="Search query")
|
|
||||||
p_st.add_argument("-n", "--num", type=int, default=10, help="Number of results")
|
|
||||||
|
|
||||||
# ── search-youtube ──
|
|
||||||
p_sy = sub.add_parser("search-youtube", help="Search YouTube")
|
|
||||||
p_sy.add_argument("query", nargs="+", help="Search query")
|
|
||||||
p_sy.add_argument("-n", "--num", type=int, default=5, help="Number of results")
|
|
||||||
|
|
||||||
# ── search-bilibili ──
|
|
||||||
p_sb = sub.add_parser("search-bilibili", help="Search Bilibili")
|
|
||||||
p_sb.add_argument("query", nargs="+", help="Search query")
|
|
||||||
p_sb.add_argument("-n", "--num", type=int, default=5, help="Number of results")
|
|
||||||
|
|
||||||
# ── search-xhs ──
|
|
||||||
p_sx = sub.add_parser("search-xhs", help="Search XiaoHongShu")
|
|
||||||
p_sx.add_argument("query", nargs="+", help="Search query")
|
|
||||||
p_sx.add_argument("-n", "--num", type=int, default=10, help="Number of results")
|
|
||||||
|
|
||||||
# ── search-instagram ──
|
|
||||||
p_si = sub.add_parser("search-instagram", help="Search Instagram")
|
|
||||||
p_si.add_argument("query", nargs="+", help="Search query")
|
|
||||||
p_si.add_argument("-n", "--num", type=int, default=10, help="Number of results")
|
|
||||||
|
|
||||||
# ── search-linkedin ──
|
|
||||||
p_sl = sub.add_parser("search-linkedin", help="Search LinkedIn")
|
|
||||||
p_sl.add_argument("query", nargs="+", help="Search query")
|
|
||||||
p_sl.add_argument("-n", "--num", type=int, default=10, help="Number of results")
|
|
||||||
|
|
||||||
# ── search-bosszhipin ──
|
|
||||||
p_sbz = sub.add_parser("search-bosszhipin", help="Search Boss直聘")
|
|
||||||
p_sbz.add_argument("query", nargs="+", help="Search query")
|
|
||||||
p_sbz.add_argument("-n", "--num", type=int, default=10, help="Number of results")
|
|
||||||
|
|
||||||
# ── setup ──
|
# ── setup ──
|
||||||
sub.add_parser("setup", help="Interactive configuration wizard")
|
sub.add_parser("setup", help="Interactive configuration wizard")
|
||||||
|
|
||||||
@@ -122,8 +61,7 @@ def main():
|
|||||||
p_conf = sub.add_parser("configure", help="Set a config value or auto-extract from browser")
|
p_conf = sub.add_parser("configure", help="Set a config value or auto-extract from browser")
|
||||||
p_conf.add_argument("key", nargs="?", default=None,
|
p_conf.add_argument("key", nargs="?", default=None,
|
||||||
choices=["proxy", "github-token", "groq-key",
|
choices=["proxy", "github-token", "groq-key",
|
||||||
"twitter-cookies", "youtube-cookies",
|
"twitter-cookies", "youtube-cookies"],
|
||||||
"instagram-cookies"],
|
|
||||||
help="What to configure (omit if using --from-browser)")
|
help="What to configure (omit if using --from-browser)")
|
||||||
p_conf.add_argument("value", nargs="*", help="The value(s) to set")
|
p_conf.add_argument("value", nargs="*", help="The value(s) to set")
|
||||||
p_conf.add_argument("--from-browser", metavar="BROWSER",
|
p_conf.add_argument("--from-browser", metavar="BROWSER",
|
||||||
@@ -167,10 +105,6 @@ def main():
|
|||||||
_cmd_install(args)
|
_cmd_install(args)
|
||||||
elif args.command == "configure":
|
elif args.command == "configure":
|
||||||
_cmd_configure(args)
|
_cmd_configure(args)
|
||||||
elif args.command == "read":
|
|
||||||
asyncio.run(_cmd_read(args))
|
|
||||||
elif args.command.startswith("search"):
|
|
||||||
asyncio.run(_cmd_search(args))
|
|
||||||
|
|
||||||
|
|
||||||
# ── Command handlers ────────────────────────────────
|
# ── Command handlers ────────────────────────────────
|
||||||
@@ -423,22 +357,18 @@ def _install_system_deps():
|
|||||||
else:
|
else:
|
||||||
print(" ⬜ bird CLI requires Node.js (optional — Twitter reading still works via Jina)")
|
print(" ⬜ bird CLI requires Node.js (optional — Twitter reading still works via Jina)")
|
||||||
|
|
||||||
# ── instaloader (for Instagram) ──
|
# ── undici (proxy support for Node.js fetch) ──
|
||||||
if shutil.which("instaloader"):
|
if shutil.which("npm"):
|
||||||
print(" ✅ instaloader already installed")
|
npm_root = subprocess.run(["npm", "root", "-g"], capture_output=True, text=True, timeout=5).stdout.strip()
|
||||||
|
undici_path = os.path.join(npm_root, "undici", "index.js") if npm_root else ""
|
||||||
|
if os.path.exists(undici_path):
|
||||||
|
print(" ✅ undici already installed (Node.js proxy support)")
|
||||||
else:
|
else:
|
||||||
print(" 📥 Installing instaloader...")
|
|
||||||
try:
|
try:
|
||||||
subprocess.run(
|
subprocess.run(["npm", "install", "-g", "undici"], capture_output=True, text=True, timeout=60)
|
||||||
[sys.executable, "-m", "pip", "install", "instaloader"],
|
print(" ✅ undici installed (Node.js proxy support)")
|
||||||
capture_output=True, text=True, timeout=120,
|
|
||||||
)
|
|
||||||
if shutil.which("instaloader"):
|
|
||||||
print(" ✅ instaloader installed (Instagram reading)")
|
|
||||||
else:
|
|
||||||
print(" ⬜ instaloader install failed (optional — try: pip install instaloader)")
|
|
||||||
except Exception:
|
except Exception:
|
||||||
print(" ⬜ instaloader install failed (optional — try: pip install instaloader)")
|
print(" ⬜ undici install failed (optional — bird may not work behind proxies)")
|
||||||
|
|
||||||
|
|
||||||
def _install_system_deps_safe():
|
def _install_system_deps_safe():
|
||||||
@@ -451,7 +381,6 @@ def _install_system_deps_safe():
|
|||||||
("gh", ["gh"], "GitHub CLI", "https://cli.github.com — or: apt install gh / brew install gh"),
|
("gh", ["gh"], "GitHub CLI", "https://cli.github.com — or: apt install gh / brew install gh"),
|
||||||
("node", ["node", "npm"], "Node.js", "https://nodejs.org — or: apt install nodejs npm"),
|
("node", ["node", "npm"], "Node.js", "https://nodejs.org — or: apt install nodejs npm"),
|
||||||
("bird", ["bird", "birdx"], "bird CLI (Twitter)", "npm install -g @steipete/bird"),
|
("bird", ["bird", "birdx"], "bird CLI (Twitter)", "npm install -g @steipete/bird"),
|
||||||
("instaloader", ["instaloader"], "instaloader (Instagram)", "pip install instaloader"),
|
|
||||||
]
|
]
|
||||||
|
|
||||||
missing = []
|
missing = []
|
||||||
@@ -482,7 +411,6 @@ def _install_system_deps_dryrun():
|
|||||||
("gh CLI", ["gh"], "apt install gh / brew install gh"),
|
("gh CLI", ["gh"], "apt install gh / brew install gh"),
|
||||||
("Node.js", ["node"], "curl NodeSource setup | bash + apt install nodejs"),
|
("Node.js", ["node"], "curl NodeSource setup | bash + apt install nodejs"),
|
||||||
("bird CLI", ["bird", "birdx"], "npm install -g @steipete/bird"),
|
("bird CLI", ["bird", "birdx"], "npm install -g @steipete/bird"),
|
||||||
("instaloader", ["instaloader"], "pip install instaloader"),
|
|
||||||
]
|
]
|
||||||
|
|
||||||
for label, binaries, method in checks:
|
for label, binaries, method in checks:
|
||||||
@@ -751,9 +679,6 @@ def _cmd_configure(args):
|
|||||||
config.set("groq_api_key", value)
|
config.set("groq_api_key", value)
|
||||||
print(f"✅ Groq key configured!")
|
print(f"✅ Groq key configured!")
|
||||||
|
|
||||||
elif args.key == "instagram-cookies":
|
|
||||||
_configure_instagram_cookies(value)
|
|
||||||
|
|
||||||
|
|
||||||
def _cmd_doctor():
|
def _cmd_doctor():
|
||||||
from agent_reach.config import Config
|
from agent_reach.config import Config
|
||||||
@@ -774,30 +699,6 @@ def _parse_cookie_header(cookie_str: str) -> dict:
|
|||||||
return cookies
|
return cookies
|
||||||
|
|
||||||
|
|
||||||
def _configure_instagram_cookies(value: str):
|
|
||||||
"""Save Instagram cookies from Cookie-Editor Header String."""
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
cookies = _parse_cookie_header(value)
|
|
||||||
if "sessionid" not in cookies:
|
|
||||||
print("❌ Cookie 里缺少 sessionid。")
|
|
||||||
print(" 确保你已登录 Instagram,然后用 Cookie-Editor 导出 Header String。")
|
|
||||||
print(' 格式: agent-reach configure instagram-cookies "sessionid=xxx; csrftoken=yyy; ..."')
|
|
||||||
return
|
|
||||||
|
|
||||||
cookie_dir = Path.home() / ".agent-reach"
|
|
||||||
cookie_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
cookie_file = cookie_dir / "instagram-cookies.txt"
|
|
||||||
cookie_file.write_text(value.strip())
|
|
||||||
cookie_file.chmod(0o600)
|
|
||||||
|
|
||||||
print(f"✅ Instagram cookies 已保存!")
|
|
||||||
print(f" sessionid: {cookies['sessionid'][:8]}...")
|
|
||||||
if "csrftoken" in cookies:
|
|
||||||
print(f" csrftoken: ✅")
|
|
||||||
if "ds_user_id" in cookies:
|
|
||||||
print(f" ds_user_id: {cookies['ds_user_id']}")
|
|
||||||
print(f" 文件: {cookie_file}")
|
|
||||||
|
|
||||||
|
|
||||||
def _cmd_setup():
|
def _cmd_setup():
|
||||||
@@ -888,100 +789,6 @@ def _cmd_setup():
|
|||||||
print()
|
print()
|
||||||
|
|
||||||
|
|
||||||
async def _cmd_read(args):
|
|
||||||
from agent_reach.core import AgentReach
|
|
||||||
eyes = AgentReach()
|
|
||||||
try:
|
|
||||||
result = await eyes.read(args.url)
|
|
||||||
if args.as_json:
|
|
||||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
|
||||||
else:
|
|
||||||
print(f"\n📖 {result.get('title', 'Untitled')}")
|
|
||||||
print(f"🔗 {result.get('url', '')}")
|
|
||||||
if result.get("author"):
|
|
||||||
print(f"👤 {result['author']}")
|
|
||||||
print(f"\n{result.get('content', '')}")
|
|
||||||
except Exception as e:
|
|
||||||
error_str = str(e)
|
|
||||||
if "400" in error_str and "Bad Request" in error_str:
|
|
||||||
print(f"❌ Invalid URL: {args.url}", file=sys.stderr)
|
|
||||||
print(" Please provide a valid URL (e.g., https://example.com)", file=sys.stderr)
|
|
||||||
elif "ConnectionError" in type(e).__name__ or "Timeout" in type(e).__name__:
|
|
||||||
print(f"❌ Could not connect to: {args.url}", file=sys.stderr)
|
|
||||||
print(" Check your internet connection or the URL.", file=sys.stderr)
|
|
||||||
else:
|
|
||||||
print(f"❌ Error: {e}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
|
|
||||||
async def _cmd_search(args):
|
|
||||||
from agent_reach.core import AgentReach
|
|
||||||
eyes = AgentReach()
|
|
||||||
query = " ".join(args.query).strip()
|
|
||||||
num = args.num
|
|
||||||
|
|
||||||
if not query:
|
|
||||||
print("Please provide a search query.", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
try:
|
|
||||||
if args.command == "search":
|
|
||||||
results = await eyes.search(query, num_results=num)
|
|
||||||
elif args.command == "search-reddit":
|
|
||||||
results = await eyes.search_reddit(query, subreddit=getattr(args, "sub", None), limit=num)
|
|
||||||
elif args.command == "search-github":
|
|
||||||
results = await eyes.search_github(query, language=getattr(args, "lang", None), limit=num)
|
|
||||||
elif args.command == "search-twitter":
|
|
||||||
results = await eyes.search_twitter(query, limit=num)
|
|
||||||
elif args.command == "search-youtube":
|
|
||||||
results = await eyes.search_youtube(query, limit=num)
|
|
||||||
elif args.command == "search-bilibili":
|
|
||||||
results = await eyes.search_bilibili(query, limit=num)
|
|
||||||
elif args.command == "search-xhs":
|
|
||||||
results = await eyes.search_xhs(query, limit=num)
|
|
||||||
elif args.command == "search-instagram":
|
|
||||||
results = await eyes.search_instagram(query, limit=num)
|
|
||||||
elif args.command == "search-linkedin":
|
|
||||||
results = await eyes.search_linkedin(query, limit=num)
|
|
||||||
elif args.command == "search-bosszhipin":
|
|
||||||
results = await eyes.search_bosszhipin(query, limit=num)
|
|
||||||
else:
|
|
||||||
print(f"Unknown command: {args.command}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
except Exception as e:
|
|
||||||
error_str = str(e)
|
|
||||||
if "401" in error_str or "Unauthorized" in error_str:
|
|
||||||
print("⚠️ Exa API key not configured or invalid.")
|
|
||||||
print("Get a free key at https://exa.ai (1000 searches/month free)")
|
|
||||||
print("Then run: agent-reach configure exa-key YOUR_KEY")
|
|
||||||
sys.exit(1)
|
|
||||||
elif "exa" in error_str.lower() or "api_key" in error_str.lower():
|
|
||||||
print("⚠️ Exa API key not configured.")
|
|
||||||
print("Get a free key at https://exa.ai")
|
|
||||||
print("Then run: agent-reach configure exa-key YOUR_KEY")
|
|
||||||
sys.exit(1)
|
|
||||||
else:
|
|
||||||
print(f"❌ Error: {e}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
if not results:
|
|
||||||
print("No results found.")
|
|
||||||
return
|
|
||||||
|
|
||||||
for i, r in enumerate(results, 1):
|
|
||||||
title = r.get("title") or r.get("name") or r.get("text", "")[:60]
|
|
||||||
url = r.get("url", "")
|
|
||||||
snippet = r.get("snippet") or r.get("description") or r.get("text", "")
|
|
||||||
print(f"\n{i}. {title}")
|
|
||||||
print(f" 🔗 {url}")
|
|
||||||
if snippet:
|
|
||||||
print(f" {snippet[:200]}")
|
|
||||||
# Extra info for GitHub
|
|
||||||
extra = r.get("extra", {})
|
|
||||||
if extra.get("stars"):
|
|
||||||
print(f" ⭐ {extra['stars']} 🍴 {extra.get('forks', 0)} 📝 {extra.get('language', '')}")
|
|
||||||
|
|
||||||
|
|
||||||
def _cmd_check_update():
|
def _cmd_check_update():
|
||||||
"""Check for newer versions on GitHub."""
|
"""Check for newer versions on GitHub."""
|
||||||
import requests
|
import requests
|
||||||
|
|||||||
+16
-116
@@ -1,126 +1,36 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
AgentReach — the unified entry point.
|
AgentReach — installer, doctor, and configuration tool.
|
||||||
|
|
||||||
Pure glue: routes URLs to the right channel, routes searches to the right engine.
|
Agent Reach helps AI agents install and configure upstream platform tools
|
||||||
Every channel is a thin wrapper around an external tool. Swap any backend anytime.
|
(bird CLI, yt-dlp, mcporter, gh CLI, etc.). After installation, agents
|
||||||
|
call the upstream tools directly — no wrapper layer needed.
|
||||||
|
|
||||||
Usage:
|
Usage:
|
||||||
from agent_reach import AgentReach
|
from agent_reach.doctor import check_all, format_report
|
||||||
|
from agent_reach.config import Config
|
||||||
|
|
||||||
eyes = AgentReach()
|
config = Config()
|
||||||
content = await eyes.read("https://github.com/openai/gpt-4")
|
results = check_all(config)
|
||||||
results = await eyes.search("AI agent framework")
|
print(format_report(results))
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
from typing import Dict, Optional
|
||||||
from typing import Any, Dict, List, Optional
|
|
||||||
|
|
||||||
from agent_reach.config import Config
|
from agent_reach.config import Config
|
||||||
from agent_reach.channels import get_channel_for_url, get_channel, get_all_channels
|
|
||||||
|
|
||||||
|
|
||||||
class AgentReach:
|
class AgentReach:
|
||||||
"""Give your AI Agent eyes to see the entire internet."""
|
"""Give your AI Agent eyes to see the entire internet.
|
||||||
|
|
||||||
|
This class provides health-check functionality.
|
||||||
|
For reading/searching, use the upstream tools directly
|
||||||
|
(see SKILL.md for commands).
|
||||||
|
"""
|
||||||
|
|
||||||
def __init__(self, config: Optional[Config] = None):
|
def __init__(self, config: Optional[Config] = None):
|
||||||
self.config = config or Config()
|
self.config = config or Config()
|
||||||
|
|
||||||
# ── Reading ─────────────────────────────────────────
|
|
||||||
|
|
||||||
async def read(self, url: str) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Read content from any URL. Auto-detects platform.
|
|
||||||
|
|
||||||
Supported: Web, GitHub, Reddit, Twitter, YouTube,
|
|
||||||
Bilibili, RSS, and more.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict with title, content, url, author, platform, etc.
|
|
||||||
"""
|
|
||||||
if not url.startswith(("http://", "https://")):
|
|
||||||
url = f"https://{url}"
|
|
||||||
|
|
||||||
channel = get_channel_for_url(url)
|
|
||||||
result = await channel.read(url, config=self.config)
|
|
||||||
return result.to_dict()
|
|
||||||
|
|
||||||
async def read_batch(self, urls: List[str]) -> List[Dict[str, Any]]:
|
|
||||||
"""Read multiple URLs concurrently."""
|
|
||||||
tasks = [self.read(url) for url in urls]
|
|
||||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
|
||||||
return [r for r in results if not isinstance(r, Exception)]
|
|
||||||
|
|
||||||
def detect_platform(self, url: str) -> str:
|
|
||||||
"""Detect what platform a URL belongs to."""
|
|
||||||
channel = get_channel_for_url(url)
|
|
||||||
return channel.name
|
|
||||||
|
|
||||||
# ── Searching ───────────────────────────────────────
|
|
||||||
|
|
||||||
async def search(self, query: str, num_results: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Semantic web search via Exa."""
|
|
||||||
ch = get_channel("exa_search")
|
|
||||||
results = await ch.search(query, config=self.config, limit=num_results)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_reddit(self, query: str, subreddit: Optional[str] = None, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Reddit via Exa (bypasses IP blocks)."""
|
|
||||||
ch = get_channel("exa_search")
|
|
||||||
q = f"site:reddit.com/r/{subreddit} {query}" if subreddit else f"site:reddit.com {query}"
|
|
||||||
results = await ch.search(q, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_github(self, query: str, language: Optional[str] = None, limit: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Search GitHub repositories."""
|
|
||||||
ch = get_channel("github")
|
|
||||||
results = await ch.search(query, config=self.config, language=language, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_twitter(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Twitter. Uses bird CLI if available, else Exa."""
|
|
||||||
ch = get_channel("twitter")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_youtube(self, query: str, limit: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Search YouTube via yt-dlp."""
|
|
||||||
ch = get_channel("youtube")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_bilibili(self, query: str, limit: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Bilibili. Tries yt-dlp first, falls back to Exa."""
|
|
||||||
ch = get_channel("bilibili")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_xhs(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search XiaoHongShu via mcporter."""
|
|
||||||
ch = get_channel("xiaohongshu")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_instagram(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Instagram via Exa."""
|
|
||||||
ch = get_channel("instagram")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_linkedin(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search LinkedIn via MCP or Exa."""
|
|
||||||
ch = get_channel("linkedin")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
async def search_bosszhipin(self, query: str, limit: int = 10) -> List[Dict[str, Any]]:
|
|
||||||
"""Search Boss直聘 via MCP or Exa."""
|
|
||||||
ch = get_channel("bosszhipin")
|
|
||||||
results = await ch.search(query, config=self.config, limit=limit)
|
|
||||||
return [r.to_dict() for r in results]
|
|
||||||
|
|
||||||
# ── Health ──────────────────────────────────────────
|
|
||||||
|
|
||||||
def doctor(self) -> Dict[str, dict]:
|
def doctor(self) -> Dict[str, dict]:
|
||||||
"""Check all channel availability."""
|
"""Check all channel availability."""
|
||||||
from agent_reach.doctor import check_all
|
from agent_reach.doctor import check_all
|
||||||
@@ -130,13 +40,3 @@ class AgentReach:
|
|||||||
"""Get formatted health report."""
|
"""Get formatted health report."""
|
||||||
from agent_reach.doctor import check_all, format_report
|
from agent_reach.doctor import check_all, format_report
|
||||||
return format_report(check_all(self.config))
|
return format_report(check_all(self.config))
|
||||||
|
|
||||||
# ── Sync wrappers ───────────────────────────────────
|
|
||||||
|
|
||||||
def read_sync(self, url: str) -> Dict[str, Any]:
|
|
||||||
"""Synchronous version of read()."""
|
|
||||||
return asyncio.run(self.read(url))
|
|
||||||
|
|
||||||
def search_sync(self, query: str, num_results: int = 5) -> List[Dict[str, Any]]:
|
|
||||||
"""Synchronous version of search()."""
|
|
||||||
return asyncio.run(self.search(query, num_results))
|
|
||||||
|
|||||||
@@ -74,4 +74,18 @@ def format_report(results: Dict[str, dict]) -> str:
|
|||||||
if ok_count < total:
|
if ok_count < total:
|
||||||
lines.append("运行 `agent-reach setup` 解锁更多渠道")
|
lines.append("运行 `agent-reach setup` 解锁更多渠道")
|
||||||
|
|
||||||
|
# Security check: config file permissions
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
config_path = Config.CONFIG_DIR / "config.yaml"
|
||||||
|
if config_path.exists():
|
||||||
|
try:
|
||||||
|
mode = config_path.stat().st_mode
|
||||||
|
if mode & (stat.S_IRGRP | stat.S_IROTH):
|
||||||
|
lines.append("")
|
||||||
|
lines.append("⚠️ 安全提示:config.yaml 权限过宽(其他用户可读)")
|
||||||
|
lines.append(" 修复:chmod 600 ~/.agent-reach/config.yaml")
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|||||||
@@ -22,8 +22,8 @@ mcporter config add exa https://mcp.exa.ai/mcp
|
|||||||
|
|
||||||
### 3. 验证
|
### 3. 验证
|
||||||
```bash
|
```bash
|
||||||
agent-reach doctor | grep "Web Search"
|
agent-reach doctor | grep "Search"
|
||||||
agent-reach search "test" -n 1
|
mcporter call 'exa.web_search_exa(query: "test", numResults: 1)'
|
||||||
```
|
```
|
||||||
|
|
||||||
## 需要用户手动做的步骤
|
## 需要用户手动做的步骤
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ playwright install chromium
|
|||||||
|
|
||||||
3. 安装完成后测试:
|
3. 安装完成后测试:
|
||||||
```bash
|
```bash
|
||||||
agent-reach read "https://mp.weixin.qq.com/s/一个测试链接"
|
curl -s "https://r.jina.ai/https://mp.weixin.qq.com/s/一个测试链接" -H "Accept: text/markdown"
|
||||||
```
|
```
|
||||||
|
|
||||||
## 需要用户手动做的步骤
|
## 需要用户手动做的步骤
|
||||||
|
|||||||
@@ -1,10 +1,11 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
"""
|
"""
|
||||||
Agent Reach MCP Server — expose all capabilities as MCP tools.
|
Agent Reach MCP Server — expose doctor/status as MCP tool.
|
||||||
|
|
||||||
Run: python -m agent_reach.integrations.mcp_server
|
Run: python -m agent_reach.integrations.mcp_server
|
||||||
|
|
||||||
8 tools for any MCP-compatible AI Agent.
|
Agent Reach is an installer + doctor tool. For actual reading/searching,
|
||||||
|
agents should call upstream tools directly (bird, yt-dlp, mcporter, etc.).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
@@ -35,50 +36,15 @@ def create_server():
|
|||||||
@server.list_tools()
|
@server.list_tools()
|
||||||
async def list_tools():
|
async def list_tools():
|
||||||
return [
|
return [
|
||||||
Tool(name="read_url",
|
|
||||||
description="Read content from any URL. Supports: web, GitHub, Reddit, Twitter, YouTube, Bilibili, RSS.",
|
|
||||||
inputSchema={"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}),
|
|
||||||
Tool(name="read_batch",
|
|
||||||
description="Read multiple URLs concurrently.",
|
|
||||||
inputSchema={"type": "object", "properties": {"urls": {"type": "array", "items": {"type": "string"}}}, "required": ["urls"]}),
|
|
||||||
Tool(name="detect_platform",
|
|
||||||
description="Detect what platform a URL belongs to.",
|
|
||||||
inputSchema={"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}),
|
|
||||||
Tool(name="search",
|
|
||||||
description="Semantic web search via Exa.",
|
|
||||||
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "num_results": {"type": "integer", "default": 5}}, "required": ["query"]}),
|
|
||||||
Tool(name="search_reddit",
|
|
||||||
description="Search Reddit posts.",
|
|
||||||
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "subreddit": {"type": "string"}, "limit": {"type": "integer", "default": 10}}, "required": ["query"]}),
|
|
||||||
Tool(name="search_github",
|
|
||||||
description="Search GitHub repositories.",
|
|
||||||
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "language": {"type": "string"}, "limit": {"type": "integer", "default": 5}}, "required": ["query"]}),
|
|
||||||
Tool(name="search_twitter",
|
|
||||||
description="Search Twitter/X posts.",
|
|
||||||
inputSchema={"type": "object", "properties": {"query": {"type": "string"}, "limit": {"type": "integer", "default": 10}}, "required": ["query"]}),
|
|
||||||
Tool(name="get_status",
|
Tool(name="get_status",
|
||||||
description="Get Agent Reach status: which channels are active.",
|
description="Get Agent Reach status: which channels are installed and active.",
|
||||||
inputSchema={"type": "object", "properties": {}}),
|
inputSchema={"type": "object", "properties": {}}),
|
||||||
]
|
]
|
||||||
|
|
||||||
@server.call_tool()
|
@server.call_tool()
|
||||||
async def call_tool(name: str, arguments: dict):
|
async def call_tool(name: str, arguments: dict):
|
||||||
try:
|
try:
|
||||||
if name == "read_url":
|
if name == "get_status":
|
||||||
result = await eyes.read(arguments["url"])
|
|
||||||
elif name == "read_batch":
|
|
||||||
result = await eyes.read_batch(arguments["urls"])
|
|
||||||
elif name == "detect_platform":
|
|
||||||
result = eyes.detect_platform(arguments["url"])
|
|
||||||
elif name == "search":
|
|
||||||
result = await eyes.search(arguments["query"], arguments.get("num_results", 5))
|
|
||||||
elif name == "search_reddit":
|
|
||||||
result = await eyes.search_reddit(arguments["query"], arguments.get("subreddit"), arguments.get("limit", 10))
|
|
||||||
elif name == "search_github":
|
|
||||||
result = await eyes.search_github(arguments["query"], arguments.get("language"), arguments.get("limit", 5))
|
|
||||||
elif name == "search_twitter":
|
|
||||||
result = await eyes.search_twitter(arguments["query"], arguments.get("limit", 10))
|
|
||||||
elif name == "get_status":
|
|
||||||
result = eyes.doctor_report()
|
result = eyes.doctor_report()
|
||||||
else:
|
else:
|
||||||
result = f"Unknown tool: {name}"
|
result = f"Unknown tool: {name}"
|
||||||
|
|||||||
@@ -1,65 +0,0 @@
|
|||||||
---
|
|
||||||
name: agent-reach
|
|
||||||
description: "Give your AI Agent eyes to see the entire internet. Search + read 10+ platforms with one install."
|
|
||||||
install: pip install git+https://github.com/Panniantong/agent-reach.git
|
|
||||||
---
|
|
||||||
|
|
||||||
# Agent Reach — 给 Agent 装上眼睛 👁️
|
|
||||||
|
|
||||||
安装后你就能搜索和阅读全网内容。
|
|
||||||
|
|
||||||
## 安装
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install git+https://github.com/Panniantong/agent-reach.git
|
|
||||||
```
|
|
||||||
|
|
||||||
## 装完直接能用的(零配置)
|
|
||||||
|
|
||||||
- `agent-reach read <URL>` — 读取任意网页、GitHub、B站、YouTube、RSS
|
|
||||||
- `agent-reach search-github "query"` — 搜索 GitHub 仓库
|
|
||||||
- `agent-reach doctor` — 查看所有平台状态
|
|
||||||
|
|
||||||
## 解锁更多功能
|
|
||||||
|
|
||||||
运行 `agent-reach configure` 进行交互式配置,或者读取以下指南:
|
|
||||||
|
|
||||||
| 功能 | 指南文件 |
|
|
||||||
|------|---------|
|
|
||||||
| 全网搜索 + Reddit搜索 + Twitter搜索 | `agent_reach/guides/setup-exa.md` |
|
|
||||||
| Reddit 完整阅读 | `agent_reach/guides/setup-reddit.md` |
|
|
||||||
| Twitter 高级(时间线/线程) | `agent_reach/guides/setup-twitter.md` |
|
|
||||||
| 小红书 | `agent_reach/guides/setup-xiaohongshu.md` |
|
|
||||||
| 微信公众号 | `agent_reach/guides/setup-wechat.md` |
|
|
||||||
| 视频语音转文字 | `agent_reach/guides/setup-groq.md` |
|
|
||||||
|
|
||||||
每个指南里有详细的步骤说明,你(Agent)可以照着做,只有需要人类操作的部分(登录、复制 key)才需要问用户。
|
|
||||||
|
|
||||||
## MCP Server
|
|
||||||
|
|
||||||
如果你的 Agent 平台支持 MCP:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install agent-reach[mcp]
|
|
||||||
python -m agent_reach.integrations.mcp_server
|
|
||||||
```
|
|
||||||
|
|
||||||
提供 8 个工具:read_url, read_batch, detect_platform, search, search_reddit, search_github, search_twitter, get_status
|
|
||||||
|
|
||||||
## Python API
|
|
||||||
|
|
||||||
```python
|
|
||||||
from agent_reach import AgentReach
|
|
||||||
import asyncio
|
|
||||||
|
|
||||||
eyes = AgentReach()
|
|
||||||
|
|
||||||
# 读取
|
|
||||||
result = asyncio.run(eyes.read("https://github.com/openai/gpt-4"))
|
|
||||||
|
|
||||||
# 搜索
|
|
||||||
results = asyncio.run(eyes.search("AI agent framework"))
|
|
||||||
|
|
||||||
# 健康检查
|
|
||||||
print(eyes.doctor_report())
|
|
||||||
```
|
|
||||||
+224
-55
@@ -1,87 +1,256 @@
|
|||||||
---
|
---
|
||||||
name: agent-reach
|
name: agent-reach
|
||||||
description: >
|
description: >
|
||||||
Give your AI agent eyes to see the entire internet. Read and search across
|
Give your AI agent eyes to see the entire internet. Install and configure
|
||||||
Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, RSS, and any web page
|
upstream tools for Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu,
|
||||||
— all from a single CLI. Use when: (1) reading content from URLs (tweets, Reddit posts,
|
Douyin, LinkedIn, Boss直聘, RSS, and any web page — then call them directly.
|
||||||
articles, videos), (2) searching across platforms (web, Twitter, Reddit, GitHub, YouTube,
|
Use when: (1) setting up platform access tools for the first time,
|
||||||
Bilibili, XiaoHongShu), (3) checking channel health or updating Agent Reach.
|
(2) checking which platforms are available,
|
||||||
Triggers: "search Twitter/Reddit/YouTube", "read this URL", "find posts about",
|
(3) user asks to configure/enable a platform channel.
|
||||||
"搜索", "读取", "查一下", "看看这个链接".
|
Triggers: "帮我配", "帮我添加", "帮我安装", "agent reach", "install channels",
|
||||||
|
"configure twitter", "enable reddit".
|
||||||
---
|
---
|
||||||
|
|
||||||
# Agent Reach
|
# Agent Reach
|
||||||
|
|
||||||
Read and search the internet across 9+ platforms via unified CLI.
|
Install and configure upstream tools for 12+ platforms. After setup, call them directly — no wrapper layer.
|
||||||
|
|
||||||
## Setup
|
## Setup
|
||||||
|
|
||||||
First check if agent-reach is installed:
|
|
||||||
```bash
|
|
||||||
agent-reach doctor
|
|
||||||
```
|
|
||||||
|
|
||||||
If command not found, install it:
|
|
||||||
```bash
|
```bash
|
||||||
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
||||||
agent-reach install --env=auto
|
agent-reach install --env=auto
|
||||||
|
agent-reach doctor
|
||||||
```
|
```
|
||||||
|
|
||||||
`install` auto-detects your environment and installs all dependencies (Node.js, mcporter, bird CLI, gh CLI). Read the output and run `agent-reach doctor` to see what's active.
|
`install` auto-detects your environment and installs core dependencies (Node.js, mcporter, bird CLI, gh CLI, yt-dlp, feedparser). Run `doctor` to see what's active.
|
||||||
|
|
||||||
For channels that need user input, ask the user. See the full setup guide:
|
## Management
|
||||||
https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
|
||||||
|
|
||||||
## Commands
|
|
||||||
|
|
||||||
### Read any URL
|
|
||||||
```bash
|
|
||||||
agent-reach read <url>
|
|
||||||
agent-reach read <url> --json # structured output
|
|
||||||
```
|
|
||||||
Handles: tweets, Reddit posts, articles, YouTube (transcripts), GitHub repos, etc.
|
|
||||||
|
|
||||||
### Search
|
|
||||||
|
|
||||||
```bash
|
|
||||||
agent-reach search "query" # web search (Exa)
|
|
||||||
agent-reach search-twitter "query" # Twitter/X
|
|
||||||
agent-reach search-reddit "query" # Reddit (--sub <subreddit>)
|
|
||||||
agent-reach search-github "query" # GitHub (--lang <language>)
|
|
||||||
agent-reach search-youtube "query" # YouTube
|
|
||||||
agent-reach search-bilibili "query" # Bilibili (B站)
|
|
||||||
agent-reach search-xhs "query" # XiaoHongShu (小红书)
|
|
||||||
```
|
|
||||||
|
|
||||||
All search commands support `-n <count>` for number of results.
|
|
||||||
|
|
||||||
### Management
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
agent-reach doctor # channel status overview
|
agent-reach doctor # channel status overview
|
||||||
agent-reach watch # quick health + update check (for scheduled tasks)
|
agent-reach watch # quick health + update check
|
||||||
agent-reach check-update # check for new versions
|
agent-reach check-update # check for new versions
|
||||||
```
|
```
|
||||||
|
|
||||||
### Configure channels
|
## Configure channels
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
|
agent-reach configure twitter-cookies "auth_token=xxx; ct0=yyy"
|
||||||
agent-reach configure proxy http://user:pass@ip:port
|
agent-reach configure proxy http://user:pass@ip:port
|
||||||
agent-reach configure --from-browser chrome # auto-extract cookies
|
agent-reach configure --from-browser chrome # auto-extract cookies from local browser
|
||||||
```
|
```
|
||||||
|
|
||||||
## Channel Status Tiers
|
## Configuring a channel ("帮我配 XXX")
|
||||||
|
|
||||||
- **Tier 0 (zero config):** Web, YouTube, RSS, Twitter (read-only via Jina)
|
When a user asks to configure/enable any channel:
|
||||||
- **Tier 1 (free setup):** Exa web search (mcporter required)
|
|
||||||
- **Tier 2 (user config):** Twitter search (cookie), Reddit full (proxy), GitHub (token), Bilibili (proxy), XiaoHongShu (MCP)
|
|
||||||
|
|
||||||
Run `agent-reach doctor` to see which channels are active.
|
1. Run `agent-reach doctor`
|
||||||
|
2. Find the channel — it shows status (✅/⚠️/⬜) and **what to do next**
|
||||||
|
3. Execute what you can automatically (install packages, start services)
|
||||||
|
4. For human-required steps (paste cookies), tell the user what to do
|
||||||
|
5. Run `agent-reach doctor` again to verify
|
||||||
|
|
||||||
## Tips
|
**Do NOT memorize per-channel steps.** Always rely on `doctor` output.
|
||||||
|
|
||||||
- Always try `agent-reach read <url>` first for any URL — it auto-detects the platform
|
### Cookie 导入(所有需要登录的平台通用)
|
||||||
- For Twitter cookies, recommend the user install [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome extension
|
|
||||||
- Reddit and Bilibili block server IPs — suggest a residential proxy (~$1/month) if on a server
|
所有需要 Cookie 的平台(Twitter、小红书等),**优先使用 Cookie-Editor 导入**:
|
||||||
- If a channel breaks, run `agent-reach doctor` to diagnose
|
|
||||||
|
1. 用户在自己的浏览器登录对应平台
|
||||||
|
2. 安装 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome 插件
|
||||||
|
3. 点击插件 → Export → Header String
|
||||||
|
4. 把导出的字符串发给 Agent
|
||||||
|
|
||||||
|
本地电脑用户也可以用 `agent-reach configure --from-browser chrome` 一键自动提取。
|
||||||
|
|
||||||
|
扫码登录是备选方案(仅限本地有浏览器的情况),Cookie-Editor 更简单可靠。
|
||||||
|
|
||||||
|
### Other human actions
|
||||||
|
|
||||||
|
- **Proxy:** Reddit/Bilibili/XiaoHongShu may block server IPs — suggest a residential proxy if on a server
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Using Upstream Tools Directly
|
||||||
|
|
||||||
|
After `agent-reach install`, call the upstream tools directly. No need for `agent-reach read` or `agent-reach search`.
|
||||||
|
|
||||||
|
### Twitter/X (bird CLI)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Search tweets
|
||||||
|
bird search "query" --json -n 10
|
||||||
|
|
||||||
|
# Read a specific tweet
|
||||||
|
bird read https://x.com/user/status/123 --json
|
||||||
|
|
||||||
|
# Read a user's timeline
|
||||||
|
bird timeline @username --json -n 20
|
||||||
|
```
|
||||||
|
|
||||||
|
### YouTube (yt-dlp)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Get video metadata
|
||||||
|
yt-dlp --dump-json "https://www.youtube.com/watch?v=xxx"
|
||||||
|
|
||||||
|
# Download subtitles only
|
||||||
|
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --skip-download -o "/tmp/%(id)s" "URL"
|
||||||
|
# Then read the .vtt file
|
||||||
|
|
||||||
|
# Search (yt-dlp ytsearch)
|
||||||
|
yt-dlp --dump-json "ytsearch5:query"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Bilibili (yt-dlp)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Get video metadata
|
||||||
|
yt-dlp --dump-json "https://www.bilibili.com/video/BVxxx"
|
||||||
|
|
||||||
|
# Download subtitles
|
||||||
|
yt-dlp --write-sub --write-auto-sub --sub-lang "zh-Hans,zh,en" --convert-subs vtt --skip-download -o "/tmp/%(id)s" "URL"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Reddit (JSON API)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Read a subreddit
|
||||||
|
curl -s "https://www.reddit.com/r/python/hot.json?limit=10" -H "User-Agent: agent-reach/1.0"
|
||||||
|
|
||||||
|
# Read a post with comments
|
||||||
|
curl -s "https://www.reddit.com/r/python/comments/POST_ID.json" -H "User-Agent: agent-reach/1.0"
|
||||||
|
|
||||||
|
# Search
|
||||||
|
curl -s "https://www.reddit.com/search.json?q=query&limit=10" -H "User-Agent: agent-reach/1.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
Note: On servers, Reddit may block your IP. Use proxy or search via Exa instead.
|
||||||
|
|
||||||
|
### 小红书 / XiaoHongShu (mcporter + xiaohongshu-mcp)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Search notes
|
||||||
|
mcporter call 'xiaohongshu.search_feeds(keyword: "query")'
|
||||||
|
|
||||||
|
# Read a note
|
||||||
|
mcporter call 'xiaohongshu.get_feed_detail(feed_id: "xxx", xsec_token: "yyy")'
|
||||||
|
|
||||||
|
# Get comments
|
||||||
|
mcporter call 'xiaohongshu.get_feed_comments(feed_id: "xxx", xsec_token: "yyy")'
|
||||||
|
|
||||||
|
# Post a note
|
||||||
|
mcporter call 'xiaohongshu.create_image_feed(title: "标题", desc: "内容", image_paths: ["/path/to/img.jpg"])'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 抖音 / Douyin (mcporter + douyin-mcp-server)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 解析抖音视频信息(分享链接 → 标题、作者、无水印视频URL等)
|
||||||
|
mcporter call 'douyin.parse_douyin_video_info(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
|
||||||
|
# 获取无水印视频下载链接
|
||||||
|
mcporter call 'douyin.get_douyin_download_link(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
|
||||||
|
# AI 提取视频语音文案(需要配置硅基流动 API Key)
|
||||||
|
mcporter call 'douyin.extract_douyin_text(share_link: "https://v.douyin.com/xxx/")'
|
||||||
|
```
|
||||||
|
|
||||||
|
> 无需登录即可解析视频。支持抖音分享链接和直接链接。
|
||||||
|
|
||||||
|
### GitHub (gh CLI)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Search repos
|
||||||
|
gh search repos "query" --sort stars --limit 10
|
||||||
|
|
||||||
|
# View a repo
|
||||||
|
gh repo view owner/repo
|
||||||
|
|
||||||
|
# Search code
|
||||||
|
gh search code "query" --language python
|
||||||
|
|
||||||
|
# List issues
|
||||||
|
gh issue list -R owner/repo --state open
|
||||||
|
|
||||||
|
# View a specific issue/PR
|
||||||
|
gh issue view 123 -R owner/repo
|
||||||
|
```
|
||||||
|
|
||||||
|
### Web — Any URL (Jina Reader)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Read any webpage as markdown
|
||||||
|
curl -s "https://r.jina.ai/URL" -H "Accept: text/markdown"
|
||||||
|
|
||||||
|
# Search the web
|
||||||
|
curl -s "https://s.jina.ai/query" -H "Accept: text/markdown"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Exa Search (mcporter + exa MCP)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Web search
|
||||||
|
mcporter call 'exa.web_search_exa(query: "query", numResults: 5)'
|
||||||
|
|
||||||
|
# Code search (GitHub, StackOverflow, docs)
|
||||||
|
mcporter call 'exa.get_code_context_exa(query: "how to parse JSON in Python", tokensNum: 3000)'
|
||||||
|
|
||||||
|
# Company research
|
||||||
|
mcporter call 'exa.company_research_exa(companyName: "OpenAI")'
|
||||||
|
```
|
||||||
|
|
||||||
|
### LinkedIn (mcporter + linkedin-scraper-mcp)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# View a profile
|
||||||
|
mcporter call 'linkedin.get_person_profile(linkedin_url: "https://linkedin.com/in/username")'
|
||||||
|
|
||||||
|
# Search people
|
||||||
|
mcporter call 'linkedin.search_people(keyword: "AI engineer", limit: 10)'
|
||||||
|
|
||||||
|
# View company
|
||||||
|
mcporter call 'linkedin.get_company_profile(linkedin_url: "https://linkedin.com/company/xxx")'
|
||||||
|
```
|
||||||
|
|
||||||
|
Fallback: `curl -s "https://r.jina.ai/https://linkedin.com/in/username"`
|
||||||
|
|
||||||
|
### Boss直聘 (mcporter + mcp-bosszp)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Browse recommended jobs
|
||||||
|
mcporter call 'bosszhipin.get_recommend_jobs_tool(page: 1)'
|
||||||
|
|
||||||
|
# Search jobs
|
||||||
|
mcporter call 'bosszhipin.search_jobs_tool(keyword: "Python", city: "北京", page: 1)'
|
||||||
|
|
||||||
|
# View job details
|
||||||
|
mcporter call 'bosszhipin.get_job_detail_tool(job_url: "https://www.zhipin.com/job_detail/xxx")'
|
||||||
|
```
|
||||||
|
|
||||||
|
Fallback: `curl -s "https://r.jina.ai/https://www.zhipin.com/job_detail/xxx"`
|
||||||
|
|
||||||
|
### RSS (feedparser)
|
||||||
|
|
||||||
|
```python
|
||||||
|
python3 -c "
|
||||||
|
import feedparser
|
||||||
|
d = feedparser.parse('https://example.com/feed')
|
||||||
|
for e in d.entries[:5]:
|
||||||
|
print(f'{e.title} — {e.link}')
|
||||||
|
"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Twitter "fetch failed"
|
||||||
|
|
||||||
|
bird CLI uses Node.js native `fetch()`, which doesn't respect `HTTP_PROXY`. Solutions:
|
||||||
|
1. Ensure `undici` is installed: `npm install -g undici`
|
||||||
|
2. Configure proxy: `agent-reach configure proxy http://user:pass@ip:port`
|
||||||
|
3. If still failing, use transparent proxy (Clash TUN, Proxifier)
|
||||||
|
|
||||||
|
### Channel broken?
|
||||||
|
|
||||||
|
Run `agent-reach doctor` — it shows what's wrong and how to fix it.
|
||||||
|
|||||||
+73
-15
@@ -58,9 +58,9 @@ Copy that to your Agent. A few minutes later, it can read tweets, search Reddit,
|
|||||||
| 🌐 **Web** | Read | Zero config | Any URL → clean Markdown ([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) |
|
| 🌐 **Web** | Read | Zero config | Any URL → clean Markdown ([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) |
|
||||||
| 🐦 **Twitter/X** | Read · Search | Zero config / Cookie | Single tweets readable out of the box. Cookie unlocks search, timeline, posting ([bird](https://github.com/steipete/bird)) |
|
| 🐦 **Twitter/X** | Read · Search | Zero config / Cookie | Single tweets readable out of the box. Cookie unlocks search, timeline, posting ([bird](https://github.com/steipete/bird)) |
|
||||||
| 📕 **XiaoHongShu** | Read · Search · **Post · Comment · Like** | mcporter | Via [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) internal API, install and go |
|
| 📕 **XiaoHongShu** | Read · Search · **Post · Comment · Like** | mcporter | Via [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) internal API, install and go |
|
||||||
| 📷 **Instagram** | Read · Search | instaloader | Posts, profiles, hashtags ([instaloader](https://github.com/instaloader/instaloader) ⭐9.8K) |
|
| 🎵 **Douyin** | Video parsing · Watermark-free download | mcporter | Via [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server), no login needed |
|
||||||
| 💼 **LinkedIn** | Read · Search | mcporter / Jina | Profiles, companies, job search ([linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) ⭐900+) |
|
| 💼 **LinkedIn** | Jina Reader (public pages) | Full profiles, companies, job search | Tell your Agent "help me set up LinkedIn" |
|
||||||
| 🏢 **Boss直聘** | Read · Search | mcporter / Jina | Job search, greet recruiters ([mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)) |
|
| 🏢 **Boss直聘** | Jina Reader (job pages) | Job search, greet recruiters | Tell your Agent "help me set up Boss直聘" |
|
||||||
| 🔍 **Web Search** | Search | Auto-configured | Auto-configured during install, free, no API key ([Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter)) |
|
| 🔍 **Web Search** | Search | Auto-configured | Auto-configured during install, free, no API key ([Exa](https://exa.ai) via [mcporter](https://github.com/nicepkg/mcporter)) |
|
||||||
| 📦 **GitHub** | Read · Search | Zero config | [gh CLI](https://cli.github.com) powered. Public repos work immediately. `gh auth login` unlocks Fork, Issue, PR |
|
| 📦 **GitHub** | Read · Search | Zero config | [gh CLI](https://cli.github.com) powered. Public repos work immediately. `gh auth login` unlocks Fork, Issue, PR |
|
||||||
| 📺 **YouTube** | Read · **Search** | Zero config | Subtitles + search across 1800+ video sites ([yt-dlp](https://github.com/yt-dlp/yt-dlp) ⭐148K) |
|
| 📺 **YouTube** | Read · **Search** | Zero config | Subtitles + search across 1800+ video sites ([yt-dlp](https://github.com/yt-dlp/yt-dlp) ⭐148K) |
|
||||||
@@ -109,14 +109,14 @@ After the Skill is installed, the Agent will auto-detect whether `agent-reach` C
|
|||||||
|
|
||||||
No configuration needed — just tell your Agent:
|
No configuration needed — just tell your Agent:
|
||||||
|
|
||||||
- "Read this link" → any web page
|
- "Read this link" → `curl https://r.jina.ai/URL` for any web page
|
||||||
- "What's this GitHub repo about?" → repos, issues, code
|
- "What's this GitHub repo about?" → `gh repo view owner/repo`
|
||||||
- "What does this video cover?" → YouTube / Bilibili subtitles
|
- "What does this video cover?" → `yt-dlp --dump-json URL` for subtitles
|
||||||
- "Read this tweet" → Twitter posts
|
- "Read this tweet" → `bird read URL --json`
|
||||||
- "Subscribe to this RSS" → RSS / Atom feeds
|
- "Subscribe to this RSS" → `feedparser` to parse feeds
|
||||||
- "Search GitHub for LLM frameworks" → GitHub search
|
- "Search GitHub for LLM frameworks" → `gh search repos "LLM framework"`
|
||||||
|
|
||||||
**No commands to remember.** The Agent knows what to call.
|
**No commands to remember.** The Agent reads SKILL.md and knows what to call.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -172,9 +172,11 @@ Every time you spin up a new Agent, you spend time finding tools, installing dep
|
|||||||
|
|
||||||
Agent Reach does one simple thing: **it makes those tool selection and configuration decisions for you.**
|
Agent Reach does one simple thing: **it makes those tool selection and configuration decisions for you.**
|
||||||
|
|
||||||
|
After installation, your Agent calls the upstream tools directly (bird CLI, yt-dlp, mcporter, gh CLI, etc.) — no wrapper layer in between.
|
||||||
|
|
||||||
### 🔌 Every Channel is Pluggable
|
### 🔌 Every Channel is Pluggable
|
||||||
|
|
||||||
Each platform is a single Python file implementing a unified interface. **Backends can be swapped anytime** — when a better tool comes along, change one file and nothing else breaks.
|
Each platform maps to an upstream tool. **Don't like one? Swap it out.**
|
||||||
|
|
||||||
```
|
```
|
||||||
channels/
|
channels/
|
||||||
@@ -185,14 +187,16 @@ channels/
|
|||||||
├── bilibili.py → yt-dlp ← swap to bilibili-api…
|
├── bilibili.py → yt-dlp ← swap to bilibili-api…
|
||||||
├── reddit.py → JSON API + Exa ← swap to PRAW, Pushshift…
|
├── reddit.py → JSON API + Exa ← swap to PRAW, Pushshift…
|
||||||
├── xiaohongshu.py → mcporter MCP ← swap to other XHS tools…
|
├── xiaohongshu.py → mcporter MCP ← swap to other XHS tools…
|
||||||
├── instagram.py → instaloader ← swap to instagrapi, official API…
|
├── douyin.py → mcporter MCP ← swap to other Douyin tools…
|
||||||
├── linkedin.py → linkedin-mcp ← swap to LinkedIn API…
|
├── linkedin.py → linkedin-mcp ← swap to LinkedIn API…
|
||||||
├── bosszhipin.py → mcp-bosszp ← swap to other job tools…
|
├── bosszhipin.py → mcp-bosszp ← swap to other job tools…
|
||||||
├── rss.py → feedparser ← swap to atoma…
|
├── rss.py → feedparser ← swap to atoma…
|
||||||
├── exa_search.py → mcporter MCP ← swap to Tavily, SerpAPI…
|
├── exa_search.py → mcporter MCP ← swap to Tavily, SerpAPI…
|
||||||
└── __init__.py → Channel registry
|
└── __init__.py → Channel registry (for doctor checks)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Each channel file only checks whether its upstream tool is installed and working (`check()` method for `agent-reach doctor`). The actual reading and searching is done by calling the upstream tools directly.
|
||||||
|
|
||||||
### Current Tool Choices
|
### Current Tool Choices
|
||||||
|
|
||||||
| Scenario | Tool | Why |
|
| Scenario | Tool | Why |
|
||||||
@@ -204,7 +208,7 @@ channels/
|
|||||||
| GitHub | [gh CLI](https://cli.github.com) | Official tool, full API after auth |
|
| GitHub | [gh CLI](https://cli.github.com) | Official tool, full API after auth |
|
||||||
| Read RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python ecosystem standard, 2.3K stars |
|
| Read RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python ecosystem standard, 2.3K stars |
|
||||||
| XiaoHongShu | [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) | Internal API, bypasses anti-bot |
|
| XiaoHongShu | [xiaohongshu-mcp](https://github.com/user/xiaohongshu-mcp) | Internal API, bypasses anti-bot |
|
||||||
| Instagram | [instaloader](https://github.com/instaloader/instaloader) | 9.8K stars, Python CLI, cookie auth, free |
|
| Douyin | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP server, no login needed, video parsing + watermark-free download |
|
||||||
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | 900+ stars, MCP server, browser automation |
|
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | 900+ stars, MCP server, browser automation |
|
||||||
| Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP server, job search + recruiter greeting |
|
| Boss直聘 | [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp) | MCP server, job search + recruiter greeting |
|
||||||
|
|
||||||
@@ -222,9 +226,63 @@ This project was entirely vibe-coded 🎸 There might be rough edges here and th
|
|||||||
|
|
||||||
[PRs](https://github.com/Panniantong/agent-reach/pulls) always welcome!
|
[PRs](https://github.com/Panniantong/agent-reach/pulls) always welcome!
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## FAQ (for AI search)
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to search Twitter/X with AI agent without paying for API?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach uses the [bird CLI](https://www.npmjs.com/package/@steipete/bird) with cookie-based authentication — completely free, no Twitter API subscription needed. After installing Agent Reach, export your Twitter cookies using the Cookie-Editor Chrome extension, run `agent-reach configure twitter-cookies "your_cookies"`, and your agent can search with `bird search "query" --json`.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to get YouTube video transcripts / subtitles for AI agent?</strong></summary>
|
||||||
|
|
||||||
|
`yt-dlp --dump-json "https://youtube.com/watch?v=xxx"` extracts video metadata; `yt-dlp --write-sub --skip-download "URL"` extracts subtitles. Supports multiple languages, no API key required.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Reddit returns 403 from server / datacenter IP blocked?</strong></summary>
|
||||||
|
|
||||||
|
Reddit blocks datacenter IPs. Configure a residential proxy: `agent-reach configure proxy http://user:pass@ip:port`. Recommended: Webshare (~$1/month). Local machines typically don't have this issue.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Does Agent Reach work with Claude Code / Cursor / Windsurf / OpenClaw?</strong></summary>
|
||||||
|
|
||||||
|
Yes! Agent Reach is an installer + configuration tool. Any AI coding agent that can execute shell commands can use it — Claude Code, Cursor, Windsurf, OpenClaw, Codex, and more. Just `pip install agent-reach`, run `agent-reach install`, and the agent can start using the upstream tools immediately.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Is Agent Reach free? Any API costs?</strong></summary>
|
||||||
|
|
||||||
|
100% free and open source. All backends (bird CLI, yt-dlp, Jina Reader, Exa) are free tools that don't require paid API keys. The only optional cost is a residential proxy (~$1/month) if you need Reddit/Bilibili access from a server.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>Free alternative to Twitter API for web scraping?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach uses bird CLI which accesses Twitter via cookie auth — same as your browser session. No API fees, no rate limit tiers, no developer account needed. Supports search, read tweets, read profiles, and timelines.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to read XiaoHongShu / 小红书 content programmatically?</strong></summary>
|
||||||
|
|
||||||
|
Agent Reach integrates with xiaohongshu-mcp (runs in Docker). After setup, use `mcporter call 'xiaohongshu.get_feed_detail(...)'` to read notes or `mcporter call 'xiaohongshu.search_feeds(keyword: "query")'` to search.
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary><strong>How to parse Douyin / 抖音 videos with AI agent?</strong></summary>
|
||||||
|
|
||||||
|
Install douyin-mcp-server, then your agent can use `mcporter call 'douyin.parse_douyin_video_info(share_link: "share_url")'` to parse video info and get watermark-free download links. No login required — just share the Douyin link. See https://github.com/yzfly/douyin-mcp-server
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Credits
|
## Credits
|
||||||
|
|
||||||
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://github.com/steipete/bird) · [Exa](https://exa.ai) · [feedparser](https://github.com/kurtmckee/feedparser) · [instaloader](https://github.com/instaloader/instaloader) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
|
[Jina Reader](https://github.com/jina-ai/reader) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [bird](https://github.com/steipete/bird) · [Exa](https://exa.ai) · [feedparser](https://github.com/kurtmckee/feedparser) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) · [mcp-bosszp](https://github.com/mucsbr/mcp-bosszp)
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
+74
-27
@@ -20,7 +20,7 @@ Copy this to your AI Agent:
|
|||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
|
|
||||||
Install Agent Reach and make sure **all channels are working**. Your user expects you to have full internet access capabilities when this is done — reading tweets, searching Reddit, watching YouTube, browsing GitHub, and more.
|
Install Agent Reach and all upstream tools so your user's agent has full internet access. After installation, you'll use the upstream tools directly (bird CLI, yt-dlp, mcporter, gh CLI, etc.) — Agent Reach is just the installer and health checker.
|
||||||
|
|
||||||
### ⚠️ Boundaries
|
### ⚠️ Boundaries
|
||||||
|
|
||||||
@@ -80,16 +80,38 @@ Only ask the user when you genuinely need their input (credentials, permissions,
|
|||||||
|
|
||||||
Some channels need credentials only the user can provide. Based on the doctor output, ask for what's missing:
|
Some channels need credentials only the user can provide. Based on the doctor output, ask for what's missing:
|
||||||
|
|
||||||
> 🔒 **Security tip:** For platforms that need cookies (Twitter, XiaoHongShu, Instagram), we recommend using a **dedicated/secondary account** rather than your main account. Cookie-based auth grants full account access — using a separate account limits the blast radius if credentials are ever compromised.
|
> 🔒 **Security tip:** For platforms that need cookies (Twitter, XiaoHongShu), we recommend using a **dedicated/secondary account** rather than your main account. Cookie-based auth grants full account access — using a separate account limits the blast radius if credentials are ever compromised.
|
||||||
|
|
||||||
**Twitter search & posting (server users):**
|
> 🍪 **Cookie 导入(所有需要登录的平台通用):**
|
||||||
|
>
|
||||||
|
> 所有需要 Cookie 的平台(Twitter、小红书等),**优先使用 Cookie-Editor 导入**,这是最简单最可靠的方式:
|
||||||
|
> 1. 用户在自己的浏览器上登录对应平台
|
||||||
|
> 2. 安装 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) Chrome 插件
|
||||||
|
> 3. 点击插件 → Export → Header String
|
||||||
|
> 4. 把导出的字符串发给 Agent
|
||||||
|
>
|
||||||
|
> **本地电脑用户**也可以用 `agent-reach configure --from-browser chrome` 一键自动提取(支持 Twitter + 小红书)。
|
||||||
|
|
||||||
|
**Twitter search & posting:**
|
||||||
> "To unlock Twitter search, I need your Twitter cookies. Install the Cookie-Editor Chrome extension, go to x.com/twitter.com, click the extension → Export → Header String, and paste it to me."
|
> "To unlock Twitter search, I need your Twitter cookies. Install the Cookie-Editor Chrome extension, go to x.com/twitter.com, click the extension → Export → Header String, and paste it to me."
|
||||||
> Extension: https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
agent-reach configure twitter-cookies "PASTED_STRING"
|
agent-reach configure twitter-cookies "PASTED_STRING"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **代理说明(中国大陆等需要翻墙的网络环境):**
|
||||||
|
>
|
||||||
|
> bird CLI 使用 Node.js 原生 `fetch()`,默认不走系统代理(`HTTP_PROXY`/`HTTPS_PROXY`)。
|
||||||
|
> Agent Reach 已内置解决方案:安装时会自动安装 `undici`(`npm install -g undici`),
|
||||||
|
> 运行 bird 时自动注入 `EnvHttpProxyAgent`,让 fetch 走用户配置的代理。
|
||||||
|
>
|
||||||
|
> **你(Agent)需要做的:**
|
||||||
|
> 1. 确认 `undici` 已全局安装:`npm list -g undici`,没有就装:`npm install -g undici`
|
||||||
|
> 2. 确认用户配了代理:`agent-reach configure proxy http://user:pass@ip:port`
|
||||||
|
> 3. Agent Reach 会自动处理剩下的,不需要用户做额外操作
|
||||||
|
>
|
||||||
|
> 如果用户报告 "fetch failed",参考 [troubleshooting.md](troubleshooting.md)
|
||||||
|
|
||||||
**Reddit & Bilibili full access (server users):**
|
**Reddit & Bilibili full access (server users):**
|
||||||
> "Reddit and Bilibili block server IPs. To unlock full access, I need a residential proxy. You can get one at https://webshare.io ($1/month). Send me the proxy address."
|
> "Reddit and Bilibili block server IPs. To unlock full access, I need a residential proxy. You can get one at https://webshare.io ($1/month). Send me the proxy address."
|
||||||
|
|
||||||
@@ -108,23 +130,42 @@ mcporter config add xiaohongshu http://localhost:18060/mcp
|
|||||||
> 如果在服务器上,建议加代理避免 IP 风控:
|
> 如果在服务器上,建议加代理避免 IP 风控:
|
||||||
> `docker run -d --name xiaohongshu-mcp -p 18060:18060 -e XHS_PROXY=http://user:pass@ip:port xpzouying/xiaohongshu-mcp`
|
> `docker run -d --name xiaohongshu-mcp -p 18060:18060 -e XHS_PROXY=http://user:pass@ip:port xpzouying/xiaohongshu-mcp`
|
||||||
>
|
>
|
||||||
> **登录方式:**
|
> **登录方式(优先用 Cookie-Editor,最简单):**
|
||||||
> - **本地电脑(有浏览器):** 打开 http://localhost:18060 扫码登录即可。
|
> 1. 用户在自己的浏览器登录小红书 (xiaohongshu.com)
|
||||||
> - **服务器(无 UI 界面):** 服务器上通常没有浏览器,无法直接扫码。最方便的方式是在自己的电脑上用浏览器登录小红书,然后用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 插件导出 Cookie(Header String 格式),发给 Agent 即可完成配置。详见 [Cookie 导出指南](cookie-export.md)。
|
> 2. 用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 插件导出 Cookie(Header String 格式)
|
||||||
|
> 3. 把 Cookie 字符串发给 Agent
|
||||||
|
> 4. Agent 将 Cookie 写入 MCP 服务的 cookie 文件完成登录
|
||||||
|
>
|
||||||
|
> **备选:** 本地电脑如果有浏览器,也可以打开 http://localhost:18060 扫码登录。
|
||||||
|
|
||||||
**Instagram (需要 instaloader):**
|
**抖音 / Douyin (douyin-mcp-server):**
|
||||||
> "Instagram 需要 instaloader。我来帮你安装。"
|
> "抖音视频解析需要一个 MCP 服务。安装 douyin-mcp-server 后即可解析视频、获取无水印下载链接。"
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install instaloader
|
# 1. 安装
|
||||||
|
pip install douyin-mcp-server
|
||||||
|
|
||||||
|
# 2. 启动 HTTP 服务(端口 18070)
|
||||||
|
# 方式一:用 uv(推荐)
|
||||||
|
git clone https://github.com/yzfly/douyin-mcp-server.git && cd douyin-mcp-server
|
||||||
|
uv sync && uv run python run_http.py
|
||||||
|
|
||||||
|
# 方式二:直接用 Python 启动
|
||||||
|
python -c "
|
||||||
|
from douyin_mcp_server.server import mcp
|
||||||
|
mcp.settings.host = '127.0.0.1'
|
||||||
|
mcp.settings.port = 18070
|
||||||
|
mcp.run(transport='streamable-http')
|
||||||
|
"
|
||||||
|
|
||||||
|
# 3. 注册到 mcporter
|
||||||
|
mcporter config add douyin http://localhost:18070/mcp
|
||||||
```
|
```
|
||||||
|
|
||||||
> **登录方式(解锁私密内容):**
|
> 无需认证即可解析视频信息和获取下载链接。
|
||||||
> - **方法 1(推荐):Cookie-Editor 导入:** 在浏览器登录 Instagram → 用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 导出 Header String → 粘贴:
|
> 如需 AI 语音识别提取文案功能,需要配置硅基流动 API Key(`export API_KEY="sk-xxx"`)。
|
||||||
> ```bash
|
>
|
||||||
> agent-reach configure instagram-cookies "sessionid=xxx; csrftoken=yyy; ..."
|
> 详见 https://github.com/yzfly/douyin-mcp-server
|
||||||
> ```
|
|
||||||
> - **方法 2:instaloader 命令行登录:** `instaloader --login YOUR_USERNAME`(需要输密码,有 2FA 的话还要输验证码)
|
|
||||||
|
|
||||||
**LinkedIn (可选 — linkedin-scraper-mcp):**
|
**LinkedIn (可选 — linkedin-scraper-mcp):**
|
||||||
> "LinkedIn 基本内容可通过 Jina Reader 读取。完整功能(Profile 详情、职位搜索)需要 linkedin-scraper-mcp。"
|
> "LinkedIn 基本内容可通过 Jina Reader 读取。完整功能(Profile 详情、职位搜索)需要 linkedin-scraper-mcp。"
|
||||||
@@ -229,14 +270,20 @@ If the user wants a different agent to handle it, let them choose.
|
|||||||
| `agent-reach check-update` | Check for new versions |
|
| `agent-reach check-update` | Check for new versions |
|
||||||
| `agent-reach configure twitter-cookies "..."` | Unlock Twitter search + posting |
|
| `agent-reach configure twitter-cookies "..."` | Unlock Twitter search + posting |
|
||||||
| `agent-reach configure proxy URL` | Unlock Reddit + Bilibili on servers |
|
| `agent-reach configure proxy URL` | Unlock Reddit + Bilibili on servers |
|
||||||
| `agent-reach read <url>` | Read any URL |
|
|
||||||
| `agent-reach search "query"` | Search the web |
|
After installation, use upstream tools directly. See SKILL.md for the full command reference:
|
||||||
| `agent-reach search-reddit "query"` | Search Reddit |
|
|
||||||
| `agent-reach search-twitter "query"` | Search Twitter |
|
| Platform | Upstream Tool | Example |
|
||||||
| `agent-reach search-github "query"` | Search GitHub |
|
|----------|--------------|---------|
|
||||||
| `agent-reach search-youtube "query"` | Search YouTube |
|
| Twitter/X | `bird` | `bird search "query" --json` |
|
||||||
| `agent-reach search-bilibili "query"` | Search Bilibili |
|
| YouTube | `yt-dlp` | `yt-dlp --dump-json URL` |
|
||||||
| `agent-reach search-xhs "query"` | Search XiaoHongShu |
|
| Bilibili | `yt-dlp` | `yt-dlp --dump-json URL` |
|
||||||
| `agent-reach search-instagram "query"` | Search Instagram |
|
| Reddit | `curl` | `curl -s "https://reddit.com/r/xxx.json"` |
|
||||||
| `agent-reach search-linkedin "query"` | Search LinkedIn |
|
| GitHub | `gh` | `gh search repos "query"` |
|
||||||
| `agent-reach search-bosszhipin "query"` | Search Boss直聘 |
|
| Web | `curl` + Jina | `curl -s "https://r.jina.ai/URL"` |
|
||||||
|
| Exa Search | `mcporter` | `mcporter call 'exa.web_search_exa(...)'` |
|
||||||
|
| 小红书 | `mcporter` | `mcporter call 'xiaohongshu.search_feeds(...)'` |
|
||||||
|
| 抖音 | `mcporter` | `mcporter call 'douyin.parse_douyin_video_info(...)'` |
|
||||||
|
| LinkedIn | `mcporter` | `mcporter call 'linkedin.get_person_profile(...)'` |
|
||||||
|
| Boss直聘 | `mcporter` | `mcporter call 'bosszhipin.search_jobs_tool(...)'` |
|
||||||
|
| RSS | `feedparser` | `python3 -c "import feedparser; ..."` |
|
||||||
|
|||||||
@@ -0,0 +1,69 @@
|
|||||||
|
# Troubleshooting / 常见问题
|
||||||
|
|
||||||
|
## Twitter/X: bird CLI "fetch failed"
|
||||||
|
|
||||||
|
**症状:** `bird whoami` 或 `bird search` 返回 "fetch failed"
|
||||||
|
|
||||||
|
**原因:** bird CLI 使用 Node.js 原生 `fetch()` 发请求,而 Node.js 的 fetch **不走系统代理**(不读取 `HTTP_PROXY`/`HTTPS_PROXY` 环境变量)。如果你的网络环境需要代理才能访问 x.com,bird 就连不上。
|
||||||
|
|
||||||
|
**解决方案(按推荐顺序):**
|
||||||
|
|
||||||
|
### 方案 1:使用透明代理 / TUN 模式(推荐)
|
||||||
|
|
||||||
|
让代理工具接管所有网络流量,这样 bird 的 fetch 也会走代理:
|
||||||
|
|
||||||
|
- **Clash Verge / Clash for Windows:** 开启 TUN 模式或系统代理
|
||||||
|
- **Proxifier(Windows):** 添加规则让 Node.js 进程走代理
|
||||||
|
- **macOS:** 在 Surge/ClashX Pro 中开启增强模式
|
||||||
|
|
||||||
|
### 方案 2:验证 Cookie 有效性
|
||||||
|
|
||||||
|
确认 Cookie 没过期:
|
||||||
|
|
||||||
|
1. 在浏览器里正常登录 x.com
|
||||||
|
2. 用 [Cookie-Editor](https://chromewebstore.google.com/detail/cookie-editor/hlkenndednhfkekhgcdicdfddnkalmdm) 重新导出 Header String
|
||||||
|
3. 重新配置:`agent-reach configure twitter-cookies "新的Cookie"`
|
||||||
|
|
||||||
|
### 方案 3:不用 bird,用 Exa 搜索替代
|
||||||
|
|
||||||
|
bird 不可用时,可以直接用 Exa 搜索 Twitter 内容:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mcporter call 'exa.web_search_exa(query: "site:x.com query", numResults: 10)'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 方案 4:配置 Node.js 全局代理(高级)
|
||||||
|
|
||||||
|
安装 `global-agent` 让 Node.js 的 fetch 走代理:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install -g global-agent
|
||||||
|
```
|
||||||
|
|
||||||
|
然后在运行 bird 前设置环境变量:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Linux / macOS
|
||||||
|
export GLOBAL_AGENT_HTTP_PROXY=http://127.0.0.1:7890
|
||||||
|
export NODE_OPTIONS="--require global-agent/bootstrap"
|
||||||
|
bird search "test"
|
||||||
|
|
||||||
|
# Windows (PowerShell)
|
||||||
|
$env:GLOBAL_AGENT_HTTP_PROXY = "http://127.0.0.1:7890"
|
||||||
|
$env:NODE_OPTIONS = "--require global-agent/bootstrap"
|
||||||
|
bird search "test"
|
||||||
|
```
|
||||||
|
|
||||||
|
> ⚠️ 注意:这个方案需要每次运行 bird 前都设置环境变量,不太方便。推荐用方案 1。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Boss直聘: "访问行为异常"
|
||||||
|
|
||||||
|
**症状:** mcp-bosszp 登录成功,但 API 请求返回"您的访问行为异常"
|
||||||
|
|
||||||
|
**原因:** Boss直聘的反爬机制会检测请求指纹(不只是 IP),Python requests 库的特征与真实浏览器不同。
|
||||||
|
|
||||||
|
**解决方案:**
|
||||||
|
- **本地电脑:** 正常使用,一般不会被拦
|
||||||
|
- **服务器:** 使用 Jina Reader 读取职位页面 + Exa 搜索职位信息作为替代
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Agent Reach
|
||||||
|
|
||||||
|
> Give your AI agent eyes to see the entire internet. A unified CLI tool that lets AI agents read and search across 12+ platforms — Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu, Instagram, LinkedIn, Boss直聘, RSS, and any web page. One install, zero API fees.
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
- [Installation Guide](https://github.com/Panniantong/Agent-Reach/blob/main/docs/install.md): Step-by-step setup instructions for AI agents
|
||||||
|
- [README (中文)](https://github.com/Panniantong/Agent-Reach/blob/main/README.md): Full documentation in Chinese
|
||||||
|
- [README (English)](https://github.com/Panniantong/Agent-Reach/blob/main/docs/README_en.md): Full documentation in English
|
||||||
|
|
||||||
|
## Core Commands
|
||||||
|
|
||||||
|
- [CLI Usage](https://github.com/Panniantong/Agent-Reach/blob/main/docs/install.md): `agent-reach read <url>` reads any URL, `agent-reach search-twitter "query"` searches Twitter, etc.
|
||||||
|
|
||||||
|
## Key Features
|
||||||
|
|
||||||
|
- Read any URL: tweets, Reddit posts, YouTube videos (transcripts), GitHub repos, articles, XiaoHongShu notes, Bilibili videos, RSS feeds
|
||||||
|
- Search across platforms: Twitter/X, Reddit, GitHub, YouTube, Bilibili, XiaoHongShu, Instagram, LinkedIn, Boss直聘, Web (via Exa)
|
||||||
|
- Self-diagnosis: `agent-reach doctor` checks what works and what needs setup
|
||||||
|
- Auto-installs dependencies: `agent-reach install --env=auto`
|
||||||
|
- Cookie-based auth for platforms that require login (Twitter, XiaoHongShu, Instagram)
|
||||||
|
- Proxy support for platforms that block server IPs (Reddit, Bilibili)
|
||||||
|
- Zero API fees: all backends are free and open-source (bird CLI, yt-dlp, Jina Reader, instaloader, etc.)
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- [Troubleshooting Guide](https://github.com/Panniantong/Agent-Reach/blob/main/docs/troubleshooting.md): Common issues and solutions
|
||||||
|
|
||||||
|
## Optional
|
||||||
|
|
||||||
|
- [SKILL.md](https://github.com/Panniantong/Agent-Reach/blob/main/agent_reach/skill/SKILL.md): Integration guide for AI agent frameworks (OpenClaw, Claude Code, etc.)
|
||||||
|
- [PyPI Package](https://pypi.org/project/agent-reach/): `pip install agent-reach`
|
||||||
+10
-2
@@ -1,12 +1,20 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "agent-reach"
|
name = "agent-reach"
|
||||||
version = "1.1.0"
|
version = "1.2.0"
|
||||||
description = "Give your AI Agent eyes to see the entire internet. Search + Read 10+ platforms."
|
description = "Give your AI Agent eyes to see the entire internet. Search + Read 10+ platforms."
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
license = {text = "MIT"}
|
license = {text = "MIT"}
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
authors = [{name = "Neo Reid"}]
|
authors = [{name = "Neo Reid"}]
|
||||||
keywords = ["ai-agent", "web-reader", "search", "mcp", "agent-infrastructure", "llm-tools"]
|
keywords = [
|
||||||
|
"ai-agent", "llm-tools", "agent-infrastructure", "mcp",
|
||||||
|
"web-reader", "web-scraper", "search",
|
||||||
|
"twitter-scraper", "reddit-scraper", "youtube-transcript",
|
||||||
|
"bilibili", "xiaohongshu",
|
||||||
|
"ai-search", "cli", "automation",
|
||||||
|
"claude-code", "cursor", "openai",
|
||||||
|
"free-api", "no-api-key",
|
||||||
|
]
|
||||||
classifiers = [
|
classifiers = [
|
||||||
"Development Status :: 4 - Beta",
|
"Development Status :: 4 - Beta",
|
||||||
"Intended Audience :: Developers",
|
"Intended Audience :: Developers",
|
||||||
|
|||||||
Reference in New Issue
Block a user