Compare commits
19 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Pnant
|
97e9e63f42 | ||
|
Pnant
|
853e5190e4 | ||
|
Pnant
|
7015b08063 | ||
|
Pnant
|
b0e010c6cc | ||
|
ming
|
4ca570a39f | ||
|
Andrew Barnes
|
9045fee67e | ||
|
hymansun
|
5b07d95e7e | ||
|
Alfred
|
a827452881 | ||
|
Pnant
|
781496540b | ||
|
nyxst4ck
|
bdf692c391 | ||
|
Yui(ゆい)
|
40963eb6b7 | ||
|
Alfred
|
6bc92a0ac7 | ||
|
hymansun
|
5dfd29eafd | ||
|
Yuhao Chen
|
dd1af8fc17 | ||
|
yagyesh-twofourlabs
|
7e264cbde6 | ||
|
Pnant
|
c2911cd4a2 | ||
|
Pnant
|
84b474c79b | ||
|
Pnant
|
b6241e8e67 | ||
|
nyxst4ck
|
607f0d2c56 |
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"WebFetch(domain:community.groq.com)"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -13,3 +13,6 @@
|
||||
|
||||
# Groq Whisper (optional, for video transcription) — https://console.groq.com
|
||||
# GROQ_API_KEY=gsk_your_key_here
|
||||
|
||||
# OpenAI Whisper (optional fallback when Groq is rate-limited) — https://platform.openai.com
|
||||
# OPENAI_API_KEY=sk-your_key_here
|
||||
|
||||
@@ -28,3 +28,43 @@ jobs:
|
||||
- name: Run tests
|
||||
run: |
|
||||
pytest -q
|
||||
|
||||
# Editable installs (-e) never exercise wheel packaging, so a broken wheel
|
||||
# can pass tests and still fail every real `pip install` from source.
|
||||
# This job builds the actual wheel and installs it into a clean venv.
|
||||
wheel-gate:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Build wheel
|
||||
run: |
|
||||
python -m pip install --upgrade pip build
|
||||
python -m build
|
||||
|
||||
- name: Verify wheel has no duplicate entries and ships data files
|
||||
run: |
|
||||
python - <<'PY'
|
||||
import glob, zipfile, collections
|
||||
whl = glob.glob("dist/*.whl")[0]
|
||||
names = zipfile.ZipFile(whl).namelist()
|
||||
dupes = [n for n, c in collections.Counter(names).items() if c > 1]
|
||||
assert not dupes, f"duplicate entries in wheel: {dupes}"
|
||||
assert "agent_reach/skill/SKILL.md" in names, "SKILL.md missing from wheel"
|
||||
for prefix in ("agent_reach/guides/", "agent_reach/scripts/", "agent_reach/skill/references/"):
|
||||
assert any(n.startswith(prefix) for n in names), f"{prefix} missing from wheel"
|
||||
print(f"wheel OK: {len(names)} entries, no duplicates, data files present")
|
||||
PY
|
||||
|
||||
- name: Smoke-install wheel into clean venv
|
||||
run: |
|
||||
python -m venv /tmp/smoke
|
||||
/tmp/smoke/bin/pip install --quiet dist/*.whl
|
||||
/tmp/smoke/bin/agent-reach version
|
||||
cd /tmp && /tmp/smoke/bin/python -c "import agent_reach; from importlib.resources import files; assert (files('agent_reach')/'skill'/'SKILL.md').is_file(); print('SKILL.md ships in site-packages OK')"
|
||||
|
||||
@@ -8,3 +8,6 @@ build/
|
||||
.env
|
||||
.agent-reach/
|
||||
*.log
|
||||
|
||||
# Claude Code personal permission settings — local only, never commit
|
||||
.claude/settings.local.json
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# CLAUDE.md
|
||||
|
||||
## Project
|
||||
Agent Reach — Python CLI + library that gives AI agents read/search access to 14+ internet platforms.
|
||||
Agent Reach — Python CLI + library that gives AI agents read/search access to 13 internet platforms.
|
||||
Positioning: installer + doctor + config tool. NOT a wrapper — after install, agents call upstream tools directly.
|
||||
Repo: github.com/Panniantong/Agent-Reach | License: MIT | Version: 1.3.0
|
||||
Repo: github.com/Panniantong/Agent-Reach | License: MIT | Version: 1.4.2
|
||||
|
||||
## Commands
|
||||
- `pip install -e .` — Dev install
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
<p align="center">
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="MIT License"></a>
|
||||
<a href="https://www.python.org/"><img src="https://img.shields.io/badge/Python-3.10+-green.svg?style=for-the-badge&logo=python&logoColor=white" alt="Python 3.8+"></a>
|
||||
<a href="https://www.python.org/"><img src="https://img.shields.io/badge/Python-3.10+-green.svg?style=for-the-badge&logo=python&logoColor=white" alt="Python 3.10+"></a>
|
||||
<a href="https://github.com/Panniantong/agent-reach/stargazers"><img src="https://img.shields.io/github/stars/Panniantong/agent-reach?style=for-the-badge" alt="GitHub Stars"></a>
|
||||
</p>
|
||||
|
||||
@@ -75,10 +75,7 @@ AI Agent 已经能帮你写代码、改文档、管项目——但你让它去
|
||||
| 📺 **B站** | 本地：字幕提取 + 搜索 | 服务器也能用 | 告诉 Agent「帮我配代理」 |
|
||||
| 📖 **Reddit** | 搜索 + 读帖子和评论（通过 rdt-cli） | Cookie | 需要登录认证（`rdt login`），详见 [rdt-cli](https://github.com/public-clis/rdt-cli) |
|
||||
| 📕 **小红书** | — | 阅读、搜索、发帖、评论、点赞 | 告诉 Agent「帮我配小红书」 |
|
||||
| 🎵 **抖音** | — | 视频解析、无水印下载链接获取 | 告诉 Agent「帮我配抖音」 |
|
||||
| 💼 **LinkedIn** | Jina Reader 读公开页面 | Profile 详情、公司页面、职位搜索 | 告诉 Agent「帮我配 LinkedIn」 |
|
||||
| 💬 **微信公众号** | 搜索 + 阅读公众号文章（全文 Markdown） | — | 无需配置 |
|
||||
| 📰 **微博** | 热搜、搜索内容/用户/话题、用户动态、评论 | — | 无需配置 |
|
||||
| 💻 **V2EX** | 热门帖子、节点帖子、帖子详情+回复、用户信息 | — | 无需配置 |
|
||||
| 📈 **雪球** | 股票行情、搜索股票、热门帖子、热门股票排行 | — | 告诉 Agent「帮我配雪球」 |
|
||||
| 🎙️ **小宇宙播客** | — | 播客音频转文字（Whisper 转录，免费 Key） | 告诉 Agent「帮我配小宇宙播客」 |
|
||||
@@ -175,9 +172,7 @@ channels/
|
||||
├── bilibili.py → yt-dlp ← 可以换成 bilibili-api……
|
||||
├── reddit.py → rdt-cli ← 搜索+阅读，需 Cookie 认证
|
||||
├── xiaohongshu.py → mcporter MCP ← 可以换成其他 XHS 工具……
|
||||
├── douyin.py → mcporter MCP ← 可以换成其他抖音工具……
|
||||
├── linkedin.py → linkedin-mcp ← 可以换成 LinkedIn API……
|
||||
├── wechat.py → Exa (+ Camoufox) ← 搜索+阅读微信公众号文章
|
||||
├── rss.py → feedparser ← 可以换成 atoma……
|
||||
├── exa_search.py → mcporter MCP ← 可以换成 Tavily、SerpAPI……
|
||||
└── __init__.py → 渠道注册（doctor 检测用）
|
||||
@@ -198,38 +193,10 @@ channels/
|
||||
| GitHub | [gh CLI](https://cli.github.com) | 官方工具，认证后完整 API 能力 |
|
||||
| 读 RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python 生态标准选择，2.3K Star |
|
||||
| 小红书 | [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) | 1.5K Star，pipx 一行安装，搜索/阅读/评论/发帖 |
|
||||
| 抖音 | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP 服务，无需登录，视频解析 + 无水印下载 |
|
||||
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | ⭐1.2K，MCP 服务，浏览器自动化 |
|
||||
| 微信公众号 | [Exa](https://exa.ai)（搜索+阅读）+ [Camoufox](https://github.com/daijro/camoufox)（可选） | 零配置搜索+全文阅读，Camoufox 可选增强 |
|
||||
|
||||
> 📌 这些都是「当前选型」。不满意？换掉对应文件就行。这正是脚手架的意义。
|
||||
|
||||
### 抖音 / 小红书脚本提取的可选实现
|
||||
|
||||
如果你不只是想“解析抖音视频信息”，还想统一处理：
|
||||
|
||||
- 抖音视频脚本提取
|
||||
- 小红书视频笔记脚本提取
|
||||
- 小红书图文笔记正文 + 图片文字提取
|
||||
- 固定输出 `script.md` 和 `info.json`
|
||||
|
||||
可以把 `douyin` 这个 mcporter alias 指向另一个兼容实现：
|
||||
|
||||
- [social-post-extractor-mcp](https://github.com/JNHFlow21/social-post-extractor-mcp)
|
||||
|
||||
这个实现保留了旧工具名兼容性：
|
||||
|
||||
- `parse_douyin_video_info`
|
||||
- `get_douyin_download_link`
|
||||
- `extract_douyin_text`
|
||||
|
||||
同时新增统一工具：
|
||||
|
||||
- `parse_social_post_info`
|
||||
- `extract_social_post_script`
|
||||
|
||||
所以从 Agent Reach 的视角看，它依然只是一个 `mcporter` 里的 `douyin` server，只是能力更完整。
|
||||
|
||||
---
|
||||
|
||||
## 安全性
|
||||
@@ -323,7 +290,7 @@ Agent Reach uses twitter-cli with cookie auth — zero API fees. Install with `p
|
||||
<details>
|
||||
<summary><strong>Reddit 返回 403 怎么办？</strong></summary>
|
||||
|
||||
Agent Reach 使用 [rdt-cli](https://github.com/public-clis/rdt-cli) 访问 Reddit。Reddit 自 2024 年起要求认证，安装后需运行 `rdt login` 登录。安装：`pipx install rdt-cli`，然后 `rdt login`（自动从浏览器提取 Cookie）。之后 Agent 可以用 `rdt search "关键词"` 搜索、`rdt read POST_ID` 读帖子全文和评论。
|
||||
Agent Reach 使用 [rdt-cli](https://github.com/public-clis/rdt-cli) 访问 Reddit。Reddit 自 2024 年起要求认证，安装后需运行 `rdt login` 登录。安装：`pipx install 'git+https://github.com/public-clis/rdt-cli.git'`（PyPI 版本暂时落后，从 GitHub 装），然后 `rdt login`（自动从浏览器提取 Cookie）。之后 Agent 可以用 `rdt search "关键词"` 搜索、`rdt read POST_ID` 读帖子全文和评论。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
@@ -338,12 +305,6 @@ Agent Reach 使用 [rdt-cli](https://github.com/public-clis/rdt-cli) 访问 Redd
|
||||
安装 `pipx install xiaohongshu-cli`，然后 `xhs login`（自动从浏览器提取 Cookie）。之后 Agent 就能用 `xhs search "关键词"` 搜索笔记、`xhs read NOTE_ID` 阅读详情、`xhs comments NOTE_ID` 查看评论了。不需要 Docker。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>怎么让 AI Agent 解析抖音视频？</strong></summary>
|
||||
|
||||
安装 douyin-mcp-server 后，Agent 就能用 `mcporter call 'douyin.parse_douyin_video_info(share_link: "分享链接")'` 解析视频信息、获取无水印下载链接。不需要登录，把抖音分享链接发给 Agent 就行。详见 https://github.com/yzfly/douyin-mcp-server
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Compatible with Claude Code / Cursor / OpenClaw / Windsurf?</strong></summary>
|
||||
|
||||
@@ -362,7 +323,7 @@ Yes! Agent Reach is an installer + configuration tool — any AI coding agent th
|
||||
|
||||
## 致谢
|
||||
|
||||
[twitter-cli](https://github.com/public-clis/twitter-cli) · [rdt-cli](https://github.com/public-clis/rdt-cli) · [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) · [bili-cli](https://github.com/public-clis/bilibili-cli) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [Jina Reader](https://github.com/jina-ai/reader) · [Exa](https://exa.ai) · [mcporter](https://github.com/nicobailon/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||
[twitter-cli](https://github.com/public-clis/twitter-cli) · [rdt-cli](https://github.com/public-clis/rdt-cli) · [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) · [bili-cli](https://github.com/public-clis/bilibili-cli) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [Jina Reader](https://github.com/jina-ai/reader) · [Exa](https://exa.ai) · [mcporter](https://github.com/nicobailon/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||
|
||||
## 联系
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Agent Reach — Give your AI Agent eyes to see the entire internet."""
|
||||
|
||||
__version__ = "1.4.0"
|
||||
__version__ = "1.4.2"
|
||||
__author__ = "Neo Reid"
|
||||
|
||||
from agent_reach.core import AgentReach
|
||||
|
||||
@@ -16,10 +16,7 @@ from .rss import RSSChannel
|
||||
from .bilibili import BilibiliChannel
|
||||
from .exa_search import ExaSearchChannel
|
||||
from .xiaohongshu import XiaoHongShuChannel
|
||||
from .douyin import DouyinChannel
|
||||
from .linkedin import LinkedInChannel
|
||||
from .wechat import WeChatChannel
|
||||
from .weibo import WeiboChannel
|
||||
from .xiaoyuzhou import XiaoyuzhouChannel
|
||||
from .v2ex import V2EXChannel
|
||||
from .xueqiu import XueqiuChannel
|
||||
@@ -32,10 +29,7 @@ ALL_CHANNELS: List[Channel] = [
|
||||
RedditChannel(),
|
||||
BilibiliChannel(),
|
||||
XiaoHongShuChannel(),
|
||||
DouyinChannel(),
|
||||
LinkedInChannel(),
|
||||
WeChatChannel(),
|
||||
WeiboChannel(),
|
||||
XiaoyuzhouChannel(),
|
||||
V2EXChannel(),
|
||||
XueqiuChannel(),
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Douyin (抖音) — check if mcporter + douyin-mcp-server is available."""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from .base import Channel
|
||||
|
||||
|
||||
class DouyinChannel(Channel):
|
||||
name = "douyin"
|
||||
description = "抖音短视频"
|
||||
backends = ["douyin-mcp-server"]
|
||||
tier = 2
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
from urllib.parse import urlparse
|
||||
d = urlparse(url).netloc.lower()
|
||||
return "douyin.com" in d or "iesdouyin.com" in d
|
||||
|
||||
def check(self, config=None):
|
||||
mcporter = shutil.which("mcporter")
|
||||
if not mcporter:
|
||||
return "off", (
|
||||
"需要 mcporter + douyin-mcp-server。安装步骤：\n"
|
||||
" 1. npm install -g mcporter\n"
|
||||
" 2. pip install douyin-mcp-server\n"
|
||||
" 3. 启动服务（见下方说明）\n"
|
||||
" 4. mcporter config add douyin http://localhost:18070/mcp\n"
|
||||
" 详见 https://github.com/yzfly/douyin-mcp-server"
|
||||
)
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[mcporter, "config", "list"], capture_output=True,
|
||||
encoding="utf-8", errors="replace", timeout=5
|
||||
)
|
||||
if "douyin" not in r.stdout:
|
||||
return "off", (
|
||||
"mcporter 已装但抖音 MCP 未配置。运行：\n"
|
||||
" pip install douyin-mcp-server\n"
|
||||
" # 启动服务后：\n"
|
||||
" mcporter config add douyin http://localhost:18070/mcp"
|
||||
)
|
||||
except Exception:
|
||||
return "off", "mcporter 连接异常"
|
||||
# Verify MCP connectivity by listing available tools instead of
|
||||
# calling with a hardcoded (invalid) share link that always fails.
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[mcporter, "list", "douyin"],
|
||||
capture_output=True, encoding="utf-8", errors="replace", timeout=15
|
||||
)
|
||||
if r.returncode == 0 and r.stdout.strip():
|
||||
return "ok", "完整可用（视频解析、下载链接获取）"
|
||||
return "warn", "MCP 已连接但工具列表为空，检查 douyin-mcp-server 服务是否在运行"
|
||||
except Exception:
|
||||
return "warn", "MCP 连接异常，检查 douyin-mcp-server 服务是否在运行"
|
||||
@@ -3,6 +3,9 @@
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
from agent_reach.utils.process import utf8_subprocess_env
|
||||
|
||||
from .base import Channel
|
||||
|
||||
|
||||
@@ -28,7 +31,8 @@ class LinkedInChannel(Channel):
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[mcporter, "config", "list"], capture_output=True,
|
||||
encoding="utf-8", errors="replace", timeout=5
|
||||
encoding="utf-8", errors="replace", timeout=5,
|
||||
env=utf8_subprocess_env(),
|
||||
)
|
||||
if "linkedin" in r.stdout.lower():
|
||||
return "ok", "完整可用（Profile、公司、职位搜索）"
|
||||
|
||||
@@ -13,13 +13,15 @@ import subprocess
|
||||
from .base import Channel
|
||||
|
||||
_CREDENTIAL_FILE = "~/.config/rdt-cli/credential.json"
|
||||
# Pinned to the 0.4.2 state — PyPI still only has 0.4.1 (upstream issue #10).
|
||||
_RDT_GIT_SOURCE = "git+https://github.com/public-clis/rdt-cli.git@5e4fb3720d5c174e976cd425ccc3b879d52cac66"
|
||||
|
||||
|
||||
class RedditChannel(Channel):
|
||||
name = "reddit"
|
||||
description = "Reddit 帖子和评论"
|
||||
backends = ["rdt-cli"]
|
||||
tier = 0
|
||||
tier = 1 # Reddit requires login since 2024 (rdt login) — not zero-config
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
from urllib.parse import urlparse
|
||||
@@ -31,9 +33,10 @@ class RedditChannel(Channel):
|
||||
rdt = shutil.which("rdt")
|
||||
if not rdt:
|
||||
return "off", (
|
||||
"需要安装 rdt-cli（推荐使用最新版 v0.4.2+）：\n"
|
||||
" pip install 'rdt-cli>=0.4.2'\n"
|
||||
"或：\n"
|
||||
"需要安装 rdt-cli。PyPI 版本可能暂时落后，推荐直接从 GitHub 安装：\n"
|
||||
f" pipx install '{_RDT_GIT_SOURCE}'\n"
|
||||
"如已确认 PyPI 版本已更新，也可使用：\n"
|
||||
" pipx install rdt-cli\n"
|
||||
" uv tool install rdt-cli\n"
|
||||
"最新源码：https://github.com/public-clis/rdt-cli\n"
|
||||
"安装后运行 `rdt login` 登录（需先在浏览器登录 reddit.com）"
|
||||
|
||||
@@ -1,63 +0,0 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""WeChat Official Account articles — read and search.
|
||||
|
||||
Read: Exa crawling (primary) / Camoufox stealth browser (optional)
|
||||
Search: Exa web_search with includeDomains mp.weixin.qq.com
|
||||
"""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from .base import Channel
|
||||
|
||||
|
||||
def _exa_available() -> bool:
|
||||
mcporter = shutil.which("mcporter")
|
||||
if not mcporter:
|
||||
return False
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[mcporter, "config", "list"],
|
||||
capture_output=True, encoding="utf-8", errors="replace", timeout=5,
|
||||
)
|
||||
return "exa" in r.stdout.lower()
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
class WeChatChannel(Channel):
|
||||
name = "wechat"
|
||||
description = "微信公众号文章"
|
||||
backends = ["Exa via mcporter (搜索+阅读)", "Camoufox (可选阅读)"]
|
||||
tier = 0
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
from urllib.parse import urlparse
|
||||
d = urlparse(url).netloc.lower()
|
||||
return "mp.weixin.qq.com" in d or "weixin.qq.com" in d
|
||||
|
||||
def check(self, config=None):
|
||||
has_exa = _exa_available()
|
||||
has_camoufox = False
|
||||
try:
|
||||
import camoufox # noqa: F401
|
||||
has_camoufox = True
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
if has_exa and has_camoufox:
|
||||
return "ok", "完整可用（Exa 搜索 + Exa/Camoufox 阅读公众号文章）"
|
||||
elif has_exa:
|
||||
return "ok", (
|
||||
"通过 Exa 搜索和阅读微信公众号文章（免费，无需额外配置）。"
|
||||
"可选安装 Camoufox 获得更好的全文阅读效果。"
|
||||
)
|
||||
elif has_camoufox:
|
||||
return "warn", (
|
||||
"Camoufox 可阅读公众号文章，但搜索功能需要 Exa。"
|
||||
"运行 `agent-reach install --env=auto` 安装 Exa。"
|
||||
)
|
||||
else:
|
||||
return "off", (
|
||||
"需要 mcporter + Exa MCP 来搜索和阅读微信公众号文章。\n"
|
||||
"运行 `agent-reach install --env=auto` 安装。"
|
||||
)
|
||||
@@ -1,52 +0,0 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Weibo (微博) — check if mcporter + mcp-server-weibo is available."""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from .base import Channel
|
||||
|
||||
|
||||
class WeiboChannel(Channel):
|
||||
name = "weibo"
|
||||
description = "微博动态与热搜"
|
||||
backends = ["mcp-server-weibo"]
|
||||
tier = 1
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
from urllib.parse import urlparse
|
||||
d = urlparse(url).netloc.lower()
|
||||
return "weibo.com" in d or "weibo.cn" in d
|
||||
|
||||
def check(self, config=None):
|
||||
mcporter = shutil.which("mcporter")
|
||||
if not mcporter:
|
||||
return "off", (
|
||||
"需要 mcporter + mcp-server-weibo。安装步骤：\n"
|
||||
" 1. npm install -g mcporter\n"
|
||||
" 2. pip install git+https://github.com/Panniantong/mcp-server-weibo.git\n"
|
||||
" 3. mcporter config add weibo --command 'mcp-server-weibo'\n"
|
||||
" 详见 https://github.com/Panniantong/mcp-server-weibo"
|
||||
)
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[mcporter, "config", "list"], capture_output=True,
|
||||
encoding="utf-8", errors="replace", timeout=5
|
||||
)
|
||||
if "weibo" not in r.stdout:
|
||||
return "off", (
|
||||
"mcporter 已装但微博 MCP 未配置。运行：\n"
|
||||
" pip install git+https://github.com/Panniantong/mcp-server-weibo.git\n"
|
||||
" mcporter config add weibo --command 'mcp-server-weibo'"
|
||||
)
|
||||
except Exception:
|
||||
return "off", "mcporter 连接异常"
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[mcporter, "list", "weibo"], capture_output=True,
|
||||
encoding="utf-8", errors="replace", timeout=15
|
||||
)
|
||||
if r.returncode == 0 and "search_users" in r.stdout:
|
||||
return "ok", "完整可用（热搜、搜索、用户动态、评论）"
|
||||
return "warn", "MCP 已配置但工具加载失败，检查 mcp-server-weibo 版本"
|
||||
except Exception:
|
||||
return "warn", "MCP 连接异常，检查 mcp-server-weibo 是否可用"
|
||||
@@ -9,6 +9,16 @@ from agent_reach.utils.text import read_utf8_text
|
||||
from .base import Channel
|
||||
|
||||
|
||||
def _has_js_runtime_config(config_path) -> bool:
|
||||
"""Return whether yt-dlp config explicitly enables a JS runtime."""
|
||||
try:
|
||||
if not config_path.exists():
|
||||
return False
|
||||
return "--js-runtimes" in read_utf8_text(config_path)
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
class YouTubeChannel(Channel):
|
||||
name = "youtube"
|
||||
description = "YouTube 视频和字幕"
|
||||
@@ -17,6 +27,7 @@ class YouTubeChannel(Channel):
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
from urllib.parse import urlparse
|
||||
|
||||
d = urlparse(url).netloc.lower()
|
||||
return "youtube.com" in d or "youtu.be" in d
|
||||
|
||||
@@ -35,12 +46,33 @@ class YouTubeChannel(Channel):
|
||||
has_deno = shutil.which("deno")
|
||||
if not has_deno:
|
||||
ytdlp_config = get_ytdlp_config_path()
|
||||
has_js_config = False
|
||||
if ytdlp_config.exists():
|
||||
has_js_config = "--js-runtimes" in read_utf8_text(ytdlp_config)
|
||||
if not has_js_config:
|
||||
if not _has_js_runtime_config(ytdlp_config):
|
||||
return "warn", (
|
||||
"yt-dlp 已安装但未配置 JS runtime。运行：\n"
|
||||
f" {render_ytdlp_fix_command()}"
|
||||
f"yt-dlp 已安装但未配置 JS runtime。运行：\n {render_ytdlp_fix_command()}"
|
||||
)
|
||||
return "ok", "可提取视频信息和字幕"
|
||||
# Surface transcription readiness so `doctor` reports it.
|
||||
msg = "可提取视频信息和字幕"
|
||||
if config is not None:
|
||||
providers = []
|
||||
if config.is_configured("groq_whisper"):
|
||||
providers.append("groq")
|
||||
if config.is_configured("openai_whisper"):
|
||||
providers.append("openai")
|
||||
if providers:
|
||||
if not shutil.which("ffmpeg"):
|
||||
msg += "（音频转写需安装 ffmpeg）"
|
||||
else:
|
||||
msg += f"，可转写音频（{'→'.join(providers)}）"
|
||||
return "ok", msg
|
||||
|
||||
def transcribe(self, url: str, *, provider: str = "auto", config=None) -> str:
|
||||
"""Download a YouTube video's audio and return its transcript.
|
||||
|
||||
Delegates to :func:`agent_reach.transcribe.transcribe`. Imported lazily
|
||||
so the channel module stays cheap to import for users who never
|
||||
transcribe.
|
||||
"""
|
||||
from agent_reach.transcribe import transcribe as _transcribe
|
||||
|
||||
return _transcribe(url, provider=provider, config=config)
|
||||
|
||||
|
||||
+60
-133
@@ -17,6 +17,9 @@ import time
|
||||
|
||||
from agent_reach import __version__
|
||||
|
||||
# Pinned to the 0.4.2 state — PyPI still only has 0.4.1 (upstream issue #10).
|
||||
_RDT_GIT_SOURCE = "git+https://github.com/public-clis/rdt-cli.git@5e4fb3720d5c174e976cd425ccc3b879d52cac66"
|
||||
|
||||
|
||||
def _ensure_utf8_console():
|
||||
"""Best-effort Windows console UTF-8 setup for CLI runtime only."""
|
||||
@@ -70,13 +73,13 @@ def main():
|
||||
help="Show what would be done without making any changes")
|
||||
p_install.add_argument("--channels", default="",
|
||||
help="Comma-separated optional channels to install "
|
||||
"(twitter,weibo,wechat,xiaoyuzhou,xueqiu,xiaohongshu,"
|
||||
"reddit,bilibili,douyin,linkedin,all)")
|
||||
"(twitter,xiaoyuzhou,xueqiu,xiaohongshu,"
|
||||
"reddit,bilibili,linkedin,all)")
|
||||
|
||||
# ── configure ──
|
||||
p_conf = sub.add_parser("configure", help="Set a config value or auto-extract from browser")
|
||||
p_conf.add_argument("key", nargs="?", default=None,
|
||||
choices=["proxy", "github-token", "groq-key",
|
||||
choices=["proxy", "github-token", "groq-key", "openai-key",
|
||||
"twitter-cookies", "youtube-cookies",
|
||||
"xhs-cookies"],
|
||||
help="What to configure (omit if using --from-browser)")
|
||||
@@ -86,7 +89,9 @@ def main():
|
||||
help="Auto-extract ALL platform cookies from browser (chrome/firefox/edge/brave/opera)")
|
||||
|
||||
# ── doctor ──
|
||||
sub.add_parser("doctor", help="Check platform availability")
|
||||
p_doctor = sub.add_parser("doctor", help="Check platform availability")
|
||||
p_doctor.add_argument("--json", action="store_true",
|
||||
help="Output machine-readable JSON instead of the text report")
|
||||
|
||||
# ── uninstall ──
|
||||
p_uninstall = sub.add_parser("uninstall", help="Remove all Agent Reach config, tokens, and skill files")
|
||||
@@ -108,6 +113,14 @@ def main():
|
||||
p_format.add_argument("platform", choices=["xhs"], help="Platform to format (xhs)")
|
||||
|
||||
# ── check-update ──
|
||||
# ── transcribe ──
|
||||
p_tr = sub.add_parser("transcribe", help="Transcribe a URL or local audio file (Whisper via Groq/OpenAI)")
|
||||
p_tr.add_argument("source", help="Audio/video URL or local file path")
|
||||
p_tr.add_argument("--provider", choices=["auto", "groq", "openai"], default="auto",
|
||||
help="Transcription provider (default: auto = groq → openai fallback)")
|
||||
p_tr.add_argument("-o", "--output", default=None,
|
||||
help="Write transcript to a file instead of stdout")
|
||||
|
||||
sub.add_parser("check-update", help="Check for new versions and changes")
|
||||
|
||||
# ── watch ──
|
||||
@@ -130,7 +143,7 @@ def main():
|
||||
sys.exit(0)
|
||||
|
||||
if args.command == "doctor":
|
||||
_cmd_doctor()
|
||||
_cmd_doctor(args)
|
||||
elif args.command == "check-update":
|
||||
_cmd_check_update()
|
||||
elif args.command == "watch":
|
||||
@@ -147,6 +160,8 @@ def main():
|
||||
_cmd_skill(args)
|
||||
elif args.command == "format":
|
||||
_cmd_format(args)
|
||||
elif args.command == "transcribe":
|
||||
_cmd_transcribe(args)
|
||||
|
||||
|
||||
# ── Command handlers ────────────────────────────────
|
||||
@@ -180,14 +195,12 @@ def _cmd_install(args):
|
||||
# ── Parse --channels ──
|
||||
CHANNEL_INSTALLERS = {
|
||||
"twitter": _install_twitter_deps,
|
||||
"weibo": _install_weibo_deps,
|
||||
"wechat": _install_wechat_deps,
|
||||
"xiaoyuzhou": _install_xiaoyuzhou_deps,
|
||||
"xiaohongshu": _install_xhs_deps,
|
||||
"reddit": _install_reddit_deps,
|
||||
"bilibili": _install_bili_deps,
|
||||
# xueqiu: cookie-only, no install step
|
||||
# douyin/linkedin: manual setup, no auto-install
|
||||
# linkedin: manual setup, no auto-install
|
||||
}
|
||||
COOKIE_CHANNELS = {"twitter", "xueqiu", "bilibili"}
|
||||
|
||||
@@ -195,7 +208,7 @@ def _cmd_install(args):
|
||||
if args.channels:
|
||||
raw = [c.strip().lower() for c in args.channels.split(",") if c.strip()]
|
||||
if "all" in raw:
|
||||
requested_channels = set(CHANNEL_INSTALLERS.keys()) | {"xueqiu", "douyin", "linkedin"}
|
||||
requested_channels = set(CHANNEL_INSTALLERS.keys()) | {"xueqiu", "linkedin"}
|
||||
else:
|
||||
requested_channels = set(raw)
|
||||
|
||||
@@ -307,7 +320,7 @@ def _cmd_install(args):
|
||||
# First install — hint about optional channels
|
||||
print()
|
||||
print("More channels available! Use --channels to install:")
|
||||
print(" agent-reach install --channels=twitter,weibo,xiaohongshu,...")
|
||||
print(" agent-reach install --channels=twitter,xiaohongshu,reddit,...")
|
||||
print(" agent-reach install --channels=all (install everything)")
|
||||
|
||||
# Star reminder
|
||||
@@ -603,7 +616,7 @@ def _install_system_deps():
|
||||
except Exception:
|
||||
print(" -- Could not configure yt-dlp JS runtime (YouTube may not work)")
|
||||
|
||||
# NOTE: twitter-cli, weibo, xiaoyuzhou, wechat, xhs-cli etc. are optional.
|
||||
# NOTE: twitter-cli, xiaoyuzhou, xhs-cli etc. are optional.
|
||||
# They are installed via --channels flag, not here.
|
||||
# See CHANNEL_INSTALLERS in _cmd_install().
|
||||
|
||||
@@ -706,8 +719,10 @@ def _install_reddit_deps():
|
||||
if shutil.which("rdt"):
|
||||
print(" ✅ rdt-cli already installed")
|
||||
return
|
||||
for tool, cmd in [("pipx", ["pipx", "install", "rdt-cli"]),
|
||||
("uv", ["uv", "tool", "install", "rdt-cli"])]:
|
||||
for tool, cmd in [
|
||||
("pipx", ["pipx", "install", _RDT_GIT_SOURCE]),
|
||||
("uv", ["uv", "tool", "install", "--from", _RDT_GIT_SOURCE, "rdt-cli"]),
|
||||
]:
|
||||
if shutil.which(tool):
|
||||
try:
|
||||
subprocess.run(cmd, capture_output=True, encoding="utf-8",
|
||||
@@ -717,7 +732,7 @@ def _install_reddit_deps():
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
print(" [!] rdt-cli install failed. Run: pipx install rdt-cli")
|
||||
print(f" [!] rdt-cli install failed. Run: pipx install '{_RDT_GIT_SOURCE}'")
|
||||
|
||||
|
||||
def _install_bili_deps():
|
||||
@@ -743,123 +758,6 @@ def _install_bili_deps():
|
||||
print(" [!] bili-cli install failed. Run: pipx install bilibili-cli")
|
||||
|
||||
|
||||
def _install_weibo_deps():
|
||||
"""Install Weibo MCP server (Panniantong fork with visitor passport auth)."""
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
print("Setting up Weibo MCP server...")
|
||||
|
||||
# Check if already installed and working
|
||||
mcporter = shutil.which("mcporter")
|
||||
if mcporter:
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[mcporter, "config", "list"], capture_output=True,
|
||||
encoding="utf-8", errors="replace", timeout=5
|
||||
)
|
||||
if "weibo" in r.stdout:
|
||||
print(" ✅ Weibo MCP already configured")
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Install from our fork (has visitor passport auth fix)
|
||||
try:
|
||||
subprocess.run(
|
||||
[sys.executable, "-m", "pip", "install", "-q",
|
||||
"git+https://github.com/Panniantong/mcp-server-weibo.git"],
|
||||
check=True, timeout=120
|
||||
)
|
||||
print(" ✅ mcp-server-weibo installed (Panniantong fork)")
|
||||
except Exception as e:
|
||||
print(f" [!] mcp-server-weibo install failed: {e}")
|
||||
return
|
||||
|
||||
# Register with mcporter
|
||||
if mcporter:
|
||||
try:
|
||||
subprocess.run(
|
||||
[mcporter, "config", "add", "weibo", "--command", "mcp-server-weibo"],
|
||||
check=True, capture_output=True, timeout=10
|
||||
)
|
||||
print(" ✅ Weibo MCP registered with mcporter")
|
||||
except Exception:
|
||||
print(" [!] mcporter config add failed. Run manually: mcporter config add weibo --command 'mcp-server-weibo'")
|
||||
else:
|
||||
print(" -- mcporter not found, skipping MCP registration. Install mcporter first, then run: mcporter config add weibo --command 'mcp-server-weibo'")
|
||||
|
||||
|
||||
def _install_wechat_deps():
|
||||
"""Install WeChat article reading and search dependencies."""
|
||||
import subprocess
|
||||
|
||||
print("Setting up WeChat article tools...")
|
||||
|
||||
# Check if already installed
|
||||
has_camoufox = False
|
||||
has_miku = False
|
||||
try:
|
||||
import camoufox # noqa: F401
|
||||
has_camoufox = True
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
import miku_ai # noqa: F401
|
||||
has_miku = True
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# Install Python packages
|
||||
if has_camoufox and has_miku:
|
||||
print(" ✅ WeChat Python packages already installed")
|
||||
else:
|
||||
pkgs = []
|
||||
if not has_camoufox:
|
||||
pkgs.extend(["camoufox[geoip]", "markdownify", "beautifulsoup4", "httpx"])
|
||||
if not has_miku:
|
||||
pkgs.append("miku_ai")
|
||||
try:
|
||||
cmd = [sys.executable, "-m", "pip", "install", "--break-system-packages", "-q"] + pkgs
|
||||
subprocess.run(cmd, capture_output=True, encoding="utf-8", errors="replace", timeout=120)
|
||||
# Verify
|
||||
ok = True
|
||||
try:
|
||||
import importlib
|
||||
if not has_camoufox:
|
||||
importlib.import_module("camoufox")
|
||||
if not has_miku:
|
||||
importlib.import_module("miku_ai")
|
||||
except ImportError:
|
||||
ok = False
|
||||
if ok:
|
||||
print(f" ✅ WeChat Python packages installed ({', '.join(pkgs)})")
|
||||
else:
|
||||
print(f" [!] Some WeChat packages failed to install. Try: pip install {' '.join(pkgs)}")
|
||||
except Exception:
|
||||
print(f" [!] WeChat packages install failed. Try: pip install {' '.join(pkgs)}")
|
||||
|
||||
# Clone wechat-article-for-ai tool
|
||||
tools_dir = os.path.expanduser("~/.agent-reach/tools")
|
||||
wechat_dir = os.path.join(tools_dir, "wechat-article-for-ai")
|
||||
if os.path.isfile(os.path.join(wechat_dir, "main.py")):
|
||||
print(" ✅ wechat-article-for-ai tool already installed")
|
||||
else:
|
||||
try:
|
||||
os.makedirs(tools_dir, exist_ok=True)
|
||||
subprocess.run(
|
||||
["git", "clone", "--depth", "1",
|
||||
"https://github.com/Panniantong/wechat-article-for-ai.git", wechat_dir],
|
||||
capture_output=True, encoding="utf-8", errors="replace", timeout=60,
|
||||
)
|
||||
if os.path.isfile(os.path.join(wechat_dir, "main.py")):
|
||||
print(" ✅ wechat-article-for-ai tool installed")
|
||||
else:
|
||||
print(" [!] wechat-article-for-ai clone failed. Try: git clone https://github.com/Panniantong/wechat-article-for-ai.git " + wechat_dir)
|
||||
except Exception:
|
||||
print(" [!] wechat-article-for-ai clone failed. Try: git clone https://github.com/Panniantong/wechat-article-for-ai.git " + wechat_dir)
|
||||
|
||||
|
||||
def _install_system_deps_safe():
|
||||
"""Safe mode: check what's installed, print instructions for what's missing."""
|
||||
import shutil
|
||||
@@ -1120,6 +1018,29 @@ def _cmd_configure(args):
|
||||
config.set("groq_api_key", value)
|
||||
print(f"✅ Groq key configured!")
|
||||
|
||||
elif args.key == "openai-key":
|
||||
config.set("openai_api_key", value)
|
||||
print(f"✅ OpenAI key configured!")
|
||||
|
||||
|
||||
def _cmd_transcribe(args):
|
||||
"""Transcribe a URL or local audio file via Whisper (Groq → OpenAI fallback)."""
|
||||
from pathlib import Path
|
||||
|
||||
from agent_reach.transcribe import TranscribeError, transcribe
|
||||
|
||||
try:
|
||||
text = transcribe(args.source, provider=args.provider)
|
||||
except TranscribeError as e:
|
||||
print(f"❌ {e}")
|
||||
sys.exit(1)
|
||||
|
||||
if args.output:
|
||||
Path(args.output).write_text(text + "\n", encoding="utf-8")
|
||||
print(f"✅ Transcript written to {args.output}")
|
||||
else:
|
||||
print(text)
|
||||
|
||||
|
||||
def _parse_twitter_cookie_input(value: str):
|
||||
"""Parse Twitter cookie input from either separate values or a cookie header."""
|
||||
@@ -1416,7 +1337,7 @@ def _cmd_uninstall(args):
|
||||
print(" npm uninstall -g undici")
|
||||
|
||||
|
||||
def _cmd_doctor():
|
||||
def _cmd_doctor(args=None):
|
||||
from agent_reach.config import Config
|
||||
from agent_reach.doctor import check_all, format_report
|
||||
try:
|
||||
@@ -1425,6 +1346,11 @@ def _cmd_doctor():
|
||||
rprint = print
|
||||
config = Config()
|
||||
results = check_all(config)
|
||||
|
||||
if args is not None and getattr(args, "json", False):
|
||||
print(json.dumps(results, ensure_ascii=False, indent=2))
|
||||
return
|
||||
|
||||
rprint(format_report(results))
|
||||
|
||||
# Auto-install skill if not already present (fixes #154)
|
||||
@@ -1495,7 +1421,8 @@ def _cmd_setup():
|
||||
|
||||
# Step 3: Reddit — rdt-cli
|
||||
print("【信息】Reddit — 通过 rdt-cli 搜索和阅读，无需配置")
|
||||
print(" 安装：pipx install rdt-cli")
|
||||
print(f" 安装：pipx install '{_RDT_GIT_SOURCE}'")
|
||||
print(" 然后运行：rdt login")
|
||||
print()
|
||||
|
||||
# Step 4: Groq (Whisper)
|
||||
|
||||
@@ -21,8 +21,9 @@ class Config:
|
||||
# Feature → required config keys
|
||||
FEATURE_REQUIREMENTS = {
|
||||
"exa_search": ["exa_api_key"],
|
||||
"twitter_xreach": ["twitter_auth_token", "twitter_ct0"], # legacy key name; used by bird CLI
|
||||
"twitter_xreach": ["twitter_auth_token", "twitter_ct0"], # legacy key name; used by twitter-cli
|
||||
"groq_whisper": ["groq_api_key"],
|
||||
"openai_whisper": ["openai_api_key"],
|
||||
"github_token": ["github_token"],
|
||||
}
|
||||
|
||||
|
||||
@@ -34,6 +34,7 @@ def format_report(results: Dict[str, dict]) -> str:
|
||||
lines = []
|
||||
lines.append("[bold cyan]Agent Reach 状态[/bold cyan]")
|
||||
lines.append("[cyan]" + "=" * 40 + "[/cyan]")
|
||||
lines.append("图例：[green]✅[/green] 可用 [yellow][!][/yellow] 已装但需配置/登录 [red][X][/red] 未安装")
|
||||
|
||||
ok_count = sum(1 for r in results.values() if r["status"] == "ok")
|
||||
total = len(results)
|
||||
|
||||
@@ -8,7 +8,7 @@ Agent Reach 通过 **rdt-cli** 实现 Reddit 的搜索和阅读功能：
|
||||
- **搜索**：`rdt search "关键词"`
|
||||
- **阅读完整帖子+评论**：`rdt read POST_ID`
|
||||
|
||||
免费，无需代理，无需 API Key，无需登录。
|
||||
免费，无需代理，无需 API Key。需要登录认证（`rdt login`，自动从浏览器提取 Cookie）。
|
||||
|
||||
## Agent 可自动完成的步骤
|
||||
|
||||
@@ -17,9 +17,9 @@ Agent Reach 通过 **rdt-cli** 实现 Reddit 的搜索和阅读功能：
|
||||
which rdt && echo "installed" || echo "not installed"
|
||||
```
|
||||
|
||||
2. 如果未安装，自动安装：
|
||||
2. 如果未安装，自动安装（PyPI 版本暂时落后，从 GitHub 安装最新版）：
|
||||
```bash
|
||||
pipx install rdt-cli
|
||||
pipx install 'git+https://github.com/public-clis/rdt-cli.git'
|
||||
```
|
||||
|
||||
或一键安装：
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
# 微信公众号配置指南
|
||||
|
||||
## 功能说明
|
||||
读取微信公众号文章。需要 Playwright 来处理微信的反爬机制。
|
||||
|
||||
## Agent 可自动完成的步骤
|
||||
|
||||
1. 检查 Playwright 是否安装：
|
||||
```bash
|
||||
python3 -c "import playwright; print('installed')" 2>&1
|
||||
```
|
||||
|
||||
2. 安装 Playwright + 浏览器：
|
||||
```bash
|
||||
pip install playwright
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
3. 安装完成后测试：
|
||||
```bash
|
||||
curl -s "https://r.jina.ai/https://mp.weixin.qq.com/s/一个测试链接" -H "Accept: text/markdown"
|
||||
```
|
||||
|
||||
## 需要用户手动做的步骤
|
||||
|
||||
请告诉用户：
|
||||
|
||||
> 微信公众号的配置很简单，只需要安装一个浏览器组件（约 150MB）。
|
||||
>
|
||||
> 我来帮你安装，你不需要做任何事情。安装过程大约 1-2 分钟。
|
||||
>
|
||||
> 安装好之后就可以直接读取微信公众号文章了，不需要登录。
|
||||
|
||||
## Agent 操作流程
|
||||
|
||||
1. 安装 Playwright：`pip install playwright`
|
||||
2. 安装 Chromium：`playwright install chromium`
|
||||
3. 测试：读一篇微信文章
|
||||
4. 反馈："✅ 微信公众号已配置！发给我任何公众号文章链接，我都能读取。"
|
||||
5. 如果安装失败（空间不足等）："❌ 浏览器组件安装失败。可能是磁盘空间不足（需要约 150MB）。"
|
||||
@@ -1,11 +1,29 @@
|
||||
#!/bin/bash
|
||||
# 小宇宙播客转文字脚本
|
||||
# 用法: bash transcribe.sh <小宇宙链接> [输出文件路径]
|
||||
# 用法: bash transcribe.sh [--polish] <小宇宙链接> [输出文件路径]
|
||||
# 环境变量: GROQ_API_KEY (必须)
|
||||
#
|
||||
# --polish: 转录后调用 Groq Llama 3.3 70B 给文稿补中文标点+合理分段
|
||||
# （Whisper 对中文标点支持较弱，开启后阅读体验显著更好）
|
||||
|
||||
set -e
|
||||
|
||||
URL="${1:?用法: bash transcribe.sh <小宇宙链接> [输出文件路径]}"
|
||||
POLISH=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--polish) POLISH=1; shift ;;
|
||||
--) shift; break ;;
|
||||
-h|--help)
|
||||
echo "用法: bash transcribe.sh [--polish] <小宇宙链接> [输出文件路径]"
|
||||
exit 0 ;;
|
||||
--*)
|
||||
echo "未知选项: $1" >&2
|
||||
exit 1 ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
|
||||
URL="${1:?用法: bash transcribe.sh [--polish] <小宇宙链接> [输出文件路径]}"
|
||||
OUTPUT="${2:-/tmp/podcast_transcript.txt}"
|
||||
TMPDIR="/tmp/xiaoyuzhou_$$"
|
||||
|
||||
@@ -35,8 +53,8 @@ echo "===================="
|
||||
# Step 1: 提取音频 URL 和标题
|
||||
echo "🔍 正在解析页面..."
|
||||
PAGE=$(curl -s "$URL")
|
||||
AUDIO_URL=$(echo "$PAGE" | grep -oP 'https://media\.xyzcdn\.net/[^"]*\.(m4a|mp3)' | head -1)
|
||||
TITLE=$(echo "$PAGE" | grep -oP '"title":"[^"]*"' | head -1 | sed 's/"title":"//;s/"//')
|
||||
AUDIO_URL=$(echo "$PAGE" | perl -ne 'while (/(https:\/\/media\.xyzcdn\.net\/[^"]*\.(?:m4a|mp3))/gi) { print "$1\n" }' | head -1)
|
||||
TITLE=$(echo "$PAGE" | perl -ne 'if (/"title":"([^"]*)"/) { print "$1\n"; last }' | head -1)
|
||||
|
||||
if [ -z "$AUDIO_URL" ]; then
|
||||
echo "❌ 无法从页面提取音频链接"
|
||||
@@ -99,6 +117,7 @@ for i in $(seq 0 $((NUM_CHUNKS - 1))); do
|
||||
-F file="@$TMPDIR/chunk_${i}.mp3" \
|
||||
-F model="whisper-large-v3" \
|
||||
-F language="zh" \
|
||||
-F prompt="以下是一段中文普通话播客录音，请输出包含完整中文标点（，。？！：；“”‘’）的转写文本。" \
|
||||
-F response_format="text")
|
||||
|
||||
HTTP_CODE=$(echo "$RESPONSE" | tail -1)
|
||||
@@ -111,7 +130,7 @@ for i in $(seq 0 $((NUM_CHUNKS - 1))); do
|
||||
# 如果是速率限制，等待后重试
|
||||
if [ "$HTTP_CODE" = "429" ]; then
|
||||
# 从错误信息中提取等待时间，默认 120 秒
|
||||
WAIT_SEC=$(echo "$BODY" | grep -oP 'in \K[0-9]+m' | sed 's/m//' | head -1)
|
||||
WAIT_SEC=$(echo "$BODY" | perl -ne 'if (/in (\d+)m/) { print "$1\n"; exit }')
|
||||
WAIT_SEC=${WAIT_SEC:-2}
|
||||
WAIT_SEC=$((WAIT_SEC * 60 + 30))
|
||||
echo " ⏳ 速率限制，等待 ${WAIT_SEC} 秒后重试..."
|
||||
@@ -122,6 +141,7 @@ for i in $(seq 0 $((NUM_CHUNKS - 1))); do
|
||||
-F file="@$TMPDIR/chunk_${i}.mp3" \
|
||||
-F model="whisper-large-v3" \
|
||||
-F language="zh" \
|
||||
-F prompt="以下是一段中文普通话播客录音，请输出包含完整中文标点（，。？！：；“”‘’）的转写文本。" \
|
||||
-F response_format="text")
|
||||
HTTP_CODE=$(echo "$RESPONSE" | tail -1)
|
||||
BODY=$(echo "$RESPONSE" | sed '$d')
|
||||
@@ -140,6 +160,81 @@ for i in $(seq 0 $((NUM_CHUNKS - 1))); do
|
||||
echo "✅ ($CHARS 字)"
|
||||
done
|
||||
|
||||
# Step 6.5 (可选): 用 Llama 3.3 70B 给文稿补标点+分段
|
||||
if [ "$POLISH" = "1" ]; then
|
||||
echo "✨ 正在润色（Llama 3.3 70B 加标点+分段）..."
|
||||
for i in $(seq 0 $((NUM_CHUNKS - 1))); do
|
||||
echo -n " 段 $((i+1))/$NUM_CHUNKS... "
|
||||
IN_FILE="$TMPDIR/transcript_${i}.txt" \
|
||||
OUT_FILE="$TMPDIR/polished_${i}.txt" \
|
||||
GROQ_API_KEY="$GROQ_API_KEY" \
|
||||
python3 <<'PY'
|
||||
import json, os, sys, urllib.request, urllib.error
|
||||
|
||||
KEY = os.environ["GROQ_API_KEY"]
|
||||
IN = os.environ["IN_FILE"]
|
||||
OUT = os.environ["OUT_FILE"]
|
||||
|
||||
MODEL = "llama-3.3-70b-versatile"
|
||||
MAX_DEPTH = 3
|
||||
PROMPT_TMPL = (
|
||||
"以下是一段中文普通话播客的语音转写片段，由于 Whisper 对中文标点支持较弱，"
|
||||
"整段几乎没有标点。请你**只做一件事**：在合适位置补充中文标点（，。！？：；），"
|
||||
"可以适度分段。\n\n"
|
||||
"**严格要求**：\n"
|
||||
"- 不得修改、删除、增加任何汉字或英文/数字\n"
|
||||
"- 不得改写、润色、总结\n"
|
||||
"- 不得添加任何解释、前言、后记\n"
|
||||
"- 直接输出加好标点+合理分段后的全文\n\n"
|
||||
"原文：\n{}"
|
||||
)
|
||||
|
||||
def call_groq(text):
|
||||
body = json.dumps({
|
||||
"model": MODEL,
|
||||
"temperature": 0.2,
|
||||
"max_completion_tokens": 8192,
|
||||
"messages": [{"role": "user", "content": PROMPT_TMPL.format(text)}],
|
||||
}).encode()
|
||||
req = urllib.request.Request(
|
||||
"https://api.groq.com/openai/v1/chat/completions",
|
||||
data=body,
|
||||
headers={
|
||||
"Authorization": f"Bearer {KEY}",
|
||||
"Content-Type": "application/json",
|
||||
"User-Agent": "agent-reach-xiaoyuzhou/1.0",
|
||||
},
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=180) as r:
|
||||
resp = json.load(r)
|
||||
return (
|
||||
resp["choices"][0]["message"]["content"].strip(),
|
||||
resp["choices"][0].get("finish_reason"),
|
||||
)
|
||||
|
||||
def polish(text, depth=0):
|
||||
try:
|
||||
out, fr = call_groq(text)
|
||||
except urllib.error.HTTPError as e:
|
||||
sys.stderr.write(f"polish HTTP {e.code}: {e.read().decode(errors='replace')[:200]}\n")
|
||||
return text # fallback to raw
|
||||
except Exception as e:
|
||||
sys.stderr.write(f"polish error: {e}\n")
|
||||
return text
|
||||
if fr != "length" or depth >= MAX_DEPTH:
|
||||
return out
|
||||
# 输出被截断：从中点切两半递归处理
|
||||
mid = len(text) // 2
|
||||
return polish(text[:mid], depth + 1) + polish(text[mid:], depth + 1)
|
||||
|
||||
content = open(IN, encoding="utf-8").read().strip()
|
||||
result = polish(content)
|
||||
open(OUT, "w", encoding="utf-8").write(result + "\n")
|
||||
print(f"✅ ({len(result)} 字)")
|
||||
PY
|
||||
done
|
||||
fi
|
||||
|
||||
# Step 7: 合并输出
|
||||
echo "📄 正在合并文字稿..."
|
||||
|
||||
@@ -149,12 +244,19 @@ echo "📄 正在合并文字稿..."
|
||||
echo "来源: $URL"
|
||||
echo "时长: ${DURATION_MIN}分${DURATION_SEC}秒"
|
||||
echo "转录时间: $(date '+%Y-%m-%d %H:%M')"
|
||||
if [ "$POLISH" = "1" ]; then
|
||||
echo "润色: Groq Llama 3.3 70B"
|
||||
fi
|
||||
echo ""
|
||||
echo "---"
|
||||
echo ""
|
||||
|
||||
|
||||
for i in $(seq 0 $((NUM_CHUNKS - 1))); do
|
||||
cat "$TMPDIR/transcript_${i}.txt"
|
||||
if [ "$POLISH" = "1" ] && [ -f "$TMPDIR/polished_${i}.txt" ]; then
|
||||
cat "$TMPDIR/polished_${i}.txt"
|
||||
else
|
||||
cat "$TMPDIR/transcript_${i}.txt"
|
||||
fi
|
||||
echo ""
|
||||
done
|
||||
} > "$OUTPUT"
|
||||
|
||||
+16
-16
@@ -1,28 +1,28 @@
|
||||
---
|
||||
name: agent-reach
|
||||
description: >
|
||||
Give your AI agent eyes to see the entire internet.
|
||||
17 platforms via CLI, MCP, curl, and Python scripts.
|
||||
Zero config for 8 channels.
|
||||
MUST USE when user asks to search, browse, read, or interact with content from any of these platforms:
|
||||
小红书/xiaohongshu/xhs, Twitter/推特/X, B站/bilibili,
|
||||
V2EX, Reddit, LinkedIn/领英, YouTube, GitHub code search,
|
||||
小宇宙播客, 雪球/股票行情, RSS feeds, or any web URL.
|
||||
|
||||
Also MUST USE for: web搜索/搜/查/找/look up/research, 招聘/求职/jobs, 分享的链接/URL.
|
||||
Routes to CLI tools: xhs-cli, twitter-cli, rdt-cli, gh, yt-dlp, curl+Jina, mcporter.
|
||||
13 platforms. Zero config for 6 channels.
|
||||
|
||||
【路由方式】SKILL.md 包含路由表和常用命令，复杂场景需按需阅读对应分类的 references/*.md。
|
||||
分类：search / social (小红书/抖音/微博/推特/B站/V2EX/Reddit) / career(LinkedIn) / dev(github) / web(网页/文章/公众号/RSS) / video(YouTube/B站/播客).
|
||||
|
||||
Use when user asks to search, read, or interact on any supported platform,
|
||||
shares a URL, or asks to search the web.
|
||||
分类：search / social (小红书/推特/B站/V2EX/Reddit) / career(LinkedIn) / dev(github) / web(网页/文章/RSS) / video(YouTube/B站/播客)。
|
||||
triggers:
|
||||
- search: 搜/查/找/search/搜索/查一下/帮我搜
|
||||
- social:
|
||||
- 小红书: xiaohongshu/xhs/小红书/红书
|
||||
- 抖音: douyin/抖音
|
||||
- Twitter: twitter/推特/x.com/推文
|
||||
- 微博: weibo/微博
|
||||
- B站: bilibili/b站/哔哩哔哩
|
||||
- V2EX: v2ex
|
||||
- Reddit: reddit
|
||||
- career: 招聘/职位/求职/linkedin/领英/找工作
|
||||
- dev: github/代码/仓库/gh/issue/pr/分支/commit
|
||||
- web: 网页/链接/文章/公众号/微信文章/rss/读一下/打开这个
|
||||
- web: 网页/链接/文章/rss/读一下/打开这个
|
||||
- video: youtube/视频/播客/字幕/小宇宙/转录/yt
|
||||
- finance: 雪球/股票/stock/xueqiu/行情/基金
|
||||
metadata:
|
||||
@@ -32,17 +32,17 @@ metadata:
|
||||
|
||||
# Agent Reach — 路由器
|
||||
|
||||
17 平台工具集合。根据用户意图选择对应分类。
|
||||
13 平台工具集合。根据用户意图选择对应分类。
|
||||
|
||||
## 路由表
|
||||
|
||||
| 用户意图 | 分类 | 详细文档 |
|
||||
|---------|------|---------|
|
||||
| 网页搜索/代码搜索 | search | [references/search.md](references/search.md) |
|
||||
| 小红书/抖音/微博/推特/B站/V2EX/Reddit | social | [references/social.md](references/social.md) |
|
||||
| 小红书/推特/B站/V2EX/Reddit | social | [references/social.md](references/social.md) |
|
||||
| 招聘/职位/LinkedIn | career | [references/career.md](references/career.md) |
|
||||
| GitHub/代码 | dev | [references/dev.md](references/dev.md) |
|
||||
| 网页/文章/公众号/RSS | web | [references/web.md](references/web.md) |
|
||||
| 网页/文章/RSS | web | [references/web.md](references/web.md) |
|
||||
| YouTube/B站/播客字幕 | video | [references/video.md](references/video.md) |
|
||||
|
||||
## 零配置快速命令
|
||||
@@ -58,7 +58,7 @@ curl -s "https://r.jina.ai/URL"
|
||||
gh search repos "query" --sort stars --limit 10
|
||||
|
||||
# Twitter 搜索
|
||||
twitter search "query" --limit 10
|
||||
twitter search "query" -n 10
|
||||
|
||||
# YouTube/B站字幕
|
||||
yt-dlp --write-sub --skip-download -o "/tmp/%(id)s" "URL"
|
||||
@@ -92,10 +92,10 @@ mcporter_list_servers()
|
||||
根据用户需求，阅读对应的详细文档：
|
||||
|
||||
- [搜索工具](references/search.md) — Exa AI 搜索
|
||||
- [社交媒体](references/social.md) — 小红书, 抖音, Twitter, B站, V2EX, Reddit
|
||||
- [社交媒体](references/social.md) — 小红书, Twitter, B站, V2EX, Reddit
|
||||
- [职场招聘](references/career.md) — LinkedIn
|
||||
- [开发工具](references/dev.md) — GitHub CLI
|
||||
- [网页阅读](references/web.md) — Jina Reader, 微信公众号, RSS
|
||||
- [网页阅读](references/web.md) — Jina Reader, RSS
|
||||
- [视频播客](references/video.md) — YouTube, B站, 小宇宙
|
||||
|
||||
## 配置渠道
|
||||
|
||||
@@ -1,18 +1,18 @@
|
||||
---
|
||||
name: agent-reach
|
||||
description: >
|
||||
Give your AI agent eyes to see the entire internet.
|
||||
Search and read 17 platforms: Twitter/X, Reddit, YouTube, GitHub, Bilibili,
|
||||
XiaoHongShu, Douyin, Weibo, WeChat Articles, Xiaoyuzhou Podcast, LinkedIn,
|
||||
V2EX, Xueqiu, RSS, Exa web search, and any web page.
|
||||
Zero config for 8 channels. Use when the user asks to search, read, or interact
|
||||
on any supported platform, shares a URL, or asks to search the web.
|
||||
MUST USE when user asks to search, browse, read, or interact with content from any supported platform:
|
||||
Twitter/X, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu,
|
||||
Xiaoyuzhou Podcast, LinkedIn, V2EX, Xueqiu (stocks), RSS, or any web URL.
|
||||
|
||||
Also MUST USE for: web search, look up, research, find, share a URL/link, jobs/recruiting.
|
||||
Routes to CLI tools: xhs-cli, twitter-cli, rdt-cli, gh, yt-dlp, curl+Jina, mcporter.
|
||||
13 platforms, zero config for 6 channels.
|
||||
|
||||
Triggers: "search twitter", "search xiaohongshu", "watch this video",
|
||||
"search the web", "look this up", "research", "youtube transcript",
|
||||
"search reddit", "read this link", "bilibili", "douyin video",
|
||||
"wechat article", "wechat official account", "weibo", "V2EX",
|
||||
"xiaoyuzhou", "podcast", "xueqiu", "stock quote",
|
||||
"install agent reach".
|
||||
"search reddit", "read this link", "bilibili", "V2EX",
|
||||
"xiaoyuzhou", "podcast", "xueqiu", "stock quote", "雪球", "股票".
|
||||
metadata:
|
||||
openclaw:
|
||||
homepage: https://github.com/Panniantong/Agent-Reach
|
||||
@@ -20,7 +20,7 @@ metadata:
|
||||
|
||||
# Agent Reach — Usage Guide
|
||||
|
||||
Upstream tools for 17 platforms. Call them directly.
|
||||
Upstream tools for 13 platforms. Call them directly.
|
||||
|
||||
Run `agent-reach doctor` to check which channels are available.
|
||||
|
||||
@@ -41,15 +41,18 @@ mcporter call 'exa.web_search_exa(query: "query", numResults: 5)'
|
||||
mcporter call 'exa.get_code_context_exa(query: "code question", tokensNum: 3000)'
|
||||
```
|
||||
|
||||
## Twitter/X (bird)
|
||||
## Twitter/X (twitter-cli)
|
||||
|
||||
```bash
|
||||
bird search "query" -n 10 # search
|
||||
bird read URL_OR_ID # read tweet (supports /status/ and /article/ URLs)
|
||||
bird user-tweets @username -n 20 # user timeline
|
||||
bird thread URL_OR_ID # full thread
|
||||
twitter -c search "query" -n 10 # search (-c = compact JSON, LLM-friendly)
|
||||
twitter -c tweet URL_OR_ID # read tweet + replies (supports /status/ URLs)
|
||||
twitter -c article URL_OR_ID # read a Twitter Article
|
||||
twitter -c user-posts @username -n 20 # user timeline
|
||||
twitter -c feed -n 20 # home timeline
|
||||
```
|
||||
|
||||
> Binary is `twitter` (`pipx install twitter-cli`, ≥ 0.8.5). The `bird` name in older docs has been retired. If `search` returns 404, run `pipx upgrade twitter-cli`.
|
||||
|
||||
## YouTube (yt-dlp)
|
||||
|
||||
```bash
|
||||
@@ -105,75 +108,6 @@ mcporter call 'xiaohongshu.publish_content(title: "Title", content: "Body text",
|
||||
> ```
|
||||
> This keeps only: title, content, author, engagement counts, image URLs, and tags.
|
||||
|
||||
## Douyin (mcporter)
|
||||
|
||||
```bash
|
||||
mcporter call 'douyin.parse_douyin_video_info(share_link: "https://v.douyin.com/xxx/")'
|
||||
mcporter call 'douyin.get_douyin_download_link(share_link: "https://v.douyin.com/xxx/")'
|
||||
```
|
||||
|
||||
> No login needed.
|
||||
|
||||
## WeChat Articles
|
||||
|
||||
**Search** (`miku_ai`):
|
||||
```bash
|
||||
# miku_ai is installed inside the agent-reach Python environment.
|
||||
# Use the same interpreter that runs agent-reach (handles pipx / venv installs):
|
||||
AGENT_REACH_PYTHON=$(python3 -c "import agent_reach, sys; print(sys.executable)" 2>/dev/null || echo python3)
|
||||
$AGENT_REACH_PYTHON -c "
|
||||
import asyncio
|
||||
from miku_ai import get_wexin_article
|
||||
async def s():
|
||||
for a in await get_wexin_article('query', 5):
|
||||
print(f'{a[\"title\"]} | {a[\"url\"]}')
|
||||
asyncio.run(s())
|
||||
"
|
||||
```
|
||||
|
||||
**Read** (Camoufox — bypasses WeChat anti-bot):
|
||||
```bash
|
||||
cd ~/.agent-reach/tools/wechat-article-for-ai && python3 main.py "https://mp.weixin.qq.com/s/ARTICLE_ID"
|
||||
```
|
||||
|
||||
> WeChat articles cannot be read with Jina Reader or curl. Use Camoufox.
|
||||
|
||||
## Weibo (mcporter)
|
||||
|
||||
```bash
|
||||
# Trending topics
|
||||
mcporter call 'weibo.get_trendings(limit: 20)'
|
||||
|
||||
# Search users
|
||||
mcporter call 'weibo.search_users(keyword: "Lei Jun", limit: 10)'
|
||||
|
||||
# Get a user profile
|
||||
mcporter call 'weibo.get_profile(uid: "1195230310")'
|
||||
|
||||
# Get a user's feed
|
||||
mcporter call 'weibo.get_feeds(uid: "1195230310", limit: 20)'
|
||||
|
||||
# Get a user's hot posts
|
||||
mcporter call 'weibo.get_hot_feeds(uid: "1195230310", limit: 10)'
|
||||
|
||||
# Search post content
|
||||
mcporter call 'weibo.search_content(keyword: "artificial intelligence", limit: 20)'
|
||||
|
||||
# Search topics
|
||||
mcporter call 'weibo.search_topics(keyword: "AI", limit: 10)'
|
||||
|
||||
# Get post comments
|
||||
mcporter call 'weibo.get_comments(mid: "5099916367123456", limit: 50)'
|
||||
|
||||
# Get fans
|
||||
mcporter call 'weibo.get_fans(uid: "1195230310", limit: 20)'
|
||||
|
||||
# Get followings
|
||||
mcporter call 'weibo.get_followers(uid: "1195230310", limit: 20)'
|
||||
```
|
||||
|
||||
> Zero config. No login needed. Uses the mobile API with auto-generated visitor cookies.
|
||||
|
||||
## Xiaoyuzhou Podcast (groq-whisper + ffmpeg)
|
||||
|
||||
```bash
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# 社交媒体 & 社区
|
||||
|
||||
小红书、抖音、Twitter/X、微博、B站、V2EX、Reddit。
|
||||
小红书、Twitter/X、B站、V2EX、Reddit。
|
||||
|
||||
## 小红书 / XiaoHongShu (xhs-cli)
|
||||
|
||||
@@ -42,42 +42,6 @@ xhs favorites # 可能返回 API error
|
||||
>
|
||||
> **POST 操作风险**: 发帖(post)、评论(comment)、点赞(like) 等写操作在 v0.6.x 可能因签名问题返回 406。如需使用，建议降级到 v0.3.5 (`pipx install xiaohongshu-cli==0.3.5`)。
|
||||
|
||||
## 抖音 / Douyin
|
||||
|
||||
### 安装与配置
|
||||
|
||||
`douyin-mcp-server` 是 **stdio 模式**的 MCP server，需先安装再注册到 mcporter：
|
||||
|
||||
```bash
|
||||
# 1. 安装
|
||||
pipx install douyin-mcp-server
|
||||
|
||||
# 2. 查找安装路径
|
||||
pipx runpip douyin-mcp-server show -f 2>/dev/null | grep "Location" \
|
||||
|| find ~/.local -name "douyin-mcp-server" 2>/dev/null | head -1
|
||||
|
||||
# 3. 注册到 mcporter（使用 stdio 模式，将路径替换为上一步的输出）
|
||||
mcporter config add douyin --command "/path/to/douyin-mcp-server" --scope home
|
||||
```
|
||||
|
||||
> **注意**：`agent-reach install --channels douyin` 暂不支持抖音渠道（抖音在"可选渠道待解锁"列表）。
|
||||
> HTTP 模式（`mcporter config add douyin http://localhost:18070/mcp`）**无法正常工作**，请使用上方 stdio 方式。
|
||||
|
||||
### 用法
|
||||
|
||||
```bash
|
||||
# 解析视频信息
|
||||
mcporter call 'douyin.parse_douyin_video_info(share_link: "https://v.douyin.com/xxx/")'
|
||||
|
||||
# 获取无水印下载链接
|
||||
mcporter call 'douyin.get_douyin_download_link(share_link: "https://v.douyin.com/xxx/")'
|
||||
|
||||
# 提取视频文案
|
||||
mcporter call 'douyin.extract_douyin_text(share_link: "https://v.douyin.com/xxx/")'
|
||||
```
|
||||
|
||||
> **无需登录**
|
||||
|
||||
## Twitter/X (twitter-cli)
|
||||
|
||||
### 稳定命令
|
||||
@@ -122,15 +86,6 @@ twitter likes
|
||||
>
|
||||
> **输出格式**: 建议用 `--yaml` 或 `--json` 获得结构化输出，对 AI agent 更友好。
|
||||
|
||||
## 微博 / Weibo
|
||||
|
||||
```bash
|
||||
# 使用 Jina Reader 读取
|
||||
curl -s "https://r.jina.ai/https://weibo.com/USER_ID/POST_ID"
|
||||
```
|
||||
|
||||
> 微博主要通过网页抓取，推荐使用通用网页读取方式。
|
||||
|
||||
## B站 / Bilibili
|
||||
|
||||
```bash
|
||||
@@ -223,6 +178,6 @@ rdt popular --limit 10
|
||||
rdt all --limit 10
|
||||
```
|
||||
|
||||
> **安装**: `pipx install rdt-cli`（确保 v0.4.2+）。无需登录即可搜索和阅读。
|
||||
> **安装**: `pipx install 'git+https://github.com/public-clis/rdt-cli.git'`（PyPI 版本暂时落后，需从 GitHub 装 v0.4.2+）。需要先登录（`rdt login`）才能搜索和阅读。
|
||||
> 需要登录的功能：`rdt feed --subs-only`（订阅列表）、`rdt saved`（收藏）。
|
||||
> 建议使用 `--yaml` 输出，对 AI agent 更友好。
|
||||
|
||||
@@ -39,6 +39,17 @@ yt-dlp --dump-json "ytsearch5:query"
|
||||
> **字幕注意**: 手动上传的字幕提取可靠；自动生成字幕可能存在行间重复，需后处理。
|
||||
> **评论注意**: `--write-comments` 基于网页抓取（非 YouTube Data API），部分评论可能丢失。
|
||||
|
||||
### 无字幕兜底：Whisper 音频转写
|
||||
|
||||
```bash
|
||||
# 视频没有字幕时的兜底：下载音频并用 Whisper 转写（Groq 免费 key 即可）
|
||||
agent-reach transcribe "https://www.youtube.com/watch?v=VIDEO_ID"
|
||||
agent-reach transcribe ./local_audio.mp3 -o /tmp/transcript.txt
|
||||
```
|
||||
|
||||
> 需要先配置 key：`agent-reach configure groq-key gsk_xxx`（免费，console.groq.com）
|
||||
> 或 `agent-reach configure openai-key sk-xxx`。默认 auto 模式：groq 失败自动降级 openai。
|
||||
|
||||
## B站 / Bilibili (yt-dlp + bili-cli)
|
||||
|
||||
### 视频元数据 (yt-dlp)
|
||||
@@ -71,13 +82,15 @@ bili rank -n 10
|
||||
|
||||
## 小宇宙播客 / Xiaoyuzhou Podcast
|
||||
|
||||
### 转录单集播客
|
||||
### 转录单集播客（可选 --polish 增强标点）
|
||||
|
||||
```bash
|
||||
# 输出 Markdown 文件到 /tmp/
|
||||
~/.agent-reach/tools/xiaoyuzhou/transcribe.sh "https://www.xiaoyuzhoufm.com/episode/EPISODE_ID"
|
||||
# 输出 Markdown 文件到 /tmp/。--polish 让 Llama 3.3 70B 给文稿补中文标点+合理分段
|
||||
~/.agent-reach/tools/xiaoyuzhou/transcribe.sh --polish "https://www.xiaoyuzhoufm.com/episode/EPISODE_ID"
|
||||
```
|
||||
|
||||
> 转写 prompt 已要求 Whisper 输出中文标点；若标点效果仍不理想，可加 `--polish` 用 Groq 上免费的 Llama 3.3 70B 补标点+合理分段（9 分钟播客约多 ~7 秒）。每次转写多一轮 LLM 调用，按需使用。
|
||||
|
||||
### 前置要求
|
||||
|
||||
1. **ffmpeg**: `brew install ffmpeg`
|
||||
@@ -93,18 +106,6 @@ agent-reach doctor
|
||||
|
||||
> 输出 Markdown 文件默认保存到 `/tmp/`。
|
||||
|
||||
## 抖音视频解析
|
||||
|
||||
```bash
|
||||
# 解析视频信息
|
||||
mcporter call 'douyin.parse_douyin_video_info(share_link: "https://v.douyin.com/xxx/")'
|
||||
|
||||
# 获取无水印下载链接
|
||||
mcporter call 'douyin.get_douyin_download_link(share_link: "https://v.douyin.com/xxx/")'
|
||||
```
|
||||
|
||||
> 详见 [social.md](social.md#抖音--douyin)
|
||||
|
||||
## 选择指南
|
||||
|
||||
| 场景 | 推荐工具 |
|
||||
@@ -112,4 +113,4 @@ mcporter call 'douyin.get_douyin_download_link(share_link: "https://v.douyin.com
|
||||
| YouTube 字幕 | yt-dlp |
|
||||
| B站字幕 | yt-dlp |
|
||||
| 播客转录 | 小宇宙 transcribe.sh |
|
||||
| 抖音视频解析 | douyin MCP |
|
||||
| 无字幕音视频 | agent-reach transcribe |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# 网页阅读
|
||||
|
||||
通用网页、微信公众号、RSS。
|
||||
通用网页、RSS。
|
||||
|
||||
## 通用网页 (Jina Reader)
|
||||
|
||||
@@ -29,30 +29,6 @@ mcporter call 'web-reader.webReader(url: "https://example.com", return_format: "
|
||||
|
||||
**适用场景**: 需要更精确控制输出格式时使用。
|
||||
|
||||
## 微信公众号 / WeChat Articles
|
||||
|
||||
### 搜索公众号文章（通过 Exa）
|
||||
|
||||
```bash
|
||||
# 搜索微信公众号文章
|
||||
mcporter call 'exa.web_search_exa(query: "搜索关键词", numResults: 5, includeDomains: ["mp.weixin.qq.com"])'
|
||||
```
|
||||
|
||||
### 阅读公众号文章全文（通过 Exa）
|
||||
|
||||
```bash
|
||||
# 抓取文章全文
|
||||
mcporter call 'exa.crawling_exa(urls: ["https://mp.weixin.qq.com/s/ARTICLE_ID"], maxCharacters: 10000)'
|
||||
```
|
||||
|
||||
### 可选：Camoufox 阅读（反爬更强）
|
||||
|
||||
```bash
|
||||
cd ~/.agent-reach/tools/wechat-article-for-ai && python3 main.py "https://mp.weixin.qq.com/s/ARTICLE_ID"
|
||||
```
|
||||
|
||||
> **注意**: Jina Reader 无法读取微信文章（被 CAPTCHA 拦截），推荐用 Exa。
|
||||
|
||||
## RSS (feedparser)
|
||||
|
||||
```python
|
||||
@@ -71,6 +47,4 @@ for e in feedparser.parse('FEED_URL').entries[:5]:
|
||||
|-----|---------|
|
||||
| 通用网页 | Jina Reader (`curl r.jina.ai`) |
|
||||
| 需要图片/格式控制 | web-reader MCP |
|
||||
| 微信公众号 | Exa (搜索+阅读) / Camoufox (可选阅读) |
|
||||
| RSS 订阅 | feedparser |
|
||||
| 微博/知乎等 | Jina Reader |
|
||||
|
||||
@@ -0,0 +1,253 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Whisper audio transcription with Groq → OpenAI fallback.
|
||||
|
||||
Downloads audio (yt-dlp), compresses + chunks (ffmpeg), and posts to a
|
||||
Whisper-compatible API. Defaults to Groq's free `whisper-large-v3` and falls
|
||||
back to OpenAI's `whisper-1` on HTTP error.
|
||||
|
||||
Public entry point:
|
||||
transcribe(source, *, provider="auto", out_dir=None, config=None) -> str
|
||||
|
||||
Designed to be importable from channels (e.g. YouTubeChannel.transcribe).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
import requests
|
||||
|
||||
from agent_reach.config import Config
|
||||
|
||||
# Whisper API limit is 25MB; leave headroom for multipart overhead.
|
||||
SIZE_LIMIT_BYTES = 24 * 1024 * 1024
|
||||
CHUNK_SECONDS = 600 # 10 min — small enough that boundary cuts rarely lose meaning
|
||||
|
||||
PROVIDERS = {
|
||||
"groq": {
|
||||
"endpoint": "https://api.groq.com/openai/v1/audio/transcriptions",
|
||||
"model": "whisper-large-v3",
|
||||
"key_field": "groq_api_key",
|
||||
},
|
||||
"openai": {
|
||||
"endpoint": "https://api.openai.com/v1/audio/transcriptions",
|
||||
"model": "whisper-1",
|
||||
"key_field": "openai_api_key",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class TranscribeError(RuntimeError):
|
||||
"""Raised when transcription cannot complete."""
|
||||
|
||||
|
||||
class MissingDependency(TranscribeError):
|
||||
"""Raised when a required external binary is missing."""
|
||||
|
||||
|
||||
class NoProviderConfigured(TranscribeError):
|
||||
"""Raised when no provider has an API key configured."""
|
||||
|
||||
|
||||
def _require(binary: str) -> None:
|
||||
if not shutil.which(binary):
|
||||
raise MissingDependency(f"{binary} not found in PATH")
|
||||
|
||||
|
||||
def _run(cmd: List[str]) -> None:
|
||||
"""Run a subprocess, raising TranscribeError on nonzero exit."""
|
||||
proc = subprocess.run(cmd, capture_output=True, text=True)
|
||||
if proc.returncode != 0:
|
||||
raise TranscribeError(
|
||||
f"{cmd[0]} failed (exit {proc.returncode}): {proc.stderr.strip()[:300]}"
|
||||
)
|
||||
|
||||
|
||||
def download_audio(url: str, out_dir: Path) -> Path:
|
||||
"""Download audio with yt-dlp into out_dir; return the resulting file path."""
|
||||
_require("yt-dlp")
|
||||
template = out_dir / "source.%(ext)s"
|
||||
_run(
|
||||
[
|
||||
"yt-dlp",
|
||||
"-x",
|
||||
"--audio-format",
|
||||
"m4a",
|
||||
"--audio-quality",
|
||||
"0",
|
||||
"-o",
|
||||
str(template),
|
||||
url,
|
||||
]
|
||||
)
|
||||
files = sorted(out_dir.glob("source.*"))
|
||||
if not files:
|
||||
raise TranscribeError("yt-dlp produced no output file")
|
||||
return files[0]
|
||||
|
||||
|
||||
def compress_audio(src: Path, out_dir: Path) -> Path:
|
||||
"""Re-encode to mono / 16kHz / 32kbps m4a — keeps most content under 25MB."""
|
||||
_require("ffmpeg")
|
||||
dst = out_dir / "compressed.m4a"
|
||||
_run(
|
||||
[
|
||||
"ffmpeg",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-y",
|
||||
"-i",
|
||||
str(src),
|
||||
"-vn",
|
||||
"-ac",
|
||||
"1",
|
||||
"-ar",
|
||||
"16000",
|
||||
"-b:a",
|
||||
"32k",
|
||||
str(dst),
|
||||
]
|
||||
)
|
||||
return dst
|
||||
|
||||
|
||||
def chunk_audio(src: Path, out_dir: Path, segment_seconds: int = CHUNK_SECONDS) -> List[Path]:
|
||||
"""Split src into segments. Re-encodes each segment so cuts align to keyframes."""
|
||||
_require("ffmpeg")
|
||||
pattern = out_dir / "chunk_%03d.m4a"
|
||||
_run(
|
||||
[
|
||||
"ffmpeg",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-y",
|
||||
"-i",
|
||||
str(src),
|
||||
"-f",
|
||||
"segment",
|
||||
"-segment_time",
|
||||
str(segment_seconds),
|
||||
"-ac",
|
||||
"1",
|
||||
"-ar",
|
||||
"16000",
|
||||
"-b:a",
|
||||
"32k",
|
||||
str(pattern),
|
||||
]
|
||||
)
|
||||
chunks = sorted(out_dir.glob("chunk_*.m4a"))
|
||||
if not chunks:
|
||||
raise TranscribeError("ffmpeg produced no chunks")
|
||||
return chunks
|
||||
|
||||
|
||||
def _provider_key(provider: str, config: Config) -> Optional[str]:
|
||||
field = PROVIDERS[provider]["key_field"]
|
||||
val = config.get(field)
|
||||
return val or None
|
||||
|
||||
|
||||
def transcribe_chunk(
|
||||
chunk: Path,
|
||||
provider: str,
|
||||
*,
|
||||
config: Optional[Config] = None,
|
||||
timeout: int = 120,
|
||||
) -> str:
|
||||
"""Transcribe one chunk via the named provider. Raises TranscribeError on failure."""
|
||||
if provider not in PROVIDERS:
|
||||
raise TranscribeError(f"unknown provider: {provider}")
|
||||
cfg = config or Config()
|
||||
key = _provider_key(provider, cfg)
|
||||
if not key:
|
||||
raise NoProviderConfigured(
|
||||
f"{provider}: missing {PROVIDERS[provider]['key_field']} "
|
||||
f"(configure with `agent-reach configure {provider}-key ...`)"
|
||||
)
|
||||
|
||||
info = PROVIDERS[provider]
|
||||
with chunk.open("rb") as fh:
|
||||
try:
|
||||
resp = requests.post(
|
||||
info["endpoint"],
|
||||
headers={"Authorization": f"Bearer {key}"},
|
||||
files={"file": (chunk.name, fh, "audio/m4a")},
|
||||
data={"model": info["model"], "response_format": "text"},
|
||||
timeout=timeout,
|
||||
)
|
||||
except requests.RequestException as e:
|
||||
raise TranscribeError(f"{provider}: network error: {e}") from e
|
||||
|
||||
if not resp.ok:
|
||||
raise TranscribeError(f"{provider}: HTTP {resp.status_code}: {resp.text[:300]}")
|
||||
return resp.text
|
||||
|
||||
|
||||
def _provider_order(provider: str) -> List[str]:
|
||||
if provider == "auto":
|
||||
return ["groq", "openai"]
|
||||
if provider in PROVIDERS:
|
||||
return [provider]
|
||||
raise TranscribeError(f"unknown provider: {provider} (use groq|openai|auto)")
|
||||
|
||||
|
||||
def transcribe(
|
||||
source: str,
|
||||
*,
|
||||
provider: str = "auto",
|
||||
out_dir: Optional[Path] = None,
|
||||
config: Optional[Config] = None,
|
||||
) -> str:
|
||||
"""Transcribe a URL or local file path. Returns the joined transcript text.
|
||||
|
||||
`provider` is one of `auto` (groq → openai), `groq`, or `openai`.
|
||||
`out_dir` defaults to a fresh temp directory; intermediate files stay there.
|
||||
"""
|
||||
cfg = config or Config()
|
||||
order = _provider_order(provider)
|
||||
|
||||
# Validate at least one provider is configured before doing expensive work.
|
||||
if not any(_provider_key(p, cfg) for p in order):
|
||||
names = ", ".join(PROVIDERS[p]["key_field"] for p in order)
|
||||
raise NoProviderConfigured(f"no provider key configured (need one of: {names})")
|
||||
|
||||
work_dir = Path(out_dir) if out_dir else Path(tempfile.mkdtemp(prefix="transcribe-"))
|
||||
work_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
src_path = Path(source)
|
||||
if src_path.is_file():
|
||||
audio = src_path
|
||||
else:
|
||||
audio = download_audio(source, work_dir)
|
||||
|
||||
compressed = compress_audio(audio, work_dir)
|
||||
if compressed.stat().st_size <= SIZE_LIMIT_BYTES:
|
||||
chunks = [compressed]
|
||||
else:
|
||||
chunks = chunk_audio(compressed, work_dir)
|
||||
|
||||
pieces: List[str] = []
|
||||
for chunk in chunks:
|
||||
text = _transcribe_with_fallback(chunk, order, cfg)
|
||||
pieces.append(text.strip())
|
||||
return "\n".join(p for p in pieces if p)
|
||||
|
||||
|
||||
def _transcribe_with_fallback(chunk: Path, order: List[str], config: Config) -> str:
|
||||
"""Try each provider in order; return first success or raise the last error."""
|
||||
last_err: Optional[Exception] = None
|
||||
for p in order:
|
||||
if not _provider_key(p, config):
|
||||
# Skip silently — caller already validated at least one is configured.
|
||||
continue
|
||||
try:
|
||||
return transcribe_chunk(chunk, p, config=config)
|
||||
except TranscribeError as e:
|
||||
last_err = e
|
||||
continue
|
||||
raise TranscribeError(f"all providers failed for {chunk.name}: {last_err}")
|
||||
@@ -0,0 +1,26 @@
|
||||
"""Subprocess helpers for consistent cross-platform text handling."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from collections.abc import Mapping
|
||||
|
||||
UTF8_ENV = {
|
||||
"PYTHONUTF8": "1",
|
||||
"PYTHONIOENCODING": "utf-8",
|
||||
}
|
||||
|
||||
|
||||
def utf8_subprocess_env(base: Mapping[str, str] | None = None) -> dict[str, str]:
|
||||
"""Return an environment that forces Python child processes into UTF-8 mode."""
|
||||
env = dict(base or os.environ)
|
||||
env.update(UTF8_ENV)
|
||||
return env
|
||||
|
||||
|
||||
def mcporter_utf8_env_args() -> list[str]:
|
||||
"""Return mcporter --env arguments for UTF-8 Python stdio servers."""
|
||||
args = []
|
||||
for key, value in UTF8_ENV.items():
|
||||
args.extend(["--env", f"{key}={value}"])
|
||||
return args
|
||||
+16
-41
@@ -64,10 +64,7 @@ Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/ma
|
||||
| 🌐 **Web** | Read | Zero config | Any URL → clean Markdown ([Jina Reader](https://github.com/jina-ai/reader) ⭐9.8K) |
|
||||
| 🐦 **Twitter/X** | Read · Search | Cookie | Cookie unlocks search, timeline, tweet reading, articles ([twitter-cli](https://github.com/public-clis/twitter-cli)) |
|
||||
| 📕 **XiaoHongShu** | Read · Search · **Post · Comment · Like** | Cookie | `pipx install xiaohongshu-cli` + `xhs login` ([xhs-cli](https://github.com/jackwener/xiaohongshu-cli)) |
|
||||
| 🎵 **Douyin** | Video parsing · Watermark-free download | mcporter | Via [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server), no login needed |
|
||||
| 💼 **LinkedIn** | Jina Reader (public pages) | Full profiles, companies, job search | Tell your Agent "help me set up LinkedIn" |
|
||||
| 💬 **WeChat Articles** | Search + Read | Zero config | Search + read WeChat Official Account articles via Exa (zero config) + optional [Camoufox](https://github.com/daijro/camoufox) |
|
||||
| 📰 **Weibo** | Trending · Search · Feeds · Comments | Zero config | Hot search, content/user/topic search, feeds, comments ([mcp-server-weibo](https://github.com/Panniantong/mcp-server-weibo)) |
|
||||
| 💻 **V2EX** | Hot topics · Node topics · Topic detail + replies · User profile | Zero config | Public JSON API, no auth required. Great for tech community content |
|
||||
| 📈 **Xueqiu (雪球)** | Stock quotes · Search · Hot posts · Hot stocks | Browser cookie | Tell your Agent "help me set up Xueqiu" |
|
||||
| 🎙️ **Xiaoyuzhou Podcast** | Transcription | Free API key | Podcast audio → full text transcript via Groq Whisper (free) |
|
||||
@@ -84,6 +81,15 @@ Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/ma
|
||||
|
||||
## Quick Start
|
||||
|
||||
> ⚠️ **OpenClaw users: enable `exec` permission first**
|
||||
>
|
||||
> Agent Reach relies on the Agent running shell commands (`pip install`, `mcporter`, `twitter`, etc.). If your OpenClaw uses the default `messaging` tool profile, the Agent won't be able to run them. **Enable `exec` before installing:**
|
||||
>
|
||||
> ```bash
|
||||
> openclaw config set tools.profile "coding"
|
||||
> ```
|
||||
> Or set `"tools": { "profile": "coding" }` in `~/.openclaw/openclaw.json`. After changing it, restart the Gateway (`openclaw gateway restart`) and start a new conversation. Other platforms (Claude Code, Cursor, Windsurf, etc.) are not affected.
|
||||
|
||||
Copy this to your AI Agent (Claude Code, OpenClaw, Cursor, etc.):
|
||||
|
||||
```
|
||||
@@ -97,6 +103,12 @@ The Agent auto-installs, detects your environment, and tells you what's ready.
|
||||
> Update Agent Reach: https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/update.md
|
||||
> ```
|
||||
|
||||
> 🛡️ **Worried about security?** Use safe mode — it won't auto-install system packages, it only tells you what you need:
|
||||
> ```
|
||||
> Install Agent Reach (safe mode): https://raw.githubusercontent.com/Panniantong/agent-reach/main/docs/install.md
|
||||
> Use the --safe flag during install
|
||||
> ```
|
||||
|
||||
<details>
|
||||
<summary>Manual install</summary>
|
||||
|
||||
@@ -207,7 +219,6 @@ channels/
|
||||
├── bilibili.py → yt-dlp ← swap to bilibili-api…
|
||||
├── reddit.py → rdt-cli ← search + read, cookie auth required
|
||||
├── xiaohongshu.py → mcporter MCP ← swap to other XHS tools…
|
||||
├── douyin.py → mcporter MCP ← swap to other Douyin tools…
|
||||
├── linkedin.py → linkedin-mcp ← swap to LinkedIn API…
|
||||
├── rss.py → feedparser ← swap to atoma…
|
||||
├── exa_search.py → mcporter MCP ← swap to Tavily, SerpAPI…
|
||||
@@ -229,10 +240,7 @@ Each channel file only checks whether its upstream tool is installed and working
|
||||
| GitHub | [gh CLI](https://cli.github.com) | Official tool, full API after auth |
|
||||
| Read RSS | [feedparser](https://github.com/kurtmckee/feedparser) | Python ecosystem standard, 2.3K stars |
|
||||
| XiaoHongShu | [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) | 1.5K stars, pipx install, search/read/comment/post |
|
||||
| Douyin | [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) | MCP server, no login needed, video parsing + watermark-free download |
|
||||
| LinkedIn | [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server) | 1.2K stars, MCP server, browser automation |
|
||||
| WeChat Articles | [Exa](https://exa.ai) (search + read) + [Camoufox](https://github.com/daijro/camoufox) (optional) | Zero-config search + full article reading |
|
||||
| Weibo | `mcporter` | `mcporter call 'weibo.get_trendings(limit: 10)'` |
|
||||
| Xiaoyuzhou Podcast | `transcribe.sh` | `bash ~/.agent-reach/tools/xiaoyuzhou/transcribe.sh <URL>` |
|
||||
|
||||
> 📌 These are the *current* choices. Don't like one? Swap out the file. That's the whole point of scaffolding.
|
||||
@@ -295,44 +303,11 @@ Agent Reach uses twitter-cli which accesses Twitter via cookie auth — same as
|
||||
Install `pipx install xiaohongshu-cli`, then `xhs login` (auto-extracts cookies from browser). Your agent can then use `xhs search "query"` to search notes, `xhs read NOTE_ID` to read details, `xhs comments NOTE_ID` to view comments. No Docker needed.
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>How to parse Douyin / 抖音 videos with AI agent?</strong></summary>
|
||||
|
||||
Install douyin-mcp-server, then your agent can use `mcporter call 'douyin.parse_douyin_video_info(share_link: "share_url")'` to parse video info and get watermark-free download links. No login required — just share the Douyin link. See https://github.com/yzfly/douyin-mcp-server
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>How to extract scripts from both Douyin and XiaoHongShu with one MCP?</strong></summary>
|
||||
|
||||
If you want one MCP server that can handle:
|
||||
|
||||
- Douyin videos
|
||||
- XiaoHongShu video notes
|
||||
- XiaoHongShu image notes
|
||||
|
||||
and directly write `script.md` + `info.json`, you can point the existing `douyin` mcporter alias at:
|
||||
|
||||
- https://github.com/JNHFlow21/social-post-extractor-mcp
|
||||
|
||||
It keeps backward compatibility with:
|
||||
|
||||
- `parse_douyin_video_info`
|
||||
- `get_douyin_download_link`
|
||||
- `extract_douyin_text`
|
||||
|
||||
and adds unified tools:
|
||||
|
||||
- `parse_social_post_info`
|
||||
- `extract_social_post_script`
|
||||
|
||||
This is useful when your agent workflow is “paste a link, get a script file”.
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Credits
|
||||
|
||||
[twitter-cli](https://github.com/public-clis/twitter-cli) · [rdt-cli](https://github.com/public-clis/rdt-cli) · [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) · [bili-cli](https://github.com/public-clis/bilibili-cli) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [Jina Reader](https://github.com/jina-ai/reader) · [Exa](https://exa.ai) · [mcporter](https://github.com/nicobailon/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [douyin-mcp-server](https://github.com/yzfly/douyin-mcp-server) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||
[twitter-cli](https://github.com/public-clis/twitter-cli) · [rdt-cli](https://github.com/public-clis/rdt-cli) · [xhs-cli](https://github.com/jackwener/xiaohongshu-cli) · [bili-cli](https://github.com/public-clis/bilibili-cli) · [yt-dlp](https://github.com/yt-dlp/yt-dlp) · [Jina Reader](https://github.com/jina-ai/reader) · [Exa](https://exa.ai) · [mcporter](https://github.com/nicobailon/mcporter) · [feedparser](https://github.com/kurtmckee/feedparser) · [linkedin-scraper-mcp](https://github.com/stickerdaniel/linkedin-mcp-server)
|
||||
|
||||
## Contact
|
||||
|
||||
|
||||
+18
-91
@@ -40,7 +40,7 @@ All Agent Reach files go in dedicated directories — **never in the agent works
|
||||
| Purpose | Directory | Example |
|
||||
|---------|-----------|---------|
|
||||
| Config & tokens | `~/.agent-reach/` | `~/.agent-reach/config.json` |
|
||||
| Upstream tool repos | `~/.agent-reach/tools/` | `~/.agent-reach/tools/douyin-mcp-server/` |
|
||||
| Upstream tool repos | `~/.agent-reach/tools/` | `~/.agent-reach/tools/xiaoyuzhou/` |
|
||||
| Temporary files | `/tmp/` | `/tmp/yt-dlp-output/` |
|
||||
| Skills | `~/.openclaw/skills/agent-reach/` | SKILL.md |
|
||||
|
||||
@@ -61,6 +61,19 @@ pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
||||
agent-reach install --env=auto
|
||||
```
|
||||
|
||||
> 💡 **Windows / Microsoft Store Python alias?**
|
||||
> 如果 `python3 --version` 打开 Microsoft Store，或 `where python3` 指向
|
||||
> `...\AppData\Local\Microsoft\WindowsApps\python3.exe`，说明 `python3` 是 Windows
|
||||
> 的 Store alias，不是可用的 Python 安装。请改用 Python Launcher `py -3`，或实际安装目录里的 `python.exe`。
|
||||
>
|
||||
> PowerShell 示例：
|
||||
> ```powershell
|
||||
> py -3 -m venv $env:USERPROFILE\.agent-reach-venv
|
||||
> $env:USERPROFILE\.agent-reach-venv\Scripts\Activate.ps1
|
||||
> python -m pip install https://github.com/Panniantong/agent-reach/archive/main.zip
|
||||
> agent-reach install --env=auto
|
||||
> ```
|
||||
|
||||
This installs core infrastructure (gh CLI, Node.js, mcporter, Exa search, yt-dlp config) and activates these zero-config channels:
|
||||
|
||||
- Web (Jina Reader), YouTube, GitHub, RSS, Exa Search, V2EX, Bilibili (basic)
|
||||
@@ -84,26 +97,23 @@ After installing the basics, **ask the user** which additional channels they nee
|
||||
> 还有这些可选渠道，你需要哪些？
|
||||
>
|
||||
> - 🐦 **Twitter/X** — 搜推文、看时间线（需要登录 Cookie）
|
||||
> - 📰 **微博** — 热搜、搜索、用户动态
|
||||
> - 💬 **微信公众号** — 搜索和阅读公众号文章
|
||||
> - 📈 **雪球** — 股票行情、热门帖子（需要登录 Cookie）
|
||||
> - 🎙️ **小宇宙播客** — 音频转文字（需要免费 Groq Key）
|
||||
> - 📕 **小红书** — 阅读、搜索、发帖（需要登录）
|
||||
> - 📖 **Reddit** — 搜索和阅读帖子
|
||||
> - 📺 **B站完整版** — 热门、排行、搜索
|
||||
> - 🎵 **抖音** — 视频解析
|
||||
> - 💼 **LinkedIn** — Profile、职位搜索
|
||||
>
|
||||
> 告诉我你要哪些，比如"帮我装微博和 Twitter"。或者说"全部装"。
|
||||
> 告诉我你要哪些，比如"帮我装小红书和 Twitter"。或者说"全部装"。
|
||||
|
||||
Based on the user's choice, run:
|
||||
|
||||
```bash
|
||||
agent-reach install --env=auto --channels=twitter,weibo # Example: user chose Twitter + Weibo
|
||||
agent-reach install --env=auto --channels=twitter,xiaohongshu # Example: user chose Twitter + XHS
|
||||
agent-reach install --env=auto --channels=all # User wants everything
|
||||
```
|
||||
|
||||
Supported channel names: `twitter`, `weibo`, `wechat`, `xiaoyuzhou`, `xueqiu`, `xiaohongshu`, `reddit`, `bilibili`, `douyin`, `linkedin`, `all`
|
||||
Supported channel names: `twitter`, `xiaoyuzhou`, `xueqiu`, `xiaohongshu`, `reddit`, `bilibili`, `linkedin`, `all`
|
||||
|
||||
### Step 3: Fix what's broken
|
||||
|
||||
@@ -189,18 +199,6 @@ xhs login
|
||||
> mcporter config add xiaohongshu http://localhost:18060/mcp
|
||||
> ```
|
||||
|
||||
**微博 / Weibo (mcp-server-weibo):**
|
||||
> "微博已默认安装，装好即用。可搜索微博内容、查看热搜、获取用户动态和评论。"
|
||||
|
||||
如果自动安装失败，手动安装：
|
||||
|
||||
```bash
|
||||
pip install git+https://github.com/Panniantong/mcp-server-weibo.git
|
||||
mcporter config add weibo --command 'mcp-server-weibo'
|
||||
```
|
||||
|
||||
> 无需登录、无需 Cookie、无需代理。海外服务器也可以直接访问。
|
||||
|
||||
**雪球 / Xueqiu (股票行情 + 热门帖子):**
|
||||
> "雪球需要登录后的 Cookie。请先在 Chrome 里登录 xueqiu.com，然后运行："
|
||||
|
||||
@@ -239,75 +237,6 @@ agent-reach configure groq-key gsk_xxxxx
|
||||
> - 转录质量高（Whisper large-v3），但不区分说话人
|
||||
> - 2 小时以上的播客建议分批处理
|
||||
|
||||
**抖音 / Douyin (douyin-mcp-server):**
|
||||
> "抖音视频解析需要一个 MCP 服务。安装 douyin-mcp-server 后即可解析视频、获取无水印下载链接。"
|
||||
|
||||
```bash
|
||||
# 1. 安装
|
||||
pip install douyin-mcp-server
|
||||
|
||||
# 2. 启动 HTTP 服务（端口 18070）
|
||||
# 方式一：用 uv（推荐）
|
||||
mkdir -p ~/.agent-reach/tools && cd ~/.agent-reach/tools
|
||||
git clone https://github.com/yzfly/douyin-mcp-server.git && cd douyin-mcp-server
|
||||
uv sync && uv run python run_http.py
|
||||
|
||||
# 方式二：直接用 Python 启动
|
||||
python -c "
|
||||
from douyin_mcp_server.server import mcp
|
||||
mcp.settings.host = '127.0.0.1'
|
||||
mcp.settings.port = 18070
|
||||
mcp.run(transport='streamable-http')
|
||||
"
|
||||
|
||||
# 3. 注册到 mcporter
|
||||
mcporter config add douyin http://localhost:18070/mcp
|
||||
```
|
||||
|
||||
> 无需认证即可解析视频信息和获取下载链接。
|
||||
> 如需 AI 语音识别提取文案功能，需要配置硅基流动 API Key（`export API_KEY="sk-xxx"`）。
|
||||
>
|
||||
> 详见 https://github.com/yzfly/douyin-mcp-server
|
||||
|
||||
**可选实现：Douyin + XiaoHongShu unified extractor**
|
||||
> "如果你想把抖音和小红书统一成一个 MCP，并直接输出 `script.md` 和 `info.json`，可以改用 social-post-extractor-mcp。"
|
||||
|
||||
适用场景：
|
||||
|
||||
- 抖音视频转文字稿
|
||||
- 小红书视频笔记转文字稿
|
||||
- 小红书图文笔记正文 + 图片文字提取
|
||||
|
||||
兼容性：
|
||||
|
||||
- 仍然可以注册成 `douyin` 这个 mcporter server 名称
|
||||
- 兼容旧工具名 `parse_douyin_video_info` / `get_douyin_download_link` / `extract_douyin_text`
|
||||
- 同时新增 `parse_social_post_info` / `extract_social_post_script`
|
||||
|
||||
示例配置：
|
||||
|
||||
```bash
|
||||
git clone https://github.com/JNHFlow21/social-post-extractor-mcp.git
|
||||
cd social-post-extractor-mcp
|
||||
uv sync
|
||||
|
||||
mcporter config add douyin \
|
||||
--command /bin/zsh \
|
||||
--arg -lc \
|
||||
--arg "cd '$PWD' && exec '.venv/bin/python' -m social_post_extractor_mcp" \
|
||||
--env ASR_PROVIDER=bailian \
|
||||
--env ASR_MODEL=paraformer-v2 \
|
||||
--env VISION_PROVIDER=bailian \
|
||||
--env VISION_MODEL=qwen3-vl-flash \
|
||||
--env CLEAN_PROVIDER=bailian \
|
||||
--env CLEAN_MODEL=qwen-flash \
|
||||
--env BAILIAN_API_KEY=YOUR_BAILIAN_API_KEY
|
||||
```
|
||||
|
||||
> 这个实现更适合“把链接直接交给 Agent，然后拿到脚本文件”的工作流。
|
||||
>
|
||||
> 详见 https://github.com/JNHFlow21/social-post-extractor-mcp
|
||||
|
||||
**LinkedIn (可选 — linkedin-scraper-mcp):**
|
||||
> "LinkedIn 基本内容可通过 Jina Reader 读取。完整功能（Profile 详情、职位搜索）需要 linkedin-scraper-mcp。"
|
||||
|
||||
@@ -375,7 +304,7 @@ If the user wants a different agent to handle it, let them choose.
|
||||
| Command | What it does |
|
||||
|---------|-------------|
|
||||
| `agent-reach install --env=auto` | Install core channels (lightweight, zero-config) |
|
||||
| `agent-reach install --env=auto --channels=twitter,weibo` | Install core + optional channels |
|
||||
| `agent-reach install --env=auto --channels=twitter,xiaohongshu` | Install core + optional channels |
|
||||
| `agent-reach install --env=auto --channels=all` | Install everything |
|
||||
| `agent-reach install --env=auto --safe` | Safe setup (no auto system changes) |
|
||||
| `agent-reach install --env=auto --dry-run` | Preview what would be done |
|
||||
@@ -398,8 +327,6 @@ After installation, use upstream tools directly. See SKILL.md for the full comma
|
||||
| Web | `curl` + Jina | `curl -s "https://r.jina.ai/URL"` |
|
||||
| Exa Search | `mcporter` | `mcporter call 'exa.web_search_exa(...)'` |
|
||||
| 小红书 | `mcporter` | `mcporter call 'xiaohongshu.search_feeds(...)'` |
|
||||
| 微博 | `mcporter` | `mcporter call 'weibo.get_trendings(limit: 10)'` |
|
||||
| 小宇宙播客 | `transcribe.sh` | `bash ~/.agent-reach/tools/xiaoyuzhou/transcribe.sh <URL>` |
|
||||
| 抖音 | `mcporter` | `mcporter call 'douyin.parse_douyin_video_info(...)'` |
|
||||
| LinkedIn | `mcporter` | `mcporter call 'linkedin.get_person_profile(...)'` |
|
||||
| RSS | `feedparser` | `python3 -c "import feedparser; ..."` |
|
||||
|
||||
+2
-2
@@ -50,8 +50,8 @@ Run these commands to ensure all upstream CLI tools are installed. Skip any that
|
||||
# Twitter/X — twitter-cli (replaces deprecated bird CLI)
|
||||
which twitter >/dev/null 2>&1 || pipx install twitter-cli 2>/dev/null || uv tool install twitter-cli 2>/dev/null
|
||||
|
||||
# Reddit — rdt-cli (replaces Exa-based approach)
|
||||
which rdt >/dev/null 2>&1 || pipx install rdt-cli 2>/dev/null || uv tool install rdt-cli 2>/dev/null
|
||||
# Reddit — rdt-cli (replaces Exa-based approach; PyPI lags, install from GitHub)
|
||||
which rdt >/dev/null 2>&1 || pipx install 'git+https://github.com/public-clis/rdt-cli.git' 2>/dev/null || uv tool install --from 'git+https://github.com/public-clis/rdt-cli.git' rdt-cli 2>/dev/null
|
||||
|
||||
# XiaoHongShu — xhs-cli (replaces Docker MCP)
|
||||
which xhs >/dev/null 2>&1 || pipx install xiaohongshu-cli 2>/dev/null || uv tool install xiaohongshu-cli 2>/dev/null
|
||||
|
||||
+1
-7
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "agent-reach"
|
||||
version = "1.4.0"
|
||||
version = "1.4.2"
|
||||
description = "Give your AI Agent eyes to see the entire internet. Search + Read 10+ platforms."
|
||||
readme = "README.md"
|
||||
license = {text = "MIT"}
|
||||
@@ -64,12 +64,6 @@ build-backend = "hatchling.build"
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["agent_reach"]
|
||||
|
||||
[tool.hatch.build.targets.wheel.force-include]
|
||||
"agent_reach/guides" = "agent_reach/guides"
|
||||
# Keep the whole skill directory so SKILL.md, SKILL_en.md, and references/ ship together.
|
||||
"agent_reach/skill" = "agent_reach/skill"
|
||||
"agent_reach/scripts" = "agent_reach/scripts"
|
||||
|
||||
[tool.ruff]
|
||||
target-version = "py310"
|
||||
line-length = 100
|
||||
|
||||
@@ -90,49 +90,6 @@ def test_youtube_ok_when_deno_installed(monkeypatch):
|
||||
assert status == "ok"
|
||||
|
||||
|
||||
def test_douyin_check_does_not_call_with_invalid_url(monkeypatch, tmp_path):
|
||||
"""Douyin check should use 'mcporter list' instead of calling with a hardcoded URL."""
|
||||
import subprocess
|
||||
|
||||
from agent_reach.channels.douyin import DouyinChannel
|
||||
|
||||
calls = []
|
||||
original_run = subprocess.run
|
||||
|
||||
def tracking_run(cmd, **kwargs):
|
||||
calls.append(cmd)
|
||||
# Simulate mcporter config list returning douyin
|
||||
if "config" in cmd and "list" in cmd:
|
||||
|
||||
class R:
|
||||
stdout = "douyin http://localhost:18070/mcp"
|
||||
returncode = 0
|
||||
|
||||
return R()
|
||||
# Simulate mcporter list douyin returning tools
|
||||
if "list" in cmd and "douyin" in cmd:
|
||||
|
||||
class R:
|
||||
stdout = "parse_douyin_video_info"
|
||||
returncode = 0
|
||||
|
||||
return R()
|
||||
return original_run(cmd, **kwargs)
|
||||
|
||||
monkeypatch.setattr(
|
||||
"shutil.which", lambda cmd: "/usr/bin/mcporter" if cmd == "mcporter" else None
|
||||
)
|
||||
monkeypatch.setattr("subprocess.run", tracking_run)
|
||||
|
||||
ch = DouyinChannel()
|
||||
status, _msg = ch.check()
|
||||
|
||||
# Should NOT contain any hardcoded douyin.com URL in subprocess calls
|
||||
for call in calls:
|
||||
call_str = " ".join(call) if isinstance(call, list) else str(call)
|
||||
assert "https://www.douyin.com" not in call_str
|
||||
|
||||
|
||||
def test_channel_can_handle_contract():
|
||||
url_samples = {
|
||||
"github": "https://github.com/panniantong/agent-reach",
|
||||
@@ -141,9 +98,7 @@ def test_channel_can_handle_contract():
|
||||
"reddit": "https://reddit.com/r/python",
|
||||
"bilibili": "https://www.bilibili.com/video/BV1xx411",
|
||||
"xiaohongshu": "https://www.xiaohongshu.com/explore/123",
|
||||
"douyin": "https://www.douyin.com/video/123",
|
||||
"linkedin": "https://www.linkedin.com/in/test",
|
||||
"weibo": "https://weibo.com/u/1749127163",
|
||||
"rss": "https://example.com/feed.xml",
|
||||
"xueqiu": "https://xueqiu.com/S/SH600519",
|
||||
"exa_search": "https://example.com",
|
||||
|
||||
@@ -654,6 +654,8 @@ class TestRedditChannel:
|
||||
assert status == "off"
|
||||
assert "rdt-cli" in msg
|
||||
assert "public-clis/rdt-cli" in msg
|
||||
assert "git+https://github.com/public-clis/rdt-cli.git" in msg
|
||||
assert "rdt-cli>=0.4.2" not in msg
|
||||
|
||||
def test_reports_ok_when_authenticated(self, monkeypatch):
|
||||
monkeypatch.setattr(shutil, "which", lambda _: "/usr/local/bin/rdt")
|
||||
|
||||
+45
-2
@@ -1,9 +1,12 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Tests for Agent Reach CLI."""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
from unittest.mock import patch
|
||||
import agent_reach.cli as cli
|
||||
from agent_reach.cli import main
|
||||
|
||||
@@ -30,6 +33,21 @@ class TestCLI:
|
||||
assert "Agent Reach" in captured.out
|
||||
assert "✅" in captured.out
|
||||
|
||||
def test_transcribe_command_prints_text(self, capsys):
|
||||
with patch("agent_reach.transcribe.transcribe", return_value="hello transcript"):
|
||||
with patch("sys.argv", ["agent-reach", "transcribe", "audio.mp3"]):
|
||||
main()
|
||||
captured = capsys.readouterr()
|
||||
assert "hello transcript" in captured.out
|
||||
|
||||
def test_transcribe_command_writes_output_file(self, capsys, tmp_path):
|
||||
out_file = tmp_path / "t.txt"
|
||||
with patch("agent_reach.transcribe.transcribe", return_value="saved text"):
|
||||
with patch("sys.argv", ["agent-reach", "transcribe", "audio.mp3", "-o", str(out_file)]):
|
||||
main()
|
||||
assert out_file.read_text(encoding="utf-8").strip() == "saved text"
|
||||
assert "Transcript written" in capsys.readouterr().out
|
||||
|
||||
def test_parse_twitter_cookie_input_separate_values(self):
|
||||
auth_token, ct0 = cli._parse_twitter_cookie_input("token123 ct0abc")
|
||||
assert auth_token == "token123"
|
||||
@@ -42,6 +60,31 @@ class TestCLI:
|
||||
assert auth_token == "token123"
|
||||
assert ct0 == "ct0abc"
|
||||
|
||||
def test_install_reddit_deps_prefers_github_source(self, monkeypatch, capsys):
|
||||
state = {"rdt_installed": False}
|
||||
commands = []
|
||||
|
||||
def fake_which(name):
|
||||
if name == "rdt":
|
||||
return "/usr/local/bin/rdt" if state["rdt_installed"] else None
|
||||
if name == "pipx":
|
||||
return "/usr/local/bin/pipx"
|
||||
return None
|
||||
|
||||
def fake_run(cmd, **kwargs):
|
||||
commands.append(cmd)
|
||||
state["rdt_installed"] = True
|
||||
return subprocess.CompletedProcess(cmd, 0, "", "")
|
||||
|
||||
monkeypatch.setattr(shutil, "which", fake_which)
|
||||
monkeypatch.setattr(subprocess, "run", fake_run)
|
||||
|
||||
cli._install_reddit_deps()
|
||||
|
||||
out = capsys.readouterr().out
|
||||
assert commands == [["pipx", "install", cli._RDT_GIT_SOURCE]]
|
||||
assert "✅ rdt-cli installed" in out
|
||||
|
||||
|
||||
class TestCheckUpdateRetry:
|
||||
def test_retry_timeout_classification(self):
|
||||
@@ -89,7 +132,7 @@ class TestCheckUpdateRetry:
|
||||
|
||||
sequence = [
|
||||
R(429, headers={"Retry-After": "3"}),
|
||||
R(200, payload={"tag_name": "v1.4.0"}),
|
||||
R(200, payload={"tag_name": "v1.4.2"}),
|
||||
]
|
||||
|
||||
with patch("requests.get", side_effect=sequence):
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
from agent_reach.utils.process import mcporter_utf8_env_args, utf8_subprocess_env
|
||||
|
||||
|
||||
def test_utf8_subprocess_env_forces_python_utf8():
|
||||
env = utf8_subprocess_env({"PYTHONUTF8": "0", "OTHER": "value"})
|
||||
|
||||
assert env["PYTHONUTF8"] == "1"
|
||||
assert env["PYTHONIOENCODING"] == "utf-8"
|
||||
assert env["OTHER"] == "value"
|
||||
|
||||
|
||||
def test_mcporter_utf8_env_args():
|
||||
assert mcporter_utf8_env_args() == [
|
||||
"--env",
|
||||
"PYTHONUTF8=1",
|
||||
"--env",
|
||||
"PYTHONIOENCODING=utf-8",
|
||||
]
|
||||
@@ -39,14 +39,11 @@ class TestSkillCommand(unittest.TestCase):
|
||||
with patch.dict(os.environ, env, clear=True):
|
||||
_install_skill()
|
||||
|
||||
target = os.path.join(skill_dir, "agent-reach", "SKILL.md")
|
||||
# Check at least one known skill dir pattern
|
||||
found = False
|
||||
for dirpath, _, filenames in os.walk(tmpdir):
|
||||
if "SKILL.md" in filenames:
|
||||
found = True
|
||||
# Verify content is non-empty
|
||||
with open(os.path.join(dirpath, "SKILL.md")) as f:
|
||||
with open(os.path.join(dirpath, "SKILL.md"), encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
self.assertIn("Agent Reach", content)
|
||||
# _install_skill may or may not find dirs depending on mock; just ensure no crash
|
||||
@@ -58,7 +55,7 @@ class TestSkillCommand(unittest.TestCase):
|
||||
# Create a fake skill installation
|
||||
skill_path = os.path.join(tmpdir, ".openclaw", "skills", "agent-reach")
|
||||
os.makedirs(skill_path)
|
||||
with open(os.path.join(skill_path, "SKILL.md"), "w") as f:
|
||||
with open(os.path.join(skill_path, "SKILL.md"), "w", encoding="utf-8") as f:
|
||||
f.write("test")
|
||||
|
||||
self.assertTrue(os.path.exists(skill_path))
|
||||
@@ -92,7 +89,7 @@ class TestSkillCommand(unittest.TestCase):
|
||||
|
||||
target = os.path.join(skill_parent, "agent-reach", "SKILL.md")
|
||||
self.assertTrue(os.path.exists(target))
|
||||
with open(target) as f:
|
||||
with open(target, encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
self.assertIn("Agent Reach", content)
|
||||
|
||||
@@ -114,10 +111,10 @@ class TestSkillCommand(unittest.TestCase):
|
||||
|
||||
target = os.path.join(skill_parent, "agent-reach", "SKILL.md")
|
||||
self.assertTrue(os.path.exists(target))
|
||||
with open(target) as f:
|
||||
with open(target, encoding="utf-8") as f:
|
||||
content = f.read()
|
||||
self.assertTrue(content.strip())
|
||||
self.assertIn("Give your AI agent eyes to see the entire internet.", content)
|
||||
self.assertIn("Xiaoyuzhou Podcast, LinkedIn", content)
|
||||
self.assertNotIn("搜推特", content)
|
||||
self.assertTrue(
|
||||
os.path.exists(os.path.join(skill_parent, "agent-reach", "references"))
|
||||
|
||||
@@ -0,0 +1,262 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Tests for agent_reach.transcribe — provider routing, fallback, and errors."""
|
||||
|
||||
from typing import List
|
||||
|
||||
import pytest
|
||||
|
||||
from agent_reach import transcribe as tr
|
||||
from agent_reach.config import Config
|
||||
|
||||
# --- Fixtures ----------------------------------------------------------- #
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake_config(tmp_path, monkeypatch):
|
||||
"""A Config that writes to a temp dir and never touches the user's HOME."""
|
||||
cfg_path = tmp_path / "config.yaml"
|
||||
monkeypatch.setattr(Config, "CONFIG_DIR", tmp_path)
|
||||
monkeypatch.setattr(Config, "CONFIG_FILE", cfg_path)
|
||||
cfg = Config(config_path=cfg_path)
|
||||
return cfg
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def chunk_file(tmp_path):
|
||||
p = tmp_path / "chunk.m4a"
|
||||
p.write_bytes(b"\x00fake-m4a-bytes")
|
||||
return p
|
||||
|
||||
|
||||
class FakeResponse:
|
||||
def __init__(self, status_code: int, text: str = ""):
|
||||
self.status_code = status_code
|
||||
self.text = text
|
||||
|
||||
@property
|
||||
def ok(self) -> bool:
|
||||
return 200 <= self.status_code < 300
|
||||
|
||||
|
||||
# --- transcribe_chunk: provider routing -------------------------------- #
|
||||
|
||||
|
||||
class TestTranscribeChunk:
|
||||
def test_routes_to_groq_endpoint(self, monkeypatch, fake_config, chunk_file):
|
||||
fake_config.set("groq_api_key", "gsk_test")
|
||||
captured = {}
|
||||
|
||||
def fake_post(url, headers=None, files=None, data=None, timeout=None):
|
||||
captured["url"] = url
|
||||
captured["headers"] = headers
|
||||
captured["model"] = data["model"]
|
||||
return FakeResponse(200, "hello world")
|
||||
|
||||
monkeypatch.setattr(tr.requests, "post", fake_post)
|
||||
text = tr.transcribe_chunk(chunk_file, "groq", config=fake_config)
|
||||
assert text == "hello world"
|
||||
assert captured["url"] == tr.PROVIDERS["groq"]["endpoint"]
|
||||
assert captured["model"] == "whisper-large-v3"
|
||||
assert captured["headers"]["Authorization"] == "Bearer gsk_test"
|
||||
|
||||
def test_routes_to_openai_endpoint(self, monkeypatch, fake_config, chunk_file):
|
||||
fake_config.set("openai_api_key", "sk-test")
|
||||
captured = {}
|
||||
|
||||
def fake_post(url, headers=None, files=None, data=None, timeout=None):
|
||||
captured["url"] = url
|
||||
captured["model"] = data["model"]
|
||||
return FakeResponse(200, "openai output")
|
||||
|
||||
monkeypatch.setattr(tr.requests, "post", fake_post)
|
||||
text = tr.transcribe_chunk(chunk_file, "openai", config=fake_config)
|
||||
assert text == "openai output"
|
||||
assert captured["url"] == tr.PROVIDERS["openai"]["endpoint"]
|
||||
assert captured["model"] == "whisper-1"
|
||||
|
||||
def test_raises_when_key_missing(self, fake_config, chunk_file):
|
||||
with pytest.raises(tr.NoProviderConfigured):
|
||||
tr.transcribe_chunk(chunk_file, "groq", config=fake_config)
|
||||
|
||||
def test_raises_on_http_error(self, monkeypatch, fake_config, chunk_file):
|
||||
fake_config.set("groq_api_key", "gsk_test")
|
||||
monkeypatch.setattr(
|
||||
tr.requests,
|
||||
"post",
|
||||
lambda *a, **k: FakeResponse(429, "rate limited"),
|
||||
)
|
||||
with pytest.raises(tr.TranscribeError, match="HTTP 429"):
|
||||
tr.transcribe_chunk(chunk_file, "groq", config=fake_config)
|
||||
|
||||
def test_unknown_provider(self, fake_config, chunk_file):
|
||||
with pytest.raises(tr.TranscribeError, match="unknown provider"):
|
||||
tr.transcribe_chunk(chunk_file, "azure", config=fake_config)
|
||||
|
||||
|
||||
# --- _transcribe_with_fallback ----------------------------------------- #
|
||||
|
||||
|
||||
class TestFallback:
|
||||
def test_groq_succeeds_no_openai_call(self, monkeypatch, fake_config, chunk_file):
|
||||
fake_config.set("groq_api_key", "gsk_test")
|
||||
fake_config.set("openai_api_key", "sk-test")
|
||||
calls: List[str] = []
|
||||
|
||||
def fake_post(url, headers=None, files=None, data=None, timeout=None):
|
||||
calls.append(url)
|
||||
return FakeResponse(200, "from-groq")
|
||||
|
||||
monkeypatch.setattr(tr.requests, "post", fake_post)
|
||||
text = tr._transcribe_with_fallback(chunk_file, ["groq", "openai"], fake_config)
|
||||
assert text == "from-groq"
|
||||
assert calls == [tr.PROVIDERS["groq"]["endpoint"]]
|
||||
|
||||
def test_groq_429_falls_back_to_openai(self, monkeypatch, fake_config, chunk_file):
|
||||
fake_config.set("groq_api_key", "gsk_test")
|
||||
fake_config.set("openai_api_key", "sk-test")
|
||||
calls: List[str] = []
|
||||
|
||||
def fake_post(url, headers=None, files=None, data=None, timeout=None):
|
||||
calls.append(url)
|
||||
if url == tr.PROVIDERS["groq"]["endpoint"]:
|
||||
return FakeResponse(429, "rate limited")
|
||||
return FakeResponse(200, "from-openai")
|
||||
|
||||
monkeypatch.setattr(tr.requests, "post", fake_post)
|
||||
text = tr._transcribe_with_fallback(chunk_file, ["groq", "openai"], fake_config)
|
||||
assert text == "from-openai"
|
||||
assert calls == [
|
||||
tr.PROVIDERS["groq"]["endpoint"],
|
||||
tr.PROVIDERS["openai"]["endpoint"],
|
||||
]
|
||||
|
||||
def test_skip_unconfigured_provider(self, monkeypatch, fake_config, chunk_file):
|
||||
# Only openai key configured — fallback should skip groq silently.
|
||||
fake_config.set("openai_api_key", "sk-test")
|
||||
calls: List[str] = []
|
||||
|
||||
def fake_post(url, headers=None, files=None, data=None, timeout=None):
|
||||
calls.append(url)
|
||||
return FakeResponse(200, "via-openai")
|
||||
|
||||
monkeypatch.setattr(tr.requests, "post", fake_post)
|
||||
text = tr._transcribe_with_fallback(chunk_file, ["groq", "openai"], fake_config)
|
||||
assert text == "via-openai"
|
||||
assert calls == [tr.PROVIDERS["openai"]["endpoint"]]
|
||||
|
||||
def test_all_fail_raises_with_last_error(self, monkeypatch, fake_config, chunk_file):
|
||||
fake_config.set("groq_api_key", "gsk_test")
|
||||
fake_config.set("openai_api_key", "sk-test")
|
||||
monkeypatch.setattr(
|
||||
tr.requests,
|
||||
"post",
|
||||
lambda *a, **k: FakeResponse(500, "boom"),
|
||||
)
|
||||
with pytest.raises(tr.TranscribeError, match="all providers failed"):
|
||||
tr._transcribe_with_fallback(chunk_file, ["groq", "openai"], fake_config)
|
||||
|
||||
|
||||
# --- transcribe (orchestrator) ---------------------------------------- #
|
||||
|
||||
|
||||
class TestOrchestrator:
|
||||
def test_local_file_skips_yt_dlp(self, monkeypatch, fake_config, tmp_path, chunk_file):
|
||||
fake_config.set("groq_api_key", "gsk_test")
|
||||
|
||||
def boom_download(*a, **k):
|
||||
raise AssertionError("yt-dlp must not be called for local files")
|
||||
|
||||
# Stub heavy external steps to no-ops that keep file paths valid.
|
||||
compressed = tmp_path / "compressed.m4a"
|
||||
compressed.write_bytes(b"x" * 1024)
|
||||
|
||||
def fake_compress(src, out_dir):
|
||||
return compressed
|
||||
|
||||
monkeypatch.setattr(tr, "download_audio", boom_download)
|
||||
monkeypatch.setattr(tr, "compress_audio", fake_compress)
|
||||
monkeypatch.setattr(
|
||||
tr.requests,
|
||||
"post",
|
||||
lambda *a, **k: FakeResponse(200, "transcript text"),
|
||||
)
|
||||
|
||||
text = tr.transcribe(
|
||||
str(chunk_file),
|
||||
out_dir=tmp_path / "work",
|
||||
config=fake_config,
|
||||
)
|
||||
assert text == "transcript text"
|
||||
|
||||
def test_chunks_concatenated_with_newlines(
|
||||
self, monkeypatch, fake_config, tmp_path, chunk_file
|
||||
):
|
||||
fake_config.set("groq_api_key", "gsk_test")
|
||||
# Force the "needs chunking" path by writing a file above the size limit.
|
||||
big = tmp_path / "compressed.m4a"
|
||||
big.write_bytes(b"x" * (tr.SIZE_LIMIT_BYTES + 1))
|
||||
monkeypatch.setattr(tr, "compress_audio", lambda src, out_dir: big)
|
||||
c1 = tmp_path / "chunk_001.m4a"
|
||||
c2 = tmp_path / "chunk_002.m4a"
|
||||
c1.write_bytes(b"a")
|
||||
c2.write_bytes(b"b")
|
||||
monkeypatch.setattr(tr, "chunk_audio", lambda src, out_dir: [c1, c2])
|
||||
|
||||
responses = iter(["part one ", "part two "])
|
||||
monkeypatch.setattr(
|
||||
tr.requests,
|
||||
"post",
|
||||
lambda *a, **k: FakeResponse(200, next(responses)),
|
||||
)
|
||||
|
||||
text = tr.transcribe(
|
||||
str(chunk_file),
|
||||
out_dir=tmp_path / "work",
|
||||
config=fake_config,
|
||||
)
|
||||
assert text == "part one\npart two"
|
||||
|
||||
def test_no_provider_configured_fails_fast(self, fake_config, chunk_file):
|
||||
with pytest.raises(tr.NoProviderConfigured):
|
||||
tr.transcribe(str(chunk_file), config=fake_config)
|
||||
|
||||
def test_invalid_provider_string(self, fake_config, chunk_file):
|
||||
with pytest.raises(tr.TranscribeError, match="unknown provider"):
|
||||
tr.transcribe(str(chunk_file), provider="azure", config=fake_config)
|
||||
|
||||
|
||||
# --- YouTubeChannel integration --------------------------------------- #
|
||||
|
||||
|
||||
class TestYouTubeChannelTranscribe:
|
||||
def test_delegates_to_transcribe(self, monkeypatch, fake_config):
|
||||
from agent_reach.channels.youtube import YouTubeChannel
|
||||
|
||||
captured = {}
|
||||
|
||||
def fake_transcribe(source, *, provider="auto", out_dir=None, config=None):
|
||||
captured["source"] = source
|
||||
captured["provider"] = provider
|
||||
captured["config"] = config
|
||||
return "delegated text"
|
||||
|
||||
monkeypatch.setattr(tr, "transcribe", fake_transcribe)
|
||||
out = YouTubeChannel().transcribe(
|
||||
"https://youtu.be/abc", provider="groq", config=fake_config
|
||||
)
|
||||
assert out == "delegated text"
|
||||
assert captured["source"] == "https://youtu.be/abc"
|
||||
assert captured["provider"] == "groq"
|
||||
assert captured["config"] is fake_config
|
||||
|
||||
|
||||
# --- Config feature requirement --------------------------------------- #
|
||||
|
||||
|
||||
class TestConfigOpenAIWhisper:
|
||||
def test_openai_whisper_feature_registered(self, fake_config):
|
||||
assert "openai_whisper" in Config.FEATURE_REQUIREMENTS
|
||||
assert Config.FEATURE_REQUIREMENTS["openai_whisper"] == ["openai_api_key"]
|
||||
assert not fake_config.is_configured("openai_whisper")
|
||||
fake_config.set("openai_api_key", "sk-test")
|
||||
assert fake_config.is_configured("openai_whisper")
|
||||
Reference in New Issue
Block a user