- Remove 3 dead modules: academic/, wechat/, social/ (merged into engines/) - Remove 4 stale news scripts (tech_news.py, wallstreetcn.py, cls.py, rss.py) - scripts/core/base.py: canonical SearchResult + BaseEngine (was in engines/base.py) - scripts/engines/base.py: re-exports from core.base for compatibility - scripts/engines/news_engine.py: Bing News + RSS unified (was in scripts/news/rss.py) - scripts/engines/academic_engine.py: ArXiv + CrossRef unified (was in scripts/academic/arxiv.py) - scripts/cli.py: complete rewrite — 10 commands (search/web/news/academic/wechat/social/rss/baidu/health/sources) - scripts/unified_search.py: clean async Python API (was a stub) - ws.py: lightweight launcher - README.md + SKILL.md: updated to reflect actual 15+ engines - All EngineResult → SearchResult - All engines return list[SearchResult]
3.7 KiB
3.7 KiB
name, description, version, license, compatibility, metadata
| name | description | version | license | compatibility | metadata | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| web-search | Multi-source web search — 15+ engines (web, news, academic, social, RSS) running concurrently in <5s. No API key needed for most engines. Use when user asks to search the web, find news, look up papers, check social media, or search WeChat. | 3.0.0 | MIT | Python 3.10+, httpx, beautifulsoup4, lxml |
|
Web Search Skills
Triggers
Use this skill when the user says:
| Intent | Example |
|---|---|
| Web search | "搜索一下 xxx", "search the web for xxx" |
| News | "查一下新闻", "搜索最新的科技新闻" |
| Academic | "搜论文 xxx", "查学术资料" |
| "搜微信公众号 xxx" | |
| Social | "看看 Twitter 上关于" |
| Health check | "检查一下搜索引擎状态" |
| List sources | "有哪些搜索引擎可以用" |
Commands
# Full search (all 15 engines in parallel, ~10s)
web-search search "AI news" -n 10
# Web only (6 engines, ~5s)
web-search web "Python async" -n 5
# News (Bing News + 15 RSS, ~8s)
web-search news "量子计算" -n 5
# Academic (ArXiv + CrossRef, ~6s)
web-search academic "machine learning"
# WeChat (Sogou, ~5s)
web-search wechat 蒙面财经
# Social (Nitter, ~5s)
web-search social "AI regulation" -n 10
# RSS feeds only
web-search rss "AI newsletter" -n 5
# Baidu Qianfan API (needs BAIDU_API_KEY)
web-search baidu "中美元首会晤"
# Engine health check
web-search health
# List all sources
web-search sources
Decision Tree
用户说了什么?
│
├─ "微信公众号" / "wechat"
│ → web-search wechat <query>
│
├─ "学术" / "论文" / "paper" / "ArXiv"
│ → web-search academic <query>
│
├─ "Twitter" / "X" / "社交"
│ → web-search social <query>
│
├─ "新闻" / "news"
│ → web-search news <query>
│
├─ "健康检查" / "health"
│ → web-search health
│
├─ "有哪些源" / "sources"
│ → web-search sources
│
└─ 通用搜索
→ web-search search <query> -n 10
Python API
from scripts.unified_search import UnifiedSearch
async def main():
us = UnifiedSearch()
resp = await us.search(
"quantum computing",
sources=["bing", "arxiv"], # or None for all
max_results=5,
)
for r in resp.results:
print(f"[{r.rank}] {r.title}")
print(f" {r.url}")
print(f" {r.snippet[:100]}")
print(f"Engines used: {resp.metadata.engines_used}")
print(f"Duration: {resp.metadata.duration_ms}ms")
await us.close()
SearchResult Fields
| Field | Type | Description |
|---|---|---|
rank |
int | Position after deduplication (1-based) |
title |
str | Article/title text |
url |
str | Direct URL |
snippet |
str | Summary (~300 chars) |
source |
str | Engine name (e.g. "bing", "arxiv") |
category |
str | "web" | "news" | "academic" | "github" |
published |
str | ISO date (optional) |
extra |
dict | Engine-specific metadata |
Error Handling
| Symptom | Cause | Action |
|---|---|---|
| Empty Bing News | Anti-scraping / region | Not critical, skip |
| Baidu Qianfan empty | Missing BAIDU_API_KEY |
Set env var |
| Sogou WeChat empty | Sogou anti-scraping | Show URL fallback |
| Nitter all fail | Instance blocked | Show direct twitter.com link |
Notes
- Most engines require no API key
- Baidu Qianfan needs
BAIDU_API_KEYenv var - All engines run concurrently via
asyncio.gather - Max 5 concurrent engines (semaphore)
- Per-engine timeout: 15s
- Results are deduplicated by URL before ranking