- Remove 3 dead modules: academic/, wechat/, social/ (merged into engines/) - Remove 4 stale news scripts (tech_news.py, wallstreetcn.py, cls.py, rss.py) - scripts/core/base.py: canonical SearchResult + BaseEngine (was in engines/base.py) - scripts/engines/base.py: re-exports from core.base for compatibility - scripts/engines/news_engine.py: Bing News + RSS unified (was in scripts/news/rss.py) - scripts/engines/academic_engine.py: ArXiv + CrossRef unified (was in scripts/academic/arxiv.py) - scripts/cli.py: complete rewrite — 10 commands (search/web/news/academic/wechat/social/rss/baidu/health/sources) - scripts/unified_search.py: clean async Python API (was a stub) - ws.py: lightweight launcher - README.md + SKILL.md: updated to reflect actual 15+ engines - All EngineResult → SearchResult - All engines return list[SearchResult]
141 lines
3.7 KiB
Markdown
141 lines
3.7 KiB
Markdown
---
|
|
name: web-search
|
|
description: "Multi-source web search — 15+ engines (web, news, academic, social, RSS) running concurrently in <5s. No API key needed for most engines. Use when user asks to search the web, find news, look up papers, check social media, or search WeChat."
|
|
version: 3.0.0
|
|
license: MIT
|
|
compatibility: Python 3.10+, httpx, beautifulsoup4, lxml
|
|
metadata:
|
|
author: openbot-coder
|
|
category: search
|
|
tags: [search, web, news, academic, social, rss, cli, async]
|
|
---
|
|
|
|
# Web Search Skills
|
|
|
|
## Triggers
|
|
|
|
Use this skill when the user says:
|
|
|
|
| Intent | Example |
|
|
|--------|---------|
|
|
| Web search | "搜索一下 xxx", "search the web for xxx" |
|
|
| News | "查一下新闻", "搜索最新的科技新闻" |
|
|
| Academic | "搜论文 xxx", "查学术资料" |
|
|
| WeChat | "搜微信公众号 xxx" |
|
|
| Social | "看看 Twitter 上关于" |
|
|
| Health check | "检查一下搜索引擎状态" |
|
|
| List sources | "有哪些搜索引擎可以用" |
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
# Full search (all 15 engines in parallel, ~10s)
|
|
web-search search "AI news" -n 10
|
|
|
|
# Web only (6 engines, ~5s)
|
|
web-search web "Python async" -n 5
|
|
|
|
# News (Bing News + 15 RSS, ~8s)
|
|
web-search news "量子计算" -n 5
|
|
|
|
# Academic (ArXiv + CrossRef, ~6s)
|
|
web-search academic "machine learning"
|
|
|
|
# WeChat (Sogou, ~5s)
|
|
web-search wechat 蒙面财经
|
|
|
|
# Social (Nitter, ~5s)
|
|
web-search social "AI regulation" -n 10
|
|
|
|
# RSS feeds only
|
|
web-search rss "AI newsletter" -n 5
|
|
|
|
# Baidu Qianfan API (needs BAIDU_API_KEY)
|
|
web-search baidu "中美元首会晤"
|
|
|
|
# Engine health check
|
|
web-search health
|
|
|
|
# List all sources
|
|
web-search sources
|
|
```
|
|
|
|
## Decision Tree
|
|
|
|
```
|
|
用户说了什么?
|
|
│
|
|
├─ "微信公众号" / "wechat"
|
|
│ → web-search wechat <query>
|
|
│
|
|
├─ "学术" / "论文" / "paper" / "ArXiv"
|
|
│ → web-search academic <query>
|
|
│
|
|
├─ "Twitter" / "X" / "社交"
|
|
│ → web-search social <query>
|
|
│
|
|
├─ "新闻" / "news"
|
|
│ → web-search news <query>
|
|
│
|
|
├─ "健康检查" / "health"
|
|
│ → web-search health
|
|
│
|
|
├─ "有哪些源" / "sources"
|
|
│ → web-search sources
|
|
│
|
|
└─ 通用搜索
|
|
→ web-search search <query> -n 10
|
|
```
|
|
|
|
## Python API
|
|
|
|
```python
|
|
from scripts.unified_search import UnifiedSearch
|
|
|
|
async def main():
|
|
us = UnifiedSearch()
|
|
resp = await us.search(
|
|
"quantum computing",
|
|
sources=["bing", "arxiv"], # or None for all
|
|
max_results=5,
|
|
)
|
|
for r in resp.results:
|
|
print(f"[{r.rank}] {r.title}")
|
|
print(f" {r.url}")
|
|
print(f" {r.snippet[:100]}")
|
|
print(f"Engines used: {resp.metadata.engines_used}")
|
|
print(f"Duration: {resp.metadata.duration_ms}ms")
|
|
await us.close()
|
|
```
|
|
|
|
## SearchResult Fields
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `rank` | int | Position after deduplication (1-based) |
|
|
| `title` | str | Article/title text |
|
|
| `url` | str | Direct URL |
|
|
| `snippet` | str | Summary (~300 chars) |
|
|
| `source` | str | Engine name (e.g. "bing", "arxiv") |
|
|
| `category` | str | "web" \| "news" \| "academic" \| "github" |
|
|
| `published` | str | ISO date (optional) |
|
|
| `extra` | dict | Engine-specific metadata |
|
|
|
|
## Error Handling
|
|
|
|
| Symptom | Cause | Action |
|
|
|---------|-------|--------|
|
|
| Empty Bing News | Anti-scraping / region | Not critical, skip |
|
|
| Baidu Qianfan empty | Missing `BAIDU_API_KEY` | Set env var |
|
|
| Sogou WeChat empty | Sogou anti-scraping | Show URL fallback |
|
|
| Nitter all fail | Instance blocked | Show direct twitter.com link |
|
|
|
|
## Notes
|
|
|
|
- Most engines require **no API key**
|
|
- Baidu Qianfan needs `BAIDU_API_KEY` env var
|
|
- All engines run concurrently via `asyncio.gather`
|
|
- Max 5 concurrent engines (semaphore)
|
|
- Per-engine timeout: 15s
|
|
- Results are deduplicated by URL before ranking
|