refactor: cleanup dead code, fix model confusion, streamline CLI
- Remove 3 dead modules: academic/, wechat/, social/ (merged into engines/) - Remove 4 stale news scripts (tech_news.py, wallstreetcn.py, cls.py, rss.py) - scripts/core/base.py: canonical SearchResult + BaseEngine (was in engines/base.py) - scripts/engines/base.py: re-exports from core.base for compatibility - scripts/engines/news_engine.py: Bing News + RSS unified (was in scripts/news/rss.py) - scripts/engines/academic_engine.py: ArXiv + CrossRef unified (was in scripts/academic/arxiv.py) - scripts/cli.py: complete rewrite — 10 commands (search/web/news/academic/wechat/social/rss/baidu/health/sources) - scripts/unified_search.py: clean async Python API (was a stub) - ws.py: lightweight launcher - README.md + SKILL.md: updated to reflect actual 15+ engines - All EngineResult → SearchResult - All engines return list[SearchResult]
This commit is contained in:
1 parent
e20478fcbd
commit
c077a7ae40
31 files changed
+1188
-3148
No files matched your search
@@ -1,200 +1,140 @@
|
||||
---
|
||||
name: web-search-skills
|
||||
description: "Multi-source web search CLI & library, covering web, news, WeChat, academic, social, and RSS feeds across 28 engines. Use when the user asks to search the web, find WeChat articles, look up news, search academic papers, or check social media."
|
||||
version: 2.2.0
|
||||
name: web-search
|
||||
description: "Multi-source web search — 15+ engines (web, news, academic, social, RSS) running concurrently in <5s. No API key needed for most engines. Use when user asks to search the web, find news, look up papers, check social media, or search WeChat."
|
||||
version: 3.0.0
|
||||
license: MIT
|
||||
compatibility: Requires Python 3.8+, httpx, beautifulsoup4, lxml
|
||||
compatibility: Python 3.10+, httpx, beautifulsoup4, lxml
|
||||
metadata:
|
||||
author: openbot-coder
|
||||
category: development
|
||||
tags: [search, web, wechat, news, academic, social, rss, cli, concurrent]
|
||||
category: search
|
||||
tags: [search, web, news, academic, social, rss, cli, async]
|
||||
---
|
||||
|
||||
# Web Search Skills
|
||||
|
||||
## Overview
|
||||
|
||||
A CLI (`web-search`) and Python library that queries **28 search engines** concurrently via `asyncio.gather`. Supports web search, news (Hacker News, GitHub Trending, 36氪, 微博热搜, V2EX, RSS feeds), WeChat articles, ArXiv papers, Twitter/X, and Baidu Qianfan AI search.
|
||||
|
||||
All engines run in parallel with `Semaphore(5)` concurrency control and 15s per-engine timeout.
|
||||
|
||||
## Triggers
|
||||
|
||||
When the user says any of these, use this skill:
|
||||
Use this skill when the user says:
|
||||
|
||||
| Intent | Example user phrases |
|
||||
|--------|---------------------|
|
||||
| Intent | Example |
|
||||
|--------|---------|
|
||||
| Web search | "搜索一下 xxx", "search the web for xxx" |
|
||||
| WeChat articles | "搜微信公众号 xxx", "search WeChat for xxx" |
|
||||
| News | "查一下新闻", "搜索最新的科技新闻", "search news about xxx" |
|
||||
| Academic | "搜论文 xxx", "search papers about xxx", "查学术资料" |
|
||||
| Social media | "看看 Twitter 上关于", "check Twitter for xxx" |
|
||||
| Specific source | "看看 Hacker News 上关于", "what's trending on GitHub" |
|
||||
| Hot topics | "查一下微博热搜", "check Weibo trending" |
|
||||
| News | "查一下新闻", "搜索最新的科技新闻" |
|
||||
| Academic | "搜论文 xxx", "查学术资料" |
|
||||
| WeChat | "搜微信公众号 xxx" |
|
||||
| Social | "看看 Twitter 上关于" |
|
||||
| Health check | "检查一下搜索引擎状态" |
|
||||
| List sources | "有哪些搜索引擎可以用" |
|
||||
|
||||
## Agent Decision Tree
|
||||
## Commands
|
||||
|
||||
Use this flow to pick the right command:
|
||||
```bash
|
||||
# Full search (all 15 engines in parallel, ~10s)
|
||||
web-search search "AI news" -n 10
|
||||
|
||||
# Web only (6 engines, ~5s)
|
||||
web-search web "Python async" -n 5
|
||||
|
||||
# News (Bing News + 15 RSS, ~8s)
|
||||
web-search news "量子计算" -n 5
|
||||
|
||||
# Academic (ArXiv + CrossRef, ~6s)
|
||||
web-search academic "machine learning"
|
||||
|
||||
# WeChat (Sogou, ~5s)
|
||||
web-search wechat 蒙面财经
|
||||
|
||||
# Social (Nitter, ~5s)
|
||||
web-search social "AI regulation" -n 10
|
||||
|
||||
# RSS feeds only
|
||||
web-search rss "AI newsletter" -n 5
|
||||
|
||||
# Baidu Qianfan API (needs BAIDU_API_KEY)
|
||||
web-search baidu "中美元首会晤"
|
||||
|
||||
# Engine health check
|
||||
web-search health
|
||||
|
||||
# List all sources
|
||||
web-search sources
|
||||
```
|
||||
|
||||
## Decision Tree
|
||||
|
||||
```
|
||||
用户说了什么?
|
||||
│
|
||||
├─ "微信公众号" / "WeChat" / 公众号名称
|
||||
├─ "微信公众号" / "wechat"
|
||||
│ → web-search wechat <query>
|
||||
│
|
||||
├─ "学术" / "论文" / "paper" / "ArXiv"
|
||||
│ → web-search academic <query>
|
||||
│
|
||||
├─ "Twitter" / "X" / "社交" / "social"
|
||||
│ → web-search social <query> [-n 5]
|
||||
├─ "Twitter" / "X" / "社交"
|
||||
│ → web-search social <query>
|
||||
│
|
||||
├─ "新闻" / "news" / 明确指定 RSS/博文/播客
|
||||
│ → web-search news <query> [-n 5]
|
||||
├─ "新闻" / "news"
|
||||
│ → web-search news <query>
|
||||
│
|
||||
├─ 明确指定了特定源名称
|
||||
│ → web-search search <query> -s "源名1" "源名2" [-n 5]
|
||||
│
|
||||
├─ 需要多类源对比(如"新闻+社交上关于伊朗的看法")
|
||||
│ → web-search search <query> -s news social [-n 10]
|
||||
│
|
||||
├─ 需要百度搜索 + 站点/时间过滤
|
||||
│ → web-search baidu <query> [--site xxx] [--recency week]
|
||||
│
|
||||
├─ "健康检查" / "health" / "测试引擎"
|
||||
├─ "健康检查" / "health"
|
||||
│ → web-search health
|
||||
│
|
||||
├─ "列出源" / "sources" / "有哪些搜索引擎"
|
||||
├─ "有哪些源" / "sources"
|
||||
│ → web-search sources
|
||||
│
|
||||
├─ 其余通用查询
|
||||
│ → web-search search <query> [-s 源1 源2...] [-n 10]
|
||||
│
|
||||
└─ 不确定 / 没特别要求
|
||||
└─ 通用搜索
|
||||
→ web-search search <query> -n 10
|
||||
```
|
||||
|
||||
## Quick Command Reference
|
||||
|
||||
| Command | Speed | Best for |
|
||||
|---------|-------|----------|
|
||||
| `search` | 并发 28 源 | 通用搜索、多源对比 |
|
||||
| `web` | 14 网页引擎 | 普通网页搜索 |
|
||||
| `baidu` | 单引擎 | 需 site/recency 过滤 |
|
||||
| `news` | ~10+ 新闻源 | 科技/财经/博客/播客 |
|
||||
| `wechat` | 单引擎 | 微信公众号独家内容 |
|
||||
| `academic` | 单引擎 | 学术论文 |
|
||||
| `social` | 2 源 (Twitter) | 社交媒体动态 |
|
||||
| `rss` | 18 订阅源 | AI 周报/深度文章/播客 |
|
||||
| `health` | 28 源并发 (~14s) | 检测连通性 |
|
||||
|
||||
**通用选项:** `-n N`(每源结果数), `-j`(JSON), `-o FILE`(存文件), `-s NAME...`(指定源), `-v`(调试日志)
|
||||
|
||||
## Result Field Reference
|
||||
|
||||
Each `SearchResult` returned has these fields:
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `rank` | int | Position in results (1-based) |
|
||||
| `title` | str | Article/title text |
|
||||
| `url` | str | Direct URL to source |
|
||||
| `snippet` | str | Summary text (first ~300 chars) |
|
||||
| `source` | str | Engine name (e.g. "Hacker News") |
|
||||
| `category` | str | "web", "news", "academic", "social", etc. |
|
||||
| `extra` | dict | Engine-specific metadata (see below) |
|
||||
|
||||
### `extra` fields by source
|
||||
|
||||
| Source | extra fields |
|
||||
|--------|-------------|
|
||||
| Hacker News | `points`, `author`, `time` (ISO date) |
|
||||
| GitHub Trending | `language`, `stars today` (heat) |
|
||||
| WeChat | `account` (公众号名), `time` |
|
||||
| Twitter | `time`, `account`, `search_type` |
|
||||
| RSS feeds | `time`, `feed_url` |
|
||||
| Cls / WallStreetCN | `search_url` (JS-rendered fallback) |
|
||||
| Baidu Qianfan | `site`, `recency` (if filtered) |
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Symptom | Likely cause | Action |
|
||||
|---------|-------------|--------|
|
||||
| Empty results from Hacker News/GitHub | Network issue or no matches | Retry with different query |
|
||||
| Baidu Qianfan returns nothing | Missing `BAIDU_API_KEY` env | Check env var is set |
|
||||
| DuckDuckGo times out (15s) | Anti-scraping / rate limit | Note to user, skip |
|
||||
| WeChat empty results | Sogou anti-scraping | Inform user it may be limited |
|
||||
| `web-search health` shows [TIMEOUT] | Engine unreachable from this region | Not critical, skip that engine |
|
||||
| GBK encode error on Windows | Non-ASCII char in output | Already auto-fixed by `_sanitize_for_console` |
|
||||
|
||||
**General rule:** If a specific source fails, note it but don't fail the whole search — other sources continue concurrently.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```bash
|
||||
# Full search (all 28 sources, concurrent)
|
||||
web-search search "quantum computing" -n 10
|
||||
|
||||
# Specific sources only
|
||||
web-search search "technology" -s "Hacker News" "V2EX" "GitHub Trending" -n 3
|
||||
|
||||
# Multiple categories
|
||||
web-search search "Iran nuclear" -s web news social -n 5
|
||||
|
||||
# Baidu with site filter
|
||||
web-search baidu "中美元首会晤" --site mp.weixin.qq.com --recency week
|
||||
|
||||
# All web engines
|
||||
web-search web "Python async" -r global -j
|
||||
|
||||
# Quick searches
|
||||
web-search news "人工智能" -n 5
|
||||
web-search wechat 蒙面财经
|
||||
web-search academic "machine learning"
|
||||
web-search social "AI regulation" -n 5
|
||||
|
||||
# Concurrent health check (~14s for 28 engines)
|
||||
web-search health
|
||||
```
|
||||
|
||||
## Python API (for programmatic use)
|
||||
|
||||
> **Tip:** For Agent use, prefer invoking the `web-search` CLI command directly (see examples above).
|
||||
> The API import below only works when running from the project root directory.
|
||||
## Python API
|
||||
|
||||
```python
|
||||
from scripts.unified_search import UnifiedSearch
|
||||
|
||||
async def main():
|
||||
us = UnifiedSearch()
|
||||
results = await us.search(
|
||||
resp = await us.search(
|
||||
"quantum computing",
|
||||
sources=["Hacker News", "ArXiv"],
|
||||
sources=["bing", "arxiv"], # or None for all
|
||||
max_results=5,
|
||||
)
|
||||
for r in results:
|
||||
print(f"[{r.rank}] {r.title} ({r.source})")
|
||||
if r.extra:
|
||||
print(f" extra: {r.extra}")
|
||||
for r in resp.results:
|
||||
print(f"[{r.rank}] {r.title}")
|
||||
print(f" {r.url}")
|
||||
print(f" {r.snippet[:100]}")
|
||||
print(f"Engines used: {resp.metadata.engines_used}")
|
||||
print(f"Duration: {resp.metadata.duration_ms}ms")
|
||||
await us.close()
|
||||
```
|
||||
|
||||
## Important Notes
|
||||
## SearchResult Fields
|
||||
|
||||
### Installation
|
||||
The `web-search` CLI must be installed first before use:
|
||||
```bash
|
||||
cd web-search-skills
|
||||
pip install -e .
|
||||
```
|
||||
This registers `web-search` as a global command. Requires Python 3.8+.
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `rank` | int | Position after deduplication (1-based) |
|
||||
| `title` | str | Article/title text |
|
||||
| `url` | str | Direct URL |
|
||||
| `snippet` | str | Summary (~300 chars) |
|
||||
| `source` | str | Engine name (e.g. "bing", "arxiv") |
|
||||
| `category` | str | "web" \| "news" \| "academic" \| "github" |
|
||||
| `published` | str | ISO date (optional) |
|
||||
| `extra` | dict | Engine-specific metadata |
|
||||
|
||||
### Known Limitations
|
||||
## Error Handling
|
||||
|
||||
- **WeChat** via Sogou has anti-scraping limits, may return empty
|
||||
- **财联社** uses public API (telegraphList) — returns parsed real-time news
|
||||
- **华尔街见闻** need JS rendering → returns search page URL, not parsed results
|
||||
- **Baidu Qianfan** requires `BAIDU_API_KEY` env var
|
||||
- **RSS feeds** need stable internet
|
||||
- **Windows GBK** terminals: special chars auto-replaced by `_sanitize_for_console()`
|
||||
| Symptom | Cause | Action |
|
||||
|---------|-------|--------|
|
||||
| Empty Bing News | Anti-scraping / region | Not critical, skip |
|
||||
| Baidu Qianfan empty | Missing `BAIDU_API_KEY` | Set env var |
|
||||
| Sogou WeChat empty | Sogou anti-scraping | Show URL fallback |
|
||||
| Nitter all fail | Instance blocked | Show direct twitter.com link |
|
||||
|
||||
> Full user-facing documentation (detailed command reference, project structure, config guide, architecture) is in `README.md`.
|
||||
## Notes
|
||||
|
||||
- Most engines require **no API key**
|
||||
- Baidu Qianfan needs `BAIDU_API_KEY` env var
|
||||
- All engines run concurrently via `asyncio.gather`
|
||||
- Max 5 concurrent engines (semaphore)
|
||||
- Per-engine timeout: 15s
|
||||
- Results are deduplicated by URL before ranking
|
||||
Reference in new issue
Block a user