- Remove 3 dead modules: academic/, wechat/, social/ (merged into engines/) - Remove 4 stale news scripts (tech_news.py, wallstreetcn.py, cls.py, rss.py) - scripts/core/base.py: canonical SearchResult + BaseEngine (was in engines/base.py) - scripts/engines/base.py: re-exports from core.base for compatibility - scripts/engines/news_engine.py: Bing News + RSS unified (was in scripts/news/rss.py) - scripts/engines/academic_engine.py: ArXiv + CrossRef unified (was in scripts/academic/arxiv.py) - scripts/cli.py: complete rewrite — 10 commands (search/web/news/academic/wechat/social/rss/baidu/health/sources) - scripts/unified_search.py: clean async Python API (was a stub) - ws.py: lightweight launcher - README.md + SKILL.md: updated to reflect actual 15+ engines - All EngineResult → SearchResult - All engines return list[SearchResult]
138 lines
3.8 KiB
Markdown
138 lines
3.8 KiB
Markdown
# web-search-skills
|
|
|
|
> Multi-source web search CLI & Python library — 15+ engines, <5s, no API key required.
|
|
|
|
## Features
|
|
|
|
- **15+ engines** across 5 categories: Web, News, Academic, Social, RSS
|
|
- **Concurrent execution** via `asyncio.gather` with semaphore (max 5 parallel)
|
|
- **Free by default** — most engines work without any API key
|
|
- **Python API** for programmatic use (`UnifiedSearch`)
|
|
- **Single model** — `SearchResult` dataclass across all engines
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Install
|
|
pip install -e .
|
|
|
|
# CLI
|
|
web-search search "AI news" -n 10
|
|
web-search news "quantum computing" -n 5
|
|
web-search academic "machine learning"
|
|
web-search wechat 蒙面财经
|
|
web-search health
|
|
```
|
|
|
|
## Commands
|
|
|
|
| Command | Description | Speed |
|
|
|---------|------------|-------|
|
|
| `search` | All 15 engines in parallel | ~10s |
|
|
| `web` | Web engines only (6) | ~5s |
|
|
| `news` | Bing News + 15 RSS feeds | ~8s |
|
|
| `academic` | ArXiv + CrossRef in parallel | ~6s |
|
|
| `wechat` | WeChat via Sogou | ~5s |
|
|
| `social` | Twitter/X via Nitter | ~5s |
|
|
| `rss` | RSS feed aggregation | ~8s |
|
|
| `baidu` | Baidu Qianfan API (needs key) | varies |
|
|
| `health` | Engine connectivity check | ~10s |
|
|
| `sources` | List all sources | instant |
|
|
|
|
## Options
|
|
|
|
- `-n N` — max results per engine (default: 10)
|
|
- `-s SRC...` — run specific sources only
|
|
- `-c CAT` — run by category (web/news/academic/social/rss)
|
|
- `-r cn|global` — region (default: cn)
|
|
- `-j` — JSON output
|
|
|
|
## Python API
|
|
|
|
```python
|
|
from scripts.unified_search import UnifiedSearch
|
|
|
|
async def main():
|
|
us = UnifiedSearch()
|
|
resp = await us.search("quantum computing", sources=["bing", "arxiv"], max_results=5)
|
|
for r in resp.results:
|
|
print(f"[{r.rank}] {r.title}")
|
|
print(f" {r.url}")
|
|
await us.close()
|
|
```
|
|
|
|
## Architecture
|
|
|
|
```
|
|
scripts/
|
|
cli.py # CLI entry point
|
|
unified_search.py # Python API
|
|
models.py # Data models
|
|
config.py # Settings
|
|
core/
|
|
base.py # SearchResult + BaseEngine (canonical)
|
|
aggregator.py # Deduplication + ranking
|
|
execution_log.py # Timing + error tracking
|
|
engines/
|
|
bing_scraper.py # Bing web search
|
|
sogou_scraper.py # Sogou web search
|
|
baidu_scraper.py # Baidu web search
|
|
search360_scraper.py # 360 search
|
|
parser_engines.py # DuckDuckGo + Brave (HTML parser)
|
|
news_engine.py # Bing News + RSS aggregator
|
|
academic_engine.py # ArXiv + CrossRef
|
|
github_engine.py # GitHub API
|
|
baidu_qianfan.py # Baidu Qianfan API (needs key)
|
|
parallel_search.py # Parallel orchestrator
|
|
```
|
|
|
|
## Engines
|
|
|
|
### Web (无需 API key)
|
|
| Engine | URL | Notes |
|
|
|--------|-----|-------|
|
|
| Bing CN | cn.bing.com | Primary |
|
|
| Bing INT | www.bing.com | Global fallback |
|
|
| Sogou | sogou.com | Primary CN |
|
|
| Baidu | baidu.com | Primary CN |
|
|
| 360 | so.com | CN fallback |
|
|
| DuckDuckGo | html.duckduckgo.com | Global |
|
|
| Brave | search.brave.com | Global |
|
|
|
|
### News
|
|
| Engine | Source |
|
|
|--------|--------|
|
|
| Bing News | bing.com/news |
|
|
| RSS (15 feeds) | 36kr, IT之家, 钛媒体, HackerNews, TechCrunch... |
|
|
|
|
### Academic (Free APIs)
|
|
| Engine | API |
|
|
|--------|-----|
|
|
| ArXiv | export.arxiv.org/api/query |
|
|
| CrossRef | api.crossref.org |
|
|
|
|
### Social
|
|
| Engine | Notes |
|
|
|--------|-------|
|
|
| Nitter | twitter.com 免登录 (多个实例自动切换) |
|
|
|
|
### API (需配置密钥)
|
|
| Engine | Env var |
|
|
|--------|---------|
|
|
| Baidu Qianfan | `BAIDU_API_KEY` |
|
|
|
|
## SearchResult Model
|
|
|
|
```python
|
|
@dataclass
|
|
class SearchResult:
|
|
title: str # Article/title text
|
|
url: str # Direct URL
|
|
snippet: str # Summary text (~300 chars)
|
|
source: str # Engine name (e.g. "bing", "arxiv")
|
|
category: str # "web" | "news" | "academic" | "github"
|
|
rank: int # Position after deduplication
|
|
published: str # ISO date (optional)
|
|
extra: dict # Engine-specific metadata
|
|
```
|