main
refactor: cleanup dead code, fix model confusion, streamline CLI
web-search-skills
Multi-source web search CLI & Python library — 15+ engines, <5s, no API key required.
Features
- 15+ engines across 5 categories: Web, News, Academic, Social, RSS
- Concurrent execution via
asyncio.gatherwith semaphore (max 5 parallel) - Free by default — most engines work without any API key
- Python API for programmatic use (
UnifiedSearch) - Single model —
SearchResultdataclass across all engines
Quick Start
# Install
pip install -e .
# CLI
web-search search "AI news" -n 10
web-search news "quantum computing" -n 5
web-search academic "machine learning"
web-search wechat 蒙面财经
web-search health
Commands
| Command | Description | Speed |
|---|---|---|
search |
All 15 engines in parallel | ~10s |
web |
Web engines only (6) | ~5s |
news |
Bing News + 15 RSS feeds | ~8s |
academic |
ArXiv + CrossRef in parallel | ~6s |
wechat |
WeChat via Sogou | ~5s |
social |
Twitter/X via Nitter | ~5s |
rss |
RSS feed aggregation | ~8s |
baidu |
Baidu Qianfan API (needs key) | varies |
health |
Engine connectivity check | ~10s |
sources |
List all sources | instant |
Options
-n N— max results per engine (default: 10)-s SRC...— run specific sources only-c CAT— run by category (web/news/academic/social/rss)-r cn|global— region (default: cn)-j— JSON output
Python API
from scripts.unified_search import UnifiedSearch
async def main():
us = UnifiedSearch()
resp = await us.search("quantum computing", sources=["bing", "arxiv"], max_results=5)
for r in resp.results:
print(f"[{r.rank}] {r.title}")
print(f" {r.url}")
await us.close()
Architecture
scripts/
cli.py # CLI entry point
unified_search.py # Python API
models.py # Data models
config.py # Settings
core/
base.py # SearchResult + BaseEngine (canonical)
aggregator.py # Deduplication + ranking
execution_log.py # Timing + error tracking
engines/
bing_scraper.py # Bing web search
sogou_scraper.py # Sogou web search
baidu_scraper.py # Baidu web search
search360_scraper.py # 360 search
parser_engines.py # DuckDuckGo + Brave (HTML parser)
news_engine.py # Bing News + RSS aggregator
academic_engine.py # ArXiv + CrossRef
github_engine.py # GitHub API
baidu_qianfan.py # Baidu Qianfan API (needs key)
parallel_search.py # Parallel orchestrator
Engines
Web (无需 API key)
| Engine | URL | Notes |
|---|---|---|
| Bing CN | cn.bing.com | Primary |
| Bing INT | www.bing.com | Global fallback |
| Sogou | sogou.com | Primary CN |
| Baidu | baidu.com | Primary CN |
| 360 | so.com | CN fallback |
| DuckDuckGo | html.duckduckgo.com | Global |
| Brave | search.brave.com | Global |
News
| Engine | Source |
|---|---|
| Bing News | bing.com/news |
| RSS (15 feeds) | 36kr, IT之家, 钛媒体, HackerNews, TechCrunch... |
Academic (Free APIs)
| Engine | API |
|---|---|
| ArXiv | export.arxiv.org/api/query |
| CrossRef | api.crossref.org |
Social
| Engine | Notes |
|---|---|
| Nitter | twitter.com 免登录 (多个实例自动切换) |
API (需配置密钥)
| Engine | Env var |
|---|---|
| Baidu Qianfan | BAIDU_API_KEY |
SearchResult Model
@dataclass
class SearchResult:
title: str # Article/title text
url: str # Direct URL
snippet: str # Summary text (~300 chars)
source: str # Engine name (e.g. "bing", "arxiv")
category: str # "web" | "news" | "academic" | "github"
rank: int # Position after deduplication
published: str # ISO date (optional)
extra: dict # Engine-specific metadata
Description
A collection of web search tools and skills for enhancing search capabilities
136 KiB
0 Stars
3 Watchers
0 Forks
Languages
Python
100%