diff --git a/README.md b/README.md index f2e55e5..d912fb6 100644 --- a/README.md +++ b/README.md @@ -1,164 +1,45 @@ -# Web Search Skills +# web-search-skills -Multi-source web search CLI tool — search across **28 engines** from your terminal: web search, news, WeChat articles, academic papers, social media, RSS feeds, and AI-powered Baidu search. +Web search agent skills for nanobot. -## Quick Start +## Features + +- Multi-engine search: Bing + Sogou (primary), Baidu Qianfan API (backup) +- News search: BingNews + 14 RSS feeds in parallel +- Academic search: ArXiv + CrossRef API +- GitHub search via public API +- Parallel dispatch with URL-based deduplication +- Result aggregation with relevance scoring +- In-memory LRU cache (500 items, 1h TTL) +- Execution logging to `logs/search_log.jsonl` + +## Installation ```bash -# Option 1: pip install (recommended for CodeBuddy Agent use) -cd web-search-skills -pip install -e . - -# Option 2: uv tool install (alternative) -uv tool install -e . - -# Now use globally from anywhere -web-search wechat 蒙面财经 -web-search news "AI" -n 5 -web-search search "量子计算" -o results.json +cd skills/web-search +npm install ``` -Or run directly without install: -```bash -python ws.py wechat 蒙面财经 -``` - -> Requires Python 3.8+. After `pip install -e .`, the `web-search` command is available globally. - ## Usage ```bash -web-search [options] +nanobot skill run web-search --query "your search query" ``` -| Command | Description | -|------------|-----------------------------------------------------| -| `search` | Search 28 sources in parallel (default) | -| `web` | Web search engines (Google, Baidu, DDG, Yahoo…) | -| `news` | News (Hacker News, GitHub Trending, 36氪, RSS…) | -| `baidu` | AI-powered Baidu search (site/recency filters) | -| `wechat` | WeChat public account articles (via Sogou) | -| `academic` | Academic papers (ArXiv) | -| `social` | Social media (Twitter/X) | -| `rss` | RSS feeds (AI newsletters, essays, podcasts…) | -| `health` | Test connectivity of all 28 engines (concurrent) | -| `sources` | List all available search sources | -| `urls` | Generate search URLs without executing | +## Architecture -### Options +- `unified_search()` — main entry, handles query planning and engine routing +- `ResultAggregator` — deduplication + relevance scoring +- `CacheLayer` — LRU in-memory cache +- `EnginePool` — manages engine health and selection -| Flag | Description | -|-------------------|----------------------------------------------| -| `-n N` | Max results per source (default: 10) | -| `-j` | JSON output | -| `-o FILE` | Save to JSON file (path restricted to current directory) | -| `-s NAME [...]` | Specific source(s) — engine names or categories (e.g. `-s "Hacker News" V2EX`, `-s web news`) | -| `-r cn/global` | Region filter (web search) | -| `--site SITE` | Restrict to site (baidu command only) | -| `--recency RECENCY` | Time filter: day/week/month/semiyear/year | -| `-v` | Verbose/debug logs | +## Engines -### Examples - -```bash -# Search specific engines concurrently (new!) -web-search search "technology" -s "Hacker News" "V2EX" "GitHub Trending" -n 3 - -# Search two categories concurrently (new!) -web-search search "AI" -s web news -n 3 - -# Baidu with site & recency filters -web-search baidu "中美元首会晤" --site mp.weixin.qq.com --recency week - -# Quick search -web-search news "人工智能" -n 5 -web-search wechat 蒙面财经 -web-search academic "quantum computing" -web-search search "latest AI news" -o results.json - -# Check engine health (concurrent, ~14s for 28 engines) -web-search health -``` - -## Sources (28 engines) - -| Category | Sources | Count | -|----------|---------|-------| -| **Web** | Baidu, Bing CN, Bing INT, 360, Sogou, Shenma, Google, Google HK, DuckDuckGo, Yahoo, Startpage, Brave, Ecosia, Qwant | 14 | -| **News (CN)** | 财联社, 华尔街见闻, 36氪, 微博热搜, V2EX, 腾讯新闻 | 6 | -| **News (Global)** | Hacker News, GitHub Trending | 2 | -| **RSS** | Reuters, AP, BBC, FT, CNN, Al Jazeera + AI newsletters (Latent Space, ChinAI, KDnuggets…), essays (Paul Graham, James Clear…), podcasts (Lex Fridman…) | 18 | -| **Academic** | ArXiv | 1 | -| **WeChat** | Sogou WeChat | 1 | -| **Social** | Twitter, Twitter Latest | 2 | -| **API** | Baidu Qianfan (AI search with site/recency) | 1 | -| **Special** | WolframAlpha | 1 | - -## Project Structure - -``` -web-search-skills/ -├── scripts/ -│ ├── cli.py # CLI entry point -│ ├── unified_search.py # Unified search orchestrator (async + concurrent) -│ ├── core/ -│ │ ├── base.py # SearchResult & SearchEngine base class -│ │ ├── config_loader.py # Load engine config from JSON -│ │ └── thread_utils.py # Shared thread pool for HTML parsing -│ ├── engines/ -│ │ ├── url_engine.py # URL-template search engines -│ │ ├── parser_engines.py # HTML-parsed search (DuckDuckGo) -│ │ └── baidu_qianfan.py # Baidu Qianfan AI Search API -│ ├── news/ -│ │ ├── cls.py # 财联社 -│ │ ├── wallstreetcn.py # 华尔街见闻 -│ │ ├── rss.py # RSS feeds (18 sources) -│ │ └── tech_news.py # Tech news engines (Hacker News, GitHub Trending, 36氪, etc.) -│ ├── academic/ -│ │ └── arxiv.py # ArXiv papers -│ ├── wechat/ -│ │ └── sogou_weixin.py # Sogou WeChat search -│ └── social/ -│ └── twitter.py # Twitter/X search -├── config/ -│ └── engines.json # Search engine definitions -├── ws.py # Direct run entry point -├── pyproject.toml # Package config -├── SKILL.md # CodeBuddy skill definition -├── LICENSE -└── README.md -``` - -## Python API - -```python -from scripts.unified_search import UnifiedSearch - -async def main(): - us = UnifiedSearch() - # Search specific sources concurrently - results = await us.search("quantum computing", - sources=["Hacker News", "ArXiv"], - max_results=5) - for r in results: - print(f"[{r.rank}] {r.title}: {r.url}") - await us.close() -``` - -## Performance - -- **Concurrent engine execution**: All 28 engines run in parallel via `asyncio.gather`, with `Semaphore(5)` for gentle concurrency -- **Health check**: 28 engines checked in ~14 seconds (vs ~84s without concurrency) -- **Thread pool**: HTML parsing offloaded to `ThreadPoolExecutor` via `scripts/core/thread_utils.py`, preventing event loop blocking -- **Timeout**: 15s per engine timeout prevents hanging - -## Notes - -- Sogou WeChat has anti-scraping limits, may return empty results -- 财联社 and 华尔街见闻 require JavaScript rendering; `UnifiedSearch` returns search page links for these -- RSS feeds require stable network connectivity -- Baidu Qianfan requires `BAIDU_API_KEY` environment variable - -## License - -MIT +| Engine | Type | Priority | +|--------|------|----------| +| Bing | Web (cn.bing.com / www.bing.com) | Primary | +| Sogou | Web (cn.sogou.com) | Primary | +| Baidu Qianfan | API (official) | Backup | +| BingNews | News | Primary | +| ArXiv | Academic | Primary | +| CrossRef | Academic | Primary |