refactor: cleanup dead code, fix model confusion, streamline CLI

- Remove 3 dead modules: academic/, wechat/, social/ (merged into engines/)
- Remove 4 stale news scripts (tech_news.py, wallstreetcn.py, cls.py, rss.py)
- scripts/core/base.py: canonical SearchResult + BaseEngine (was in engines/base.py)
- scripts/engines/base.py: re-exports from core.base for compatibility
- scripts/engines/news_engine.py: Bing News + RSS unified (was in scripts/news/rss.py)
- scripts/engines/academic_engine.py: ArXiv + CrossRef unified (was in scripts/academic/arxiv.py)
- scripts/cli.py: complete rewrite — 10 commands (search/web/news/academic/wechat/social/rss/baidu/health/sources)
- scripts/unified_search.py: clean async Python API (was a stub)
- ws.py: lightweight launcher
- README.md + SKILL.md: updated to reflect actual 15+ engines
- All EngineResult → SearchResult
- All engines return list[SearchResult]
This commit is contained in:
openbot-coder committed 2026-05-30 16:25:41 +08:00
1 parent e20478fcbd
commit c077a7ae40
31 files changed
+1188 -3148

No files matched your search

+119 -27
View File
@@ -1,45 +1,137 @@
# web-search-skills
Web search agent skills for nanobot.
> Multi-source web search CLI & Python library — 15+ engines, <5s, no API key required.
## Features
- Multi-engine search: Bing + Sogou (primary), Baidu Qianfan API (backup)
- News search: BingNews + 14 RSS feeds in parallel
- Academic search: ArXiv + CrossRef API
- GitHub search via public API
- Parallel dispatch with URL-based deduplication
- Result aggregation with relevance scoring
- In-memory LRU cache (500 items, 1h TTL)
- Execution logging to `logs/search_log.jsonl`
- **15+ engines** across 5 categories: Web, News, Academic, Social, RSS
- **Concurrent execution** via `asyncio.gather` with semaphore (max 5 parallel)
- **Free by default** — most engines work without any API key
- **Python API** for programmatic use (`UnifiedSearch`)
- **Single model** — `SearchResult` dataclass across all engines
## Installation
## Quick Start
```bash
cd skills/web-search
npm install
# Install
pip install -e .
# CLI
web-search search "AI news" -n 10
web-search news "quantum computing" -n 5
web-search academic "machine learning"
web-search wechat 蒙面财经
web-search health
```
## Usage
## Commands
```bash
nanobot skill run web-search --query "your search query"
| Command | Description | Speed |
|---------|------------|-------|
| `search` | All 15 engines in parallel | ~10s |
| `web` | Web engines only (6) | ~5s |
| `news` | Bing News + 15 RSS feeds | ~8s |
| `academic` | ArXiv + CrossRef in parallel | ~6s |
| `wechat` | WeChat via Sogou | ~5s |
| `social` | Twitter/X via Nitter | ~5s |
| `rss` | RSS feed aggregation | ~8s |
| `baidu` | Baidu Qianfan API (needs key) | varies |
| `health` | Engine connectivity check | ~10s |
| `sources` | List all sources | instant |
## Options
- `-n N` — max results per engine (default: 10)
- `-s SRC...` — run specific sources only
- `-c CAT` — run by category (web/news/academic/social/rss)
- `-r cn|global` — region (default: cn)
- `-j` — JSON output
## Python API
```python
from scripts.unified_search import UnifiedSearch
async def main():
us = UnifiedSearch()
resp = await us.search("quantum computing", sources=["bing", "arxiv"], max_results=5)
for r in resp.results:
print(f"[{r.rank}] {r.title}")
print(f" {r.url}")
await us.close()
```
## Architecture
- `unified_search()` — main entry, handles query planning and engine routing
- `ResultAggregator` — deduplication + relevance scoring
- `CacheLayer` — LRU in-memory cache
- `EnginePool` — manages engine health and selection
```
scripts/
cli.py # CLI entry point
unified_search.py # Python API
models.py # Data models
config.py # Settings
core/
base.py # SearchResult + BaseEngine (canonical)
aggregator.py # Deduplication + ranking
execution_log.py # Timing + error tracking
engines/
bing_scraper.py # Bing web search
sogou_scraper.py # Sogou web search
baidu_scraper.py # Baidu web search
search360_scraper.py # 360 search
parser_engines.py # DuckDuckGo + Brave (HTML parser)
news_engine.py # Bing News + RSS aggregator
academic_engine.py # ArXiv + CrossRef
github_engine.py # GitHub API
baidu_qianfan.py # Baidu Qianfan API (needs key)
parallel_search.py # Parallel orchestrator
```
## Engines
| Engine | Type | Priority |
|--------|------|----------|
| Bing | Web (cn.bing.com / www.bing.com) | Primary |
| Sogou | Web (cn.sogou.com) | Primary |
| Baidu Qianfan | API (official) | Backup |
| BingNews | News | Primary |
| ArXiv | Academic | Primary |
| CrossRef | Academic | Primary |
### Web (无需 API key)
| Engine | URL | Notes |
|--------|-----|-------|
| Bing CN | cn.bing.com | Primary |
| Bing INT | www.bing.com | Global fallback |
| Sogou | sogou.com | Primary CN |
| Baidu | baidu.com | Primary CN |
| 360 | so.com | CN fallback |
| DuckDuckGo | html.duckduckgo.com | Global |
| Brave | search.brave.com | Global |
### News
| Engine | Source |
|--------|--------|
| Bing News | bing.com/news |
| RSS (15 feeds) | 36kr, IT之家, 钛媒体, HackerNews, TechCrunch... |
### Academic (Free APIs)
| Engine | API |
|--------|-----|
| ArXiv | export.arxiv.org/api/query |
| CrossRef | api.crossref.org |
### Social
| Engine | Notes |
|--------|-------|
| Nitter | twitter.com 免登录 (多个实例自动切换) |
### API (需配置密钥)
| Engine | Env var |
|--------|---------|
| Baidu Qianfan | `BAIDU_API_KEY` |
## SearchResult Model
```python
@dataclass
class SearchResult:
title: str # Article/title text
url: str # Direct URL
snippet: str # Summary text (~300 chars)
source: str # Engine name (e.g. "bing", "arxiv")
category: str # "web" | "news" | "academic" | "github"
rank: int # Position after deduplication
published: str # ISO date (optional)
extra: dict # Engine-specific metadata
```