refactor: cleanup dead code, fix model confusion, streamline CLI
- Remove 3 dead modules: academic/, wechat/, social/ (merged into engines/) - Remove 4 stale news scripts (tech_news.py, wallstreetcn.py, cls.py, rss.py) - scripts/core/base.py: canonical SearchResult + BaseEngine (was in engines/base.py) - scripts/engines/base.py: re-exports from core.base for compatibility - scripts/engines/news_engine.py: Bing News + RSS unified (was in scripts/news/rss.py) - scripts/engines/academic_engine.py: ArXiv + CrossRef unified (was in scripts/academic/arxiv.py) - scripts/cli.py: complete rewrite — 10 commands (search/web/news/academic/wechat/social/rss/baidu/health/sources) - scripts/unified_search.py: clean async Python API (was a stub) - ws.py: lightweight launcher - README.md + SKILL.md: updated to reflect actual 15+ engines - All EngineResult → SearchResult - All engines return list[SearchResult]
This commit is contained in:
1 parent
e20478fcbd
commit
c077a7ae40
31 files changed
+1188
-3148
No files matched your search
@@ -1,45 +1,137 @@
|
||||
# web-search-skills
|
||||
|
||||
Web search agent skills for nanobot.
|
||||
> Multi-source web search CLI & Python library — 15+ engines, <5s, no API key required.
|
||||
|
||||
## Features
|
||||
|
||||
- Multi-engine search: Bing + Sogou (primary), Baidu Qianfan API (backup)
|
||||
- News search: BingNews + 14 RSS feeds in parallel
|
||||
- Academic search: ArXiv + CrossRef API
|
||||
- GitHub search via public API
|
||||
- Parallel dispatch with URL-based deduplication
|
||||
- Result aggregation with relevance scoring
|
||||
- In-memory LRU cache (500 items, 1h TTL)
|
||||
- Execution logging to `logs/search_log.jsonl`
|
||||
- **15+ engines** across 5 categories: Web, News, Academic, Social, RSS
|
||||
- **Concurrent execution** via `asyncio.gather` with semaphore (max 5 parallel)
|
||||
- **Free by default** — most engines work without any API key
|
||||
- **Python API** for programmatic use (`UnifiedSearch`)
|
||||
- **Single model** — `SearchResult` dataclass across all engines
|
||||
|
||||
## Installation
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
cd skills/web-search
|
||||
npm install
|
||||
# Install
|
||||
pip install -e .
|
||||
|
||||
# CLI
|
||||
web-search search "AI news" -n 10
|
||||
web-search news "quantum computing" -n 5
|
||||
web-search academic "machine learning"
|
||||
web-search wechat 蒙面财经
|
||||
web-search health
|
||||
```
|
||||
|
||||
## Usage
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
nanobot skill run web-search --query "your search query"
|
||||
| Command | Description | Speed |
|
||||
|---------|------------|-------|
|
||||
| `search` | All 15 engines in parallel | ~10s |
|
||||
| `web` | Web engines only (6) | ~5s |
|
||||
| `news` | Bing News + 15 RSS feeds | ~8s |
|
||||
| `academic` | ArXiv + CrossRef in parallel | ~6s |
|
||||
| `wechat` | WeChat via Sogou | ~5s |
|
||||
| `social` | Twitter/X via Nitter | ~5s |
|
||||
| `rss` | RSS feed aggregation | ~8s |
|
||||
| `baidu` | Baidu Qianfan API (needs key) | varies |
|
||||
| `health` | Engine connectivity check | ~10s |
|
||||
| `sources` | List all sources | instant |
|
||||
|
||||
## Options
|
||||
|
||||
- `-n N` — max results per engine (default: 10)
|
||||
- `-s SRC...` — run specific sources only
|
||||
- `-c CAT` — run by category (web/news/academic/social/rss)
|
||||
- `-r cn|global` — region (default: cn)
|
||||
- `-j` — JSON output
|
||||
|
||||
## Python API
|
||||
|
||||
```python
|
||||
from scripts.unified_search import UnifiedSearch
|
||||
|
||||
async def main():
|
||||
us = UnifiedSearch()
|
||||
resp = await us.search("quantum computing", sources=["bing", "arxiv"], max_results=5)
|
||||
for r in resp.results:
|
||||
print(f"[{r.rank}] {r.title}")
|
||||
print(f" {r.url}")
|
||||
await us.close()
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
- `unified_search()` — main entry, handles query planning and engine routing
|
||||
- `ResultAggregator` — deduplication + relevance scoring
|
||||
- `CacheLayer` — LRU in-memory cache
|
||||
- `EnginePool` — manages engine health and selection
|
||||
```
|
||||
scripts/
|
||||
cli.py # CLI entry point
|
||||
unified_search.py # Python API
|
||||
models.py # Data models
|
||||
config.py # Settings
|
||||
core/
|
||||
base.py # SearchResult + BaseEngine (canonical)
|
||||
aggregator.py # Deduplication + ranking
|
||||
execution_log.py # Timing + error tracking
|
||||
engines/
|
||||
bing_scraper.py # Bing web search
|
||||
sogou_scraper.py # Sogou web search
|
||||
baidu_scraper.py # Baidu web search
|
||||
search360_scraper.py # 360 search
|
||||
parser_engines.py # DuckDuckGo + Brave (HTML parser)
|
||||
news_engine.py # Bing News + RSS aggregator
|
||||
academic_engine.py # ArXiv + CrossRef
|
||||
github_engine.py # GitHub API
|
||||
baidu_qianfan.py # Baidu Qianfan API (needs key)
|
||||
parallel_search.py # Parallel orchestrator
|
||||
```
|
||||
|
||||
## Engines
|
||||
|
||||
| Engine | Type | Priority |
|
||||
|--------|------|----------|
|
||||
| Bing | Web (cn.bing.com / www.bing.com) | Primary |
|
||||
| Sogou | Web (cn.sogou.com) | Primary |
|
||||
| Baidu Qianfan | API (official) | Backup |
|
||||
| BingNews | News | Primary |
|
||||
| ArXiv | Academic | Primary |
|
||||
| CrossRef | Academic | Primary |
|
||||
### Web (无需 API key)
|
||||
| Engine | URL | Notes |
|
||||
|--------|-----|-------|
|
||||
| Bing CN | cn.bing.com | Primary |
|
||||
| Bing INT | www.bing.com | Global fallback |
|
||||
| Sogou | sogou.com | Primary CN |
|
||||
| Baidu | baidu.com | Primary CN |
|
||||
| 360 | so.com | CN fallback |
|
||||
| DuckDuckGo | html.duckduckgo.com | Global |
|
||||
| Brave | search.brave.com | Global |
|
||||
|
||||
### News
|
||||
| Engine | Source |
|
||||
|--------|--------|
|
||||
| Bing News | bing.com/news |
|
||||
| RSS (15 feeds) | 36kr, IT之家, 钛媒体, HackerNews, TechCrunch... |
|
||||
|
||||
### Academic (Free APIs)
|
||||
| Engine | API |
|
||||
|--------|-----|
|
||||
| ArXiv | export.arxiv.org/api/query |
|
||||
| CrossRef | api.crossref.org |
|
||||
|
||||
### Social
|
||||
| Engine | Notes |
|
||||
|--------|-------|
|
||||
| Nitter | twitter.com 免登录 (多个实例自动切换) |
|
||||
|
||||
### API (需配置密钥)
|
||||
| Engine | Env var |
|
||||
|--------|---------|
|
||||
| Baidu Qianfan | `BAIDU_API_KEY` |
|
||||
|
||||
## SearchResult Model
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class SearchResult:
|
||||
title: str # Article/title text
|
||||
url: str # Direct URL
|
||||
snippet: str # Summary text (~300 chars)
|
||||
source: str # Engine name (e.g. "bing", "arxiv")
|
||||
category: str # "web" | "news" | "academic" | "github"
|
||||
rank: int # Position after deduplication
|
||||
published: str # ISO date (optional)
|
||||
extra: dict # Engine-specific metadata
|
||||
```
|
||||
Reference in new issue
Block a user