docs: 重构 SKILL.md 为 Agent 决策驱动,更新文档版本号及 CLI docstring
This commit is contained in:
1 parent
39c01ebf69
commit
a81661d828
9 files changed
+912
-163
No files matched your search
@@ -1,6 +1,6 @@
|
|||||||
# Web Search Skills
|
# Web Search Skills
|
||||||
|
|
||||||
Multi-source web search CLI tool — search across 20+ engines from your terminal.
|
Multi-source web search CLI tool — search across **28 engines** from your terminal: web search, news, WeChat articles, academic papers, social media, RSS feeds, and AI-powered Baidu search.
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
@@ -27,51 +27,94 @@ web-search <command> <query> [options]
|
|||||||
```
|
```
|
||||||
|
|
||||||
| Command | Description |
|
| Command | Description |
|
||||||
|------------|------------------------------------------|
|
|------------|-----------------------------------------------------|
|
||||||
| `search` | Search all sources (default) |
|
| `search` | Search 28 sources in parallel (default) |
|
||||||
| `web` | Web search engines (Google, Baidu, DDG) |
|
| `web` | Web search engines (Google, Baidu, DDG, Yahoo…) |
|
||||||
| `news` | News (财联社, 华尔街见闻, RSS) |
|
| `news` | News (Hacker News, GitHub Trending, 36氪, RSS…) |
|
||||||
| `wechat` | WeChat public account articles |
|
| `baidu` | AI-powered Baidu search (site/recency filters) |
|
||||||
|
| `wechat` | WeChat public account articles (via Sogou) |
|
||||||
| `academic` | Academic papers (ArXiv) |
|
| `academic` | Academic papers (ArXiv) |
|
||||||
| `social` | Social media (Twitter/X) |
|
| `social` | Social media (Twitter/X) |
|
||||||
| `rss` | RSS news feeds |
|
| `rss` | RSS feeds (AI newsletters, essays, podcasts…) |
|
||||||
|
| `health` | Test connectivity of all 28 engines (concurrent) |
|
||||||
| `sources` | List all available search sources |
|
| `sources` | List all available search sources |
|
||||||
| `urls` | Generate search URLs without executing |
|
| `urls` | Generate search URLs without executing |
|
||||||
|
|
||||||
Options:
|
### Options
|
||||||
|
|
||||||
| Flag | Description |
|
| Flag | Description |
|
||||||
|------|-------------|
|
|-------------------|----------------------------------------------|
|
||||||
| `-n N` | Max results per source (default: 10) |
|
| `-n N` | Max results per source (default: 10) |
|
||||||
| `-j` | JSON output |
|
| `-j` | JSON output |
|
||||||
| `-o FILE` | Save to JSON file |
|
| `-o FILE` | Save to JSON file |
|
||||||
| `-s NAME` | Specific source name(s) |
|
| `-s NAME [...]` | Specific source(s) — engine names or categories (e.g. `-s "Hacker News" V2EX`, `-s web news`) |
|
||||||
| `-r cn/global` | Region filter (web search) |
|
| `-r cn/global` | Region filter (web search) |
|
||||||
|
| `--site SITE` | Restrict to site (baidu command only) |
|
||||||
|
| `--recency RECENCY` | Time filter: day/week/month/semiyear/year |
|
||||||
| `-v` | Verbose/debug logs |
|
| `-v` | Verbose/debug logs |
|
||||||
|
|
||||||
|
### Examples
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Search specific engines concurrently (new!)
|
||||||
|
web-search search "technology" -s "Hacker News" "V2EX" "GitHub Trending" -n 3
|
||||||
|
|
||||||
|
# Search two categories concurrently (new!)
|
||||||
|
web-search search "AI" -s web news -n 3
|
||||||
|
|
||||||
|
# Baidu with site & recency filters
|
||||||
|
web-search baidu "中美元首会晤" --site mp.weixin.qq.com --recency week
|
||||||
|
|
||||||
|
# Quick search
|
||||||
|
web-search news "人工智能" -n 5
|
||||||
|
web-search wechat 蒙面财经
|
||||||
|
web-search academic "quantum computing"
|
||||||
|
web-search search "latest AI news" -o results.json
|
||||||
|
|
||||||
|
# Check engine health (concurrent, ~14s for 28 engines)
|
||||||
|
web-search health
|
||||||
|
```
|
||||||
|
|
||||||
|
## Sources (28 engines)
|
||||||
|
|
||||||
|
| Category | Sources | Count |
|
||||||
|
|----------|---------|-------|
|
||||||
|
| **Web** | Baidu, Bing CN, Bing INT, 360, Sogou, Shenma, Google, Google HK, DuckDuckGo, Yahoo, Startpage, Brave, Ecosia, Qwant | 14 |
|
||||||
|
| **News (CN)** | 财联社, 华尔街见闻, 36氪, 微博热搜, V2EX, 腾讯新闻 | 6 |
|
||||||
|
| **News (Global)** | Hacker News, GitHub Trending | 2 |
|
||||||
|
| **RSS** | Reuters, AP, BBC, FT, CNN, Al Jazeera + AI newsletters (Latent Space, ChinAI, KDnuggets…), essays (Paul Graham, James Clear…), podcasts (Lex Fridman…) | 18 |
|
||||||
|
| **Academic** | ArXiv | 1 |
|
||||||
|
| **WeChat** | Sogou WeChat | 1 |
|
||||||
|
| **Social** | Twitter, Twitter Latest | 2 |
|
||||||
|
| **API** | Baidu Qianfan (AI search with site/recency) | 1 |
|
||||||
|
| **Special** | WolframAlpha | 1 |
|
||||||
|
|
||||||
## Project Structure
|
## Project Structure
|
||||||
|
|
||||||
```
|
```
|
||||||
web-search-skills/
|
web-search-skills/
|
||||||
├── scripts/
|
├── scripts/
|
||||||
│ ├── cli.py # CLI entry point
|
│ ├── cli.py # CLI entry point
|
||||||
│ ├── unified_search.py # Unified search orchestrator
|
│ ├── unified_search.py # Unified search orchestrator (async + concurrent)
|
||||||
│ ├── core/
|
│ ├── core/
|
||||||
│ │ ├── base.py # SearchResult & SearchEngine base class
|
│ │ ├── base.py # SearchResult & SearchEngine base class
|
||||||
│ │ └── config_loader.py # Load engine config from JSON
|
│ │ ├── config_loader.py # Load engine config from JSON
|
||||||
|
│ │ └── thread_utils.py # Shared thread pool for HTML parsing
|
||||||
│ ├── engines/
|
│ ├── engines/
|
||||||
│ │ ├── url_engine.py # URL-template search engines
|
│ │ ├── url_engine.py # URL-template search engines
|
||||||
│ │ └── parser_engines.py # HTML-parsed search (DuckDuckGo)
|
│ │ ├── parser_engines.py # HTML-parsed search (DuckDuckGo)
|
||||||
|
│ │ └── baidu_qianfan.py # Baidu Qianfan AI Search API
|
||||||
│ ├── news/
|
│ ├── news/
|
||||||
│ │ ├── cls.py # 财联社
|
│ │ ├── cls.py # 财联社
|
||||||
│ │ ├── wallstreetcn.py # 华尔街见闻
|
│ │ ├── wallstreetcn.py # 华尔街见闻
|
||||||
│ │ └── rss.py # RSS feeds
|
│ │ ├── rss.py # RSS feeds (18 sources)
|
||||||
|
│ │ └── tech_news.py # Tech news engines (Hacker News, GitHub Trending, 36氪, etc.)
|
||||||
│ ├── academic/
|
│ ├── academic/
|
||||||
│ │ └── arxiv.py # ArXiv papers
|
│ │ └── arxiv.py # ArXiv papers
|
||||||
│ ├── wechat/
|
│ ├── wechat/
|
||||||
│ │ └── sogou_weixin.py # WeChat (via Sogou)
|
│ │ └── sogou_weixin.py # Sogou WeChat search
|
||||||
│ └── social/
|
│ └── social/
|
||||||
│ └── twitter.py # Twitter/X
|
│ └── twitter.py # Twitter/X search
|
||||||
├── config/
|
├── config/
|
||||||
│ └── engines.json # Search engine definitions
|
│ └── engines.json # Search engine definitions
|
||||||
├── ws.py # Direct run entry point
|
├── ws.py # Direct run entry point
|
||||||
@@ -81,6 +124,36 @@ web-search-skills/
|
|||||||
└── README.md
|
└── README.md
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Python API
|
||||||
|
|
||||||
|
```python
|
||||||
|
from scripts.unified_search import UnifiedSearch
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
us = UnifiedSearch()
|
||||||
|
# Search specific sources concurrently
|
||||||
|
results = await us.search("quantum computing",
|
||||||
|
sources=["Hacker News", "ArXiv"],
|
||||||
|
max_results=5)
|
||||||
|
for r in results:
|
||||||
|
print(f"[{r.rank}] {r.title}: {r.url}")
|
||||||
|
await us.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
- **Concurrent engine execution**: All 28 engines run in parallel via `asyncio.gather`, with `Semaphore(5)` for gentle concurrency
|
||||||
|
- **Health check**: 28 engines checked in ~14 seconds (vs ~84s without concurrency)
|
||||||
|
- **Thread pool**: HTML parsing offloaded to `ThreadPoolExecutor` via `scripts/core/thread_utils.py`, preventing event loop blocking
|
||||||
|
- **Timeout**: 15s per engine timeout prevents hanging
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Sogou WeChat has anti-scraping limits, may return empty results
|
||||||
|
- 财联社 and 华尔街见闻 require JavaScript rendering; `UnifiedSearch` returns search page links for these
|
||||||
|
- RSS feeds require stable network connectivity
|
||||||
|
- Baidu Qianfan requires `BAIDU_API_KEY` environment variable
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
MIT
|
MIT
|
||||||
+7
-1
@@ -23,7 +23,13 @@
|
|||||||
],
|
],
|
||||||
"news_sources": [
|
"news_sources": [
|
||||||
{"name": "cls", "url": "https://www.cls.cn/searchPage?keyword={keyword}&type=all", "region": "cn", "type": "news"},
|
{"name": "cls", "url": "https://www.cls.cn/searchPage?keyword={keyword}&type=all", "region": "cn", "type": "news"},
|
||||||
{"name": "wallstreetcn", "url": "https://wallstreetcn.com/search?q={keyword}", "region": "cn", "type": "news"}
|
{"name": "wallstreetcn", "url": "https://wallstreetcn.com/search?q={keyword}", "region": "cn", "type": "news"},
|
||||||
|
{"name": "Hacker News", "type": "news", "region": "global"},
|
||||||
|
{"name": "GitHub Trending", "type": "news", "region": "global"},
|
||||||
|
{"name": "36氪", "type": "news", "region": "cn"},
|
||||||
|
{"name": "微博热搜", "type": "news", "region": "cn"},
|
||||||
|
{"name": "V2EX", "type": "news", "region": "cn"},
|
||||||
|
{"name": "腾讯新闻", "type": "news", "region": "cn"}
|
||||||
],
|
],
|
||||||
"academic_sources": [
|
"academic_sources": [
|
||||||
{"name": "arxiv", "url": "https://export.arxiv.org/api/query?search_query=all:{keyword}&max_results=10", "region": "global", "type": "api"}
|
{"name": "arxiv", "url": "https://export.arxiv.org/api/query?search_query=all:{keyword}&max_results=10", "region": "global", "type": "api"}
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "web-search-skills"
|
name = "web-search-skills"
|
||||||
version = "2.1.0"
|
version = "2.2.0"
|
||||||
description = "Multi-source web search CLI tool"
|
description = "Multi-source web search CLI tool"
|
||||||
requires-python = ">=3.8"
|
requires-python = ">=3.8"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
|||||||
+27
-20
@@ -1,41 +1,45 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""
|
"""
|
||||||
Web Search Skills CLI — search across 20+ sources from the command line.
|
Web Search Skills CLI — search across 28 sources from the command line.
|
||||||
|
|
||||||
Usage:
|
Usage:
|
||||||
web-search <command> <query> [options]
|
web-search <command> <query> [options]
|
||||||
|
|
||||||
Commands:
|
Commands:
|
||||||
search Search all sources (default)
|
search Search all 28 sources in parallel (default)
|
||||||
web Search web search engines (Google, Baidu, DuckDuckGo, etc.)
|
web Web search engines (Google, Baidu, DuckDuckGo, etc.)
|
||||||
baidu Search via Baidu Qianfan AI Search API (需配置 BAIDU_API_KEY)
|
baidu Baidu Qianfan AI Search API (需配置 BAIDU_API_KEY)
|
||||||
news Search news (财联社, 华尔街见闻, RSS)
|
news News (Hacker News, GitHub Trending, 36氪, RSS, etc.)
|
||||||
wechat Search WeChat public account articles (via Sougou)
|
wechat WeChat public account articles (via Sogou)
|
||||||
academic Search academic papers (ArXiv)
|
academic Academic papers (ArXiv)
|
||||||
social Search social media (Twitter/X)
|
social Social media (Twitter/X)
|
||||||
rss Search RSS news feeds
|
rss RSS feeds (AI newsletters, essays, podcasts)
|
||||||
health Test engine connectivity
|
health Test connectivity of all 28 engines (concurrent)
|
||||||
sources List all available sources
|
sources List all available sources
|
||||||
urls Get search URLs without executing
|
urls Generate search URLs without executing
|
||||||
|
|
||||||
Options:
|
Options:
|
||||||
-n, --max-results N Max results per source (default: 10)
|
-n, --max-results N Max results per source (default: 10)
|
||||||
-j, --json Output as JSON
|
-j, --json Output as JSON
|
||||||
-o, --output FILE Save results to file
|
-o, --output FILE Save results to file
|
||||||
-s, --source NAME Specific source name(s)
|
-s, --source NAME Specific source name(s), support multiple
|
||||||
-r, --region REGION Region filter (cn/global)
|
-r, --region REGION Region filter: cn / global (web only)
|
||||||
--site SITE Restrict to specific site (baidu command only, e.g. mp.weixin.qq.com)
|
--site SITE Restrict to site (baidu only, e.g. mp.weixin.qq.com)
|
||||||
--recency RECENCY Time filter: day, week, month, semiyear, year (baidu command only)
|
--recency RECENCY Time filter: day/week/month/semiyear/year (baidu only)
|
||||||
-v, --verbose Show debug logs
|
-v, --verbose Show debug logs
|
||||||
-h, --help Show this help
|
-h, --help Show this help
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
web-search wechat 蒙面财经
|
|
||||||
web-search baidu 今日新闻 --recency week
|
|
||||||
web-search baidu "央视新闻 中美元首会晤" --site mp.weixin.qq.com --recency week
|
|
||||||
web-search news "人工智能" -n 5
|
|
||||||
web-search web "Python async" -r global -j
|
|
||||||
web-search search "量子计算" -o results.json
|
web-search search "量子计算" -o results.json
|
||||||
|
web-search search "AI" -s web news -n 5
|
||||||
|
web-search search "technology" -s "Hacker News" "V2EX" -n 3
|
||||||
|
web-search web "Python async" -r global -j
|
||||||
|
web-search baidu "今日新闻" --recency week
|
||||||
|
web-search baidu "中美元首会晤" --site mp.weixin.qq.com
|
||||||
|
web-search news "人工智能" -n 5
|
||||||
|
web-search wechat 蒙面财经
|
||||||
|
web-search academic "quantum computing"
|
||||||
|
web-search social "Iran nuclear" -n 5
|
||||||
web-search health
|
web-search health
|
||||||
web-search sources
|
web-search sources
|
||||||
web-search urls "机器学习"
|
web-search urls "机器学习"
|
||||||
@@ -377,6 +381,9 @@ async def _run_search(args):
|
|||||||
kwargs["site"] = args.site
|
kwargs["site"] = args.site
|
||||||
if args.recency:
|
if args.recency:
|
||||||
kwargs["recency"] = args.recency
|
kwargs["recency"] = args.recency
|
||||||
|
# --source / -s: pass specific source names/categories to search
|
||||||
|
if args.source:
|
||||||
|
kwargs["sources"] = args.source
|
||||||
|
|
||||||
try:
|
try:
|
||||||
results = await search_fn(
|
results = await search_fn(
|
||||||
|
|||||||
@@ -0,0 +1,52 @@
|
|||||||
|
"""
|
||||||
|
Thread pool utilities for CPU-bound operations.
|
||||||
|
|
||||||
|
BeautifulSoup/HTML parsing is CPU-bound and can block the async event loop.
|
||||||
|
Use this shared thread pool to offload parsing work to background threads.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from concurrent.futures import ThreadPoolExecutor
|
||||||
|
from typing import Callable, TypeVar
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
T = TypeVar("T")
|
||||||
|
|
||||||
|
# Shared thread pool - max_workers=4 balances parallelism with overhead
|
||||||
|
_PARSE_POOL: ThreadPoolExecutor | None = None
|
||||||
|
_MAX_WORKERS = 4
|
||||||
|
|
||||||
|
|
||||||
|
def get_parse_pool() -> ThreadPoolExecutor:
|
||||||
|
"""Get the shared thread pool for parsing operations."""
|
||||||
|
global _PARSE_POOL
|
||||||
|
if _PARSE_POOL is None:
|
||||||
|
_PARSE_POOL = ThreadPoolExecutor(
|
||||||
|
max_workers=_MAX_WORKERS,
|
||||||
|
thread_name_prefix="parse",
|
||||||
|
)
|
||||||
|
return _PARSE_POOL
|
||||||
|
|
||||||
|
|
||||||
|
async def run_in_thread(func: Callable[..., T], *args, **kwargs) -> T:
|
||||||
|
"""
|
||||||
|
Run a CPU-bound function in a thread pool to avoid blocking the event loop.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
soup = await run_in_thread(BeautifulSoup, html, "lxml")
|
||||||
|
results = await run_in_thread(parse_func, html, max_results)
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
|
return await loop.run_in_executor(get_parse_pool(), func, *args, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def shutdown_pool():
|
||||||
|
"""Clean up the thread pool (call during application shutdown)."""
|
||||||
|
global _PARSE_POOL
|
||||||
|
if _PARSE_POOL:
|
||||||
|
_PARSE_POOL.shutdown(wait=True)
|
||||||
|
_PARSE_POOL = None
|
||||||
+80
-1
@@ -27,7 +27,8 @@ from scripts.core.base import SearchEngine, SearchResult
|
|||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
# RSS feed definitions (from news-summary skill + additional sources)
|
# RSS feed definitions
|
||||||
|
# Core sources (general news) + AI newsletters + essays + podcasts
|
||||||
RSS_FEEDS = {
|
RSS_FEEDS = {
|
||||||
"BBC World": {
|
"BBC World": {
|
||||||
"url": "https://feeds.bbci.co.uk/news/world/rss.xml",
|
"url": "https://feeds.bbci.co.uk/news/world/rss.xml",
|
||||||
@@ -64,6 +65,84 @@ RSS_FEEDS = {
|
|||||||
"category": "news",
|
"category": "news",
|
||||||
"lang": "en",
|
"lang": "en",
|
||||||
},
|
},
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
# AI Newsletters (inspired by news-aggregator-skill)
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
"Latent Space": {
|
||||||
|
"url": "https://feeds.transistor.fm/latent-space-podcast",
|
||||||
|
"category": "technology",
|
||||||
|
"lang": "en",
|
||||||
|
"subcategory": "ai_newsletter",
|
||||||
|
},
|
||||||
|
"ChinAI": {
|
||||||
|
"url": "https://chinai.substack.com/feed",
|
||||||
|
"category": "technology",
|
||||||
|
"lang": "en",
|
||||||
|
"subcategory": "ai_newsletter",
|
||||||
|
},
|
||||||
|
"Memia": {
|
||||||
|
"url": "https://memia.substack.com/feed",
|
||||||
|
"category": "technology",
|
||||||
|
"lang": "en",
|
||||||
|
"subcategory": "ai_newsletter",
|
||||||
|
},
|
||||||
|
"Interconnects": {
|
||||||
|
"url": "https://interconnects.substack.com/feed",
|
||||||
|
"category": "technology",
|
||||||
|
"lang": "en",
|
||||||
|
"subcategory": "ai_newsletter",
|
||||||
|
},
|
||||||
|
"KDnuggets": {
|
||||||
|
"url": "https://www.kdnuggets.com/feed",
|
||||||
|
"category": "technology",
|
||||||
|
"lang": "en",
|
||||||
|
"subcategory": "ai_newsletter",
|
||||||
|
},
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
# Deep Essays & Thinkers (inspired by news-aggregator-skill)
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
"Paul Graham": {
|
||||||
|
"url": "https://paulgraham.com/rss.xml",
|
||||||
|
"category": "technology",
|
||||||
|
"lang": "en",
|
||||||
|
"subcategory": "essay",
|
||||||
|
},
|
||||||
|
"James Clear": {
|
||||||
|
"url": "https://jamesclear.com/feed",
|
||||||
|
"category": "news",
|
||||||
|
"lang": "en",
|
||||||
|
"subcategory": "essay",
|
||||||
|
},
|
||||||
|
"Wait But Why": {
|
||||||
|
"url": "https://waitbutwhy.com/feed",
|
||||||
|
"category": "news",
|
||||||
|
"lang": "en",
|
||||||
|
"subcategory": "essay",
|
||||||
|
},
|
||||||
|
"Farnam Street": {
|
||||||
|
"url": "https://fs.blog/feed/",
|
||||||
|
"category": "news",
|
||||||
|
"lang": "en",
|
||||||
|
"subcategory": "essay",
|
||||||
|
},
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
# Podcasts (inspired by news-aggregator-skill)
|
||||||
|
# -----------------------------------------------------------------------
|
||||||
|
"Lex Fridman Podcast": {
|
||||||
|
"url": "https://lexfridman.com/feed/podcast/",
|
||||||
|
"category": "technology",
|
||||||
|
"lang": "en",
|
||||||
|
"subcategory": "podcast",
|
||||||
|
},
|
||||||
|
"80,000 Hours": {
|
||||||
|
"url": "https://80000hours.org/feed/podcast/",
|
||||||
|
"category": "news",
|
||||||
|
"lang": "en",
|
||||||
|
"subcategory": "podcast",
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,461 @@
|
|||||||
|
"""
|
||||||
|
Tech & social news aggregation engines.
|
||||||
|
|
||||||
|
Implements non-Playwright news sources inspired by news-aggregator-skill:
|
||||||
|
- Hacker News (via Algolia API)
|
||||||
|
- GitHub Trending (HTML scrape)
|
||||||
|
- 36氪 (HTML scrape)
|
||||||
|
- 微博热搜 (Ajax API)
|
||||||
|
- V2EX (Public API)
|
||||||
|
- 腾讯新闻 (API)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Any, Optional
|
||||||
|
from urllib.parse import quote_plus
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
|
||||||
|
from scripts.core.base import SearchEngine, SearchResult
|
||||||
|
from scripts.core.thread_utils import run_in_thread
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Shared HTTP client settings
|
||||||
|
_DEFAULT_TIMEOUT = 15
|
||||||
|
_COMMON_HEADERS = {
|
||||||
|
"User-Agent": (
|
||||||
|
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
||||||
|
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||||
|
"Chrome/120.0.0.0 Safari/537.36"
|
||||||
|
),
|
||||||
|
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||||
|
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Hacker News (via Algolia API)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class HackerNewsEngine(SearchEngine):
|
||||||
|
"""Hacker News search via Algolia public API."""
|
||||||
|
|
||||||
|
SEARCH_URL = "https://hn.algolia.com/api/v1/search"
|
||||||
|
|
||||||
|
def __init__(self, config: Optional[dict] = None):
|
||||||
|
super().__init__("Hacker News", config)
|
||||||
|
self._client = httpx.AsyncClient(
|
||||||
|
timeout=_DEFAULT_TIMEOUT,
|
||||||
|
headers={"User-Agent": _COMMON_HEADERS["User-Agent"]},
|
||||||
|
follow_redirects=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10, **kwargs) -> list[SearchResult]:
|
||||||
|
try:
|
||||||
|
params = {
|
||||||
|
"query": query,
|
||||||
|
"hitsPerPage": max_results,
|
||||||
|
"tags": "story",
|
||||||
|
}
|
||||||
|
response = await self._client.get(self.SEARCH_URL, params=params)
|
||||||
|
response.raise_for_status()
|
||||||
|
data = response.json()
|
||||||
|
return self._parse_results(data, max_results)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Hacker News search failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse_results(self, data: dict, max_results: int) -> list[SearchResult]:
|
||||||
|
results = []
|
||||||
|
hits = data.get("hits", [])[:max_results]
|
||||||
|
for i, hit in enumerate(hits):
|
||||||
|
title = hit.get("title", "")
|
||||||
|
url = hit.get("url") or f"https://news.ycombinator.com/item?id={hit.get('objectID', '')}"
|
||||||
|
snippet = hit.get("story_text", "") or ""
|
||||||
|
# Clean HTML from snippet
|
||||||
|
snippet = re.sub(r"<[^>]+>", "", snippet)[:200]
|
||||||
|
points = hit.get("points", 0)
|
||||||
|
author = hit.get("author", "")
|
||||||
|
created_at = hit.get("created_at", "")
|
||||||
|
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title,
|
||||||
|
url=url,
|
||||||
|
snippet=snippet or f"[{points} points] by {author}",
|
||||||
|
source=self.name,
|
||||||
|
rank=i + 1,
|
||||||
|
category="news",
|
||||||
|
extra={
|
||||||
|
"points": points,
|
||||||
|
"author": author,
|
||||||
|
"time": created_at,
|
||||||
|
"heat": f"{points} points",
|
||||||
|
},
|
||||||
|
))
|
||||||
|
return results
|
||||||
|
|
||||||
|
async def close(self):
|
||||||
|
await self._client.aclose()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# GitHub Trending
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class GitHubTrendingEngine(SearchEngine):
|
||||||
|
"""GitHub Trending repositories."""
|
||||||
|
|
||||||
|
TRENDING_URL = "https://github.com/trending"
|
||||||
|
|
||||||
|
def __init__(self, config: Optional[dict] = None):
|
||||||
|
super().__init__("GitHub Trending", config)
|
||||||
|
self._client = httpx.AsyncClient(
|
||||||
|
timeout=_DEFAULT_TIMEOUT,
|
||||||
|
headers=_COMMON_HEADERS,
|
||||||
|
follow_redirects=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def search(self, query: str = "", max_results: int = 10, **kwargs) -> list[SearchResult]:
|
||||||
|
"""GitHub Trending doesn't support keyword search; returns top trending repos."""
|
||||||
|
url = self.TRENDING_URL
|
||||||
|
if query:
|
||||||
|
url += f"?spoken_language_code=&q={quote_plus(query)}"
|
||||||
|
try:
|
||||||
|
response = await self._client.get(url)
|
||||||
|
response.raise_for_status()
|
||||||
|
return await run_in_thread(self._parse_html, response.text, max_results)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"GitHub Trending failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse_html(self, html: str, max_results: int) -> list[SearchResult]:
|
||||||
|
soup = BeautifulSoup(html, "lxml")
|
||||||
|
results = []
|
||||||
|
articles = soup.select("article.Box-row")[:max_results]
|
||||||
|
for i, article in enumerate(articles):
|
||||||
|
h2 = article.select_one("h2")
|
||||||
|
if not h2:
|
||||||
|
continue
|
||||||
|
repo_path = h2.get_text(strip=True).replace(" ", "")
|
||||||
|
repo_url = f"https://github.com/{repo_path}"
|
||||||
|
|
||||||
|
desc_el = article.select_one("p")
|
||||||
|
description = desc_el.get_text(strip=True) if desc_el else ""
|
||||||
|
|
||||||
|
# Language
|
||||||
|
lang_el = article.select_one("[itemprop='programmingLanguage']")
|
||||||
|
language = lang_el.get_text(strip=True) if lang_el else ""
|
||||||
|
|
||||||
|
# Stars today
|
||||||
|
stars_el = article.select_one(".float-sm-right")
|
||||||
|
stars_today = stars_el.get_text(strip=True) if stars_el else ""
|
||||||
|
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=repo_path,
|
||||||
|
url=repo_url,
|
||||||
|
snippet=description[:200] if description else language,
|
||||||
|
source=self.name,
|
||||||
|
rank=i + 1,
|
||||||
|
category="news",
|
||||||
|
extra={
|
||||||
|
"language": language,
|
||||||
|
"heat": stars_today,
|
||||||
|
"time": "",
|
||||||
|
},
|
||||||
|
))
|
||||||
|
return results
|
||||||
|
|
||||||
|
async def close(self):
|
||||||
|
await self._client.aclose()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 36氪 (36kr.com)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class Kr36Engine(SearchEngine):
|
||||||
|
"""36氪 tech news search."""
|
||||||
|
|
||||||
|
SEARCH_URL = "https://36kr.com/search/articles/{keyword}"
|
||||||
|
|
||||||
|
def __init__(self, config: Optional[dict] = None):
|
||||||
|
super().__init__("36氪", config)
|
||||||
|
self._client = httpx.AsyncClient(
|
||||||
|
timeout=_DEFAULT_TIMEOUT,
|
||||||
|
headers={**_COMMON_HEADERS, "Referer": "https://36kr.com/"},
|
||||||
|
follow_redirects=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10, **kwargs) -> list[SearchResult]:
|
||||||
|
url = self.SEARCH_URL.format(keyword=quote_plus(query))
|
||||||
|
try:
|
||||||
|
response = await self._client.get(url)
|
||||||
|
response.raise_for_status()
|
||||||
|
return await run_in_thread(self._parse_html, response.text, query, max_results)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"36氪 search failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse_html(self, html: str, query: str, max_results: int) -> list[SearchResult]:
|
||||||
|
soup = BeautifulSoup(html, "lxml")
|
||||||
|
results = []
|
||||||
|
|
||||||
|
# Look for article cards
|
||||||
|
for i, item in enumerate(soup.select("a[href*='/p/'], .article-item, .search-result-item, [class*='article']")[:max_results]):
|
||||||
|
title_el = item.select_one("h3, h4, .title, [class*='title']")
|
||||||
|
link = item.get("href", "")
|
||||||
|
snippet_el = item.select_one(".abstract, .summary, .desc, p, [class*='desc']")
|
||||||
|
|
||||||
|
title = title_el.get_text(strip=True) if title_el else ""
|
||||||
|
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
||||||
|
|
||||||
|
if not title and not link:
|
||||||
|
continue
|
||||||
|
|
||||||
|
full_url = link if link.startswith("http") else f"https://36kr.com{link}"
|
||||||
|
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title or f"36氪: {query}",
|
||||||
|
url=full_url,
|
||||||
|
snippet=snippet[:200] if snippet else "",
|
||||||
|
source=self.name,
|
||||||
|
rank=i + 1,
|
||||||
|
category="news",
|
||||||
|
extra={"region": "cn"},
|
||||||
|
))
|
||||||
|
|
||||||
|
if not results:
|
||||||
|
search_url = f"https://36kr.com/search/articles/{quote_plus(query)}"
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=f"36氪: {query}",
|
||||||
|
url=search_url,
|
||||||
|
snippet="Search 36氪 (site may require JS)",
|
||||||
|
source=self.name,
|
||||||
|
rank=1,
|
||||||
|
category="news",
|
||||||
|
extra={"search_url": search_url, "note": "JS-rendered page"},
|
||||||
|
))
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
async def close(self):
|
||||||
|
await self._client.aclose()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 微博热搜 (Weibo Trending)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class WeiboEngine(SearchEngine):
|
||||||
|
"""微博热搜 trending topics."""
|
||||||
|
|
||||||
|
HOT_URL = "https://weibo.com/ajax/side/hotSearch"
|
||||||
|
|
||||||
|
def __init__(self, config: Optional[dict] = None):
|
||||||
|
super().__init__("微博热搜", config)
|
||||||
|
self._client = httpx.AsyncClient(
|
||||||
|
timeout=_DEFAULT_TIMEOUT,
|
||||||
|
headers={**_COMMON_HEADERS, "Referer": "https://weibo.com/"},
|
||||||
|
follow_redirects=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def search(self, query: str = "", max_results: int = 10, **kwargs) -> list[SearchResult]:
|
||||||
|
try:
|
||||||
|
response = await self._client.get(self.HOT_URL)
|
||||||
|
response.raise_for_status()
|
||||||
|
data = response.json()
|
||||||
|
return self._parse_results(data, query, max_results)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"微博热搜 failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse_results(self, data: dict, query: str, max_results: int) -> list[SearchResult]:
|
||||||
|
results = []
|
||||||
|
realtime = data.get("data", {}).get("realtime", [])[:max_results]
|
||||||
|
for i, item in enumerate(realtime):
|
||||||
|
word = item.get("word", "")
|
||||||
|
raw_hot = item.get("raw_hot", 0)
|
||||||
|
url = f"https://s.weibo.com/weibo?q={quote_plus(word)}"
|
||||||
|
|
||||||
|
# Filter by keyword if query provided
|
||||||
|
if query and query.lower() not in word.lower():
|
||||||
|
continue
|
||||||
|
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=f"#{word}#",
|
||||||
|
url=url,
|
||||||
|
snippet=f"热度: {raw_hot:,}" if raw_hot else "",
|
||||||
|
source=self.name,
|
||||||
|
rank=i + 1,
|
||||||
|
category="news",
|
||||||
|
extra={
|
||||||
|
"heat": f"{raw_hot:,}" if raw_hot else "",
|
||||||
|
"region": "cn",
|
||||||
|
},
|
||||||
|
))
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
async def close(self):
|
||||||
|
await self._client.aclose()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# V2EX (via public API)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class V2EXEngine(SearchEngine):
|
||||||
|
"""V2EX tech community hot topics."""
|
||||||
|
|
||||||
|
HOT_API = "https://www.v2ex.com/api/v2/topics/hot"
|
||||||
|
# Fallback to older API if v2 needs auth
|
||||||
|
HOT_API_FALLBACK = "https://www.v2ex.com/api/topics/hot.json"
|
||||||
|
|
||||||
|
def __init__(self, config: Optional[dict] = None):
|
||||||
|
super().__init__("V2EX", config)
|
||||||
|
self._client = httpx.AsyncClient(
|
||||||
|
timeout=_DEFAULT_TIMEOUT,
|
||||||
|
headers=_COMMON_HEADERS,
|
||||||
|
follow_redirects=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def search(self, query: str = "", max_results: int = 10, **kwargs) -> list[SearchResult]:
|
||||||
|
# Try v2 API first, fallback to old API
|
||||||
|
for api_url in [self.HOT_API, self.HOT_API_FALLBACK]:
|
||||||
|
try:
|
||||||
|
response = await self._client.get(api_url)
|
||||||
|
if response.status_code == 200:
|
||||||
|
data = response.json()
|
||||||
|
# v2 API returns list directly; old API also returns list
|
||||||
|
topics = data if isinstance(data, list) else data.get("data", [])
|
||||||
|
return self._parse_results(topics, query, max_results)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
logger.error("V2EX API failed (both v2 and fallback)")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse_results(self, topics: list, query: str, max_results: int) -> list[SearchResult]:
|
||||||
|
results = []
|
||||||
|
for i, topic in enumerate(topics[:max_results]):
|
||||||
|
title = topic.get("title", "")
|
||||||
|
node_name = ""
|
||||||
|
if "node" in topic and isinstance(topic["node"], dict):
|
||||||
|
node_name = topic["node"].get("title", "")
|
||||||
|
topic_id = topic.get("id", "")
|
||||||
|
url = f"https://www.v2ex.com/t/{topic_id}"
|
||||||
|
replies = topic.get("replies", 0)
|
||||||
|
created = topic.get("created", "")
|
||||||
|
member = ""
|
||||||
|
if "member" in topic and isinstance(topic["member"], dict):
|
||||||
|
member = topic["member"].get("username", "")
|
||||||
|
|
||||||
|
# Filter by keyword
|
||||||
|
if query and query.lower() not in title.lower():
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Format time
|
||||||
|
time_str = ""
|
||||||
|
if created:
|
||||||
|
try:
|
||||||
|
dt = datetime.fromtimestamp(int(created), tz=timezone.utc)
|
||||||
|
time_str = dt.strftime("%Y-%m-%d %H:%M")
|
||||||
|
except (ValueError, OSError):
|
||||||
|
time_str = str(created)
|
||||||
|
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title,
|
||||||
|
url=url,
|
||||||
|
snippet=f"[{node_name}] by {member}" if node_name and member else f"by {member}" if member else "",
|
||||||
|
source=self.name,
|
||||||
|
rank=i + 1,
|
||||||
|
category="news",
|
||||||
|
extra={
|
||||||
|
"heat": f"{replies} replies",
|
||||||
|
"author": member,
|
||||||
|
"time": time_str,
|
||||||
|
"region": "cn",
|
||||||
|
},
|
||||||
|
))
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
async def close(self):
|
||||||
|
await self._client.aclose()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 腾讯新闻 (news.qq.com)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
class TencentNewsEngine(SearchEngine):
|
||||||
|
"""腾讯新闻 search."""
|
||||||
|
|
||||||
|
SEARCH_URL = "https://news.qq.com/search"
|
||||||
|
|
||||||
|
def __init__(self, config: Optional[dict] = None):
|
||||||
|
super().__init__("腾讯新闻", config)
|
||||||
|
self._client = httpx.AsyncClient(
|
||||||
|
timeout=_DEFAULT_TIMEOUT,
|
||||||
|
headers={**_COMMON_HEADERS, "Referer": "https://news.qq.com/"},
|
||||||
|
follow_redirects=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10, **kwargs) -> list[SearchResult]:
|
||||||
|
url = f"{self.SEARCH_URL}?q={quote_plus(query)}"
|
||||||
|
try:
|
||||||
|
response = await self._client.get(url)
|
||||||
|
response.raise_for_status()
|
||||||
|
return await run_in_thread(self._parse_html, response.text, query, max_results)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"腾讯新闻 search failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse_html(self, html: str, query: str, max_results: int) -> list[SearchResult]:
|
||||||
|
soup = BeautifulSoup(html, "lxml")
|
||||||
|
results = []
|
||||||
|
|
||||||
|
for i, item in enumerate(soup.select("a[href*='news.qq.com'], .list-item, [class*='result'], article")[:max_results]):
|
||||||
|
title_el = item.select_one("h3, h4, .title, a")
|
||||||
|
link = item.get("href", "") or (title_el.get("href", "") if title_el else "")
|
||||||
|
snippet_el = item.select_one(".summary, .desc, .abstract, p, .text")
|
||||||
|
|
||||||
|
title = title_el.get_text(strip=True) if title_el else ""
|
||||||
|
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
||||||
|
|
||||||
|
if not title:
|
||||||
|
continue
|
||||||
|
|
||||||
|
full_url = link if link.startswith("http") else f"https:{link}" if link.startswith("//") else link
|
||||||
|
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title,
|
||||||
|
url=full_url,
|
||||||
|
snippet=snippet[:200] if snippet else "",
|
||||||
|
source=self.name,
|
||||||
|
rank=i + 1,
|
||||||
|
category="news",
|
||||||
|
extra={"region": "cn"},
|
||||||
|
))
|
||||||
|
|
||||||
|
if not results:
|
||||||
|
search_url = f"{self.SEARCH_URL}?q={quote_plus(query)}"
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=f"腾讯新闻: {query}",
|
||||||
|
url=search_url,
|
||||||
|
snippet=f"Search Tencent News for '{query}'",
|
||||||
|
source=self.name,
|
||||||
|
rank=1,
|
||||||
|
category="news",
|
||||||
|
extra={"search_url": search_url, "note": "JS-rendered page"},
|
||||||
|
))
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
async def close(self):
|
||||||
|
await self._client.aclose()
|
||||||
+40
-11
@@ -21,6 +21,10 @@ from scripts.engines.parser_engines import DuckDuckGoEngine
|
|||||||
from scripts.news.cls import CLSNewsEngine
|
from scripts.news.cls import CLSNewsEngine
|
||||||
from scripts.news.wallstreetcn import WallStreetCNEngine
|
from scripts.news.wallstreetcn import WallStreetCNEngine
|
||||||
from scripts.news.rss import RSSNewsEngine
|
from scripts.news.rss import RSSNewsEngine
|
||||||
|
from scripts.news.tech_news import (
|
||||||
|
HackerNewsEngine, GitHubTrendingEngine, Kr36Engine,
|
||||||
|
WeiboEngine, V2EXEngine, TencentNewsEngine,
|
||||||
|
)
|
||||||
from scripts.academic.arxiv import ArXivEngine
|
from scripts.academic.arxiv import ArXivEngine
|
||||||
from scripts.wechat.sogou_weixin import SogouWeChatEngine
|
from scripts.wechat.sogou_weixin import SogouWeChatEngine
|
||||||
from scripts.social.twitter import TwitterSearchEngine
|
from scripts.social.twitter import TwitterSearchEngine
|
||||||
@@ -132,31 +136,44 @@ class UnifiedSearch:
|
|||||||
return []
|
return []
|
||||||
|
|
||||||
async def search_health(self) -> list[dict[str, Any]]:
|
async def search_health(self) -> list[dict[str, Any]]:
|
||||||
"""Check health of all configured engines."""
|
"""Check health of all configured engines (concurrent)."""
|
||||||
statuses = []
|
seen_names: set[str] = set()
|
||||||
seen_names = set()
|
tasks: list[asyncio.Task] = []
|
||||||
for cat_key, cat_fn in SOURCE_CATEGORIES.items():
|
for cat_fn in SOURCE_CATEGORIES.values():
|
||||||
for engine_def in cat_fn():
|
for engine_def in cat_fn():
|
||||||
name = engine_def.get("name", "?")
|
name = engine_def.get("name", "?")
|
||||||
if name in seen_names:
|
if name in seen_names:
|
||||||
continue
|
continue
|
||||||
seen_names.add(name)
|
seen_names.add(name)
|
||||||
|
tasks.append(asyncio.create_task(self._check_one_engine(engine_def, name)))
|
||||||
|
|
||||||
|
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||||
|
statuses: list[dict[str, Any]] = []
|
||||||
|
for r in results:
|
||||||
|
if isinstance(r, Exception):
|
||||||
|
statuses.append({"name": "?", "status": "error", "detail": str(r)[:80]})
|
||||||
|
else:
|
||||||
|
statuses.append(r)
|
||||||
|
return statuses
|
||||||
|
|
||||||
|
async def _check_one_engine(
|
||||||
|
self, engine_def: dict[str, Any], name: str,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Check health of a single engine."""
|
||||||
engine = self._get_engine(engine_def)
|
engine = self._get_engine(engine_def)
|
||||||
if engine is None:
|
if engine is None:
|
||||||
statuses.append({"name": name, "status": "error", "detail": "engine_init_failed"})
|
return {"name": name, "status": "error", "detail": "engine_init_failed"}
|
||||||
continue
|
|
||||||
try:
|
try:
|
||||||
async with self._semaphore:
|
async with self._semaphore:
|
||||||
await asyncio.wait_for(
|
await asyncio.wait_for(
|
||||||
engine.search("health check", max_results=1),
|
engine.search("health check", max_results=1),
|
||||||
timeout=8,
|
timeout=8,
|
||||||
)
|
)
|
||||||
statuses.append({"name": name, "status": "ok"})
|
return {"name": name, "status": "ok"}
|
||||||
except asyncio.TimeoutError:
|
except asyncio.TimeoutError:
|
||||||
statuses.append({"name": name, "status": "timeout"})
|
return {"name": name, "status": "timeout"}
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
statuses.append({"name": name, "status": "error", "detail": str(e)[:80]})
|
return {"name": name, "status": "error", "detail": str(e)[:80]}
|
||||||
return statuses
|
|
||||||
|
|
||||||
def _get_engine(self, engine_def: dict[str, Any]):
|
def _get_engine(self, engine_def: dict[str, Any]):
|
||||||
"""Get or create an engine instance by definition."""
|
"""Get or create an engine instance by definition."""
|
||||||
@@ -180,7 +197,19 @@ class UnifiedSearch:
|
|||||||
engine = SogouWeChatEngine(self.config)
|
engine = SogouWeChatEngine(self.config)
|
||||||
elif "twitter" in name.lower():
|
elif "twitter" in name.lower():
|
||||||
engine = TwitterSearchEngine(self.config)
|
engine = TwitterSearchEngine(self.config)
|
||||||
elif engine_type == "news" and name not in ("cls", "wallstreetcn"):
|
elif name == "Hacker News":
|
||||||
|
engine = HackerNewsEngine(self.config)
|
||||||
|
elif name == "GitHub Trending":
|
||||||
|
engine = GitHubTrendingEngine(self.config)
|
||||||
|
elif name == "36氪":
|
||||||
|
engine = Kr36Engine(self.config)
|
||||||
|
elif name == "微博热搜":
|
||||||
|
engine = WeiboEngine(self.config)
|
||||||
|
elif name == "V2EX":
|
||||||
|
engine = V2EXEngine(self.config)
|
||||||
|
elif name == "腾讯新闻":
|
||||||
|
engine = TencentNewsEngine(self.config)
|
||||||
|
elif engine_type == "news" and name not in ("cls", "wallstreetcn", "Hacker News", "GitHub Trending", "36氪", "微博热搜", "V2EX", "腾讯新闻"):
|
||||||
engine = RSSNewsEngine(self.config)
|
engine = RSSNewsEngine(self.config)
|
||||||
elif engine_type == "social":
|
elif engine_type == "social":
|
||||||
engine = TwitterSearchEngine(self.config) if "twitter" in name.lower() else UrlSearchEngine(name, engine_def)
|
engine = TwitterSearchEngine(self.config) if "twitter" in name.lower() else UrlSearchEngine(name, engine_def)
|
||||||
|
|||||||
@@ -1,144 +1,186 @@
|
|||||||
---
|
---
|
||||||
name: web-search-skills
|
name: web-search-skills
|
||||||
description: "多源搜索 CLI 工具,支持 20+ 搜索引擎。搜索微信公众号文章、新闻资讯、学术论文、社交媒体的综合搜索技能。当用户需要搜索网络信息、查找微信公众号文章、搜索财经新闻、查询学术论文时使用。"
|
description: "Multi-source web search CLI & library, covering web, news, WeChat, academic, social, and RSS feeds across 28 engines. Use when the user asks to search the web, find WeChat articles, look up news, search academic papers, or check social media."
|
||||||
version: 2.1.0
|
version: 2.2.0
|
||||||
license: MIT
|
license: MIT
|
||||||
compatibility: Requires Python 3.8+, httpx, beautifulsoup4, lxml
|
compatibility: Requires Python 3.8+, httpx, beautifulsoup4, lxml
|
||||||
metadata:
|
metadata:
|
||||||
author: openbot-coder
|
author: openbot-coder
|
||||||
category: development
|
category: development
|
||||||
tags: [search, web, wechat, news, academic, cli]
|
tags: [search, web, wechat, news, academic, social, rss, cli, concurrent]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Web Search Skills
|
# Web Search Skills
|
||||||
|
|
||||||
## 概述
|
## Overview
|
||||||
|
|
||||||
Multi-source web search CLI and library supporting 20+ search engines, including web search, news (财联社, 华尔街见闻), WeChat articles, academic papers (ArXiv), social media (Twitter/X), and RSS feeds.
|
A CLI (`web-search`) and Python library that queries **28 search engines** concurrently via `asyncio.gather`. Supports web search, news (Hacker News, GitHub Trending, 36氪, 微博热搜, V2EX, RSS feeds), WeChat articles, ArXiv papers, Twitter/X, and Baidu Qianfan AI search.
|
||||||
|
|
||||||
## 触发词 Triggers
|
All engines run in parallel with `Semaphore(5)` concurrency control and 15s per-engine timeout.
|
||||||
|
|
||||||
中文触发词:
|
## Triggers
|
||||||
- "搜索微信公众号..."
|
|
||||||
- "搜一下微信公众号..."
|
|
||||||
- "查一下新闻..."
|
|
||||||
- "搜索新闻关于..."
|
|
||||||
- "搜论文..."
|
|
||||||
- "查学术资料..."
|
|
||||||
- "搜索一下..."
|
|
||||||
|
|
||||||
English Triggers:
|
When the user says any of these, use this skill:
|
||||||
- "search the web for..."
|
|
||||||
- "find information about..."
|
|
||||||
- "look up..."
|
|
||||||
- "search WeChat for..."
|
|
||||||
- "search news about..."
|
|
||||||
- Any query requiring real-time web data
|
|
||||||
|
|
||||||
## 安装 Installation
|
| Intent | Example user phrases |
|
||||||
|
|--------|---------------------|
|
||||||
|
| Web search | "搜索一下 xxx", "search the web for xxx" |
|
||||||
|
| WeChat articles | "搜微信公众号 xxx", "search WeChat for xxx" |
|
||||||
|
| News | "查一下新闻", "搜索最新的科技新闻", "search news about xxx" |
|
||||||
|
| Academic | "搜论文 xxx", "search papers about xxx", "查学术资料" |
|
||||||
|
| Social media | "看看 Twitter 上关于", "check Twitter for xxx" |
|
||||||
|
| Specific source | "看看 Hacker News 上关于", "what's trending on GitHub" |
|
||||||
|
| Hot topics | "查一下微博热搜", "check Weibo trending" |
|
||||||
|
|
||||||
|
## Agent Decision Tree
|
||||||
|
|
||||||
|
Use this flow to pick the right command:
|
||||||
|
|
||||||
|
```
|
||||||
|
用户说了什么?
|
||||||
|
│
|
||||||
|
├─ "微信公众号" / "WeChat" / 公众号名称
|
||||||
|
│ → web-search wechat <query>
|
||||||
|
│
|
||||||
|
├─ "学术" / "论文" / "paper" / "ArXiv"
|
||||||
|
│ → web-search academic <query>
|
||||||
|
│
|
||||||
|
├─ "Twitter" / "X" / "社交" / "social"
|
||||||
|
│ → web-search social <query> [-n 5]
|
||||||
|
│
|
||||||
|
├─ "新闻" / "news" / 明确指定 RSS/博文/播客
|
||||||
|
│ → web-search news <query> [-n 5]
|
||||||
|
│
|
||||||
|
├─ 明确指定了特定源名称
|
||||||
|
│ → web-search search <query> -s "源名1" "源名2" [-n 5]
|
||||||
|
│
|
||||||
|
├─ 需要多类源对比(如"新闻+社交上关于伊朗的看法")
|
||||||
|
│ → web-search search <query> -s news social [-n 10]
|
||||||
|
│
|
||||||
|
├─ 需要百度搜索 + 站点/时间过滤
|
||||||
|
│ → web-search baidu <query> [--site xxx] [--recency week]
|
||||||
|
│
|
||||||
|
├─ "健康检查" / "health" / "测试引擎"
|
||||||
|
│ → web-search health
|
||||||
|
│
|
||||||
|
├─ "列出源" / "sources" / "有哪些搜索引擎"
|
||||||
|
│ → web-search sources
|
||||||
|
│
|
||||||
|
├─ 其余通用查询
|
||||||
|
│ → web-search search <query> [-s 源1 源2...] [-n 10]
|
||||||
|
│
|
||||||
|
└─ 不确定 / 没特别要求
|
||||||
|
→ web-search search <query> -n 10
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick Command Reference
|
||||||
|
|
||||||
|
| Command | Speed | Best for |
|
||||||
|
|---------|-------|----------|
|
||||||
|
| `search` | 并发 28 源 | 通用搜索、多源对比 |
|
||||||
|
| `web` | 14 网页引擎 | 普通网页搜索 |
|
||||||
|
| `baidu` | 单引擎 | 需 site/recency 过滤 |
|
||||||
|
| `news` | ~10+ 新闻源 | 科技/财经/博客/播客 |
|
||||||
|
| `wechat` | 单引擎 | 微信公众号独家内容 |
|
||||||
|
| `academic` | 单引擎 | 学术论文 |
|
||||||
|
| `social` | 2 源 (Twitter) | 社交媒体动态 |
|
||||||
|
| `rss` | 18 订阅源 | AI 周报/深度文章/播客 |
|
||||||
|
| `health` | 28 源并发 (~14s) | 检测连通性 |
|
||||||
|
|
||||||
|
**通用选项:** `-n N`(每源结果数), `-j`(JSON), `-o FILE`(存文件), `-s NAME...`(指定源), `-v`(调试日志)
|
||||||
|
|
||||||
|
## Result Field Reference
|
||||||
|
|
||||||
|
Each `SearchResult` returned has these fields:
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `rank` | int | Position in results (1-based) |
|
||||||
|
| `title` | str | Article/title text |
|
||||||
|
| `url` | str | Direct URL to source |
|
||||||
|
| `snippet` | str | Summary text (first ~300 chars) |
|
||||||
|
| `source` | str | Engine name (e.g. "Hacker News") |
|
||||||
|
| `category` | str | "web", "news", "academic", "social", etc. |
|
||||||
|
| `extra` | dict | Engine-specific metadata (see below) |
|
||||||
|
|
||||||
|
### `extra` fields by source
|
||||||
|
|
||||||
|
| Source | extra fields |
|
||||||
|
|--------|-------------|
|
||||||
|
| Hacker News | `points`, `author`, `time` (ISO date) |
|
||||||
|
| GitHub Trending | `language`, `stars today` (heat) |
|
||||||
|
| WeChat | `account` (公众号名), `time` |
|
||||||
|
| Twitter | `time`, `account`, `search_type` |
|
||||||
|
| RSS feeds | `time`, `feed_url` |
|
||||||
|
| Cls / WallStreetCN | `search_url` (JS-rendered fallback) |
|
||||||
|
| Baidu Qianfan | `site`, `recency` (if filtered) |
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
| Symptom | Likely cause | Action |
|
||||||
|
|---------|-------------|--------|
|
||||||
|
| Empty results from Hacker News/GitHub | Network issue or no matches | Retry with different query |
|
||||||
|
| Baidu Qianfan returns nothing | Missing `BAIDU_API_KEY` env | Check env var is set |
|
||||||
|
| DuckDuckGo times out (15s) | Anti-scraping / rate limit | Note to user, skip |
|
||||||
|
| WeChat empty results | Sogou anti-scraping | Inform user it may be limited |
|
||||||
|
| `web-search health` shows [TIMEOUT] | Engine unreachable from this region | Not critical, skip that engine |
|
||||||
|
| GBK encode error on Windows | Non-ASCII char in output | Already auto-fixed by `_sanitize_for_console` |
|
||||||
|
|
||||||
|
**General rule:** If a specific source fails, note it but don't fail the whole search — other sources continue concurrently.
|
||||||
|
|
||||||
|
## Usage Examples
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd web-search-skills
|
# Full search (all 28 sources, concurrent)
|
||||||
uv tool install -e .
|
web-search search "quantum computing" -n 10
|
||||||
|
|
||||||
|
# Specific sources only
|
||||||
|
web-search search "technology" -s "Hacker News" "V2EX" "GitHub Trending" -n 3
|
||||||
|
|
||||||
|
# Multiple categories
|
||||||
|
web-search search "Iran nuclear" -s web news social -n 5
|
||||||
|
|
||||||
|
# Baidu with site filter
|
||||||
|
web-search baidu "中美元首会晤" --site mp.weixin.qq.com --recency week
|
||||||
|
|
||||||
|
# All web engines
|
||||||
|
web-search web "Python async" -r global -j
|
||||||
|
|
||||||
|
# Quick searches
|
||||||
|
web-search news "人工智能" -n 5
|
||||||
|
web-search wechat 蒙面财经
|
||||||
|
web-search academic "machine learning"
|
||||||
|
web-search social "AI regulation" -n 5
|
||||||
|
|
||||||
|
# Concurrent health check (~14s for 28 engines)
|
||||||
|
web-search health
|
||||||
```
|
```
|
||||||
|
|
||||||
安装后 `web-search` 命令全局可用。
|
## Python API (for programmatic use)
|
||||||
|
|
||||||
## 命令 Commands
|
|
||||||
|
|
||||||
| 命令 Commands | 说明 Description |
|
|
||||||
|---------------|------------------|
|
|
||||||
| `web-search search <query>` | 全源搜索 Search all sources |
|
|
||||||
| `web-search web <query>` | 网页搜索 Web search engines |
|
|
||||||
| `web-search news <query>` | 新闻搜索 News (财联社, 华尔街见闻) |
|
|
||||||
| `web-search wechat <query>` | 微信公众号搜索 WeChat articles |
|
|
||||||
| `web-search academic <query>` | 学术搜索 Academic papers (ArXiv) |
|
|
||||||
| `web-search social <query>` | 社交搜索 Social media (Twitter/X) |
|
|
||||||
| `web-search rss <query>` | RSS 源搜索 RSS feeds |
|
|
||||||
| `web-search sources` | 列出所有搜索源 List all sources |
|
|
||||||
| `web-search urls <query>` | 只生成搜索链接 Search URLs only |
|
|
||||||
|
|
||||||
### 通用选项 Options
|
|
||||||
|
|
||||||
| 选项 | 说明 |
|
|
||||||
|------|------|
|
|
||||||
| `-n N` | 每源最大结果数 (默认: 10) |
|
|
||||||
| `-j` | JSON 格式输出 |
|
|
||||||
| `-o FILE` | 保存结果到 JSON 文件 |
|
|
||||||
| `-s NAME` | 指定来源名称 |
|
|
||||||
| `-r cn/global` | 区域过滤 (web 搜索) |
|
|
||||||
| `-v` | 调试日志 |
|
|
||||||
|
|
||||||
## 项目结构 Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
web-search-skills/
|
|
||||||
├── scripts/ # Python 脚本目录
|
|
||||||
│ ├── cli.py # CLI 入口
|
|
||||||
│ ├── unified_search.py # 统一搜索编排器
|
|
||||||
│ ├── core/
|
|
||||||
│ │ ├── base.py # SearchResult & SearchEngine 基类
|
|
||||||
│ │ └── config_loader.py# 从 JSON 加载搜索引擎配置
|
|
||||||
│ ├── engines/
|
|
||||||
│ │ ├── url_engine.py # URL 模板搜索引擎
|
|
||||||
│ │ └── parser_engines.py# HTML 解析搜索 (DuckDuckGo)
|
|
||||||
│ ├── news/
|
|
||||||
│ │ ├── cls.py # 财联社新闻引擎
|
|
||||||
│ │ ├── wallstreetcn.py # 华尔街见闻引擎
|
|
||||||
│ │ └── rss.py # RSS 聚合引擎
|
|
||||||
│ ├── academic/
|
|
||||||
│ │ └── arxiv.py # ArXiv 学术论文引擎
|
|
||||||
│ ├── wechat/
|
|
||||||
│ │ └── sogou_weixin.py # 搜狗微信搜索引擎
|
|
||||||
│ └── social/
|
|
||||||
│ └── twitter.py # Twitter/X 搜索引擎
|
|
||||||
├── config/
|
|
||||||
│ └── engines.json # 搜索引擎定义配置
|
|
||||||
├── ws.py # 本地运行入口
|
|
||||||
├── pyproject.toml # 包构建配置
|
|
||||||
├── SKILL.md # 本技能定义文件
|
|
||||||
├── LICENSE # MIT 许可证
|
|
||||||
└── README.md # 项目文档
|
|
||||||
```
|
|
||||||
|
|
||||||
## 配置 Configuration
|
|
||||||
|
|
||||||
所有搜索引擎定义在 `config/engines.json` 中,支持 6 类源:
|
|
||||||
|
|
||||||
| 类别 | 数量 |
|
|
||||||
|------|------|
|
|
||||||
| Web 搜索引擎 | 14 个 (Baidu, Google, DuckDuckGo 等) |
|
|
||||||
| 新闻源 | 2 个 (财联社, 华尔街见闻) |
|
|
||||||
| 学术源 | 1 个 (ArXiv) |
|
|
||||||
| 微信源 | 1 个 (搜狗微信) |
|
|
||||||
| 社交源 | 2 个 (Twitter, Twitter Latest) |
|
|
||||||
| 特殊源 | 1 个 (WolframAlpha) |
|
|
||||||
|
|
||||||
每个引擎定义格式:
|
|
||||||
```json
|
|
||||||
{"name": "Baidu", "url": "https://www.baidu.com/s?wd={keyword}", "region": "cn", "type": "url"}
|
|
||||||
```
|
|
||||||
|
|
||||||
- `{keyword}` 会被替换为搜索关键词
|
|
||||||
- `type: url` 表示只生成 URL,`type: parser` 表示会解析 HTML 提取结果
|
|
||||||
|
|
||||||
## Python API
|
|
||||||
|
|
||||||
也可在代码中直接调用:
|
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from scripts.unified_search import UnifiedSearch
|
from scripts.unified_search import UnifiedSearch
|
||||||
|
|
||||||
async def main():
|
async def main():
|
||||||
us = UnifiedSearch()
|
us = UnifiedSearch()
|
||||||
results = await us.search("量子计算", sources=["web", "news"], max_results=5)
|
results = await us.search(
|
||||||
|
"quantum computing",
|
||||||
|
sources=["Hacker News", "ArXiv"],
|
||||||
|
max_results=5,
|
||||||
|
)
|
||||||
for r in results:
|
for r in results:
|
||||||
print(f"[{r.rank}] {r.title}: {r.url}")
|
print(f"[{r.rank}] {r.title} ({r.source})")
|
||||||
|
if r.extra:
|
||||||
|
print(f" extra: {r.extra}")
|
||||||
await us.close()
|
await us.close()
|
||||||
```
|
```
|
||||||
|
|
||||||
## 注意事项 Notes
|
## Important Notes
|
||||||
|
|
||||||
- 搜狗微信搜索有反爬限制,可能返回空结果
|
- **WeChat** via Sogou has anti-scraping limits, may return empty
|
||||||
- 财联社和华尔街见闻需要 JavaScript 渲染,`UnifiedSearch` 返回搜索页面链接
|
- **财联社 / 华尔街见闻** need JS rendering → returns search page URL, not parsed results
|
||||||
- RSS 源需要稳定的网络连接
|
- **Baidu Qianfan** requires `BAIDU_API_KEY` env var
|
||||||
|
- **RSS feeds** need stable internet
|
||||||
|
- **Windows GBK** terminals: special chars auto-replaced by `_sanitize_for_console()`
|
||||||
|
|
||||||
|
> Full user-facing documentation (detailed command reference, project structure, config guide, architecture) is in `README.md`.
|
||||||
Reference in new issue
Block a user