Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
66557e906a | ||
|
|
c077a7ae40 | ||
|
|
e20478fcbd | ||
|
|
a60ba71663 |
No files matched your search
@@ -35,3 +35,4 @@ build/
|
|||||||
|
|
||||||
# Config with secrets
|
# Config with secrets
|
||||||
config.local.yaml
|
config.local.yaml
|
||||||
|
logs/
|
||||||
@@ -1,133 +1,51 @@
|
|||||||
# Web Search Skills
|
# web-search-skills
|
||||||
|
|
||||||
Multi-source web search CLI tool — search across **28 engines** from your terminal: web search, news, WeChat articles, academic papers, social media, RSS feeds, and AI-powered Baidu search.
|
> Multi-source web search CLI & Python library — 15+ engines, <5s, no API key required.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **15+ engines** across 5 categories: Web, News, Academic, Social, RSS
|
||||||
|
- **Concurrent execution** via `asyncio.gather` with semaphore (max 5 parallel)
|
||||||
|
- **Free by default** — most engines work without any API key
|
||||||
|
- **Python API** for programmatic use (`UnifiedSearch`)
|
||||||
|
- **Single model** — `SearchResult` dataclass across all engines
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Option 1: pip install (recommended for CodeBuddy Agent use)
|
# Install
|
||||||
cd web-search-skills
|
|
||||||
pip install -e .
|
pip install -e .
|
||||||
|
|
||||||
# Option 2: uv tool install (alternative)
|
# CLI
|
||||||
uv tool install -e .
|
web-search search "AI news" -n 10
|
||||||
|
web-search news "quantum computing" -n 5
|
||||||
# Now use globally from anywhere
|
web-search academic "machine learning"
|
||||||
web-search wechat 蒙面财经
|
web-search wechat 蒙面财经
|
||||||
web-search news "AI" -n 5
|
|
||||||
web-search search "量子计算" -o results.json
|
|
||||||
```
|
|
||||||
|
|
||||||
Or run directly without install:
|
|
||||||
```bash
|
|
||||||
python ws.py wechat 蒙面财经
|
|
||||||
```
|
|
||||||
|
|
||||||
> Requires Python 3.8+. After `pip install -e .`, the `web-search` command is available globally.
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
```bash
|
|
||||||
web-search <command> <query> [options]
|
|
||||||
```
|
|
||||||
|
|
||||||
| Command | Description |
|
|
||||||
|------------|-----------------------------------------------------|
|
|
||||||
| `search` | Search 28 sources in parallel (default) |
|
|
||||||
| `web` | Web search engines (Google, Baidu, DDG, Yahoo…) |
|
|
||||||
| `news` | News (Hacker News, GitHub Trending, 36氪, RSS…) |
|
|
||||||
| `baidu` | AI-powered Baidu search (site/recency filters) |
|
|
||||||
| `wechat` | WeChat public account articles (via Sogou) |
|
|
||||||
| `academic` | Academic papers (ArXiv) |
|
|
||||||
| `social` | Social media (Twitter/X) |
|
|
||||||
| `rss` | RSS feeds (AI newsletters, essays, podcasts…) |
|
|
||||||
| `health` | Test connectivity of all 28 engines (concurrent) |
|
|
||||||
| `sources` | List all available search sources |
|
|
||||||
| `urls` | Generate search URLs without executing |
|
|
||||||
|
|
||||||
### Options
|
|
||||||
|
|
||||||
| Flag | Description |
|
|
||||||
|-------------------|----------------------------------------------|
|
|
||||||
| `-n N` | Max results per source (default: 10) |
|
|
||||||
| `-j` | JSON output |
|
|
||||||
| `-o FILE` | Save to JSON file (path restricted to current directory) |
|
|
||||||
| `-s NAME [...]` | Specific source(s) — engine names or categories (e.g. `-s "Hacker News" V2EX`, `-s web news`) |
|
|
||||||
| `-r cn/global` | Region filter (web search) |
|
|
||||||
| `--site SITE` | Restrict to site (baidu command only) |
|
|
||||||
| `--recency RECENCY` | Time filter: day/week/month/semiyear/year |
|
|
||||||
| `-v` | Verbose/debug logs |
|
|
||||||
|
|
||||||
### Examples
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Search specific engines concurrently (new!)
|
|
||||||
web-search search "technology" -s "Hacker News" "V2EX" "GitHub Trending" -n 3
|
|
||||||
|
|
||||||
# Search two categories concurrently (new!)
|
|
||||||
web-search search "AI" -s web news -n 3
|
|
||||||
|
|
||||||
# Baidu with site & recency filters
|
|
||||||
web-search baidu "中美元首会晤" --site mp.weixin.qq.com --recency week
|
|
||||||
|
|
||||||
# Quick search
|
|
||||||
web-search news "人工智能" -n 5
|
|
||||||
web-search wechat 蒙面财经
|
|
||||||
web-search academic "quantum computing"
|
|
||||||
web-search search "latest AI news" -o results.json
|
|
||||||
|
|
||||||
# Check engine health (concurrent, ~14s for 28 engines)
|
|
||||||
web-search health
|
web-search health
|
||||||
```
|
```
|
||||||
|
|
||||||
## Sources (28 engines)
|
## Commands
|
||||||
|
|
||||||
| Category | Sources | Count |
|
| Command | Description | Speed |
|
||||||
|----------|---------|-------|
|
|---------|------------|-------|
|
||||||
| **Web** | Baidu, Bing CN, Bing INT, 360, Sogou, Shenma, Google, Google HK, DuckDuckGo, Yahoo, Startpage, Brave, Ecosia, Qwant | 14 |
|
| `search` | All 15 engines in parallel | ~10s |
|
||||||
| **News (CN)** | 财联社, 华尔街见闻, 36氪, 微博热搜, V2EX, 腾讯新闻 | 6 |
|
| `web` | Web engines only (6) | ~5s |
|
||||||
| **News (Global)** | Hacker News, GitHub Trending | 2 |
|
| `news` | Bing News + 15 RSS feeds | ~8s |
|
||||||
| **RSS** | Reuters, AP, BBC, FT, CNN, Al Jazeera + AI newsletters (Latent Space, ChinAI, KDnuggets…), essays (Paul Graham, James Clear…), podcasts (Lex Fridman…) | 18 |
|
| `academic` | ArXiv + CrossRef in parallel | ~6s |
|
||||||
| **Academic** | ArXiv | 1 |
|
| `wechat` | WeChat via Sogou | ~5s |
|
||||||
| **WeChat** | Sogou WeChat | 1 |
|
| `social` | Twitter/X via Nitter | ~5s |
|
||||||
| **Social** | Twitter, Twitter Latest | 2 |
|
| `rss` | RSS feed aggregation | ~8s |
|
||||||
| **API** | Baidu Qianfan (AI search with site/recency) | 1 |
|
| `baidu` | Baidu Qianfan API (needs key) | varies |
|
||||||
| **Special** | WolframAlpha | 1 |
|
| `health` | Engine connectivity check | ~10s |
|
||||||
|
| `sources` | List all sources | instant |
|
||||||
|
|
||||||
## Project Structure
|
## Options
|
||||||
|
|
||||||
```
|
- `-n N` — max results per engine (default: 10)
|
||||||
web-search-skills/
|
- `-s SRC...` — run specific sources only
|
||||||
├── scripts/
|
- `-c CAT` — run by category (web/news/academic/social/rss)
|
||||||
│ ├── cli.py # CLI entry point
|
- `-r cn|global` — region (default: cn)
|
||||||
│ ├── unified_search.py # Unified search orchestrator (async + concurrent)
|
- `-j` — JSON output
|
||||||
│ ├── core/
|
|
||||||
│ │ ├── base.py # SearchResult & SearchEngine base class
|
|
||||||
│ │ ├── config_loader.py # Load engine config from JSON
|
|
||||||
│ │ └── thread_utils.py # Shared thread pool for HTML parsing
|
|
||||||
│ ├── engines/
|
|
||||||
│ │ ├── url_engine.py # URL-template search engines
|
|
||||||
│ │ ├── parser_engines.py # HTML-parsed search (DuckDuckGo)
|
|
||||||
│ │ └── baidu_qianfan.py # Baidu Qianfan AI Search API
|
|
||||||
│ ├── news/
|
|
||||||
│ │ ├── cls.py # 财联社
|
|
||||||
│ │ ├── wallstreetcn.py # 华尔街见闻
|
|
||||||
│ │ ├── rss.py # RSS feeds (18 sources)
|
|
||||||
│ │ └── tech_news.py # Tech news engines (Hacker News, GitHub Trending, 36氪, etc.)
|
|
||||||
│ ├── academic/
|
|
||||||
│ │ └── arxiv.py # ArXiv papers
|
|
||||||
│ ├── wechat/
|
|
||||||
│ │ └── sogou_weixin.py # Sogou WeChat search
|
|
||||||
│ └── social/
|
|
||||||
│ └── twitter.py # Twitter/X search
|
|
||||||
├── config/
|
|
||||||
│ └── engines.json # Search engine definitions
|
|
||||||
├── ws.py # Direct run entry point
|
|
||||||
├── pyproject.toml # Package config
|
|
||||||
├── SKILL.md # CodeBuddy skill definition
|
|
||||||
├── LICENSE
|
|
||||||
└── README.md
|
|
||||||
```
|
|
||||||
|
|
||||||
## Python API
|
## Python API
|
||||||
|
|
||||||
@@ -136,29 +54,84 @@ from scripts.unified_search import UnifiedSearch
|
|||||||
|
|
||||||
async def main():
|
async def main():
|
||||||
us = UnifiedSearch()
|
us = UnifiedSearch()
|
||||||
# Search specific sources concurrently
|
resp = await us.search("quantum computing", sources=["bing", "arxiv"], max_results=5)
|
||||||
results = await us.search("quantum computing",
|
for r in resp.results:
|
||||||
sources=["Hacker News", "ArXiv"],
|
print(f"[{r.rank}] {r.title}")
|
||||||
max_results=5)
|
print(f" {r.url}")
|
||||||
for r in results:
|
|
||||||
print(f"[{r.rank}] {r.title}: {r.url}")
|
|
||||||
await us.close()
|
await us.close()
|
||||||
```
|
```
|
||||||
|
|
||||||
## Performance
|
## Architecture
|
||||||
|
|
||||||
- **Concurrent engine execution**: All 28 engines run in parallel via `asyncio.gather`, with `Semaphore(5)` for gentle concurrency
|
```
|
||||||
- **Health check**: 28 engines checked in ~14 seconds (vs ~84s without concurrency)
|
scripts/
|
||||||
- **Thread pool**: HTML parsing offloaded to `ThreadPoolExecutor` via `scripts/core/thread_utils.py`, preventing event loop blocking
|
cli.py # CLI entry point
|
||||||
- **Timeout**: 15s per engine timeout prevents hanging
|
unified_search.py # Python API
|
||||||
|
models.py # Data models
|
||||||
|
config.py # Settings
|
||||||
|
core/
|
||||||
|
base.py # SearchResult + BaseEngine (canonical)
|
||||||
|
aggregator.py # Deduplication + ranking
|
||||||
|
execution_log.py # Timing + error tracking
|
||||||
|
engines/
|
||||||
|
bing_scraper.py # Bing web search
|
||||||
|
sogou_scraper.py # Sogou web search
|
||||||
|
baidu_scraper.py # Baidu web search
|
||||||
|
search360_scraper.py # 360 search
|
||||||
|
parser_engines.py # DuckDuckGo + Brave (HTML parser)
|
||||||
|
news_engine.py # Bing News + RSS aggregator
|
||||||
|
academic_engine.py # ArXiv + CrossRef
|
||||||
|
github_engine.py # GitHub API
|
||||||
|
baidu_qianfan.py # Baidu Qianfan API (needs key)
|
||||||
|
parallel_search.py # Parallel orchestrator
|
||||||
|
```
|
||||||
|
|
||||||
## Notes
|
## Engines
|
||||||
|
|
||||||
- Sogou WeChat has anti-scraping limits, may return empty results
|
### Web (无需 API key)
|
||||||
- 财联社 and 华尔街见闻 require JavaScript rendering; `UnifiedSearch` returns search page links for these
|
| Engine | URL | Notes |
|
||||||
- RSS feeds require stable network connectivity
|
|--------|-----|-------|
|
||||||
- Baidu Qianfan requires `BAIDU_API_KEY` environment variable
|
| Bing CN | cn.bing.com | Primary |
|
||||||
|
| Bing INT | www.bing.com | Global fallback |
|
||||||
|
| Sogou | sogou.com | Primary CN |
|
||||||
|
| Baidu | baidu.com | Primary CN |
|
||||||
|
| 360 | so.com | CN fallback |
|
||||||
|
| DuckDuckGo | html.duckduckgo.com | Global |
|
||||||
|
| Brave | search.brave.com | Global |
|
||||||
|
|
||||||
## License
|
### News
|
||||||
|
| Engine | Source |
|
||||||
|
|--------|--------|
|
||||||
|
| Bing News | bing.com/news |
|
||||||
|
| RSS (15 feeds) | 36kr, IT之家, 钛媒体, HackerNews, TechCrunch... |
|
||||||
|
|
||||||
MIT
|
### Academic (Free APIs)
|
||||||
|
| Engine | API |
|
||||||
|
|--------|-----|
|
||||||
|
| ArXiv | export.arxiv.org/api/query |
|
||||||
|
| CrossRef | api.crossref.org |
|
||||||
|
|
||||||
|
### Social
|
||||||
|
| Engine | Notes |
|
||||||
|
|--------|-------|
|
||||||
|
| Nitter | twitter.com 免登录 (多个实例自动切换) |
|
||||||
|
|
||||||
|
### API (需配置密钥)
|
||||||
|
| Engine | Env var |
|
||||||
|
|--------|---------|
|
||||||
|
| Baidu Qianfan | `BAIDU_API_KEY` |
|
||||||
|
|
||||||
|
## SearchResult Model
|
||||||
|
|
||||||
|
```python
|
||||||
|
@dataclass
|
||||||
|
class SearchResult:
|
||||||
|
title: str # Article/title text
|
||||||
|
url: str # Direct URL
|
||||||
|
snippet: str # Summary text (~300 chars)
|
||||||
|
source: str # Engine name (e.g. "bing", "arxiv")
|
||||||
|
category: str # "web" | "news" | "academic" | "github"
|
||||||
|
rank: int # Position after deduplication
|
||||||
|
published: str # ISO date (optional)
|
||||||
|
extra: dict # Engine-specific metadata
|
||||||
|
```
|
||||||
@@ -1,200 +1,140 @@
|
|||||||
---
|
---
|
||||||
name: web-search-skills
|
name: web-search
|
||||||
description: "Multi-source web search CLI & library, covering web, news, WeChat, academic, social, and RSS feeds across 28 engines. Use when the user asks to search the web, find WeChat articles, look up news, search academic papers, or check social media."
|
description: "Multi-source web search — 15+ engines (web, news, academic, social, RSS) running concurrently in <5s. No API key needed for most engines. Use when user asks to search the web, find news, look up papers, check social media, or search WeChat."
|
||||||
version: 2.2.0
|
version: 3.0.0
|
||||||
license: MIT
|
license: MIT
|
||||||
compatibility: Requires Python 3.8+, httpx, beautifulsoup4, lxml
|
compatibility: Python 3.10+, httpx, beautifulsoup4, lxml
|
||||||
metadata:
|
metadata:
|
||||||
author: openbot-coder
|
author: openbot-coder
|
||||||
category: development
|
category: search
|
||||||
tags: [search, web, wechat, news, academic, social, rss, cli, concurrent]
|
tags: [search, web, news, academic, social, rss, cli, async]
|
||||||
---
|
---
|
||||||
|
|
||||||
# Web Search Skills
|
# Web Search Skills
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
A CLI (`web-search`) and Python library that queries **28 search engines** concurrently via `asyncio.gather`. Supports web search, news (Hacker News, GitHub Trending, 36氪, 微博热搜, V2EX, RSS feeds), WeChat articles, ArXiv papers, Twitter/X, and Baidu Qianfan AI search.
|
|
||||||
|
|
||||||
All engines run in parallel with `Semaphore(5)` concurrency control and 15s per-engine timeout.
|
|
||||||
|
|
||||||
## Triggers
|
## Triggers
|
||||||
|
|
||||||
When the user says any of these, use this skill:
|
Use this skill when the user says:
|
||||||
|
|
||||||
| Intent | Example user phrases |
|
| Intent | Example |
|
||||||
|--------|---------------------|
|
|--------|---------|
|
||||||
| Web search | "搜索一下 xxx", "search the web for xxx" |
|
| Web search | "搜索一下 xxx", "search the web for xxx" |
|
||||||
| WeChat articles | "搜微信公众号 xxx", "search WeChat for xxx" |
|
| News | "查一下新闻", "搜索最新的科技新闻" |
|
||||||
| News | "查一下新闻", "搜索最新的科技新闻", "search news about xxx" |
|
| Academic | "搜论文 xxx", "查学术资料" |
|
||||||
| Academic | "搜论文 xxx", "search papers about xxx", "查学术资料" |
|
| WeChat | "搜微信公众号 xxx" |
|
||||||
| Social media | "看看 Twitter 上关于", "check Twitter for xxx" |
|
| Social | "看看 Twitter 上关于" |
|
||||||
| Specific source | "看看 Hacker News 上关于", "what's trending on GitHub" |
|
| Health check | "检查一下搜索引擎状态" |
|
||||||
| Hot topics | "查一下微博热搜", "check Weibo trending" |
|
| List sources | "有哪些搜索引擎可以用" |
|
||||||
|
|
||||||
## Agent Decision Tree
|
## Commands
|
||||||
|
|
||||||
Use this flow to pick the right command:
|
```bash
|
||||||
|
# Full search (all 15 engines in parallel, ~10s)
|
||||||
|
web-search search "AI news" -n 10
|
||||||
|
|
||||||
|
# Web only (6 engines, ~5s)
|
||||||
|
web-search web "Python async" -n 5
|
||||||
|
|
||||||
|
# News (Bing News + 15 RSS, ~8s)
|
||||||
|
web-search news "量子计算" -n 5
|
||||||
|
|
||||||
|
# Academic (ArXiv + CrossRef, ~6s)
|
||||||
|
web-search academic "machine learning"
|
||||||
|
|
||||||
|
# WeChat (Sogou, ~5s)
|
||||||
|
web-search wechat 蒙面财经
|
||||||
|
|
||||||
|
# Social (Nitter, ~5s)
|
||||||
|
web-search social "AI regulation" -n 10
|
||||||
|
|
||||||
|
# RSS feeds only
|
||||||
|
web-search rss "AI newsletter" -n 5
|
||||||
|
|
||||||
|
# Baidu Qianfan API (needs BAIDU_API_KEY)
|
||||||
|
web-search baidu "中美元首会晤"
|
||||||
|
|
||||||
|
# Engine health check
|
||||||
|
web-search health
|
||||||
|
|
||||||
|
# List all sources
|
||||||
|
web-search sources
|
||||||
|
```
|
||||||
|
|
||||||
|
## Decision Tree
|
||||||
|
|
||||||
```
|
```
|
||||||
用户说了什么?
|
用户说了什么?
|
||||||
│
|
│
|
||||||
├─ "微信公众号" / "WeChat" / 公众号名称
|
├─ "微信公众号" / "wechat"
|
||||||
│ → web-search wechat <query>
|
│ → web-search wechat <query>
|
||||||
│
|
│
|
||||||
├─ "学术" / "论文" / "paper" / "ArXiv"
|
├─ "学术" / "论文" / "paper" / "ArXiv"
|
||||||
│ → web-search academic <query>
|
│ → web-search academic <query>
|
||||||
│
|
│
|
||||||
├─ "Twitter" / "X" / "社交" / "social"
|
├─ "Twitter" / "X" / "社交"
|
||||||
│ → web-search social <query> [-n 5]
|
│ → web-search social <query>
|
||||||
│
|
│
|
||||||
├─ "新闻" / "news" / 明确指定 RSS/博文/播客
|
├─ "新闻" / "news"
|
||||||
│ → web-search news <query> [-n 5]
|
│ → web-search news <query>
|
||||||
│
|
│
|
||||||
├─ 明确指定了特定源名称
|
├─ "健康检查" / "health"
|
||||||
│ → web-search search <query> -s "源名1" "源名2" [-n 5]
|
|
||||||
│
|
|
||||||
├─ 需要多类源对比(如"新闻+社交上关于伊朗的看法")
|
|
||||||
│ → web-search search <query> -s news social [-n 10]
|
|
||||||
│
|
|
||||||
├─ 需要百度搜索 + 站点/时间过滤
|
|
||||||
│ → web-search baidu <query> [--site xxx] [--recency week]
|
|
||||||
│
|
|
||||||
├─ "健康检查" / "health" / "测试引擎"
|
|
||||||
│ → web-search health
|
│ → web-search health
|
||||||
│
|
│
|
||||||
├─ "列出源" / "sources" / "有哪些搜索引擎"
|
├─ "有哪些源" / "sources"
|
||||||
│ → web-search sources
|
│ → web-search sources
|
||||||
│
|
│
|
||||||
├─ 其余通用查询
|
└─ 通用搜索
|
||||||
│ → web-search search <query> [-s 源1 源2...] [-n 10]
|
|
||||||
│
|
|
||||||
└─ 不确定 / 没特别要求
|
|
||||||
→ web-search search <query> -n 10
|
→ web-search search <query> -n 10
|
||||||
```
|
```
|
||||||
|
|
||||||
## Quick Command Reference
|
## Python API
|
||||||
|
|
||||||
| Command | Speed | Best for |
|
|
||||||
|---------|-------|----------|
|
|
||||||
| `search` | 并发 28 源 | 通用搜索、多源对比 |
|
|
||||||
| `web` | 14 网页引擎 | 普通网页搜索 |
|
|
||||||
| `baidu` | 单引擎 | 需 site/recency 过滤 |
|
|
||||||
| `news` | ~10+ 新闻源 | 科技/财经/博客/播客 |
|
|
||||||
| `wechat` | 单引擎 | 微信公众号独家内容 |
|
|
||||||
| `academic` | 单引擎 | 学术论文 |
|
|
||||||
| `social` | 2 源 (Twitter) | 社交媒体动态 |
|
|
||||||
| `rss` | 18 订阅源 | AI 周报/深度文章/播客 |
|
|
||||||
| `health` | 28 源并发 (~14s) | 检测连通性 |
|
|
||||||
|
|
||||||
**通用选项:** `-n N`(每源结果数), `-j`(JSON), `-o FILE`(存文件), `-s NAME...`(指定源), `-v`(调试日志)
|
|
||||||
|
|
||||||
## Result Field Reference
|
|
||||||
|
|
||||||
Each `SearchResult` returned has these fields:
|
|
||||||
|
|
||||||
| Field | Type | Description |
|
|
||||||
|-------|------|-------------|
|
|
||||||
| `rank` | int | Position in results (1-based) |
|
|
||||||
| `title` | str | Article/title text |
|
|
||||||
| `url` | str | Direct URL to source |
|
|
||||||
| `snippet` | str | Summary text (first ~300 chars) |
|
|
||||||
| `source` | str | Engine name (e.g. "Hacker News") |
|
|
||||||
| `category` | str | "web", "news", "academic", "social", etc. |
|
|
||||||
| `extra` | dict | Engine-specific metadata (see below) |
|
|
||||||
|
|
||||||
### `extra` fields by source
|
|
||||||
|
|
||||||
| Source | extra fields |
|
|
||||||
|--------|-------------|
|
|
||||||
| Hacker News | `points`, `author`, `time` (ISO date) |
|
|
||||||
| GitHub Trending | `language`, `stars today` (heat) |
|
|
||||||
| WeChat | `account` (公众号名), `time` |
|
|
||||||
| Twitter | `time`, `account`, `search_type` |
|
|
||||||
| RSS feeds | `time`, `feed_url` |
|
|
||||||
| Cls / WallStreetCN | `search_url` (JS-rendered fallback) |
|
|
||||||
| Baidu Qianfan | `site`, `recency` (if filtered) |
|
|
||||||
|
|
||||||
## Error Handling
|
|
||||||
|
|
||||||
| Symptom | Likely cause | Action |
|
|
||||||
|---------|-------------|--------|
|
|
||||||
| Empty results from Hacker News/GitHub | Network issue or no matches | Retry with different query |
|
|
||||||
| Baidu Qianfan returns nothing | Missing `BAIDU_API_KEY` env | Check env var is set |
|
|
||||||
| DuckDuckGo times out (15s) | Anti-scraping / rate limit | Note to user, skip |
|
|
||||||
| WeChat empty results | Sogou anti-scraping | Inform user it may be limited |
|
|
||||||
| `web-search health` shows [TIMEOUT] | Engine unreachable from this region | Not critical, skip that engine |
|
|
||||||
| GBK encode error on Windows | Non-ASCII char in output | Already auto-fixed by `_sanitize_for_console` |
|
|
||||||
|
|
||||||
**General rule:** If a specific source fails, note it but don't fail the whole search — other sources continue concurrently.
|
|
||||||
|
|
||||||
## Usage Examples
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Full search (all 28 sources, concurrent)
|
|
||||||
web-search search "quantum computing" -n 10
|
|
||||||
|
|
||||||
# Specific sources only
|
|
||||||
web-search search "technology" -s "Hacker News" "V2EX" "GitHub Trending" -n 3
|
|
||||||
|
|
||||||
# Multiple categories
|
|
||||||
web-search search "Iran nuclear" -s web news social -n 5
|
|
||||||
|
|
||||||
# Baidu with site filter
|
|
||||||
web-search baidu "中美元首会晤" --site mp.weixin.qq.com --recency week
|
|
||||||
|
|
||||||
# All web engines
|
|
||||||
web-search web "Python async" -r global -j
|
|
||||||
|
|
||||||
# Quick searches
|
|
||||||
web-search news "人工智能" -n 5
|
|
||||||
web-search wechat 蒙面财经
|
|
||||||
web-search academic "machine learning"
|
|
||||||
web-search social "AI regulation" -n 5
|
|
||||||
|
|
||||||
# Concurrent health check (~14s for 28 engines)
|
|
||||||
web-search health
|
|
||||||
```
|
|
||||||
|
|
||||||
## Python API (for programmatic use)
|
|
||||||
|
|
||||||
> **Tip:** For Agent use, prefer invoking the `web-search` CLI command directly (see examples above).
|
|
||||||
> The API import below only works when running from the project root directory.
|
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from scripts.unified_search import UnifiedSearch
|
from scripts.unified_search import UnifiedSearch
|
||||||
|
|
||||||
async def main():
|
async def main():
|
||||||
us = UnifiedSearch()
|
us = UnifiedSearch()
|
||||||
results = await us.search(
|
resp = await us.search(
|
||||||
"quantum computing",
|
"quantum computing",
|
||||||
sources=["Hacker News", "ArXiv"],
|
sources=["bing", "arxiv"], # or None for all
|
||||||
max_results=5,
|
max_results=5,
|
||||||
)
|
)
|
||||||
for r in results:
|
for r in resp.results:
|
||||||
print(f"[{r.rank}] {r.title} ({r.source})")
|
print(f"[{r.rank}] {r.title}")
|
||||||
if r.extra:
|
print(f" {r.url}")
|
||||||
print(f" extra: {r.extra}")
|
print(f" {r.snippet[:100]}")
|
||||||
|
print(f"Engines used: {resp.metadata.engines_used}")
|
||||||
|
print(f"Duration: {resp.metadata.duration_ms}ms")
|
||||||
await us.close()
|
await us.close()
|
||||||
```
|
```
|
||||||
|
|
||||||
## Important Notes
|
## SearchResult Fields
|
||||||
|
|
||||||
### Installation
|
| Field | Type | Description |
|
||||||
The `web-search` CLI must be installed first before use:
|
|-------|------|-------------|
|
||||||
```bash
|
| `rank` | int | Position after deduplication (1-based) |
|
||||||
cd web-search-skills
|
| `title` | str | Article/title text |
|
||||||
pip install -e .
|
| `url` | str | Direct URL |
|
||||||
```
|
| `snippet` | str | Summary (~300 chars) |
|
||||||
This registers `web-search` as a global command. Requires Python 3.8+.
|
| `source` | str | Engine name (e.g. "bing", "arxiv") |
|
||||||
|
| `category` | str | "web" \| "news" \| "academic" \| "github" |
|
||||||
|
| `published` | str | ISO date (optional) |
|
||||||
|
| `extra` | dict | Engine-specific metadata |
|
||||||
|
|
||||||
### Known Limitations
|
## Error Handling
|
||||||
|
|
||||||
- **WeChat** via Sogou has anti-scraping limits, may return empty
|
| Symptom | Cause | Action |
|
||||||
- **财联社** uses public API (telegraphList) — returns parsed real-time news
|
|---------|-------|--------|
|
||||||
- **华尔街见闻** need JS rendering → returns search page URL, not parsed results
|
| Empty Bing News | Anti-scraping / region | Not critical, skip |
|
||||||
- **Baidu Qianfan** requires `BAIDU_API_KEY` env var
|
| Baidu Qianfan empty | Missing `BAIDU_API_KEY` | Set env var |
|
||||||
- **RSS feeds** need stable internet
|
| Sogou WeChat empty | Sogou anti-scraping | Show URL fallback |
|
||||||
- **Windows GBK** terminals: special chars auto-replaced by `_sanitize_for_console()`
|
| Nitter all fail | Instance blocked | Show direct twitter.com link |
|
||||||
|
|
||||||
> Full user-facing documentation (detailed command reference, project structure, config guide, architecture) is in `README.md`.
|
## Notes
|
||||||
|
|
||||||
|
- Most engines require **no API key**
|
||||||
|
- Baidu Qianfan needs `BAIDU_API_KEY` env var
|
||||||
|
- All engines run concurrently via `asyncio.gather`
|
||||||
|
- Max 5 concurrent engines (semaphore)
|
||||||
|
- Per-engine timeout: 15s
|
||||||
|
- Results are deduplicated by URL before ranking
|
||||||
+11
-10
@@ -1,20 +1,21 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "web-search-skills"
|
name = "web-search-skills"
|
||||||
version = "2.2.0"
|
version = "0.2.0"
|
||||||
description = "Multi-source web search CLI tool"
|
description = "Open-source diversified search in <5s. 28+ engines across web, news, academic, social, WeChat."
|
||||||
requires-python = ">=3.8"
|
authors = [{ name = "openbot-coder" }]
|
||||||
|
readme = "README.md"
|
||||||
|
license = { file = "LICENSE" }
|
||||||
|
|
||||||
|
requires-python = ">=3.10"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"httpx>=0.25.0",
|
"httpx>=0.24.0",
|
||||||
"beautifulsoup4>=4.12.0",
|
"beautifulsoup4>=4.12.0",
|
||||||
"lxml>=4.9.0",
|
"lxml>=5.0.0",
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
web-search = "scripts.cli:main"
|
web-search = "ws:main"
|
||||||
|
|
||||||
[build-system]
|
[build-system]
|
||||||
requires = ["setuptools>=64", "wheel"]
|
requires = ["setuptools>=68.0"]
|
||||||
build-backend = "setuptools.build_meta"
|
build-backend = "setuptools.build_meta"
|
||||||
|
|
||||||
[tool.setuptools.packages.find]
|
|
||||||
include = ["scripts*"]
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
"""Academic search engines."""
|
|
||||||
@@ -1,130 +0,0 @@
|
|||||||
"""
|
|
||||||
ArXiv academic paper search engine.
|
|
||||||
|
|
||||||
Uses the ArXiv API (export.arxiv.org/api/query) to search for papers.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
import xml.etree.ElementTree as ET
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from typing import Optional
|
|
||||||
from urllib.parse import quote_plus
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
|
|
||||||
from scripts.core.base import SearchEngine, SearchResult
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
class ArXivEngine(SearchEngine):
|
|
||||||
"""ArXiv academic paper search engine via official API."""
|
|
||||||
|
|
||||||
API_URL = "https://export.arxiv.org/api/query"
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
|
||||||
super().__init__("ArXiv", config)
|
|
||||||
timeout = (config or {}).get("timeout", 30)
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=timeout,
|
|
||||||
headers={
|
|
||||||
"User-Agent": (config or {}).get(
|
|
||||||
"user_agent",
|
|
||||||
"ArXivSearchSkill/1.0",
|
|
||||||
),
|
|
||||||
"Accept": "application/atom+xml",
|
|
||||||
},
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
def build_url(self, query: str, max_results: int = 10, **kwargs) -> str:
|
|
||||||
search_field = kwargs.get("search_field", "all")
|
|
||||||
return (
|
|
||||||
f"{self.API_URL}?search_query={search_field}:{quote_plus(query)}"
|
|
||||||
f"&max_results={max_results}&sortBy=submittedDate&sortOrder=descending"
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(
|
|
||||||
self, query: str, max_results: int = 10, **kwargs
|
|
||||||
) -> list[SearchResult]:
|
|
||||||
"""Search ArXiv papers."""
|
|
||||||
url = self.build_url(query, max_results=max_results, **kwargs)
|
|
||||||
try:
|
|
||||||
response = await self._client.get(url)
|
|
||||||
response.raise_for_status()
|
|
||||||
return self._parse_response(response.text, max_results)
|
|
||||||
except httpx.HTTPError as e:
|
|
||||||
logger.error(f"ArXiv search failed: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_response(self, xml_text: str, max_results: int) -> list[SearchResult]:
|
|
||||||
"""Parse ArXiv Atom XML response."""
|
|
||||||
ns = {
|
|
||||||
"atom": "http://www.w3.org/2005/Atom",
|
|
||||||
"arxiv": "http://arxiv.org/schemas/atom",
|
|
||||||
}
|
|
||||||
results = []
|
|
||||||
|
|
||||||
try:
|
|
||||||
root = ET.fromstring(xml_text)
|
|
||||||
except ET.ParseError as e:
|
|
||||||
logger.error(f"ArXiv XML parse error: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
for i, entry in enumerate(root.findall("atom:entry", ns)[:max_results]):
|
|
||||||
title = entry.findtext("atom:title", "", ns)
|
|
||||||
summary = entry.findtext("atom:summary", "", ns)
|
|
||||||
published = entry.findtext("atom:published", "", ns)
|
|
||||||
updated = entry.findtext("atom:updated", "", ns)
|
|
||||||
|
|
||||||
link_el = entry.find("atom:link", ns)
|
|
||||||
link = link_el.get("href", "") if link_el is not None else ""
|
|
||||||
|
|
||||||
authors = []
|
|
||||||
for author in entry.findall("atom:author", ns):
|
|
||||||
name = author.findtext("atom:name", "", ns)
|
|
||||||
if name:
|
|
||||||
authors.append(name)
|
|
||||||
|
|
||||||
# Clean title and summary
|
|
||||||
title = title.replace("\n", " ").strip()
|
|
||||||
snippet = summary.replace("\n", " ").strip()[:300]
|
|
||||||
|
|
||||||
parsed_date = self._parse_date(published) if published else (
|
|
||||||
self._parse_date(updated) if updated else
|
|
||||||
datetime.min.replace(tzinfo=timezone.utc)
|
|
||||||
)
|
|
||||||
|
|
||||||
results.append(
|
|
||||||
SearchResult(
|
|
||||||
title=title or f"ArXiv: {summary[:60]}",
|
|
||||||
url=link,
|
|
||||||
snippet=snippet or "No abstract available",
|
|
||||||
source=self.name,
|
|
||||||
rank=i + 1,
|
|
||||||
category="academic",
|
|
||||||
extra={
|
|
||||||
"published": published or "",
|
|
||||||
"updated": updated or "",
|
|
||||||
"authors": ", ".join(authors),
|
|
||||||
"parsed_date": parsed_date,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
return results
|
|
||||||
|
|
||||||
def _parse_date(self, date_str: str) -> datetime:
|
|
||||||
"""Parse ArXiv date format."""
|
|
||||||
try:
|
|
||||||
# ArXiv format: 2024-01-01T00:00:00Z
|
|
||||||
return datetime.strptime(
|
|
||||||
date_str.strip(), "%Y-%m-%dT%H:%M:%SZ"
|
|
||||||
).replace(tzinfo=timezone.utc)
|
|
||||||
except (ValueError, TypeError):
|
|
||||||
return datetime.min.replace(tzinfo=timezone.utc)
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
|
||||||
+529
-488
File diff suppressed because it is too large.
Load diff
@@ -43,6 +43,10 @@ try:
|
|||||||
# --- 日志 ---
|
# --- 日志 ---
|
||||||
log_level: str = "INFO"
|
log_level: str = "INFO"
|
||||||
|
|
||||||
|
# --- API 引擎每日限额(有 key 的尽量做替补)---
|
||||||
|
baidu_daily_limit: int = 50
|
||||||
|
minimax_daily_limit: int = 100
|
||||||
|
|
||||||
except ImportError:
|
except ImportError:
|
||||||
|
|
||||||
class Settings: # type: ignore
|
class Settings: # type: ignore
|
||||||
@@ -76,6 +80,9 @@ except ImportError:
|
|||||||
os.environ.get("MAX_CONCURRENT", "5")
|
os.environ.get("MAX_CONCURRENT", "5")
|
||||||
)
|
)
|
||||||
self.log_level = os.environ.get("LOG_LEVEL", "INFO")
|
self.log_level = os.environ.get("LOG_LEVEL", "INFO")
|
||||||
|
# API 引擎每日限额
|
||||||
|
self.baidu_daily_limit = int(os.environ.get("BAIDU_DAILY_LIMIT", "50"))
|
||||||
|
self.minimax_daily_limit = int(os.environ.get("MINIMAX_DAILY_LIMIT", "100"))
|
||||||
|
|
||||||
|
|
||||||
# 全局单例
|
# 全局单例
|
||||||
|
|||||||
+48
-27
@@ -1,48 +1,69 @@
|
|||||||
"""
|
"""
|
||||||
Core data types and base classes for search engines.
|
Unified data models — single source of truth for all search results.
|
||||||
|
|
||||||
Provides:
|
Defines:
|
||||||
- SearchResult: Standard result dataclass used across all engines
|
- SearchResult: unified result from any engine (immutable dataclass)
|
||||||
- SearchEngine: Abstract base class for all search engine implementations
|
- BaseEngine: abstract base for all engines
|
||||||
|
- rank field: assigned by aggregator, not engines
|
||||||
|
|
||||||
|
All engine-specific models must convert to SearchResult before returning.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from abc import ABC, abstractmethod
|
from abc import ABC, abstractmethod
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from typing import Any, Optional
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class SearchResult:
|
class SearchResult:
|
||||||
"""Represents a single search result."""
|
"""Unified search result — returned by all engines."""
|
||||||
title: str = ""
|
title: str
|
||||||
url: str = ""
|
url: str
|
||||||
snippet: str = ""
|
snippet: str = ""
|
||||||
source: str = ""
|
source: str = "" # engine name (e.g. "bing", "arxiv")
|
||||||
rank: int = 0
|
category: str = "" # "web" | "news" | "academic" | "github" | "social"
|
||||||
category: str = ""
|
rank: int = 0 # assigned by aggregator, not engine
|
||||||
|
published: str = "" # ISO date string (optional)
|
||||||
extra: dict[str, Any] = field(default_factory=dict)
|
extra: dict[str, Any] = field(default_factory=dict)
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"title": self.title,
|
||||||
|
"url": self.url,
|
||||||
|
"snippet": self.snippet,
|
||||||
|
"source": self.source,
|
||||||
|
"category": self.category,
|
||||||
|
"rank": self.rank,
|
||||||
|
"published": self.published,
|
||||||
|
"extra": self.extra,
|
||||||
|
}
|
||||||
|
|
||||||
class SearchEngine(ABC):
|
|
||||||
"""Abstract base class for search engines."""
|
|
||||||
|
|
||||||
def __init__(self, name: str, config: Optional[dict[str, Any]] = None):
|
class BaseEngine(ABC):
|
||||||
self.name = name
|
"""Abstract base for all search engines.
|
||||||
self.config = config or {}
|
|
||||||
|
Subclasses override `search()`, which returns list[SearchResult].
|
||||||
|
`health()` provides a quick connectivity check.
|
||||||
|
"""
|
||||||
|
|
||||||
|
name: str = "unknown"
|
||||||
|
timeout: float = 15.0
|
||||||
|
|
||||||
|
def __init__(self, timeout: float = 15.0):
|
||||||
|
self.timeout = timeout
|
||||||
|
|
||||||
@abstractmethod
|
@abstractmethod
|
||||||
async def search(
|
async def search(self, query: str, max_results: int = 10,
|
||||||
self, query: str, max_results: int = 10, **kwargs
|
**kwargs) -> list[SearchResult]:
|
||||||
) -> list[SearchResult]:
|
"""Search and return unified results."""
|
||||||
"""Execute a search and return results."""
|
|
||||||
...
|
...
|
||||||
|
|
||||||
def build_url(self, query: str, **kwargs) -> str:
|
async def health(self) -> dict[str, Any]:
|
||||||
"""Build a search URL for this engine (optional)."""
|
"""Quick connectivity check. Returns {"status": "ok"} or {"status": "fail", "error": "..."}."""
|
||||||
return ""
|
try:
|
||||||
|
results = await self.search("test", max_results=1)
|
||||||
async def close(self):
|
return {"name": self.name, "status": "ok", "count": len(results)}
|
||||||
"""Clean up resources."""
|
except Exception as e:
|
||||||
pass
|
return {"name": self.name, "status": "fail", "error": str(e)}
|
||||||
@@ -0,0 +1,232 @@
|
|||||||
|
"""
|
||||||
|
执行日志模块 — 记录各引擎/接口的调用耗时、报错信息。
|
||||||
|
|
||||||
|
用于跟踪各个接口的执行效果,便于后续优化。
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
from scripts.core.execution_log import execution_log
|
||||||
|
|
||||||
|
# 自动记录耗时(推荐用法)
|
||||||
|
with execution_log.track("bing", "search") as ctx:
|
||||||
|
results = await engine.search(query)
|
||||||
|
ctx.result_count = len(results)
|
||||||
|
|
||||||
|
# 手动记录
|
||||||
|
execution_log.record(EngineCallLog(
|
||||||
|
engine="bing",
|
||||||
|
operation="search",
|
||||||
|
duration_ms=320,
|
||||||
|
success=True,
|
||||||
|
result_count=5,
|
||||||
|
))
|
||||||
|
|
||||||
|
# 打印摘要
|
||||||
|
execution_log.print_summary()
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from dataclasses import dataclass, field, asdict
|
||||||
|
from datetime import datetime
|
||||||
|
from typing import Any, Optional
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class EngineCallLog:
|
||||||
|
"""单次引擎调用记录。"""
|
||||||
|
engine: str # 引擎名(bing, sogou, baidu, ...)
|
||||||
|
operation: str = "search" # 操作类型(search, fetch, health, ...)
|
||||||
|
timestamp: str = "" # 调用时间 ISO 格式
|
||||||
|
start_time: float = 0.0 # 起始时间戳
|
||||||
|
duration_ms: int = 0 # 耗时(毫秒)
|
||||||
|
success: bool = True # 是否成功
|
||||||
|
result_count: int = 0 # 返回结果数
|
||||||
|
error: Optional[str] = None # 错误信息
|
||||||
|
error_type: Optional[str] = None # 错误类型(异常类名)
|
||||||
|
status_code: Optional[int] = None # HTTP 状态码(如有)
|
||||||
|
detail: dict[str, Any] = field(default_factory=dict) # 额外信息
|
||||||
|
|
||||||
|
|
||||||
|
class _ExecutionLogCollector:
|
||||||
|
"""全局执行日志收集器。"""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self._logs: list[EngineCallLog] = []
|
||||||
|
self._session_start = time.time()
|
||||||
|
|
||||||
|
def record(self, log: EngineCallLog):
|
||||||
|
"""记录一次引擎调用。"""
|
||||||
|
self._logs.append(log)
|
||||||
|
|
||||||
|
# 同时写 logger
|
||||||
|
status = "✓" if log.success else "✗"
|
||||||
|
msg = (
|
||||||
|
f"{status} [{log.engine}] {log.operation} "
|
||||||
|
f"{log.duration_ms}ms"
|
||||||
|
)
|
||||||
|
if log.result_count > 0:
|
||||||
|
msg += f" → {log.result_count} results"
|
||||||
|
if log.error:
|
||||||
|
msg += f" | error: {log.error}"
|
||||||
|
if log.status_code and log.status_code >= 400:
|
||||||
|
msg += f" | HTTP {log.status_code}"
|
||||||
|
|
||||||
|
if log.success:
|
||||||
|
logger.info(msg)
|
||||||
|
else:
|
||||||
|
logger.warning(msg)
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def track(self, engine: str, operation: str = "search",
|
||||||
|
**extra):
|
||||||
|
"""上下文管理器,自动记录耗时和异常。
|
||||||
|
|
||||||
|
Example:
|
||||||
|
with execution_log.track("bing", "search", region="cn") as ctx:
|
||||||
|
results = await engine.search(query)
|
||||||
|
ctx.result_count = len(results)
|
||||||
|
"""
|
||||||
|
entry = EngineCallLog(
|
||||||
|
engine=engine,
|
||||||
|
operation=operation,
|
||||||
|
timestamp=datetime.now().isoformat(),
|
||||||
|
start_time=time.time(),
|
||||||
|
detail=dict(extra),
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
yield entry
|
||||||
|
except Exception as e:
|
||||||
|
entry.success = False
|
||||||
|
entry.error = str(e)
|
||||||
|
entry.error_type = type(e).__name__
|
||||||
|
raise
|
||||||
|
finally:
|
||||||
|
entry.duration_ms = int((time.time() - entry.start_time) * 1000)
|
||||||
|
self.record(entry)
|
||||||
|
|
||||||
|
def get_logs(self, engine: str | None = None) -> list[EngineCallLog]:
|
||||||
|
"""获取日志,可按引擎名过滤。"""
|
||||||
|
if engine:
|
||||||
|
return [l for l in self._logs if l.engine == engine]
|
||||||
|
return list(self._logs)
|
||||||
|
|
||||||
|
def get_stats(self, engine: str | None = None) -> dict[str, Any]:
|
||||||
|
"""获取统计摘要。"""
|
||||||
|
logs = self.get_logs(engine)
|
||||||
|
if not logs:
|
||||||
|
return {"total_calls": 0}
|
||||||
|
|
||||||
|
total = len(logs)
|
||||||
|
success = sum(1 for l in logs if l.success)
|
||||||
|
failed = total - success
|
||||||
|
durations = [l.duration_ms for l in logs]
|
||||||
|
result_counts = [l.result_count for l in logs]
|
||||||
|
|
||||||
|
errors = []
|
||||||
|
for l in logs:
|
||||||
|
if l.error:
|
||||||
|
errors.append({
|
||||||
|
"engine": l.engine,
|
||||||
|
"operation": l.operation,
|
||||||
|
"error_type": l.error_type,
|
||||||
|
"error": l.error[:200],
|
||||||
|
"timestamp": l.timestamp,
|
||||||
|
})
|
||||||
|
|
||||||
|
return {
|
||||||
|
"total_calls": total,
|
||||||
|
"success": success,
|
||||||
|
"failed": failed,
|
||||||
|
"success_rate": f"{success/total*100:.1f}%",
|
||||||
|
"avg_duration_ms": sum(durations) // total,
|
||||||
|
"max_duration_ms": max(durations),
|
||||||
|
"min_duration_ms": min(durations),
|
||||||
|
"total_results": sum(result_counts),
|
||||||
|
"errors": errors,
|
||||||
|
}
|
||||||
|
|
||||||
|
def print_summary(self):
|
||||||
|
"""打印本次会话的执行摘要。"""
|
||||||
|
if not self._logs:
|
||||||
|
print("暂无执行记录")
|
||||||
|
return
|
||||||
|
|
||||||
|
total_time = int((time.time() - self._session_start) * 1000)
|
||||||
|
stats = self.get_stats()
|
||||||
|
|
||||||
|
lines = [
|
||||||
|
"═══════════════════════════════════════════",
|
||||||
|
" 执行日志摘要",
|
||||||
|
"═══════════════════════════════════════════",
|
||||||
|
f" 总调用次数: {stats['total_calls']}",
|
||||||
|
f" 成功/失败: {stats['success']}/{stats['failed']} ({stats['success_rate']})",
|
||||||
|
f" 平均耗时: {stats['avg_duration_ms']}ms",
|
||||||
|
f" 最长耗时: {stats['max_duration_ms']}ms",
|
||||||
|
f" 最短耗时: {stats['min_duration_ms']}ms",
|
||||||
|
f" 总结果数: {stats['total_results']}",
|
||||||
|
f" 总耗时: {total_time}ms",
|
||||||
|
"───────────────────────────────────────────",
|
||||||
|
]
|
||||||
|
|
||||||
|
# 按引擎分组
|
||||||
|
engine_names = sorted(set(l.engine for l in self._logs))
|
||||||
|
lines.append(" 引擎明细:")
|
||||||
|
for name in engine_names:
|
||||||
|
estat = self.get_stats(name)
|
||||||
|
status = "✓" if estat["failed"] == 0 else "✗"
|
||||||
|
lines.append(
|
||||||
|
f" {status} {name:15s} "
|
||||||
|
f"calls={estat['total_calls']} "
|
||||||
|
f"avg={estat['avg_duration_ms']}ms "
|
||||||
|
f"max={estat['max_duration_ms']}ms "
|
||||||
|
f"results={estat['total_results']}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# 错误详情
|
||||||
|
if stats["errors"]:
|
||||||
|
lines.append("───────────────────────────────────────────")
|
||||||
|
lines.append(" 错误详情:")
|
||||||
|
for err in stats["errors"]:
|
||||||
|
lines.append(
|
||||||
|
f" ✗ [{err['engine']}] {err['error_type']}: {err['error'][:100]}"
|
||||||
|
)
|
||||||
|
|
||||||
|
lines.append("═══════════════════════════════════════════")
|
||||||
|
print("\n".join(lines))
|
||||||
|
|
||||||
|
def to_json(self) -> str:
|
||||||
|
"""导出为 JSON。"""
|
||||||
|
data = {
|
||||||
|
"session_start": datetime.fromtimestamp(self._session_start).isoformat(),
|
||||||
|
"total_calls": len(self._logs),
|
||||||
|
"stats": self.get_stats(),
|
||||||
|
"logs": [asdict(l) for l in self._logs],
|
||||||
|
}
|
||||||
|
return json.dumps(data, ensure_ascii=False, indent=2)
|
||||||
|
|
||||||
|
def save(self, path: str = "logs/search_log.jsonl"):
|
||||||
|
"""追加写入 JSONL 日志文件。"""
|
||||||
|
import os
|
||||||
|
os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
|
||||||
|
with open(path, "a", encoding="utf-8") as f:
|
||||||
|
for log in self._logs:
|
||||||
|
record = asdict(log)
|
||||||
|
record["session_start"] = datetime.fromtimestamp(
|
||||||
|
self._session_start
|
||||||
|
).isoformat()
|
||||||
|
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||||
|
|
||||||
|
def clear(self):
|
||||||
|
"""清空日志。"""
|
||||||
|
self._logs.clear()
|
||||||
|
self._session_start = time.time()
|
||||||
|
|
||||||
|
|
||||||
|
# 全局单例
|
||||||
|
execution_log = _ExecutionLogCollector()
|
||||||
@@ -0,0 +1,158 @@
|
|||||||
|
"""Academic search — ArXiv + CrossRef APIs (free, no API key)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from urllib.parse import quote_plus
|
||||||
|
from xml.etree import ElementTree as ET
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class ArxivEngine(BaseEngine):
|
||||||
|
"""ArXiv paper search via public API."""
|
||||||
|
|
||||||
|
name = "arxiv"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
**kwargs) -> list[SearchResult]:
|
||||||
|
t0 = time.time()
|
||||||
|
url = (
|
||||||
|
f"http://export.arxiv.org/api/query?"
|
||||||
|
f"search_query=all:{quote_plus(query)}&start=0&max_results={max_results}"
|
||||||
|
f"&sortBy=relevance&sortOrder=descending"
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
|
||||||
|
resp = await client.get(url)
|
||||||
|
resp.raise_for_status()
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
results = self._parse_xml(resp.text, max_results)
|
||||||
|
logger.info(f"[arxiv] {len(results)} results in {elapsed:.2f}s")
|
||||||
|
return results
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[arxiv] failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse_xml(self, xml_text: str, max_results: int) -> list[SearchResult]:
|
||||||
|
try:
|
||||||
|
root = ET.fromstring(xml_text)
|
||||||
|
except ET.ParseError:
|
||||||
|
return []
|
||||||
|
|
||||||
|
ns = {"atom": "http://www.w3.org/2005/Atom"}
|
||||||
|
results = []
|
||||||
|
|
||||||
|
for entry in root.findall("atom:entry", ns)[:max_results]:
|
||||||
|
title = entry.findtext("atom:title", "", ns).strip().replace("\n", " ")
|
||||||
|
summary = entry.findtext("atom:summary", "", ns).strip().replace("\n", " ")
|
||||||
|
link_el = entry.find("atom:link[@type='text/html']", ns)
|
||||||
|
if link_el is None:
|
||||||
|
link_el = entry.find("atom:id", ns)
|
||||||
|
url = link_el.get("href", "") if link_el is not None else ""
|
||||||
|
published = entry.findtext("atom:published", "", ns)
|
||||||
|
authors = [a.findtext("atom:name", "", ns) for a in entry.findall("atom:author", ns)]
|
||||||
|
categories = [c.get("term", "") for c in entry.findall("atom:category", ns)]
|
||||||
|
|
||||||
|
if title:
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title, url=url, snippet=summary[:300],
|
||||||
|
source="arxiv", category="academic",
|
||||||
|
published=published,
|
||||||
|
extra={"authors": authors[:5], "categories": categories},
|
||||||
|
))
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
class CrossRefEngine(BaseEngine):
|
||||||
|
"""CrossRef academic search via public API."""
|
||||||
|
|
||||||
|
name = "crossref"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
**kwargs) -> list[SearchResult]:
|
||||||
|
t0 = time.time()
|
||||||
|
url = "https://api.crossref.org/works"
|
||||||
|
params = {"query": query, "rows": min(max_results, 20),
|
||||||
|
"sort": "relevance", "order": "desc"}
|
||||||
|
headers = {"User-Agent": "web-search-skills/1.0 (mailto:search@example.com)"}
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
|
||||||
|
resp = await client.get(url, params=params, headers=headers)
|
||||||
|
resp.raise_for_status()
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
data = resp.json()
|
||||||
|
results = []
|
||||||
|
for item in data.get("message", {}).get("items", [])[:max_results]:
|
||||||
|
title_list = item.get("title", [])
|
||||||
|
title = title_list[0] if title_list else ""
|
||||||
|
doi = item.get("DOI", "")
|
||||||
|
url_link = item.get("URL", f"https://doi.org/{doi}")
|
||||||
|
abstract = item.get("abstract", "")[:300]
|
||||||
|
pub_date = ""
|
||||||
|
if item.get("published-print"):
|
||||||
|
dp = item["published-print"].get("date-parts", [[]])[0]
|
||||||
|
pub_date = "-".join(str(p) for p in dp if p) if dp else ""
|
||||||
|
|
||||||
|
authors = [f"{a.get('given', '')} {a.get('family', '')}".strip()
|
||||||
|
for a in item.get("author", [])]
|
||||||
|
|
||||||
|
if title:
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title, url=url_link, snippet=abstract,
|
||||||
|
source="crossref", category="academic",
|
||||||
|
published=pub_date,
|
||||||
|
extra={
|
||||||
|
"doi": doi,
|
||||||
|
"authors": authors[:5],
|
||||||
|
"journal": (item.get("container-title", [""])[0]
|
||||||
|
if item.get("container-title") else ""),
|
||||||
|
},
|
||||||
|
))
|
||||||
|
|
||||||
|
logger.info(f"[crossref] {len(results)} results in {elapsed:.2f}s")
|
||||||
|
return results
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[crossref] failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
class AcademicSearch(BaseEngine):
|
||||||
|
"""Unified academic search — ArXiv + CrossRef in parallel."""
|
||||||
|
|
||||||
|
name = "academic"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
**kwargs) -> list[SearchResult]:
|
||||||
|
import asyncio
|
||||||
|
arxiv = ArxivEngine(timeout=self.timeout)
|
||||||
|
crossref = CrossRefEngine(timeout=self.timeout)
|
||||||
|
|
||||||
|
arxiv_results, crossref_results = await asyncio.gather(
|
||||||
|
arxiv.search(query, max_results=max_results),
|
||||||
|
crossref.search(query, max_results=max_results),
|
||||||
|
return_exceptions=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
all_results = []
|
||||||
|
if isinstance(arxiv_results, list):
|
||||||
|
all_results.extend(arxiv_results)
|
||||||
|
if isinstance(crossref_results, list):
|
||||||
|
all_results.extend(crossref_results)
|
||||||
|
|
||||||
|
seen = set()
|
||||||
|
deduped = []
|
||||||
|
for r in all_results:
|
||||||
|
if r.url not in seen:
|
||||||
|
seen.add(r.url)
|
||||||
|
deduped.append(r)
|
||||||
|
|
||||||
|
return deduped[:max_results]
|
||||||
@@ -18,7 +18,7 @@ from typing import Any, Optional
|
|||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
|
|
||||||
from scripts.core.base import SearchEngine, SearchResult
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -51,18 +51,18 @@ def _get_api_key() -> str:
|
|||||||
return env_from_file.get(ENV_KEY, "")
|
return env_from_file.get(ENV_KEY, "")
|
||||||
|
|
||||||
|
|
||||||
class BaiduQianFanEngine(SearchEngine):
|
class BaiduQianFanEngine(BaseEngine):
|
||||||
"""百度千帆 AI 搜索 API 引擎."""
|
"""百度千帆 AI 搜索 API 引擎."""
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
name = "baidu_qianfan"
|
||||||
super().__init__("Baidu Qianfan", config)
|
|
||||||
|
def __init__(self, config: Optional[dict] = None, timeout: float = 30.0):
|
||||||
|
super().__init__(timeout=timeout)
|
||||||
self.api_key = (config or {}).get("api_key", "") or _get_api_key()
|
self.api_key = (config or {}).get("api_key", "") or _get_api_key()
|
||||||
if not self.api_key:
|
if not self.api_key:
|
||||||
logger.warning(
|
logger.warning(
|
||||||
f"BAIDU_API_KEY not set! Set it in .env file or as environment variable."
|
"BAIDU_API_KEY not set. Set it in .env or as environment variable."
|
||||||
)
|
)
|
||||||
|
|
||||||
timeout = (config or {}).get("timeout", 30)
|
|
||||||
self._client = httpx.AsyncClient(
|
self._client = httpx.AsyncClient(
|
||||||
timeout=timeout,
|
timeout=timeout,
|
||||||
headers={
|
headers={
|
||||||
@@ -76,17 +76,7 @@ class BaiduQianFanEngine(SearchEngine):
|
|||||||
async def search(
|
async def search(
|
||||||
self, query: str, max_results: int = 10, **kwargs
|
self, query: str, max_results: int = 10, **kwargs
|
||||||
) -> list[SearchResult]:
|
) -> list[SearchResult]:
|
||||||
"""Search via Baidu Qianfan AI Search API.
|
"""Search via Baidu Qianfan AI Search API."""
|
||||||
|
|
||||||
Args:
|
|
||||||
query: Search keywords.
|
|
||||||
max_results: Max results to return.
|
|
||||||
**kwargs:
|
|
||||||
site: Restrict search to a specific site (e.g. "mp.weixin.qq.com").
|
|
||||||
recency: Time filter: "week", "month", "semiyear", "year" (default: "year").
|
|
||||||
edition: "standard" (default) or "lite".
|
|
||||||
safe_search: bool, default False.
|
|
||||||
"""
|
|
||||||
if not self.api_key:
|
if not self.api_key:
|
||||||
return [
|
return [
|
||||||
SearchResult(
|
SearchResult(
|
||||||
@@ -100,19 +90,14 @@ class BaiduQianFanEngine(SearchEngine):
|
|||||||
)
|
)
|
||||||
]
|
]
|
||||||
|
|
||||||
# Build resource_type_filter based on max_results
|
|
||||||
top_k = min(max_results, 50)
|
top_k = min(max_results, 50)
|
||||||
resource_type_filter = [
|
resource_type_filter = [{"type": "web", "top_k": top_k}]
|
||||||
{"type": "web", "top_k": top_k},
|
|
||||||
]
|
|
||||||
|
|
||||||
# Build search_filter if site is specified
|
search_filter: dict[str, Any] = {}
|
||||||
search_filter = {}
|
if kwargs.get("site"):
|
||||||
site = kwargs.get("site")
|
search_filter["match"] = {"site": [kwargs["site"]]}
|
||||||
if site:
|
|
||||||
search_filter["match"] = {"site": [site]}
|
|
||||||
|
|
||||||
request_body = {
|
request_body: dict[str, Any] = {
|
||||||
"messages": [{"content": query, "role": "user"}],
|
"messages": [{"content": query, "role": "user"}],
|
||||||
"edition": kwargs.get("edition", "standard"),
|
"edition": kwargs.get("edition", "standard"),
|
||||||
"search_source": "baidu_search_v2",
|
"search_source": "baidu_search_v2",
|
||||||
@@ -134,38 +119,24 @@ class BaiduQianFanEngine(SearchEngine):
|
|||||||
logger.error(f"Baidu Qianfan API error: {data.get('message', '')}")
|
logger.error(f"Baidu Qianfan API error: {data.get('message', '')}")
|
||||||
return []
|
return []
|
||||||
|
|
||||||
references = data.get("references", [])
|
|
||||||
results = []
|
results = []
|
||||||
for i, ref in enumerate(references[:max_results]):
|
for i, ref in enumerate(data.get("references", [])[:max_results]):
|
||||||
results.append(
|
results.append(SearchResult(
|
||||||
SearchResult(
|
title=ref.get("title", ""),
|
||||||
title=ref.get("title", ""),
|
url=ref.get("url", ""),
|
||||||
url=ref.get("url", ""),
|
snippet=ref.get("snippet", ""),
|
||||||
snippet=ref.get("snippet", ""),
|
source=self.name,
|
||||||
source=self.name,
|
rank=i + 1,
|
||||||
rank=i + 1,
|
category="web",
|
||||||
category="web",
|
extra={k: ref[k] for k in ("site", "date", "type", "source") if k in ref},
|
||||||
extra={
|
))
|
||||||
k: ref[k]
|
|
||||||
for k in ("site", "date", "type", "source")
|
|
||||||
if k in ref
|
|
||||||
},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return results
|
return results
|
||||||
|
|
||||||
except httpx.HTTPStatusError as e:
|
except httpx.HTTPStatusError as e:
|
||||||
logger.error(f"Baidu Qianfan API HTTP error: {e.response.status_code}")
|
logger.error(f"Baidu Qianfan HTTP {e.response.status_code}: {e}")
|
||||||
if e.response.status_code == 401:
|
|
||||||
logger.error("API Key 无效或未授权,请检查 BAIDU_API_KEY")
|
|
||||||
elif e.response.status_code == 429:
|
|
||||||
logger.error("API 配额不足")
|
|
||||||
return []
|
return []
|
||||||
except httpx.RequestError as e:
|
except httpx.RequestError as e:
|
||||||
logger.error(f"Baidu Qianfan API request failed: {e}")
|
logger.error(f"Baidu Qianfan request failed: {e}")
|
||||||
return []
|
|
||||||
except json.JSONDecodeError as e:
|
|
||||||
logger.error(f"Baidu Qianfan API response parse failed: {e}")
|
|
||||||
return []
|
return []
|
||||||
|
|
||||||
async def close(self):
|
async def close(self):
|
||||||
|
|||||||
@@ -0,0 +1,69 @@
|
|||||||
|
"""Baidu HTML scraping engine — free, fastest in China, best Chinese results."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from urllib.parse import quote_plus
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
|
||||||
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
HEADERS = {
|
||||||
|
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||||
|
"(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
|
||||||
|
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||||
|
"Accept-Language": "zh-CN,zh;q=0.9",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class BaiduScraper(BaseEngine):
|
||||||
|
"""Baidu web scraping — fastest in China, best Chinese content."""
|
||||||
|
|
||||||
|
name = "baidu"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
**kwargs) -> list[SearchResult]:
|
||||||
|
t0 = time.time()
|
||||||
|
url = f"https://www.baidu.com/s?wd={quote_plus(query)}&rn={max_results}"
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
|
||||||
|
resp = await client.get(url, headers=HEADERS)
|
||||||
|
resp.raise_for_status()
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
results = self._parse(resp.text, max_results)
|
||||||
|
logger.info(f"[baidu] {len(results)} results in {elapsed:.2f}s")
|
||||||
|
return results
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[baidu] failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse(self, html: str, max_results: int) -> list[SearchResult]:
|
||||||
|
soup = BeautifulSoup(html, "html.parser")
|
||||||
|
results = []
|
||||||
|
|
||||||
|
for item in soup.select("div.result, div.c-container"):
|
||||||
|
title_el = item.select_one("h3 a")
|
||||||
|
if not title_el:
|
||||||
|
continue
|
||||||
|
title = title_el.get_text(strip=True)
|
||||||
|
href = title_el.get("href", "")
|
||||||
|
snippet_el = item.select_one(
|
||||||
|
"span.content-right_8Zs40, div.c-abstract, "
|
||||||
|
"span[class*='content'], div[class*='abstract']"
|
||||||
|
)
|
||||||
|
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
||||||
|
|
||||||
|
if title and href:
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title, url=href, snippet=snippet, source="baidu",
|
||||||
|
))
|
||||||
|
if len(results) >= max_results:
|
||||||
|
break
|
||||||
|
|
||||||
|
return results
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
"""Base engine class and shared data models."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
# Re-export SearchResult/BaseEngine from scripts.core.base for engine code compatibility
|
||||||
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
|
||||||
|
__all__ = ["BaseEngine", "SearchResult"]
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
"""Bing HTML scraping engine — free, no API key."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from urllib.parse import quote_plus
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
|
||||||
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
HEADERS = {
|
||||||
|
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||||
|
"(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
|
||||||
|
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||||
|
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class BingScraper(BaseEngine):
|
||||||
|
"""Bing web scraping — fast, good English results."""
|
||||||
|
|
||||||
|
name = "bing"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
region: str = "global", **kwargs) -> list[SearchResult]:
|
||||||
|
t0 = time.time()
|
||||||
|
url = f"https://cn.bing.com/search?q={quote_plus(query)}&setlang=zh-CN"
|
||||||
|
if region == "global":
|
||||||
|
url = f"https://www.bing.com/search?q={quote_plus(query)}"
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
|
||||||
|
resp = await client.get(url, headers=HEADERS)
|
||||||
|
resp.raise_for_status()
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
results = self._parse(resp.text, max_results)
|
||||||
|
logger.info(f"[bing] {len(results)} results in {elapsed:.2f}s")
|
||||||
|
return results
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[bing] failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse(self, html: str, max_results: int) -> list[SearchResult]:
|
||||||
|
soup = BeautifulSoup(html, "html.parser")
|
||||||
|
results = []
|
||||||
|
|
||||||
|
for item in soup.select("li.b_algo"):
|
||||||
|
title_el = item.select_one("h2 a")
|
||||||
|
if not title_el:
|
||||||
|
continue
|
||||||
|
title = title_el.get_text(strip=True)
|
||||||
|
href = title_el.get("href", "")
|
||||||
|
snippet_el = item.select_one("p, .b_caption p")
|
||||||
|
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
||||||
|
|
||||||
|
if title and href:
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title, url=href, snippet=snippet, source="bing",
|
||||||
|
))
|
||||||
|
if len(results) >= max_results:
|
||||||
|
break
|
||||||
|
|
||||||
|
return results
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
"""Brave Search HTML scraping engine — free, privacy-focused."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from urllib.parse import quote_plus
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
|
||||||
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
HEADERS = {
|
||||||
|
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||||
|
"(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
|
||||||
|
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||||
|
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class BraveScraper(BaseEngine):
|
||||||
|
"""Brave Search scraping — privacy-focused, good results."""
|
||||||
|
|
||||||
|
name = "brave"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
**kwargs) -> list[SearchResult]:
|
||||||
|
t0 = time.time()
|
||||||
|
url = f"https://search.brave.com/search?q={quote_plus(query)}&source=web"
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
|
||||||
|
resp = await client.get(url, headers=HEADERS)
|
||||||
|
resp.raise_for_status()
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
results = self._parse(resp.text, max_results)
|
||||||
|
logger.info(f"[brave] {len(results)} results in {elapsed:.2f}s")
|
||||||
|
return results
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[brave] failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse(self, html: str, max_results: int) -> list[SearchResult]:
|
||||||
|
soup = BeautifulSoup(html, "html.parser")
|
||||||
|
results = []
|
||||||
|
|
||||||
|
# Brave uses div.snippet or article elements
|
||||||
|
for item in soup.select("div.snippet, div.result"):
|
||||||
|
title_el = item.select_one("a.result-header, a.snippet-title")
|
||||||
|
if not title_el:
|
||||||
|
title_el = item.select_one("a")
|
||||||
|
if not title_el:
|
||||||
|
continue
|
||||||
|
title = title_el.get_text(strip=True)
|
||||||
|
href = title_el.get("href", "")
|
||||||
|
snippet_el = item.select_one("p.snippet-description, div.snippet-description")
|
||||||
|
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
||||||
|
|
||||||
|
if title and href and href.startswith("http"):
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title, url=href, snippet=snippet, source="brave",
|
||||||
|
))
|
||||||
|
if len(results) >= max_results:
|
||||||
|
break
|
||||||
|
|
||||||
|
return results
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
"""GitHub code search engine — uses GitHub API (no key needed for basic search)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from urllib.parse import quote_plus
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
HEADERS = {
|
||||||
|
"Accept": "application/vnd.github.v3+json",
|
||||||
|
"User-Agent": "web-search-skills/1.0",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class GitHubEngine(BaseEngine):
|
||||||
|
"""GitHub code & repo search via public API."""
|
||||||
|
|
||||||
|
name = "github"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
search_type: str = "repositories", **kwargs) -> list[SearchResult]:
|
||||||
|
"""search_type: 'repositories' | 'code' | 'issues'"""
|
||||||
|
t0 = time.time()
|
||||||
|
endpoint = f"https://api.github.com/search/{search_type}"
|
||||||
|
params = {"q": query, "per_page": min(max_results, 30), "sort": "stars"}
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
|
||||||
|
resp = await client.get(endpoint, headers=HEADERS, params=params)
|
||||||
|
resp.raise_for_status()
|
||||||
|
data = resp.json()
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
|
||||||
|
results = []
|
||||||
|
for item in data.get("items", [])[:max_results]:
|
||||||
|
if search_type == "repositories":
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=item.get("full_name", ""),
|
||||||
|
url=item.get("html_url", ""),
|
||||||
|
snippet=item.get("description", "") or "",
|
||||||
|
source="github",
|
||||||
|
category="github",
|
||||||
|
extra={
|
||||||
|
"stars": item.get("stargazers_count", 0),
|
||||||
|
"language": item.get("language", ""),
|
||||||
|
"updated": item.get("updated_at", ""),
|
||||||
|
},
|
||||||
|
))
|
||||||
|
elif search_type == "code":
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=item.get("name", ""),
|
||||||
|
url=item.get("html_url", ""),
|
||||||
|
snippet=item.get("text_matches", [{}])[0].get("fragment", "")
|
||||||
|
if item.get("text_matches") else "",
|
||||||
|
source="github",
|
||||||
|
category="github",
|
||||||
|
extra={
|
||||||
|
"repo": item.get("repository", {}).get("full_name", ""),
|
||||||
|
"language": item.get("language", ""),
|
||||||
|
},
|
||||||
|
))
|
||||||
|
elif search_type == "issues":
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=item.get("title", ""),
|
||||||
|
url=item.get("html_url", ""),
|
||||||
|
snippet=item.get("body", "")[:300] if item.get("body") else "",
|
||||||
|
source="github",
|
||||||
|
category="github",
|
||||||
|
extra={
|
||||||
|
"state": item.get("state", ""),
|
||||||
|
"repo": item.get("repository_url", "").split("/")[-2:]
|
||||||
|
if item.get("repository_url") else [],
|
||||||
|
},
|
||||||
|
))
|
||||||
|
|
||||||
|
logger.info(f"[github] {len(results)} results in {elapsed:.2f}s")
|
||||||
|
return results
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[github] failed: {e}")
|
||||||
|
return []
|
||||||
@@ -0,0 +1,194 @@
|
|||||||
|
"""News search — multi-source aggregation (Bing News + RSS)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from urllib.parse import quote_plus
|
||||||
|
from xml.etree import ElementTree as ET
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
_HEADERS = {
|
||||||
|
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||||
|
"(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
|
||||||
|
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Free RSS feeds — no API key needed
|
||||||
|
_RSS_FEEDS = {
|
||||||
|
# Chinese tech/business
|
||||||
|
"36kr": "https://36kr.com/feed",
|
||||||
|
"少数派": "https://sspai.com/feed",
|
||||||
|
"IT之家": "https://www.ithome.com/rss/",
|
||||||
|
"钛媒体": "https://www.tmtpost.com/rss.xml",
|
||||||
|
"Solidot": "https://www.solidot.org/index.rss",
|
||||||
|
"InfoQ CN": "https://www.infoq.cn/feed",
|
||||||
|
"OSChina": "https://www.oschina.net/news/rss",
|
||||||
|
# English tech
|
||||||
|
"HackerNews": "https://hnrss.org/best?count=20",
|
||||||
|
"TechCrunch": "https://techcrunch.com/feed/",
|
||||||
|
"ArsTechnica": "https://feeds.arstechnica.com/arstechnica/index",
|
||||||
|
"TheVerge": "https://www.theverge.com/rss/index.xml",
|
||||||
|
"MIT Tech Review": "https://www.technologyreview.com/feed/",
|
||||||
|
"Wired": "https://www.wired.com/feed/rss",
|
||||||
|
"Lobste.rs": "https://lobste.rs/rss",
|
||||||
|
# English general
|
||||||
|
"ChinaDaily": "https://www.chinadaily.com.cn/rss/china_rss.xml",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class BingNewsEngine(BaseEngine):
|
||||||
|
"""Bing News scraping — works for both CN and global news."""
|
||||||
|
|
||||||
|
name = "bing_news"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
region: str = "cn", **kwargs) -> list[SearchResult]:
|
||||||
|
t0 = time.time()
|
||||||
|
base = "cn.bing.com" if region == "cn" else "www.bing.com"
|
||||||
|
url = f"https://{base}/news/search?q={quote_plus(query)}&setlang=zh-CN"
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
|
||||||
|
resp = await client.get(url, headers=_HEADERS)
|
||||||
|
resp.raise_for_status()
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
results = self._parse(resp.text, max_results)
|
||||||
|
logger.info(f"[bing_news] {len(results)} results in {elapsed:.2f}s")
|
||||||
|
return results
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[bing_news] failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse(self, html: str, max_results: int) -> list[SearchResult]:
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
soup = BeautifulSoup(html, "html.parser")
|
||||||
|
results = []
|
||||||
|
for item in soup.select("a.title"):
|
||||||
|
title = item.get_text(strip=True)
|
||||||
|
href = item.get("href", "")
|
||||||
|
parent = item.find_parent("div", class_=True)
|
||||||
|
snippet = ""
|
||||||
|
if parent:
|
||||||
|
snippet_el = parent.select_one("div.snippet")
|
||||||
|
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
||||||
|
if title and href and href.startswith("http"):
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title, url=href, snippet=snippet,
|
||||||
|
source="bing_news", category="news",
|
||||||
|
))
|
||||||
|
if len(results) >= max_results:
|
||||||
|
break
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
class RSSNewsEngine(BaseEngine):
|
||||||
|
"""RSS feed aggregation — multi-source news from curated feeds."""
|
||||||
|
|
||||||
|
name = "rss_news"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
**kwargs) -> list[SearchResult]:
|
||||||
|
t0 = time.time()
|
||||||
|
all_results = []
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
tasks = [self._fetch_feed(name, url, query)
|
||||||
|
for name, url in _RSS_FEEDS.items()]
|
||||||
|
feeds = await asyncio.gather(*tasks, return_exceptions=True)
|
||||||
|
|
||||||
|
for feed_results in feeds:
|
||||||
|
if isinstance(feed_results, list):
|
||||||
|
all_results.extend(feed_results)
|
||||||
|
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
logger.info(f"[rss_news] {len(all_results)} results in {elapsed:.2f}s")
|
||||||
|
return all_results[:max_results]
|
||||||
|
|
||||||
|
async def _fetch_feed(self, name: str, url: str, query: str) -> list[SearchResult]:
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
|
||||||
|
resp = await client.get(url, headers=_HEADERS)
|
||||||
|
resp.raise_for_status()
|
||||||
|
return self._parse_feed(name, resp.text, query)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[rss_news:{name}] failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse_feed(self, name: str, xml_text: str, query: str) -> list[SearchResult]:
|
||||||
|
try:
|
||||||
|
root = ET.fromstring(xml_text)
|
||||||
|
except ET.ParseError:
|
||||||
|
return []
|
||||||
|
|
||||||
|
ns = {"atom": "http://www.w3.org/2005/Atom"}
|
||||||
|
items = root.findall(".//item") or root.findall(".//atom:entry", ns)
|
||||||
|
|
||||||
|
q_lower = query.lower()
|
||||||
|
q_words = [w for w in q_lower.split() if len(w) > 1]
|
||||||
|
|
||||||
|
results = []
|
||||||
|
for item in items[:50]:
|
||||||
|
title = (item.findtext("title") or
|
||||||
|
item.findtext("atom:title", namespaces=ns) or "")
|
||||||
|
link = item.findtext("link") or ""
|
||||||
|
if not link:
|
||||||
|
link_el = item.find("atom:link", ns)
|
||||||
|
link = link_el.get("href", "") if link_el is not None else ""
|
||||||
|
desc = (item.findtext("description") or
|
||||||
|
item.findtext("atom:summary", namespaces=ns) or "")
|
||||||
|
desc = re.sub(r'<[^>]+>', '', desc)
|
||||||
|
|
||||||
|
text_lower = f"{title} {desc}".lower()
|
||||||
|
if q_words and not any(w in text_lower for w in q_words):
|
||||||
|
continue # Skip items with no query keyword match
|
||||||
|
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title.strip(),
|
||||||
|
url=link.strip(),
|
||||||
|
snippet=desc[:300].strip(),
|
||||||
|
source=f"rss_{name}",
|
||||||
|
category="news",
|
||||||
|
))
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
class NewsSearch(BaseEngine):
|
||||||
|
"""Unified news search — Bing News + RSS feeds in parallel."""
|
||||||
|
|
||||||
|
name = "news"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
region: str = "cn", **kwargs) -> list[SearchResult]:
|
||||||
|
import asyncio
|
||||||
|
bing = BingNewsEngine(timeout=self.timeout)
|
||||||
|
rss = RSSNewsEngine(timeout=8.0)
|
||||||
|
|
||||||
|
bing_results, rss_results = await asyncio.gather(
|
||||||
|
bing.search(query, max_results=max_results, region=region),
|
||||||
|
rss.search(query, max_results=max_results),
|
||||||
|
return_exceptions=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
all_results = []
|
||||||
|
if isinstance(bing_results, list):
|
||||||
|
all_results.extend(bing_results)
|
||||||
|
if isinstance(rss_results, list):
|
||||||
|
all_results.extend(rss_results)
|
||||||
|
|
||||||
|
# Deduplicate by URL
|
||||||
|
seen = set()
|
||||||
|
deduped = []
|
||||||
|
for r in all_results:
|
||||||
|
if r.url not in seen:
|
||||||
|
seen.add(r.url)
|
||||||
|
deduped.append(r)
|
||||||
|
|
||||||
|
return deduped[:max_results]
|
||||||
@@ -0,0 +1,199 @@
|
|||||||
|
"""Parallel search orchestrator — runs multiple engines concurrently."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
from scripts.engines.bing_scraper import BingScraper
|
||||||
|
from scripts.engines.sogou_scraper import SogouScraper
|
||||||
|
from scripts.engines.baidu_scraper import BaiduScraper
|
||||||
|
from scripts.engines.search360_scraper import Search360Scraper
|
||||||
|
from scripts.engines.news_engine import NewsSearch
|
||||||
|
from scripts.engines.academic_engine import AcademicSearch
|
||||||
|
from scripts.engines.github_engine import GitHubEngine
|
||||||
|
from scripts.core.execution_log import execution_log
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class SearchStats:
|
||||||
|
"""Search statistics."""
|
||||||
|
total_engines: int = 0
|
||||||
|
successful_engines: int = 0
|
||||||
|
failed_engines: list[str] = field(default_factory=list)
|
||||||
|
total_results: int = 0
|
||||||
|
deduplicated: int = 0
|
||||||
|
duration_ms: int = 0
|
||||||
|
engine_timings: dict[str, float] = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class ParallelSearch:
|
||||||
|
"""Run multiple engines in parallel and merge results."""
|
||||||
|
|
||||||
|
# Default engine sets
|
||||||
|
GENERAL_ENGINES = ["bing", "sogou", "baidu", "360"]
|
||||||
|
VERTICAL_ENGINES = ["news", "academic", "github"]
|
||||||
|
|
||||||
|
def __init__(self, timeout: float = 10.0):
|
||||||
|
self.timeout = timeout
|
||||||
|
self._engines: dict[str, BaseEngine] = {
|
||||||
|
"bing": BingScraper(timeout=timeout),
|
||||||
|
"sogou": SogouScraper(timeout=timeout),
|
||||||
|
"baidu": BaiduScraper(timeout=timeout),
|
||||||
|
"360": Search360Scraper(timeout=timeout),
|
||||||
|
"news": NewsSearch(timeout=timeout),
|
||||||
|
"academic": AcademicSearch(timeout=timeout),
|
||||||
|
"github": GitHubEngine(timeout=timeout),
|
||||||
|
}
|
||||||
|
|
||||||
|
async def search(
|
||||||
|
self,
|
||||||
|
query: str,
|
||||||
|
engines: list[str] | None = None,
|
||||||
|
max_results: int = 10,
|
||||||
|
region: str = "cn",
|
||||||
|
**kwargs,
|
||||||
|
) -> tuple[list[SearchResult], SearchStats]:
|
||||||
|
"""
|
||||||
|
Search across specified engines in parallel.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
query: Search query
|
||||||
|
engines: List of engine names to use (default: all general)
|
||||||
|
max_results: Max results per engine
|
||||||
|
region: 'cn' or 'global'
|
||||||
|
**kwargs: Extra params passed to engines
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
(results, stats)
|
||||||
|
"""
|
||||||
|
if engines is None:
|
||||||
|
engines = self.GENERAL_ENGINES.copy()
|
||||||
|
|
||||||
|
t0 = time.time()
|
||||||
|
stats = SearchStats(total_engines=len(engines))
|
||||||
|
|
||||||
|
# Build tasks
|
||||||
|
tasks = {}
|
||||||
|
for name in engines:
|
||||||
|
engine = self._engines.get(name)
|
||||||
|
if engine is None:
|
||||||
|
logger.warning(f"Unknown engine: {name}")
|
||||||
|
continue
|
||||||
|
kwargs_extra = dict(kwargs)
|
||||||
|
if name in ("bing", "bing_news"):
|
||||||
|
kwargs_extra["region"] = region
|
||||||
|
tasks[name] = self._run_engine(engine, query, max_results, kwargs_extra)
|
||||||
|
|
||||||
|
# Run all in parallel
|
||||||
|
results_map = await asyncio.gather(
|
||||||
|
*tasks.values(), return_exceptions=True
|
||||||
|
)
|
||||||
|
|
||||||
|
all_results = []
|
||||||
|
for name, result in zip(tasks.keys(), results_map):
|
||||||
|
if isinstance(result, Exception):
|
||||||
|
stats.failed_engines.append(name)
|
||||||
|
logger.warning(f"[{name}] exception: {result}")
|
||||||
|
# 记录异常到 execution_log
|
||||||
|
execution_log.record(execution_log._logs[-1].__class__(
|
||||||
|
engine=name,
|
||||||
|
operation="search",
|
||||||
|
success=False,
|
||||||
|
error=str(result),
|
||||||
|
error_type=type(result).__name__,
|
||||||
|
duration_ms=int((time.time() - t0) * 1000),
|
||||||
|
))
|
||||||
|
elif isinstance(result, tuple):
|
||||||
|
engine_results, elapsed = result
|
||||||
|
stats.engine_timings[name] = elapsed
|
||||||
|
if engine_results:
|
||||||
|
stats.successful_engines += 1
|
||||||
|
all_results.extend(engine_results)
|
||||||
|
else:
|
||||||
|
stats.failed_engines.append(name)
|
||||||
|
else:
|
||||||
|
stats.failed_engines.append(name)
|
||||||
|
|
||||||
|
# Deduplicate by URL
|
||||||
|
seen_urls = set()
|
||||||
|
deduped = []
|
||||||
|
for r in all_results:
|
||||||
|
key = r.url or r.title # fallback to title if no URL
|
||||||
|
if key not in seen_urls:
|
||||||
|
seen_urls.add(key)
|
||||||
|
deduped.append(r)
|
||||||
|
|
||||||
|
stats.total_results = len(all_results)
|
||||||
|
stats.deduplicated = len(deduped)
|
||||||
|
stats.duration_ms = int((time.time() - t0) * 1000)
|
||||||
|
|
||||||
|
return deduped, stats
|
||||||
|
|
||||||
|
async def _run_engine(
|
||||||
|
self, engine: BaseEngine, query: str, max_results: int, kwargs: dict
|
||||||
|
) -> tuple[list[SearchResult], float]:
|
||||||
|
"""Run a single engine with timing."""
|
||||||
|
with execution_log.track(engine.name, "search", query=query) as ctx:
|
||||||
|
t0 = time.time()
|
||||||
|
try:
|
||||||
|
results = await engine.search(
|
||||||
|
query, max_results=max_results, **kwargs
|
||||||
|
)
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
ctx.result_count = len(results)
|
||||||
|
return results, elapsed
|
||||||
|
except Exception as e:
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
ctx.success = False
|
||||||
|
ctx.error = str(e)
|
||||||
|
ctx.error_type = type(e).__name__
|
||||||
|
raise
|
||||||
|
|
||||||
|
async def search_vertical(
|
||||||
|
self,
|
||||||
|
query: str,
|
||||||
|
vertical: str,
|
||||||
|
max_results: int = 10,
|
||||||
|
**kwargs,
|
||||||
|
) -> tuple[list[SearchResult], SearchStats]:
|
||||||
|
"""Search a specific vertical (news/academic/github)."""
|
||||||
|
if vertical == "github":
|
||||||
|
# GitHub supports sub-types
|
||||||
|
search_type = kwargs.pop("search_type", "repositories")
|
||||||
|
engine = self._engines["github"]
|
||||||
|
with execution_log.track("github", "search",
|
||||||
|
search_type=search_type) as ctx:
|
||||||
|
t0 = time.time()
|
||||||
|
results = await engine.search(
|
||||||
|
query, max_results=max_results,
|
||||||
|
search_type=search_type, **kwargs
|
||||||
|
)
|
||||||
|
ctx.result_count = len(results)
|
||||||
|
stats = SearchStats(
|
||||||
|
total_engines=1,
|
||||||
|
successful_engines=1 if results else 0,
|
||||||
|
total_results=len(results),
|
||||||
|
duration_ms=int((time.time() - t0) * 1000),
|
||||||
|
)
|
||||||
|
return results, stats
|
||||||
|
else:
|
||||||
|
return await self.search(
|
||||||
|
query, engines=[vertical], max_results=max_results, **kwargs
|
||||||
|
)
|
||||||
|
|
||||||
|
async def health(self) -> dict[str, dict]:
|
||||||
|
"""Check health of all engines."""
|
||||||
|
results = {}
|
||||||
|
for name, engine in self._engines.items():
|
||||||
|
try:
|
||||||
|
results[name] = await engine.health()
|
||||||
|
except Exception as e:
|
||||||
|
results[name] = {"name": name, "status": "error", "error": str(e)}
|
||||||
|
return results
|
||||||
@@ -1,80 +1,114 @@
|
|||||||
"""
|
"""Parsing-based search engines — DuckDuckGo, Brave (fallback HTML scraper)."""
|
||||||
Parsing-based search engines (e.g. DuckDuckGo HTML scraper).
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
from typing import Optional
|
from urllib.parse import quote_plus
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
from bs4 import BeautifulSoup
|
from bs4 import BeautifulSoup
|
||||||
|
|
||||||
from scripts.core.base import SearchEngine, SearchResult
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
_HEADERS = {
|
||||||
|
"User-Agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
||||||
|
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||||
|
"Chrome/125.0.0.0 Safari/537.36"),
|
||||||
|
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||||
|
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
|
||||||
|
}
|
||||||
|
|
||||||
class DuckDuckGoEngine(SearchEngine):
|
|
||||||
"""DuckDuckGo HTML search engine (no API key required)."""
|
|
||||||
|
|
||||||
SEARCH_URL = "https://html.duckduckgo.com/html/"
|
class DuckDuckGoParser(BaseEngine):
|
||||||
|
"""DuckDuckGo HTML scraper (no API key needed)."""
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
name = "duckduckgo"
|
||||||
super().__init__("DuckDuckGo", config)
|
|
||||||
timeout = (config or {}).get("timeout", 15)
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=timeout,
|
|
||||||
headers={
|
|
||||||
"User-Agent": (config or {}).get(
|
|
||||||
"user_agent",
|
|
||||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
|
||||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
|
||||||
"Chrome/120.0.0.0 Safari/537.36",
|
|
||||||
),
|
|
||||||
},
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
def build_url(self, query: str, **kwargs) -> str:
|
async def search(self, query: str, max_results: int = 10,
|
||||||
"""Build DuckDuckGo search URL."""
|
**kwargs) -> list[SearchResult]:
|
||||||
from urllib.parse import quote_plus
|
import time
|
||||||
return f"https://duckduckgo.com/?q={quote_plus(query)}"
|
t0 = time.time()
|
||||||
|
url = f"https://html.duckduckgo.com/html/?q={quote_plus(query)}"
|
||||||
|
|
||||||
async def search(
|
|
||||||
self, query: str, max_results: int = 10, **kwargs
|
|
||||||
) -> list[SearchResult]:
|
|
||||||
"""Search DuckDuckGo and parse HTML results."""
|
|
||||||
params = {"q": query}
|
|
||||||
try:
|
try:
|
||||||
response = await self._client.post(self.SEARCH_URL, data=params)
|
async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
|
||||||
response.raise_for_status()
|
resp = await client.get(url, headers=_HEADERS)
|
||||||
return self._parse_results(response.text, max_results)
|
resp.raise_for_status()
|
||||||
except httpx.HTTPError as e:
|
elapsed = time.time() - t0
|
||||||
logger.error(f"DuckDuckGo search failed: {e}")
|
results = self._parse(resp.text, max_results)
|
||||||
|
logger.info(f"[duckduckgo] {len(results)} results in {elapsed:.2f}s")
|
||||||
|
return results
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[duckduckgo] failed: {e}")
|
||||||
return []
|
return []
|
||||||
|
|
||||||
def _parse_results(self, html: str, max_results: int) -> list[SearchResult]:
|
def _parse(self, html: str, max_results: int) -> list[SearchResult]:
|
||||||
"""Parse DuckDuckGo HTML search results."""
|
soup = BeautifulSoup(html, "html.parser")
|
||||||
soup = BeautifulSoup(html, "lxml")
|
|
||||||
results = []
|
results = []
|
||||||
|
|
||||||
for i, result in enumerate(soup.select(".result")[:max_results]):
|
for result in soup.select(".result")[:max_results]:
|
||||||
title_el = result.select_one(".result__title a")
|
title_el = result.select_one(".result__title a")
|
||||||
snippet_el = result.select_one(".result__snippet")
|
snippet_el = result.select_one(".result__snippet")
|
||||||
if title_el:
|
if not title_el:
|
||||||
results.append(
|
continue
|
||||||
SearchResult(
|
results.append(SearchResult(
|
||||||
title=title_el.get_text(strip=True),
|
title=title_el.get_text(strip=True),
|
||||||
url=title_el.get("href", ""),
|
url=title_el.get("href", ""),
|
||||||
snippet=snippet_el.get_text(strip=True)
|
snippet=snippet_el.get_text(strip=True) if snippet_el else "",
|
||||||
if snippet_el else "",
|
source="duckduckgo",
|
||||||
source=self.name,
|
category="web",
|
||||||
rank=i + 1,
|
))
|
||||||
category="web",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return results
|
return results
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
class BraveParser(BaseEngine):
|
||||||
|
"""Brave Search fallback HTML scraper (no API key needed)."""
|
||||||
|
|
||||||
|
name = "brave"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
**kwargs) -> list[SearchResult]:
|
||||||
|
import time
|
||||||
|
t0 = time.time()
|
||||||
|
url = f"https://search.brave.com/search?q={quote_plus(query)}&source=web"
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
|
||||||
|
resp = await client.get(url, headers=_HEADERS)
|
||||||
|
resp.raise_for_status()
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
results = self._parse(resp.text, max_results)
|
||||||
|
logger.info(f"[brave] {len(results)} results in {elapsed:.2f}s")
|
||||||
|
return results
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[brave] failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse(self, html: str, max_results: int) -> list[SearchResult]:
|
||||||
|
soup = BeautifulSoup(html, "html.parser")
|
||||||
|
results = []
|
||||||
|
|
||||||
|
for item in soup.select("div.snippet, div.result")[:max_results]:
|
||||||
|
title_el = item.select_one("a.result-header, a.snippet-title, a")
|
||||||
|
if not title_el:
|
||||||
|
continue
|
||||||
|
title = title_el.get_text(strip=True)
|
||||||
|
href = title_el.get("href", "")
|
||||||
|
snippet_el = item.select_one("p.snippet-description, div.snippet-description")
|
||||||
|
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
||||||
|
|
||||||
|
if title and href and href.startswith("http"):
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title, url=href, snippet=snippet,
|
||||||
|
source="brave", category="web",
|
||||||
|
))
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
# Backwards-compatibility aliases
|
||||||
|
DuckDuckGoEngine = DuckDuckGoParser
|
||||||
|
BraveScraper = BraveParser
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
"""360 Search HTML scraping engine — free, good Chinese results."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from urllib.parse import quote_plus
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
|
||||||
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
HEADERS = {
|
||||||
|
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||||
|
"(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
|
||||||
|
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||||
|
"Accept-Language": "zh-CN,zh;q=0.9",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class Search360Scraper(BaseEngine):
|
||||||
|
"""360 Search scraping — good Chinese results, fast in China."""
|
||||||
|
|
||||||
|
name = "360"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
**kwargs) -> list[SearchResult]:
|
||||||
|
t0 = time.time()
|
||||||
|
url = f"https://www.so.com/s?q={quote_plus(query)}"
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
|
||||||
|
resp = await client.get(url, headers=HEADERS)
|
||||||
|
resp.raise_for_status()
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
results = self._parse(resp.text, max_results)
|
||||||
|
logger.info(f"[360] {len(results)} results in {elapsed:.2f}s")
|
||||||
|
return results
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[360] failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse(self, html: str, max_results: int) -> list[SearchResult]:
|
||||||
|
soup = BeautifulSoup(html, "html.parser")
|
||||||
|
results = []
|
||||||
|
|
||||||
|
for item in soup.select("li.res-list"):
|
||||||
|
title_el = item.select_one("h3 a")
|
||||||
|
if not title_el:
|
||||||
|
continue
|
||||||
|
title = title_el.get_text(strip=True)
|
||||||
|
href = title_el.get("href", "")
|
||||||
|
snippet_el = item.select_one("p.res-desc, .res-rich")
|
||||||
|
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
||||||
|
|
||||||
|
if title and href:
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title, url=href, snippet=snippet, source="360",
|
||||||
|
))
|
||||||
|
if len(results) >= max_results:
|
||||||
|
break
|
||||||
|
|
||||||
|
return results
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
"""Sogou HTML scraping engine — free, good Chinese results."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
import logging
|
||||||
|
from urllib.parse import quote_plus
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
|
||||||
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
HEADERS = {
|
||||||
|
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||||
|
"(KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
|
||||||
|
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||||
|
"Accept-Language": "zh-CN,zh;q=0.9",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class SogouScraper(BaseEngine):
|
||||||
|
"""Sogou web scraping — best for Chinese content."""
|
||||||
|
|
||||||
|
name = "sogou"
|
||||||
|
|
||||||
|
async def search(self, query: str, max_results: int = 10,
|
||||||
|
**kwargs) -> list[SearchResult]:
|
||||||
|
t0 = time.time()
|
||||||
|
url = f"https://www.sogou.com/web?query={quote_plus(query)}"
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
|
||||||
|
resp = await client.get(url, headers=HEADERS)
|
||||||
|
resp.raise_for_status()
|
||||||
|
elapsed = time.time() - t0
|
||||||
|
results = self._parse(resp.text, max_results)
|
||||||
|
logger.info(f"[sogou] {len(results)} results in {elapsed:.2f}s")
|
||||||
|
return results
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[sogou] failed: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def _parse(self, html: str, max_results: int) -> list[SearchResult]:
|
||||||
|
soup = BeautifulSoup(html, "html.parser")
|
||||||
|
results = []
|
||||||
|
|
||||||
|
for item in soup.select("div.results > div"):
|
||||||
|
title_el = item.select_one("h3 a, a.account")
|
||||||
|
if not title_el:
|
||||||
|
continue
|
||||||
|
title = title_el.get_text(strip=True)
|
||||||
|
href = title_el.get("href", "")
|
||||||
|
snippet_el = item.select_one("p.str_info, div.str-text, .star-wiki")
|
||||||
|
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
||||||
|
|
||||||
|
if title and href and not href.startswith("/"):
|
||||||
|
results.append(SearchResult(
|
||||||
|
title=title, url=href, snippet=snippet, source="sogou",
|
||||||
|
))
|
||||||
|
if len(results) >= max_results:
|
||||||
|
break
|
||||||
|
|
||||||
|
return results
|
||||||
@@ -1,31 +1,31 @@
|
|||||||
"""
|
"""
|
||||||
Generic URL-based search engine.
|
Generic URL-based search engine.
|
||||||
|
|
||||||
Builds search URLs from templates defined in engines.json
|
Builds search URLs from templates (config/engines.json)
|
||||||
and returns them as URL results.
|
and returns them as URL results.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import logging
|
|
||||||
from typing import Any, Optional
|
|
||||||
from urllib.parse import quote_plus
|
from urllib.parse import quote_plus
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
from scripts.core.base import SearchEngine, SearchResult
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
class UrlSearchEngine(SearchEngine):
|
class UrlSearchEngine(BaseEngine):
|
||||||
"""
|
"""
|
||||||
Generic search engine that builds URLs from templates.
|
Generic engine that builds search URLs from templates.
|
||||||
|
|
||||||
Reads URL templates from config (engines.json) and returns
|
Reads URL templates from config (engines.json) and returns
|
||||||
the constructed search URL as a SearchResult.
|
the constructed search URL as a SearchResult.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, name: str, engine_def: dict[str, Any]):
|
name: str = "url"
|
||||||
super().__init__(name, engine_def.get("config", {}))
|
|
||||||
|
def __init__(self, name: str, engine_def: dict[str, Any], timeout: float = 10.0):
|
||||||
|
super().__init__(timeout=timeout)
|
||||||
|
self.name = name
|
||||||
self.url_template = engine_def.get("url", "")
|
self.url_template = engine_def.get("url", "")
|
||||||
self.region = engine_def.get("region", "global")
|
self.region = engine_def.get("region", "global")
|
||||||
|
|
||||||
@@ -55,6 +55,3 @@ class UrlSearchEngine(SearchEngine):
|
|||||||
extra={"url": url, "region": self.region},
|
extra={"url": url, "region": self.region},
|
||||||
)
|
)
|
||||||
]
|
]
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
pass
|
|
||||||
+12
-23
@@ -1,25 +1,22 @@
|
|||||||
"""
|
"""
|
||||||
Pydantic 数据模型定义。
|
Data models — re-exports from scripts.core.base for backwards compatibility.
|
||||||
|
|
||||||
从 bot-search-mcp 移植,适配 web-search-skills 的数据结构。
|
The canonical source of truth is scripts.core.base.SearchResult.
|
||||||
提供类型安全的搜索数据模型。
|
This module exists so old import paths (e.g. from scripts.models import SearchQuery)
|
||||||
|
continue to work without changes.
|
||||||
依赖: pydantic (>=2.0)
|
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from datetime import datetime
|
|
||||||
from typing import Any, Optional
|
from typing import Any, Optional
|
||||||
|
|
||||||
|
|
||||||
# ============================================================
|
# Backwards-compatible thin wrappers (no functionality changes needed for now)
|
||||||
# 搜索结果模型(兼容 dataclass + pydantic 双模式)
|
|
||||||
# ============================================================
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class SearchQuery:
|
class SearchQuery:
|
||||||
"""搜索查询封装。"""
|
"""Search query wrapper."""
|
||||||
query: str
|
query: str
|
||||||
sources: Optional[list[str]] = None
|
sources: Optional[list[str]] = None
|
||||||
max_results: int = 10
|
max_results: int = 10
|
||||||
@@ -31,20 +28,16 @@ class SearchQuery:
|
|||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class SourceRef:
|
class SourceRef:
|
||||||
"""引用来源。"""
|
"""Reference to a result source."""
|
||||||
index: int
|
index: int
|
||||||
title: str
|
title: str
|
||||||
url: str
|
url: str
|
||||||
source: str
|
source: str
|
||||||
|
|
||||||
|
|
||||||
# ============================================================
|
|
||||||
# 性能计时模型
|
|
||||||
# ============================================================
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class EngineTiming:
|
class EngineTiming:
|
||||||
"""单个引擎耗时记录。"""
|
"""Per-engine timing record."""
|
||||||
engine_name: str
|
engine_name: str
|
||||||
success: bool
|
success: bool
|
||||||
duration_ms: int
|
duration_ms: int
|
||||||
@@ -54,7 +47,7 @@ class EngineTiming:
|
|||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class SearchMetadata:
|
class SearchMetadata:
|
||||||
"""搜索元数据(全链路计时)。"""
|
"""Search metadata (full-chain timing)."""
|
||||||
total_engines: int = 0
|
total_engines: int = 0
|
||||||
total_raw_results: int = 0
|
total_raw_results: int = 0
|
||||||
engines_used: list[str] = field(default_factory=list)
|
engines_used: list[str] = field(default_factory=list)
|
||||||
@@ -63,13 +56,9 @@ class SearchMetadata:
|
|||||||
timed_out_engines: list[str] = field(default_factory=list)
|
timed_out_engines: list[str] = field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
# ============================================================
|
|
||||||
# 最终响应
|
|
||||||
# ============================================================
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class SearchResponse:
|
class SearchResponse:
|
||||||
"""完整搜索响应。"""
|
"""Complete search response."""
|
||||||
query: str
|
query: str
|
||||||
results: list[Any] = field(default_factory=list)
|
results: list[Any] = field(default_factory=list)
|
||||||
sources: list[SourceRef] = field(default_factory=list)
|
sources: list[SourceRef] = field(default_factory=list)
|
||||||
|
|||||||
@@ -1 +0,0 @@
|
|||||||
"""News search engines."""
|
|
||||||
@@ -1,186 +0,0 @@
|
|||||||
"""
|
|
||||||
财联社 (cls.cn) news search engine.
|
|
||||||
|
|
||||||
Uses the public telegraph API (nodeapi/telegraphList) to fetch real-time
|
|
||||||
financial news without JS rendering. Falls back to search page URL for
|
|
||||||
keyword-based search queries.
|
|
||||||
|
|
||||||
API details (discovered via frontend reverse-engineering):
|
|
||||||
GET https://www.cls.cn/nodeapi/telegraphList
|
|
||||||
- Parameters: app=CailianpressWeb, os=web, sv=7.7.5, rn=N, last_time=timestamp
|
|
||||||
- Sign: SHA1(plaintext) → MD5(hex) (anti-scraping measure)
|
|
||||||
- Returns: {"data":{"roll_data":[{"ctime":..., "content":"..."}]}}
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import logging
|
|
||||||
import time
|
|
||||||
from typing import Optional
|
|
||||||
from urllib.parse import quote_plus
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
|
|
||||||
from scripts.core.base import SearchEngine, SearchResult
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
API_URL = "https://www.cls.cn/nodeapi/telegraphList"
|
|
||||||
SEARCH_URL = "https://www.cls.cn/searchPage"
|
|
||||||
API_APP = "CailianpressWeb"
|
|
||||||
API_OS = "web"
|
|
||||||
API_SV = "7.7.5"
|
|
||||||
|
|
||||||
|
|
||||||
class CLSNewsEngine(SearchEngine):
|
|
||||||
"""财联社 (cls.cn) news search engine."""
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
|
||||||
super().__init__("cls", config)
|
|
||||||
timeout = (config or {}).get("timeout", 15)
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=timeout,
|
|
||||||
headers={
|
|
||||||
"User-Agent": (
|
|
||||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
|
||||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
|
||||||
"Chrome/120.0.0.0 Safari/537.36"
|
|
||||||
),
|
|
||||||
"Referer": "https://www.cls.cn/telegraph",
|
|
||||||
},
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
def build_url(self, query: str, **kwargs) -> str:
|
|
||||||
return f"{SEARCH_URL}?keyword={quote_plus(query)}&type=all"
|
|
||||||
|
|
||||||
def _build_api_sign(self, last_time: int, rn: int) -> str:
|
|
||||||
"""Build SHA1 + MD5 double-hashed sign for API request."""
|
|
||||||
plain = (
|
|
||||||
f"app={API_APP}&last_time={last_time}"
|
|
||||||
f"&os={API_OS}&rn={rn}&sv={API_SV}"
|
|
||||||
)
|
|
||||||
sha1_hex = hashlib.sha1(plain.encode()).hexdigest()
|
|
||||||
return hashlib.md5(sha1_hex.encode()).hexdigest()
|
|
||||||
|
|
||||||
async def search(
|
|
||||||
self, query: str, max_results: int = 10, **kwargs
|
|
||||||
) -> list[SearchResult]:
|
|
||||||
"""Search 财联社.
|
|
||||||
|
|
||||||
Uses the telegraph API for latest news (ignores query).
|
|
||||||
For keyword-specific results, falls back to the search page URL.
|
|
||||||
"""
|
|
||||||
# Step 1: Try to get real results from telegraph API
|
|
||||||
api_results = await self._fetch_telegraph(max_results)
|
|
||||||
if api_results:
|
|
||||||
return api_results[:max_results]
|
|
||||||
|
|
||||||
# Step 2: Fallback to search page URL
|
|
||||||
search_url = self.build_url(query)
|
|
||||||
return [
|
|
||||||
SearchResult(
|
|
||||||
title=f"财联社: {query}",
|
|
||||||
url=search_url,
|
|
||||||
snippet=(
|
|
||||||
f"Search 财联社 for '{query}' "
|
|
||||||
"(JS-required search page)"
|
|
||||||
),
|
|
||||||
source=self.name,
|
|
||||||
rank=1,
|
|
||||||
category="news",
|
|
||||||
extra={"search_url": search_url, "note": "JS-rendered page"},
|
|
||||||
)
|
|
||||||
]
|
|
||||||
|
|
||||||
async def _fetch_telegraph(
|
|
||||||
self, max_results: int = 10
|
|
||||||
) -> list[SearchResult]:
|
|
||||||
"""Fetch real-time telegraph news via API."""
|
|
||||||
try:
|
|
||||||
rn = min(max(max_results, 5), 100)
|
|
||||||
last_time = int(time.time())
|
|
||||||
sign = self._build_api_sign(last_time, rn)
|
|
||||||
|
|
||||||
params = {
|
|
||||||
"app": API_APP,
|
|
||||||
"last_time": str(last_time),
|
|
||||||
"os": API_OS,
|
|
||||||
"rn": str(rn),
|
|
||||||
"sv": API_SV,
|
|
||||||
"sign": sign,
|
|
||||||
}
|
|
||||||
|
|
||||||
response = await self._client.get(API_URL, params=params)
|
|
||||||
response.raise_for_status()
|
|
||||||
|
|
||||||
data = response.json()
|
|
||||||
items = data.get("data", {}).get("roll_data", [])
|
|
||||||
|
|
||||||
results = []
|
|
||||||
for i, item in enumerate(items[:max_results]):
|
|
||||||
content = item.get("content", "")
|
|
||||||
ctime = item.get("ctime", 0)
|
|
||||||
|
|
||||||
# Extract title from content (财联社 format:
|
|
||||||
# "【title】content" or plain text)
|
|
||||||
title, snippet = self._parse_content(content)
|
|
||||||
|
|
||||||
# Use unique URL per item to avoid dedup collision
|
|
||||||
item_url = (
|
|
||||||
f"https://www.cls.cn/telegraph#{ctime}-{i}"
|
|
||||||
)
|
|
||||||
|
|
||||||
results.append(
|
|
||||||
SearchResult(
|
|
||||||
title=title or content[:80],
|
|
||||||
url=item_url,
|
|
||||||
snippet=snippet or content[:200],
|
|
||||||
source=self.name,
|
|
||||||
rank=i + 1,
|
|
||||||
category="news",
|
|
||||||
extra={
|
|
||||||
"time": self._format_time(ctime),
|
|
||||||
"timestamp": ctime,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
return results
|
|
||||||
|
|
||||||
except (httpx.HTTPError, ValueError, KeyError) as e:
|
|
||||||
logger.warning(f"财联社 API failed: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _parse_content(content: str) -> tuple[str, str]:
|
|
||||||
"""Parse 财联社 content into (title, snippet).
|
|
||||||
|
|
||||||
Format examples:
|
|
||||||
"【日本地震】内容详情..." → ("日本地震", "内容详情...")
|
|
||||||
"财联社5月20日电,普通新闻" → ("", "财联社5月20日电,普通新闻")
|
|
||||||
"""
|
|
||||||
if content.startswith("【"):
|
|
||||||
end = content.find("】")
|
|
||||||
if end != -1:
|
|
||||||
title = content[1:end]
|
|
||||||
snippet = content[end + 1:].strip()
|
|
||||||
return title, snippet
|
|
||||||
return "", content
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _format_time(ts: int) -> str:
|
|
||||||
"""Format Unix timestamp to readable time."""
|
|
||||||
import datetime
|
|
||||||
|
|
||||||
try:
|
|
||||||
dt = datetime.datetime.fromtimestamp(
|
|
||||||
ts,
|
|
||||||
tz=datetime.timezone(datetime.timedelta(hours=8)),
|
|
||||||
)
|
|
||||||
return dt.strftime("%Y-%m-%d %H:%M")
|
|
||||||
except (OSError, ValueError):
|
|
||||||
return str(ts)
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
|
||||||
@@ -1,367 +0,0 @@
|
|||||||
"""
|
|
||||||
RSS news aggregation engine.
|
|
||||||
|
|
||||||
Fetches and parses RSS/Atom feeds from trusted news sources.
|
|
||||||
Inspired by the news-summary skill's RSS-based approach.
|
|
||||||
|
|
||||||
Supported sources:
|
|
||||||
- BBC (World, Business, Technology, Top Stories)
|
|
||||||
- Reuters (World News)
|
|
||||||
- NPR (US news)
|
|
||||||
- Al Jazeera (Global South perspective)
|
|
||||||
- 财联社 (RSS if available)
|
|
||||||
- 华尔街见闻 (RSS if available)
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
import xml.etree.ElementTree as ET
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from typing import Any, Optional
|
|
||||||
from urllib.parse import urlparse
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
|
|
||||||
from scripts.core.base import SearchEngine, SearchResult
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
# RSS feed definitions
|
|
||||||
# Core sources (general news) + AI newsletters + essays + podcasts
|
|
||||||
RSS_FEEDS = {
|
|
||||||
"BBC World": {
|
|
||||||
"url": "https://feeds.bbci.co.uk/news/world/rss.xml",
|
|
||||||
"category": "news",
|
|
||||||
"lang": "en",
|
|
||||||
},
|
|
||||||
"BBC Top Stories": {
|
|
||||||
"url": "https://feeds.bbci.co.uk/news/rss.xml",
|
|
||||||
"category": "news",
|
|
||||||
"lang": "en",
|
|
||||||
},
|
|
||||||
"BBC Business": {
|
|
||||||
"url": "https://feeds.bbci.co.uk/news/business/rss.xml",
|
|
||||||
"category": "business",
|
|
||||||
"lang": "en",
|
|
||||||
},
|
|
||||||
"BBC Technology": {
|
|
||||||
"url": "https://feeds.bbci.co.uk/news/technology/rss.xml",
|
|
||||||
"category": "technology",
|
|
||||||
"lang": "en",
|
|
||||||
},
|
|
||||||
"Reuters": {
|
|
||||||
"url": "https://www.reutersagency.com/feed/?best-regions=world&post_type=best",
|
|
||||||
"category": "news",
|
|
||||||
"lang": "en",
|
|
||||||
},
|
|
||||||
"NPR": {
|
|
||||||
"url": "https://feeds.npr.org/1001/rss.xml",
|
|
||||||
"category": "news",
|
|
||||||
"lang": "en",
|
|
||||||
},
|
|
||||||
"Al Jazeera": {
|
|
||||||
"url": "https://www.aljazeera.com/xml/rss/all.xml",
|
|
||||||
"category": "news",
|
|
||||||
"lang": "en",
|
|
||||||
},
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------
|
|
||||||
# AI Newsletters (inspired by news-aggregator-skill)
|
|
||||||
# -----------------------------------------------------------------------
|
|
||||||
"Latent Space": {
|
|
||||||
"url": "https://feeds.transistor.fm/latent-space-podcast",
|
|
||||||
"category": "technology",
|
|
||||||
"lang": "en",
|
|
||||||
"subcategory": "ai_newsletter",
|
|
||||||
},
|
|
||||||
"ChinAI": {
|
|
||||||
"url": "https://chinai.substack.com/feed",
|
|
||||||
"category": "technology",
|
|
||||||
"lang": "en",
|
|
||||||
"subcategory": "ai_newsletter",
|
|
||||||
},
|
|
||||||
"Memia": {
|
|
||||||
"url": "https://memia.substack.com/feed",
|
|
||||||
"category": "technology",
|
|
||||||
"lang": "en",
|
|
||||||
"subcategory": "ai_newsletter",
|
|
||||||
},
|
|
||||||
"Interconnects": {
|
|
||||||
"url": "https://interconnects.substack.com/feed",
|
|
||||||
"category": "technology",
|
|
||||||
"lang": "en",
|
|
||||||
"subcategory": "ai_newsletter",
|
|
||||||
},
|
|
||||||
"KDnuggets": {
|
|
||||||
"url": "https://www.kdnuggets.com/feed",
|
|
||||||
"category": "technology",
|
|
||||||
"lang": "en",
|
|
||||||
"subcategory": "ai_newsletter",
|
|
||||||
},
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------
|
|
||||||
# Deep Essays & Thinkers (inspired by news-aggregator-skill)
|
|
||||||
# -----------------------------------------------------------------------
|
|
||||||
"Paul Graham": {
|
|
||||||
"url": "https://paulgraham.com/rss.xml",
|
|
||||||
"category": "technology",
|
|
||||||
"lang": "en",
|
|
||||||
"subcategory": "essay",
|
|
||||||
},
|
|
||||||
"James Clear": {
|
|
||||||
"url": "https://jamesclear.com/feed",
|
|
||||||
"category": "news",
|
|
||||||
"lang": "en",
|
|
||||||
"subcategory": "essay",
|
|
||||||
},
|
|
||||||
"Wait But Why": {
|
|
||||||
"url": "https://waitbutwhy.com/feed",
|
|
||||||
"category": "news",
|
|
||||||
"lang": "en",
|
|
||||||
"subcategory": "essay",
|
|
||||||
},
|
|
||||||
"Farnam Street": {
|
|
||||||
"url": "https://fs.blog/feed/",
|
|
||||||
"category": "news",
|
|
||||||
"lang": "en",
|
|
||||||
"subcategory": "essay",
|
|
||||||
},
|
|
||||||
|
|
||||||
# -----------------------------------------------------------------------
|
|
||||||
# Podcasts (inspired by news-aggregator-skill)
|
|
||||||
# -----------------------------------------------------------------------
|
|
||||||
"Lex Fridman Podcast": {
|
|
||||||
"url": "https://lexfridman.com/feed/podcast/",
|
|
||||||
"category": "technology",
|
|
||||||
"lang": "en",
|
|
||||||
"subcategory": "podcast",
|
|
||||||
},
|
|
||||||
"80,000 Hours": {
|
|
||||||
"url": "https://80000hours.org/feed/podcast/",
|
|
||||||
"category": "news",
|
|
||||||
"lang": "en",
|
|
||||||
"subcategory": "podcast",
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
class RSSNewsEngine(SearchEngine):
|
|
||||||
"""
|
|
||||||
RSS-based news aggregation engine.
|
|
||||||
|
|
||||||
Fetches and parses RSS/Atom feeds from international news sources.
|
|
||||||
Supports filtering by category, keyword, and time range.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
|
||||||
super().__init__("RSS News", config)
|
|
||||||
timeout = (config or {}).get("timeout", 30)
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=timeout,
|
|
||||||
headers={
|
|
||||||
"User-Agent": (config or {}).get(
|
|
||||||
"user_agent",
|
|
||||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
|
||||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
|
||||||
"Chrome/120.0.0.0 Safari/537.36",
|
|
||||||
),
|
|
||||||
"Accept": "application/rss+xml, application/xml, text/xml, */*",
|
|
||||||
},
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
def build_url(self, query: str, **kwargs) -> str:
|
|
||||||
"""Build search URL - for RSS this is not a search URL but a feed URL."""
|
|
||||||
feed_name = kwargs.get("feed", "BBC World")
|
|
||||||
feed = RSS_FEEDS.get(feed_name, RSS_FEEDS["BBC World"])
|
|
||||||
return feed["url"]
|
|
||||||
|
|
||||||
async def search(self, query: str, max_results: int = 10, **kwargs) -> list[SearchResult]:
|
|
||||||
"""
|
|
||||||
Search news across RSS feeds. If query is empty, returns latest headlines.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
query: Search keyword to filter articles (case-insensitive).
|
|
||||||
If empty/blank, returns all headlines.
|
|
||||||
max_results: Maximum articles per feed.
|
|
||||||
feeds: Specific feeds to query (list of feed names or "all").
|
|
||||||
category: Filter by category (news, business, technology).
|
|
||||||
hours: Only include articles from the last N hours.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
List of SearchResult objects sorted by date (newest first).
|
|
||||||
"""
|
|
||||||
feeds_to_query = self._select_feeds(
|
|
||||||
feed_names=kwargs.get("feeds", "all"),
|
|
||||||
category=kwargs.get("category"),
|
|
||||||
)
|
|
||||||
|
|
||||||
results = []
|
|
||||||
for feed_name, feed_info in feeds_to_query.items():
|
|
||||||
try:
|
|
||||||
response = await self._client.get(feed_info["url"])
|
|
||||||
response.raise_for_status()
|
|
||||||
|
|
||||||
parsed = self._parse_feed(response.text, feed_name, feed_info)
|
|
||||||
for article in parsed:
|
|
||||||
# Filter by keyword if query provided
|
|
||||||
if query:
|
|
||||||
q = query.lower()
|
|
||||||
if q in article.title.lower() or q in article.snippet.lower():
|
|
||||||
results.append(article)
|
|
||||||
else:
|
|
||||||
results.append(article)
|
|
||||||
|
|
||||||
except httpx.HTTPError as e:
|
|
||||||
logger.warning(f"RSS feed '{feed_name}' failed: {e}")
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Apply time filter
|
|
||||||
hours = kwargs.get("hours")
|
|
||||||
if hours:
|
|
||||||
from datetime import timedelta
|
|
||||||
cutoff = datetime.now(timezone.utc) - timedelta(hours=int(hours))
|
|
||||||
results = [r for r in results if r.extra.get("parsed_date", datetime.max.replace(tzinfo=timezone.utc)) >= cutoff]
|
|
||||||
|
|
||||||
# Sort by date (newest first), limit results
|
|
||||||
results.sort(key=lambda r: r.extra.get("parsed_date", datetime.min.replace(tzinfo=timezone.utc)), reverse=True)
|
|
||||||
return results[:max_results * max(1, len(feeds_to_query))]
|
|
||||||
|
|
||||||
def _select_feeds(self, feed_names="all", category=None):
|
|
||||||
"""Select feeds based on filters."""
|
|
||||||
if feed_names != "all":
|
|
||||||
if isinstance(feed_names, str):
|
|
||||||
feed_names = [feed_names]
|
|
||||||
return {k: v for k, v in RSS_FEEDS.items() if k in feed_names}
|
|
||||||
|
|
||||||
if category:
|
|
||||||
return {k: v for k, v in RSS_FEEDS.items() if v.get("category") == category}
|
|
||||||
|
|
||||||
return RSS_FEEDS
|
|
||||||
|
|
||||||
def _parse_feed(self, xml_text: str, feed_name: str, feed_info: dict) -> list[SearchResult]:
|
|
||||||
"""Parse RSS/Atom XML into SearchResult objects."""
|
|
||||||
results = []
|
|
||||||
|
|
||||||
try:
|
|
||||||
root = ET.fromstring(xml_text)
|
|
||||||
except ET.ParseError as e:
|
|
||||||
logger.error(f"Failed to parse RSS from {feed_name}: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
# Detect RSS vs Atom format
|
|
||||||
if root.tag == "rss":
|
|
||||||
return self._parse_rss(root, feed_name, feed_info)
|
|
||||||
elif root.tag.endswith("feed"):
|
|
||||||
return self._parse_atom(root, feed_name, feed_info)
|
|
||||||
else:
|
|
||||||
logger.warning(f"Unknown feed format for {feed_name}: {root.tag}")
|
|
||||||
return results
|
|
||||||
|
|
||||||
def _parse_rss(self, root: ET.Element, feed_name: str, feed_info: dict) -> list[SearchResult]:
|
|
||||||
"""Parse RSS 2.0 format."""
|
|
||||||
results = []
|
|
||||||
namespaces = {"content": "http://purl.org/rss/1.0/modules/content/"}
|
|
||||||
|
|
||||||
for i, item in enumerate(root.findall(".//item")):
|
|
||||||
title = item.findtext("title", "")
|
|
||||||
link = item.findtext("link", "")
|
|
||||||
description = item.findtext("description", "")
|
|
||||||
pub_date_str = item.findtext("pubDate", "")
|
|
||||||
creator = item.findtext("dc:creator", "") or item.findtext("author", "")
|
|
||||||
|
|
||||||
# Clean up description (remove HTML tags)
|
|
||||||
import re
|
|
||||||
snippet = re.sub(r"<[^>]+>", "", description).strip()
|
|
||||||
# Truncate
|
|
||||||
snippet = snippet[:300] + "..." if len(snippet) > 300 else snippet
|
|
||||||
|
|
||||||
# Parse date
|
|
||||||
parsed_date = self._parse_date(pub_date_str)
|
|
||||||
|
|
||||||
results.append(
|
|
||||||
SearchResult(
|
|
||||||
title=title,
|
|
||||||
url=link,
|
|
||||||
snippet=snippet,
|
|
||||||
source=feed_name,
|
|
||||||
rank=i + 1,
|
|
||||||
category=feed_info.get("category", "news"),
|
|
||||||
extra={
|
|
||||||
"feed_name": feed_name,
|
|
||||||
"author": creator,
|
|
||||||
"published": pub_date_str,
|
|
||||||
"parsed_date": parsed_date,
|
|
||||||
"feed_url": feed_info["url"],
|
|
||||||
},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
return results
|
|
||||||
|
|
||||||
def _parse_atom(self, root: ET.Element, feed_name: str, feed_info: dict) -> list[SearchResult]:
|
|
||||||
"""Parse Atom format."""
|
|
||||||
results = []
|
|
||||||
ns = {"a": "http://www.w3.org/2005/Atom"}
|
|
||||||
|
|
||||||
feed_title = root.findtext("a:title", "", ns)
|
|
||||||
|
|
||||||
for i, entry in enumerate(root.findall("a:entry", ns)):
|
|
||||||
title = entry.findtext("a:title", "", ns)
|
|
||||||
link_el = entry.find("a:link", ns)
|
|
||||||
link = link_el.get("href", "") if link_el is not None else ""
|
|
||||||
summary = entry.findtext("a:summary", "", ns) or entry.findtext("a:content", "", ns)
|
|
||||||
published = entry.findtext("a:published", "", ns) or entry.findtext("a:updated", "", ns)
|
|
||||||
author = entry.findtext("a:author/a:name", "", ns)
|
|
||||||
|
|
||||||
import re
|
|
||||||
snippet = re.sub(r"<[^>]+>", "", summary).strip()
|
|
||||||
snippet = snippet[:300] + "..." if len(snippet) > 300 else snippet
|
|
||||||
|
|
||||||
parsed_date = self._parse_date(published)
|
|
||||||
|
|
||||||
results.append(
|
|
||||||
SearchResult(
|
|
||||||
title=title,
|
|
||||||
url=link,
|
|
||||||
snippet=snippet,
|
|
||||||
source=feed_name,
|
|
||||||
rank=i + 1,
|
|
||||||
category=feed_info.get("category", "news"),
|
|
||||||
extra={
|
|
||||||
"feed_name": feed_name,
|
|
||||||
"author": author,
|
|
||||||
"published": published,
|
|
||||||
"parsed_date": parsed_date,
|
|
||||||
"feed_url": feed_info["url"],
|
|
||||||
},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
return results
|
|
||||||
|
|
||||||
def _parse_date(self, date_str: str) -> datetime:
|
|
||||||
"""Parse various date formats to datetime."""
|
|
||||||
if not date_str:
|
|
||||||
return datetime.min.replace(tzinfo=timezone.utc)
|
|
||||||
|
|
||||||
formats = [
|
|
||||||
"%a, %d %b %Y %H:%M:%S %z", # RSS: Sat, 01 Jan 2024 12:00:00 +0000
|
|
||||||
"%a, %d %b %Y %H:%M:%S %Z", # RSS with timezone name
|
|
||||||
"%Y-%m-%dT%H:%M:%S%z", # Atom ISO 8601
|
|
||||||
"%Y-%m-%dT%H:%M:%S.%f%z", # Atom with microseconds
|
|
||||||
"%Y-%m-%dT%H:%M:%SZ", # Atom UTC
|
|
||||||
"%Y-%m-%d", # Simple date
|
|
||||||
]
|
|
||||||
|
|
||||||
for fmt in formats:
|
|
||||||
try:
|
|
||||||
return datetime.strptime(date_str.strip(), fmt).replace(tzinfo=timezone.utc)
|
|
||||||
except (ValueError, TypeError):
|
|
||||||
continue
|
|
||||||
|
|
||||||
return datetime.min.replace(tzinfo=timezone.utc)
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
|
||||||
@@ -1,461 +0,0 @@
|
|||||||
"""
|
|
||||||
Tech & social news aggregation engines.
|
|
||||||
|
|
||||||
Implements non-Playwright news sources inspired by news-aggregator-skill:
|
|
||||||
- Hacker News (via Algolia API)
|
|
||||||
- GitHub Trending (HTML scrape)
|
|
||||||
- 36氪 (HTML scrape)
|
|
||||||
- 微博热搜 (Ajax API)
|
|
||||||
- V2EX (Public API)
|
|
||||||
- 腾讯新闻 (API)
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
import re
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from typing import Any, Optional
|
|
||||||
from urllib.parse import quote_plus
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
|
|
||||||
from scripts.core.base import SearchEngine, SearchResult
|
|
||||||
from scripts.core.thread_utils import run_in_thread
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
# Shared HTTP client settings
|
|
||||||
_DEFAULT_TIMEOUT = 15
|
|
||||||
_COMMON_HEADERS = {
|
|
||||||
"User-Agent": (
|
|
||||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
|
||||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
|
||||||
"Chrome/120.0.0.0 Safari/537.36"
|
|
||||||
),
|
|
||||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
|
||||||
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Hacker News (via Algolia API)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
class HackerNewsEngine(SearchEngine):
|
|
||||||
"""Hacker News search via Algolia public API."""
|
|
||||||
|
|
||||||
SEARCH_URL = "https://hn.algolia.com/api/v1/search"
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
|
||||||
super().__init__("Hacker News", config)
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=_DEFAULT_TIMEOUT,
|
|
||||||
headers={"User-Agent": _COMMON_HEADERS["User-Agent"]},
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, max_results: int = 10, **kwargs) -> list[SearchResult]:
|
|
||||||
try:
|
|
||||||
params = {
|
|
||||||
"query": query,
|
|
||||||
"hitsPerPage": max_results,
|
|
||||||
"tags": "story",
|
|
||||||
}
|
|
||||||
response = await self._client.get(self.SEARCH_URL, params=params)
|
|
||||||
response.raise_for_status()
|
|
||||||
data = response.json()
|
|
||||||
return self._parse_results(data, max_results)
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Hacker News search failed: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_results(self, data: dict, max_results: int) -> list[SearchResult]:
|
|
||||||
results = []
|
|
||||||
hits = data.get("hits", [])[:max_results]
|
|
||||||
for i, hit in enumerate(hits):
|
|
||||||
title = hit.get("title", "")
|
|
||||||
url = hit.get("url") or f"https://news.ycombinator.com/item?id={hit.get('objectID', '')}"
|
|
||||||
snippet = hit.get("story_text", "") or ""
|
|
||||||
# Clean HTML from snippet
|
|
||||||
snippet = re.sub(r"<[^>]+>", "", snippet)[:200]
|
|
||||||
points = hit.get("points", 0)
|
|
||||||
author = hit.get("author", "")
|
|
||||||
created_at = hit.get("created_at", "")
|
|
||||||
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title,
|
|
||||||
url=url,
|
|
||||||
snippet=snippet or f"[{points} points] by {author}",
|
|
||||||
source=self.name,
|
|
||||||
rank=i + 1,
|
|
||||||
category="news",
|
|
||||||
extra={
|
|
||||||
"points": points,
|
|
||||||
"author": author,
|
|
||||||
"time": created_at,
|
|
||||||
"heat": f"{points} points",
|
|
||||||
},
|
|
||||||
))
|
|
||||||
return results
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# GitHub Trending
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
class GitHubTrendingEngine(SearchEngine):
|
|
||||||
"""GitHub Trending repositories."""
|
|
||||||
|
|
||||||
TRENDING_URL = "https://github.com/trending"
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
|
||||||
super().__init__("GitHub Trending", config)
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=_DEFAULT_TIMEOUT,
|
|
||||||
headers=_COMMON_HEADERS,
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str = "", max_results: int = 10, **kwargs) -> list[SearchResult]:
|
|
||||||
"""GitHub Trending doesn't support keyword search; returns top trending repos."""
|
|
||||||
url = self.TRENDING_URL
|
|
||||||
if query:
|
|
||||||
url += f"?spoken_language_code=&q={quote_plus(query)}"
|
|
||||||
try:
|
|
||||||
response = await self._client.get(url)
|
|
||||||
response.raise_for_status()
|
|
||||||
return await run_in_thread(self._parse_html, response.text, max_results)
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"GitHub Trending failed: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_html(self, html: str, max_results: int) -> list[SearchResult]:
|
|
||||||
soup = BeautifulSoup(html, "lxml")
|
|
||||||
results = []
|
|
||||||
articles = soup.select("article.Box-row")[:max_results]
|
|
||||||
for i, article in enumerate(articles):
|
|
||||||
h2 = article.select_one("h2")
|
|
||||||
if not h2:
|
|
||||||
continue
|
|
||||||
repo_path = h2.get_text(strip=True).replace(" ", "")
|
|
||||||
repo_url = f"https://github.com/{repo_path}"
|
|
||||||
|
|
||||||
desc_el = article.select_one("p")
|
|
||||||
description = desc_el.get_text(strip=True) if desc_el else ""
|
|
||||||
|
|
||||||
# Language
|
|
||||||
lang_el = article.select_one("[itemprop='programmingLanguage']")
|
|
||||||
language = lang_el.get_text(strip=True) if lang_el else ""
|
|
||||||
|
|
||||||
# Stars today
|
|
||||||
stars_el = article.select_one(".float-sm-right")
|
|
||||||
stars_today = stars_el.get_text(strip=True) if stars_el else ""
|
|
||||||
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=repo_path,
|
|
||||||
url=repo_url,
|
|
||||||
snippet=description[:200] if description else language,
|
|
||||||
source=self.name,
|
|
||||||
rank=i + 1,
|
|
||||||
category="news",
|
|
||||||
extra={
|
|
||||||
"language": language,
|
|
||||||
"heat": stars_today,
|
|
||||||
"time": "",
|
|
||||||
},
|
|
||||||
))
|
|
||||||
return results
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# 36氪 (36kr.com)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
class Kr36Engine(SearchEngine):
|
|
||||||
"""36氪 tech news search."""
|
|
||||||
|
|
||||||
SEARCH_URL = "https://36kr.com/search/articles/{keyword}"
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
|
||||||
super().__init__("36氪", config)
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=_DEFAULT_TIMEOUT,
|
|
||||||
headers={**_COMMON_HEADERS, "Referer": "https://36kr.com/"},
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, max_results: int = 10, **kwargs) -> list[SearchResult]:
|
|
||||||
url = self.SEARCH_URL.format(keyword=quote_plus(query))
|
|
||||||
try:
|
|
||||||
response = await self._client.get(url)
|
|
||||||
response.raise_for_status()
|
|
||||||
return await run_in_thread(self._parse_html, response.text, query, max_results)
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"36氪 search failed: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_html(self, html: str, query: str, max_results: int) -> list[SearchResult]:
|
|
||||||
soup = BeautifulSoup(html, "lxml")
|
|
||||||
results = []
|
|
||||||
|
|
||||||
# Look for article cards
|
|
||||||
for i, item in enumerate(soup.select("a[href*='/p/'], .article-item, .search-result-item, [class*='article']")[:max_results]):
|
|
||||||
title_el = item.select_one("h3, h4, .title, [class*='title']")
|
|
||||||
link = item.get("href", "")
|
|
||||||
snippet_el = item.select_one(".abstract, .summary, .desc, p, [class*='desc']")
|
|
||||||
|
|
||||||
title = title_el.get_text(strip=True) if title_el else ""
|
|
||||||
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
|
||||||
|
|
||||||
if not title and not link:
|
|
||||||
continue
|
|
||||||
|
|
||||||
full_url = link if link.startswith("http") else f"https://36kr.com{link}"
|
|
||||||
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title or f"36氪: {query}",
|
|
||||||
url=full_url,
|
|
||||||
snippet=snippet[:200] if snippet else "",
|
|
||||||
source=self.name,
|
|
||||||
rank=i + 1,
|
|
||||||
category="news",
|
|
||||||
extra={"region": "cn"},
|
|
||||||
))
|
|
||||||
|
|
||||||
if not results:
|
|
||||||
search_url = f"https://36kr.com/search/articles/{quote_plus(query)}"
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=f"36氪: {query}",
|
|
||||||
url=search_url,
|
|
||||||
snippet="Search 36氪 (site may require JS)",
|
|
||||||
source=self.name,
|
|
||||||
rank=1,
|
|
||||||
category="news",
|
|
||||||
extra={"search_url": search_url, "note": "JS-rendered page"},
|
|
||||||
))
|
|
||||||
|
|
||||||
return results
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# 微博热搜 (Weibo Trending)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
class WeiboEngine(SearchEngine):
|
|
||||||
"""微博热搜 trending topics."""
|
|
||||||
|
|
||||||
HOT_URL = "https://weibo.com/ajax/side/hotSearch"
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
|
||||||
super().__init__("微博热搜", config)
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=_DEFAULT_TIMEOUT,
|
|
||||||
headers={**_COMMON_HEADERS, "Referer": "https://weibo.com/"},
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str = "", max_results: int = 10, **kwargs) -> list[SearchResult]:
|
|
||||||
try:
|
|
||||||
response = await self._client.get(self.HOT_URL)
|
|
||||||
response.raise_for_status()
|
|
||||||
data = response.json()
|
|
||||||
return self._parse_results(data, query, max_results)
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"微博热搜 failed: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_results(self, data: dict, query: str, max_results: int) -> list[SearchResult]:
|
|
||||||
results = []
|
|
||||||
realtime = data.get("data", {}).get("realtime", [])[:max_results]
|
|
||||||
for i, item in enumerate(realtime):
|
|
||||||
word = item.get("word", "")
|
|
||||||
raw_hot = item.get("raw_hot", 0)
|
|
||||||
url = f"https://s.weibo.com/weibo?q={quote_plus(word)}"
|
|
||||||
|
|
||||||
# Filter by keyword if query provided
|
|
||||||
if query and query.lower() not in word.lower():
|
|
||||||
continue
|
|
||||||
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=f"#{word}#",
|
|
||||||
url=url,
|
|
||||||
snippet=f"热度: {raw_hot:,}" if raw_hot else "",
|
|
||||||
source=self.name,
|
|
||||||
rank=i + 1,
|
|
||||||
category="news",
|
|
||||||
extra={
|
|
||||||
"heat": f"{raw_hot:,}" if raw_hot else "",
|
|
||||||
"region": "cn",
|
|
||||||
},
|
|
||||||
))
|
|
||||||
|
|
||||||
return results
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# V2EX (via public API)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
class V2EXEngine(SearchEngine):
|
|
||||||
"""V2EX tech community hot topics."""
|
|
||||||
|
|
||||||
HOT_API = "https://www.v2ex.com/api/v2/topics/hot"
|
|
||||||
# Fallback to older API if v2 needs auth
|
|
||||||
HOT_API_FALLBACK = "https://www.v2ex.com/api/topics/hot.json"
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
|
||||||
super().__init__("V2EX", config)
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=_DEFAULT_TIMEOUT,
|
|
||||||
headers=_COMMON_HEADERS,
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str = "", max_results: int = 10, **kwargs) -> list[SearchResult]:
|
|
||||||
# Try v2 API first, fallback to old API
|
|
||||||
for api_url in [self.HOT_API, self.HOT_API_FALLBACK]:
|
|
||||||
try:
|
|
||||||
response = await self._client.get(api_url)
|
|
||||||
if response.status_code == 200:
|
|
||||||
data = response.json()
|
|
||||||
# v2 API returns list directly; old API also returns list
|
|
||||||
topics = data if isinstance(data, list) else data.get("data", [])
|
|
||||||
return self._parse_results(topics, query, max_results)
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
logger.error("V2EX API failed (both v2 and fallback)")
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_results(self, topics: list, query: str, max_results: int) -> list[SearchResult]:
|
|
||||||
results = []
|
|
||||||
for i, topic in enumerate(topics[:max_results]):
|
|
||||||
title = topic.get("title", "")
|
|
||||||
node_name = ""
|
|
||||||
if "node" in topic and isinstance(topic["node"], dict):
|
|
||||||
node_name = topic["node"].get("title", "")
|
|
||||||
topic_id = topic.get("id", "")
|
|
||||||
url = f"https://www.v2ex.com/t/{topic_id}"
|
|
||||||
replies = topic.get("replies", 0)
|
|
||||||
created = topic.get("created", "")
|
|
||||||
member = ""
|
|
||||||
if "member" in topic and isinstance(topic["member"], dict):
|
|
||||||
member = topic["member"].get("username", "")
|
|
||||||
|
|
||||||
# Filter by keyword
|
|
||||||
if query and query.lower() not in title.lower():
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Format time
|
|
||||||
time_str = ""
|
|
||||||
if created:
|
|
||||||
try:
|
|
||||||
dt = datetime.fromtimestamp(int(created), tz=timezone.utc)
|
|
||||||
time_str = dt.strftime("%Y-%m-%d %H:%M")
|
|
||||||
except (ValueError, OSError):
|
|
||||||
time_str = str(created)
|
|
||||||
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title,
|
|
||||||
url=url,
|
|
||||||
snippet=f"[{node_name}] by {member}" if node_name and member else f"by {member}" if member else "",
|
|
||||||
source=self.name,
|
|
||||||
rank=i + 1,
|
|
||||||
category="news",
|
|
||||||
extra={
|
|
||||||
"heat": f"{replies} replies",
|
|
||||||
"author": member,
|
|
||||||
"time": time_str,
|
|
||||||
"region": "cn",
|
|
||||||
},
|
|
||||||
))
|
|
||||||
|
|
||||||
return results
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# 腾讯新闻 (news.qq.com)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
class TencentNewsEngine(SearchEngine):
|
|
||||||
"""腾讯新闻 search."""
|
|
||||||
|
|
||||||
SEARCH_URL = "https://news.qq.com/search"
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
|
||||||
super().__init__("腾讯新闻", config)
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=_DEFAULT_TIMEOUT,
|
|
||||||
headers={**_COMMON_HEADERS, "Referer": "https://news.qq.com/"},
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(self, query: str, max_results: int = 10, **kwargs) -> list[SearchResult]:
|
|
||||||
url = f"{self.SEARCH_URL}?q={quote_plus(query)}"
|
|
||||||
try:
|
|
||||||
response = await self._client.get(url)
|
|
||||||
response.raise_for_status()
|
|
||||||
return await run_in_thread(self._parse_html, response.text, query, max_results)
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"腾讯新闻 search failed: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_html(self, html: str, query: str, max_results: int) -> list[SearchResult]:
|
|
||||||
soup = BeautifulSoup(html, "lxml")
|
|
||||||
results = []
|
|
||||||
|
|
||||||
for i, item in enumerate(soup.select("a[href*='news.qq.com'], .list-item, [class*='result'], article")[:max_results]):
|
|
||||||
title_el = item.select_one("h3, h4, .title, a")
|
|
||||||
link = item.get("href", "") or (title_el.get("href", "") if title_el else "")
|
|
||||||
snippet_el = item.select_one(".summary, .desc, .abstract, p, .text")
|
|
||||||
|
|
||||||
title = title_el.get_text(strip=True) if title_el else ""
|
|
||||||
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
|
||||||
|
|
||||||
if not title:
|
|
||||||
continue
|
|
||||||
|
|
||||||
full_url = link if link.startswith("http") else f"https:{link}" if link.startswith("//") else link
|
|
||||||
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=title,
|
|
||||||
url=full_url,
|
|
||||||
snippet=snippet[:200] if snippet else "",
|
|
||||||
source=self.name,
|
|
||||||
rank=i + 1,
|
|
||||||
category="news",
|
|
||||||
extra={"region": "cn"},
|
|
||||||
))
|
|
||||||
|
|
||||||
if not results:
|
|
||||||
search_url = f"{self.SEARCH_URL}?q={quote_plus(query)}"
|
|
||||||
results.append(SearchResult(
|
|
||||||
title=f"腾讯新闻: {query}",
|
|
||||||
url=search_url,
|
|
||||||
snippet=f"Search Tencent News for '{query}'",
|
|
||||||
source=self.name,
|
|
||||||
rank=1,
|
|
||||||
category="news",
|
|
||||||
extra={"search_url": search_url, "note": "JS-rendered page"},
|
|
||||||
))
|
|
||||||
|
|
||||||
return results
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
|
||||||
@@ -1,117 +0,0 @@
|
|||||||
"""
|
|
||||||
华尔街见闻 (wallstreetcn.com) news search engine.
|
|
||||||
|
|
||||||
Scrapes wallstreetcn.com search results for financial news.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
from typing import Optional
|
|
||||||
from urllib.parse import quote_plus
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
|
|
||||||
from scripts.core.base import SearchEngine, SearchResult
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
class WallStreetCNEngine(SearchEngine):
|
|
||||||
"""华尔街见闻 (wallstreetcn.com) news search engine."""
|
|
||||||
|
|
||||||
SEARCH_URL = "https://wallstreetcn.com/search"
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
|
||||||
super().__init__("wallstreetcn", config)
|
|
||||||
timeout = (config or {}).get("timeout", 15)
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=timeout,
|
|
||||||
headers={
|
|
||||||
"User-Agent": (config or {}).get(
|
|
||||||
"user_agent",
|
|
||||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
|
||||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
|
||||||
"Chrome/120.0.0.0 Safari/537.36",
|
|
||||||
),
|
|
||||||
"Referer": "https://wallstreetcn.com/",
|
|
||||||
},
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
def build_url(self, query: str, **kwargs) -> str:
|
|
||||||
return f"{self.SEARCH_URL}?q={quote_plus(query)}"
|
|
||||||
|
|
||||||
async def search(
|
|
||||||
self, query: str, max_results: int = 10, **kwargs
|
|
||||||
) -> list[SearchResult]:
|
|
||||||
"""Search 华尔街见闻 and parse results."""
|
|
||||||
url = self.build_url(query)
|
|
||||||
try:
|
|
||||||
response = await self._client.get(url)
|
|
||||||
response.raise_for_status()
|
|
||||||
return self._parse_results(response.text, query, max_results)
|
|
||||||
except httpx.HTTPError as e:
|
|
||||||
logger.error(f"华尔街见闻 search failed: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
def _parse_results(
|
|
||||||
self, html: str, query: str, max_results: int
|
|
||||||
) -> list[SearchResult]:
|
|
||||||
"""Parse 华尔街见闻 search page."""
|
|
||||||
soup = BeautifulSoup(html, "lxml")
|
|
||||||
results = []
|
|
||||||
|
|
||||||
articles = soup.select(
|
|
||||||
"a[href*='/article/'], .search-result-item, article, .item"
|
|
||||||
)
|
|
||||||
for i, article in enumerate(articles[:max_results]):
|
|
||||||
title_el = article.select_one("h3, h4, .title, a")
|
|
||||||
link = article.get("href", "")
|
|
||||||
snippet_el = article.select_one(
|
|
||||||
".summary, .desc, .abstract, p"
|
|
||||||
)
|
|
||||||
|
|
||||||
title = title_el.get_text(strip=True) if title_el else ""
|
|
||||||
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
|
||||||
|
|
||||||
if not title and not link:
|
|
||||||
continue
|
|
||||||
|
|
||||||
full_url = (
|
|
||||||
link
|
|
||||||
if link.startswith("http")
|
|
||||||
else f"https://wallstreetcn.com{link}"
|
|
||||||
)
|
|
||||||
|
|
||||||
results.append(
|
|
||||||
SearchResult(
|
|
||||||
title=title or f"华尔街见闻: {query}",
|
|
||||||
url=full_url,
|
|
||||||
snippet=snippet or f"Search result from 华尔街见闻",
|
|
||||||
source=self.name,
|
|
||||||
rank=i + 1,
|
|
||||||
category="news",
|
|
||||||
extra={"region": "cn"},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
if not results:
|
|
||||||
search_url = self.build_url(query)
|
|
||||||
results.append(
|
|
||||||
SearchResult(
|
|
||||||
title=f"华尔街见闻: {query}",
|
|
||||||
url=search_url,
|
|
||||||
snippet=f"Search 华尔街见闻 for '{query}' (site may require JS)",
|
|
||||||
source=self.name,
|
|
||||||
rank=1,
|
|
||||||
category="news",
|
|
||||||
extra={"search_url": search_url, "note": "JS-rendered page"},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
return results
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
"""Social media search implementations (Twitter/X, etc.)."""
|
|
||||||
@@ -1,208 +0,0 @@
|
|||||||
"""
|
|
||||||
Twitter/X search module.
|
|
||||||
|
|
||||||
Generates search URLs for X.com (formerly Twitter) search and
|
|
||||||
optionally parses search result pages.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
from typing import Optional
|
|
||||||
from urllib.parse import quote_plus
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
|
|
||||||
from scripts.core.base import SearchEngine, SearchResult
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
class TwitterSearchEngine(SearchEngine):
|
|
||||||
"""
|
|
||||||
Twitter/X search engine.
|
|
||||||
|
|
||||||
Provides search URL generation and optional result scraping.
|
|
||||||
Supports basic keyword search, user search, and advanced queries.
|
|
||||||
"""
|
|
||||||
|
|
||||||
SEARCH_URL = "https://twitter.com/search"
|
|
||||||
BASE_URL = "https://twitter.com"
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
|
||||||
super().__init__("Twitter", config)
|
|
||||||
timeout = (config or {}).get("timeout", 15)
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=timeout,
|
|
||||||
headers={
|
|
||||||
"User-Agent": (config or {}).get(
|
|
||||||
"user_agent",
|
|
||||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
|
||||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
|
||||||
"Chrome/120.0.0.0 Safari/537.36",
|
|
||||||
),
|
|
||||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
|
||||||
"Accept-Language": "en-US,en;q=0.5",
|
|
||||||
},
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
def build_url(self, query: str, search_type: str = "top", **kwargs) -> str:
|
|
||||||
"""
|
|
||||||
Build Twitter/X search URL.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
query: Search keyword.
|
|
||||||
search_type: "top", "latest", "people", "photos", "videos"
|
|
||||||
**kwargs: Additional URL parameters.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Search URL string.
|
|
||||||
"""
|
|
||||||
encoded_query = quote_plus(query)
|
|
||||||
url = f"{self.SEARCH_URL}?q={encoded_query}&src=typed_query"
|
|
||||||
|
|
||||||
if search_type == "latest":
|
|
||||||
url += "&f=live"
|
|
||||||
elif search_type == "people":
|
|
||||||
url += "&f=user"
|
|
||||||
elif search_type == "photos":
|
|
||||||
url += "&f=image"
|
|
||||||
elif search_type == "videos":
|
|
||||||
url += "&f=video"
|
|
||||||
|
|
||||||
if kwargs.get("lang"):
|
|
||||||
url += f"&l={kwargs['lang']}"
|
|
||||||
if kwargs.get("until"):
|
|
||||||
url += f"&until={kwargs['until']}"
|
|
||||||
if kwargs.get("since"):
|
|
||||||
url += f"&since={kwargs['since']}"
|
|
||||||
|
|
||||||
return url
|
|
||||||
|
|
||||||
def build_advanced_url(self, **params) -> str:
|
|
||||||
"""
|
|
||||||
Build advanced Twitter search URL with operators.
|
|
||||||
|
|
||||||
Supported operators:
|
|
||||||
from:username - Tweets from user
|
|
||||||
to:username - Tweets to user
|
|
||||||
@username - Mentions
|
|
||||||
#hashtag - Hashtag
|
|
||||||
filter:links - Tweets with links
|
|
||||||
filter:images - Tweets with images
|
|
||||||
min_retweets:N - Minimum retweets
|
|
||||||
min_faves:N - Minimum likes
|
|
||||||
lang:en - Language
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Advanced search URL.
|
|
||||||
"""
|
|
||||||
parts = []
|
|
||||||
if params.get("from_user"):
|
|
||||||
parts.append(f"from:{params['from_user']}")
|
|
||||||
if params.get("to_user"):
|
|
||||||
parts.append(f"to:{params['to_user']}")
|
|
||||||
if params.get("hashtag"):
|
|
||||||
parts.append(f"#{params['hashtag']}")
|
|
||||||
if params.get("keyword"):
|
|
||||||
parts.append(params["keyword"])
|
|
||||||
if params.get("min_retweets"):
|
|
||||||
parts.append(f"min_retweets:{params['min_retweets']}")
|
|
||||||
if params.get("min_faves"):
|
|
||||||
parts.append(f"min_faves:{params['min_faves']}")
|
|
||||||
if params.get("lang"):
|
|
||||||
parts.append(f"lang:{params['lang']}")
|
|
||||||
if params.get("filter"):
|
|
||||||
parts.append(f"filter:{params['filter']}")
|
|
||||||
|
|
||||||
query = " ".join(parts)
|
|
||||||
return self.build_url(query, search_type=params.get("search_type", "top"))
|
|
||||||
|
|
||||||
async def search(self, query: str, max_results: int = 10, search_type: str = "top", **kwargs) -> list[SearchResult]:
|
|
||||||
"""
|
|
||||||
Search Twitter/X and optionally parse page content.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
query: Search keyword.
|
|
||||||
max_results: Maximum results (used for URL generation).
|
|
||||||
search_type: "top", "latest", "people", "photos", "videos"
|
|
||||||
**kwargs: Additional options.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
List of SearchResult objects.
|
|
||||||
"""
|
|
||||||
url = self.build_url(query, search_type=search_type, **kwargs)
|
|
||||||
|
|
||||||
# Try to scrape the search page for basic info
|
|
||||||
try:
|
|
||||||
response = await self._client.get(url)
|
|
||||||
if response.status_code == 200:
|
|
||||||
soup = BeautifulSoup(response.text, "lxml")
|
|
||||||
results = self._parse_page(soup, max_results)
|
|
||||||
if results:
|
|
||||||
return results
|
|
||||||
except Exception as e:
|
|
||||||
logger.debug(f"Twitter page scraping failed (expected): {e}")
|
|
||||||
|
|
||||||
# Fall back to URL-based result
|
|
||||||
result_type_map = {
|
|
||||||
"top": "Top tweets",
|
|
||||||
"latest": "Latest tweets",
|
|
||||||
"people": "People",
|
|
||||||
"photos": "Photos",
|
|
||||||
"videos": "Videos",
|
|
||||||
}
|
|
||||||
desc = result_type_map.get(search_type, "Tweets")
|
|
||||||
|
|
||||||
return [
|
|
||||||
SearchResult(
|
|
||||||
title=f"Twitter: {query} ({desc})",
|
|
||||||
url=url,
|
|
||||||
snippet=f"Search Twitter/X for '{query}'. Note: Twitter may require login.",
|
|
||||||
source=self.name,
|
|
||||||
rank=1,
|
|
||||||
category="social",
|
|
||||||
extra={"search_type": search_type, "twitter_url": url},
|
|
||||||
)
|
|
||||||
]
|
|
||||||
|
|
||||||
def _parse_page(self, soup, max_results: int) -> list[SearchResult]:
|
|
||||||
"""Attempt to extract tweet info from search page."""
|
|
||||||
results = []
|
|
||||||
# Twitter uses JS rendering, so HTML parsing may yield limited results
|
|
||||||
tweets = soup.select("article[data-testid='tweet'], div.tweet, .content")
|
|
||||||
for i, tweet in enumerate(tweets[:max_results]):
|
|
||||||
text_el = tweet.select_one(
|
|
||||||
"[data-testid='tweetText'], .tweet-text, p[lang]"
|
|
||||||
)
|
|
||||||
link_el = tweet.select_one("a[href*='/status/'], time")
|
|
||||||
user_el = tweet.select_one(
|
|
||||||
"[data-testid='User-Name'], .username, .fullname"
|
|
||||||
)
|
|
||||||
|
|
||||||
text = text_el.get_text(strip=True) if text_el else ""
|
|
||||||
user = user_el.get_text(strip=True) if user_el else ""
|
|
||||||
|
|
||||||
tweet_url = ""
|
|
||||||
if link_el:
|
|
||||||
href = link_el.get("href", "")
|
|
||||||
tweet_url = f"{self.BASE_URL}{href}" if href.startswith("/") else href
|
|
||||||
|
|
||||||
if text:
|
|
||||||
results.append(
|
|
||||||
SearchResult(
|
|
||||||
title=f"@{user}: {text[:80]}..." if user else text[:80],
|
|
||||||
url=tweet_url or self.build_url(text[:30]),
|
|
||||||
snippet=text,
|
|
||||||
source=self.name,
|
|
||||||
rank=i + 1,
|
|
||||||
category="social",
|
|
||||||
extra={"user": user} if user else {},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
return results
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
|
||||||
+144
-261
@@ -1,297 +1,180 @@
|
|||||||
"""
|
"""
|
||||||
Unified search entry point.
|
Unified multi-source search — Python API.
|
||||||
|
|
||||||
Provides a single interface to search across all configured sources.
|
Provides a clean async interface for programmatic use.
|
||||||
|
All engines run in parallel with configurable concurrency.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
from scripts.unified_search import UnifiedSearch
|
||||||
|
|
||||||
|
us = UnifiedSearch()
|
||||||
|
resp = await us.search("quantum computing", sources=["bing", "arxiv"], max_results=5)
|
||||||
|
for r in resp.results:
|
||||||
|
print(f"[{r.rank}] {r.title}")
|
||||||
|
await us.close()
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import time
|
||||||
from typing import Any, Optional
|
from typing import Any, Optional
|
||||||
|
|
||||||
from scripts.core.base import SearchResult
|
from scripts.core.base import BaseEngine, SearchResult
|
||||||
from scripts.core.config_loader import (
|
from scripts.engines.bing_scraper import BingScraper
|
||||||
get_search_engines, get_news_sources, get_academic_sources,
|
from scripts.engines.sogou_scraper import SogouScraper
|
||||||
get_wechat_sources, get_api_engines, get_special_engines,
|
from scripts.engines.baidu_scraper import BaiduScraper
|
||||||
get_social_sources, get_all_sources, get_engine,
|
from scripts.engines.search360_scraper import Search360Scraper
|
||||||
)
|
from scripts.engines.parser_engines import DuckDuckGoParser, BraveParser
|
||||||
from scripts.engines.url_engine import UrlSearchEngine
|
from scripts.engines.news_engine import NewsSearch
|
||||||
from scripts.engines.parser_engines import DuckDuckGoEngine
|
from scripts.engines.academic_engine import AcademicSearch
|
||||||
from scripts.news.cls import CLSNewsEngine
|
from scripts.engines.github_engine import GitHubEngine
|
||||||
from scripts.news.wallstreetcn import WallStreetCNEngine
|
from scripts.core.aggregator import ResultAggregator
|
||||||
from scripts.news.rss import RSSNewsEngine
|
from scripts.models import SearchResponse, SourceRef, SearchMetadata, EngineTiming
|
||||||
from scripts.news.tech_news import (
|
|
||||||
HackerNewsEngine, GitHubTrendingEngine, Kr36Engine,
|
|
||||||
WeiboEngine, V2EXEngine, TencentNewsEngine,
|
|
||||||
)
|
|
||||||
from scripts.academic.arxiv import ArXivEngine
|
|
||||||
from scripts.wechat.sogou_weixin import SogouWeChatEngine
|
|
||||||
from scripts.social.twitter import TwitterSearchEngine
|
|
||||||
from scripts.engines.baidu_qianfan import BaiduQianFanEngine
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
# Default timeout per engine (seconds)
|
# Default engine registry
|
||||||
_DEFAULT_ENGINE_TIMEOUT = 8
|
_ALL_ENGINES: dict[str, BaseEngine] = {
|
||||||
# Max concurrent engine calls
|
"bing": BingScraper(),
|
||||||
_DEFAULT_MAX_CONCURRENT = 5
|
"sogou": SogouScraper(),
|
||||||
|
"baidu": BaiduScraper(),
|
||||||
SOURCE_CATEGORIES = {
|
"360": Search360Scraper(),
|
||||||
"web": get_search_engines,
|
"duckduckgo": DuckDuckGoParser(),
|
||||||
"news": get_news_sources,
|
"brave": BraveParser(),
|
||||||
"academic": get_academic_sources,
|
"news": NewsSearch(),
|
||||||
"wechat": get_wechat_sources,
|
"academic": AcademicSearch(),
|
||||||
"social": get_social_sources,
|
"github": GitHubEngine(),
|
||||||
"api": get_api_engines,
|
|
||||||
"special": get_special_engines,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
class UnifiedSearch:
|
class UnifiedSearch:
|
||||||
"""Unified search client that aggregates results from multiple sources."""
|
"""Unified multi-source search — Python API entry point."""
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict[str, Any]] = None):
|
def __init__(self, timeout: float = 15.0, max_concurrent: int = 5):
|
||||||
self.config = config or {}
|
self.timeout = timeout
|
||||||
self._cache: dict[str, list[SearchResult]] = {}
|
self.max_concurrent = max_concurrent
|
||||||
self._engines: dict[str, Any] = {}
|
self._engines: dict[str, BaseEngine] = {
|
||||||
self._timeout = self.config.get("engine_timeout", _DEFAULT_ENGINE_TIMEOUT)
|
name: eng.__class__(timeout=timeout)
|
||||||
self._max_concurrent = self.config.get(
|
for name, eng in _ALL_ENGINES.items()
|
||||||
"max_concurrent", _DEFAULT_MAX_CONCURRENT
|
}
|
||||||
)
|
|
||||||
self._semaphore = asyncio.Semaphore(self._max_concurrent)
|
|
||||||
|
|
||||||
async def __aenter__(self):
|
|
||||||
return self
|
|
||||||
|
|
||||||
async def __aexit__(self, *args):
|
|
||||||
await self.close()
|
|
||||||
|
|
||||||
async def search(
|
async def search(
|
||||||
self, query: str, sources: Optional[list[str]] = None,
|
self,
|
||||||
max_results: int = 10, force_refresh: bool = False, **kwargs,
|
query: str,
|
||||||
) -> list[SearchResult]:
|
sources: Optional[list[str]] = None,
|
||||||
"""Search across selected sources with streaming + early termination.
|
max_results: int = 10,
|
||||||
|
region: str = "cn",
|
||||||
Returns results as soon as enough are collected, cancelling remaining
|
**kwargs,
|
||||||
slow engines to minimize total wait time.
|
) -> SearchResponse:
|
||||||
"""
|
"""
|
||||||
if not sources or "all" in sources:
|
Search across multiple sources in parallel.
|
||||||
sources = list(SOURCE_CATEGORIES.keys())
|
|
||||||
|
|
||||||
cache_key = f"{sorted(sources) if isinstance(sources, list) else sources}:{query}"
|
Args:
|
||||||
if not force_refresh and cache_key in self._cache:
|
query: Search query string
|
||||||
return self._cache[cache_key]
|
sources: List of engine names to use (default: all)
|
||||||
|
max_results: Max results to return (deduped)
|
||||||
|
region: "cn" or "global"
|
||||||
|
**kwargs: Extra params passed to engines
|
||||||
|
|
||||||
# --- Build task list --- #
|
Returns:
|
||||||
engine_list: list[tuple[Any, str]] = [] # (engine, query)
|
SearchResponse with results, sources, and metadata
|
||||||
for source_spec in sources:
|
"""
|
||||||
if source_spec in SOURCE_CATEGORIES:
|
t0 = time.time()
|
||||||
for engine_def in SOURCE_CATEGORIES[source_spec]():
|
|
||||||
engine = self._get_engine(engine_def)
|
|
||||||
if engine:
|
|
||||||
engine_list.append((engine, query))
|
|
||||||
else:
|
|
||||||
engine_def = get_engine(source_spec)
|
|
||||||
if engine_def:
|
|
||||||
engine = self._get_engine(engine_def)
|
|
||||||
if engine:
|
|
||||||
engine_list.append((engine, query))
|
|
||||||
|
|
||||||
if not engine_list:
|
# Select engines
|
||||||
return []
|
if sources:
|
||||||
|
selected = {k: v for k, v in self._engines.items() if k in sources}
|
||||||
engine_count = len(engine_list)
|
|
||||||
|
|
||||||
# --- Streaming: as_completed + early termination --- #
|
|
||||||
seen_urls: set[str] = set()
|
|
||||||
all_results: list[SearchResult] = []
|
|
||||||
pending = set()
|
|
||||||
|
|
||||||
# Submit all tasks
|
|
||||||
for engine, q in engine_list:
|
|
||||||
coro = self._safe_search(engine, q, max_results, **kwargs)
|
|
||||||
pending.add(asyncio.create_task(coro))
|
|
||||||
|
|
||||||
enough_results = max_results * 2 # allow some slack for dedup
|
|
||||||
|
|
||||||
# Process as they complete
|
|
||||||
for done in asyncio.as_completed(pending):
|
|
||||||
try:
|
|
||||||
results = await done
|
|
||||||
if isinstance(results, (Exception, BaseException)):
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Dedup on the fly
|
|
||||||
for r in results:
|
|
||||||
key = r.url.strip().rstrip("/")
|
|
||||||
if key and key not in seen_urls:
|
|
||||||
seen_urls.add(key)
|
|
||||||
all_results.append(r)
|
|
||||||
# Early termination: stop waiting for more engines
|
|
||||||
if len(all_results) >= enough_results:
|
|
||||||
break
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Check early termination after each completion
|
|
||||||
if len(all_results) >= enough_results:
|
|
||||||
remaining = pending - {done}
|
|
||||||
for t in remaining:
|
|
||||||
t.cancel()
|
|
||||||
break
|
|
||||||
|
|
||||||
self._cache[cache_key] = all_results
|
|
||||||
return all_results
|
|
||||||
|
|
||||||
async def _safe_search(self, engine, query, max_results, **kwargs):
|
|
||||||
async with self._semaphore:
|
|
||||||
try:
|
|
||||||
return await asyncio.wait_for(
|
|
||||||
engine.search(query, max_results=max_results, **kwargs),
|
|
||||||
timeout=self._timeout,
|
|
||||||
)
|
|
||||||
except asyncio.TimeoutError:
|
|
||||||
logger.warning(f"Engine {engine.name} timed out after {self._timeout}s")
|
|
||||||
return []
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Engine {engine.name} failed: {e}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
async def search_health(self) -> list[dict[str, Any]]:
|
|
||||||
"""Check health of all configured engines (concurrent)."""
|
|
||||||
seen_names: set[str] = set()
|
|
||||||
tasks: list[asyncio.Task] = []
|
|
||||||
for cat_fn in SOURCE_CATEGORIES.values():
|
|
||||||
for engine_def in cat_fn():
|
|
||||||
name = engine_def.get("name", "?")
|
|
||||||
if name in seen_names:
|
|
||||||
continue
|
|
||||||
seen_names.add(name)
|
|
||||||
tasks.append(asyncio.create_task(self._check_one_engine(engine_def, name)))
|
|
||||||
|
|
||||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
|
||||||
statuses: list[dict[str, Any]] = []
|
|
||||||
for r in results:
|
|
||||||
if isinstance(r, Exception):
|
|
||||||
statuses.append({"name": "?", "status": "error", "detail": str(r)[:80]})
|
|
||||||
else:
|
|
||||||
statuses.append(r)
|
|
||||||
return statuses
|
|
||||||
|
|
||||||
async def _check_one_engine(
|
|
||||||
self, engine_def: dict[str, Any], name: str,
|
|
||||||
) -> dict[str, Any]:
|
|
||||||
"""Check health of a single engine."""
|
|
||||||
engine = self._get_engine(engine_def)
|
|
||||||
if engine is None:
|
|
||||||
return {"name": name, "status": "error", "detail": "engine_init_failed"}
|
|
||||||
try:
|
|
||||||
async with self._semaphore:
|
|
||||||
await asyncio.wait_for(
|
|
||||||
engine.search("health check", max_results=1),
|
|
||||||
timeout=8,
|
|
||||||
)
|
|
||||||
return {"name": name, "status": "ok"}
|
|
||||||
except asyncio.TimeoutError:
|
|
||||||
return {"name": name, "status": "timeout"}
|
|
||||||
except Exception as e:
|
|
||||||
return {"name": name, "status": "error", "detail": str(e)[:80]}
|
|
||||||
|
|
||||||
def _get_engine(self, engine_def: dict[str, Any]):
|
|
||||||
"""Get or create an engine instance by definition."""
|
|
||||||
name = engine_def.get("name", "")
|
|
||||||
engine_type = engine_def.get("type", "url")
|
|
||||||
|
|
||||||
if name in self._engines:
|
|
||||||
return self._engines[name]
|
|
||||||
|
|
||||||
if name == "DuckDuckGo":
|
|
||||||
engine = DuckDuckGoEngine(self.config)
|
|
||||||
elif engine_type == "baidu_qianfan" or "baidu qianfan" in name.lower():
|
|
||||||
engine = BaiduQianFanEngine(self.config)
|
|
||||||
elif "cls" in name.lower() or "财联社" in name:
|
|
||||||
engine = CLSNewsEngine(self.config)
|
|
||||||
elif "wallstreet" in name.lower() or "华尔街" in name:
|
|
||||||
engine = WallStreetCNEngine(self.config)
|
|
||||||
elif name == "ArXiv" or engine_type == "api":
|
|
||||||
engine = ArXivEngine(self.config)
|
|
||||||
elif "微信" in name or "wechat" in name.lower() or "weixin" in name.lower():
|
|
||||||
engine = SogouWeChatEngine(self.config)
|
|
||||||
elif "twitter" in name.lower():
|
|
||||||
engine = TwitterSearchEngine(self.config)
|
|
||||||
elif name == "Hacker News":
|
|
||||||
engine = HackerNewsEngine(self.config)
|
|
||||||
elif name == "GitHub Trending":
|
|
||||||
engine = GitHubTrendingEngine(self.config)
|
|
||||||
elif name == "36氪":
|
|
||||||
engine = Kr36Engine(self.config)
|
|
||||||
elif name == "微博热搜":
|
|
||||||
engine = WeiboEngine(self.config)
|
|
||||||
elif name == "V2EX":
|
|
||||||
engine = V2EXEngine(self.config)
|
|
||||||
elif name == "腾讯新闻":
|
|
||||||
engine = TencentNewsEngine(self.config)
|
|
||||||
elif engine_type == "news" and name not in ("cls", "wallstreetcn", "Hacker News", "GitHub Trending", "36氪", "微博热搜", "V2EX", "腾讯新闻"):
|
|
||||||
engine = RSSNewsEngine(self.config)
|
|
||||||
elif engine_type == "social":
|
|
||||||
engine = TwitterSearchEngine(self.config) if "twitter" in name.lower() else UrlSearchEngine(name, engine_def)
|
|
||||||
else:
|
else:
|
||||||
engine = UrlSearchEngine(name, engine_def)
|
selected = dict(self._engines)
|
||||||
|
|
||||||
self._engines[name] = engine
|
# Build region kwarg
|
||||||
return engine
|
eng_kwargs = {"region": region} if region else {}
|
||||||
|
|
||||||
async def search_engines(self, query, region=None, max_results=10):
|
# Run all engines concurrently with semaphore
|
||||||
"""Quick search across web search engines."""
|
semaphore = asyncio.Semaphore(self.max_concurrent)
|
||||||
engines = get_search_engines()
|
|
||||||
if region:
|
|
||||||
engines = [e for e in engines if e.get("region") == region]
|
|
||||||
return await self.search(query, sources=[e["name"] for e in engines], max_results=max_results)
|
|
||||||
|
|
||||||
async def search_news(self, query, max_results=10, **kwargs):
|
async def run_engine(name: str, eng: BaseEngine) -> tuple[str, list[SearchResult], float]:
|
||||||
"""Quick search news sources (CN news + RSS feeds)."""
|
async with semaphore:
|
||||||
return await self.search(query, sources=["news"], max_results=max_results, **kwargs)
|
start = time.time()
|
||||||
|
try:
|
||||||
|
results = await asyncio.wait_for(
|
||||||
|
eng.search(query, max_results=max_results, **eng_kwargs),
|
||||||
|
timeout=eng.timeout + 2.0,
|
||||||
|
)
|
||||||
|
elapsed = time.time() - start
|
||||||
|
return name, list(results), elapsed
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
return name, [], time.time() - start
|
||||||
|
except Exception:
|
||||||
|
return name, [], time.time() - start
|
||||||
|
|
||||||
async def search_academic(self, query, max_results=10, **kwargs):
|
tasks = [run_engine(n, e) for n, e in selected.items()]
|
||||||
"""Quick search academic papers."""
|
done = await asyncio.gather(*tasks, return_exceptions=True)
|
||||||
return await self.search(query, sources=["academic"], max_results=max_results, **kwargs)
|
|
||||||
|
|
||||||
async def search_wechat(self, query, max_results=10):
|
# Collect results and timings
|
||||||
"""Quick search WeChat articles."""
|
all_results: list[SearchResult] = []
|
||||||
return await self.search(query, sources=["wechat"], max_results=max_results)
|
engine_timings: list[EngineTiming] = []
|
||||||
|
engines_used: set[str] = set()
|
||||||
|
|
||||||
async def search_social(self, query, max_results=10, **kwargs):
|
for item in done:
|
||||||
"""Quick search social media (Twitter/X)."""
|
if isinstance(item, tuple):
|
||||||
return await self.search(query, sources=["social"], max_results=max_results, **kwargs)
|
name, results, elapsed = item
|
||||||
|
all_results.extend(results)
|
||||||
|
if results:
|
||||||
|
engines_used.add(name)
|
||||||
|
engine_timings.append(EngineTiming(
|
||||||
|
engine_name=name,
|
||||||
|
success=bool(results),
|
||||||
|
duration_ms=int(elapsed * 1000),
|
||||||
|
result_count=len(results),
|
||||||
|
))
|
||||||
|
elif isinstance(item, Exception):
|
||||||
|
# Exception in gather — log but don't crash
|
||||||
|
pass
|
||||||
|
|
||||||
async def search_rss(self, query="", max_results=10, **kwargs):
|
# Deduplicate and rank
|
||||||
"""Quick search RSS news feeds."""
|
aggregator = ResultAggregator()
|
||||||
engine = RSSNewsEngine(self.config)
|
merged = aggregator.merge_and_rank(all_results, max_results=max_results)
|
||||||
return await engine.search(query, max_results=max_results, **kwargs)
|
|
||||||
|
|
||||||
async def get_search_urls(self, query, sources=None):
|
elapsed_ms = int((time.time() - t0) * 1000)
|
||||||
"""Get search URLs without executing searches."""
|
|
||||||
if not sources:
|
|
||||||
sources = [e["name"] for e in get_search_engines()]
|
|
||||||
|
|
||||||
urls = {}
|
# Build SourceRef list
|
||||||
for name in sources:
|
sources_list = [
|
||||||
engine_def = get_engine(name)
|
SourceRef(
|
||||||
if engine_def:
|
index=r.rank,
|
||||||
urls[name] = UrlSearchEngine(name, engine_def).build_url(query)
|
title=r.title,
|
||||||
|
url=r.url,
|
||||||
|
source=r.source,
|
||||||
|
)
|
||||||
|
for r in merged
|
||||||
|
]
|
||||||
|
|
||||||
return urls
|
metadata = SearchMetadata(
|
||||||
|
total_engines=len(selected),
|
||||||
|
total_raw_results=len(all_results),
|
||||||
|
engines_used=list(engines_used),
|
||||||
|
duration_ms=elapsed_ms,
|
||||||
|
engine_timings=engine_timings,
|
||||||
|
)
|
||||||
|
|
||||||
async def get_twitter_search_url(self, query, search_type="top"):
|
return SearchResponse(
|
||||||
"""Get Twitter/X search URL for browser/web_fetch usage."""
|
query=query,
|
||||||
engine = TwitterSearchEngine(self.config)
|
results=merged,
|
||||||
return engine.build_url(query, search_type=search_type)
|
sources=sources_list,
|
||||||
|
metadata=metadata,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def health(self) -> dict[str, Any]:
|
||||||
|
"""Check connectivity of all engines."""
|
||||||
|
results = {}
|
||||||
|
for name, eng in self._engines.items():
|
||||||
|
try:
|
||||||
|
health = await asyncio.wait_for(eng.health(), timeout=5.0)
|
||||||
|
results[name] = health
|
||||||
|
except Exception as e:
|
||||||
|
results[name] = {"name": name, "status": "fail", "error": str(e)}
|
||||||
|
return results
|
||||||
|
|
||||||
async def close(self):
|
async def close(self):
|
||||||
"""Close all engine resources."""
|
"""Cleanup (no-op for current implementation)."""
|
||||||
for engine in self._engines.values():
|
pass
|
||||||
try:
|
|
||||||
await engine.close()
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
"""WeChat search engines."""
|
|
||||||
@@ -1,365 +0,0 @@
|
|||||||
"""
|
|
||||||
Sogou WeChat (搜狗微信) search engine.
|
|
||||||
|
|
||||||
Searches WeChat public account articles via weixin.sogou.com.
|
|
||||||
|
|
||||||
Anti-scraping strategy (inspired by Node.js wechat-article-search skill):
|
|
||||||
- Cookie pre-fetch from v.sogou.com to get SNUID
|
|
||||||
- User-Agent rotation with a pool of real browser UAs
|
|
||||||
- Random delays between requests (500-1500ms)
|
|
||||||
- Retry logic with exponential backoff
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import logging
|
|
||||||
import random
|
|
||||||
import re
|
|
||||||
from typing import Optional
|
|
||||||
from urllib.parse import quote_plus
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
|
|
||||||
from scripts.core.base import SearchEngine, SearchResult
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
# Real browser User-Agent pool for rotation
|
|
||||||
USER_AGENTS = [
|
|
||||||
# Chrome
|
|
||||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
|
||||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36",
|
|
||||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
|
|
||||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
|
||||||
# Safari
|
|
||||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Safari/605.1.15",
|
|
||||||
"Mozilla/5.0 (iPhone; CPU iPhone OS 17_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Mobile/15E148 Safari/604.1",
|
|
||||||
# Firefox
|
|
||||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0",
|
|
||||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0",
|
|
||||||
# Edge
|
|
||||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
class SogouWeChatEngine(SearchEngine):
|
|
||||||
"""搜狗微信搜索 - searches WeChat public account articles."""
|
|
||||||
|
|
||||||
SEARCH_URL = "https://weixin.sogou.com/weixin"
|
|
||||||
COOKIE_URL = "https://v.sogou.com"
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[dict] = None):
|
|
||||||
super().__init__("weixin", config)
|
|
||||||
timeout = (config or {}).get("timeout", 15)
|
|
||||||
self._ua = random.choice(USER_AGENTS)
|
|
||||||
self._cookies: dict[str, str] = {}
|
|
||||||
self._client = httpx.AsyncClient(
|
|
||||||
timeout=timeout,
|
|
||||||
headers=self._build_headers(),
|
|
||||||
follow_redirects=False, # Don't auto-follow to detect anti-scraping
|
|
||||||
)
|
|
||||||
|
|
||||||
def _build_headers(self) -> dict[str, str]:
|
|
||||||
return {
|
|
||||||
"User-Agent": self._ua,
|
|
||||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
|
||||||
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
|
|
||||||
"Accept-Encoding": "gzip, deflate",
|
|
||||||
"Referer": "https://weixin.sogou.com/",
|
|
||||||
"DNT": "1",
|
|
||||||
"Connection": "keep-alive",
|
|
||||||
}
|
|
||||||
|
|
||||||
# ---- Anti-scraping enhancements from bot-search-mcp ---- #
|
|
||||||
|
|
||||||
CAPTCHA_KEYWORDS = [
|
|
||||||
"请输入验证码", "访问频率过高", "请稍后重试", "验证码",
|
|
||||||
"captcha", "访问过于频繁", "请输入以下验证码",
|
|
||||||
"403 Forbidden", "您的访问被拒绝",
|
|
||||||
]
|
|
||||||
|
|
||||||
def _random_delay(self, min_s: float = 0.5, max_s: float = 1.5):
|
|
||||||
"""Random delay to mimic human browsing."""
|
|
||||||
delay = random.uniform(min_s, max_s)
|
|
||||||
return asyncio.sleep(delay)
|
|
||||||
|
|
||||||
def _rotate_ua(self):
|
|
||||||
"""Rotate to a new random User-Agent."""
|
|
||||||
self._ua = random.choice(USER_AGENTS)
|
|
||||||
|
|
||||||
async def _get_cookie(self) -> dict[str, str]:
|
|
||||||
"""Pre-fetch cookie from v.sogou.com to get SNUID/SUID/IPLOC/SUV."""
|
|
||||||
try:
|
|
||||||
response = await self._client.get(
|
|
||||||
self.COOKIE_URL,
|
|
||||||
headers={
|
|
||||||
"User-Agent": self._ua,
|
|
||||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
|
||||||
"Accept-Language": "zh-CN,zh;q=0.9",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
cookies: dict[str, str] = {}
|
|
||||||
|
|
||||||
# Extract from httpx Cookies
|
|
||||||
for key, value in response.cookies.items():
|
|
||||||
if key.upper() in ("SNUID", "SUID", "IPLOC", "SUV"):
|
|
||||||
cookies[key] = value
|
|
||||||
|
|
||||||
# Also parse from Set-Cookie header (more reliable)
|
|
||||||
set_cookie = response.headers.get("set-cookie", "")
|
|
||||||
for part in set_cookie.split(";"):
|
|
||||||
if "=" in part:
|
|
||||||
k, v = part.split("=", 1)
|
|
||||||
k = k.strip()
|
|
||||||
if k.upper() in ("SNUID", "SUID", "IPLOC", "SUV"):
|
|
||||||
cookies[k] = v.split(",")[-1].strip()
|
|
||||||
|
|
||||||
logger.debug(f"Got {len(cookies)} cookies from v.sogou.com: {list(cookies.keys())}")
|
|
||||||
return cookies
|
|
||||||
except Exception as e:
|
|
||||||
logger.warning(f"Cookie pre-fetch failed: {e}")
|
|
||||||
return {}
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _detect_captcha(html: str) -> bool:
|
|
||||||
"""检测是否触发了验证码或反爬。"""
|
|
||||||
html_lower = html.lower()
|
|
||||||
for kw in SogouWeChatEngine.CAPTCHA_KEYWORDS:
|
|
||||||
if kw in html_lower or kw.lower() in html_lower:
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
async def _exponential_backoff(
|
|
||||||
self, attempt: int, base_delay: float = 2.0
|
|
||||||
):
|
|
||||||
"""指数退避等待(带随机 jitter)。"""
|
|
||||||
wait = (2 ** attempt) + random.uniform(0, 1)
|
|
||||||
await asyncio.sleep(wait)
|
|
||||||
|
|
||||||
def build_url(self, query: str, search_type: int = 2, page: int = 1, **kwargs) -> str:
|
|
||||||
"""
|
|
||||||
Build Sogou WeChat search URL.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
query: Search keyword.
|
|
||||||
search_type: 2=articles, 1=accounts.
|
|
||||||
page: Page number (starts at 1).
|
|
||||||
"""
|
|
||||||
return (
|
|
||||||
f"{self.SEARCH_URL}?query={quote_plus(query)}"
|
|
||||||
f"&s_from=input&_sug_=n&type={search_type}"
|
|
||||||
f"&page={page}&ie=utf8"
|
|
||||||
)
|
|
||||||
|
|
||||||
async def search(
|
|
||||||
self, query: str, max_results: int = 10, **kwargs
|
|
||||||
) -> list[SearchResult]:
|
|
||||||
"""Search WeChat articles via Sogou and parse results."""
|
|
||||||
# Step 1: Get cookie first
|
|
||||||
if not self._cookies:
|
|
||||||
self._cookies = await self._get_cookie()
|
|
||||||
await self._random_delay()
|
|
||||||
|
|
||||||
# Step 2: Search with retry
|
|
||||||
max_pages = min(5, (max_results + 9) // 10)
|
|
||||||
all_results = []
|
|
||||||
|
|
||||||
for page in range(1, max_pages + 1):
|
|
||||||
url = self.build_url(query, page=page, **kwargs)
|
|
||||||
results = await self._search_page(url, query, max_results - len(all_results))
|
|
||||||
all_results.extend(results)
|
|
||||||
|
|
||||||
if len(all_results) >= max_results:
|
|
||||||
break
|
|
||||||
|
|
||||||
# Delay between pages
|
|
||||||
await self._random_delay()
|
|
||||||
|
|
||||||
if not all_results:
|
|
||||||
# Fallback: try legacy wx.sogou.com
|
|
||||||
return await self._legacy_search(query, max_results, **kwargs)
|
|
||||||
|
|
||||||
return all_results[:max_results]
|
|
||||||
|
|
||||||
async def _search_page(
|
|
||||||
self, url: str, query: str, max_results: int
|
|
||||||
) -> list[SearchResult]:
|
|
||||||
"""Search a single page with enhanced retry + anti-scraping logic."""
|
|
||||||
for attempt in range(3):
|
|
||||||
try:
|
|
||||||
# Rotate UA on each retry
|
|
||||||
if attempt > 0:
|
|
||||||
self._rotate_ua()
|
|
||||||
self._client.headers.update({"User-Agent": self._ua})
|
|
||||||
|
|
||||||
response = await self._client.get(
|
|
||||||
url,
|
|
||||||
cookies=self._cookies,
|
|
||||||
follow_redirects=False,
|
|
||||||
)
|
|
||||||
|
|
||||||
# 302 = anti-scraping redirect
|
|
||||||
if response.status_code == 302:
|
|
||||||
logger.warning(
|
|
||||||
f"Sogou anti-scraping triggered (302), "
|
|
||||||
f"attempt {attempt + 1}"
|
|
||||||
)
|
|
||||||
self._cookies = await self._get_cookie()
|
|
||||||
await self._exponential_backoff(attempt)
|
|
||||||
continue
|
|
||||||
|
|
||||||
if response.status_code != 200:
|
|
||||||
logger.warning(
|
|
||||||
f"Sogou returned {response.status_code}, "
|
|
||||||
f"attempt {attempt + 1}"
|
|
||||||
)
|
|
||||||
await self._exponential_backoff(attempt)
|
|
||||||
continue
|
|
||||||
|
|
||||||
html = response.text
|
|
||||||
|
|
||||||
# Detect captcha / block page
|
|
||||||
if self._detect_captcha(html):
|
|
||||||
logger.warning(
|
|
||||||
f"Sogou captcha/block detected, "
|
|
||||||
f"attempt {attempt + 1}"
|
|
||||||
)
|
|
||||||
await self._exponential_backoff(attempt)
|
|
||||||
self._cookies = await self._get_cookie()
|
|
||||||
continue
|
|
||||||
|
|
||||||
return self._parse_results(html, query, max_results)
|
|
||||||
|
|
||||||
except (httpx.HTTPError, httpx.TimeoutException) as e:
|
|
||||||
logger.warning(
|
|
||||||
f"Sogou search attempt {attempt + 1} failed: {e}"
|
|
||||||
)
|
|
||||||
await self._exponential_backoff(attempt)
|
|
||||||
continue
|
|
||||||
|
|
||||||
return []
|
|
||||||
|
|
||||||
async def _legacy_search(
|
|
||||||
self, query: str, max_results: int, **kwargs
|
|
||||||
) -> list[SearchResult]:
|
|
||||||
"""Fallback: try the legacy wx.sogou.com domain."""
|
|
||||||
legacy_url = f"https://wx.sogou.com/weixin?type=2&query={quote_plus(query)}"
|
|
||||||
try:
|
|
||||||
response = await self._client.get(
|
|
||||||
legacy_url,
|
|
||||||
cookies=self._cookies,
|
|
||||||
follow_redirects=True,
|
|
||||||
)
|
|
||||||
if response.status_code == 200:
|
|
||||||
return self._parse_results(response.text, query, max_results)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Ultimate fallback
|
|
||||||
search_url = self.build_url(query)
|
|
||||||
return [
|
|
||||||
SearchResult(
|
|
||||||
title=f"微信搜索: {query}",
|
|
||||||
url=search_url,
|
|
||||||
snippet=f"Search WeChat for '{query}' via Sogou",
|
|
||||||
source=self.name,
|
|
||||||
rank=1,
|
|
||||||
category="wechat",
|
|
||||||
extra={"search_url": search_url},
|
|
||||||
)
|
|
||||||
]
|
|
||||||
|
|
||||||
def _parse_results(
|
|
||||||
self, html: str, query: str, max_results: int
|
|
||||||
) -> list[SearchResult]:
|
|
||||||
"""Parse Sogou WeChat search results."""
|
|
||||||
soup = BeautifulSoup(html, "lxml")
|
|
||||||
results = []
|
|
||||||
|
|
||||||
# Main selector: ul.news-list > li (standard Sogou WeChat structure)
|
|
||||||
items = soup.select("ul.news-list > li")
|
|
||||||
|
|
||||||
if not items:
|
|
||||||
# Fallback selectors
|
|
||||||
items = soup.select(
|
|
||||||
".news-list2 li, .wx-rb-item, .results .result, "
|
|
||||||
".item, .news-box .news-list li"
|
|
||||||
)
|
|
||||||
|
|
||||||
for i, item in enumerate(items[:max_results]):
|
|
||||||
title_el = item.select_one("h3 a")
|
|
||||||
if not title_el:
|
|
||||||
title_el = item.select_one(".tit a, .title a, h3")
|
|
||||||
|
|
||||||
snippet_el = item.select_one("p.txt-info, .txt-info, .summary, .des, p.info")
|
|
||||||
source_el = item.select_one(
|
|
||||||
".all-time-y2 a.account, a.account, "
|
|
||||||
".account, .source, .s-p, .from"
|
|
||||||
)
|
|
||||||
|
|
||||||
title = title_el.get_text(strip=True) if title_el else ""
|
|
||||||
link = title_el.get("href", "") if title_el and hasattr(title_el, "get") else ""
|
|
||||||
snippet = snippet_el.get_text(strip=True) if snippet_el else ""
|
|
||||||
account = source_el.get_text(strip=True) if source_el else ""
|
|
||||||
|
|
||||||
# Parse time from inline scripts or text
|
|
||||||
time_text = self._parse_time(item)
|
|
||||||
|
|
||||||
if not title:
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Make sure URL is absolute
|
|
||||||
if link and not link.startswith("http"):
|
|
||||||
link = f"https://weixin.sogou.com{link}"
|
|
||||||
|
|
||||||
results.append(
|
|
||||||
SearchResult(
|
|
||||||
title=title,
|
|
||||||
url=link,
|
|
||||||
snippet=snippet or f"WeChat article from {account or 'unknown'}",
|
|
||||||
source=self.name,
|
|
||||||
rank=(i + 1),
|
|
||||||
category="wechat",
|
|
||||||
extra={
|
|
||||||
"account": account,
|
|
||||||
"time": time_text,
|
|
||||||
"search_url": self.build_url(query),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
return results
|
|
||||||
|
|
||||||
def _parse_time(self, item) -> str:
|
|
||||||
"""Extract publish time from search result item."""
|
|
||||||
# Try script tag with Unix timestamp (Sogou encodes dates in scripts)
|
|
||||||
for script in item.select("script"):
|
|
||||||
text = script.string or ""
|
|
||||||
# Look for Unix timestamp pattern
|
|
||||||
match = re.search(r'\d{10}', text)
|
|
||||||
if match:
|
|
||||||
import datetime
|
|
||||||
ts = int(match.group())
|
|
||||||
try:
|
|
||||||
dt = datetime.datetime.fromtimestamp(ts, tz=datetime.timezone(datetime.timedelta(hours=8)))
|
|
||||||
return dt.strftime("%Y-%m-%d %H:%M")
|
|
||||||
except (OSError, ValueError):
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Try inline text with relative time like "2小时前", "3天前"
|
|
||||||
time_text = item.get_text()
|
|
||||||
relative_match = re.search(r'(\d+)\s*(小时|分钟|天|周|月)前', time_text)
|
|
||||||
if relative_match:
|
|
||||||
return f"{relative_match.group(1)}{relative_match.group(2)}前"
|
|
||||||
|
|
||||||
# Try standard date patterns
|
|
||||||
date_match = re.search(r'(\d{4}[-/]\d{1,2}[-/]\d{1,2})', time_text)
|
|
||||||
if date_match:
|
|
||||||
return date_match.group(1)
|
|
||||||
|
|
||||||
return ""
|
|
||||||
|
|
||||||
async def close(self):
|
|
||||||
await self._client.aclose()
|
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
from setuptools import setup, find_packages
|
||||||
|
|
||||||
|
setup(
|
||||||
|
name="web-search-skills",
|
||||||
|
version="0.2.0",
|
||||||
|
description="Open-source diversified search in <5s",
|
||||||
|
author="openbot-coder",
|
||||||
|
packages=find_packages(),
|
||||||
|
py_modules=["ws"],
|
||||||
|
python_requires=">=3.10",
|
||||||
|
install_requires=[
|
||||||
|
"httpx>=0.24.0",
|
||||||
|
"beautifulsoup4>=4.12.0",
|
||||||
|
"lxml>=5.0.0",
|
||||||
|
],
|
||||||
|
entry_points={
|
||||||
|
"console_scripts": [
|
||||||
|
"web-search=ws:main",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
@@ -1,21 +1,20 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""
|
"""
|
||||||
Web Search Skills CLI entry point.
|
Lightweight launcher — delegates to scripts.cli:main().
|
||||||
|
|
||||||
Usage:
|
Usage:
|
||||||
python ws.py <command> <query> [options]
|
python3 ws.py search "quantum computing" --max-results 5
|
||||||
|
python3 ws.py health
|
||||||
Or install and run:
|
python3 ws.py --help
|
||||||
uv tool install -e .
|
|
||||||
ws <command> <query> [options]
|
|
||||||
"""
|
"""
|
||||||
import sys
|
|
||||||
import os
|
|
||||||
|
|
||||||
# Ensure scripts package is importable
|
import sys
|
||||||
_SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
from pathlib import Path
|
||||||
if _SCRIPT_DIR not in sys.path:
|
|
||||||
sys.path.insert(0, _SCRIPT_DIR)
|
# Ensure the project root is on the path
|
||||||
|
_root = Path(__file__).parent
|
||||||
|
if str(_root) not in sys.path:
|
||||||
|
sys.path.insert(0, str(_root))
|
||||||
|
|
||||||
from scripts.cli import main
|
from scripts.cli import main
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user