Files
openbot-coder c077a7ae40 refactor: cleanup dead code, fix model confusion, streamline CLI
- Remove 3 dead modules: academic/, wechat/, social/ (merged into engines/)
- Remove 4 stale news scripts (tech_news.py, wallstreetcn.py, cls.py, rss.py)
- scripts/core/base.py: canonical SearchResult + BaseEngine (was in engines/base.py)
- scripts/engines/base.py: re-exports from core.base for compatibility
- scripts/engines/news_engine.py: Bing News + RSS unified (was in scripts/news/rss.py)
- scripts/engines/academic_engine.py: ArXiv + CrossRef unified (was in scripts/academic/arxiv.py)
- scripts/cli.py: complete rewrite — 10 commands (search/web/news/academic/wechat/social/rss/baidu/health/sources)
- scripts/unified_search.py: clean async Python API (was a stub)
- ws.py: lightweight launcher
- README.md + SKILL.md: updated to reflect actual 15+ engines
- All EngineResult → SearchResult
- All engines return list[SearchResult]
2026-05-30 16:25:41 +08:00

3.8 KiB

web-search-skills

Multi-source web search CLI & Python library — 15+ engines, <5s, no API key required.

Features

  • 15+ engines across 5 categories: Web, News, Academic, Social, RSS
  • Concurrent execution via asyncio.gather with semaphore (max 5 parallel)
  • Free by default — most engines work without any API key
  • Python API for programmatic use (UnifiedSearch)
  • Single model — SearchResult dataclass across all engines

Quick Start

# Install
pip install -e .

# CLI
web-search search "AI news" -n 10
web-search news "quantum computing" -n 5
web-search academic "machine learning"
web-search wechat 蒙面财经
web-search health

Commands

Command Description Speed
search All 15 engines in parallel ~10s
web Web engines only (6) ~5s
news Bing News + 15 RSS feeds ~8s
academic ArXiv + CrossRef in parallel ~6s
wechat WeChat via Sogou ~5s
social Twitter/X via Nitter ~5s
rss RSS feed aggregation ~8s
baidu Baidu Qianfan API (needs key) varies
health Engine connectivity check ~10s
sources List all sources instant

Options

  • -n N — max results per engine (default: 10)
  • -s SRC... — run specific sources only
  • -c CAT — run by category (web/news/academic/social/rss)
  • -r cn|global — region (default: cn)
  • -j — JSON output

Python API

from scripts.unified_search import UnifiedSearch

async def main():
    us = UnifiedSearch()
    resp = await us.search("quantum computing", sources=["bing", "arxiv"], max_results=5)
    for r in resp.results:
        print(f"[{r.rank}] {r.title}")
        print(f"    {r.url}")
    await us.close()

Architecture

scripts/
  cli.py              # CLI entry point
  unified_search.py   # Python API
  models.py           # Data models
  config.py           # Settings
  core/
    base.py           # SearchResult + BaseEngine (canonical)
    aggregator.py     # Deduplication + ranking
    execution_log.py  # Timing + error tracking
  engines/
    bing_scraper.py     # Bing web search
    sogou_scraper.py    # Sogou web search
    baidu_scraper.py    # Baidu web search
    search360_scraper.py # 360 search
    parser_engines.py   # DuckDuckGo + Brave (HTML parser)
    news_engine.py      # Bing News + RSS aggregator
    academic_engine.py   # ArXiv + CrossRef
    github_engine.py    # GitHub API
    baidu_qianfan.py    # Baidu Qianfan API (needs key)
    parallel_search.py  # Parallel orchestrator

Engines

Web (无需 API key)

Engine URL Notes
Bing CN cn.bing.com Primary
Bing INT www.bing.com Global fallback
Sogou sogou.com Primary CN
Baidu baidu.com Primary CN
360 so.com CN fallback
DuckDuckGo html.duckduckgo.com Global
Brave search.brave.com Global

News

Engine Source
Bing News bing.com/news
RSS (15 feeds) 36kr, IT之家, 钛媒体, HackerNews, TechCrunch...

Academic (Free APIs)

Engine API
ArXiv export.arxiv.org/api/query
CrossRef api.crossref.org

Social

Engine Notes
Nitter twitter.com 免登录 (多个实例自动切换)

API (需配置密钥)

Engine Env var
Baidu Qianfan BAIDU_API_KEY

SearchResult Model

@dataclass
class SearchResult:
    title: str       # Article/title text
    url: str         # Direct URL
    snippet: str     # Summary text (~300 chars)
    source: str     # Engine name (e.g. "bing", "arxiv")
    category: str    # "web" | "news" | "academic" | "github"
    rank: int        # Position after deduplication
    published: str   # ISO date (optional)
    extra: dict      # Engine-specific metadata