refactor: restructure project with CLI tool, multi-engine support, and skill-builder compliant SKILL.md
- Replace setup.py with pyproject.toml (uv-compatible) - Add CLI entry point (web-search) with 9 subcommands - Add multi-engine architecture (14 web + 7 special sources) - Add WeChat/CLS/WallStreetCN/ArXiv/Twitter search engines - Remove old src/ and examples/ directories - Remove redundant requirements.txt and setup.py - Update README.md and SKILL.md with full documentation - SKILL.md now compliant with skill-builder standards (frontmatter, triggers, structure)
This commit is contained in:
1 parent
80ac14491e
commit
87935b59ce
29 files changed
+2364
-354
No files matched your search
@@ -1,95 +1,86 @@
|
||||
# Web Search Skills
|
||||
|
||||
[](LICENSE)
|
||||
Multi-source web search CLI tool — search across 20+ engines from your terminal.
|
||||
|
||||
A collection of web search tools and skills for AI agents, designed to enhance search capabilities with customizable configurations, multiple search engines support, and intelligent result processing.
|
||||
|
||||
## Features
|
||||
|
||||
- **Multi-Engine Support** — Built-in support for Google, Bing, DuckDuckGo, and custom search backends
|
||||
- **Intelligent Result Processing** — Extract, summarize, and rank search results automatically
|
||||
- **Configurable Filters** — Filter by date, domain, language, region, and content type
|
||||
- **Async & Concurrent** — Non-blocking requests with concurrent search execution
|
||||
- **Extensible Architecture** — Plugin-based design for easy integration of new search sources
|
||||
- **Agent-Ready** — Designed as a CodeBuddy Agent Skill for seamless AI integration
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.8+
|
||||
- pip / poetry
|
||||
|
||||
### Installation
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
git clone https://github.com/openbot-coder/web-search-skills.git
|
||||
# Install with uv (recommended)
|
||||
cd web-search-skills
|
||||
pip install -r requirements.txt
|
||||
uv tool install -e .
|
||||
|
||||
# Now use globally from anywhere
|
||||
web-search wechat 蒙面财经
|
||||
web-search news "AI" -n 5
|
||||
web-search search "量子计算" -o results.json
|
||||
```
|
||||
|
||||
### Quick Start
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from src.web_search import search_web
|
||||
|
||||
results = asyncio.run(search_web("latest AI developments"))
|
||||
for result in results:
|
||||
print(f"- {result.title}: {result.url}")
|
||||
Or run directly without install:
|
||||
```bash
|
||||
python ws.py wechat 蒙面财经
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
web-search <command> <query> [options]
|
||||
```
|
||||
|
||||
| Command | Description |
|
||||
|------------|------------------------------------------|
|
||||
| `search` | Search all sources (default) |
|
||||
| `web` | Web search engines (Google, Baidu, DDG) |
|
||||
| `news` | News (财联社, 华尔街见闻, RSS) |
|
||||
| `wechat` | WeChat public account articles |
|
||||
| `academic` | Academic papers (ArXiv) |
|
||||
| `social` | Social media (Twitter/X) |
|
||||
| `rss` | RSS news feeds |
|
||||
| `sources` | List all available search sources |
|
||||
| `urls` | Generate search URLs without executing |
|
||||
|
||||
Options:
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `-n N` | Max results per source (default: 10) |
|
||||
| `-j` | JSON output |
|
||||
| `-o FILE` | Save to JSON file |
|
||||
| `-s NAME` | Specific source name(s) |
|
||||
| `-r cn/global` | Region filter (web search) |
|
||||
| `-v` | Verbose/debug logs |
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
web-search-skills/
|
||||
├── README.md # Project documentation
|
||||
├── LICENSE # MIT License
|
||||
├── skill.md # CodeBuddy Agent Skill definition
|
||||
├── requirements.txt # Python dependencies
|
||||
├── src/ # Source code
|
||||
│ └── web_search.py # Core search implementation
|
||||
└── examples/ # Usage examples
|
||||
└── basic_search.py
|
||||
├── scripts/
|
||||
│ ├── cli.py # CLI entry point
|
||||
│ ├── unified_search.py # Unified search orchestrator
|
||||
│ ├── core/
|
||||
│ │ ├── base.py # SearchResult & SearchEngine base class
|
||||
│ │ └── config_loader.py # Load engine config from JSON
|
||||
│ ├── engines/
|
||||
│ │ ├── url_engine.py # URL-template search engines
|
||||
│ │ └── parser_engines.py # HTML-parsed search (DuckDuckGo)
|
||||
│ ├── news/
|
||||
│ │ ├── cls.py # 财联社
|
||||
│ │ ├── wallstreetcn.py # 华尔街见闻
|
||||
│ │ └── rss.py # RSS feeds
|
||||
│ ├── academic/
|
||||
│ │ └── arxiv.py # ArXiv papers
|
||||
│ ├── wechat/
|
||||
│ │ └── sogou_weixin.py # WeChat (via Sogou)
|
||||
│ └── social/
|
||||
│ └── twitter.py # Twitter/X
|
||||
├── config/
|
||||
│ └── engines.json # Search engine definitions
|
||||
├── ws.py # Direct run entry point
|
||||
├── pyproject.toml # Package config
|
||||
├── SKILL.md # CodeBuddy skill definition
|
||||
├── LICENSE
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### As a CodeBuddy Agent Skill
|
||||
|
||||
This project is designed as a CodeBuddy Agent Skill. To install:
|
||||
|
||||
1. Copy `skill.md` to your CodeBuddy skills directory
|
||||
2. Configure your search engine API keys
|
||||
3. The skill is now available to your AI agent
|
||||
|
||||
### API Reference
|
||||
|
||||
```python
|
||||
# Core class
|
||||
searcher = WebSearch(config=None)
|
||||
results = await searcher.search(query, engine="duckduckgo", max_results=10)
|
||||
content = await searcher.extract_content(url)
|
||||
|
||||
# Convenience function
|
||||
results = await search_web(query, engine="duckduckgo", max_results=10)
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
|
||||
|
||||
1. Fork the repository
|
||||
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
|
||||
3. Commit your changes
|
||||
4. Push to the branch
|
||||
5. Open a Pull Request
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
- Built for the CodeBuddy ecosystem
|
||||
- Inspired by the need for better web search capabilities in AI agents
|
||||
MIT
|
||||
Reference in new issue
Block a user