fix(pool): 重写当日测速缓存:全量候选 + 用户级缓存 + 真实数据探针
原实现在 v0.5.0 已随包发布,但有三处会"静默失效"的缺陷,等于没真正生效: - 候选集自指:连接池把自己已过滤后的成员当探针输入,实测只重测了 9 台, 另外 20+ 台候选(含快节点)永远测不到。改为由 probe_cache 自行枚举 全量候选(包内清单 ∪ 代码常量 ∪ env),不再接受调用方传入的"当前池成员"。 - 写回安装目录:结果写回 src/pytdxdata/servers.json,容器/系统级安装下只读会 静默失败、多用户共用一个安装会互相覆盖,且污染随包分发的数据。改为写入 用户级缓存目录(Win %LOCALAPPDATA% / macOS ~/Library/Caches / Linux $XDG_CACHE_HOME), 运行时绝不再改写包内清单。 - 探测太弱:只做握手,测不出「握手正常但 K 线空响应」的坏节点。改为发真实业务 命令并要求返回非空(标准 GetSecurityBarsCmd / MAC MacKlineCmd / EX GetExInstrumentBarsCmd), EX 池(4 台,7727)也一并纳入。 无感保证:后台 task 不阻塞 start()、日志全 debug 级、绝不抛异常(取消除外)、 原子写 tmp + os.replace、进程内单例 + asyncio.Lock 单飞;全不可达时不写缓存 (否则"今天全挂"会被缓存一整天)。close() 会取消并 await 后台探针, 避免解释器退出时报 "Task was destroyed but it is pending"。 新增环境变量 PYPYTDX_DATA_CACHE_DIR(覆盖缓存目录)、 PYPYTDX_DATA_DISABLE_PROBE=1(关闭测速)。新增 tests/test_probe_cache.py(37 项)。
This commit is contained in:
1 parent
3e0beb27a6
commit
18d9e9a006
9 files changed
+941
-134
No files matched your search
@@ -2,6 +2,38 @@
|
|||||||
|
|
||||||
本项目遵循 [语义化版本](https://semver.org/lang/zh-CN/)。
|
本项目遵循 [语义化版本](https://semver.org/lang/zh-CN/)。
|
||||||
|
|
||||||
|
## [Unreleased]
|
||||||
|
|
||||||
|
### 变更
|
||||||
|
|
||||||
|
- **重写「当日首次启动测速缓存」,改为对用户完全无感**。原实现在 v0.5.0 已随包发布,但存在
|
||||||
|
三个会让它**静默失效**的缺陷,本次一并修正:
|
||||||
|
|
||||||
|
| 缺陷 | 原实现 | 现在 |
|
||||||
|
|---|---|---|
|
||||||
|
| 候选集自指 | 连接池只把自己**已过滤后的成员**交给探针 → 只重测了 9 台,另外 20+ 台候选(含快节点)永远测不到 | 探针内部取**全部候选**(包内清单 ∪ 代码常量 ∪ env) |
|
||||||
|
| 写回安装目录 | 结果写回 `src/pytdxdata/servers.json` → 容器 / 系统级安装只读、多用户互相覆盖 | 改写入**用户级缓存目录**,包内清单运行时**绝不被改写** |
|
||||||
|
| 探测太弱 | 只做握手(`TdxConnection.test`),测不出「握手正常但 K 线空响应」的坏节点 | 发**真实业务命令**并要求返回非空(标准 / MAC / EX 各一) |
|
||||||
|
| EX 池漏测 | EX 的 4 台从未被探测 | EX 一并纳入(7727 + 各自握手模式) |
|
||||||
|
|
||||||
|
- 测速结果按**实测延迟升序**被 `config.get_standard_hosts()` 等采用;当天已有缓存则**不再刷新**。
|
||||||
|
后台执行,不阻塞 `start()`、不打印、失败静默降级。
|
||||||
|
|
||||||
|
### 新增
|
||||||
|
|
||||||
|
- 环境变量 `PYPYTDX_DATA_CACHE_DIR`(覆盖测速缓存目录)、
|
||||||
|
`PYPYTDX_DATA_DISABLE_PROBE`(关闭测速,CI / 测试用)
|
||||||
|
- `tests/test_probe_cache.py`(37 项):当日缓存不刷新、跨日重测、
|
||||||
|
原子写入且不留 tmp、**不改写包内 `servers.json`(逐字节校验)**、
|
||||||
|
单飞并发、非法主机过滤、`DISABLE_PROBE`、池 `start()` 不阻塞 / `close()` 取消后台探针
|
||||||
|
|
||||||
|
### 文档
|
||||||
|
|
||||||
|
- [安装与配置](https://openbot-coder.github.io/pytdxdata/getting-started/#当日自动测速无感):
|
||||||
|
新增「当日自动测速(无感)」小节——缓存路径、优先级链、环境变量一览
|
||||||
|
- [通道与数据源](https://openbot-coder.github.io/pytdxdata/internals/routing/#daily-probe):
|
||||||
|
新增「运行时:当日测速缓存」小节,逐条说明必须踩对的实现要点与反例
|
||||||
|
|
||||||
## [0.6.0] - 2026-09-16
|
## [0.6.0] - 2026-09-16
|
||||||
|
|
||||||
### 新增
|
### 新增
|
||||||
|
|||||||
@@ -184,6 +184,7 @@ await td.get_bars("cffex:IFL0", KlinePeriod.DAY, count=700) # 期货合约
|
|||||||
| 循环里逐个 `await get_bars` | 慢 | 用列表传多个标的,自动并发 |
|
| 循环里逐个 `await get_bars` | 慢 | 用列表传多个标的,自动并发 |
|
||||||
| 港股/美股五档 bid/ask | 没有(协议不支持) | `fields` 字段位通道有 name 但无五档 |
|
| 港股/美股五档 bid/ask | 没有(协议不支持) | `fields` 字段位通道有 name 但无五档 |
|
||||||
| 依赖实时性做信号 | 数据是 15 分钟前的 | 换券商实时源 |
|
| 依赖实时性做信号 | 数据是 15 分钟前的 | 换券商实时源 |
|
||||||
|
| 手动改包内 `servers.json` / 写死 IP | 容器只读、升级被覆盖、快节点捞不回来 | 日常靠内置的**当日测速缓存**自动选路;要强制指定就用 `PYPYTDX_DATA_{STANDARD,MAC,EX}_HOSTS` 环境变量 |
|
||||||
|
|
||||||
## 深入参考
|
## 深入参考
|
||||||
|
|
||||||
|
|||||||
@@ -121,6 +121,45 @@ from pytdxdata.config import get_standard_hosts, get_mac_hosts, get_ex_hosts
|
|||||||
print(len(get_standard_hosts()), len(get_mac_hosts()), len(get_ex_hosts()))
|
print(len(get_standard_hosts()), len(get_mac_hosts()), len(get_ex_hosts()))
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### 当日自动测速(无感)
|
||||||
|
|
||||||
|
除随包发布的静态清单外,`TdxData` 还在**当日首次启动**时做一次**后台全量测速**:
|
||||||
|
|
||||||
|
- 覆盖**全部候选**(包内清单 ∪ 代码内置常量 ∪ 环境变量覆盖),而不是只测当前池里的少数节点
|
||||||
|
——否则「上次慢、这次快」的节点永远捞不回来
|
||||||
|
- 用**真实 K 线命令**判定可达(标准 / MAC / EX 各发一条业务命令并要求非空),
|
||||||
|
能剔除「握手正常但返回空数据」的坏节点
|
||||||
|
- 结果按**实测延迟升序**写入**用户级缓存**,当天后续启动直接复用、**不再重测**
|
||||||
|
- 全程在后台进行:**不打印、不阻塞 `start()`、失败静默降级**,对调用方完全无感
|
||||||
|
|
||||||
|
缓存位置(可用 `PYPYTDX_DATA_CACHE_DIR` 覆盖):
|
||||||
|
|
||||||
|
| 平台 | 路径 |
|
||||||
|
|---|---|
|
||||||
|
| Windows | `%LOCALAPPDATA%\pytdxdata\hosts-today.json` |
|
||||||
|
| macOS | `~/Library/Caches/pytdxdata/hosts-today.json` |
|
||||||
|
| Linux / 其他 | `$XDG_CACHE_HOME/pytdxdata/hosts-today.json`(默认 `~/.cache/…`) |
|
||||||
|
|
||||||
|
主机选择的优先级(各协议独立):
|
||||||
|
|
||||||
|
1. `PYPYTDX_DATA_STANDARD_HOSTS` / `…_MAC_HOSTS` / `…_EX_HOSTS` 环境变量(手工指定,最高优先)
|
||||||
|
2. **当日测速缓存**中该协议实测可达的主机(按延迟升序)
|
||||||
|
3. 包内清单里 `status=ok` 的 `fast_pool` 主机
|
||||||
|
4. 包内清单里 `status=ok` 的主机
|
||||||
|
5. 代码内置常量
|
||||||
|
|
||||||
|
!!! note "缓存绝不写回安装目录"
|
||||||
|
测速结果只写入上面的**用户级缓存目录**。`site-packages` 在容器 / 系统级安装 / `pipx`
|
||||||
|
下可能只读,多用户共用一个安装时还会互相覆盖,因此包内 `servers.json` 不会被运行时改写。
|
||||||
|
|
||||||
|
环境变量一览:
|
||||||
|
|
||||||
|
| 变量 | 作用 |
|
||||||
|
|---|---|
|
||||||
|
| `PYPYTDX_DATA_STANDARD_HOSTS` / `…_MAC_HOSTS` / `…_EX_HOSTS` | 逗号分隔,手工指定主机(优先级最高,直接跳过测速缓存) |
|
||||||
|
| `PYPYTDX_DATA_CACHE_DIR` | 覆盖当日测速缓存目录 |
|
||||||
|
| `PYPYTDX_DATA_DISABLE_PROBE=1` | 关闭当日测速与缓存(CI / 测试用,保证确定性) |
|
||||||
|
|
||||||
## 生命周期
|
## 生命周期
|
||||||
|
|
||||||
`TdxData` 必须在 `async with` 内使用,或手动 `await td.start()` / `await td.close()`:
|
`TdxData` 必须在 `async with` 内使用,或手动 `await td.start()` / `await td.close()`:
|
||||||
|
|||||||
@@ -57,6 +57,29 @@ python -m pytdxdata.server_probe
|
|||||||
普通「只 ping 握手」的探测会把这类坏节点放进来,症状是**偶发的空返回**——
|
普通「只 ping 握手」的探测会把这类坏节点放进来,症状是**偶发的空返回**——
|
||||||
比直接连不上更难查。所以探测必须校验数据。
|
比直接连不上更难查。所以探测必须校验数据。
|
||||||
|
|
||||||
|
## 运行时:当日测速缓存 {#daily-probe}
|
||||||
|
|
||||||
|
`python -m pytdxdata.server_probe` 是**离线**重生成随包清单的工具;运行时还有一条**在线**的
|
||||||
|
轻量路径——`TdxData.start()` 会异步触发一次全量测速,把结果缓存到**用户级目录**
|
||||||
|
(`…/pytdxdata/hosts-today.json`),当天后续启动直接复用,不再重测。
|
||||||
|
|
||||||
|
设计与几个必须踩对的点:
|
||||||
|
|
||||||
|
| 关注点 | 正确做法 | 反例(会静默出错) |
|
||||||
|
|---|---|---|
|
||||||
|
| 候选集 | 取**全部候选**(清单 ∪ 常量 ∪ env) | 只把「当前池成员」丢给探针 → 上次慢的节点永远捞不回来 |
|
||||||
|
| 可达判定 | 发**真实 K 线命令**并要求非空 | 只握手 / 只读报文头 → 测不出「空响应」坏节点 |
|
||||||
|
| 写盘位置 | 用户级缓存目录 | 写回包内 `servers.json` → 容器只读、多用户互相覆盖 |
|
||||||
|
| 对调用方 | 后台 task + debug 日志 + 绝不抛异常 | 在 `start()` 里 `await` 或打印 → 启动变慢、污染用户输出 |
|
||||||
|
| 并发 | 进程内单例 + `asyncio.Lock` | 三个池各测一遍 → 三倍流量 |
|
||||||
|
| 写入 | 临时文件 + `os.replace` 原子替换 | 直接覆写 → 并发进程读到半截 JSON |
|
||||||
|
|
||||||
|
全部不可达时**不写缓存**——否则「今天全挂了」会被缓存一整天,明天也不重试。
|
||||||
|
|
||||||
|
缓存存在时,`config.get_standard_hosts()` 等按**实测延迟升序**返回该协议的可达主机;
|
||||||
|
缓存缺失 / 跨日 / 版本不符 / 已损坏时,才回落到包内清单。开关见
|
||||||
|
[安装与配置 → 环境变量](../getting-started.md#当日自动测速无感)。
|
||||||
|
|
||||||
## EX 通道细节
|
## EX 通道细节
|
||||||
|
|
||||||
- 端口固定 **7727**,与标准 / MAC 的 7709 不同
|
- 端口固定 **7727**,与标准 / MAC 的 7709 不同
|
||||||
|
|||||||
@@ -59,6 +59,19 @@ asyncio.run(main())
|
|||||||
> 旧的分通道/市场方法(`get_kline` / `get_transactions` / `get_goods_*` 等共 33 个)保留为
|
> 旧的分通道/市场方法(`get_kline` / `get_transactions` / `get_goods_*` 等共 33 个)保留为
|
||||||
> deprecated 别名,1.0 移除;完整对照见[接口重构审查](https://openbot-coder.github.io/pytdxdata/refactor-audit/)与 CHANGELOG。
|
> deprecated 别名,1.0 移除;完整对照见[接口重构审查](https://openbot-coder.github.io/pytdxdata/refactor-audit/)与 CHANGELOG。
|
||||||
|
|
||||||
|
## 服务器与配置
|
||||||
|
|
||||||
|
服务器主机来源有一条优先级链(各协议独立):环境变量 → **当日测速缓存** → 包内清单
|
||||||
|
`fast_pool` → 包内清单 → 代码内置常量。
|
||||||
|
|
||||||
|
- `TdxData.start()` 会在**当日首次启动**后台触发一次全量测速,用**真实 K 线命令**判定可达
|
||||||
|
(不是只握手),结果按延迟升序写入**用户级缓存**(Windows `%LOCALAPPDATA%\pytdxdata`、
|
||||||
|
macOS `~/Library/Caches/pytdxdata`、Linux `$XDG_CACHE_HOME/pytdxdata`),当天复用、不再重测。
|
||||||
|
- 该操作对调用方**完全无感**:后台执行、不打印、不阻塞 `start()`、失败静默降级。
|
||||||
|
运行时**绝不改写包内 `servers.json`**。
|
||||||
|
- 环境变量:`PYPYTDX_DATA_STANDARD_HOSTS` / `…_MAC_HOSTS` / `…_EX_HOSTS`(逗号分隔,手工指定,最高优先)、
|
||||||
|
`PYPYTDX_DATA_CACHE_DIR`(覆盖缓存目录)、`PYPYTDX_DATA_DISABLE_PROBE=1`(关闭测速)。
|
||||||
|
|
||||||
## CLI
|
## CLI
|
||||||
|
|
||||||
安装后提供 `tdx` 命令,19 个子命令,不写代码即可取数:
|
安装后提供 `tdx` 命令,19 个子命令,不写代码即可取数:
|
||||||
|
|||||||
+178
-13
@@ -1,12 +1,28 @@
|
|||||||
"""服务器清单与默认配置
|
"""服务器清单与默认配置
|
||||||
|
|
||||||
服务器清单存于包内 servers.json,由 server_probe.py 导入时自动探测可用性与延时。
|
服务器清单存于包内 ``servers.json`` —— 它是**随包分发的基线清单,运行期只读**。
|
||||||
config.py 加载时只采用 status="ok" 的服务器;环境变量可覆盖;代码内嵌清单为兜底。
|
每日首次启动的后台测速结果写入**用户级缓存目录**,运行期优先采用当日实测结果。
|
||||||
|
|
||||||
|
主机选取优先级(每个协议独立)::
|
||||||
|
|
||||||
|
环境变量 PYPYTDX_DATA_*_HOSTS > 当日测速缓存 > 包内清单 fast_pool
|
||||||
|
> 包内清单 ok > 代码内嵌常量
|
||||||
|
|
||||||
|
相关环境变量:
|
||||||
|
|
||||||
|
- ``PYPYTDX_DATA_CACHE_DIR`` 覆盖用户级缓存目录(默认见 :func:`probe_cache_dir`)
|
||||||
|
- ``PYPYTDX_DATA_DISABLE_PROBE`` 置真值时既不读当日缓存也不测速(CI / 测试用,保证确定性)
|
||||||
|
|
||||||
|
为什么不把测速结果写回包内 ``servers.json``:安装目录可能是只读的(容器 / 系统级
|
||||||
|
site-packages / pipx),且多用户共用一份安装时彼此覆盖,运行期改写分发数据本身也不妥。
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import ipaddress
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import sys
|
||||||
|
from datetime import date
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
DEFAULT_PORT = 7709
|
DEFAULT_PORT = 7709
|
||||||
@@ -14,19 +30,29 @@ DEFAULT_TIMEOUT = 10.0
|
|||||||
|
|
||||||
_SERVERS_FILE = Path(__file__).resolve().parent / "servers.json" # 包内清单(随包分发)
|
_SERVERS_FILE = Path(__file__).resolve().parent / "servers.json" # 包内清单(随包分发)
|
||||||
|
|
||||||
|
# 当日测速缓存(结构变更时递增,旧版本缓存自动失效)
|
||||||
|
PROBE_CACHE_VERSION = 1
|
||||||
|
_PROBE_CACHE_NAME = "hosts-today.json"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 包内清单(只读基线) ----------
|
||||||
|
def _read_servers_json() -> dict:
|
||||||
|
"""读取包内清单;文件缺失 / 内容损坏 / 顶层非对象 → 空 dict。"""
|
||||||
|
try:
|
||||||
|
cfg = json.loads(_SERVERS_FILE.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return {}
|
||||||
|
return cfg if isinstance(cfg, dict) else {}
|
||||||
|
|
||||||
|
|
||||||
def _load_ok_hosts(protocol: str, *, fast_only: bool = False) -> list[str]:
|
def _load_ok_hosts(protocol: str, *, fast_only: bool = False) -> list[str]:
|
||||||
"""从 servers.json 加载 status=ok 且支持指定协议的服务器。
|
"""从 servers.json 加载 status=ok 且支持指定协议的服务器。
|
||||||
|
|
||||||
fast_only=True 时只返回当日测速入池(fast_pool)的服务器(排名前N或延迟<阈值),
|
fast_only=True 时只返回测速入池(fast_pool)的服务器(排名前N或延迟<阈值),
|
||||||
用于动态连接池优先使用优质节点。
|
用于动态连接池优先使用优质节点。
|
||||||
"""
|
"""
|
||||||
try:
|
|
||||||
cfg = json.loads(_SERVERS_FILE.read_text(encoding="utf-8"))
|
|
||||||
except (OSError, ValueError):
|
|
||||||
return []
|
|
||||||
out = []
|
out = []
|
||||||
for s in cfg.get("servers", []):
|
for s in _read_servers_json().get("servers", []):
|
||||||
if s.get("status") != "ok":
|
if s.get("status") != "ok":
|
||||||
continue
|
continue
|
||||||
if protocol not in s.get("protocols", []):
|
if protocol not in s.get("protocols", []):
|
||||||
@@ -38,13 +64,90 @@ def _load_ok_hosts(protocol: str, *, fast_only: bool = False) -> list[str]:
|
|||||||
|
|
||||||
|
|
||||||
def _load_ex_handshakes() -> dict[str, str]:
|
def _load_ex_handshakes() -> dict[str, str]:
|
||||||
try:
|
"""包内清单里 EX 服务器的握手模式。"""
|
||||||
cfg = json.loads(_SERVERS_FILE.read_text(encoding="utf-8"))
|
return {s["ip"]: s.get("handshake", "setup")
|
||||||
except (OSError, ValueError):
|
for s in _read_servers_json().get("servers", [])
|
||||||
return {}
|
|
||||||
return {s["ip"]: s.get("handshake", "setup") for s in cfg.get("servers", [])
|
|
||||||
if "ex" in s.get("protocols", []) and s.get("status") == "ok"}
|
if "ex" in s.get("protocols", []) and s.get("status") == "ok"}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 用户级当日测速缓存 ----------
|
||||||
|
def probe_cache_dir() -> Path:
|
||||||
|
"""用户级缓存目录(保证可写,绝不指向安装目录)。
|
||||||
|
|
||||||
|
``PYPYTDX_DATA_CACHE_DIR`` 可覆盖(测试指向临时目录)。按平台约定:
|
||||||
|
|
||||||
|
- Windows: ``%LOCALAPPDATA%\\pytdxdata``
|
||||||
|
- macOS: ``~/Library/Caches/pytdxdata``
|
||||||
|
- 其他: ``$XDG_CACHE_HOME/pytdxdata`` 或 ``~/.cache/pytdxdata``
|
||||||
|
"""
|
||||||
|
env = (os.environ.get("PYPYTDX_DATA_CACHE_DIR") or "").strip()
|
||||||
|
if env:
|
||||||
|
return Path(env).expanduser()
|
||||||
|
if os.name == "nt":
|
||||||
|
base = os.environ.get("LOCALAPPDATA") or os.environ.get("APPDATA")
|
||||||
|
return (Path(base) if base else Path.home() / "AppData" / "Local") / "pytdxdata"
|
||||||
|
if sys.platform == "darwin":
|
||||||
|
return Path.home() / "Library" / "Caches" / "pytdxdata"
|
||||||
|
base = os.environ.get("XDG_CACHE_HOME")
|
||||||
|
return (Path(base) if base else Path.home() / ".cache") / "pytdxdata"
|
||||||
|
|
||||||
|
|
||||||
|
def probe_cache_file() -> Path:
|
||||||
|
"""当日测速缓存文件路径(每次调用重新解析,便于测试改环境变量后立即生效)。"""
|
||||||
|
return probe_cache_dir() / _PROBE_CACHE_NAME
|
||||||
|
|
||||||
|
|
||||||
|
def probe_disabled() -> bool:
|
||||||
|
"""是否已通过 ``PYPYTDX_DATA_DISABLE_PROBE`` 关闭当日测速缓存。"""
|
||||||
|
return os.environ.get("PYPYTDX_DATA_DISABLE_PROBE", "").strip().lower() in {
|
||||||
|
"1", "true", "yes", "on",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_probe_cache(today: str | None = None,
|
||||||
|
path: str | Path | None = None) -> dict | None:
|
||||||
|
"""读取**当日**测速缓存;缺失 / 损坏 / 非今日 / 版本不符 → ``None``。
|
||||||
|
|
||||||
|
返回非 None 即表示"今天已经测过速了",调用方应直接复用而不再刷新。
|
||||||
|
``path`` 可显式指定缓存文件(默认 :func:`probe_cache_file`),便于注入与测试。
|
||||||
|
"""
|
||||||
|
if probe_disabled():
|
||||||
|
return None
|
||||||
|
target = Path(path) if path is not None else probe_cache_file()
|
||||||
|
try:
|
||||||
|
cfg = json.loads(target.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return None
|
||||||
|
if not isinstance(cfg, dict) or cfg.get("version") != PROBE_CACHE_VERSION:
|
||||||
|
return None
|
||||||
|
if str(cfg.get("date", "")) != (today or date.today().isoformat()):
|
||||||
|
return None
|
||||||
|
results = cfg.get("results")
|
||||||
|
if not isinstance(results, dict) or not results:
|
||||||
|
return None
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
def _load_cached_hosts(protocol: str) -> list[str]:
|
||||||
|
"""当日测速缓存中该协议**实测可达**的主机,按延迟升序(越快越靠前)。"""
|
||||||
|
cfg = load_probe_cache()
|
||||||
|
if cfg is None:
|
||||||
|
return []
|
||||||
|
rows: list[tuple[float, str]] = []
|
||||||
|
for ip, per_proto in cfg["results"].items():
|
||||||
|
if not isinstance(per_proto, dict):
|
||||||
|
continue
|
||||||
|
lat = per_proto.get(protocol)
|
||||||
|
if isinstance(lat, bool) or not isinstance(lat, (int, float)):
|
||||||
|
continue
|
||||||
|
if lat < 0:
|
||||||
|
continue
|
||||||
|
rows.append((float(lat), str(ip)))
|
||||||
|
rows.sort(key=lambda kv: (kv[0], kv[1]))
|
||||||
|
return [ip for _, ip in rows]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 代码内嵌兜底清单 ----------
|
||||||
# 标准行情服务器 —— 真实K线探测验证可用(2026-08-12, GetSecurityBarsCmd 600000 日K)
|
# 标准行情服务器 —— 真实K线探测验证可用(2026-08-12, GetSecurityBarsCmd 600000 日K)
|
||||||
# 27台可用 = pytdxwrapper券商服务器(国泰君安/中信短线宝/华林/安信等) + 通达信官方主站
|
# 27台可用 = pytdxwrapper券商服务器(国泰君安/中信短线宝/华林/安信等) + 通达信官方主站
|
||||||
# 来源①: pytdxwrapper config.py TDX_HOSTS(券商+官方, 51台探测27可用)
|
# 来源①: pytdxwrapper config.py TDX_HOSTS(券商+官方, 51台探测27可用)
|
||||||
@@ -91,10 +194,66 @@ EX_HANDSHAKES: dict[str, str] = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 测速候选枚举 ----------
|
||||||
|
def is_valid_host(host: object) -> bool:
|
||||||
|
"""是否是可用作服务器地址的合法主机(IPv4/IPv6 字面量,或含点的域名)。
|
||||||
|
|
||||||
|
用于挡住测试 mock 混进来的伪主机名(如 ``"s1"``)被写进缓存 / 清单。
|
||||||
|
"""
|
||||||
|
if not isinstance(host, str):
|
||||||
|
return False
|
||||||
|
h = host.strip()
|
||||||
|
if not h or len(h) > 253 or any(c.isspace() for c in h):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
ipaddress.ip_address(h)
|
||||||
|
return True
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
return "." in h and all(
|
||||||
|
part and part[0] != "-" and part[-1] != "-" and part.replace("-", "").isalnum()
|
||||||
|
for part in h.split(".")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def all_probe_candidates() -> dict[str, list[str]]:
|
||||||
|
"""[测速用] 返回 ``{协议: [候选主机]}`` 全量清单(已校验、去重、保序)。
|
||||||
|
|
||||||
|
来源 = 包内清单申报的协议 ∪ 代码内嵌常量 ∪ 环境变量覆盖。
|
||||||
|
|
||||||
|
与 ``get_*_hosts()`` 的关键区别:这里**不做 status / 延迟过滤**。
|
||||||
|
测速必须能看到全部候选,否则"上次慢、这次快"的节点永远捞不回来,
|
||||||
|
形成自指循环(只重测已在池里的少数节点,其余记录一直过期)。
|
||||||
|
"""
|
||||||
|
out: dict[str, list[str]] = {"standard": [], "mac": [], "ex": []}
|
||||||
|
for s in _read_servers_json().get("servers", []):
|
||||||
|
ip = s.get("ip")
|
||||||
|
for protocol in s.get("protocols", []):
|
||||||
|
if protocol in out:
|
||||||
|
out[protocol].append(ip)
|
||||||
|
out["standard"] += STANDARD_HOSTS
|
||||||
|
out["mac"] += MAC_HOSTS
|
||||||
|
out["ex"] += EX_HOSTS
|
||||||
|
for protocol, var in (("standard", "PYPYTDX_DATA_STANDARD_HOSTS"),
|
||||||
|
("mac", "PYPYTDX_DATA_MAC_HOSTS"),
|
||||||
|
("ex", "PYPYTDX_DATA_EX_HOSTS")):
|
||||||
|
env = os.environ.get(var)
|
||||||
|
if env:
|
||||||
|
out[protocol] += env.split(",")
|
||||||
|
return {
|
||||||
|
protocol: [h for h in dict.fromkeys(hosts) if is_valid_host(h)]
|
||||||
|
for protocol, hosts in out.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 对外取主机 ----------
|
||||||
def get_standard_hosts() -> list[str]:
|
def get_standard_hosts() -> list[str]:
|
||||||
env = os.environ.get("PYPYTDX_DATA_STANDARD_HOSTS")
|
env = os.environ.get("PYPYTDX_DATA_STANDARD_HOSTS")
|
||||||
if env:
|
if env:
|
||||||
return [h.strip() for h in env.split(",") if h.strip()]
|
return [h.strip() for h in env.split(",") if h.strip()]
|
||||||
|
cached = _load_cached_hosts("standard")
|
||||||
|
if cached:
|
||||||
|
return cached
|
||||||
fast = _load_ok_hosts("standard", fast_only=True)
|
fast = _load_ok_hosts("standard", fast_only=True)
|
||||||
if fast:
|
if fast:
|
||||||
return fast
|
return fast
|
||||||
@@ -106,6 +265,9 @@ def get_mac_hosts() -> list[str]:
|
|||||||
env = os.environ.get("PYPYTDX_DATA_MAC_HOSTS")
|
env = os.environ.get("PYPYTDX_DATA_MAC_HOSTS")
|
||||||
if env:
|
if env:
|
||||||
return [h.strip() for h in env.split(",") if h.strip()]
|
return [h.strip() for h in env.split(",") if h.strip()]
|
||||||
|
cached = _load_cached_hosts("mac")
|
||||||
|
if cached:
|
||||||
|
return cached
|
||||||
fast = _load_ok_hosts("mac", fast_only=True)
|
fast = _load_ok_hosts("mac", fast_only=True)
|
||||||
if fast:
|
if fast:
|
||||||
return fast
|
return fast
|
||||||
@@ -117,6 +279,9 @@ def get_ex_hosts() -> list[str]:
|
|||||||
env = os.environ.get("PYPYTDX_DATA_EX_HOSTS")
|
env = os.environ.get("PYPYTDX_DATA_EX_HOSTS")
|
||||||
if env:
|
if env:
|
||||||
return [h.strip() for h in env.split(",") if h.strip()]
|
return [h.strip() for h in env.split(",") if h.strip()]
|
||||||
|
cached = _load_cached_hosts("ex")
|
||||||
|
if cached:
|
||||||
|
return cached
|
||||||
from_json = _load_ok_hosts("ex")
|
from_json = _load_ok_hosts("ex")
|
||||||
return from_json or list(EX_HOSTS)
|
return from_json or list(EX_HOSTS)
|
||||||
|
|
||||||
|
|||||||
@@ -6,8 +6,9 @@
|
|||||||
- 动态:支持运行时 add() 新服务器;支持后台全量测速入库(当日首次启动)。
|
- 动态:支持运行时 add() 新服务器;支持后台全量测速入库(当日首次启动)。
|
||||||
- 单连接约束(核心需求):per_server_cap 默认 1 —— 一台服务器只允许建立 1 个连接,
|
- 单连接约束(核心需求):per_server_cap 默认 1 —— 一台服务器只允许建立 1 个连接,
|
||||||
不允许多开,避免对单一券商服务器造成并发压力被限流/封禁。
|
不允许多开,避免对单一券商服务器造成并发压力被限流/封禁。
|
||||||
- 后台测速(核心需求):当日首次启动时,在后台对全量候选服务器测速,按
|
- 后台测速(核心需求):当日首次启动时,在后台对**全量候选**服务器测速(覆盖标准/MAC/EX
|
||||||
「排名前 FAST_POOL_TOP_N 或 延迟 < FAST_POOL_MAX_LATENCY」筛选入库,结果写回 servers.json。
|
三个协议,而不是只测当前池内节点),结果缓存到用户级缓存目录;当天已有缓存则跳过、不再刷新。
|
||||||
|
全程不阻塞 start()、不打印、不抛错(对用户无感)。详见 pool/probe_cache.py。
|
||||||
- 弹性:空闲连接超时后由 reaper 缩容回 min_size;支持 stats 统计。
|
- 弹性:空闲连接超时后由 reaper 缩容回 min_size;支持 stats 统计。
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -19,7 +20,7 @@ import time
|
|||||||
|
|
||||||
from .connection import TdxConnection
|
from .connection import TdxConnection
|
||||||
from .health import HealthEngine
|
from .health import HealthEngine
|
||||||
from .probe_cache import get_probe_cache
|
from .probe_cache import schedule_probe
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -101,6 +102,7 @@ class DynamicPool:
|
|||||||
self._lock = asyncio.Lock()
|
self._lock = asyncio.Lock()
|
||||||
self._cond = asyncio.Condition(self._lock)
|
self._cond = asyncio.Condition(self._lock)
|
||||||
self._reaper_task: asyncio.Task | None = None
|
self._reaper_task: asyncio.Task | None = None
|
||||||
|
self._probe_task: asyncio.Task | None = None
|
||||||
|
|
||||||
# ---------- 生命周期 ----------
|
# ---------- 生命周期 ----------
|
||||||
async def start(self) -> None:
|
async def start(self) -> None:
|
||||||
@@ -108,13 +110,11 @@ class DynamicPool:
|
|||||||
return
|
return
|
||||||
self._started = True
|
self._started = True
|
||||||
|
|
||||||
# 需求①:当日首次启动 → 后台全量测速入库(不阻塞主流程)
|
# 需求①:当日首次启动 → 后台全量测速(不阻塞主流程;当天已有缓存则自动跳过)
|
||||||
|
# 候选清单由 probe_cache 自行枚举全量主机(本池的 servers 只是其中一个子集,
|
||||||
|
# 不能直接当作测速输入,否则只会重测已在池内的少数节点、其余记录永久过期)。
|
||||||
if self.auto_probe:
|
if self.auto_probe:
|
||||||
hosts = list(self.servers)
|
self._probe_task = schedule_probe()
|
||||||
asyncio.create_task(
|
|
||||||
get_probe_cache().ensure_probe(hosts),
|
|
||||||
name="pytdxdata-daily-probe",
|
|
||||||
)
|
|
||||||
|
|
||||||
targets = self._pick_servers(self.min_size)
|
targets = self._pick_servers(self.min_size)
|
||||||
for host in targets:
|
for host in targets:
|
||||||
@@ -175,6 +175,15 @@ class DynamicPool:
|
|||||||
if self._reaper_task is not None:
|
if self._reaper_task is not None:
|
||||||
self._reaper_task.cancel()
|
self._reaper_task.cancel()
|
||||||
self._reaper_task = None
|
self._reaper_task = None
|
||||||
|
# 后台测速仍可能持有 socket;取消并等它退场,避免事件循环关闭时冒出
|
||||||
|
# "Task was destroyed but it is pending"(那会破坏"无感")
|
||||||
|
if self._probe_task is not None:
|
||||||
|
self._probe_task.cancel()
|
||||||
|
try:
|
||||||
|
await self._probe_task
|
||||||
|
except (asyncio.CancelledError, Exception):
|
||||||
|
pass
|
||||||
|
self._probe_task = None
|
||||||
async with self._lock:
|
async with self._lock:
|
||||||
for pc in list(self._conns):
|
for pc in list(self._conns):
|
||||||
try:
|
try:
|
||||||
|
|||||||
+272
-112
@@ -1,162 +1,322 @@
|
|||||||
"""当日首次启动的全量服务器测速与入库缓存。
|
"""当日首次启动的全量服务器测速与缓存(对用户完全无感)。
|
||||||
|
|
||||||
设计要点:
|
行为
|
||||||
- 仅在"当日首次启动"时触发一次后台全量测速(按本地日期判断是否已探测)。
|
----
|
||||||
- 测速采用真实握手连通(TdxConnection.test),比单纯 ping 更能反映通达信可用性。
|
- **当日首次启动**触发一次后台全量测速;当天已有缓存则直接复用,**不再刷新**。
|
||||||
- 入库规则:延迟排名前 FAST_POOL_TOP_N(10) **或** 延迟 < FAST_POOL_MAX_LATENCY(0.2s) 的服务器
|
- 测速覆盖**全部候选**(包内清单 ∪ 代码常量 ∪ env 覆盖,见
|
||||||
才标记为 status="ok" 并写入 servers.json,其余降权(保留但不优先)。
|
:func:`pytdxdata.config.all_probe_candidates`),而不是只测当前已在池里的少数节点 ——
|
||||||
- 结果持久化到 servers.json,供 config / 连接池直接使用。
|
否则"上次慢、这次快"的节点永远捞不回来。
|
||||||
|
- 用**真实业务命令**判定可达(标准: GetSecurityBarsCmd / MAC: MacKlineCmd /
|
||||||
|
EX: GetExInstrumentBarsCmd)并要求返回非空数据,而不是只做握手回显:
|
||||||
|
存在"握手正常但 K线空响应"的坏节点(见 ``config.py`` 注释),只有真数据能测出来。
|
||||||
|
- 结果写入**用户级缓存目录**(见 :func:`pytdxdata.config.probe_cache_file`),
|
||||||
|
不写安装目录:site-packages 在容器 / 系统级安装 / pipx 下可能只读,
|
||||||
|
多用户共用一份安装时还会互相覆盖。
|
||||||
|
- 结果被 :func:`pytdxdata.config.get_standard_hosts` 等按**实测延迟升序**采用。
|
||||||
|
|
||||||
|
无感保证
|
||||||
|
--------
|
||||||
|
- 不打印任何东西;日志一律 ``debug`` 级(默认不输出)。
|
||||||
|
- 不抛异常(取消除外);失败只落 debug 日志,下次启动自然重试。
|
||||||
|
- 由连接池以后台 task 方式调用,不阻塞 ``start()`` 与任何用户请求。
|
||||||
|
|
||||||
|
一致性与并发
|
||||||
|
------------
|
||||||
|
- 进程内单例 + ``asyncio.Lock``:标准 / MAC / EX 三个池共用一次真正测速。
|
||||||
|
- 原子写入(tmp + ``os.replace``):进程被杀或多进程并发都不会读到半截 JSON。
|
||||||
|
- 全部不可达时**不写缓存**,避免把"全挂"状态缓存一整天。
|
||||||
|
|
||||||
|
开关
|
||||||
|
----
|
||||||
|
- ``PYPYTDX_DATA_DISABLE_PROBE=1`` 关闭(CI / 测试用,保证确定性)。
|
||||||
|
- ``PYPYTDX_DATA_CACHE_DIR`` 覆盖缓存目录(测试指向临时目录)。
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
|
import os
|
||||||
import time
|
import time
|
||||||
|
from collections import defaultdict
|
||||||
|
from collections.abc import Awaitable, Callable
|
||||||
from datetime import date
|
from datetime import date
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from ..config import _SERVERS_FILE
|
from ..config import (
|
||||||
|
PROBE_CACHE_VERSION,
|
||||||
|
all_probe_candidates,
|
||||||
|
get_ex_handshakes,
|
||||||
|
is_valid_host,
|
||||||
|
load_probe_cache,
|
||||||
|
probe_cache_file,
|
||||||
|
probe_disabled,
|
||||||
|
)
|
||||||
|
from ..models import Adjust, KlinePeriod
|
||||||
|
from ..protocol.commands.ex_proto import GetExInstrumentBarsCmd
|
||||||
|
from ..protocol.commands.kline import GetSecurityBarsCmd
|
||||||
|
from ..protocol.commands.mac import MacKlineCmd
|
||||||
|
from .connection import TdxConnection
|
||||||
|
from .ex_connection import ExTdxConnection
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
# 入池阈值
|
# 入池阈值(仅用于缓存里的 fast 诊断标记;运行时按实测延迟整体排序,不做硬切)
|
||||||
FAST_POOL_TOP_N = 10 # 延迟排名前 N
|
FAST_POOL_TOP_N = 10
|
||||||
FAST_POOL_MAX_LATENCY = 0.2 # 延迟 < 200ms(秒)也可入池
|
FAST_POOL_MAX_LATENCY = 0.2
|
||||||
|
# 探针参数
|
||||||
PROBE_PORT = 7709
|
PROBE_PORT = 7709
|
||||||
|
EX_PORT = 7727
|
||||||
PROBE_TIMEOUT = 3.0
|
PROBE_TIMEOUT = 3.0
|
||||||
PROBE_CONCURRENCY = 24
|
PROBE_CONCURRENCY = 12
|
||||||
|
PROBE_BUDGET = 60.0 # 整轮测速硬上限(秒),防止极端网络下长期占着后台
|
||||||
|
|
||||||
|
ProbeFn = Callable[[str, float], Awaitable[float | None]]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 三个协议各自的真实数据探针 ----------
|
||||||
|
async def _probe_standard(host: str, timeout: float) -> float | None:
|
||||||
|
"""标准通道 7709:取 600000 日K,非空才算可达。"""
|
||||||
|
conn = TdxConnection(host, PROBE_PORT, timeout)
|
||||||
|
t0 = time.monotonic()
|
||||||
|
try:
|
||||||
|
await conn.connect()
|
||||||
|
bars = await conn.execute(GetSecurityBarsCmd(1, "600000", KlinePeriod.DAY, 0, 100))
|
||||||
|
return time.monotonic() - t0 if bars else None
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
finally:
|
||||||
|
await conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
async def _probe_mac(host: str, timeout: float) -> float | None:
|
||||||
|
"""MAC 通道 7709:MAC 帧与标准帧不同,用 MacKlineCmd 取日K。"""
|
||||||
|
conn = TdxConnection(host, PROBE_PORT, timeout)
|
||||||
|
t0 = time.monotonic()
|
||||||
|
try:
|
||||||
|
await conn.connect()
|
||||||
|
bars = await conn.execute(
|
||||||
|
MacKlineCmd(1, "600000", KlinePeriod.DAY, Adjust.NONE, 0, 100)
|
||||||
|
)
|
||||||
|
return time.monotonic() - t0 if bars else None
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
finally:
|
||||||
|
await conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
async def _probe_ex(host: str, timeout: float) -> float | None:
|
||||||
|
"""EX 扩展市场 7727:取港股 00700,握手模式按主机取(login / setup)。"""
|
||||||
|
conn = ExTdxConnection(host, EX_PORT, timeout, get_ex_handshakes().get(host, "setup"))
|
||||||
|
t0 = time.monotonic()
|
||||||
|
try:
|
||||||
|
await conn.connect()
|
||||||
|
bars = await conn.execute(GetExInstrumentBarsCmd(31, "00700", 0, 0, 5))
|
||||||
|
return time.monotonic() - t0 if bars else None
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
finally:
|
||||||
|
await conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
_DEFAULT_PROBERS: dict[str, ProbeFn] = {
|
||||||
|
"standard": _probe_standard,
|
||||||
|
"mac": _probe_mac,
|
||||||
|
"ex": _probe_ex,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
class ProbeCache:
|
class ProbeCache:
|
||||||
"""管理当日测速缓存与 servers.json 入库。
|
"""当日全量测速的进程内协调器。
|
||||||
|
|
||||||
一台服务器不论协议(standard/mac/ex),测速结果共享:
|
一台主机不论申报几个协议,都按协议分别探测;缓存里记录
|
||||||
只要任一端口可达即视为该主机可用,并以最快端口延迟作为延迟基准。
|
``{host: {protocol: 延迟或 None}}``,``None`` 表示该协议本次不可达。
|
||||||
|
|
||||||
|
参数(后 5 个主要供测试注入):
|
||||||
|
cache_file: 显式指定缓存文件;``None`` 时按 config 的平台规则解析。
|
||||||
|
probers: 协议 → 探针函数;默认使用真实网络探针。
|
||||||
|
today_fn: 返回 ``YYYY-MM-DD``;测试可注入假日期来验证跨日重测。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, servers_file: Path | None = None) -> None:
|
def __init__(self, cache_file: Path | str | None = None, *,
|
||||||
self._file = servers_file or _SERVERS_FILE
|
timeout: float = PROBE_TIMEOUT,
|
||||||
|
concurrency: int = PROBE_CONCURRENCY,
|
||||||
|
budget: float = PROBE_BUDGET,
|
||||||
|
probers: dict[str, ProbeFn] | None = None,
|
||||||
|
today_fn: Callable[[], str] | None = None) -> None:
|
||||||
|
self._file = Path(cache_file) if cache_file is not None else None
|
||||||
|
self._timeout = timeout
|
||||||
|
self._concurrency = concurrency
|
||||||
|
self._budget = budget
|
||||||
|
self._probers = dict(probers or _DEFAULT_PROBERS)
|
||||||
|
self._today_fn = today_fn or (lambda: date.today().isoformat())
|
||||||
self._lock = asyncio.Lock()
|
self._lock = asyncio.Lock()
|
||||||
self._last_probe_date: str | None = None
|
self._handled = False # 本进程是否已处理过今日测速(含"已有缓存,跳过")
|
||||||
self._probed_today = False
|
self._task: asyncio.Task | None = None
|
||||||
|
|
||||||
|
# ---------- 路径与状态 ----------
|
||||||
|
def cache_file(self) -> Path:
|
||||||
|
"""缓存文件路径(构造时显式传入的优先,否则按平台规则解析)。"""
|
||||||
|
return self._file if self._file is not None else probe_cache_file()
|
||||||
|
|
||||||
|
def today(self) -> str:
|
||||||
|
return self._today_fn()
|
||||||
|
|
||||||
|
def read_cache(self) -> dict | None:
|
||||||
|
"""读取当日缓存;``None`` 表示"今天还没测过"。"""
|
||||||
|
return load_probe_cache(self.today(), self.cache_file())
|
||||||
|
|
||||||
# ---- 公开 API ----
|
|
||||||
def needs_probe_today(self) -> bool:
|
def needs_probe_today(self) -> bool:
|
||||||
"""是否有必要在今日执行一次全量测速(文件不存在 / 日期不符 / 无 ok 项)。"""
|
"""今日是否仍需测速(缓存缺失 / 非今日 / 损坏 / 版本不符 / 无任何可用项)。"""
|
||||||
if not self._file.exists():
|
if probe_disabled():
|
||||||
return True
|
|
||||||
try:
|
|
||||||
cfg = json.loads(self._file.read_text(encoding="utf-8"))
|
|
||||||
except (OSError, ValueError):
|
|
||||||
return True
|
|
||||||
checked = str(cfg.get("checked_date", ""))
|
|
||||||
today = date.today().isoformat()
|
|
||||||
if checked != today:
|
|
||||||
return True
|
|
||||||
servers = cfg.get("servers", [])
|
|
||||||
if not any(s.get("status") == "ok" for s in servers):
|
|
||||||
return True
|
|
||||||
return False
|
return False
|
||||||
|
return self.read_cache() is None
|
||||||
|
|
||||||
async def ensure_probe(self, hosts: list[str]) -> None:
|
def reset(self) -> None:
|
||||||
"""若今日尚未测速,则后台执行一次全量测速并入库(幂等)。"""
|
"""清除"本进程已处理"标记(仅供测试复用单例)。"""
|
||||||
|
self._handled = False
|
||||||
|
|
||||||
|
# ---------- 主入口 ----------
|
||||||
|
async def ensure_probe(self, hosts: list[str] | None = None) -> None:
|
||||||
|
"""幂等入口:当日尚未测速则测一次;当天已有缓存则立即返回。
|
||||||
|
|
||||||
|
- 不阻塞调用方:正常由连接池以后台 task 调用。
|
||||||
|
- 绝不抛异常(``CancelledError`` 除外):测速 / 写盘失败只落 debug 日志。
|
||||||
|
- ``hosts``:为兼容旧签名保留,作为**额外的标准通道候选**并入,
|
||||||
|
且必须通过 :func:`~pytdxdata.config.is_valid_host` 校验(挡住 mock 脏数据)。
|
||||||
|
"""
|
||||||
async with self._lock:
|
async with self._lock:
|
||||||
if self._probed_today:
|
if self._handled:
|
||||||
return
|
return
|
||||||
|
self._handled = True # 先占位:并发调用方不必排队等网络
|
||||||
if not self.needs_probe_today():
|
if not self.needs_probe_today():
|
||||||
self._probed_today = True
|
log.debug("当日测速缓存已存在,跳过测速: %s", self.cache_file())
|
||||||
return
|
return
|
||||||
self._probed_today = True
|
|
||||||
|
|
||||||
# 在锁外执行(耗时网络 IO),允许并发进入的调用方共享本次结果
|
|
||||||
try:
|
try:
|
||||||
await self._run_full_probe(hosts)
|
payload = await self._run_probe(hosts)
|
||||||
except Exception as e: # 测速失败不能阻塞主流程
|
if payload is not None:
|
||||||
log.warning("后台全量测速失败,沿用既有清单: %s", str(e)[:80])
|
self._write(payload)
|
||||||
|
except asyncio.CancelledError:
|
||||||
|
raise
|
||||||
|
except Exception as exc: # 测速绝不能影响主流程
|
||||||
|
log.debug("后台测速失败,沿用既有清单: %s", str(exc)[:120])
|
||||||
|
|
||||||
# ---- 内部实现 ----
|
# ---------- 内部实现 ----------
|
||||||
async def _run_full_probe(self, hosts: list[str]) -> None:
|
async def _run_probe(self, extra_hosts: list[str] | None = None) -> dict | None:
|
||||||
from .connection import TdxConnection
|
"""全量并发测速,返回待写入的缓存 payload;无任何可达主机时返回 None。"""
|
||||||
|
candidates = all_probe_candidates()
|
||||||
|
if extra_hosts:
|
||||||
|
extra = [h.strip() for h in extra_hosts
|
||||||
|
if isinstance(h, str) and is_valid_host(h)]
|
||||||
|
if extra:
|
||||||
|
candidates["standard"] = list(dict.fromkeys(candidates["standard"] + extra))
|
||||||
|
jobs = [(proto, ip) for proto, ips in candidates.items() for ip in ips]
|
||||||
|
if not jobs:
|
||||||
|
log.debug("无测速候选,跳过")
|
||||||
|
return None
|
||||||
|
|
||||||
log.info("开始当日全量服务器测速 (%d 台)...", len(hosts))
|
log.debug("开始当日全量测速: %d 台主机 / %d 个协议探针",
|
||||||
sem = asyncio.Semaphore(PROBE_CONCURRENCY)
|
len({ip for _, ip in jobs}), len(jobs))
|
||||||
results: dict[str, float | None] = {}
|
sem = asyncio.Semaphore(self._concurrency)
|
||||||
|
# {host: {protocol: 延迟或 None}}
|
||||||
|
results: dict[str, dict[str, float | None]] = defaultdict(dict)
|
||||||
|
|
||||||
async def _probe_one(host: str) -> None:
|
async def _one(protocol: str, host: str) -> None:
|
||||||
async with sem:
|
async with sem:
|
||||||
try:
|
try:
|
||||||
# 使用连接自带的 ping():建连+握手+发1条命令测往返延迟
|
lat = await asyncio.wait_for(
|
||||||
conn = TdxConnection(host, PROBE_PORT, PROBE_TIMEOUT)
|
self._probers[protocol](host, self._timeout),
|
||||||
dt = await asyncio.wait_for(conn.ping(), timeout=PROBE_TIMEOUT + 1.0)
|
timeout=self._timeout + 2.0,
|
||||||
results[host] = dt # 失败时为 None
|
)
|
||||||
except Exception:
|
except Exception:
|
||||||
results[host] = None
|
lat = None
|
||||||
|
results[host][protocol] = lat
|
||||||
|
|
||||||
await asyncio.gather(*(_probe_one(h) for h in hosts))
|
t0 = time.monotonic()
|
||||||
|
|
||||||
ok_hosts = {h: lt for h, lt in results.items() if lt is not None}
|
|
||||||
# 按延迟升序排名
|
|
||||||
ranked = sorted(ok_hosts.items(), key=lambda kv: kv[1])
|
|
||||||
fast = set()
|
|
||||||
for i, (h, lt) in enumerate(ranked):
|
|
||||||
if i < FAST_POOL_TOP_N or lt < FAST_POOL_MAX_LATENCY:
|
|
||||||
fast.add(h)
|
|
||||||
|
|
||||||
log.info("测速完成: 可用 %d 台, 入池 %d 台 (前%d或<%dms)",
|
|
||||||
len(ok_hosts), len(fast), FAST_POOL_TOP_N,
|
|
||||||
int(FAST_POOL_MAX_LATENCY * 1000))
|
|
||||||
self._write_results(results, fast)
|
|
||||||
|
|
||||||
def _write_results(self, results: dict[str, float | None], fast: set[str]) -> None:
|
|
||||||
try:
|
try:
|
||||||
cfg = json.loads(self._file.read_text(encoding="utf-8"))
|
await asyncio.wait_for(
|
||||||
except (OSError, ValueError):
|
asyncio.gather(*(_one(p, ip) for p, ip in jobs)),
|
||||||
cfg = {"servers": []}
|
timeout=self._budget,
|
||||||
servers = cfg.get("servers", [])
|
|
||||||
by_ip = {s.get("ip"): s for s in servers}
|
|
||||||
|
|
||||||
today = date.today().isoformat()
|
|
||||||
now = time.strftime("%Y-%m-%d %H:%M:%S")
|
|
||||||
checked_any = False
|
|
||||||
for host, lt in results.items():
|
|
||||||
entry = by_ip.get(host)
|
|
||||||
if entry is None:
|
|
||||||
entry = {"ip": host, "name": host, "protocols": ["standard"]}
|
|
||||||
servers.append(entry)
|
|
||||||
by_ip[host] = entry
|
|
||||||
if lt is not None:
|
|
||||||
checked_any = True
|
|
||||||
entry["status"] = "ok" if host in fast else "slow"
|
|
||||||
entry["latency"] = round(lt, 3)
|
|
||||||
entry["fast_pool"] = host in fast
|
|
||||||
else:
|
|
||||||
# 探测不可达:不降级已有 ok(避免误杀),仅标记本次不可达
|
|
||||||
entry.setdefault("status", "unknown")
|
|
||||||
entry["latency"] = None
|
|
||||||
entry["fast_pool"] = False
|
|
||||||
entry["checked_date"] = today
|
|
||||||
entry["checked_at"] = now
|
|
||||||
|
|
||||||
# 若有 ok 服务器写入可入池日期;否则保留今日以免反复重试
|
|
||||||
cfg["checked_date"] = today
|
|
||||||
cfg["note"] = (
|
|
||||||
f"当日全量测速入库({today}): 入池={len(fast)}台 "
|
|
||||||
f"(排名前{FAST_POOL_TOP_N}或延迟<{int(FAST_POOL_MAX_LATENCY*1000)}ms)"
|
|
||||||
)
|
)
|
||||||
if not checked_any:
|
except TimeoutError:
|
||||||
# 全部不可达时不写 checked_date,下次启动重试
|
# 超时:采用已完成的部分结果(部分可用也强于完全没有)
|
||||||
cfg.pop("checked_date", None)
|
log.debug("测速整体超过 %.0fs 预算,采用已完成的部分结果", self._budget)
|
||||||
|
except Exception as exc:
|
||||||
|
log.debug("测速过程异常: %s", str(exc)[:120])
|
||||||
|
elapsed = time.monotonic() - t0
|
||||||
|
|
||||||
self._file.write_text(
|
responsive = {
|
||||||
json.dumps(cfg, ensure_ascii=False, indent=2), encoding="utf-8"
|
ip: {p: lat for p, lat in per.items() if lat is not None}
|
||||||
|
for ip, per in results.items()
|
||||||
|
}
|
||||||
|
responsive = {ip: per for ip, per in responsive.items() if per}
|
||||||
|
if not responsive:
|
||||||
|
log.debug("测速结束: 无任何可达主机,不写缓存(下次启动重试)")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _ordered(protocol: str) -> list[str]:
|
||||||
|
rows = [(per[protocol], ip) for ip, per in responsive.items()
|
||||||
|
if protocol in per]
|
||||||
|
rows.sort(key=lambda kv: (kv[0], kv[1]))
|
||||||
|
return [ip for _, ip in rows]
|
||||||
|
|
||||||
|
hosts_by_proto = {p: _ordered(p) for p in ("standard", "mac", "ex")}
|
||||||
|
overall = sorted(
|
||||||
|
(min(per.values()), ip) for ip, per in responsive.items()
|
||||||
)
|
)
|
||||||
log.info("测速结果已写入 %s", self._file)
|
fast = {ip for i, (lat, ip) in enumerate(overall)
|
||||||
|
if i < FAST_POOL_TOP_N or lat < FAST_POOL_MAX_LATENCY}
|
||||||
|
|
||||||
|
log.debug("测速完成: 候选 %d 台, 可达 %d 台, fast %d 台, 耗时 %.2fs",
|
||||||
|
len({ip for _, ip in jobs}), len(responsive), len(fast), elapsed)
|
||||||
|
return {
|
||||||
|
"version": PROBE_CACHE_VERSION,
|
||||||
|
"date": self.today(),
|
||||||
|
"probed_at": time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||||
|
"elapsed": round(elapsed, 3),
|
||||||
|
# 每个协议的完整探测结果(含 None = 本次不可达),便于事后诊断
|
||||||
|
"results": {ip: dict(per) for ip, per in results.items() if per},
|
||||||
|
# 按延迟升序的可达主机(运行时即按此顺序取用)
|
||||||
|
"hosts": hosts_by_proto,
|
||||||
|
# 诊断用:符合"前N名或<200ms"的可达主机
|
||||||
|
"fast": {p: [h for h in ips if h in fast]
|
||||||
|
for p, ips in hosts_by_proto.items()},
|
||||||
|
}
|
||||||
|
|
||||||
|
def _write(self, payload: dict) -> None:
|
||||||
|
"""原子写入:先写临时文件再 ``os.replace``,避免读到半截 JSON。"""
|
||||||
|
target = self.cache_file()
|
||||||
|
tmp = target.with_name(f"{target.name}.tmp{os.getpid()}")
|
||||||
|
try:
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
tmp.write_text(json.dumps(payload, ensure_ascii=False, indent=2),
|
||||||
|
encoding="utf-8")
|
||||||
|
os.replace(tmp, target)
|
||||||
|
log.debug("当日测速结果已缓存: %s", target)
|
||||||
|
except OSError as exc:
|
||||||
|
log.debug("测速结果写入失败(忽略): %s", str(exc)[:120])
|
||||||
|
try:
|
||||||
|
tmp.unlink(missing_ok=True)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
# 进程级单例(跨池共享,确保当日仅测一次)
|
# 进程级单例(标准 / MAC / EX 三池共享,确保当日只真正测一次)
|
||||||
_probe_cache = ProbeCache()
|
_probe_cache = ProbeCache()
|
||||||
|
|
||||||
|
|
||||||
def get_probe_cache() -> ProbeCache:
|
def get_probe_cache() -> ProbeCache:
|
||||||
return _probe_cache
|
return _probe_cache
|
||||||
|
|
||||||
|
|
||||||
|
def schedule_probe(hosts: list[str] | None = None) -> asyncio.Task | None:
|
||||||
|
"""[连接池用] 把当日测速挂到后台并返回 Task,供调用方持有 / 取消。
|
||||||
|
|
||||||
|
返回 ``None`` 表示当前没有运行中的事件循环(此时无法起后台任务)。
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
return asyncio.create_task(
|
||||||
|
get_probe_cache().ensure_probe(hosts), name="pytdxdata-daily-probe"
|
||||||
|
)
|
||||||
|
except RuntimeError:
|
||||||
|
return None
|
||||||
@@ -0,0 +1,365 @@
|
|||||||
|
"""当日测速缓存单元测试(注入假探针与假日期,不连真实网络)
|
||||||
|
|
||||||
|
覆盖点:
|
||||||
|
- 当日首次启动测一次;当天已有缓存则**不刷新**(探针一次都不发起)
|
||||||
|
- 跨日 / 缓存损坏 / 版本不符 → 重新测速
|
||||||
|
- 结果按实测延迟升序被 config 采用
|
||||||
|
- **绝不改写包内 servers.json**,缓存落在用户级缓存目录
|
||||||
|
- 全部不可达 → 不写缓存(下次启动重试)
|
||||||
|
- 并发调用只真正测一次(单飞)
|
||||||
|
- 非法主机名(测试 mock 混入的 "s1" 等)被挡在测速之外
|
||||||
|
- 连接池 start() 不阻塞、close() 会取消后台测速
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
from datetime import date, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from pytdxdata import config
|
||||||
|
from pytdxdata.pool.probe_cache import ProbeCache
|
||||||
|
|
||||||
|
TODAY = date.today().isoformat()
|
||||||
|
YESTERDAY = (date.today() - timedelta(days=1)).isoformat()
|
||||||
|
PROTOS = ("standard", "mac", "ex")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def isolated_cache(tmp_path, monkeypatch):
|
||||||
|
"""把用户级缓存目录指到临时目录,并复位所有相关环境变量。"""
|
||||||
|
monkeypatch.setenv("PYPYTDX_DATA_CACHE_DIR", str(tmp_path))
|
||||||
|
monkeypatch.delenv("PYPYTDX_DATA_DISABLE_PROBE", raising=False)
|
||||||
|
for var in ("PYPYTDX_DATA_STANDARD_HOSTS", "PYPYTDX_DATA_MAC_HOSTS",
|
||||||
|
"PYPYTDX_DATA_EX_HOSTS"):
|
||||||
|
monkeypatch.delenv(var, raising=False)
|
||||||
|
return tmp_path
|
||||||
|
|
||||||
|
|
||||||
|
def fake_probers(latencies: dict[str, dict[str, float | None]], calls: list):
|
||||||
|
"""构造假探针;latencies = {协议: {主机: 延迟或 None}}。"""
|
||||||
|
def _make(protocol: str):
|
||||||
|
async def _probe(host: str, timeout: float) -> float | None:
|
||||||
|
calls.append((protocol, host))
|
||||||
|
return latencies.get(protocol, {}).get(host)
|
||||||
|
return _probe
|
||||||
|
return {p: _make(p) for p in PROTOS}
|
||||||
|
|
||||||
|
|
||||||
|
def patch_candidates(monkeypatch, mapping: dict[str, list[str]]) -> None:
|
||||||
|
"""把"全量候选"换成受控清单,避免依赖真实服务器清单。"""
|
||||||
|
monkeypatch.setattr("pytdxdata.pool.probe_cache.all_probe_candidates",
|
||||||
|
lambda: mapping)
|
||||||
|
|
||||||
|
|
||||||
|
def write_cache(path: Path, day: str, results: dict, version: int = 1) -> None:
|
||||||
|
path.write_text(json.dumps({"version": version, "date": day, "results": results}),
|
||||||
|
encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 是否该测速 ----------
|
||||||
|
|
||||||
|
def test_needs_probe_when_no_cache(tmp_path):
|
||||||
|
pc = ProbeCache(tmp_path / "c.json", today_fn=lambda: TODAY)
|
||||||
|
assert pc.needs_probe_today() is True
|
||||||
|
assert pc.read_cache() is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_needs_probe_false_for_today_cache(tmp_path):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
write_cache(cache, TODAY, {"1.1.1.1": {"standard": 0.05}})
|
||||||
|
pc = ProbeCache(cache, today_fn=lambda: TODAY)
|
||||||
|
assert pc.needs_probe_today() is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_stale_cache_needs_probe(tmp_path):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
write_cache(cache, YESTERDAY, {"1.1.1.1": {"standard": 0.05}})
|
||||||
|
pc = ProbeCache(cache, today_fn=lambda: TODAY)
|
||||||
|
assert pc.needs_probe_today() is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_version_mismatch_needs_probe(tmp_path):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
write_cache(cache, TODAY, {"1.1.1.1": {"standard": 0.05}}, version=999)
|
||||||
|
pc = ProbeCache(cache, today_fn=lambda: TODAY)
|
||||||
|
assert pc.needs_probe_today() is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("content", ["{not json", "[]", '{"version":1}', ""])
|
||||||
|
def test_broken_cache_is_ignored_not_crash(tmp_path, content):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
cache.write_text(content, encoding="utf-8")
|
||||||
|
pc = ProbeCache(cache, today_fn=lambda: TODAY)
|
||||||
|
assert pc.read_cache() is None
|
||||||
|
assert pc.needs_probe_today() is True
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 核心行为:当天已有缓存就不刷新 ----------
|
||||||
|
|
||||||
|
async def test_today_cache_is_not_refreshed(tmp_path, monkeypatch):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
write_cache(cache, TODAY, {"1.1.1.1": {"standard": 0.05}})
|
||||||
|
before = cache.read_text(encoding="utf-8")
|
||||||
|
calls: list = []
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
|
||||||
|
pc = ProbeCache(cache, probers=fake_probers({}, calls), today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
|
||||||
|
assert calls == [] # 未发起任何测速
|
||||||
|
assert cache.read_text(encoding="utf-8") == before # 缓存内容原封不动
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stale_cache_is_refreshed(tmp_path, monkeypatch):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
write_cache(cache, YESTERDAY, {"9.9.9.9": {"standard": 9.0}})
|
||||||
|
calls: list = []
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
|
||||||
|
pc = ProbeCache(cache,
|
||||||
|
probers=fake_probers({"standard": {"1.1.1.1": 0.02}}, calls),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
|
||||||
|
assert calls == [("standard", "1.1.1.1")]
|
||||||
|
data = json.loads(cache.read_text(encoding="utf-8"))
|
||||||
|
assert data["date"] == TODAY and data["version"] == config.PROBE_CACHE_VERSION
|
||||||
|
assert data["results"] == {"1.1.1.1": {"standard": 0.02}}
|
||||||
|
assert data["hosts"]["standard"] == ["1.1.1.1"]
|
||||||
|
assert "9.9.9.9" not in data["results"] # 旧结果被整体替换,不残留
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 缓存写入 ----------
|
||||||
|
|
||||||
|
async def test_write_is_atomic_and_leaves_no_tmp(tmp_path, monkeypatch):
|
||||||
|
cache = tmp_path / "sub" / "c.json" # 目录不存在也要能建
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
pc = ProbeCache(cache,
|
||||||
|
probers=fake_probers({"standard": {"1.1.1.1": 0.03}}, []),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
|
||||||
|
assert cache.exists()
|
||||||
|
assert list(cache.parent.glob("*.tmp*")) == []
|
||||||
|
assert json.loads(cache.read_text(encoding="utf-8"))["probed_at"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_all_unreachable_writes_nothing(tmp_path, monkeypatch):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
pc = ProbeCache(cache, probers=fake_probers({}, []), today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
|
||||||
|
assert not cache.exists() # 不把"全挂"缓存一整天
|
||||||
|
assert pc.needs_probe_today() is True
|
||||||
|
|
||||||
|
|
||||||
|
async def test_never_touches_package_servers_json(tmp_path, monkeypatch):
|
||||||
|
"""最关键的一条:测速**绝不能**改写随包分发的 servers.json。"""
|
||||||
|
servers_json = config._SERVERS_FILE
|
||||||
|
before_bytes = servers_json.read_bytes()
|
||||||
|
before_mtime = servers_json.stat().st_mtime_ns
|
||||||
|
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
pc = ProbeCache(probers=fake_probers({"standard": {"1.1.1.1": 0.04}}, []),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
|
||||||
|
assert servers_json.read_bytes() == before_bytes
|
||||||
|
assert servers_json.stat().st_mtime_ns == before_mtime
|
||||||
|
assert pc.cache_file() == tmp_path / "hosts-today.json" # 落在用户缓存目录
|
||||||
|
assert Path(config.probe_cache_file()).parent == tmp_path
|
||||||
|
|
||||||
|
|
||||||
|
async def test_results_are_recorded_per_protocol(tmp_path, monkeypatch):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
patch_candidates(monkeypatch, {
|
||||||
|
"standard": ["1.1.1.1", "2.2.2.2"],
|
||||||
|
"mac": ["1.1.1.1", "2.2.2.2"],
|
||||||
|
"ex": ["3.3.3.3"],
|
||||||
|
})
|
||||||
|
lats = {"standard": {"1.1.1.1": 0.30, "2.2.2.2": 0.10},
|
||||||
|
"mac": {"1.1.1.1": 0.20}, # 2.2.2.2 的 mac 故意不给 → 应记为 None
|
||||||
|
"ex": {"3.3.3.3": 0.05}}
|
||||||
|
pc = ProbeCache(cache, probers=fake_probers(lats, []), today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
|
||||||
|
data = json.loads(cache.read_text(encoding="utf-8"))
|
||||||
|
assert data["hosts"]["standard"] == ["2.2.2.2", "1.1.1.1"] # 按延迟升序
|
||||||
|
assert data["hosts"]["mac"] == ["1.1.1.1"] # 不可达的 2.2.2.2 被排除
|
||||||
|
assert data["hosts"]["ex"] == ["3.3.3.3"]
|
||||||
|
# 2.2.2.2 的 mac 探针返回 None,也要如实记录(便于事后诊断)
|
||||||
|
assert data["results"]["2.2.2.2"]["mac"] is None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 单飞 / 开关 / 脏数据 ----------
|
||||||
|
|
||||||
|
async def test_concurrent_ensure_probe_runs_once(tmp_path, monkeypatch):
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1", "2.2.2.2"], "mac": [], "ex": []})
|
||||||
|
calls: list = []
|
||||||
|
pc = ProbeCache(tmp_path / "c.json",
|
||||||
|
probers=fake_probers({"standard": {"1.1.1.1": 0.01,
|
||||||
|
"2.2.2.2": 0.02}}, calls),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
await asyncio.gather(pc.ensure_probe(), pc.ensure_probe(), pc.ensure_probe())
|
||||||
|
|
||||||
|
assert sorted(calls) == [("standard", "1.1.1.1"), ("standard", "2.2.2.2")]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_disabled_env_skips_everything(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("PYPYTDX_DATA_DISABLE_PROBE", "1")
|
||||||
|
calls: list = []
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
pc = ProbeCache(tmp_path / "c.json",
|
||||||
|
probers=fake_probers({"standard": {"1.1.1.1": 0.01}}, calls),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
|
||||||
|
assert pc.needs_probe_today() is False
|
||||||
|
await pc.ensure_probe()
|
||||||
|
assert calls == []
|
||||||
|
assert not (tmp_path / "c.json").exists()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_illegal_hosts_are_filtered_out(tmp_path, monkeypatch):
|
||||||
|
"""测试 mock 常传入 "s1"/"m1" 这类伪主机名,必须挡在缓存之外。"""
|
||||||
|
patch_candidates(monkeypatch, {"standard": [], "mac": [], "ex": []})
|
||||||
|
calls: list = []
|
||||||
|
pc = ProbeCache(tmp_path / "c.json",
|
||||||
|
probers=fake_probers({"standard": {"1.1.1.1": 0.01}}, calls),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe(["s1", "m1", "1.1.1.1", ""])
|
||||||
|
|
||||||
|
assert calls == [("standard", "1.1.1.1")]
|
||||||
|
data = json.loads((tmp_path / "c.json").read_text(encoding="utf-8"))
|
||||||
|
assert "s1" not in data["results"] and "m1" not in data["results"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_reset_allows_recheck(tmp_path, monkeypatch):
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
calls: list = []
|
||||||
|
pc = ProbeCache(tmp_path / "c.json",
|
||||||
|
probers=fake_probers({"standard": {"1.1.1.1": 0.01}}, calls),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
await pc.ensure_probe()
|
||||||
|
assert len(calls) == 1 # 第二次被单飞挡掉
|
||||||
|
|
||||||
|
pc.reset()
|
||||||
|
await pc.ensure_probe()
|
||||||
|
assert len(calls) == 1 # 重置后当天已有缓存,仍不重测
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- config 侧:采用顺序与兜底 ----------
|
||||||
|
|
||||||
|
def test_config_getters_prefer_cache_and_order_by_latency(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("PYPYTDX_DATA_CACHE_DIR", str(tmp_path))
|
||||||
|
(tmp_path / "hosts-today.json").write_text(json.dumps({
|
||||||
|
"version": config.PROBE_CACHE_VERSION, "date": TODAY,
|
||||||
|
"results": {
|
||||||
|
"3.3.3.3": {"standard": 0.9},
|
||||||
|
"1.1.1.1": {"standard": 0.05},
|
||||||
|
"2.2.2.2": {"standard": 0.4, "mac": 0.1},
|
||||||
|
"9.9.9.9": {"ex": 0.2},
|
||||||
|
},
|
||||||
|
}), encoding="utf-8")
|
||||||
|
|
||||||
|
assert config.get_standard_hosts() == ["1.1.1.1", "2.2.2.2", "3.3.3.3"]
|
||||||
|
assert config.get_mac_hosts() == ["2.2.2.2"]
|
||||||
|
assert config.get_ex_hosts() == ["9.9.9.9"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_config_getters_fall_back_without_cache(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("PYPYTDX_DATA_CACHE_DIR", str(tmp_path))
|
||||||
|
assert config.get_standard_hosts() == config._load_ok_hosts("standard", fast_only=True)
|
||||||
|
assert config.get_mac_hosts() == config._load_ok_hosts("mac", fast_only=True)
|
||||||
|
assert config.get_ex_hosts() == config._load_ok_hosts("ex")
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_override_beats_cache(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("PYPYTDX_DATA_CACHE_DIR", str(tmp_path))
|
||||||
|
(tmp_path / "hosts-today.json").write_text(json.dumps({
|
||||||
|
"version": config.PROBE_CACHE_VERSION, "date": TODAY,
|
||||||
|
"results": {"1.1.1.1": {"standard": 0.05}},
|
||||||
|
}), encoding="utf-8")
|
||||||
|
monkeypatch.setenv("PYPYTDX_DATA_STANDARD_HOSTS", "8.8.8.8")
|
||||||
|
assert config.get_standard_hosts() == ["8.8.8.8"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_all_probe_candidates_covers_full_set_not_filtered():
|
||||||
|
cand = config.all_probe_candidates()
|
||||||
|
assert set(config.STANDARD_HOSTS) <= set(cand["standard"])
|
||||||
|
assert set(config.MAC_HOSTS) <= set(cand["mac"])
|
||||||
|
assert set(config.EX_HOSTS) <= set(cand["ex"])
|
||||||
|
assert "117.34.114.17" in cand["standard"] # 未申报 MAC 的节点也要被测到
|
||||||
|
assert "s1" not in cand["standard"] # 包内清单里的脏数据被挡
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(("host", "expected"), [
|
||||||
|
("1.2.3.4", True),
|
||||||
|
("117.34.114.17", True),
|
||||||
|
("a.b", True),
|
||||||
|
("good-host.com", True),
|
||||||
|
("s1", False),
|
||||||
|
("m1", False),
|
||||||
|
("", False),
|
||||||
|
(" ", False),
|
||||||
|
("-bad.com", False),
|
||||||
|
("bad-.com", False),
|
||||||
|
("has space.com", False),
|
||||||
|
(123, False),
|
||||||
|
(None, False),
|
||||||
|
])
|
||||||
|
def test_is_valid_host(host, expected):
|
||||||
|
assert config.is_valid_host(host) is expected
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 连接池集成 ----------
|
||||||
|
|
||||||
|
async def test_pool_start_does_not_block_and_close_cancels_probe(monkeypatch):
|
||||||
|
from pytdxdata.pool import pool as pool_mod
|
||||||
|
|
||||||
|
cancel_seen = asyncio.Event()
|
||||||
|
created: list[asyncio.Task] = []
|
||||||
|
|
||||||
|
async def _hang(hosts=None):
|
||||||
|
try:
|
||||||
|
await asyncio.sleep(30) # 模拟耗时测速
|
||||||
|
except asyncio.CancelledError:
|
||||||
|
cancel_seen.set()
|
||||||
|
raise
|
||||||
|
|
||||||
|
def _fake_schedule(hosts=None):
|
||||||
|
task = asyncio.create_task(_hang())
|
||||||
|
created.append(task)
|
||||||
|
return task
|
||||||
|
|
||||||
|
monkeypatch.setattr(pool_mod, "schedule_probe", _fake_schedule)
|
||||||
|
|
||||||
|
p = pool_mod.DynamicPool([], min_size=1, max_size=1)
|
||||||
|
t0 = time.monotonic()
|
||||||
|
await p.start()
|
||||||
|
assert time.monotonic() - t0 < 1.0 # start() 不等测速
|
||||||
|
assert p._probe_task is created[0]
|
||||||
|
|
||||||
|
await asyncio.sleep(0) # 让测速任务真正跑起来再关闭
|
||||||
|
await p.close()
|
||||||
|
assert p._probe_task is None
|
||||||
|
assert created[0].cancelled()
|
||||||
|
assert cancel_seen.is_set()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_pool_auto_probe_false_schedules_nothing(monkeypatch):
|
||||||
|
from pytdxdata.pool import pool as pool_mod
|
||||||
|
|
||||||
|
def _boom(hosts=None): # pragma: no cover - 不应被调用
|
||||||
|
raise AssertionError("auto_probe=False 时不应调度测速")
|
||||||
|
|
||||||
|
monkeypatch.setattr(pool_mod, "schedule_probe", _boom)
|
||||||
|
p = pool_mod.DynamicPool([], min_size=1, max_size=1, auto_probe=False)
|
||||||
|
await p.start()
|
||||||
|
assert p._probe_task is None
|
||||||
|
await p.close()
|
||||||
Reference in new issue
Block a user