Compare commits
14
Commits
v0.6.0
..
feec86bf13
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
feec86bf13 | ||
|
|
0618062e54 | ||
|
|
9c97fea71b | ||
|
|
2d62d82b51 | ||
|
|
7e72f70eb2 | ||
|
|
b2f899df49 | ||
|
|
8193d560da | ||
|
|
5408903a4c | ||
|
|
8951ec2678 | ||
|
|
b78d0fabf5 | ||
|
|
d422385877 | ||
|
|
7e830c7579 | ||
|
|
c7c6c70333 | ||
|
|
18d9e9a006 |
No files matched your search
+117
@@ -2,6 +2,123 @@
|
|||||||
|
|
||||||
本项目遵循 [语义化版本](https://semver.org/lang/zh-CN/)。
|
本项目遵循 [语义化版本](https://semver.org/lang/zh-CN/)。
|
||||||
|
|
||||||
|
## [0.6.7] - 2026-10-22
|
||||||
|
|
||||||
|
### 新增
|
||||||
|
- `get_index_members` / `get_index_trade_stats` / `get_index_trade_stats_multi`:指数成分股与逐笔聚合统计(分钟级增量;北交所成分无笔数自动剔除)
|
||||||
|
- `sync_decimal_points` / `decimal_point_status`:权威价格小数位表运行时同步(报价标度不再依赖硬编码前缀规则),`start()` 自动装载
|
||||||
|
- `decimal_sync` 模块
|
||||||
|
|
||||||
|
### 修复
|
||||||
|
- **指数/ETF/LOF 逐笔的 `trade_count` 语义修正**:协议字段在指数上是「均价×1000」、ETF/LOF 上是「价格×1000」,**不是笔数**;新增 `TransactionRecord.trade_count_available`(判定见 `symbols.has_order_count`),CLI 对无笔数口径标的显示 `(-)` 而非假笔数
|
||||||
|
- 集合竞价委托(bs_flag=8)不再混入逐笔结果(会被覆盖、Σvol 虚高 20%~160%、不同主机条数不一致)
|
||||||
|
- 缓存管理器 / 连接池 / 分页调度若干健壮性修复
|
||||||
|
|
||||||
|
### 说明
|
||||||
|
- 0.6.4~0.6.6 为中间迭代,未单独发版记录;本次合并 PyPI sdist 与仓库,源码恢复双仓一致
|
||||||
|
|
||||||
|
|
||||||
|
- fix(pool): ExPool acquire 背压等待(Condition 排队,默认 30s 超时)替代 3x0.05s 快速失败;超时异常带池状态
|
||||||
|
- feat(api): 新增 ex_pool_cap 参数(默认 2),单 EX 服务器场景不再静默退化为 1 并发(fixes #1)
|
||||||
|
|
||||||
|
## [0.6.2] - 2026-10-13
|
||||||
|
|
||||||
|
### 修复
|
||||||
|
|
||||||
|
- **美股分钟级 K 线的 `datetime` 全部错位**(`get_bars("usAAPL", KlinePeriod.MIN_1)`)。
|
||||||
|
|
||||||
|
通达信 EX 通道返回的美股分钟线时间戳,把**两个时区拼进了同一个五元组**:日期分量是
|
||||||
|
**美东交易日**(协议里的 `zipday`),时分分量却是**北京时间**(`tminutes`)。直拼出的
|
||||||
|
`datetime` 会把同一交易日的**下午半场排到上午半场之前**——单日 390 根不仅首末颠倒,
|
||||||
|
而且全部落在同一个北京自然日里。于是任何依赖 `datetime` 的行为都是错的:排序、
|
||||||
|
`count` 切片("取最新 N 根")、日期区间二分与过滤、增量更新水位线。
|
||||||
|
|
||||||
|
| 观测项 | 修复前 | 修复后 |
|
||||||
|
|---|---|---|
|
||||||
|
| 单日 390 根的日期分布 | **全部**落在 10-12(hour 0~23 混在一起) | 10-12 的 21:31~23:59(149 根)+ 10-13 的 00:00~04:00(241 根) |
|
||||||
|
| 时间轴 | 上午半场被排到下午半场**之前** | 严格单调递增 |
|
||||||
|
| `count=30` 是否取到真正最新的 30 根 | 否(拿到上一交易日的 23:30~23:59) | 是(当日收盘段 03:31~04:00) |
|
||||||
|
| 每组首根 `open` / 末根 `close` 对日线 | 对不上 | **逐位相同**(10-07/08/09/12 四个交易日实测) |
|
||||||
|
|
||||||
|
现在 `SecurityBar.datetime` **统一为北京时间**:美股凌晨段(北京 hour < 12)的日期
|
||||||
|
自动 +1 天(含月末 / 年末进位)。A股 / 港股 / 期货原始即为北京时间,**零行为变化**。
|
||||||
|
|
||||||
|
新增两个只读属性:
|
||||||
|
|
||||||
|
| 属性 | 用途 |
|
||||||
|
|---|---|
|
||||||
|
| `SecurityBar.trading_date` | 该 bar 归属的交易日 `YYYY-MM-DD`。美股一个交易日横跨两个北京自然日,**按交易日聚合(分钟线合成日线、日内统计)必须用它,不能用 `datetime.date()`** |
|
||||||
|
| `SecurityBar.is_us` | 是否美股(EX 通道 `US_STOCK`) |
|
||||||
|
|
||||||
|
归一化在 `SecurityBar.__post_init__` 中实现,因此 **EX 的两条 K 线命令路径**
|
||||||
|
(`_bars_ex` 的 0x23FF 与日期区间二分所依赖的 MAC 0x122E)自动共用同一份口径,
|
||||||
|
不会出现"取数按一套时间、二分按另一套时间"的错位。
|
||||||
|
|
||||||
|
### 说明
|
||||||
|
|
||||||
|
- 分钟线缓存 TTL 为 **60 秒**,升级后旧缓存最多残留 1 分钟即自然失效,**无需手动清理**;
|
||||||
|
日线(`period >= DAY`)不受本缺陷影响。
|
||||||
|
- 本版只修 K 线。**同日发现的同类问题未包含在内**:EX 分时(`get_minutes`)对美股返回的
|
||||||
|
`MinuteBar.hour` 是**未取模的连续分钟数**(实测可达 27,即次日 03:xx),且 `MinuteBar`
|
||||||
|
无日期分量可承载跨日信息——修法涉及"是否新增字段 / 日期如何归属"的取舍,留待下一版决策。
|
||||||
|
- 新增 9 项针对性回归测试(分界点、月末与年末进位、390 根整日单调性、MAC 路径同样归一化)。
|
||||||
|
|
||||||
|
## [0.6.1] - 2026-09-21
|
||||||
|
|
||||||
|
### 变更
|
||||||
|
|
||||||
|
- **重写「当日首次启动测速缓存」,改为对用户完全无感**。原实现在 v0.5.0 已随包发布,但存在
|
||||||
|
三个会让它**静默失效**的缺陷,本次一并修正:
|
||||||
|
|
||||||
|
| 缺陷 | 原实现 | 现在 |
|
||||||
|
|---|---|---|
|
||||||
|
| 候选集自指 | 连接池只把自己**已过滤后的成员**交给探针 → 只重测了 9 台,另外 20+ 台候选(含快节点)永远测不到 | 探针内部取**全部候选**(包内清单 ∪ 代码常量 ∪ env) |
|
||||||
|
| 写回安装目录 | 结果写回 `src/pytdxdata/servers.json` → 容器 / 系统级安装只读、多用户互相覆盖 | 改写入**用户级缓存目录**,包内清单运行时**绝不被改写** |
|
||||||
|
| 探测太弱 | 只做握手(`TdxConnection.test`),测不出「握手正常但 K 线空响应」的坏节点 | 发**真实业务命令**并要求返回非空(标准 / MAC / EX 各一) |
|
||||||
|
| EX 池漏测 | EX 的 4 台从未被探测 | EX 一并纳入(7727 + 各自握手模式) |
|
||||||
|
|
||||||
|
- 测速结果按**实测延迟升序**被 `config.get_standard_hosts()` 等采用;当天已有缓存则**不再刷新**。
|
||||||
|
后台执行,不阻塞 `start()`、不打印、失败静默降级。
|
||||||
|
|
||||||
|
- 本版基于 **0.6.0 的统一对外接口**构建(`get_bars` + 字符串标的);同时把 0.6.0
|
||||||
|
全接口回归测试里对旧探针入口(`pool.get_probe_cache`)的替身改为新入口(`pool.schedule_probe`),
|
||||||
|
保证单测不触发真实网络测速。
|
||||||
|
|
||||||
|
### 修复
|
||||||
|
|
||||||
|
- **A股分时缩略采样 100% 失败**(`get_minutes(sampling=True)` / `get_chart_sampling()`):
|
||||||
|
旧实现走 `MacChartSamplingCmd`(**0x254D**),但该 opcode 属于 **EX 协议**,7709 标准 /
|
||||||
|
MAC 主机对它一律不响应——实测 3 台主机各 10s 超时,叠加连接池 3 次重试约 24s 才抛错。
|
||||||
|
现改为**由分时(0x122D)数据派生**价格点(分时本身就是 240 点,语义一致),
|
||||||
|
实测 4 个 A股 场景均在 0.1s 内返回 240 点;EX 通道仍用 0x254D(84 点)。
|
||||||
|
|
||||||
|
- **历史专业财报整条链路不可用**(`get_financial_records_parsed()`):
|
||||||
|
`tdxfin/gpcw.txt` 索引里写的是裸 `gpcwYYYYMMDD.zip`,而服务器**实际只提供未压缩的**
|
||||||
|
`tdxfin/gpcwYYYYMMDD.dat`。旧实现把索引名直接丢给 `zipfile` → 必然 `BadZipFile`;
|
||||||
|
废弃别名 `get_financial_records()` 更危险,会**静默返回 0~1 字节空壳**而不报错。
|
||||||
|
现按候选名(`tdxfin/*.zip` → `tdxfin/*.dat` → 原名)逐个尝试并校验数据形态,
|
||||||
|
zip 与未压缩 dat 都支持。实测 `gpcw20260630.zip` → **5568 条**、
|
||||||
|
`gpcw20260331.zip` → **5547 条**(样本 000001,584 个 float 字段),
|
||||||
|
别名返回 **13,068,116 字节**真实数据;尚未生成的占位条目
|
||||||
|
(如 `gpcw20260930.zip`,filesize=164)优雅返回空列表而非报错。
|
||||||
|
|
||||||
|
### 新增
|
||||||
|
|
||||||
|
- `codec.financial` 新增 `looks_like_gpcw_dat()` / `extract_gpcw_dat()`:
|
||||||
|
财报字节的形态判定与「zip / 未压缩 dat」统一提取(纯标准库,可脱网单测)
|
||||||
|
- 环境变量 `PYPYTDX_DATA_CACHE_DIR`(覆盖测速缓存目录)、
|
||||||
|
`PYPYTDX_DATA_DISABLE_PROBE`(关闭测速,CI / 测试用)
|
||||||
|
- `tests/test_probe_cache.py`(37 项):当日缓存不刷新、跨日重测、
|
||||||
|
原子写入且不留 tmp、**不改写包内 `servers.json`(逐字节校验)**、
|
||||||
|
单飞并发、非法主机过滤、`DISABLE_PROBE`、池 `start()` 不阻塞 / `close()` 取消后台探针
|
||||||
|
|
||||||
|
### 文档
|
||||||
|
|
||||||
|
- [安装与配置](https://openbot-coder.github.io/pytdxdata/getting-started/#当日自动测速无感):
|
||||||
|
新增「当日自动测速(无感)」小节——缓存路径、优先级链、环境变量一览
|
||||||
|
- [通道与数据源](https://openbot-coder.github.io/pytdxdata/internals/routing/#daily-probe):
|
||||||
|
新增「运行时:当日测速缓存」小节,逐条说明必须踩对的实现要点与反例
|
||||||
|
|
||||||
## [0.6.0] - 2026-09-16
|
## [0.6.0] - 2026-09-16
|
||||||
|
|
||||||
### 新增
|
### 新增
|
||||||
|
|||||||
@@ -65,7 +65,7 @@ df = pd.DataFrame([asdict(b) for b in bars])
|
|||||||
| 🔀 双通道整合 | 标准通道(经典行情)+ MAC 通道(逐笔 / 复权 / 1 分钟 K 线),按命令自动选路 |
|
| 🔀 双通道整合 | 标准通道(经典行情)+ MAC 通道(逐笔 / 复权 / 1 分钟 K 线),按命令自动选路 |
|
||||||
| 📦 零重型依赖 | 纯 asyncio + stdlib(socket / zlib / struct / sqlite3) |
|
| 📦 零重型依赖 | 纯 asyncio + stdlib(socket / zlib / struct / sqlite3) |
|
||||||
| 📊 全市场覆盖 | A股、港股、美股、期货、期权(46 个扩展市场) |
|
| 📊 全市场覆盖 | A股、港股、美股、期货、期权(46 个扩展市场) |
|
||||||
| 📄 服务器文件解析 | 板块 `block_*.dat` / 行业 `tdxhy.cfg` / 财报 `gpcw*.zip` **下载即解析**成 dataclass |
|
| 📄 服务器文件解析 | 板块 `block_*.dat` / 行业 `tdxhy.cfg` / 财报 `gpcw*` **下载即解析**成 dataclass(索引名与服务器实际文件名不一致时自动纠正) |
|
||||||
| 🤖 AI 助手友好 | 内置 [`SKILL.md`](https://github.com/openbot-coder/pytdxdata/blob/main/SKILL.md) 与 [`llms.txt`](https://openbot-coder.github.io/pytdxdata/llms.txt) |
|
| 🤖 AI 助手友好 | 内置 [`SKILL.md`](https://github.com/openbot-coder/pytdxdata/blob/main/SKILL.md) 与 [`llms.txt`](https://openbot-coder.github.io/pytdxdata/llms.txt) |
|
||||||
|
|
||||||
## 文档导航
|
## 文档导航
|
||||||
@@ -104,7 +104,7 @@ curl -o ~/.workbuddy/skills/pytdxdata/SKILL.md --create-dirs \
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv sync # 建虚拟环境 + 注册 tdx 命令
|
uv sync # 建虚拟环境 + 注册 tdx 命令
|
||||||
uv run pytest tests/ # 127 个测试用例(协议 / 池 / 缓存 / 分页 / 全接口回归 / 文件解析 / CLI / 文档漂移)
|
uv run pytest tests/ # 191 个测试用例(协议 / 池 / 缓存 / 分页 / 全接口回归 / 文件解析 / CLI / 文档漂移 / 当日测速缓存)
|
||||||
uv sync --group docs # 文档站依赖(mkdocs-material + mkdocstrings)
|
uv sync --group docs # 文档站依赖(mkdocs-material + mkdocstrings)
|
||||||
uv run mkdocs serve # 本地预览文档站
|
uv run mkdocs serve # 本地预览文档站
|
||||||
```
|
```
|
||||||
@@ -114,7 +114,12 @@ uv run mkdocs serve # 本地预览文档站
|
|||||||
- 行情来自通达信公开服务器,**15 分钟延时**;需实时数据请使用券商行情
|
- 行情来自通达信公开服务器,**15 分钟延时**;需实时数据请使用券商行情
|
||||||
- 标准通道 K 线 `count < 100` 部分服务器返回空 → 分页器已强制 `min_page=100` 规避
|
- 标准通道 K 线 `count < 100` 部分服务器返回空 → 分页器已强制 `min_page=100` 规避
|
||||||
- 港股 / 美股期权在通达信协议内无数据
|
- 港股 / 美股期权在通达信协议内无数据
|
||||||
- 服务器可用性随时间变化,可用 `python -m pytdxdata.server_probe` 重新探测
|
- **K 线时间戳统一为北京时间**(v0.6.2 起)。美股原始报文把「美东交易日日期」与「北京时间时分」
|
||||||
|
拼在一起,库已自动归一化,`bar.datetime` 可直接排序 / 过滤;但**美股一个交易日横跨两个北京
|
||||||
|
自然日**,按交易日聚合请用 `SecurityBar.trading_date`,不要用 `datetime.date()`
|
||||||
|
- 服务器可用性随时间变化。**当日首次启动会自动在后台测速并缓存**(对调用方无感,当天不再重测);
|
||||||
|
也可用 `python -m pytdxdata.server_probe` 离线重生成随包清单。
|
||||||
|
缓存路径与三个开关见 [安装与配置 → 当日自动测速](https://openbot-coder.github.io/pytdxdata/getting-started/#当日自动测速无感)
|
||||||
|
|
||||||
## 免责声明
|
## 免责声明
|
||||||
|
|
||||||
|
|||||||
@@ -164,6 +164,7 @@ await td.get_bars("cffex:IFL0", KlinePeriod.DAY, count=700) # 期货合约
|
|||||||
|
|
||||||
- **数据有 15 分钟延时**(通达信公开服务器特性)。要实时数据请用券商行情。
|
- **数据有 15 分钟延时**(通达信公开服务器特性)。要实时数据请用券商行情。
|
||||||
- **复权 K 线与 1 分钟 K 线走 MAC 通道**。`get_bars(..., adjust=Adjust.QFQ)` 自动切换。
|
- **复权 K 线与 1 分钟 K 线走 MAC 通道**。`get_bars(..., adjust=Adjust.QFQ)` 自动切换。
|
||||||
|
- **K 线时间戳统一为北京时间**(v0.6.2 起)。美股原始报文里日期是美东交易日、时分是北京时间(两个时区拼在一起),库已自动归一化,`bar.datetime` 可直接排序 / 比较 / 过滤。但**美股一个交易日横跨两个北京自然日**,按交易日聚合(分钟线合成日线、日内统计)必须用 `bar.trading_date`。
|
||||||
- **`count` 很大时会自动多连接并发分页**,无需手动翻页。但标准通道 K 线 `count < 100` 部分服务器会返回空——内部已强制 `min_page=100` 规避。
|
- **`count` 很大时会自动多连接并发分页**,无需手动翻页。但标准通道 K 线 `count < 100` 部分服务器会返回空——内部已强制 `min_page=100` 规避。
|
||||||
- **`get_quotes` 一次 >80 只自动切批**;走 MAC 字段位通道(`fields=`)时上限为 80 只/次。
|
- **`get_quotes` 一次 >80 只自动切批**;走 MAC 字段位通道(`fields=`)时上限为 80 只/次。
|
||||||
- **历史逐笔没有笔数字段**,只有当日逐笔的 `trade_count` 有效。
|
- **历史逐笔没有笔数字段**,只有当日逐笔的 `trade_count` 有效。
|
||||||
@@ -183,7 +184,9 @@ await td.get_bars("cffex:IFL0", KlinePeriod.DAY, count=700) # 期货合约
|
|||||||
| 用 `get_block_info` 又自己拆二进制 | 白写解析器 | 用 `get_block_parsed`,已经解析好了 |
|
| 用 `get_block_info` 又自己拆二进制 | 白写解析器 | 用 `get_block_parsed`,已经解析好了 |
|
||||||
| 循环里逐个 `await get_bars` | 慢 | 用列表传多个标的,自动并发 |
|
| 循环里逐个 `await get_bars` | 慢 | 用列表传多个标的,自动并发 |
|
||||||
| 港股/美股五档 bid/ask | 没有(协议不支持) | `fields` 字段位通道有 name 但无五档 |
|
| 港股/美股五档 bid/ask | 没有(协议不支持) | `fields` 字段位通道有 name 但无五档 |
|
||||||
|
| 美股分钟线用 `datetime.date()` 分组做日聚合 | 一个交易日被拆成两天(美股跨北京零点) | 用 `bar.trading_date`;`bar.datetime` 是归一化后的北京时间 |
|
||||||
| 依赖实时性做信号 | 数据是 15 分钟前的 | 换券商实时源 |
|
| 依赖实时性做信号 | 数据是 15 分钟前的 | 换券商实时源 |
|
||||||
|
| 手动改包内 `servers.json` / 写死 IP | 容器只读、升级被覆盖、快节点捞不回来 | 日常靠内置的**当日测速缓存**自动选路;要强制指定就用 `PYPYTDX_DATA_{STANDARD,MAC,EX}_HOSTS` 环境变量 |
|
||||||
|
|
||||||
## 深入参考
|
## 深入参考
|
||||||
|
|
||||||
|
|||||||
+44
-10
@@ -70,7 +70,7 @@ from pytdxdata import (
|
|||||||
| 方法 | 返回 | 说明 |
|
| 方法 | 返回 | 说明 |
|
||||||
|------|------|------|
|
|------|------|------|
|
||||||
| `get_ticks(symbols, *, date=None, start=0, count=None, concurrency=6)` | `list[TransactionRecord]` | 跨市场逐笔成交 |
|
| `get_ticks(symbols, *, date=None, start=0, count=None, concurrency=6)` | `list[TransactionRecord]` | 跨市场逐笔成交 |
|
||||||
| `get_minutes(symbols, *, date=None, sampling=False, concurrency=6)` | `list[MinuteBar]` | 跨市场分时(sampling=True 取缩略采样) |
|
| `get_minutes(symbols, *, date=None, sampling=False, concurrency=6)` | `list[MinuteBar]` | 跨市场分时(sampling=True 取缩略采样:A股由 0x122D 派生,扩展市场 0x254D) |
|
||||||
|
|
||||||
#### 公司信息/文件/板块
|
#### 公司信息/文件/板块
|
||||||
|
|
||||||
@@ -82,6 +82,21 @@ from pytdxdata import (
|
|||||||
| `get_board_members(board, *, fields=None, count=100000)` | `list[SecurityQuote]` | 板块成分股报价(归一模型) |
|
| `get_board_members(board, *, fields=None, count=100000)` | `list[SecurityQuote]` | 板块成分股报价(归一模型) |
|
||||||
| `get_board_of(symbols)` | `list[BelongBoard]` | 个股所属板块 |
|
| `get_board_of(symbols)` | `list[BelongBoard]` | 个股所属板块 |
|
||||||
|
|
||||||
|
#### 指数成分与统计(v0.6.7)
|
||||||
|
|
||||||
|
| 方法 | 返回 | 说明 |
|
||||||
|
|------|------|------|
|
||||||
|
| `get_index_members(index, *, count=100000, fields=None)` | `list[SecurityQuote]` | 指数成分股含实时报价(MAC 0x122C,指数码→板块码映射) |
|
||||||
|
| `get_index_trade_stats(index, *, date=None, members=None, batch_size=50, concurrency=24, skip_errors=True)` | `IndexTradeStats` | 指数逐笔聚合统计(分钟级增量 vol/trade_count/amount;北交所成分无笔数自动剔除) |
|
||||||
|
| `get_index_trade_stats_multi(index, dates, *, concurrency=24, batch_size=50, skip_errors=True, on_progress=None)` | `dict[int, IndexTradeStats]` | 多日批量版(按日期键) |
|
||||||
|
|
||||||
|
#### 价格小数位(v0.6.7)
|
||||||
|
|
||||||
|
| 方法 | 返回 | 说明 |
|
||||||
|
|------|------|------|
|
||||||
|
| `sync_decimal_points(markets=("sh","sz"))` | `dict[str, int]` | 从证券清单同步权威价格小数位表(`start()` 已自动处理,此为手动刷新口) |
|
||||||
|
| `decimal_point_status()` | `dict` | 小数位表当前状态(source/stock/syncing,排障用) |
|
||||||
|
|
||||||
### 2.2 K线(旧接口,共 6 个) {#legacy-kline}
|
### 2.2 K线(旧接口,共 6 个) {#legacy-kline}
|
||||||
|
|
||||||
| 方法 | 返回 | 说明 |
|
| 方法 | 返回 | 说明 |
|
||||||
@@ -109,7 +124,7 @@ from pytdxdata import (
|
|||||||
| `get_transactions(market, code, *, date=None, start=0, count=2000)` | `list[TransactionRecord]` | 逐笔成交(MAC,含 trade_count),支持历史日期 |
|
| `get_transactions(market, code, *, date=None, start=0, count=2000)` | `list[TransactionRecord]` | 逐笔成交(MAC,含 trade_count),支持历史日期 |
|
||||||
| `get_minute(market, code, *, date=None)` | `list[MinuteBar]` | 分时(MAC 0x122D,可历史) |
|
| `get_minute(market, code, *, date=None)` | `list[MinuteBar]` | 分时(MAC 0x122D,可历史) |
|
||||||
| `get_minute_batch(stocks, *, date=None, concurrency=6)` | `dict[str, list[MinuteBar]]` | 批量分时 |
|
| `get_minute_batch(stocks, *, date=None, concurrency=6)` | `dict[str, list[MinuteBar]]` | 批量分时 |
|
||||||
| `get_chart_sampling(market, code)` | `list[float]` | 分时缩略采样(A股,MAC 0x254D) |
|
| `get_chart_sampling(market, code)` | `list[float]` | 分时缩略采样(A股由 MAC **0x122D** 派生;0x254D 仅 EX 通道可用) |
|
||||||
|
|
||||||
### 2.5 证券列表(旧接口,共 3 个) {#legacy-list}
|
### 2.5 证券列表(旧接口,共 3 个) {#legacy-list}
|
||||||
|
|
||||||
@@ -131,13 +146,13 @@ from pytdxdata import (
|
|||||||
| `get_block_info(filename)` | `bytes` | 板块文件**原始字节**(先查元数据再分块下载) |
|
| `get_block_info(filename)` | `bytes` | 板块文件**原始字节**(先查元数据再分块下载) |
|
||||||
| `get_report_file(filename)` | `bytes` | 报告/财务文件**原始字节**(标准 0x06B9) |
|
| `get_report_file(filename)` | `bytes` | 报告/财务文件**原始字节**(标准 0x06B9) |
|
||||||
| `get_financial_file_list()` | `list[str]` | 财务文件列表**原始行**(tdxfin/gpcw.txt) |
|
| `get_financial_file_list()` | `list[str]` | 财务文件列表**原始行**(tdxfin/gpcw.txt) |
|
||||||
| `get_financial_records(filename)` | `bytes` | 财务文件内容(gpcw*.zip 原始字节) |
|
| `get_financial_records(filename)` | `bytes` | 财报 `.dat` 原始字节(索引写 `.zip` 但服务器只有 `.dat` 时自动回退;取不到返回 `b""`) |
|
||||||
| `get_file_meta(filename)` | `FileMeta` | 远程文件元信息(MAC 0x1215) |
|
| `get_file_meta(filename)` | `FileMeta` | 远程文件元信息(MAC 0x1215) |
|
||||||
| `download_file(filename, filesize=0)` | `bytes` | 下载完整远程文件(MAC 0x1217,分块) |
|
| `download_file(filename, filesize=0)` | `bytes` | 下载完整远程文件(MAC 0x1217,分块) |
|
||||||
| **`get_block_parsed(filename)`** | `list[TdxBlock]` | 板块文件**下载+解析**为板块成分(category 按文件名推断:0行业/2概念/3风格) |
|
| **`get_block_parsed(filename)`** | `list[TdxBlock]` | 板块文件**下载+解析**为板块成分(category 按文件名推断:0行业/2概念/3风格) |
|
||||||
| **`get_industry_map()`** | `dict[str, IndustryInfo]` | tdxhy.cfg **下载+解析**:`{代码: 通达信行业+申万行业}` |
|
| **`get_industry_map()`** | `dict[str, IndustryInfo]` | tdxhy.cfg **下载+解析**:`{代码: 通达信行业+申万行业}` |
|
||||||
| **`get_financial_file_infos()`** | `list[FinancialFileInfo]` | gpcw.txt **下载+解析**:文件名/MD5/大小 |
|
| **`get_financial_file_infos()`** | `list[FinancialFileInfo]` | gpcw.txt **下载+解析**:文件名/MD5/大小 |
|
||||||
| **`get_financial_records_parsed(filename)`** | `list[FinancialRecord]` | gpcw*.zip **解压+解析**:每股原始 float 字段(报告期取自文件名,缺失回退头部) |
|
| **`get_financial_records_parsed(filename)`** | `list[FinancialRecord]` | 财报**解析**:每股原始 float 字段。可直接传 `get_financial_file_infos()` 的 `filename`——自动纠正服务器实际文件名(`.zip`/`.dat` 两种形态都支持);报告期取自文件名,缺失回退头部 |
|
||||||
|
|
||||||
> 解析层独立于网络层:解析器位于 `pytdxdata.codec`(纯标准库 `struct`/`zipfile`),可脱网单测;
|
> 解析层独立于网络层:解析器位于 `pytdxdata.codec`(纯标准库 `struct`/`zipfile`),可脱网单测;
|
||||||
> 原 `get_*` 返回 `bytes`/`list[str]` 的方法保持**向后兼容**,解析版为**新增**方法。
|
> 原 `get_*` 返回 `bytes`/`list[str]` 的方法保持**向后兼容**,解析版为**新增**方法。
|
||||||
@@ -232,17 +247,36 @@ from pytdxdata import (
|
|||||||
| `boll(values, period=20, mult=2.0)` | 布林带 |
|
| `boll(values, period=20, mult=2.0)` | 布林带 |
|
||||||
| `compute_indicators(high, low, close, indicators, params=None, tail=None)` | 统一入口(被 `get_stock_kline_with_indicators` 调用) |
|
| `compute_indicators(high, low, close, indicators, params=None, tail=None)` | 统一入口(被 `get_stock_kline_with_indicators` 调用) |
|
||||||
|
|
||||||
**`pytdxdata.config`**(服务器清单)
|
**`pytdxdata.config`**(服务器清单与选路)
|
||||||
|
|
||||||
| 函数 | 说明 |
|
主机来源优先级链(各协议独立):`PYPYTDX_DATA_*_HOSTS` 环境变量 → **当日测速缓存** →
|
||||||
|
包内清单 `fast_pool` → 包内清单 → 代码内置常量。
|
||||||
|
|
||||||
|
| 函数 / 常量 | 说明 |
|
||||||
|------|------|
|
|------|------|
|
||||||
| `get_standard_hosts()` | 标准通道服务器列表 |
|
| `get_standard_hosts()` | 标准通道服务器列表(按上述优先级链解析) |
|
||||||
| `get_mac_hosts()` | MAC 通道服务器列表 |
|
| `get_mac_hosts()` | MAC 通道服务器列表 |
|
||||||
| `get_ex_hosts()` | EX 扩展市场服务器列表 |
|
| `get_ex_hosts()` | EX 扩展市场服务器列表 |
|
||||||
| `get_ex_handshakes()` | EX 握手方式映射 |
|
| `get_ex_handshakes()` | EX 握手方式映射 |
|
||||||
|
| `all_probe_candidates()` | 全量候选主机 `{standard, mac, ex}`(清单 ∪ 常量 ∪ env,**不按状态过滤**) |
|
||||||
|
| `is_valid_host(host)` | 主机名合法性校验(挡 mock 脏数据 / 非法条目) |
|
||||||
|
| `probe_cache_dir()` / `probe_cache_file()` | 用户级当日测速缓存目录 / 文件路径 |
|
||||||
|
| `load_probe_cache(today=None, path=None)` | 读当日测速缓存;缺失 / 损坏 / 非今日 / 版本不符 → `None` |
|
||||||
|
| `probe_disabled()` | 是否已用 `PYPYTDX_DATA_DISABLE_PROBE` 关闭测速 |
|
||||||
|
| `PROBE_CACHE_VERSION` | 测速缓存 payload 版本号 |
|
||||||
|
|
||||||
**`pytdxdata.server_probe`**(服务器探测,可 `python -m pytdxdata.server_probe` 运行)
|
**`pytdxdata.pool.probe_cache`**(当日测速:后台执行、对用户无感)
|
||||||
`load_servers()` / `save_servers(cfg)` / `SERVERS_FILE`
|
|
||||||
|
| 符号 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| `ProbeCache` | 进程内协调器;`ensure_probe()` 为幂等入口(单飞 + 原子写) |
|
||||||
|
| `schedule_probe(hosts=None)` | 连接池用:挂后台 task 并返回该 task;无事件循环时返回 `None` |
|
||||||
|
| `get_probe_cache()` | 取进程级单例 |
|
||||||
|
|
||||||
|
**`pytdxdata.server_probe`**(离线服务器探测,`python -m pytdxdata.server_probe`)
|
||||||
|
`load_servers()` / `save_servers(cfg)` / `SERVERS_FILE` / `probe_one(server, timeout, discover)`
|
||||||
|
CLI 开关:`--discover`(三种协议全测并据实修正 `protocols`,同时剔除脏记录)/
|
||||||
|
`--group standard|mac|ex` / `--refresh`(只重测 `status=ok` 的)/ `--timeout`
|
||||||
|
|
||||||
## 五、CLI(`tdx` 命令,来自 `python -m pytdxdata.cli`)
|
## 五、CLI(`tdx` 命令,来自 `python -m pytdxdata.cli`)
|
||||||
|
|
||||||
@@ -264,7 +298,7 @@ from pytdxdata import (
|
|||||||
| `tdx block [filename] [--limit 30] [--codes]` | 板块文件 下载+解析(block_*.dat → 成分股) |
|
| `tdx block [filename] [--limit 30] [--codes]` | 板块文件 下载+解析(block_*.dat → 成分股) |
|
||||||
| `tdx industry [codes...] [--limit 30] [--search 词]` | 行业分类 下载+解析(tdxhy.cfg) |
|
| `tdx industry [codes...] [--limit 30] [--search 词]` | 行业分类 下载+解析(tdxhy.cfg) |
|
||||||
| `tdx financial-list [--limit 30]` | 历史专业财报文件索引(gpcw.txt) |
|
| `tdx financial-list [--limit 30]` | 历史专业财报文件索引(gpcw.txt) |
|
||||||
| `tdx financial <filename> [--code X] [--limit 5] [--fields 4]` | 历史专业财报记录 下载+解析(gpcw*.zip) |
|
| `tdx financial <filename> [--code X] [--limit 5] [--fields 4]` | 历史专业财报记录 下载+解析(文件名自动纠正,见 `get_financial_records_parsed`) |
|
||||||
| `tdx server-info` | 服务器交易时段 |
|
| `tdx server-info` | 服务器交易时段 |
|
||||||
| `tdx market-stat` | 市场统计 |
|
| `tdx market-stat` | 市场统计 |
|
||||||
|
|
||||||
|
|||||||
+2
-2
@@ -18,8 +18,8 @@ from pytdxdata.models import Adjust, ExMarket, KlinePeriod, Market
|
|||||||
| `Market.BJ` | `2` | 北京(北交所) |
|
| `Market.BJ` | `2` | 北京(北交所) |
|
||||||
|
|
||||||
!!! warning "代码不唯一"
|
!!! warning "代码不唯一"
|
||||||
**市场 + 代码**才唯一。`(SH, "000001")` 是上证指数,`(SZ, "000001")` 是平安银行。
|
**市场 + 代码**才唯一。`sh000001` 是上证指数,`sz000001` 是平安银行。
|
||||||
所有接口的第一个参数都是 `market`,就是为了消歧。
|
统一接口用**字符串标的**把两者一起表达(前缀即市场),避免只写 6 位数字时的歧义。
|
||||||
|
|
||||||
## KlinePeriod — K 线周期
|
## KlinePeriod — K 线周期
|
||||||
|
|
||||||
|
|||||||
@@ -62,6 +62,22 @@ from pytdxdata.models import BoardInfo, MemberQuote, AuctionItem, CapitalFlow
|
|||||||
| 结构 | 具名属性(`zongguben`、`jingzichan` ...) | `fields: list[float]`,原始字段数组 |
|
| 结构 | 具名属性(`zongguben`、`jingzichan` ...) | `fields: list[float]`,原始字段数组 |
|
||||||
| 单位 | 万元 / 万股 | 原始 float,需自行按字段语义换算 |
|
| 单位 | 万元 / 万股 | 原始 float,需自行按字段语义换算 |
|
||||||
|
|
||||||
|
!!! tip "`SecurityBar.datetime` 的时间戳口径(v0.6.2 起)"
|
||||||
|
时间统一为**北京时间**。A股 / 港股 / 期货的原始时间戳即为北京时间;美股的原始报文
|
||||||
|
混用了「美东交易日日期」与「北京时间时分」两个时区,库在构造时已自动归一化
|
||||||
|
(凌晨段日期 +1 天),因此 `bar.datetime` 可直接排序 / 比较 / 过滤。
|
||||||
|
|
||||||
|
但要注意**美股一个交易日横跨两个北京自然日**,按交易日聚合请用 `bar.trading_date`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
bars = await td.get_bars("usAAPL", KlinePeriod.MIN_1, count=390)
|
||||||
|
bars[0].datetime # 2026-10-12 21:31:00(北京时间)
|
||||||
|
bars[-1].datetime # 2026-10-13 04:00:00(次日)
|
||||||
|
bars[0].trading_date # "2026-10-12" —— 归属的美东交易日
|
||||||
|
```
|
||||||
|
|
||||||
|
`bar.is_us` 可判定是否美股。详见 [常见问题 → 美股分钟 K 线](../faq.md)。
|
||||||
|
|
||||||
## 完整字段定义
|
## 完整字段定义
|
||||||
|
|
||||||
以下由 `mkdocstrings` 从源码实时渲染(**以源码为准**):
|
以下由 `mkdocstrings` 从源码实时渲染(**以源码为准**):
|
||||||
|
|||||||
+1
-1
@@ -97,7 +97,7 @@ tdx industry --limit 30 # 只看前 30 条
|
|||||||
|
|
||||||
# 历史专业财报
|
# 历史专业财报
|
||||||
tdx financial-list # 索引:文件名 / MD5 / 大小(gpcw.txt)
|
tdx financial-list # 索引:文件名 / MD5 / 大小(gpcw.txt)
|
||||||
tdx financial gpcw20260331.zip --limit 5 # 下载 + 解压 + 解析
|
tdx financial gpcw20260331.zip --limit 5 # 下载 + 解析(服务器实际给 .dat 时自动回退)
|
||||||
tdx financial gpcw20260331.zip --code 600000 --fields 8 # 只取指定股票、打印更多字段
|
tdx financial gpcw20260331.zip --code 600000 --fields 8 # 只取指定股票、打印更多字段
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
+15
-2
@@ -11,7 +11,7 @@ from pytdxdata.models import Adjust, KlinePeriod
|
|||||||
|
|
||||||
## 全市场日线批量落库
|
## 全市场日线批量落库
|
||||||
|
|
||||||
连接池会把不同股票分配到不同服务器,这是 `*_batch` 存在的意义。
|
传多标的时,连接池会把不同股票分配到不同服务器——这正是它比逐个 `await` 快的原因。
|
||||||
|
|
||||||
```python
|
```python
|
||||||
async def dump_all_daily():
|
async def dump_all_daily():
|
||||||
@@ -118,7 +118,20 @@ async def finance_snapshot(symbols):
|
|||||||
print(fin.symbol, fin)
|
print(fin.symbol, fin)
|
||||||
```
|
```
|
||||||
|
|
||||||
历史专业财报(`gpcw*.zip`,含多报告期)走文件接口:
|
历史专业财报(`gpcw*`,含多报告期)走文件接口。
|
||||||
|
|
||||||
|
!!! warning "索引里的文件名与服务器实际不一致"
|
||||||
|
`tdxfin/gpcw.txt` 索引里写的是 `gpcwYYYYMMDD.zip`,但服务器实际只提供**未压缩的**
|
||||||
|
`tdxfin/gpcwYYYYMMDD.dat` —— 这些差异已由 `get_financial_records_parsed()` 内部处理,
|
||||||
|
直接把索引里的 `filename` 传进去即可。
|
||||||
|
|
||||||
|
另外索引里**含有尚未生成的占位条目**(`filesize` 只有百来字节),取它们会得到空列表。
|
||||||
|
想拿最新一期可用数据,先按体积过滤:
|
||||||
|
|
||||||
|
```python
|
||||||
|
usable = [fi for fi in infos if fi.filesize > 500_000] # 排除占位条目
|
||||||
|
records = await td.get_financial_records_parsed(usable[0].filename)
|
||||||
|
```
|
||||||
|
|
||||||
```python
|
```python
|
||||||
async def history_financials(code: str):
|
async def history_financials(code: str):
|
||||||
|
|||||||
+50
-9
@@ -19,7 +19,8 @@
|
|||||||
| SH | `000001` | 上证指数 |
|
| SH | `000001` | 上证指数 |
|
||||||
| SZ | `000001` | 平安银行 |
|
| SZ | `000001` | 平安银行 |
|
||||||
|
|
||||||
代码不唯一,**市场 + 代码**才唯一。这也是为什么所有接口的第一个参数都是 `market`。
|
代码不唯一,**市场 + 代码**才唯一。统一接口用**字符串标的**把两者一起表达
|
||||||
|
(`sh000001` / `sz000001`),避免只写 6 位数字时的歧义。
|
||||||
|
|
||||||
??? question "为什么报价格式有时候是 2 位小数,有时候是 3 位?"
|
??? question "为什么报价格式有时候是 2 位小数,有时候是 3 位?"
|
||||||
价格小数位按**品种**推断:股票 2 位,指数 / ETF / 可转债 3 位。
|
价格小数位按**品种**推断:股票 2 位,指数 / ETF / 可转债 3 位。
|
||||||
@@ -28,17 +29,40 @@
|
|||||||
## 连接与网络
|
## 连接与网络
|
||||||
|
|
||||||
??? question "连不上服务器 / 全部超时怎么办?"
|
??? question "连不上服务器 / 全部超时怎么办?"
|
||||||
服务器可用性会随时间变化(下线、换 IP、封禁)。按顺序排查:
|
服务器可用性会随时间变化(下线、换 IP、封禁)。本包**当日首次启动会自动后台测速**并把
|
||||||
|
可达主机按延迟排序缓存,多数情况下无需手工干预。按顺序排查:
|
||||||
|
|
||||||
1. 重新探测:`python -m pytdxdata.server_probe`
|
1. **删掉当日测速缓存**,让下次启动重新测速——缓存里可能存着今天刚挂掉的节点
|
||||||
2. 手工指定清单绕过探测结果:
|
(路径见 [安装与配置](getting-started.md#当日自动测速无感))。
|
||||||
```python
|
2. 手工指定主机。优先级最高,**直接跳过测速与缓存**:
|
||||||
async with TdxData(standard_servers=[("1.2.3.4", 7709)]) as td: ...
|
```bash
|
||||||
|
PYPYTDX_DATA_STANDARD_HOSTS=1.2.3.4,5.6.7.8 python your_script.py
|
||||||
```
|
```
|
||||||
3. 检查出口网络策略——标准 / MAC 端口 **7709**、EX 端口 **7727** 需要放通。
|
|
||||||
4. 若在用代理软件,注意有些本地代理会劫持回环 TCP,导致 asyncio 建事件循环都卡住
|
或在代码里传入(**只填主机名**,端口由 `port=` 统一指定):
|
||||||
|
```python
|
||||||
|
async with TdxData(standard_servers=["1.2.3.4"]) as td: ...
|
||||||
|
```
|
||||||
|
3. 离线重生成随包清单(该脚本会真实拉一段 K 线,剔除「握手正常但返回空」的坏节点):
|
||||||
|
```bash
|
||||||
|
python -m pytdxdata.server_probe
|
||||||
|
```
|
||||||
|
4. 检查出口网络策略——标准 / MAC 端口 **7709**、EX 端口 **7727** 需要放通。
|
||||||
|
5. 若在用代理软件,注意有些本地代理会劫持回环 TCP,导致 asyncio 建事件循环都卡住
|
||||||
(见下方「在 Windows 上测试卡死」)。
|
(见下方「在 Windows 上测试卡死」)。
|
||||||
|
|
||||||
|
??? question "启动时会变慢吗?会不会在后台偷偷联网?"
|
||||||
|
会有一次**当日首次启动测速**,但完全在后台,对调用方无感:
|
||||||
|
|
||||||
|
- **不阻塞** `start()`、**不打印**任何东西、失败**静默降级**(绝不抛给调用方)
|
||||||
|
- 结果只缓存到用户级目录(**不写安装目录**),**当天后续启动直接复用、不再重测**
|
||||||
|
- 覆盖**全部候选**主机(含暂时被挤下去的),所以「上次慢、这次快」的节点能回来
|
||||||
|
|
||||||
|
三个开关:`PYPYTDX_DATA_DISABLE_PROBE=1` 关掉测速、
|
||||||
|
`PYPYTDX_DATA_CACHE_DIR` 改缓存位置、
|
||||||
|
`PYPYTDX_DATA_{STANDARD,MAC,EX}_HOSTS` 强制指定主机。
|
||||||
|
详见 [安装与配置 → 当日自动测速](getting-started.md#当日自动测速无感)。
|
||||||
|
|
||||||
??? question "公司网络 / 代理环境下失败"
|
??? question "公司网络 / 代理环境下失败"
|
||||||
通达信协议是裸 TCP,不走 HTTP 代理。如果环境强制走代理,需要把 `7709` / `7727`
|
通达信协议是裸 TCP,不走 HTTP 代理。如果环境强制走代理,需要把 `7709` / `7727`
|
||||||
配成直连(bypass)。云服务器上常见问题是安全组只放通了 80/443。
|
配成直连(bypass)。云服务器上常见问题是安全组只放通了 80/443。
|
||||||
@@ -49,7 +73,7 @@
|
|||||||
| 场景 | 建议 |
|
| 场景 | 建议 |
|
||||||
|---|---|
|
|---|---|
|
||||||
| 偶发查询、CLI | 默认即可 |
|
| 偶发查询、CLI | 默认即可 |
|
||||||
| 批量落库(`*_batch`) | `pool_max=16`、`concurrency=8~16` |
|
| 批量落库(`get_bars` 传多标的) | `pool_max=16`、`concurrency=8~16` |
|
||||||
| Web 服务多并发 | `pool_max=24`,并确保单实例复用 |
|
| Web 服务多并发 | `pool_max=24`,并确保单实例复用 |
|
||||||
|
|
||||||
再大只是排队,不会更快——瓶颈是对端服务器的连接上限。
|
再大只是排队,不会更快——瓶颈是对端服务器的连接上限。
|
||||||
@@ -75,6 +99,23 @@
|
|||||||
2. **是否含未收盘的那根** —— 盘中最后一根是动态的
|
2. **是否含未收盘的那根** —— 盘中最后一根是动态的
|
||||||
3. 分钟线注意**数据来源深度**不同
|
3. 分钟线注意**数据来源深度**不同
|
||||||
|
|
||||||
|
??? question "美股分钟 K 线的时间为什么跨了两个日期?"
|
||||||
|
**已在上游修好(v0.6.2)**。通达信 EX 通道对美股返回的原生时间戳把两个时区拼在一起:
|
||||||
|
日期分量是**美东交易日**、时分分量是**北京时间**。修复前,同一交易日的 390 根会全部
|
||||||
|
落在同一个北京自然日里,并且下午半场被排在上午半场**之前**。
|
||||||
|
|
||||||
|
现在 `SecurityBar.datetime` 已统一为**北京时间**(凌晨段日期自动 +1 天),可直接排序 /
|
||||||
|
过滤 / 做增量水位线。但要注意**美股一个交易日横跨两个北京自然日**:
|
||||||
|
|
||||||
|
```python
|
||||||
|
bars = await td.get_bars("usAAPL", KlinePeriod.MIN_1, count=390)
|
||||||
|
print(bars[0].datetime, bars[-1].datetime) # 2026-10-12 21:31:00 → 2026-10-13 04:00:00
|
||||||
|
print({b.trading_date for b in bars}) # {'2026-10-12'} —— 同一个交易日
|
||||||
|
```
|
||||||
|
|
||||||
|
按**交易日**聚合(分钟线合成日线、日内统计)请用 `b.trading_date`,不要用
|
||||||
|
`b.datetime.date()`。A股 / 港股 / 期货原始即为北京时间,不受影响。
|
||||||
|
|
||||||
??? question "缓存会不会返回过期数据?"
|
??? question "缓存会不会返回过期数据?"
|
||||||
缓存按数据类型分级 TTL,不会返回超出 TTL 的数据:
|
缓存按数据类型分级 TTL,不会返回超出 TTL 的数据:
|
||||||
|
|
||||||
|
|||||||
+45
-2
@@ -96,12 +96,16 @@ asyncio.run(main())
|
|||||||
|
|
||||||
```python
|
```python
|
||||||
async with TdxData(
|
async with TdxData(
|
||||||
standard_servers=[("1.2.3.4", 7709), ("5.6.7.8", 7709)],
|
standard_servers=["1.2.3.4", "5.6.7.8"],
|
||||||
mac_servers=[("9.10.11.12", 7709)],
|
mac_servers=["9.10.11.12"],
|
||||||
|
port=7709,
|
||||||
) as td:
|
) as td:
|
||||||
...
|
...
|
||||||
```
|
```
|
||||||
|
|
||||||
|
主机元素**只填主机名**,端口由构造参数 `port=` 统一指定(不写在主机元素里)。
|
||||||
|
等价的免代码写法是环境变量 `PYPYTDX_DATA_STANDARD_HOSTS` / `…_MAC_HOSTS`(逗号分隔)。
|
||||||
|
|
||||||
## 服务器清单
|
## 服务器清单
|
||||||
|
|
||||||
内置清单随包发布(`src/pytdxdata/servers.json`),由探测脚本生成:
|
内置清单随包发布(`src/pytdxdata/servers.json`),由探测脚本生成:
|
||||||
@@ -121,6 +125,45 @@ from pytdxdata.config import get_standard_hosts, get_mac_hosts, get_ex_hosts
|
|||||||
print(len(get_standard_hosts()), len(get_mac_hosts()), len(get_ex_hosts()))
|
print(len(get_standard_hosts()), len(get_mac_hosts()), len(get_ex_hosts()))
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### 当日自动测速(无感)
|
||||||
|
|
||||||
|
除随包发布的静态清单外,`TdxData` 还在**当日首次启动**时做一次**后台全量测速**:
|
||||||
|
|
||||||
|
- 覆盖**全部候选**(包内清单 ∪ 代码内置常量 ∪ 环境变量覆盖),而不是只测当前池里的少数节点
|
||||||
|
——否则「上次慢、这次快」的节点永远捞不回来
|
||||||
|
- 用**真实 K 线命令**判定可达(标准 / MAC / EX 各发一条业务命令并要求非空),
|
||||||
|
能剔除「握手正常但返回空数据」的坏节点
|
||||||
|
- 结果按**实测延迟升序**写入**用户级缓存**,当天后续启动直接复用、**不再重测**
|
||||||
|
- 全程在后台进行:**不打印、不阻塞 `start()`、失败静默降级**,对调用方完全无感
|
||||||
|
|
||||||
|
缓存位置(可用 `PYPYTDX_DATA_CACHE_DIR` 覆盖):
|
||||||
|
|
||||||
|
| 平台 | 路径 |
|
||||||
|
|---|---|
|
||||||
|
| Windows | `%LOCALAPPDATA%\pytdxdata\hosts-today.json` |
|
||||||
|
| macOS | `~/Library/Caches/pytdxdata/hosts-today.json` |
|
||||||
|
| Linux / 其他 | `$XDG_CACHE_HOME/pytdxdata/hosts-today.json`(默认 `~/.cache/…`) |
|
||||||
|
|
||||||
|
主机选择的优先级(各协议独立):
|
||||||
|
|
||||||
|
1. `PYPYTDX_DATA_STANDARD_HOSTS` / `…_MAC_HOSTS` / `…_EX_HOSTS` 环境变量(手工指定,最高优先)
|
||||||
|
2. **当日测速缓存**中该协议实测可达的主机(按延迟升序)
|
||||||
|
3. 包内清单里 `status=ok` 的 `fast_pool` 主机
|
||||||
|
4. 包内清单里 `status=ok` 的主机
|
||||||
|
5. 代码内置常量
|
||||||
|
|
||||||
|
!!! note "缓存绝不写回安装目录"
|
||||||
|
测速结果只写入上面的**用户级缓存目录**。`site-packages` 在容器 / 系统级安装 / `pipx`
|
||||||
|
下可能只读,多用户共用一个安装时还会互相覆盖,因此包内 `servers.json` 不会被运行时改写。
|
||||||
|
|
||||||
|
环境变量一览:
|
||||||
|
|
||||||
|
| 变量 | 作用 |
|
||||||
|
|---|---|
|
||||||
|
| `PYPYTDX_DATA_STANDARD_HOSTS` / `…_MAC_HOSTS` / `…_EX_HOSTS` | 逗号分隔,手工指定主机(优先级最高,直接跳过测速缓存) |
|
||||||
|
| `PYPYTDX_DATA_CACHE_DIR` | 覆盖当日测速缓存目录 |
|
||||||
|
| `PYPYTDX_DATA_DISABLE_PROBE=1` | 关闭当日测速与缓存(CI / 测试用,保证确定性) |
|
||||||
|
|
||||||
## 生命周期
|
## 生命周期
|
||||||
|
|
||||||
`TdxData` 必须在 `async with` 内使用,或手动 `await td.start()` / `await td.close()`:
|
`TdxData` 必须在 `async with` 内使用,或手动 `await td.start()` / `await td.close()`:
|
||||||
|
|||||||
+3
-2
@@ -34,7 +34,7 @@ asyncio.run(main())
|
|||||||
| 🗄️ **双层 TTL 缓存** | 内存 LRU + 磁盘 SQLite(WAL);TTL 分级:报价 5s / 分钟K线 60s / 日线当天 / 复权K线 1天 / 历史逐笔永久;`single-flight` 同键并发去重;缓存「已解析记录」,重复请求 0 网络 0 解析 |
|
| 🗄️ **双层 TTL 缓存** | 内存 LRU + 磁盘 SQLite(WAL);TTL 分级:报价 5s / 分钟K线 60s / 日线当天 / 复权K线 1天 / 历史逐笔永久;`single-flight` 同键并发去重;缓存「已解析记录」,重复请求 0 网络 0 解析 |
|
||||||
| ⚡ **并发分页** | 把大请求拆成显式 `start` 偏移的页,多连接并行拉取、按序拼接、失败换连接重试 |
|
| ⚡ **并发分页** | 把大请求拆成显式 `start` 偏移的页,多连接并行拉取、按序拼接、失败换连接重试 |
|
||||||
| 🔀 **双通道整合** | 标准通道(经典行情)+ MAC 通道(逐笔 / 复权 / 1分钟K线),按命令自动选路 |
|
| 🔀 **双通道整合** | 标准通道(经典行情)+ MAC 通道(逐笔 / 复权 / 1分钟K线),按命令自动选路 |
|
||||||
| 📄 **服务器文件解析** | 板块 `block_*.dat` / 行业 `tdxhy.cfg` / 历史财报 `gpcw*.zip` **下载即解析**成 dataclass(纯 stdlib),无需自己拆二进制 |
|
| 📄 **服务器文件解析** | 板块 `block_*.dat` / 行业 `tdxhy.cfg` / 历史财报 `gpcw*` **下载即解析**成 dataclass(纯 stdlib),无需自己拆二进制;文件名与索引不一致时自动纠正 |
|
||||||
| 🚀 **批量下载** | `get_bars` / `get_minutes` 传标的列表即多股票并发,连接池自动分配到不同服务器 |
|
| 🚀 **批量下载** | `get_bars` / `get_minutes` 传标的列表即多股票并发,连接池自动分配到不同服务器 |
|
||||||
| 📦 **零重型依赖** | 返回 `list[dataclass]`,pandas / polars 由调用方按需转换 |
|
| 📦 **零重型依赖** | 返回 `list[dataclass]`,pandas / polars 由调用方按需转换 |
|
||||||
|
|
||||||
@@ -104,7 +104,8 @@ asyncio.run(main())
|
|||||||
## 数据来源与限制
|
## 数据来源与限制
|
||||||
|
|
||||||
- 行情来自通达信**公开**行情服务器(无需账号),**15 分钟延时**;需实时数据请使用券商行情
|
- 行情来自通达信**公开**行情服务器(无需账号),**15 分钟延时**;需实时数据请使用券商行情
|
||||||
- 服务器清单由 `python -m pytdxdata.server_probe` 探测生成,可用性随时间变化
|
- 服务器清单由 `python -m pytdxdata.server_probe` 探测生成;运行时**当日首次启动会自动在后台测速并缓存**
|
||||||
|
(对调用方无感,当天不再重测),可用性随时间变化
|
||||||
- 港股 / 美股 / 期货 / 期权走通达信 EX 通道(端口 7727),部分市场(美股期权、港股期权)协议内无数据
|
- 港股 / 美股 / 期货 / 期权走通达信 EX 通道(端口 7727),部分市场(美股期权、港股期权)协议内无数据
|
||||||
|
|
||||||
完整清单见 [常见问题](faq.md)。
|
完整清单见 [常见问题](faq.md)。
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ src/pytdxdata/
|
|||||||
├── api.py # 统一接口 TdxData(对外唯一入口)
|
├── api.py # 统一接口 TdxData(对外唯一入口)
|
||||||
├── symbols.py # 标的解析:字符串 → Symbol(parse_symbol / format_symbol / market_targets)
|
├── symbols.py # 标的解析:字符串 → Symbol(parse_symbol / format_symbol / market_targets)
|
||||||
├── router.py # 标准 / MAC / EX 通道选路
|
├── router.py # 标准 / MAC / EX 通道选路
|
||||||
├── config.py # 服务器清单加载(servers.json)
|
├── config.py # 服务器清单加载 + 选路优先级链 + 用户级测速缓存路径
|
||||||
├── indicator.py # 技术指标(纯函数,无依赖)
|
├── indicator.py # 技术指标(纯函数,无依赖)
|
||||||
├── server_probe.py # 服务器探测,可 python -m 运行
|
├── server_probe.py # 服务器探测,可 python -m 运行
|
||||||
├── protocol/ # 协议层
|
├── protocol/ # 协议层
|
||||||
@@ -39,7 +39,7 @@ src/pytdxdata/
|
|||||||
│ ├── pool.py # 标准 / MAC 动态池
|
│ ├── pool.py # 标准 / MAC 动态池
|
||||||
│ ├── ex_pool.py # EX 池
|
│ ├── ex_pool.py # EX 池
|
||||||
│ ├── health.py # 健康分与选路
|
│ ├── health.py # 健康分与选路
|
||||||
│ └── probe_cache.py # 探测结果缓存
|
│ └── probe_cache.py # 当日全量测速 + 用户级缓存(后台、无感)
|
||||||
├── scheduler/
|
├── scheduler/
|
||||||
│ └── paginator.py # 并发分页调度(3 连接并行)
|
│ └── paginator.py # 并发分页调度(3 连接并行)
|
||||||
├── cache/
|
├── cache/
|
||||||
|
|||||||
@@ -81,11 +81,11 @@ results = await asyncio.gather(*[
|
|||||||
|
|
||||||
EWMA(指数加权移动平均)延迟让选路对「最近变慢」敏感,而不是被历史均值拖住。
|
EWMA(指数加权移动平均)延迟让选路对「最近变慢」敏感,而不是被历史均值拖住。
|
||||||
|
|
||||||
### 为什么 MAC 池更小
|
### MAC 通道的单台连接上限
|
||||||
|
|
||||||
MAC 通道可用服务器只有数台(见 [通道与数据源](routing.md#server-groups)),
|
`mac_pool_cap` 默认 `4`,与标准池的 `std_pool_cap` 一致——这是**对端单台连接限制**,
|
||||||
所以 `mac_pool_cap` 默认只有 `4`——这是**对端限制**,不是本项目的选择。
|
不是本项目的选择。两个通道的可用台数见 [通道与数据源](routing.md#server-groups):
|
||||||
逐笔 / 分时的并发能力天然弱于日线,原因就在这里。
|
实测可达台数会随服务器上下线变化,以**当日测速缓存**为准。
|
||||||
|
|
||||||
## 并发分页
|
## 并发分页
|
||||||
|
|
||||||
@@ -110,4 +110,5 @@ count=2400 → 3 页 → 3 个连接并行 → 按序拼接
|
|||||||
## 下一步
|
## 下一步
|
||||||
|
|
||||||
- [架构总览](architecture.md) — 这些模块在整体里的位置
|
- [架构总览](architecture.md) — 这些模块在整体里的位置
|
||||||
|
- [通道与数据源](routing.md#daily-probe) — 服务器是怎么选出来的(当日测速缓存)
|
||||||
- [常见问题](../faq.md) — 缓存与超时的排查
|
- [常见问题](../faq.md) — 缓存与超时的排查
|
||||||
@@ -29,10 +29,13 @@
|
|||||||
|
|
||||||
| 池 | 数量级 | 用途 |
|
| 池 | 数量级 | 用途 |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **标准池** | 数十台 | 报价 / K线 / 列表 / 财务 |
|
| **标准池** | 9 台 | 报价 / K线 / 列表 / 财务 |
|
||||||
| **MAC 池** | 数台 | 逐笔 / 分时 / 复权 / 板块 / 资金流 |
|
| **MAC 池** | 22 台 | 逐笔 / 分时 / 复权 / 板块 / 资金流 |
|
||||||
| **EX 池** | 4 台 | 港股 / 美股 / 期货 / 期权 |
|
| **EX 池** | 4 台 | 港股 / 美股 / 期货 / 期权 |
|
||||||
|
|
||||||
|
> 上表为 2026-09-21 离线探针(`python -m pytdxdata.server_probe --discover`)实测**可达**台数。
|
||||||
|
> 服务器会下线、换 IP 或变更协议,**实际台数以当日测速缓存为准**(见下方「运行时:当日测速缓存」)。
|
||||||
|
|
||||||
EX 池内部还要再分两类,因为握手方式不同:
|
EX 池内部还要再分两类,因为握手方式不同:
|
||||||
|
|
||||||
| 类型 | 数量 | 握手 |
|
| 类型 | 数量 | 握手 |
|
||||||
@@ -42,7 +45,7 @@ EX 池内部还要再分两类,因为握手方式不同:
|
|||||||
|
|
||||||
这就是 `config.get_ex_handshakes()` 存在的原因——EX 的 4 台机器不是同一种握手。
|
这就是 `config.get_ex_handshakes()` 存在的原因——EX 的 4 台机器不是同一种握手。
|
||||||
|
|
||||||
MAC 池远小于标准池,这也解释了为什么逐笔 / 分时的并发能力天然弱于日线。
|
注意 MAC 池实测**并不比标准池小**(22 台 vs 9 台)。早期版本曾按「MAC 可用节点很少」来解释逐笔 / 分时并发弱于日线,实测已推翻该前提——两者的单台连接上限(`mac_pool_cap` / `std_pool_cap`)默认相同,都是 `4`。
|
||||||
|
|
||||||
## 为什么这些服务器会被「探测」
|
## 为什么这些服务器会被「探测」
|
||||||
|
|
||||||
@@ -51,12 +54,56 @@ MAC 池远小于标准池,这也解释了为什么逐笔 / 分时的并发能
|
|||||||
`server_probe` 的探测逻辑**不是只看 TCP 连通性**,而是真实请求一段 K 线并校验内容:
|
`server_probe` 的探测逻辑**不是只看 TCP 连通性**,而是真实请求一段 K 线并校验内容:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python -m pytdxdata.server_probe
|
python -m pytdxdata.server_probe # 按申报协议全量探测
|
||||||
|
python -m pytdxdata.server_probe --discover # 三种协议全测,并据实修正 protocols
|
||||||
|
python -m pytdxdata.server_probe --group ex # 只测 EX
|
||||||
|
python -m pytdxdata.server_probe --refresh # 只重测当前 status=ok 的
|
||||||
```
|
```
|
||||||
|
|
||||||
普通「只 ping 握手」的探测会把这类坏节点放进来,症状是**偶发的空返回**——
|
普通「只 ping 握手」的探测会把这类坏节点放进来,症状是**偶发的空返回**——
|
||||||
比直接连不上更难查。所以探测必须校验数据。
|
比直接连不上更难查。所以探测必须校验数据。
|
||||||
|
|
||||||
|
写回字段:`status` / `latency` / `checked_at` / `checked_date` / `fast_pool`,
|
||||||
|
以及顶层 `version` / `checked_date` / `note`。其中 `fast_pool` 与运行时缓存里的 `fast`
|
||||||
|
用同一条规则(整体延迟前 10 名,或延迟 < 200ms),用于**还没有当日缓存时**的冷启动排序。
|
||||||
|
|
||||||
|
### 为什么需要 `--discover`
|
||||||
|
|
||||||
|
「申报协议」会过期。实测(2026-09-21)有一批服务器的 **`0x010C` 标准 K 线变成空响应
|
||||||
|
(返回 2 字节空壳),但同端口同主机的 `0x122E` MAC 帧正常**——它们从 `standard` 退化成了
|
||||||
|
`mac`-only。只测申报协议的话,这类退化永远发现不了,标准池会一直含着一批死节点。
|
||||||
|
|
||||||
|
`--discover` 对每台主机把三种协议都测一遍,并据实测结果重写 `protocols`;
|
||||||
|
配套两个必须处理的细节:
|
||||||
|
|
||||||
|
- **失败必须重试一次**再判定不支持。单次抖动会误删协议,而 `protocols` 被收窄后不会自动恢复。
|
||||||
|
- **EX 握手模式**申报可能错,探测失败时换另一种(`login` ⇄ `setup`)再试。
|
||||||
|
- 无任何协议可达时**不动 `protocols`**,只把 `status` 置 `fail`——保留"它申报过什么"的信息。
|
||||||
|
- 顺带剔除 `ip` 非法的脏记录(历史遗留如 `"s1"`)。
|
||||||
|
|
||||||
|
## 运行时:当日测速缓存 {#daily-probe}
|
||||||
|
|
||||||
|
`python -m pytdxdata.server_probe` 是**离线**重生成随包清单的工具;运行时还有一条**在线**的
|
||||||
|
轻量路径——`TdxData.start()` 会异步触发一次全量测速,把结果缓存到**用户级目录**
|
||||||
|
(`…/pytdxdata/hosts-today.json`),当天后续启动直接复用,不再重测。
|
||||||
|
|
||||||
|
设计与几个必须踩对的点:
|
||||||
|
|
||||||
|
| 关注点 | 正确做法 | 反例(会静默出错) |
|
||||||
|
|---|---|---|
|
||||||
|
| 候选集 | 取**全部候选**(清单 ∪ 常量 ∪ env) | 只把「当前池成员」丢给探针 → 上次慢的节点永远捞不回来 |
|
||||||
|
| 可达判定 | 发**真实 K 线命令**并要求非空 | 只握手 / 只读报文头 → 测不出「空响应」坏节点 |
|
||||||
|
| 写盘位置 | 用户级缓存目录 | 写回包内 `servers.json` → 容器只读、多用户互相覆盖 |
|
||||||
|
| 对调用方 | 后台 task + debug 日志 + 绝不抛异常 | 在 `start()` 里 `await` 或打印 → 启动变慢、污染用户输出 |
|
||||||
|
| 并发 | 进程内单例 + `asyncio.Lock` | 三个池各测一遍 → 三倍流量 |
|
||||||
|
| 写入 | 临时文件 + `os.replace` 原子替换 | 直接覆写 → 并发进程读到半截 JSON |
|
||||||
|
|
||||||
|
全部不可达时**不写缓存**——否则「今天全挂了」会被缓存一整天,明天也不重试。
|
||||||
|
|
||||||
|
缓存存在时,`config.get_standard_hosts()` 等按**实测延迟升序**返回该协议的可达主机;
|
||||||
|
缓存缺失 / 跨日 / 版本不符 / 已损坏时,才回落到包内清单。开关见
|
||||||
|
[安装与配置 → 环境变量](../getting-started.md#当日自动测速无感)。
|
||||||
|
|
||||||
## EX 通道细节
|
## EX 通道细节
|
||||||
|
|
||||||
- 端口固定 **7727**,与标准 / MAC 的 7709 不同
|
- 端口固定 **7727**,与标准 / MAC 的 7709 不同
|
||||||
|
|||||||
@@ -59,6 +59,19 @@ asyncio.run(main())
|
|||||||
> 旧的分通道/市场方法(`get_kline` / `get_transactions` / `get_goods_*` 等共 33 个)保留为
|
> 旧的分通道/市场方法(`get_kline` / `get_transactions` / `get_goods_*` 等共 33 个)保留为
|
||||||
> deprecated 别名,1.0 移除;完整对照见[接口重构审查](https://openbot-coder.github.io/pytdxdata/refactor-audit/)与 CHANGELOG。
|
> deprecated 别名,1.0 移除;完整对照见[接口重构审查](https://openbot-coder.github.io/pytdxdata/refactor-audit/)与 CHANGELOG。
|
||||||
|
|
||||||
|
## 服务器与配置
|
||||||
|
|
||||||
|
服务器主机来源有一条优先级链(各协议独立):环境变量 → **当日测速缓存** → 包内清单
|
||||||
|
`fast_pool` → 包内清单 → 代码内置常量。
|
||||||
|
|
||||||
|
- `TdxData.start()` 会在**当日首次启动**后台触发一次全量测速,用**真实 K 线命令**判定可达
|
||||||
|
(不是只握手),结果按延迟升序写入**用户级缓存**(Windows `%LOCALAPPDATA%\pytdxdata`、
|
||||||
|
macOS `~/Library/Caches/pytdxdata`、Linux `$XDG_CACHE_HOME/pytdxdata`),当天复用、不再重测。
|
||||||
|
- 该操作对调用方**完全无感**:后台执行、不打印、不阻塞 `start()`、失败静默降级。
|
||||||
|
运行时**绝不改写包内 `servers.json`**。
|
||||||
|
- 环境变量:`PYPYTDX_DATA_STANDARD_HOSTS` / `…_MAC_HOSTS` / `…_EX_HOSTS`(逗号分隔,手工指定,最高优先)、
|
||||||
|
`PYPYTDX_DATA_CACHE_DIR`(覆盖缓存目录)、`PYPYTDX_DATA_DISABLE_PROBE=1`(关闭测速)。
|
||||||
|
|
||||||
## CLI
|
## CLI
|
||||||
|
|
||||||
安装后提供 `tdx` 命令,19 个子命令,不写代码即可取数:
|
安装后提供 `tdx` 命令,19 个子命令,不写代码即可取数:
|
||||||
@@ -80,6 +93,10 @@ tdx financial gpcw20260331.zip --limit 5
|
|||||||
- **`KlinePeriod.MIN_1` 走 MAC 通道并单连接串行分页**,这是刻意设计:并发分页在多服务器
|
- **`KlinePeriod.MIN_1` 走 MAC 通道并单连接串行分页**,这是刻意设计:并发分页在多服务器
|
||||||
历史深度不一致时会产生非确定性数据空洞。不要试图"优化"成并发。
|
历史深度不一致时会产生非确定性数据空洞。不要试图"优化"成并发。
|
||||||
- **复权 K 线走 MAC 通道**,标准通道不提供复权数据。
|
- **复权 K 线走 MAC 通道**,标准通道不提供复权数据。
|
||||||
|
- **K 线 `datetime` 统一为北京时间**(v0.6.2 起)。美股原始报文的日期分量是美东交易日、
|
||||||
|
时分分量是北京时间,库已自动归一化,`bar.datetime` 可直接排序 / 比较 / 过滤。
|
||||||
|
但**美股一个交易日横跨两个北京自然日**:按交易日聚合(分钟线合成日线、日内统计)
|
||||||
|
用 `bar.trading_date`,不要用 `bar.datetime.date()`。`bar.is_us` 判定美股。
|
||||||
- **标准通道部分服务器在 `count < 100` 时返回空**,分页器已强制 `min_page=100` 规避。
|
- **标准通道部分服务器在 `count < 100` 时返回空**,分页器已强制 `min_page=100` 规避。
|
||||||
- **本地 Windows 上 `asyncio.new_event_loop()` 可能永久挂起**,根因是回环 `socket.socketpair()`
|
- **本地 Windows 上 `asyncio.new_event_loop()` 可能永久挂起**,根因是回环 `socket.socketpair()`
|
||||||
被本地代理/安全软件拦截。异步测试请在 Linux/WSL 运行。
|
被本地代理/安全软件拦截。异步测试请在 Linux/WSL 运行。
|
||||||
|
|||||||
@@ -142,7 +142,7 @@
|
|||||||
|
|
||||||
## 七、验证
|
## 七、验证
|
||||||
|
|
||||||
- `pytest`:**127 passed**。其中 `tests/test_all_interfaces.py` 为全接口回归:mock 连接逐个
|
- `pytest`:**127 passed**(审查当时的快照;当前基线为 164 passed)。其中 `tests/test_all_interfaces.py` 为全接口回归:mock 连接逐个
|
||||||
调用 63 个公开方法,并以「覆盖哨兵」断言无遗漏(7 条告警为别名测试的预期 `DeprecationWarning`)。
|
调用 63 个公开方法,并以「覆盖哨兵」断言无遗漏(7 条告警为别名测试的预期 `DeprecationWarning`)。
|
||||||
- `scripts/check_docs.py`:`MUST_LIST_ALL` + `MUST_NOT_BE_STALE` 全部 `[OK]`。
|
- `scripts/check_docs.py`:`MUST_LIST_ALL` + `MUST_NOT_BE_STALE` 全部 `[OK]`。
|
||||||
- 签名核对:统一接口签名与返回类型与本文一致;全仓无 `_as_symbols` 及元组式调用残留。
|
- 签名核对:统一接口签名与返回类型与本文一致;全仓无 `_as_symbols` 及元组式调用残留。
|
||||||
+23
-6
@@ -1,14 +1,24 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "pytdxdata"
|
name = "pytdxdata"
|
||||||
version = "0.6.0"
|
version = "0.6.7"
|
||||||
description = "通达信(TongDaXin)行情数据客户端:动态连接池 + 双层TTL缓存 + 3连接并发分页,纯asyncio零运行时依赖,支持A股/港股/美股/期货/期权"
|
description = "通达信(TongDaXin)行情数据客户端:动态连接池 + 双层TTL缓存 + 3连接并发分页,纯asyncio零运行时依赖,支持A股/港股/美股/期货/期权"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
license = "MIT"
|
license = "MIT"
|
||||||
license-files = ["LICENSE"]
|
license-files = ["LICENSE"]
|
||||||
keywords = ["tdx", "trading", "stock", "kline", "quote", "通达信", "A股", "港股", "美股", "期货", "asyncio", "quant"]
|
keywords = [
|
||||||
authors = [
|
"tdx",
|
||||||
{ name = "openbot-coder", email = "openbot-coder@users.noreply.github.com" }
|
"trading",
|
||||||
|
"stock",
|
||||||
|
"kline",
|
||||||
|
"quote",
|
||||||
|
"通达信",
|
||||||
|
"A股",
|
||||||
|
"港股",
|
||||||
|
"美股",
|
||||||
|
"期货",
|
||||||
|
"asyncio",
|
||||||
|
"quant",
|
||||||
]
|
]
|
||||||
classifiers = [
|
classifiers = [
|
||||||
"Development Status :: 4 - Beta",
|
"Development Status :: 4 - Beta",
|
||||||
@@ -24,6 +34,10 @@ classifiers = [
|
|||||||
]
|
]
|
||||||
dependencies = []
|
dependencies = []
|
||||||
|
|
||||||
|
[[project.authors]]
|
||||||
|
name = "openbot-coder"
|
||||||
|
email = "openbot-coder@users.noreply.github.com"
|
||||||
|
|
||||||
[project.urls]
|
[project.urls]
|
||||||
Homepage = "https://github.com/openbot-coder/pytdxdata"
|
Homepage = "https://github.com/openbot-coder/pytdxdata"
|
||||||
Repository = "https://github.com/openbot-coder/pytdxdata"
|
Repository = "https://github.com/openbot-coder/pytdxdata"
|
||||||
@@ -31,7 +45,11 @@ Documentation = "https://openbot-coder.github.io/pytdxdata/"
|
|||||||
|
|
||||||
[project.optional-dependencies]
|
[project.optional-dependencies]
|
||||||
uvloop = ["uvloop>=0.21"]
|
uvloop = ["uvloop>=0.21"]
|
||||||
dev = ["pytest>=8.0", "pytest-asyncio>=0.24", "ruff>=0.6"]
|
dev = [
|
||||||
|
"pytest>=8.0",
|
||||||
|
"pytest-asyncio>=0.24",
|
||||||
|
"ruff>=0.6",
|
||||||
|
]
|
||||||
|
|
||||||
[project.scripts]
|
[project.scripts]
|
||||||
tdx = "pytdxdata.cli:main"
|
tdx = "pytdxdata.cli:main"
|
||||||
@@ -59,4 +77,3 @@ docs = [
|
|||||||
"ruff>=0.6",
|
"ruff>=0.6",
|
||||||
"jieba>=0.42",
|
"jieba>=0.42",
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -7,16 +7,18 @@ from .models import (
|
|||||||
Adjust,
|
Adjust,
|
||||||
ExMarket,
|
ExMarket,
|
||||||
FinanceRecord,
|
FinanceRecord,
|
||||||
|
IndexTradeStats,
|
||||||
KlinePeriod,
|
KlinePeriod,
|
||||||
Market,
|
Market,
|
||||||
MinuteBar,
|
MinuteBar,
|
||||||
|
MinuteTradeStat,
|
||||||
SecurityBar,
|
SecurityBar,
|
||||||
SecurityQuote,
|
SecurityQuote,
|
||||||
TransactionRecord,
|
TransactionRecord,
|
||||||
XdxrRecord,
|
XdxrRecord,
|
||||||
)
|
)
|
||||||
|
|
||||||
__version__ = "0.6.0"
|
__version__ = "0.6.7"
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"TdxData",
|
"TdxData",
|
||||||
"Market",
|
"Market",
|
||||||
@@ -27,6 +29,8 @@ __all__ = [
|
|||||||
"SecurityQuote",
|
"SecurityQuote",
|
||||||
"TransactionRecord",
|
"TransactionRecord",
|
||||||
"MinuteBar",
|
"MinuteBar",
|
||||||
|
"MinuteTradeStat",
|
||||||
|
"IndexTradeStats",
|
||||||
"XdxrRecord",
|
"XdxrRecord",
|
||||||
"FinanceRecord",
|
"FinanceRecord",
|
||||||
"__version__",
|
"__version__",
|
||||||
|
|||||||
+592
-64
@@ -22,11 +22,16 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
|
import re
|
||||||
|
import time
|
||||||
import warnings
|
import warnings
|
||||||
|
from collections import deque
|
||||||
|
from collections.abc import Callable
|
||||||
from datetime import date as _date, datetime
|
from datetime import date as _date, datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Sequence
|
from typing import Sequence
|
||||||
|
|
||||||
|
from . import decimal_sync
|
||||||
from .cache.disk import DiskCache
|
from .cache.disk import DiskCache
|
||||||
from .cache.key import CacheKey
|
from .cache.key import CacheKey
|
||||||
from .cache.manager import CacheManager
|
from .cache.manager import CacheManager
|
||||||
@@ -44,11 +49,13 @@ from .models import (
|
|||||||
FinancialFileInfo,
|
FinancialFileInfo,
|
||||||
FinancialRecord,
|
FinancialRecord,
|
||||||
IndicatorSet,
|
IndicatorSet,
|
||||||
|
IndexTradeStats,
|
||||||
IndustryInfo,
|
IndustryInfo,
|
||||||
KlinePeriod,
|
KlinePeriod,
|
||||||
Market,
|
Market,
|
||||||
MemberQuote,
|
MemberQuote,
|
||||||
MinuteBar,
|
MinuteBar,
|
||||||
|
MinuteTradeStat,
|
||||||
SecurityBar,
|
SecurityBar,
|
||||||
SecurityInfo,
|
SecurityInfo,
|
||||||
SecurityQuote,
|
SecurityQuote,
|
||||||
@@ -61,7 +68,7 @@ from .models import (
|
|||||||
)
|
)
|
||||||
from .pool.ex_pool import ExPool
|
from .pool.ex_pool import ExPool
|
||||||
from .pool.health import HealthEngine
|
from .pool.health import HealthEngine
|
||||||
from .pool.pool import DynamicPool
|
from .pool.pool import DynamicPool, PoolTimeoutError
|
||||||
from .protocol.commands import (
|
from .protocol.commands import (
|
||||||
GetFinanceInfoCmd,
|
GetFinanceInfoCmd,
|
||||||
GetIndexBarsCmd,
|
GetIndexBarsCmd,
|
||||||
@@ -75,6 +82,7 @@ from .protocol.commands import (
|
|||||||
MacKlineCmd,
|
MacKlineCmd,
|
||||||
MacTransactionCmd,
|
MacTransactionCmd,
|
||||||
MAX_QUOTES,
|
MAX_QUOTES,
|
||||||
|
TdxDecodeError,
|
||||||
)
|
)
|
||||||
from .protocol.commands.ex_ext import (
|
from .protocol.commands.ex_ext import (
|
||||||
GetExHistoryTransactionDataCmd,
|
GetExHistoryTransactionDataCmd,
|
||||||
@@ -140,10 +148,13 @@ _KLINE_MAX_BARS: dict[KlinePeriod, int] = {
|
|||||||
# 自定义字段报价(0x122B/0x122C)基础字段位:昨收/开/高/低/现价/量/量比/额
|
# 自定义字段报价(0x122B/0x122C)基础字段位:昨收/开/高/低/现价/量/量比/额
|
||||||
_QUOTE_BITS = [0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07]
|
_QUOTE_BITS = [0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07]
|
||||||
_QUOTE_PAGE = 80 # 0x122B/0x122C 单次上限 80 只
|
_QUOTE_PAGE = 80 # 0x122B/0x122C 单次上限 80 只
|
||||||
|
_QUOTE_PIPE_CHUNK = 3 # 单连接流水线一次取几片(实测 3 片 + 每台 2 连接最稳)
|
||||||
_EX_PAGE = 1000 # EX 商品列表单页
|
_EX_PAGE = 1000 # EX 商品列表单页
|
||||||
_TICK_PAGE = 1000 # MAC 逐笔单页
|
_TICK_PAGE = 1000 # MAC 逐笔单页
|
||||||
_EX_TICK_PAGE = 1800 # 港股逐笔单页
|
_EX_TICK_PAGE = 1800 # 港股逐笔单页
|
||||||
_ALL_PAGES_CAP = 50 # count=None(全量)时的翻页上限
|
_ALL_PAGES_CAP = 50 # count=None(全量)时的翻页上限
|
||||||
|
# 集合竞价**委托**记录标记(0x122F 第 5 字段)。不是成交,见 _ticks_one 注释。
|
||||||
|
_BS_AUCTION_ORDER = 8
|
||||||
|
|
||||||
# 港股系市场:逐笔走 EX 专用 16B 协议(0x23FC/0x2406),其余扩展市场走 MAC 0x122F
|
# 港股系市场:逐笔走 EX 专用 16B 协议(0x23FC/0x2406),其余扩展市场走 MAC 0x122F
|
||||||
_HK_MARKETS = frozenset({27, 31, 48, 49, 71, 98})
|
_HK_MARKETS = frozenset({27, 31, 48, 49, 71, 98})
|
||||||
@@ -228,14 +239,16 @@ class TdxData:
|
|||||||
default_adjust: Adjust = Adjust.NONE,
|
default_adjust: Adjust = Adjust.NONE,
|
||||||
max_inflight: int = 64,
|
max_inflight: int = 64,
|
||||||
pool_min: int = 3,
|
pool_min: int = 3,
|
||||||
pool_max: int = 12,
|
pool_max: int = 24,
|
||||||
mac_pool_cap: int = 4,
|
mac_pool_cap: int = 4,
|
||||||
std_pool_cap: int = 4,
|
std_pool_cap: int = 4,
|
||||||
|
ex_pool_cap: int = 2,
|
||||||
) -> None:
|
) -> None:
|
||||||
self.port = port
|
self.port = port
|
||||||
self.timeout = timeout
|
self.timeout = timeout
|
||||||
self.default_adjust = default_adjust
|
self.default_adjust = default_adjust
|
||||||
self.mac_pool_cap = mac_pool_cap
|
self.mac_pool_cap = mac_pool_cap
|
||||||
|
self.ex_pool_cap = ex_pool_cap
|
||||||
self._std_servers = list(standard_servers or get_standard_hosts())
|
self._std_servers = list(standard_servers or get_standard_hosts())
|
||||||
self._mac_servers = list(mac_servers or get_mac_hosts())
|
self._mac_servers = list(mac_servers or get_mac_hosts())
|
||||||
self._ex_servers = list(ex_servers or get_ex_hosts())
|
self._ex_servers = list(ex_servers or get_ex_hosts())
|
||||||
@@ -260,8 +273,11 @@ class TdxData:
|
|||||||
probe_factory=lambda: MacKlineCmd(1, "600000", KlinePeriod.DAY, Adjust.NONE, 0, 100),
|
probe_factory=lambda: MacKlineCmd(1, "600000", KlinePeriod.DAY, Adjust.NONE, 0, 100),
|
||||||
)
|
)
|
||||||
# EX扩展市场池(港股/期货/美股,端口7727;MAC EX login + pytdx EX setup 混合)
|
# EX扩展市场池(港股/期货/美股,端口7727;MAC EX login + pytdx EX setup 混合)
|
||||||
|
# 单服务器场景(测速缓存收缩后仅 1 台)下,有效并发 = 服务器数 × ex_pool_cap,
|
||||||
|
# 默认 cap=2 保证至少 2 并发;acquire 有 Condition 背压等待,不再快速失败。
|
||||||
self._ex_pool = ExPool(self._ex_servers, timeout=timeout,
|
self._ex_pool = ExPool(self._ex_servers, timeout=timeout,
|
||||||
min_size=min(pool_min, 2), max_size=min(pool_max, 6),
|
min_size=min(pool_min, 2), max_size=min(pool_max, 6),
|
||||||
|
per_server_cap=ex_pool_cap,
|
||||||
handshake_map=EX_HANDSHAKES)
|
handshake_map=EX_HANDSHAKES)
|
||||||
disk = None
|
disk = None
|
||||||
if self._cache_dir is not None:
|
if self._cache_dir is not None:
|
||||||
@@ -272,6 +288,7 @@ class TdxData:
|
|||||||
self._paginator_std = Paginator(self._std_pool)
|
self._paginator_std = Paginator(self._std_pool)
|
||||||
self._paginator_mac = Paginator(self._mac_pool)
|
self._paginator_mac = Paginator(self._mac_pool)
|
||||||
self._started = False
|
self._started = False
|
||||||
|
self._decimal_task: asyncio.Task | None = None
|
||||||
|
|
||||||
async def start(self) -> None:
|
async def start(self) -> None:
|
||||||
await self._std_pool.start()
|
await self._std_pool.start()
|
||||||
@@ -279,8 +296,13 @@ class TdxData:
|
|||||||
await self._ex_pool.start()
|
await self._ex_pool.start()
|
||||||
await self._cache.start()
|
await self._cache.start()
|
||||||
self._started = True
|
self._started = True
|
||||||
|
# 权威价格小数位表:当日缓存命中则同步装表(无网络),否则挂后台同步。
|
||||||
|
# 不阻塞 start(),失败静默退回静态前缀规则。
|
||||||
|
self._decimal_task = decimal_sync.schedule(self)
|
||||||
|
|
||||||
async def close(self) -> None:
|
async def close(self) -> None:
|
||||||
|
await decimal_sync.cancel(self._decimal_task)
|
||||||
|
self._decimal_task = None
|
||||||
await self._std_pool.close()
|
await self._std_pool.close()
|
||||||
await self._mac_pool.close()
|
await self._mac_pool.close()
|
||||||
await self._ex_pool.close()
|
await self._ex_pool.close()
|
||||||
@@ -318,6 +340,39 @@ class TdxData:
|
|||||||
out.extend(await self._universe_cn(mkt))
|
out.extend(await self._universe_cn(mkt))
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
async def sync_decimal_points(self, markets: Sequence[str] = ("sh", "sz")
|
||||||
|
) -> dict[str, int]:
|
||||||
|
"""从证券清单同步**权威价格小数位表**,返回统计。
|
||||||
|
|
||||||
|
报价接口返回的是整数,真实价格 = ``整数 / 10**小数位``。小数位不随报价下发,
|
||||||
|
但证券清单接口自带权威值 ``SecurityInfo.decimal_point`` —— 本方法把它灌进
|
||||||
|
:func:`~pytdxdata.protocol.commands.quotes.price_decimal_digits` 使用的运行时表,
|
||||||
|
从此价格标度不再依赖硬编码前缀规则(新增代码段也不会算错)。
|
||||||
|
|
||||||
|
``start()`` 已自动处理:当日缓存命中就直接装表,否则挂后台同步,**不用手动调**。
|
||||||
|
这个方法用于「立刻要权威表」「排障时强制刷新」。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
markets: 拉哪些市场,默认 ``("sh", "sz")``。证券清单走 ``get_universe``,
|
||||||
|
其本身已有缓存层,重复调用只花一次网络。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
``{"prefixes", "codes", "mixed_prefixes", "stock"}`` —— 已覆盖的三位前缀数、
|
||||||
|
例外代码数、出现混合小数位的前缀数、总条目数。
|
||||||
|
``mixed_prefixes`` > 0 说明某段代码混用了不同小数位(已自动退化为逐代码查表)。
|
||||||
|
|
||||||
|
用法::
|
||||||
|
|
||||||
|
async with TdxData() as td:
|
||||||
|
print(await td.sync_decimal_points()) # {'prefixes': 300+, 'codes': 0, ...}
|
||||||
|
q = await td.get_quotes("sh019742") # 110.228(4 位小数)
|
||||||
|
"""
|
||||||
|
return await decimal_sync.refresh(self, tuple(markets))
|
||||||
|
|
||||||
|
async def decimal_point_status(self) -> dict:
|
||||||
|
"""权威小数位表当前状态(``source`` / ``stock`` / ``syncing`` 等,排障用)。"""
|
||||||
|
return decimal_sync.status()
|
||||||
|
|
||||||
async def _universe_cn(self, market: int) -> list[SecurityInfo]:
|
async def _universe_cn(self, market: int) -> list[SecurityInfo]:
|
||||||
key = CacheKey("list", market)
|
key = CacheKey("list", market)
|
||||||
|
|
||||||
@@ -389,7 +444,9 @@ class TdxData:
|
|||||||
# ================================================================== #
|
# ================================================================== #
|
||||||
|
|
||||||
async def get_quotes(self, symbols: str | Sequence[str], *,
|
async def get_quotes(self, symbols: str | Sequence[str], *,
|
||||||
fields: Sequence[int] | None = None
|
fields: Sequence[int] | None = None,
|
||||||
|
concurrency: int | None = None,
|
||||||
|
pipeline: bool = True
|
||||||
) -> list[SecurityQuote]:
|
) -> list[SecurityQuote]:
|
||||||
"""报价,一次可跨市场、跨通道混合查询。
|
"""报价,一次可跨市场、跨通道混合查询。
|
||||||
|
|
||||||
@@ -399,6 +456,11 @@ class TdxData:
|
|||||||
fields: 可选自定义字段位(如 ``[0x00, 0x04, 0x38]``)。给了就走 MAC/EX 的
|
fields: 可选自定义字段位(如 ``[0x00, 0x04, 0x38]``)。给了就走 MAC/EX 的
|
||||||
0x122B 字段位通道(有 ``name``,无五档);不给时 A股走标准通道
|
0x122B 字段位通道(有 ``name``,无五档);不给时 A股走标准通道
|
||||||
(**有五档 bid/ask、买卖量、涨跌停价,无 name**),扩展市场走 0x122B。
|
(**有五档 bid/ask、买卖量、涨跌停价,无 name**),扩展市场走 0x122B。
|
||||||
|
concurrency: 连接并发数(默认取连接池上限)。协议硬上限 80 只/请求,
|
||||||
|
全市场约 5,200 只 A股需切 66 片;调小可降低对服务器的压力。
|
||||||
|
pipeline: 是否启用**单连接流水线**(默认开启)。开启后一条连接可一次发出
|
||||||
|
多个分片、只付一次往返,全市场 5,200 只 A股由 ~4s 降到 **~0.3s**;
|
||||||
|
个股或小批量不受影响。个别服务器不支持流水线时会自动降级为逐片。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
``list[SecurityQuote]``,按入参顺序排列(各记录带 ``symbol``)。
|
``list[SecurityQuote]``,按入参顺序排列(各记录带 ``symbol``)。
|
||||||
@@ -409,50 +471,155 @@ class TdxData:
|
|||||||
bits = [int(b) for b in fields] if fields else None
|
bits = [int(b) for b in fields] if fields else None
|
||||||
tag = "mac:" + _fields_tag(bits) if bits else "std"
|
tag = "mac:" + _fields_tag(bits) if bits else "std"
|
||||||
key = CacheKey.quote_key([s.as_pair() for s in syms], tag=tag)
|
key = CacheKey.quote_key([s.as_pair() for s in syms], tag=tag)
|
||||||
payload = await self._cache.get_or_fetch(
|
quotes = await self._cache.get_or_fetch_obj(
|
||||||
key, lambda: self._fetch_quotes(syms, bits))
|
key,
|
||||||
quotes = unpack_payload(payload)
|
lambda: self._fetch_quotes(syms, bits, concurrency, pipeline),
|
||||||
|
pack=pack_payload, unpack=unpack_payload)
|
||||||
order = {s.as_pair(): i for i, s in enumerate(syms)}
|
order = {s.as_pair(): i for i, s in enumerate(syms)}
|
||||||
quotes.sort(key=lambda q: order.get((int(q.market), q.code), len(order)))
|
quotes.sort(key=lambda q: order.get((int(q.market), q.code), len(order)))
|
||||||
return quotes
|
return quotes
|
||||||
|
|
||||||
async def _fetch_quotes(self, syms: list[Symbol], bits: list[int] | None) -> bytes:
|
async def _fetch_quotes(self, syms: list[Symbol], bits: list[int] | None,
|
||||||
|
concurrency: int | None = None,
|
||||||
|
pipeline: bool = True) -> list[SecurityQuote]:
|
||||||
out: list[SecurityQuote] = []
|
out: list[SecurityQuote] = []
|
||||||
cn = [s for s in syms if not s.is_ex]
|
cn = [s for s in syms if not s.is_ex]
|
||||||
ex = [s for s in syms if s.is_ex]
|
ex = [s for s in syms if s.is_ex]
|
||||||
if bits is None:
|
if bits is None:
|
||||||
if cn:
|
if cn:
|
||||||
out.extend(await self._quotes_std(cn))
|
out.extend(await self._quotes_std(
|
||||||
|
cn, concurrency=concurrency, pipeline=pipeline))
|
||||||
if ex:
|
if ex:
|
||||||
rows = await self._execute_ex(
|
rows = await self._execute_ex(
|
||||||
MacSymbolQuotesCmd([s.as_pair() for s in ex], None))
|
MacSymbolQuotesCmd([s.as_pair() for s in ex], None))
|
||||||
out.extend(_quote_from_member(m) for m in rows)
|
out.extend(_quote_from_member(m) for m in rows)
|
||||||
else:
|
else:
|
||||||
for i in range(0, len(syms), _QUOTE_PAGE):
|
batches = [syms[i:i + _QUOTE_PAGE]
|
||||||
batch = syms[i:i + _QUOTE_PAGE]
|
for i in range(0, len(syms), _QUOTE_PAGE)]
|
||||||
|
sem = asyncio.Semaphore(self._quote_concurrency(concurrency))
|
||||||
|
|
||||||
|
async def one(batch: list[Symbol]) -> list[SecurityQuote]:
|
||||||
|
async with sem:
|
||||||
pairs = [s.as_pair() for s in batch]
|
pairs = [s.as_pair() for s in batch]
|
||||||
cn_pairs = [p for p, s in zip(pairs, batch) if not s.is_ex]
|
cn_pairs = [p for p, s in zip(pairs, batch) if not s.is_ex]
|
||||||
ex_pairs = [p for p, s in zip(pairs, batch) if s.is_ex]
|
ex_pairs = [p for p, s in zip(pairs, batch) if s.is_ex]
|
||||||
|
got: list[SecurityQuote] = []
|
||||||
if cn_pairs:
|
if cn_pairs:
|
||||||
rows = await self._execute_mac(MacSymbolQuotesCmd(cn_pairs, bits))
|
rows = await self._execute_mac(
|
||||||
out.extend(_quote_from_member(m) for m in rows)
|
MacSymbolQuotesCmd(cn_pairs, bits))
|
||||||
|
got.extend(_quote_from_member(m) for m in rows)
|
||||||
if ex_pairs:
|
if ex_pairs:
|
||||||
rows = await self._execute_ex(MacSymbolQuotesCmd(ex_pairs, bits))
|
rows = await self._execute_ex(
|
||||||
out.extend(_quote_from_member(m) for m in rows)
|
MacSymbolQuotesCmd(ex_pairs, bits))
|
||||||
return pack_payload(out)
|
got.extend(_quote_from_member(m) for m in rows)
|
||||||
|
return got
|
||||||
|
|
||||||
async def _quotes_std(self, syms: list[Symbol]) -> list[SecurityQuote]:
|
if batches:
|
||||||
"""A股标准通道五档报价(>80 只自动切批)。"""
|
for got in await asyncio.gather(*(one(b) for b in batches)):
|
||||||
out: list[SecurityQuote] = []
|
out.extend(got)
|
||||||
for i in range(0, len(syms), MAX_QUOTES):
|
return out
|
||||||
batch = syms[i:i + MAX_QUOTES]
|
|
||||||
cmd = GetSecurityQuotesCmd([s.as_pair() for s in batch])
|
def _quote_concurrency(self, requested: int | None) -> int:
|
||||||
rows = await self._execute_standard(cmd)
|
"""连接并发数:显式指定优先,否则按「每台服务器 2 连接」估算。
|
||||||
|
|
||||||
|
实测(5,227 只 A股 / 66 片,11 台服务器):
|
||||||
|
|
||||||
|
========== ===== ====== ====== ======
|
||||||
|
并发 每台连接 流水深度 p50 max
|
||||||
|
========== ===== ====== ====== ======
|
||||||
|
11 1 6 455ms 464ms
|
||||||
|
12 1~2 3 457ms 523ms
|
||||||
|
**22** **2** **3** **325ms** **334ms**
|
||||||
|
33 3 2 328ms 345ms
|
||||||
|
========== ===== ====== ====== ======
|
||||||
|
|
||||||
|
结论:每台 2 连接 × 流水 3 片最稳;再多连接只是把同一台的队列搬个位置,
|
||||||
|
对整体无益反增抖动。池 ``per_server_cap`` 默认 4,2 连接仍留有余量。
|
||||||
|
"""
|
||||||
|
if requested is not None and requested > 0:
|
||||||
|
return max(1, int(requested))
|
||||||
|
n = len(self._std_pool.servers)
|
||||||
|
return max(1, min(self._std_pool.max_size, (n or 1) * 2))
|
||||||
|
|
||||||
|
async def _quotes_std(self, syms: list[Symbol], *,
|
||||||
|
concurrency: int | None = None,
|
||||||
|
pipeline: bool = True) -> list[SecurityQuote]:
|
||||||
|
"""A股标准通道五档报价(协议上限 80 只/请求,超出自动切批)。
|
||||||
|
|
||||||
|
切片多于 1 片时把各片**按轮转分给若干条连接**,每条连接内部用流水线
|
||||||
|
一次发出(1 次往返拿多片),不再逐片往返。
|
||||||
|
"""
|
||||||
|
if not syms:
|
||||||
|
return []
|
||||||
|
|
||||||
|
def stamp(rows: list[SecurityQuote]) -> list[SecurityQuote]:
|
||||||
for q in rows:
|
for q in rows:
|
||||||
q.symbol = format_symbol(q.market, q.code)
|
q.symbol = format_symbol(q.market, q.code)
|
||||||
out.extend(rows)
|
return rows
|
||||||
|
|
||||||
|
batches = [syms[i:i + MAX_QUOTES]
|
||||||
|
for i in range(0, len(syms), MAX_QUOTES)]
|
||||||
|
if len(batches) == 1:
|
||||||
|
cmd = GetSecurityQuotesCmd([s.as_pair() for s in batches[0]])
|
||||||
|
return stamp(await self._execute_standard(cmd))
|
||||||
|
|
||||||
|
conc = min(self._quote_concurrency(concurrency), len(batches))
|
||||||
|
|
||||||
|
# 首次大批量:先把连接并发建好,否则 acquire 会在池锁里串行握手
|
||||||
|
# (22 路并发首次调用实测多花 5~6s)
|
||||||
|
await self._std_pool.warm(conc)
|
||||||
|
|
||||||
|
if not pipeline:
|
||||||
|
sem = asyncio.Semaphore(conc)
|
||||||
|
|
||||||
|
async def one(batch: list[Symbol]) -> list[SecurityQuote]:
|
||||||
|
async with sem:
|
||||||
|
cmd = GetSecurityQuotesCmd([s.as_pair() for s in batch])
|
||||||
|
return stamp(await self._execute_standard(cmd))
|
||||||
|
|
||||||
|
groups = await asyncio.gather(*(one(b) for b in batches))
|
||||||
|
return [q for g in groups for q in g]
|
||||||
|
|
||||||
|
# 工作窃取:连接做完一批就回来再取,快的多干、慢的少干。
|
||||||
|
# 平均分配(每台固定几片)会被最慢的一台拖住 —— 实测 11 台平均分
|
||||||
|
# 时 p50 在 400~580ms 之间抖,改窃取后波动收窄。
|
||||||
|
queue = deque(batches)
|
||||||
|
gate = asyncio.Lock()
|
||||||
|
|
||||||
|
async def worker() -> list[SecurityQuote]:
|
||||||
|
got: list[SecurityQuote] = []
|
||||||
|
while True:
|
||||||
|
async with gate:
|
||||||
|
if not queue:
|
||||||
|
return got
|
||||||
|
take = min(_QUOTE_PIPE_CHUNK, len(queue))
|
||||||
|
chunk = [queue.popleft() for _ in range(take)]
|
||||||
|
for rows in await self._execute_std_pipeline(chunk):
|
||||||
|
got.extend(stamp(rows))
|
||||||
|
|
||||||
|
out: list[SecurityQuote] = []
|
||||||
|
for part in await asyncio.gather(*(worker() for _ in range(conc))):
|
||||||
|
out.extend(part)
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
async def _execute_std_pipeline(
|
||||||
|
self, batches: list[list[Symbol]]
|
||||||
|
) -> list[list[SecurityQuote]]:
|
||||||
|
"""一条连接流水线执行多片报价;节点不支持流水线时降级为逐片。"""
|
||||||
|
cmds = [GetSecurityQuotesCmd([s.as_pair() for s in b]) for b in batches]
|
||||||
|
pc = await self._std_pool.acquire()
|
||||||
|
try:
|
||||||
|
rows = await pc.execute_pipeline(cmds)
|
||||||
|
except Exception:
|
||||||
|
# 流水线中断后响应流已错配,连接必须废弃;改用逐片(每条独立借还)
|
||||||
|
await self._std_pool.release(pc, broken=True)
|
||||||
|
fallback: list[list[SecurityQuote]] = []
|
||||||
|
for cmd in cmds:
|
||||||
|
fallback.append(await self._execute_standard(cmd))
|
||||||
|
return fallback
|
||||||
|
await self._std_pool.release(pc)
|
||||||
|
return rows
|
||||||
|
|
||||||
async def get_snapshot(self, symbols: str | Sequence[str]) -> list[SymbolSnapshot]:
|
async def get_snapshot(self, symbols: str | Sequence[str]) -> list[SymbolSnapshot]:
|
||||||
"""个股特征快照(当日/换手/活跃度等,仅 A股)。"""
|
"""个股特征快照(当日/换手/活跃度等,仅 A股)。"""
|
||||||
syms = parse_symbols(symbols)
|
syms = parse_symbols(symbols)
|
||||||
@@ -788,12 +955,30 @@ class TdxData:
|
|||||||
|
|
||||||
参数:
|
参数:
|
||||||
date: ``None``=当日,``YYYYMMDD``=历史某日。
|
date: ``None``=当日,``YYYYMMDD``=历史某日。
|
||||||
start: 起始偏移。
|
start: 起始偏移,**从最新一笔往回数**(``0``=当天最后一条)。
|
||||||
|
``start=1000`` 表示跳过最近 1000 条、从更早的位置开始。
|
||||||
count: ``None``=尽量取全量(自动翻页,上限 50 页);给了就取这么多条。
|
count: ``None``=尽量取全量(自动翻页,上限 50 页);给了就取这么多条。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
``list[TransactionRecord]``,按入参标的顺序拼接(A股含 ``trade_count``,
|
``list[TransactionRecord]``,按入参标的顺序拼接,**每条标的内部按时间升序**。
|
||||||
港股为 HH:MM 时间 + ``fields["zengcang"]`` 增仓)。
|
注意服务端是按约 3 秒聚合的切片,``vol``/``trade_count`` 是切片增量
|
||||||
|
而非当日累计。
|
||||||
|
A股含 ``trade_count``,港股为 HH:MM 时间 + ``fields["zengcang"]`` 增仓。
|
||||||
|
|
||||||
|
**集合竞价委托(``bs_flag=8``)不会出现在结果里**:它们不是成交,``vol``
|
||||||
|
是累计撮合量(会被后续记录覆盖),计入会让 Σvol 比真值虚高 20%~160%,
|
||||||
|
且不同主机收录的条数不同 → 同一查询两次调用结果不一致。
|
||||||
|
|
||||||
|
**指数(``sh000016`` 等)的两个字段语义与个股不同**:
|
||||||
|
|
||||||
|
- ``vol`` 单位是**百元**(成交额/100),不是手。
|
||||||
|
- ``trade_count`` 是**当时均价 × 1000**,不是笔数。
|
||||||
|
|
||||||
|
**ETF / LOF 同样没有笔数**:``trade_count`` 是「价格 × 1000」(货币 ETF 恒为 0),
|
||||||
|
这类记录的 ``trade_count_available`` 为 ``False`` 且 ``trade_count`` 已置 0。
|
||||||
|
判定规则见 :func:`~pytdxdata.symbols.has_order_count`。
|
||||||
|
|
||||||
|
详见 :class:`~pytdxdata.models.TransactionRecord` 的类注释。
|
||||||
"""
|
"""
|
||||||
syms = parse_symbols(symbols)
|
syms = parse_symbols(symbols)
|
||||||
sem = asyncio.Semaphore(max(1, concurrency))
|
sem = asyncio.Semaphore(max(1, concurrency))
|
||||||
@@ -814,18 +999,19 @@ class TdxData:
|
|||||||
if sym.is_ex:
|
if sym.is_ex:
|
||||||
rows = await self._ticks_ex(sym, date=date, start=start, count=count)
|
rows = await self._ticks_ex(sym, date=date, start=start, count=count)
|
||||||
else:
|
else:
|
||||||
def make_cmd(s: int, c: int):
|
rows = await self._ticks_cn_serial(sym, date=date, start=start,
|
||||||
return MacTransactionCmd(sym.market, sym.code, date or 0, s, c)
|
count=count)
|
||||||
|
# ★ 过滤集合竞价**委托**记录(bs_flag=8)。必须在逐页拉取**全部结束之后**做:
|
||||||
spec = PageSpec(
|
# 分页靠 `len(rows) < page_size` 判终止,若在 parse_response 里过滤,
|
||||||
page_size=_TICK_PAGE,
|
# 满页会被剥短成"短页" → 提前终止 → 丢数据。
|
||||||
total=None if count is None else start + count,
|
# 实测(2026-10-21 sh600000):b8 是 09:15~09:25 / 14:57~15:00 竞价段的
|
||||||
min_page=100,
|
# 撮合快照,vol 是**累计值**(可被后续记录覆盖),不是成交增量;
|
||||||
stop_when_short=True,
|
# 计入后 Σvol 比真值虚高 20%~160%,且**不同主机的 b8 集合不同**
|
||||||
max_pages=_ALL_PAGES_CAP if count is None else None,
|
# (干净主机 0 条、脏主机 135 条)→ 同一查询两次调用结果不一致。
|
||||||
start_offset=start,
|
# 剔除后:Σvol 与快照 vol 精确相等(比值 1.0000),且跨主机逐条一致。
|
||||||
)
|
rows = [r for r in rows if r.bs_flag != _BS_AUCTION_ORDER]
|
||||||
rows = await self._fetch_paginated_list("mac", make_cmd, spec)
|
# 排序放在 pack 之前,让缓存直接存入正确顺序,避免每次命中都重排。
|
||||||
|
rows.sort(key=lambda r: r.time)
|
||||||
return pack_payload(rows)
|
return pack_payload(rows)
|
||||||
|
|
||||||
records = unpack_payload(await self._cache.get_or_fetch(key, fetch))
|
records = unpack_payload(await self._cache.get_or_fetch(key, fetch))
|
||||||
@@ -833,6 +1019,54 @@ class TdxData:
|
|||||||
r.symbol = sym.text
|
r.symbol = sym.text
|
||||||
return records
|
return records
|
||||||
|
|
||||||
|
async def _ticks_cn_serial(self, sym: Symbol, *, date: int | None, start: int,
|
||||||
|
count: int | None) -> list[TransactionRecord]:
|
||||||
|
"""A股逐笔(0x122F)**单连接串行游标**翻页。
|
||||||
|
|
||||||
|
为什么不用并发分页器:``0x122F`` 的 ``start`` 是「从**最新一笔**往回数的偏移」,
|
||||||
|
是个**相对锚点**,不是稳定的绝对序号。而池里各主机对集合竞价委托(``bs_flag=8``)
|
||||||
|
的收录**不一致**(实测干净主机 0 条、脏主机 135 条),同一 ``start`` 在不同主机上
|
||||||
|
落到**不同的真实成交**上。并发分页 = 各页来自不同主机 → 页边界既**重叠**
|
||||||
|
(sh600000 冗余 116 条)又**漏条**(13:03 附近丢 59 条),Σvol 偏移 −1.2%。
|
||||||
|
|
||||||
|
逐笔单日仅 5~6 页(1000 条/页),串行开销可忽略、换来精确一致,
|
||||||
|
故这里固定**单连接**、按 ``start`` 递增(= 向更早方向)推进。
|
||||||
|
|
||||||
|
返回:按页序累加后再**整体翻转**,得到时间升序的完整列表。
|
||||||
|
"""
|
||||||
|
chunks: list[list[TransactionRecord]] = []
|
||||||
|
pos = start
|
||||||
|
remaining = count
|
||||||
|
pc = await self._mac_pool.acquire()
|
||||||
|
try:
|
||||||
|
for _ in range(_ALL_PAGES_CAP):
|
||||||
|
n = _TICK_PAGE if remaining is None else min(_TICK_PAGE, remaining)
|
||||||
|
if n <= 0:
|
||||||
|
break
|
||||||
|
rows = await pc.execute(
|
||||||
|
MacTransactionCmd(sym.market, sym.code, date or 0, pos, n))
|
||||||
|
if not rows:
|
||||||
|
break
|
||||||
|
chunks.append(list(rows))
|
||||||
|
pos += len(rows)
|
||||||
|
if remaining is not None:
|
||||||
|
remaining -= len(rows)
|
||||||
|
if remaining <= 0:
|
||||||
|
break
|
||||||
|
# 短页 = 已到最早一笔
|
||||||
|
if len(rows) < n:
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
await self._mac_pool.release(pc, broken=True)
|
||||||
|
raise
|
||||||
|
await self._mac_pool.release(pc)
|
||||||
|
out: list[TransactionRecord] = []
|
||||||
|
for chunk in reversed(chunks):
|
||||||
|
out.extend(chunk)
|
||||||
|
for r in out:
|
||||||
|
r.symbol = sym.text
|
||||||
|
return out
|
||||||
|
|
||||||
async def _ticks_ex(self, sym: Symbol, *, date: int | None, start: int,
|
async def _ticks_ex(self, sym: Symbol, *, date: int | None, start: int,
|
||||||
count: int | None) -> list[TransactionRecord]:
|
count: int | None) -> list[TransactionRecord]:
|
||||||
from .protocol.commands.ex_ext import (
|
from .protocol.commands.ex_ext import (
|
||||||
@@ -841,7 +1075,7 @@ class TdxData:
|
|||||||
)
|
)
|
||||||
hk = sym.market in _HK_MARKETS
|
hk = sym.market in _HK_MARKETS
|
||||||
page = _EX_TICK_PAGE if hk else _TICK_PAGE
|
page = _EX_TICK_PAGE if hk else _TICK_PAGE
|
||||||
out: list[TransactionRecord] = []
|
chunks: list[list[TransactionRecord]] = []
|
||||||
pos = start
|
pos = start
|
||||||
remaining = count
|
remaining = count
|
||||||
for _ in range(_ALL_PAGES_CAP):
|
for _ in range(_ALL_PAGES_CAP):
|
||||||
@@ -853,22 +1087,27 @@ class TdxData:
|
|||||||
if date else
|
if date else
|
||||||
GetExTransactionDataCmd(sym.market, sym.code, pos, n))
|
GetExTransactionDataCmd(sym.market, sym.code, pos, n))
|
||||||
rows = await self._execute_ex(cmd)
|
rows = await self._execute_ex(cmd)
|
||||||
out.extend(self._ex_tick(sym, r) for r in rows)
|
chunk = [self._ex_tick(sym, r) for r in rows]
|
||||||
else:
|
else:
|
||||||
cmd = MacTransactionCmd(sym.market, sym.code, date or 0, pos, n)
|
cmd = MacTransactionCmd(sym.market, sym.code, date or 0, pos, n)
|
||||||
rows = await self._execute_ex(cmd)
|
rows = await self._execute_ex(cmd)
|
||||||
for r in rows:
|
for r in rows:
|
||||||
r.symbol = sym.text
|
r.symbol = sym.text
|
||||||
out.extend(rows)
|
chunk = list(rows)
|
||||||
if not rows:
|
if not chunk:
|
||||||
break
|
break
|
||||||
pos += len(rows)
|
# EX 通道的 pos 同样是"从最新往回数",逐页倒序累加 → 最后整体翻转。
|
||||||
|
chunks.append(chunk)
|
||||||
|
pos += len(chunk)
|
||||||
if remaining is not None:
|
if remaining is not None:
|
||||||
remaining -= len(rows)
|
remaining -= len(chunk)
|
||||||
if remaining <= 0:
|
if remaining <= 0:
|
||||||
break
|
break
|
||||||
if len(rows) < n:
|
if len(chunk) < n:
|
||||||
break
|
break
|
||||||
|
out: list[TransactionRecord] = []
|
||||||
|
for chunk in reversed(chunks):
|
||||||
|
out.extend(chunk)
|
||||||
return out
|
return out
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
@@ -885,6 +1124,7 @@ class TdxData:
|
|||||||
trade_count=0,
|
trade_count=0,
|
||||||
bs_flag=int(row.get("direction", 0) or 0),
|
bs_flag=int(row.get("direction", 0) or 0),
|
||||||
fields=fields,
|
fields=fields,
|
||||||
|
trade_count_available=False, # 扩展市场逐笔不带笔数字段
|
||||||
)
|
)
|
||||||
|
|
||||||
async def get_minutes(
|
async def get_minutes(
|
||||||
@@ -895,8 +1135,9 @@ class TdxData:
|
|||||||
|
|
||||||
参数:
|
参数:
|
||||||
date: ``None``=今天;``YYYYMMDD``=历史某日(A股支持历史分时)。
|
date: ``None``=今天;``YYYYMMDD``=历史某日(A股支持历史分时)。
|
||||||
sampling: ``True`` 取缩略采样(约240个价格点,无时间;A股 0x254D,
|
sampling: ``True`` 取缩略采样(约 240 个价格点,无时间),``vol``/
|
||||||
扩展市场同命令),``vol``/``avg_price`` 为 0,点位序号在 ``fields["i"]``。
|
``avg_price`` 为 0,点位序号在 ``fields["i"]``。A股由分时数据派生,
|
||||||
|
扩展市场走 0x254D。
|
||||||
|
|
||||||
返回:
|
返回:
|
||||||
``list[MinuteBar]``,按入参标的顺序拼接,每条带 ``symbol``/``market``/
|
``list[MinuteBar]``,按入参标的顺序拼接,每条带 ``symbol``/``market``/
|
||||||
@@ -920,8 +1161,17 @@ class TdxData:
|
|||||||
|
|
||||||
async def fetch() -> bytes:
|
async def fetch() -> bytes:
|
||||||
if sampling:
|
if sampling:
|
||||||
run = self._execute_ex if sym.is_ex else self._execute_mac
|
if sym.is_ex:
|
||||||
prices = await run(MacChartSamplingCmd(sym.market, sym.code))
|
prices = await self._execute_ex(
|
||||||
|
MacChartSamplingCmd(sym.market, sym.code))
|
||||||
|
else:
|
||||||
|
# ⚠️ 不能用 MacChartSamplingCmd(0x254D):那是 **EX 协议**命令,
|
||||||
|
# A股/标准通道的 7709 主机对它一律不响应(实测 3 台全 8s 超时,
|
||||||
|
# 叠加连接池 3 次重试 ≈24s 才报错)。改由分时(0x122D)派生价格点:
|
||||||
|
# 分时本身就是 240 点,语义与「缩略采样」一致且天然可用。
|
||||||
|
ticks = await self._execute_mac(
|
||||||
|
MacTickChartCmd(sym.market, sym.code, ymd))
|
||||||
|
prices = [b.price for b in ticks]
|
||||||
bars = [MinuteBar(price=float(p), vol=0.0, symbol=sym.text,
|
bars = [MinuteBar(price=float(p), vol=0.0, symbol=sym.text,
|
||||||
market=sym.market, code=sym.code, date=ymd,
|
market=sym.market, code=sym.code, date=ymd,
|
||||||
fields={"i": float(i)})
|
fields={"i": float(i)})
|
||||||
@@ -1204,6 +1454,248 @@ class TdxData:
|
|||||||
result.sort(key=lambda x: x["change_pct"], reverse=True)
|
result.sort(key=lambda x: x["change_pct"], reverse=True)
|
||||||
return result[:top_n]
|
return result[:top_n]
|
||||||
|
|
||||||
|
# ================================================================== #
|
||||||
|
# 七之二、指数成分成交汇总(成分股逐笔 → 每分钟成交量 / 笔数)
|
||||||
|
# ================================================================== #
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _index_pair(index: str) -> tuple[int, str]:
|
||||||
|
"""解析指数标的 → ``(市场号, 6 位代码)``。
|
||||||
|
|
||||||
|
与 :func:`~pytdxdata.symbols.parse_symbol` 的关键区别:**裸 6 位数字
|
||||||
|
一律按指数语义解析**,不做「``000xxx`` → 深市股票」的推断 ——
|
||||||
|
在指数接口里 ``000001`` 是**上证指数**,不是平安银行 ``sz000001``。
|
||||||
|
|
||||||
|
判定规则(与通达信板块码一致)::
|
||||||
|
|
||||||
|
000xxx / 999xxx / 880xxx -> 沪市(上证指数 / A股指数 / 板块指数)
|
||||||
|
399xxx -> 深市(深证成指 / 创业板指)
|
||||||
|
899xxx -> 北交所(北证50)
|
||||||
|
|
||||||
|
显式前缀 ``sh``/``sz``/``bj`` 优先,例如 ``sz399001``、``sh000300``。
|
||||||
|
"""
|
||||||
|
s = str(index).strip().lower().replace(".", "").replace(":", "")
|
||||||
|
if not s:
|
||||||
|
raise ValueError("指数标的不能为空")
|
||||||
|
prefix_table = {"sh": 1, "sz": 0, "bj": 2}
|
||||||
|
if s[:2] in prefix_table:
|
||||||
|
code = s[2:]
|
||||||
|
if not (code.isdigit() and len(code) == 6):
|
||||||
|
raise ValueError(f"非法指数标的: {index!r}(前缀后需 6 位数字)")
|
||||||
|
return prefix_table[s[:2]], code
|
||||||
|
if not (s.isdigit() and len(s) == 6):
|
||||||
|
raise ValueError(f"非法指数标的: {index!r}(需 6 位代码或 sh/sz/bj 前缀)")
|
||||||
|
if s.startswith(("000", "999", "880")):
|
||||||
|
return 1, s
|
||||||
|
if s.startswith("399"):
|
||||||
|
return 0, s
|
||||||
|
if s.startswith("899"):
|
||||||
|
return 2, s
|
||||||
|
raise ValueError(f"无法判定指数市场: {index!r}(不是 A股指数代码段)")
|
||||||
|
|
||||||
|
async def get_index_members(self, index: str, *, count: int = 100000,
|
||||||
|
fields: Sequence[int] | None = None
|
||||||
|
) -> list[SecurityQuote]:
|
||||||
|
"""指数**成分股**(含实时报价)。
|
||||||
|
|
||||||
|
走 MAC ``0x122C`` 板块成分命令,指数代码映射为服务端板块码:``000300`` →
|
||||||
|
31300、``399001`` → 30001、``899050`` → 32050(见 :meth:`_convert_board_code`)。
|
||||||
|
实测(2026-10-22):``sh000001`` 2230 只(沪市全部)、``sh000300`` 300 只、
|
||||||
|
``sh000016`` 50 只、``sh000905`` 500 只、``sh000852`` 1000 只、
|
||||||
|
``sz399006`` 100 只、``bj899050`` 50 只。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
index: 指数代码,可裸写或带前缀(``"000300"`` / ``"sh000300"``)。
|
||||||
|
count: 最多取的成分数(分页上限,默认足够覆盖全部)。
|
||||||
|
fields: 附加报价字段位(同 :meth:`get_board_members`)。
|
||||||
|
|
||||||
|
注意:返回的是**当前**成分,指数调仓后历史日期不适用。
|
||||||
|
|
||||||
|
⚠️ **综合指数的成员名单不完整**(实测 2026-10-22):``sh000001`` 只返回
|
||||||
|
2230 只(2190 A + 40 B),而当日沪市 A 股实有 2318 只;缺的 128 只
|
||||||
|
(多为近期科创板/主板新股)当日成交 282 亿,正好对上「指数日线 amount −
|
||||||
|
成员日线 amount」的 297 亿(残差 4.9%)。因此
|
||||||
|
:meth:`get_index_trade_stats` 对**综合指数**会系统性低估约 4.6%;
|
||||||
|
样本指数 ``sh000016``(50 只)/ ``sh000300``(300 只)实测比值
|
||||||
|
**1.000000**,不受影响。要精确汇总综指成交,请自行用全市场股票列表。
|
||||||
|
|
||||||
|
用法::
|
||||||
|
|
||||||
|
members = await td.get_index_members("000300")
|
||||||
|
syms = [m.symbol for m in members] # ['sz000001', ...]
|
||||||
|
"""
|
||||||
|
_market, code = self._index_pair(index)
|
||||||
|
board_code = self._convert_board_code(code)
|
||||||
|
bits = [int(b) for b in fields] if fields else list(_QUOTE_BITS)
|
||||||
|
items: list[SecurityQuote] = []
|
||||||
|
start = 0
|
||||||
|
while start < count:
|
||||||
|
batch = await self._execute_mac(MacBoardMembersQuotesCmd(
|
||||||
|
board_code, sort_type=0, start=start,
|
||||||
|
page_size=_QUOTE_PAGE, bits=bits))
|
||||||
|
if not batch:
|
||||||
|
break
|
||||||
|
items.extend(_quote_from_member(m) for m in batch)
|
||||||
|
start += len(batch)
|
||||||
|
if len(batch) < _QUOTE_PAGE:
|
||||||
|
break
|
||||||
|
return items[:count]
|
||||||
|
|
||||||
|
async def get_index_trade_stats(
|
||||||
|
self,
|
||||||
|
index: str,
|
||||||
|
date: int | None = None,
|
||||||
|
*,
|
||||||
|
concurrency: int = 24,
|
||||||
|
batch_size: int = 50,
|
||||||
|
members: Sequence[str] | None = None,
|
||||||
|
skip_errors: bool = True,
|
||||||
|
on_progress: Callable[[int, int], None] | None = None,
|
||||||
|
) -> IndexTradeStats:
|
||||||
|
"""按指数**全部成分股**的逐笔汇总出每分钟成交量与成交笔数。
|
||||||
|
|
||||||
|
为什么必须绕这一圈:**通达信协议里指数本身没有成交笔数** —— 指数逐笔的
|
||||||
|
``trade_count`` 是「当时均价 × 1000」,不是笔数(见
|
||||||
|
:class:`TransactionRecord` 类注释)。所以「指数笔数」只能用成分股逐笔累加。
|
||||||
|
|
||||||
|
口径校验:``total_amount`` 是**成分股**合计。对「成分 = 指数覆盖面」的指数
|
||||||
|
它与指数快照 ``amount`` 完全相等(``sh000016`` 661.34 亿、``sh000300``
|
||||||
|
2373.98 亿,比值 1.0000);但深市 ``399xxx`` 的指数 ``amount`` 是**全板块**
|
||||||
|
口径(``sz399001`` = 深市全部股票 5081 亿、``sz399006`` = 创业板全部),
|
||||||
|
两者不可直接比。详见 :class:`IndexTradeStats`。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
index: 指数代码(``"000300"`` / ``"sh000300"``,规则见 :meth:`_index_pair`)。
|
||||||
|
date: ``None`` = 当日;``YYYYMMDD`` = 历史某日(通达信逐笔仅保留最近
|
||||||
|
若干交易日,实测 2026-10-22 可回溯到 10-14,更早返回空)。
|
||||||
|
concurrency: 逐笔并发数(受 MAC 连接池上限约束,池满自动排队)。
|
||||||
|
batch_size: 每批处理的成分数 —— 批内并发拉取、批间串行聚合,
|
||||||
|
用于压住内存峰值(全量 tick 不驻留)。
|
||||||
|
members: 直接给成分股列表(跳过 :meth:`get_index_members`,
|
||||||
|
多日循环时可复用一次成分请求)。
|
||||||
|
skip_errors: 单只成分失败是否忽略(默认忽略并计入 ``failed_members``)。
|
||||||
|
on_progress: 进度回调 ``(已处理, 总数)``,每批结束调用一次。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
:class:`IndexTradeStats`;``minutes`` 是**每分钟增量**,
|
||||||
|
「当日截止该分钟累计」用 ``.cumulative()`` / ``.cum_trade_count()``。
|
||||||
|
|
||||||
|
性能(实测 2026-10-22,盘中):
|
||||||
|
|
||||||
|
====================== ========== ============
|
||||||
|
指数 成分数 耗时
|
||||||
|
====================== ========== ============
|
||||||
|
``sh000016`` 上证50 50 ~5 s
|
||||||
|
``sh000300`` 沪深300 300 ~10 s
|
||||||
|
``sh000001`` 上证指数 2230 ~60-90 s
|
||||||
|
====================== ========== ============
|
||||||
|
|
||||||
|
**北交所成分没有笔数**(实测 ``trade_count`` 恒 0),此时
|
||||||
|
``trade_count_available`` 为 ``False``,只有成交量可用。
|
||||||
|
|
||||||
|
用法::
|
||||||
|
|
||||||
|
st = await td.get_index_trade_stats("000300")
|
||||||
|
for hhmm, cum in st.cum_trade_count():
|
||||||
|
print(hhmm, cum)
|
||||||
|
"""
|
||||||
|
started = time.perf_counter()
|
||||||
|
market, code = self._index_pair(index)
|
||||||
|
idx_text = format_symbol(market, code)
|
||||||
|
|
||||||
|
if members is None:
|
||||||
|
members = [m.symbol for m in await self.get_index_members(code)]
|
||||||
|
syms = [s for s in members if s]
|
||||||
|
total = len(syms)
|
||||||
|
if total == 0:
|
||||||
|
raise ValueError(f"指数 {index!r} 没有取到成分股")
|
||||||
|
|
||||||
|
vol_by_min: dict[str, float] = {}
|
||||||
|
tc_by_min: dict[str, int] = {}
|
||||||
|
amt_by_min: dict[str, float] = {}
|
||||||
|
active: set[str] = set()
|
||||||
|
with_tc: set[str] = set()
|
||||||
|
failed = 0
|
||||||
|
step = max(1, int(batch_size))
|
||||||
|
done = 0
|
||||||
|
for i in range(0, total, step):
|
||||||
|
batch = syms[i:i + step]
|
||||||
|
try:
|
||||||
|
ticks = await self.get_ticks(batch, date=date, concurrency=concurrency)
|
||||||
|
except Exception:
|
||||||
|
if not skip_errors:
|
||||||
|
raise
|
||||||
|
# 整批失败 → 退化为逐只重试,避免一只坏股拖垮一批
|
||||||
|
ticks = []
|
||||||
|
for one in batch:
|
||||||
|
try:
|
||||||
|
ticks.extend(await self.get_ticks(one, date=date))
|
||||||
|
except Exception:
|
||||||
|
failed += 1
|
||||||
|
for t in ticks:
|
||||||
|
hhmm = t.time[:5]
|
||||||
|
vol_by_min[hhmm] = vol_by_min.get(hhmm, 0.0) + t.vol
|
||||||
|
# 成分里若混入 ETF/LOF(协议无笔数),不能把占位值累加进笔数
|
||||||
|
if getattr(t, "trade_count_available", True):
|
||||||
|
tc_by_min[hhmm] = tc_by_min.get(hhmm, 0) + t.trade_count
|
||||||
|
amt_by_min[hhmm] = amt_by_min.get(hhmm, 0.0) + t.price * t.vol * 100.0
|
||||||
|
active.add(t.symbol)
|
||||||
|
if getattr(t, "trade_count_available", True) and t.trade_count:
|
||||||
|
with_tc.add(t.symbol)
|
||||||
|
done += len(batch)
|
||||||
|
if on_progress is not None:
|
||||||
|
on_progress(min(done, total), total)
|
||||||
|
|
||||||
|
minutes = [
|
||||||
|
MinuteTradeStat(time=k, volume=vol_by_min[k],
|
||||||
|
trade_count=tc_by_min[k], amount=amt_by_min[k])
|
||||||
|
for k in sorted(vol_by_min)
|
||||||
|
]
|
||||||
|
return IndexTradeStats(
|
||||||
|
index=idx_text,
|
||||||
|
date=date,
|
||||||
|
member_count=total,
|
||||||
|
active_members=len(active),
|
||||||
|
trade_count_members=len(with_tc),
|
||||||
|
minutes=minutes,
|
||||||
|
total_volume=sum(vol_by_min.values()),
|
||||||
|
total_trade_count=sum(tc_by_min.values()),
|
||||||
|
total_amount=sum(amt_by_min.values()),
|
||||||
|
failed_members=failed,
|
||||||
|
elapsed=time.perf_counter() - started,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def get_index_trade_stats_multi(
|
||||||
|
self,
|
||||||
|
index: str,
|
||||||
|
dates: Sequence[int | None],
|
||||||
|
*,
|
||||||
|
concurrency: int = 24,
|
||||||
|
batch_size: int = 50,
|
||||||
|
skip_errors: bool = True,
|
||||||
|
on_progress: Callable[[int, int], None] | None = None,
|
||||||
|
) -> list[IndexTradeStats]:
|
||||||
|
"""多日批量版 :meth:`get_index_trade_stats`(成分只取一次)。
|
||||||
|
|
||||||
|
用于「近 N 日同时点累计平均笔数」这类基准曲线:一次拿到 N 天数据,
|
||||||
|
避免每天重复请求成分表。
|
||||||
|
|
||||||
|
用法::
|
||||||
|
|
||||||
|
# 最近 5 个交易日的笔数曲线(日期自己给,节假日请自行跳过)
|
||||||
|
stats = await td.get_index_trade_stats_multi(
|
||||||
|
"000300", [20261016, 20261019, 20261020, 20261021, 20261022])
|
||||||
|
"""
|
||||||
|
market, code = self._index_pair(index)
|
||||||
|
members = [m.symbol for m in await self.get_index_members(code)]
|
||||||
|
out: list[IndexTradeStats] = []
|
||||||
|
for d in dates:
|
||||||
|
out.append(await self.get_index_trade_stats(
|
||||||
|
code, d, concurrency=concurrency, batch_size=batch_size,
|
||||||
|
members=members, skip_errors=skip_errors,
|
||||||
|
on_progress=on_progress))
|
||||||
|
return out
|
||||||
|
|
||||||
# ================================================================== #
|
# ================================================================== #
|
||||||
# 八、服务器文件(原始字节 + 解析)
|
# 八、服务器文件(原始字节 + 解析)
|
||||||
# ================================================================== #
|
# ================================================================== #
|
||||||
@@ -1273,28 +1765,60 @@ class TdxData:
|
|||||||
return parse_financial_file_list(await self.get_file("tdxfin/gpcw.txt"))
|
return parse_financial_file_list(await self.get_file("tdxfin/gpcw.txt"))
|
||||||
|
|
||||||
async def get_financial_records_parsed(self, filename: str) -> list[FinancialRecord]:
|
async def get_financial_records_parsed(self, filename: str) -> list[FinancialRecord]:
|
||||||
"""财务 zip(``tdxfin/gpcw*.zip``)解压解析 -> ``list[FinancialRecord]``。
|
"""财报数据解析 -> ``list[FinancialRecord]``。
|
||||||
|
|
||||||
|
可直接把 :meth:`get_financial_file_infos` 条目里的 ``filename`` 原样传进来:
|
||||||
|
内部会自动纠正「索引名与服务器实际文件名不一致」的问题(索引写
|
||||||
|
``gpcwYYYYMMDD.zip`` 且不带 ``tdxfin/`` 前缀,服务器实际只提供未压缩的
|
||||||
|
``tdxfin/gpcwYYYYMMDD.dat``),并兼容 zip 包与未压缩 dat 两种形态。
|
||||||
|
|
||||||
报告期优先取文件名中的 8 位日期,缺失时回退到 .dat 头部内嵌值。
|
报告期优先取文件名中的 8 位日期,缺失时回退到 .dat 头部内嵌值。
|
||||||
"""
|
"""
|
||||||
import io
|
|
||||||
import re
|
|
||||||
import zipfile
|
|
||||||
|
|
||||||
from .codec.financial import parse_financial_dat
|
from .codec.financial import parse_financial_dat
|
||||||
|
|
||||||
zip_data = await self.get_file(filename)
|
dat_data = await self._fetch_financial_dat(filename)
|
||||||
if not zip_data:
|
if not dat_data:
|
||||||
return []
|
return []
|
||||||
with zipfile.ZipFile(io.BytesIO(zip_data)) as zf:
|
match = re.search(r"(\d{8})", str(filename))
|
||||||
dat_names = [n for n in zf.namelist() if n.endswith(".dat")]
|
|
||||||
if not dat_names:
|
|
||||||
return []
|
|
||||||
dat_data = zf.read(dat_names[0])
|
|
||||||
match = re.search(r"(\d{8})", filename)
|
|
||||||
report_date = int(match.group(1)) if match else 0
|
report_date = int(match.group(1)) if match else 0
|
||||||
return parse_financial_dat(dat_data, report_date)
|
return parse_financial_dat(dat_data, report_date)
|
||||||
|
|
||||||
|
async def _fetch_financial_dat(self, filename: str) -> bytes:
|
||||||
|
"""[内部] 取回 gpcw 财报 ``.dat`` 原始字节,自动纠正服务器文件名差异。
|
||||||
|
|
||||||
|
``tdxfin/gpcw.txt`` 索引声明的名字(``gpcwYYYYMMDD.zip``、无目录前缀)与服务器
|
||||||
|
实际提供的文件(``tdxfin/gpcwYYYYMMDD.dat``、未压缩)不一致,且对不存在的名字
|
||||||
|
服务器**不报错、只回 0~1 字节空壳**。这里按候选名逐个尝试,并校验取回的字节
|
||||||
|
是否真的是 ``.dat``,返回第一个有效结果;全部失败返回 ``b""``。
|
||||||
|
|
||||||
|
候选顺序:索引声明的 ``.zip`` → 服务器实际的 ``.dat`` → 调用方原样名字(兜底)。
|
||||||
|
"""
|
||||||
|
from .codec.financial import extract_gpcw_dat
|
||||||
|
|
||||||
|
raw = str(filename).strip().replace("\\", "/")
|
||||||
|
base = raw.rsplit("/", 1)[-1]
|
||||||
|
stem = re.sub(r"\.(?:zip|dat)$", "", base, flags=re.IGNORECASE)
|
||||||
|
candidates = list(dict.fromkeys([
|
||||||
|
f"tdxfin/{stem}.zip",
|
||||||
|
f"tdxfin/{stem}.dat",
|
||||||
|
base,
|
||||||
|
]))
|
||||||
|
for name in candidates:
|
||||||
|
try:
|
||||||
|
blob = await self.get_file(name)
|
||||||
|
except (ConnectionError, OSError, TimeoutError,
|
||||||
|
TdxDecodeError, ValueError, PoolTimeoutError) as exc:
|
||||||
|
# 单个候选取回失败就换下一个(服务器对不存在的名字不一定报错)
|
||||||
|
log.debug("财报候选 %s 取回失败: %s", name, str(exc)[:80])
|
||||||
|
continue
|
||||||
|
dat = extract_gpcw_dat(blob)
|
||||||
|
if dat:
|
||||||
|
log.debug("财报文件命中: %s(%d 字节)", name, len(dat))
|
||||||
|
return dat
|
||||||
|
log.debug("财报候选 %s 非有效 gpcw 数据(%d 字节),继续尝试",
|
||||||
|
name, len(blob))
|
||||||
|
return b""
|
||||||
|
|
||||||
# ================================================================== #
|
# ================================================================== #
|
||||||
# 内部执行(连接池 + 失败换连接 + 分页)
|
# 内部执行(连接池 + 失败换连接 + 分页)
|
||||||
# ================================================================== #
|
# ================================================================== #
|
||||||
@@ -1553,9 +2077,13 @@ class TdxData:
|
|||||||
return [l.strip() for l in text.splitlines() if l.strip()]
|
return [l.strip() for l in text.splitlines() if l.strip()]
|
||||||
|
|
||||||
async def get_financial_records(self, filename: str) -> bytes:
|
async def get_financial_records(self, filename: str) -> bytes:
|
||||||
"""已废弃 -> :meth:`get_financial_records_parsed`。"""
|
"""已废弃 -> :meth:`get_financial_records_parsed`。
|
||||||
|
|
||||||
|
返回 gpcw 财报 ``.dat`` 原始字节(服务器给未压缩 ``.dat`` 时自动回退,
|
||||||
|
不再像旧实现那样静默返回 0~1 字节空壳);取不到时返回 ``b""``。
|
||||||
|
"""
|
||||||
_deprecated("get_financial_records", "get_financial_records_parsed")
|
_deprecated("get_financial_records", "get_financial_records_parsed")
|
||||||
return await self.get_file(filename)
|
return await self._fetch_financial_dat(filename)
|
||||||
|
|
||||||
async def get_company_info_category(self, market, code) -> list:
|
async def get_company_info_category(self, market, code) -> list:
|
||||||
"""已废弃 -> :meth:`get_f10`。"""
|
"""已废弃 -> :meth:`get_f10`。"""
|
||||||
|
|||||||
Vendored
+32
@@ -90,6 +90,38 @@ class CacheManager:
|
|||||||
# 确保清理锁(即使 fetcher 异常也清理,避免无限增长)
|
# 确保清理锁(即使 fetcher 异常也清理,避免无限增长)
|
||||||
self._locks.pop(ks, None)
|
self._locks.pop(ks, None)
|
||||||
|
|
||||||
|
async def get_or_fetch_obj(
|
||||||
|
self,
|
||||||
|
key: CacheKey,
|
||||||
|
fetcher: Callable[[], Awaitable[Any]],
|
||||||
|
*,
|
||||||
|
pack: Callable[[Any], bytes],
|
||||||
|
unpack: Callable[[bytes], Any],
|
||||||
|
) -> Any:
|
||||||
|
"""同 :meth:`get_or_fetch`,但**回源时直接返回新取到的对象**。
|
||||||
|
|
||||||
|
用于「一次取回上千条记录」的场景(如全市场报价):常规路径对冷数据
|
||||||
|
会先 ``pack`` 再立刻 ``unpack`` 一次,纯属往返浪费 —— 5,200 条报价实测
|
||||||
|
白烧约 90ms。这里命中缓存才解序列化,回源则把对象原样返回,只在写缓存时打包。
|
||||||
|
"""
|
||||||
|
ks = key.to_str()
|
||||||
|
cached = await self.get(key)
|
||||||
|
if cached is not None:
|
||||||
|
return unpack(cached)
|
||||||
|
|
||||||
|
lock = self._locks.setdefault(ks, asyncio.Lock())
|
||||||
|
try:
|
||||||
|
async with lock:
|
||||||
|
cached = await self.get(key)
|
||||||
|
if cached is not None:
|
||||||
|
return unpack(cached)
|
||||||
|
async with self._semaphore:
|
||||||
|
obj = await fetcher()
|
||||||
|
await self.set(key, pack(obj))
|
||||||
|
return obj
|
||||||
|
finally:
|
||||||
|
self._locks.pop(ks, None)
|
||||||
|
|
||||||
async def invalidate(self, category: str, *, market: int | None = None,
|
async def invalidate(self, category: str, *, market: int | None = None,
|
||||||
code: str | None = None) -> int:
|
code: str | None = None) -> int:
|
||||||
"""按前缀批量失效,返回失效条数。"""
|
"""按前缀批量失效,返回失效条数。"""
|
||||||
|
|||||||
+112
-13
@@ -8,6 +8,7 @@
|
|||||||
tdx transactions sz000001 --count 50 # 逐笔成交
|
tdx transactions sz000001 --count 50 # 逐笔成交
|
||||||
tdx minute sz000001 # 分时(今日/历史--date)
|
tdx minute sz000001 # 分时(今日/历史--date)
|
||||||
tdx index 000001 --count 30 # 指数K线
|
tdx index 000001 --count 30 # 指数K线
|
||||||
|
tdx index-stats 000300 --top 10 # 指数成分成交汇总(每分钟成交量/笔数)
|
||||||
tdx board list [--type 1] # 板块列表(0全部/1行业/2概念)
|
tdx board list [--type 1] # 板块列表(0全部/1行业/2概念)
|
||||||
tdx board members 881001 --count 20 # 板块成分
|
tdx board members 881001 --count 20 # 板块成分
|
||||||
tdx board ranking [--top 10] # 板块涨跌幅排行
|
tdx board ranking [--top 10] # 板块涨跌幅排行
|
||||||
@@ -37,6 +38,7 @@ from .api import TdxData
|
|||||||
from dataclasses import asdict
|
from dataclasses import asdict
|
||||||
|
|
||||||
from .models import Adjust, ExMarket, KlinePeriod, Market
|
from .models import Adjust, ExMarket, KlinePeriod, Market
|
||||||
|
from .protocol.commands.quotes import price_decimal_digits
|
||||||
|
|
||||||
PERIODS = {
|
PERIODS = {
|
||||||
"min1": KlinePeriod.MIN_1, "min5": KlinePeriod.MIN_5, "min15": KlinePeriod.MIN_15,
|
"min1": KlinePeriod.MIN_1, "min5": KlinePeriod.MIN_5, "min15": KlinePeriod.MIN_15,
|
||||||
@@ -106,6 +108,40 @@ def _pct(price: float, pre: float) -> str:
|
|||||||
return "%+.2f%%" % ((price / pre - 1) * 100)
|
return "%+.2f%%" % ((price / pre - 1) * 100)
|
||||||
|
|
||||||
|
|
||||||
|
def _price_fmt(dp: int) -> str:
|
||||||
|
"""按品种小数位生成右对齐格式串。
|
||||||
|
|
||||||
|
股票 2 位、**可转债/ETF 3 位**、债券/回购 4 位。固定写 ``%8.2f`` 会把
|
||||||
|
可转债的 128.891 打成 128.89(少一位有效报价)。
|
||||||
|
"""
|
||||||
|
dp = max(0, min(int(dp or 2), 4))
|
||||||
|
return "%%%d.%df" % (6 + dp, dp)
|
||||||
|
|
||||||
|
|
||||||
|
# 可转债后缀(解码 4 位,但交易所报价步长 0.001 元,第 4 位恒为 0)
|
||||||
|
_CB_PREFIX = {1: ("110", "111", "113", "118"), 0: ("123", "127", "128")}
|
||||||
|
|
||||||
|
|
||||||
|
def _display_dp(market: int, code: str) -> int:
|
||||||
|
"""报价**显示**小数位。
|
||||||
|
|
||||||
|
注意 ``price_decimal_digits`` 给的是**解码位数**(原始整数 / 10**dp),
|
||||||
|
可转债是 4 位;但真实报价是 0.001 元的步长,第 4 位恒为 0,
|
||||||
|
按 4 位打印只会多一个 ``0``(``128.8920``)。
|
||||||
|
"""
|
||||||
|
dp = price_decimal_digits(market, code)
|
||||||
|
if str(code).startswith(_CB_PREFIX.get(int(market), ())):
|
||||||
|
return 3
|
||||||
|
return dp
|
||||||
|
|
||||||
|
|
||||||
|
def _dp_of(sym) -> int:
|
||||||
|
"""Symbol -> 价格小数位(非 A 股/指数一律 2 位)。"""
|
||||||
|
if getattr(sym, "kind", "") == "cn":
|
||||||
|
return _display_dp(sym.market, sym.code)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------ #
|
# ------------------------------------------------------------------ #
|
||||||
# 各子命令
|
# 各子命令
|
||||||
# ------------------------------------------------------------------ #
|
# ------------------------------------------------------------------ #
|
||||||
@@ -119,24 +155,31 @@ async def cmd_quotes(td: TdxData, symbols: list[str]) -> None:
|
|||||||
def _print_quote(q) -> None:
|
def _print_quote(q) -> None:
|
||||||
pct = (q.price / q.pre_close - 1) * 100 if q.pre_close else 0
|
pct = (q.price / q.pre_close - 1) * 100 if q.pre_close else 0
|
||||||
mkt = {31: "港股", 74: "美股"}.get(q.market, "")
|
mkt = {31: "港股", 74: "美股"}.get(q.market, "")
|
||||||
|
if q.market in (0, 1):
|
||||||
|
dp = _display_dp(q.market, q.code)
|
||||||
|
else:
|
||||||
|
dp = getattr(q, "decimal_point", 2) or 2
|
||||||
print("=" * 58)
|
print("=" * 58)
|
||||||
label = "%s %s" % (mkt, q.code) if mkt else "%s %s" % (q.code, q.server_time or datetime.now().strftime("%H:%M:%S"))
|
label = "%s %s" % (mkt, q.code) if mkt else "%s %s" % (q.code, q.server_time or datetime.now().strftime("%H:%M:%S"))
|
||||||
print(" %s" % label)
|
print(" %s" % label)
|
||||||
print(" 最新 %-10.2f 涨跌 %+8.2f (%+.2f%%)" % (q.price, q.price - q.pre_close, pct))
|
print((" 最新 %-10.*f 涨跌 %+8.*f (%+.2f%%)" % (dp, q.price, dp, q.price - q.pre_close, pct)))
|
||||||
print(" 今开 %-10.2f 最高 %.2f 最低 %.2f" % (q.open, q.high, q.low))
|
print((" 今开 %-10.*f 最高 %.*f 最低 %.*f" % (dp, q.open, dp, q.high, dp, q.low)))
|
||||||
print(" 昨收 %-10.2f 成交量 %s 成交额 %s" % (q.pre_close, _fmt_num(q.vol), _fmt_num(q.amount)))
|
print((" 昨收 %-10.*f 成交量 %s 成交额 %s" % (dp, q.pre_close, _fmt_num(q.vol), _fmt_num(q.amount))))
|
||||||
if q.bid and q.ask:
|
if q.bid and q.ask:
|
||||||
print(" " + "-" * 56)
|
print(" " + "-" * 56)
|
||||||
|
pf = _price_fmt(dp)
|
||||||
for i in range(max(len(q.bid), len(q.ask))):
|
for i in range(max(len(q.bid), len(q.ask))):
|
||||||
b = q.bid[i] if i < len(q.bid) else (0, 0)
|
b = q.bid[i] if i < len(q.bid) else (0, 0)
|
||||||
a = q.ask[i] if i < len(q.ask) else (0, 0)
|
a = q.ask[i] if i < len(q.ask) else (0, 0)
|
||||||
print(" 买%-1d %8.2f %10s | 卖%-1d %8.2f %10s" % (
|
print((" 买%-1d " + pf + " %10s | 卖%-1d " + pf + " %10s") % (
|
||||||
i + 1, b[0], _fmt_num(b[1]), i + 1, a[0], _fmt_num(a[1])))
|
i + 1, b[0], _fmt_num(b[1]), i + 1, a[0], _fmt_num(a[1])))
|
||||||
print()
|
print()
|
||||||
|
|
||||||
|
|
||||||
async def cmd_kline(td: TdxData, sym: str, period: KlinePeriod, count: int,
|
async def cmd_kline(td: TdxData, sym: str, period: KlinePeriod, count: int,
|
||||||
start: int, adjust: Adjust) -> None:
|
start: int, adjust: Adjust) -> None:
|
||||||
|
s = parse_symbol(sym)
|
||||||
|
pf = _price_fmt(_dp_of(s))
|
||||||
bars = await td.get_bars(sym, period, start=start, count=count, adjust=adjust)
|
bars = await td.get_bars(sym, period, start=start, count=count, adjust=adjust)
|
||||||
print("%s K线 (%s, %d根):" % (sym, period.name, len(bars)))
|
print("%s K线 (%s, %d根):" % (sym, period.name, len(bars)))
|
||||||
print(" %-12s %8s %8s %8s %8s %12s" % ("日期", "开盘", "最高", "最低", "收盘", "成交量"))
|
print(" %-12s %8s %8s %8s %8s %12s" % ("日期", "开盘", "最高", "最低", "收盘", "成交量"))
|
||||||
@@ -144,30 +187,38 @@ async def cmd_kline(td: TdxData, sym: str, period: KlinePeriod, count: int,
|
|||||||
dt = "%04d-%02d-%02d" % (b.year, b.month, b.day)
|
dt = "%04d-%02d-%02d" % (b.year, b.month, b.day)
|
||||||
if b.hour:
|
if b.hour:
|
||||||
dt += " %02d:%02d" % (b.hour, b.minute)
|
dt += " %02d:%02d" % (b.hour, b.minute)
|
||||||
print(" %-12s %8.2f %8.2f %8.2f %8.2f %12s" % (
|
print((" %-12s " + " ".join([pf] * 4) + " %12s") % (
|
||||||
dt, b.open, b.high, b.low, b.close, _fmt_num(b.vol)))
|
dt, b.open, b.high, b.low, b.close, _fmt_num(b.vol)))
|
||||||
|
|
||||||
|
|
||||||
async def cmd_transactions(td: TdxData, sym: str, count: int, date: int | None) -> None:
|
async def cmd_transactions(td: TdxData, sym: str, count: int, date: int | None) -> None:
|
||||||
s = parse_symbol(sym)
|
s = parse_symbol(sym)
|
||||||
tx = await td.get_transactions(Market(s.market), s.code, date=date, count=count)
|
pf = _price_fmt(_dp_of(s))
|
||||||
|
tx = await td.get_ticks(sym, date=date, count=count)
|
||||||
print("%s 逐笔成交 (%d条):" % (sym, len(tx)))
|
print("%s 逐笔成交 (%d条):" % (sym, len(tx)))
|
||||||
print(" 时间 价格 成交量 方向")
|
print(" 时间 价格 成交量 方向")
|
||||||
for t in tx:
|
for t in tx:
|
||||||
bs = "买" if getattr(t, "bs_flag", 0) == 1 else "卖" if getattr(t, "bs_flag", 0) == 2 else "-"
|
bs = "买" if getattr(t, "bs_flag", 0) == 1 else "卖" if getattr(t, "bs_flag", 0) == 2 else "-"
|
||||||
tc = getattr(t, "trade_count", 0)
|
if not getattr(t, "trade_count_available", True):
|
||||||
print(" %-10s %8.2f %10s %s%s" % (t.time, t.price, _fmt_num(t.vol), bs,
|
tc = " (-)" # 指数/ETF/LOF:协议不提供笔数
|
||||||
" (%d笔)" % tc if tc else ""))
|
elif getattr(t, "trade_count", 0):
|
||||||
|
tc = " (%d笔)" % t.trade_count
|
||||||
|
else:
|
||||||
|
tc = ""
|
||||||
|
print((" %-10s " + pf + " %10s %s%s") % (
|
||||||
|
t.time, t.price, _fmt_num(t.vol), bs, tc))
|
||||||
|
|
||||||
|
|
||||||
async def cmd_minute(td: TdxData, sym: str, date: int | None) -> None:
|
async def cmd_minute(td: TdxData, sym: str, date: int | None) -> None:
|
||||||
s = parse_symbol(sym)
|
s = parse_symbol(sym)
|
||||||
bars = await td.get_minute(Market(s.market), s.code, date=date)
|
pf = _price_fmt(_dp_of(s))
|
||||||
|
bars = await td.get_minutes(sym, date=date)
|
||||||
print("%s 分时 (%d条):" % (sym, len(bars)))
|
print("%s 分时 (%d条):" % (sym, len(bars)))
|
||||||
print(" 时间 价格 均价 成交量")
|
print(" 时间 价格 均价 成交量")
|
||||||
for i, b in enumerate(bars):
|
for b in bars:
|
||||||
print(" %02d:%02d %8.2f %8.2f %10s" % (
|
# 用 bar 自带时间:早期按序号推算会打成 00:00~03:59
|
||||||
i // 60, i % 60, b.price, b.avg_price, _fmt_num(b.vol)))
|
print((" %02d:%02d " + pf + " " + pf + " %10s") % (
|
||||||
|
b.hour, b.minute, b.price, b.avg_price, _fmt_num(b.vol)))
|
||||||
|
|
||||||
|
|
||||||
async def cmd_board(td: TdxData, args) -> None:
|
async def cmd_board(td: TdxData, args) -> None:
|
||||||
@@ -204,6 +255,43 @@ async def cmd_board(td: TdxData, args) -> None:
|
|||||||
print(" %-12s %+8.2f%%" % (r.get("name", ""), r.get("change_pct", 0)))
|
print(" %-12s %+8.2f%%" % (r.get("name", ""), r.get("change_pct", 0)))
|
||||||
|
|
||||||
|
|
||||||
|
async def cmd_index_stats(td: TdxData, index: str, date: int | None,
|
||||||
|
show_all: bool, top: int, concurrency: int) -> None:
|
||||||
|
"""指数成分成交汇总:每分钟成交量 / 笔数(笔数=成分股逐笔累加)。"""
|
||||||
|
total = 0
|
||||||
|
|
||||||
|
def on_progress(done: int, all_: int) -> None:
|
||||||
|
nonlocal total
|
||||||
|
total = all_
|
||||||
|
print("\r 拉取成分逐笔 %d/%d ..." % (done, all_), end="", file=sys.stderr,
|
||||||
|
flush=True)
|
||||||
|
|
||||||
|
st = await td.get_index_trade_stats(
|
||||||
|
index, date, concurrency=concurrency, on_progress=on_progress)
|
||||||
|
if total:
|
||||||
|
print("\r" + " " * 40 + "\r", end="", file=sys.stderr, flush=True)
|
||||||
|
|
||||||
|
print("%s 成分成交汇总" % st.index)
|
||||||
|
print(" 成分 %d 只 / 有成交 %d 只 / 有笔数 %d 只%s" % (
|
||||||
|
st.member_count, st.active_members, st.trade_count_members,
|
||||||
|
"" if st.trade_count_available else " ★该指数无笔数口径"))
|
||||||
|
print(" 成交量 %s 手 成交笔数 %s 笔 成交额 %.2f 亿 耗时 %.1fs" % (
|
||||||
|
_fmt_num(st.total_volume),
|
||||||
|
_fmt_num(st.total_trade_count) if st.trade_count_available else "-",
|
||||||
|
st.total_amount / 1e8, st.elapsed))
|
||||||
|
if st.failed_members:
|
||||||
|
print(" (%d 只成分逐笔拉取失败,已跳过)" % st.failed_members)
|
||||||
|
print()
|
||||||
|
print(" 时间 累计成交量 累计笔数")
|
||||||
|
rows = st.cumulative()
|
||||||
|
if not show_all:
|
||||||
|
rows = rows[-top:]
|
||||||
|
for m in rows:
|
||||||
|
print(" %-6s %12s %12s" % (
|
||||||
|
m.time, _fmt_num(m.volume),
|
||||||
|
_fmt_num(m.trade_count) if st.trade_count_available else "-"))
|
||||||
|
|
||||||
|
|
||||||
async def cmd_info(td: TdxData, sym: str) -> None:
|
async def cmd_info(td: TdxData, sym: str) -> None:
|
||||||
s = parse_symbol(sym)
|
s = parse_symbol(sym)
|
||||||
info = await td.get_symbol_info(Market(s.market), s.code)
|
info = await td.get_symbol_info(Market(s.market), s.code)
|
||||||
@@ -390,6 +478,17 @@ def main() -> None:
|
|||||||
ix.set_defaults(func=lambda td, a: cmd_kline(
|
ix.set_defaults(func=lambda td, a: cmd_kline(
|
||||||
td, a.code, KlinePeriod.DAY, a.count, 0, Adjust.NONE))
|
td, a.code, KlinePeriod.DAY, a.count, 0, Adjust.NONE))
|
||||||
|
|
||||||
|
ist = add("index-stats", "指数成分成交汇总(每分钟成交量/笔数)")
|
||||||
|
ist.add_argument("index", help="指数代码,如 000300 / 000001 / sh000016")
|
||||||
|
ist.add_argument("--date", type=str, default=None, help="YYYYMMDD,缺省=当日")
|
||||||
|
ist.add_argument("--top", type=int, default=10, help="只显示最后 N 分钟(默认10)")
|
||||||
|
ist.add_argument("--all", dest="show_all", action="store_true",
|
||||||
|
help="显示全部分钟")
|
||||||
|
ist.add_argument("--concurrency", type=int, default=24, help="逐笔并发数")
|
||||||
|
ist.set_defaults(func=lambda td, a: cmd_index_stats(
|
||||||
|
td, a.index, int(a.date) if a.date else None,
|
||||||
|
a.show_all, a.top, a.concurrency))
|
||||||
|
|
||||||
b = add("board", "板块")
|
b = add("board", "板块")
|
||||||
b.add_argument("action", choices=["list", "members", "ranking", "change"])
|
b.add_argument("action", choices=["list", "members", "ranking", "change"])
|
||||||
b.add_argument("symbol", nargs="?", default="")
|
b.add_argument("symbol", nargs="?", default="")
|
||||||
|
|||||||
@@ -17,13 +17,19 @@
|
|||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
import struct
|
import struct
|
||||||
|
import zipfile
|
||||||
|
|
||||||
from ..models import FinancialFileInfo, FinancialRecord
|
from ..models import FinancialFileInfo, FinancialRecord
|
||||||
|
|
||||||
_HEADER_FMT = "<1hI1H3L"
|
_HEADER_FMT = "<1hI1H3L"
|
||||||
_INDEX_FMT = "<6sBL"
|
_INDEX_FMT = "<6sBL"
|
||||||
|
|
||||||
|
_HEADER_SIZE = struct.calcsize(_HEADER_FMT)
|
||||||
|
_INDEX_SIZE = struct.calcsize(_INDEX_FMT)
|
||||||
|
_ZIP_MAGIC = b"PK\x03\x04"
|
||||||
|
|
||||||
|
|
||||||
def parse_financial_file_list(data: bytes) -> list[FinancialFileInfo]:
|
def parse_financial_file_list(data: bytes) -> list[FinancialFileInfo]:
|
||||||
"""解析 tdxfin/gpcw.txt 内容(每行 ``filename,md5hash,filesize``)。"""
|
"""解析 tdxfin/gpcw.txt 内容(每行 ``filename,md5hash,filesize``)。"""
|
||||||
@@ -46,8 +52,58 @@ def parse_financial_file_list(data: bytes) -> list[FinancialFileInfo]:
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def looks_like_gpcw_dat(data: bytes) -> bool:
|
||||||
|
"""粗略判定字节流是否为 gpcw ``.dat``(校验 20 字节 header 的自洽性)。
|
||||||
|
|
||||||
|
用于把「真的取到了财报数据」与「服务器对不存在的文件名回的 0/1 字节空壳」区分开:
|
||||||
|
空壳要么长度不足、要么 header 全零,一律判否。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
data: 待判定的字节流。
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
可疑时返回 True。
|
||||||
|
"""
|
||||||
|
if len(data) < _HEADER_SIZE:
|
||||||
|
return False
|
||||||
|
_unknown, _date, max_count, _u1, report_size, _u2 = struct.unpack(
|
||||||
|
_HEADER_FMT, data[:_HEADER_SIZE]
|
||||||
|
)
|
||||||
|
if max_count <= 0 or report_size <= 0 or report_size % 4 != 0:
|
||||||
|
return False
|
||||||
|
# 索引区必须放得下 max_count 条 11 字节条目
|
||||||
|
return _HEADER_SIZE + max_count * _INDEX_SIZE <= len(data)
|
||||||
|
|
||||||
|
|
||||||
|
def extract_gpcw_dat(data: bytes) -> bytes:
|
||||||
|
"""从「zip 包」或「未压缩 dat」中取出 ``.dat`` 字节;无法识别时返回 ``b""``。
|
||||||
|
|
||||||
|
服务器**实际只提供未压缩的** ``tdxfin/gpcwYYYYMMDD.dat``,而 ``tdxfin/gpcw.txt``
|
||||||
|
索引里写的是 ``gpcwYYYYMMDD.zip``,因此两种形态都要能走通。
|
||||||
|
|
||||||
|
Args:
|
||||||
|
data: zip 包字节或未压缩 ``.dat`` 字节。
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
内层 / 原始的 ``.dat`` 字节;不是有效财报数据时返回 ``b""``。
|
||||||
|
"""
|
||||||
|
if not data:
|
||||||
|
return b""
|
||||||
|
if data.startswith(_ZIP_MAGIC):
|
||||||
|
try:
|
||||||
|
with zipfile.ZipFile(io.BytesIO(data)) as zf:
|
||||||
|
names = [n for n in zf.namelist() if n.lower().endswith(".dat")]
|
||||||
|
if not names:
|
||||||
|
return b""
|
||||||
|
inner = zf.read(names[0])
|
||||||
|
except (zipfile.BadZipFile, OSError, KeyError):
|
||||||
|
return b""
|
||||||
|
return inner if looks_like_gpcw_dat(inner) else b""
|
||||||
|
return data if looks_like_gpcw_dat(data) else b""
|
||||||
|
|
||||||
|
|
||||||
def parse_financial_dat(data: bytes, report_date: int = 0) -> list[FinancialRecord]:
|
def parse_financial_dat(data: bytes, report_date: int = 0) -> list[FinancialRecord]:
|
||||||
"""解析 gpcw*.zip 内的 .dat 二进制。
|
"""解析 gpcw ``.dat`` 二进制(zip 内的或服务器直接给的未压缩形式都行)。
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
data: .dat 文件完整字节。
|
data: .dat 文件完整字节。
|
||||||
@@ -56,7 +112,7 @@ def parse_financial_dat(data: bytes, report_date: int = 0) -> list[FinancialReco
|
|||||||
Returns:
|
Returns:
|
||||||
FinancialRecord 列表;数据不足或字段非法时返回空列表。
|
FinancialRecord 列表;数据不足或字段非法时返回空列表。
|
||||||
"""
|
"""
|
||||||
header_size = struct.calcsize(_HEADER_FMT)
|
header_size = _HEADER_SIZE
|
||||||
if len(data) < header_size:
|
if len(data) < header_size:
|
||||||
return []
|
return []
|
||||||
|
|
||||||
@@ -72,7 +128,7 @@ def parse_financial_dat(data: bytes, report_date: int = 0) -> list[FinancialReco
|
|||||||
if num_fields <= 0:
|
if num_fields <= 0:
|
||||||
return []
|
return []
|
||||||
|
|
||||||
index_size = struct.calcsize(_INDEX_FMT)
|
index_size = _INDEX_SIZE
|
||||||
report_fmt = f"<{num_fields}f"
|
report_fmt = f"<{num_fields}f"
|
||||||
report_pack_size = struct.calcsize(report_fmt)
|
report_pack_size = struct.calcsize(report_fmt)
|
||||||
|
|
||||||
|
|||||||
+191
-13
@@ -1,12 +1,28 @@
|
|||||||
"""服务器清单与默认配置
|
"""服务器清单与默认配置
|
||||||
|
|
||||||
服务器清单存于包内 servers.json,由 server_probe.py 导入时自动探测可用性与延时。
|
服务器清单存于包内 ``servers.json`` —— 它是**随包分发的基线清单,运行期只读**。
|
||||||
config.py 加载时只采用 status="ok" 的服务器;环境变量可覆盖;代码内嵌清单为兜底。
|
每日首次启动的后台测速结果写入**用户级缓存目录**,运行期优先采用当日实测结果。
|
||||||
|
|
||||||
|
主机选取优先级(每个协议独立)::
|
||||||
|
|
||||||
|
环境变量 PYPYTDX_DATA_*_HOSTS > 当日测速缓存 > 包内清单 fast_pool
|
||||||
|
> 包内清单 ok > 代码内嵌常量
|
||||||
|
|
||||||
|
相关环境变量:
|
||||||
|
|
||||||
|
- ``PYPYTDX_DATA_CACHE_DIR`` 覆盖用户级缓存目录(默认见 :func:`probe_cache_dir`)
|
||||||
|
- ``PYPYTDX_DATA_DISABLE_PROBE`` 置真值时既不读当日缓存也不测速(CI / 测试用,保证确定性)
|
||||||
|
|
||||||
|
为什么不把测速结果写回包内 ``servers.json``:安装目录可能是只读的(容器 / 系统级
|
||||||
|
site-packages / pipx),且多用户共用一份安装时彼此覆盖,运行期改写分发数据本身也不妥。
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import ipaddress
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import sys
|
||||||
|
from datetime import date
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
DEFAULT_PORT = 7709
|
DEFAULT_PORT = 7709
|
||||||
@@ -14,19 +30,30 @@ DEFAULT_TIMEOUT = 10.0
|
|||||||
|
|
||||||
_SERVERS_FILE = Path(__file__).resolve().parent / "servers.json" # 包内清单(随包分发)
|
_SERVERS_FILE = Path(__file__).resolve().parent / "servers.json" # 包内清单(随包分发)
|
||||||
|
|
||||||
|
# 当日测速缓存(结构变更时递增,旧版本缓存自动失效)
|
||||||
|
PROBE_CACHE_VERSION = 1
|
||||||
|
_PROBE_CACHE_NAME = "hosts-today.json"
|
||||||
|
_DECIMAL_CACHE_NAME = "decimal-points.json"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 包内清单(只读基线) ----------
|
||||||
|
def _read_servers_json() -> dict:
|
||||||
|
"""读取包内清单;文件缺失 / 内容损坏 / 顶层非对象 → 空 dict。"""
|
||||||
|
try:
|
||||||
|
cfg = json.loads(_SERVERS_FILE.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return {}
|
||||||
|
return cfg if isinstance(cfg, dict) else {}
|
||||||
|
|
||||||
|
|
||||||
def _load_ok_hosts(protocol: str, *, fast_only: bool = False) -> list[str]:
|
def _load_ok_hosts(protocol: str, *, fast_only: bool = False) -> list[str]:
|
||||||
"""从 servers.json 加载 status=ok 且支持指定协议的服务器。
|
"""从 servers.json 加载 status=ok 且支持指定协议的服务器。
|
||||||
|
|
||||||
fast_only=True 时只返回当日测速入池(fast_pool)的服务器(排名前N或延迟<阈值),
|
fast_only=True 时只返回测速入池(fast_pool)的服务器(排名前N或延迟<阈值),
|
||||||
用于动态连接池优先使用优质节点。
|
用于动态连接池优先使用优质节点。
|
||||||
"""
|
"""
|
||||||
try:
|
|
||||||
cfg = json.loads(_SERVERS_FILE.read_text(encoding="utf-8"))
|
|
||||||
except (OSError, ValueError):
|
|
||||||
return []
|
|
||||||
out = []
|
out = []
|
||||||
for s in cfg.get("servers", []):
|
for s in _read_servers_json().get("servers", []):
|
||||||
if s.get("status") != "ok":
|
if s.get("status") != "ok":
|
||||||
continue
|
continue
|
||||||
if protocol not in s.get("protocols", []):
|
if protocol not in s.get("protocols", []):
|
||||||
@@ -38,13 +65,102 @@ def _load_ok_hosts(protocol: str, *, fast_only: bool = False) -> list[str]:
|
|||||||
|
|
||||||
|
|
||||||
def _load_ex_handshakes() -> dict[str, str]:
|
def _load_ex_handshakes() -> dict[str, str]:
|
||||||
try:
|
"""包内清单里 EX 服务器的握手模式。"""
|
||||||
cfg = json.loads(_SERVERS_FILE.read_text(encoding="utf-8"))
|
return {s["ip"]: s.get("handshake", "setup")
|
||||||
except (OSError, ValueError):
|
for s in _read_servers_json().get("servers", [])
|
||||||
return {}
|
|
||||||
return {s["ip"]: s.get("handshake", "setup") for s in cfg.get("servers", [])
|
|
||||||
if "ex" in s.get("protocols", []) and s.get("status") == "ok"}
|
if "ex" in s.get("protocols", []) and s.get("status") == "ok"}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 用户级当日测速缓存 ----------
|
||||||
|
def probe_cache_dir() -> Path:
|
||||||
|
"""用户级缓存目录(保证可写,绝不指向安装目录)。
|
||||||
|
|
||||||
|
``PYPYTDX_DATA_CACHE_DIR`` 可覆盖(测试指向临时目录)。按平台约定:
|
||||||
|
|
||||||
|
- Windows: ``%LOCALAPPDATA%\\pytdxdata``
|
||||||
|
- macOS: ``~/Library/Caches/pytdxdata``
|
||||||
|
- 其他: ``$XDG_CACHE_HOME/pytdxdata`` 或 ``~/.cache/pytdxdata``
|
||||||
|
"""
|
||||||
|
env = (os.environ.get("PYPYTDX_DATA_CACHE_DIR") or "").strip()
|
||||||
|
if env:
|
||||||
|
return Path(env).expanduser()
|
||||||
|
if os.name == "nt":
|
||||||
|
base = os.environ.get("LOCALAPPDATA") or os.environ.get("APPDATA")
|
||||||
|
return (Path(base) if base else Path.home() / "AppData" / "Local") / "pytdxdata"
|
||||||
|
if sys.platform == "darwin":
|
||||||
|
return Path.home() / "Library" / "Caches" / "pytdxdata"
|
||||||
|
base = os.environ.get("XDG_CACHE_HOME")
|
||||||
|
return (Path(base) if base else Path.home() / ".cache") / "pytdxdata"
|
||||||
|
|
||||||
|
|
||||||
|
def probe_cache_file() -> Path:
|
||||||
|
"""当日测速缓存文件路径(每次调用重新解析,便于测试改环境变量后立即生效)。"""
|
||||||
|
return probe_cache_dir() / _PROBE_CACHE_NAME
|
||||||
|
|
||||||
|
|
||||||
|
def decimal_cache_file() -> Path:
|
||||||
|
"""权威价格小数位表缓存路径(每次调用重新解析,便于测试改环境变量后生效)。"""
|
||||||
|
return probe_cache_dir() / _DECIMAL_CACHE_NAME
|
||||||
|
|
||||||
|
|
||||||
|
def decimal_sync_disabled() -> bool:
|
||||||
|
"""是否已通过 ``PYPYTDX_DATA_DISABLE_DECIMAL_SYNC`` 关闭小数位后台同步。"""
|
||||||
|
return os.environ.get("PYPYTDX_DATA_DISABLE_DECIMAL_SYNC", "").strip().lower() in {
|
||||||
|
"1", "true", "yes", "on",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def probe_disabled() -> bool:
|
||||||
|
"""是否已通过 ``PYPYTDX_DATA_DISABLE_PROBE`` 关闭当日测速缓存。"""
|
||||||
|
return os.environ.get("PYPYTDX_DATA_DISABLE_PROBE", "").strip().lower() in {
|
||||||
|
"1", "true", "yes", "on",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_probe_cache(today: str | None = None,
|
||||||
|
path: str | Path | None = None) -> dict | None:
|
||||||
|
"""读取**当日**测速缓存;缺失 / 损坏 / 非今日 / 版本不符 → ``None``。
|
||||||
|
|
||||||
|
返回非 None 即表示"今天已经测过速了",调用方应直接复用而不再刷新。
|
||||||
|
``path`` 可显式指定缓存文件(默认 :func:`probe_cache_file`),便于注入与测试。
|
||||||
|
"""
|
||||||
|
if probe_disabled():
|
||||||
|
return None
|
||||||
|
target = Path(path) if path is not None else probe_cache_file()
|
||||||
|
try:
|
||||||
|
cfg = json.loads(target.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return None
|
||||||
|
if not isinstance(cfg, dict) or cfg.get("version") != PROBE_CACHE_VERSION:
|
||||||
|
return None
|
||||||
|
if str(cfg.get("date", "")) != (today or date.today().isoformat()):
|
||||||
|
return None
|
||||||
|
results = cfg.get("results")
|
||||||
|
if not isinstance(results, dict) or not results:
|
||||||
|
return None
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
|
||||||
|
def _load_cached_hosts(protocol: str) -> list[str]:
|
||||||
|
"""当日测速缓存中该协议**实测可达**的主机,按延迟升序(越快越靠前)。"""
|
||||||
|
cfg = load_probe_cache()
|
||||||
|
if cfg is None:
|
||||||
|
return []
|
||||||
|
rows: list[tuple[float, str]] = []
|
||||||
|
for ip, per_proto in cfg["results"].items():
|
||||||
|
if not isinstance(per_proto, dict):
|
||||||
|
continue
|
||||||
|
lat = per_proto.get(protocol)
|
||||||
|
if isinstance(lat, bool) or not isinstance(lat, (int, float)):
|
||||||
|
continue
|
||||||
|
if lat < 0:
|
||||||
|
continue
|
||||||
|
rows.append((float(lat), str(ip)))
|
||||||
|
rows.sort(key=lambda kv: (kv[0], kv[1]))
|
||||||
|
return [ip for _, ip in rows]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 代码内嵌兜底清单 ----------
|
||||||
# 标准行情服务器 —— 真实K线探测验证可用(2026-08-12, GetSecurityBarsCmd 600000 日K)
|
# 标准行情服务器 —— 真实K线探测验证可用(2026-08-12, GetSecurityBarsCmd 600000 日K)
|
||||||
# 27台可用 = pytdxwrapper券商服务器(国泰君安/中信短线宝/华林/安信等) + 通达信官方主站
|
# 27台可用 = pytdxwrapper券商服务器(国泰君安/中信短线宝/华林/安信等) + 通达信官方主站
|
||||||
# 来源①: pytdxwrapper config.py TDX_HOSTS(券商+官方, 51台探测27可用)
|
# 来源①: pytdxwrapper config.py TDX_HOSTS(券商+官方, 51台探测27可用)
|
||||||
@@ -91,10 +207,66 @@ EX_HANDSHAKES: dict[str, str] = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 测速候选枚举 ----------
|
||||||
|
def is_valid_host(host: object) -> bool:
|
||||||
|
"""是否是可用作服务器地址的合法主机(IPv4/IPv6 字面量,或含点的域名)。
|
||||||
|
|
||||||
|
用于挡住测试 mock 混进来的伪主机名(如 ``"s1"``)被写进缓存 / 清单。
|
||||||
|
"""
|
||||||
|
if not isinstance(host, str):
|
||||||
|
return False
|
||||||
|
h = host.strip()
|
||||||
|
if not h or len(h) > 253 or any(c.isspace() for c in h):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
ipaddress.ip_address(h)
|
||||||
|
return True
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
return "." in h and all(
|
||||||
|
part and part[0] != "-" and part[-1] != "-" and part.replace("-", "").isalnum()
|
||||||
|
for part in h.split(".")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def all_probe_candidates() -> dict[str, list[str]]:
|
||||||
|
"""[测速用] 返回 ``{协议: [候选主机]}`` 全量清单(已校验、去重、保序)。
|
||||||
|
|
||||||
|
来源 = 包内清单申报的协议 ∪ 代码内嵌常量 ∪ 环境变量覆盖。
|
||||||
|
|
||||||
|
与 ``get_*_hosts()`` 的关键区别:这里**不做 status / 延迟过滤**。
|
||||||
|
测速必须能看到全部候选,否则"上次慢、这次快"的节点永远捞不回来,
|
||||||
|
形成自指循环(只重测已在池里的少数节点,其余记录一直过期)。
|
||||||
|
"""
|
||||||
|
out: dict[str, list[str]] = {"standard": [], "mac": [], "ex": []}
|
||||||
|
for s in _read_servers_json().get("servers", []):
|
||||||
|
ip = s.get("ip")
|
||||||
|
for protocol in s.get("protocols", []):
|
||||||
|
if protocol in out:
|
||||||
|
out[protocol].append(ip)
|
||||||
|
out["standard"] += STANDARD_HOSTS
|
||||||
|
out["mac"] += MAC_HOSTS
|
||||||
|
out["ex"] += EX_HOSTS
|
||||||
|
for protocol, var in (("standard", "PYPYTDX_DATA_STANDARD_HOSTS"),
|
||||||
|
("mac", "PYPYTDX_DATA_MAC_HOSTS"),
|
||||||
|
("ex", "PYPYTDX_DATA_EX_HOSTS")):
|
||||||
|
env = os.environ.get(var)
|
||||||
|
if env:
|
||||||
|
out[protocol] += env.split(",")
|
||||||
|
return {
|
||||||
|
protocol: [h for h in dict.fromkeys(hosts) if is_valid_host(h)]
|
||||||
|
for protocol, hosts in out.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 对外取主机 ----------
|
||||||
def get_standard_hosts() -> list[str]:
|
def get_standard_hosts() -> list[str]:
|
||||||
env = os.environ.get("PYPYTDX_DATA_STANDARD_HOSTS")
|
env = os.environ.get("PYPYTDX_DATA_STANDARD_HOSTS")
|
||||||
if env:
|
if env:
|
||||||
return [h.strip() for h in env.split(",") if h.strip()]
|
return [h.strip() for h in env.split(",") if h.strip()]
|
||||||
|
cached = _load_cached_hosts("standard")
|
||||||
|
if cached:
|
||||||
|
return cached
|
||||||
fast = _load_ok_hosts("standard", fast_only=True)
|
fast = _load_ok_hosts("standard", fast_only=True)
|
||||||
if fast:
|
if fast:
|
||||||
return fast
|
return fast
|
||||||
@@ -106,6 +278,9 @@ def get_mac_hosts() -> list[str]:
|
|||||||
env = os.environ.get("PYPYTDX_DATA_MAC_HOSTS")
|
env = os.environ.get("PYPYTDX_DATA_MAC_HOSTS")
|
||||||
if env:
|
if env:
|
||||||
return [h.strip() for h in env.split(",") if h.strip()]
|
return [h.strip() for h in env.split(",") if h.strip()]
|
||||||
|
cached = _load_cached_hosts("mac")
|
||||||
|
if cached:
|
||||||
|
return cached
|
||||||
fast = _load_ok_hosts("mac", fast_only=True)
|
fast = _load_ok_hosts("mac", fast_only=True)
|
||||||
if fast:
|
if fast:
|
||||||
return fast
|
return fast
|
||||||
@@ -117,6 +292,9 @@ def get_ex_hosts() -> list[str]:
|
|||||||
env = os.environ.get("PYPYTDX_DATA_EX_HOSTS")
|
env = os.environ.get("PYPYTDX_DATA_EX_HOSTS")
|
||||||
if env:
|
if env:
|
||||||
return [h.strip() for h in env.split(",") if h.strip()]
|
return [h.strip() for h in env.split(",") if h.strip()]
|
||||||
|
cached = _load_cached_hosts("ex")
|
||||||
|
if cached:
|
||||||
|
return cached
|
||||||
from_json = _load_ok_hosts("ex")
|
from_json = _load_ok_hosts("ex")
|
||||||
return from_json or list(EX_HOSTS)
|
return from_json or list(EX_HOSTS)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,223 @@
|
|||||||
|
"""权威价格小数位表的同步与缓存(对用户无感)。
|
||||||
|
|
||||||
|
为什么需要它
|
||||||
|
------------
|
||||||
|
``get_quotes`` 解析报价时,协议给的是**整数**,真实价格 = ``整数 / 10**小数位``。
|
||||||
|
小数位在协议里**不是**随报价一起下发的,只能靠推断(见
|
||||||
|
:func:`pytdxdata.protocol.commands.quotes.price_decimal_digits`)。
|
||||||
|
|
||||||
|
但证券清单接口(``get_universe``,底层 ``GetSecurityListCmd``)**自带**这个字段:
|
||||||
|
``SecurityInfo.decimal_point``,同时还带 ``pre_close``。这就是权威来源。本模块把它
|
||||||
|
灌进 ``price_decimal_digits`` 使用的运行时表,使价格标度不再依赖硬编码前缀规则。
|
||||||
|
|
||||||
|
行为
|
||||||
|
----
|
||||||
|
- **当日首次启动**检查缓存文件(``decimal-points.json``):命中就直接装表(纯内存,
|
||||||
|
无网络、约 10 ms);未命中则挂一个**后台 task** 拉证券清单并写缓存,
|
||||||
|
当天不再重复。
|
||||||
|
- 后台 task 不阻塞 ``start()``,不打印、不抛异常(取消除外),失败只落 debug 日志。
|
||||||
|
- 未同步成功时 ``price_decimal_digits`` 静默退回静态前缀规则,行为与 0.6.5 一致。
|
||||||
|
- 证券清单本身已被 ``cache/`` 层缓存,因此同步的实际网络开销只在**首次**
|
||||||
|
(约 26 + 25 个分页请求)。
|
||||||
|
|
||||||
|
开关
|
||||||
|
----
|
||||||
|
- ``PYPYTDX_DATA_DISABLE_DECIMAL_SYNC=1`` 关闭后台同步(CI / 测试用,保证确定性)。
|
||||||
|
- ``PYPYTDX_DATA_CACHE_DIR`` 覆盖缓存目录(测试指向临时目录)。
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
from datetime import date
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
|
from .config import decimal_cache_file, decimal_sync_disabled
|
||||||
|
from .protocol.commands.quotes import (
|
||||||
|
clear_decimal_map,
|
||||||
|
decimal_map_source,
|
||||||
|
decimal_map_stats,
|
||||||
|
install_decimal_map,
|
||||||
|
)
|
||||||
|
|
||||||
|
if TYPE_CHECKING: # pragma: no cover
|
||||||
|
from .api import TdxData
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
#: 同步哪些市场(只对 A 股有意义:扩展市场报价本身不含标度歧义)
|
||||||
|
DEFAULT_MARKETS: tuple[str, ...] = ("sh", "sz")
|
||||||
|
|
||||||
|
_CN_NAME = {"sh": 1, "sz": 0}
|
||||||
|
|
||||||
|
_state: dict[str, Any] = {"loaded": "", "task": None}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------- #
|
||||||
|
# 磁盘缓存
|
||||||
|
# ---------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def cache_file() -> Path:
|
||||||
|
"""缓存文件路径(每次重新解析,便于测试改环境变量)。"""
|
||||||
|
return decimal_cache_file()
|
||||||
|
|
||||||
|
|
||||||
|
def _read_cache(path: Path | None = None) -> dict[str, dict[str, int]] | None:
|
||||||
|
"""读缓存 → ``{"1": {code: dp}}``;缺失 / 损坏 → ``None``。"""
|
||||||
|
target = Path(path) if path is not None else cache_file()
|
||||||
|
try:
|
||||||
|
raw = json.loads(target.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return None
|
||||||
|
if not isinstance(raw, dict):
|
||||||
|
return None
|
||||||
|
if raw.get("date") != date.today().isoformat():
|
||||||
|
return None
|
||||||
|
markets = raw.get("markets")
|
||||||
|
if not isinstance(markets, dict) or not markets:
|
||||||
|
return None
|
||||||
|
out: dict[str, dict[str, int]] = {}
|
||||||
|
for k, v in markets.items():
|
||||||
|
if not isinstance(v, dict):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
out[str(int(k))] = {str(c): int(d) for c, d in v.items() if int(d) > 0}
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
return out or None
|
||||||
|
|
||||||
|
|
||||||
|
def load_cached(path: Path | None = None) -> int:
|
||||||
|
"""把当日缓存装进权威表;成功返回覆盖条目数,否则 0。"""
|
||||||
|
if decimal_sync_disabled():
|
||||||
|
return 0
|
||||||
|
markets = _read_cache(path)
|
||||||
|
if not markets:
|
||||||
|
return 0
|
||||||
|
items = [(int(m), c, d) for m, codes in markets.items() for c, d in codes.items()]
|
||||||
|
stats = install_decimal_map(items, source="cache")
|
||||||
|
_state["loaded"] = date.today().isoformat()
|
||||||
|
log.debug("小数位表已从缓存加载: %s", stats)
|
||||||
|
return int(stats["stock"])
|
||||||
|
|
||||||
|
|
||||||
|
def _write_cache(markets: dict[str, dict[str, int]], path: Path | None = None) -> None:
|
||||||
|
"""原子写缓存(tmp + ``os.replace``),失败只落 debug 日志。"""
|
||||||
|
target = Path(path) if path is not None else cache_file()
|
||||||
|
payload = {"date": date.today().isoformat(), "markets": markets}
|
||||||
|
tmp = target.with_suffix(".tmp")
|
||||||
|
try:
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
tmp.write_text(
|
||||||
|
json.dumps(payload, ensure_ascii=False, separators=(",", ":")),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
os.replace(tmp, target)
|
||||||
|
except OSError as e: # pragma: no cover - 磁盘不可写等
|
||||||
|
log.debug("小数位表缓存写入失败: %s", e)
|
||||||
|
try:
|
||||||
|
tmp.unlink(missing_ok=True)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------- #
|
||||||
|
# 同步
|
||||||
|
# ---------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
async def refresh(td: "TdxData", markets: tuple[str, ...] = DEFAULT_MARKETS,
|
||||||
|
*, path: Path | None = None) -> dict[str, int]:
|
||||||
|
"""拉证券清单 → 装表 + 写缓存。返回统计。
|
||||||
|
|
||||||
|
证券清单走 ``get_universe``(含 ``cache/`` 层缓存),只有首次真正发请求。
|
||||||
|
"""
|
||||||
|
tables: dict[str, dict[str, int]] = {}
|
||||||
|
items: list[tuple[int, str, int]] = []
|
||||||
|
for name in markets:
|
||||||
|
m = _CN_NAME.get(name)
|
||||||
|
if m is None:
|
||||||
|
continue
|
||||||
|
rows = await td.get_universe(name)
|
||||||
|
table: dict[str, int] = {}
|
||||||
|
for it in rows:
|
||||||
|
dp = int(getattr(it, "decimal_point", 0) or 0)
|
||||||
|
code = str(getattr(it, "code", "") or "").strip().rstrip("\x00")
|
||||||
|
if dp <= 0 or not code:
|
||||||
|
continue
|
||||||
|
table[code] = dp
|
||||||
|
items.append((m, code, dp))
|
||||||
|
if table:
|
||||||
|
tables[str(m)] = table
|
||||||
|
|
||||||
|
stats = install_decimal_map(items, source="universe")
|
||||||
|
_state["loaded"] = date.today().isoformat()
|
||||||
|
if tables:
|
||||||
|
_write_cache(tables, path)
|
||||||
|
log.debug("小数位表同步完成: %s", stats)
|
||||||
|
return stats
|
||||||
|
|
||||||
|
|
||||||
|
async def _run(td: "TdxData") -> dict[str, int]:
|
||||||
|
try:
|
||||||
|
return await refresh(td)
|
||||||
|
except asyncio.CancelledError:
|
||||||
|
raise
|
||||||
|
except Exception as e: # pragma: no cover - 网络/解析异常都不该影响用户
|
||||||
|
log.debug("小数位表后台同步失败: %s", e)
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def schedule(td: "TdxData") -> asyncio.Task | None:
|
||||||
|
"""[内部] 需要时把同步挂到后台并返回 Task(调用方持有 / 取消)。
|
||||||
|
|
||||||
|
- 已关闭(``PYPYTDX_DATA_DISABLE_DECIMAL_SYNC=1``)→ ``None``
|
||||||
|
- 当日缓存命中 → 立即装表(同步、无网络),返回 ``None``
|
||||||
|
- 否则创建后台 task;进程内单例,重复调用复用同一个 task
|
||||||
|
"""
|
||||||
|
if decimal_sync_disabled():
|
||||||
|
return None
|
||||||
|
if decimal_map_stats()["stock"] and _state["loaded"] == date.today().isoformat():
|
||||||
|
return None
|
||||||
|
if load_cached():
|
||||||
|
return None
|
||||||
|
task = _state.get("task")
|
||||||
|
if task is not None and not task.done():
|
||||||
|
return task
|
||||||
|
task = asyncio.get_running_loop().create_task(_run(td), name="pytdxdata-decimal-sync")
|
||||||
|
_state["task"] = task
|
||||||
|
return task
|
||||||
|
|
||||||
|
|
||||||
|
async def cancel(task: asyncio.Task | None) -> None:
|
||||||
|
"""取消并等待后台同步 task(供 ``TdxData.close()`` 调用)。"""
|
||||||
|
if task is None or task.done():
|
||||||
|
return
|
||||||
|
task.cancel()
|
||||||
|
try:
|
||||||
|
await task
|
||||||
|
except (asyncio.CancelledError, Exception): # noqa: BLE001 - 关闭路径不抛
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def status() -> dict[str, Any]:
|
||||||
|
"""当前状态快照(排障 / 测试用)。"""
|
||||||
|
task = _state.get("task")
|
||||||
|
return {
|
||||||
|
"loaded": _state["loaded"],
|
||||||
|
"source": decimal_map_source(),
|
||||||
|
"syncing": bool(task is not None and not task.done()),
|
||||||
|
**decimal_map_stats(),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def reset() -> None:
|
||||||
|
"""清空内存状态(测试用;不动磁盘缓存)。"""
|
||||||
|
task = _state.get("task")
|
||||||
|
if task is not None and not task.done():
|
||||||
|
task.cancel()
|
||||||
|
_state["task"] = None
|
||||||
|
_state["loaded"] = ""
|
||||||
|
clear_decimal_map()
|
||||||
@@ -3,7 +3,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import struct
|
import struct
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from datetime import datetime
|
from datetime import date, datetime, timedelta
|
||||||
from enum import IntEnum
|
from enum import IntEnum
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
@@ -92,17 +92,33 @@ class Adjust(IntEnum):
|
|||||||
HFQ = 2 # 后复权
|
HFQ = 2 # 后复权
|
||||||
|
|
||||||
|
|
||||||
|
# 扩展市场中「时间戳跨时区拼接」的市场号:通达信 EX 通道对美股返回的分钟级 K线
|
||||||
|
# 里,日期字段是**美东交易日**、时分字段是**北京时间**,两个时区被拼进同一个
|
||||||
|
# 五元组,必须归一化(见 SecurityBar.__post_init__)。
|
||||||
|
_US_EX_MARKETS: frozenset[int] = frozenset({int(ExMarket.US_STOCK)})
|
||||||
|
|
||||||
|
# 归一化判据:北京时间 hour >= 12 的 bar 与美东交易日属同一自然日。美股常规时段
|
||||||
|
# 09:30~16:00 EDT = 北京 21:30~次日 04:00(冬令时 22:30~次日 05:00),夏令时/
|
||||||
|
# 冬令时下分界都稳定落在 12 点。
|
||||||
|
_US_DAY_SPLIT_HOUR = 12
|
||||||
|
|
||||||
|
|
||||||
@dataclass(slots=True)
|
@dataclass(slots=True)
|
||||||
class SecurityBar:
|
class SecurityBar:
|
||||||
"""K线记录(价格单位:元)。跨市场统一模型(A股/港股/美股/期货共用)。"""
|
"""K线记录(价格单位:元)。跨市场统一模型(A股/港股/美股/期货共用)。
|
||||||
|
|
||||||
|
时间戳口径**统一为北京时间**:A股/港股/期货原始即为北京时间;美股原始是
|
||||||
|
「日期=美东交易日 + 时分=北京时间」的混合值,构造时已还原为真实北京时间
|
||||||
|
(见 ``__post_init__``)。该 bar 归属的交易日见 :attr:`trading_date`。
|
||||||
|
"""
|
||||||
market: int
|
market: int
|
||||||
code: str
|
code: str
|
||||||
open: float
|
open: float
|
||||||
high: float
|
high: float
|
||||||
low: float
|
low: float
|
||||||
close: float
|
close: float
|
||||||
vol: float # 股票=股 / 转债=手(与vipdoc一致)
|
vol: float # 股票=股 / 转债=手(1 手=10 张,与 vipdoc 一致)
|
||||||
amount: float # 元
|
amount: float # 元。⚠️ amount/vol 只在股票上等于均价;转债还要再 /10(手=10张)
|
||||||
year: int
|
year: int
|
||||||
month: int
|
month: int
|
||||||
day: int
|
day: int
|
||||||
@@ -110,10 +126,40 @@ class SecurityBar:
|
|||||||
minute: int = 0
|
minute: int = 0
|
||||||
symbol: str = "" # 规范化标的串(sz000001 / hk00700),由 API 层填充
|
symbol: str = "" # 规范化标的串(sz000001 / hk00700),由 API 层填充
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
"""美股(EX 通道)时间戳归一化:还原成真实北京时间。
|
||||||
|
|
||||||
|
原始报文里 ``zipday`` 是**美东交易日**、``tminutes`` 是**北京时间分钟数**,
|
||||||
|
直拼出的 ``datetime`` 会把同一交易日的下午半场(北京 00:00~04:00)排到
|
||||||
|
上午半场(北京 21:30~23:59)**之前** —— 单日 390 根全部错位、首末颠倒。
|
||||||
|
这里把凌晨段(北京 hour < 12)的日期 +1 天,得到严格单调的北京时间序列。
|
||||||
|
"""
|
||||||
|
if self.market in _US_EX_MARKETS and self.hour < _US_DAY_SPLIT_HOUR:
|
||||||
|
d = date(self.year, self.month, self.day) + timedelta(days=1)
|
||||||
|
self.year, self.month, self.day = d.year, d.month, d.day
|
||||||
|
|
||||||
|
@property
|
||||||
|
def is_us(self) -> bool:
|
||||||
|
"""是否美股(EX 通道 ``US_STOCK``)。"""
|
||||||
|
return self.market in _US_EX_MARKETS
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def datetime(self) -> datetime:
|
def datetime(self) -> datetime:
|
||||||
|
"""bar 的时刻(北京时间)。"""
|
||||||
return datetime(self.year, self.month, self.day, self.hour, self.minute)
|
return datetime(self.year, self.month, self.day, self.hour, self.minute)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def trading_date(self) -> str:
|
||||||
|
"""该 bar 归属的交易日,``YYYY-MM-DD``。
|
||||||
|
|
||||||
|
美股:``datetime`` 已还原为北京时间,凌晨段(北京 hour < 12)属于**前一天**
|
||||||
|
的美东交易日,故回退一天;A股/港股/期货的交易日即自然日。按交易日聚合
|
||||||
|
(如分钟线合成日线)时应当用本字段,而不是 ``datetime.date()``。
|
||||||
|
"""
|
||||||
|
if self.is_us and self.hour < _US_DAY_SPLIT_HOUR:
|
||||||
|
return (date(self.year, self.month, self.day) - timedelta(days=1)).isoformat()
|
||||||
|
return f"{self.year:04d}-{self.month:02d}-{self.day:02d}"
|
||||||
|
|
||||||
|
|
||||||
@dataclass(slots=True)
|
@dataclass(slots=True)
|
||||||
class IndexBar(SecurityBar):
|
class IndexBar(SecurityBar):
|
||||||
@@ -182,23 +228,72 @@ class SecurityInfo:
|
|||||||
|
|
||||||
@dataclass(slots=True)
|
@dataclass(slots=True)
|
||||||
class TransactionRecord:
|
class TransactionRecord:
|
||||||
"""逐笔成交(跨市场统一模型;A股 MAC 通道含 trade_count=成交笔数)"""
|
"""逐笔成交(跨市场统一模型;A股 MAC 通道含 trade_count=成交笔数)。
|
||||||
|
|
||||||
|
注意(实测 2026-10-21):
|
||||||
|
|
||||||
|
- 服务端是按**约 3 秒聚合**返回的切片,不是交易所原始逐笔成交。
|
||||||
|
- ``vol`` / ``trade_count`` 都是**该切片自身的增量**,不是当日累计值。
|
||||||
|
- ``trade_count`` 为**有符号**值:集合竞价段(``bs_flag=8``)会是负数
|
||||||
|
(实测 sh600000 09:15 段为 -874),不要对全表直接求 max/mean。
|
||||||
|
- 返回列表已按时间升序,但同秒多条时顺序不保证。
|
||||||
|
- ``bs_flag=8`` 的**竞价委托记录已被 ``get_ticks()`` 过滤掉**,不会出现在结果里。
|
||||||
|
|
||||||
|
**指数语义完全不同**(``is_cn_index`` 为真时,如 ``sh000016``):
|
||||||
|
|
||||||
|
- ``vol`` 单位是**百元**(= 成交额 / 100),不是「手」。实测
|
||||||
|
``快照amount / Σtick.vol = 100.0002``,且 ``Σtick.vol`` 与分时 Σvol **逐位相等**。
|
||||||
|
想和成分股成交量对比必须先换算,直接比会得到 ~26 倍的假比例。
|
||||||
|
- ``trade_count`` **不是笔数**,而是**当时均价 × 1000**(实测 tc/1000 与分时
|
||||||
|
``avg_price`` 的差 < 1.1)。指数没有「成交笔数」概念。
|
||||||
|
- ``bs_flag`` 语义不成立(实测返回 2,而模型只定义 0=卖 / 1=买)。
|
||||||
|
|
||||||
|
**场内基金同样没有笔数**(ETF / 多数 LOF,见
|
||||||
|
:func:`~pytdxdata.symbols.has_order_count`):
|
||||||
|
|
||||||
|
- ``trade_count`` 是 ``价格 × 1000``(实测 ``sh510310`` 收盘 tc=3994,
|
||||||
|
当时价 3.989 → 3989,偏差 σ≈2;货币 ETF ``sh511990`` 恒为 0)。
|
||||||
|
- ``vol`` 仍是**手**,``bs_flag`` 正常。
|
||||||
|
- 这类标的的 :attr:`trade_count_available` 为 ``False``,``trade_count`` 已被
|
||||||
|
置 ``0``,不要拿去求和/求均值。
|
||||||
|
|
||||||
|
``trade_count_available=False`` 表示该品种**协议上就不带笔数**,
|
||||||
|
与「本切片恰好 0 笔」不是一回事。
|
||||||
|
|
||||||
|
**可转债(110/111/113/118/123/127/128/132)三样都是真的**(实测 2026-10-22,
|
||||||
|
16 只 A股可转债当日 + 8 只历史日全部自洽):
|
||||||
|
|
||||||
|
- ``vol`` 单位是**手**(1 手 = 10 张),与股票逐笔一致;``trade_count`` 是**真实成交笔数**
|
||||||
|
(``sh111000`` 3631 条切片 / 20443 笔,可转债在
|
||||||
|
:func:`~pytdxdata.symbols.has_order_count` 里判为**可用**)。
|
||||||
|
- ``Σvol`` 与**日线** ``SecurityBar.vol`` **1:1 相等**(比值 1.000000)——
|
||||||
|
这点和股票相反:股票逐笔是「手」、日线是「股」,Σ比值是 **0.01**。
|
||||||
|
- ⚠️ 但**日线** ``amount / vol`` **不等于**均价,而是均价的 **10 倍**(``vol`` 的「手」= 10 张)。
|
||||||
|
要算可转债均价须用 ``amount / vol / 10``;股票则是 ``amount / vol`` 直接就对。
|
||||||
|
实测:``sz127025`` 逐笔 VWAP 105.9063 vs ``amount/vol`` 1059.0957,比值 10.0003。
|
||||||
|
"""
|
||||||
|
|
||||||
market: int
|
market: int
|
||||||
code: str
|
code: str
|
||||||
time: str # HH:MM:SS(扩展市场为 HH:MM)
|
time: str # HH:MM:SS(扩展市场为 HH:MM)
|
||||||
price: float
|
price: float
|
||||||
vol: float # 手(MAC逐笔)
|
vol: float # 股票/转债=手;指数=百元(成交额/100),见类注释
|
||||||
trade_count: int # 成交笔数(MAC=逐笔记录字段;标准当日逐笔协议也有但需单独解析)
|
trade_count: int # 股票=笔数增量;指数/基金=占位值,见类注释
|
||||||
bs_flag: int # 1=主动买 0=主动卖
|
bs_flag: int # 1=主动买 0=主动卖 2=中性 5=盘后固定价(指数语义不成立)
|
||||||
symbol: str = ""
|
symbol: str = ""
|
||||||
fields: dict[str, float] = field(default_factory=dict) # 扩展市场额外字段(增仓等)
|
fields: dict[str, float] = field(default_factory=dict) # 扩展市场额外字段(增仓等)
|
||||||
|
trade_count_available: bool = True # 指数/ETF/LOF 为 False(协议无笔数字段)
|
||||||
|
|
||||||
|
|
||||||
@dataclass(slots=True)
|
@dataclass(slots=True)
|
||||||
class MinuteBar:
|
class MinuteBar:
|
||||||
"""分时(跨市场统一模型;A股 MAC 0x122D / 扩展市场 0x122D 同构)"""
|
"""分时(跨市场统一模型;A股 MAC 0x122D / 扩展市场 0x122D 同构)。
|
||||||
|
|
||||||
|
``hour``/``minute`` 是**真实时钟**(``0 <= hour < 24``),跨日的分时请用
|
||||||
|
:attr:`datetime` / :attr:`trading_date`,不要自己拿 ``hour`` 拼日期。
|
||||||
|
"""
|
||||||
price: float
|
price: float
|
||||||
vol: float
|
vol: float # 手(股票/转债同);Σ分时vol 与当日日线 vol 同口径
|
||||||
avg_price: float = 0.0 # 均价(协议第2字段)
|
avg_price: float = 0.0 # 均价(协议第2字段)
|
||||||
hour: int = 0
|
hour: int = 0
|
||||||
minute: int = 0
|
minute: int = 0
|
||||||
@@ -207,6 +302,50 @@ class MinuteBar:
|
|||||||
code: str = ""
|
code: str = ""
|
||||||
date: int = 0 # YYYYMMDD
|
date: int = 0 # YYYYMMDD
|
||||||
fields: dict[str, float] = field(default_factory=dict) # 扩展市场额外字段(动量等)
|
fields: dict[str, float] = field(default_factory=dict) # 扩展市场额外字段(动量等)
|
||||||
|
day_offset: int = 0 # 相对 date 的自然日偏移:美股凌晨段为 +1
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
"""美股(EX 通道)分时时间归一化。
|
||||||
|
|
||||||
|
EX 分时的 ``minutes`` 字段是**自开盘起的连续分钟数**,不是「当天的分钟数」:
|
||||||
|
实测 ``usAAPL`` 返回 ``21:30`` ~ ``27:59``(390 条里 240 条的 ``hour`` ≥ 24)。
|
||||||
|
``27:59`` 实际是次日 ``03:59`` 北京时间。直接 ``minutes // 60`` 会得到
|
||||||
|
``hour=27`` 这种**非法时钟值**,任何 ``datetime(...)`` 拼接都会抛异常,
|
||||||
|
按小时聚合也会把凌晨段错误地归到同一天。
|
||||||
|
|
||||||
|
这里拆成「真实小时 + 日偏移」:``hour = (minutes // 60) % 24``,
|
||||||
|
``day_offset = (minutes // 60) // 24``。A股/港股/期货的 ``hour`` 本就合法
|
||||||
|
(A股合成时间 09:30~15:00、港股 09:30~16:00),**零行为变化**。
|
||||||
|
"""
|
||||||
|
if self.hour >= 24:
|
||||||
|
self.day_offset += self.hour // 24
|
||||||
|
self.hour %= 24
|
||||||
|
|
||||||
|
@property
|
||||||
|
def datetime(self) -> datetime:
|
||||||
|
"""该分时点的时刻(北京时间,已含跨日偏移)。"""
|
||||||
|
base = date(self.date // 10000, self.date // 100 % 100, self.date % 100)
|
||||||
|
d = base + timedelta(days=self.day_offset)
|
||||||
|
return datetime(d.year, d.month, d.day, self.hour, self.minute)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def is_us(self) -> bool:
|
||||||
|
"""是否美股(EX 通道 ``US_STOCK``)。"""
|
||||||
|
return self.market in _US_EX_MARKETS
|
||||||
|
|
||||||
|
@property
|
||||||
|
def trading_date(self) -> str:
|
||||||
|
"""该分时点归属的交易日,``YYYY-MM-DD``。
|
||||||
|
|
||||||
|
美股:一个交易日横跨两个北京自然日(21:30~次日 04:00),凌晨段
|
||||||
|
(北京 hour < 12)归属**前一天**的美东交易日,与
|
||||||
|
:attr:`SecurityBar.trading_date` 口径一致。A股/港股/期货即 ``date`` 本身。
|
||||||
|
按交易日聚合分时(合成日线、日内统计)**必须用它**。
|
||||||
|
"""
|
||||||
|
base = self.datetime
|
||||||
|
if self.is_us and base.hour < _US_DAY_SPLIT_HOUR:
|
||||||
|
base -= timedelta(days=1)
|
||||||
|
return base.date().isoformat()
|
||||||
|
|
||||||
|
|
||||||
@dataclass(slots=True)
|
@dataclass(slots=True)
|
||||||
@@ -448,3 +587,82 @@ class FinancialRecord:
|
|||||||
market: int # 0=深 1=沪
|
market: int # 0=深 1=沪
|
||||||
report_date: int # 报告期 YYYYMMDD
|
report_date: int # 报告期 YYYYMMDD
|
||||||
fields: list[float] = field(default_factory=list)
|
fields: list[float] = field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
# ===== 指数成分成交汇总(TdxData.get_index_trade_stats) =====
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class MinuteTradeStat:
|
||||||
|
"""指数**全部成分股**汇总后的单分钟成交**增量**(不是累计)。
|
||||||
|
|
||||||
|
累计曲线用 :meth:`IndexTradeStats.cumulative`,不要自己再对同一字段反复累加。
|
||||||
|
"""
|
||||||
|
time: str # HH:MM
|
||||||
|
volume: float # 成交量合计(手,1 手 = 100 股)
|
||||||
|
trade_count: int # 成交笔数合计
|
||||||
|
amount: float # 成交额估算(元)= Σ(逐笔价 × 手 × 100)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class IndexTradeStats:
|
||||||
|
"""某个指数「全部成分股逐笔」汇总出的成交曲线。
|
||||||
|
|
||||||
|
为什么需要它:**通达信协议里指数本身没有成交笔数** —— 指数逐笔
|
||||||
|
(``sh000001``)的 ``trade_count`` 字段实际是「当时均价 × 1000」(见
|
||||||
|
:class:`TransactionRecord`)。所以「指数笔数」唯一可行的口径是
|
||||||
|
**把成分股的逐笔笔数累加**,本类即该口径。
|
||||||
|
|
||||||
|
口径校验(2026-10-22 实测):
|
||||||
|
|
||||||
|
- ``total_amount`` 是**成分股**合计。对「成分恰好就是指数覆盖面」的指数,它与
|
||||||
|
指数快照 ``amount`` **完全相等**:``sh000016`` 661.34 亿 = 661.34 亿、
|
||||||
|
``sh000300`` 2373.98 亿 = 2373.98 亿(比值 1.0000)。
|
||||||
|
- **但深市 ``399xxx`` 的指数成交额字段是「全板块」口径**,不是样本股:
|
||||||
|
实测 ``sz399001`` 快照 5081.40 亿 = 深市 2908 只股票合计 5081.01 亿、
|
||||||
|
``sz399006`` 2568.60 亿 = 创业板全部股票合计。所以拿深市指数 ``amount``
|
||||||
|
与 ``total_amount`` 比会得到 ~0.43 这类无意义比值——不是数据错,是口径不同。
|
||||||
|
- ``total_volume`` 单位是**手**(与指数逐笔的「百元」单位不同,勿混用)。
|
||||||
|
- 北交所成分(``bj`` 前缀)**没有笔数字段**(实测 ``trade_count`` 恒为 0):
|
||||||
|
此时 ``trade_count_members = 0``、:attr:`trade_count_available` 为 ``False``,
|
||||||
|
只有成交量可用。``sh``/``sz`` 成分均正常返回笔数。
|
||||||
|
|
||||||
|
``minutes`` 只包含**实际有成交**的分钟(午休 11:30~13:00 之间无桶),
|
||||||
|
画累计曲线时请按 ``time`` 自行补齐或直接连线。
|
||||||
|
"""
|
||||||
|
index: str # 规范化指数标的(如 sh000300)
|
||||||
|
date: int | None # None=当日;否则 YYYYMMDD
|
||||||
|
member_count: int # 指数成分数
|
||||||
|
active_members: int # 实际有逐笔成交的成分数(停牌/无成交不计)
|
||||||
|
trade_count_members: int # 返回了笔数的成分数(北交所为 0)
|
||||||
|
minutes: list[MinuteTradeStat] = field(default_factory=list)
|
||||||
|
total_volume: float = 0.0
|
||||||
|
total_trade_count: int = 0
|
||||||
|
total_amount: float = 0.0
|
||||||
|
failed_members: int = 0 # 逐笔拉取失败的成分数(skip_errors=True 时)
|
||||||
|
elapsed: float = 0.0 # 本次汇总耗时(秒)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def trade_count_available(self) -> bool:
|
||||||
|
"""是否有笔数口径(北交所成分为 ``False``,此时 ``trade_count`` 全 0)。"""
|
||||||
|
return self.trade_count_members > 0
|
||||||
|
|
||||||
|
def cumulative(self) -> list[MinuteTradeStat]:
|
||||||
|
"""累计曲线:当日**截止该分钟**的累计量,时间升序。
|
||||||
|
|
||||||
|
这是「当日截止该分钟累计成交笔数」的直接数据源。
|
||||||
|
"""
|
||||||
|
out: list[MinuteTradeStat] = []
|
||||||
|
vol = 0.0
|
||||||
|
tc = 0
|
||||||
|
amt = 0.0
|
||||||
|
for m in self.minutes:
|
||||||
|
vol += m.volume
|
||||||
|
tc += m.trade_count
|
||||||
|
amt += m.amount
|
||||||
|
out.append(MinuteTradeStat(time=m.time, volume=vol,
|
||||||
|
trade_count=tc, amount=amt))
|
||||||
|
return out
|
||||||
|
|
||||||
|
def cum_trade_count(self) -> list[tuple[str, int]]:
|
||||||
|
"""``[(HH:MM, 累计笔数)]``,画「截止该分钟累计笔数」曲线用。"""
|
||||||
|
return [(m.time, m.trade_count) for m in self.cumulative()]
|
||||||
@@ -3,6 +3,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import time
|
import time
|
||||||
|
from collections.abc import Sequence
|
||||||
from typing import Generic, TypeVar
|
from typing import Generic, TypeVar
|
||||||
|
|
||||||
from ..protocol.commands.base import BaseCommand
|
from ..protocol.commands.base import BaseCommand
|
||||||
@@ -22,6 +23,7 @@ class TdxConnection(Generic[T]):
|
|||||||
self._reader: asyncio.StreamReader | None = None
|
self._reader: asyncio.StreamReader | None = None
|
||||||
self._writer: asyncio.StreamWriter | None = None
|
self._writer: asyncio.StreamWriter | None = None
|
||||||
self._last_result: T | None = None
|
self._last_result: T | None = None
|
||||||
|
self._pipeline_result: list[T] = []
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def alive(self) -> bool:
|
def alive(self) -> bool:
|
||||||
@@ -107,6 +109,58 @@ class TdxConnection(Generic[T]):
|
|||||||
body = decompress_body(header, raw_body)
|
body = decompress_body(header, raw_body)
|
||||||
self._last_result: T = cmd.parse_response(body)
|
self._last_result: T = cmd.parse_response(body)
|
||||||
|
|
||||||
|
async def execute_pipeline(
|
||||||
|
self, cmds: Sequence[BaseCommand[T]], *, timeout: float | None = None
|
||||||
|
) -> list[T]:
|
||||||
|
"""单连接**流水线**:一次性发出全部请求,再按序读取全部响应。
|
||||||
|
|
||||||
|
通达信协议是自描述长度帧(16 字节帧头 + body),且服务器**按序**应答,
|
||||||
|
因此可以在同一 TCP 连接上连续下发多条请求、省掉 ``(N-1)`` 次往返。
|
||||||
|
实测 66 个「80 只报价」请求:逐条 7,372ms → 流水线 526ms(14 倍)。
|
||||||
|
|
||||||
|
与 :meth:`execute` 的关键差异:``execute`` 逐条严格配对,失败可小范围重试;
|
||||||
|
流水线一旦中途失败,**响应流已错配**,剩余响应无法再与请求对齐,
|
||||||
|
故此处强制断开连接(上层应换连接重试)。
|
||||||
|
|
||||||
|
参数:
|
||||||
|
cmds: 同一连接的请求序列(通常为同一类命令的不同分片)。
|
||||||
|
timeout: 整体超时(默认 ``self.timeout * 2``,流水线耗时随条数增长)。
|
||||||
|
|
||||||
|
返回:
|
||||||
|
与 ``cmds`` **等长且同序**的结果列表。
|
||||||
|
"""
|
||||||
|
if not cmds:
|
||||||
|
return []
|
||||||
|
if not self.alive:
|
||||||
|
raise ConnectionError(f"连接未建立或已关闭: {self.host}")
|
||||||
|
assert self._reader is not None and self._writer is not None
|
||||||
|
budget = timeout if timeout is not None else self.timeout * 2
|
||||||
|
try:
|
||||||
|
try:
|
||||||
|
await asyncio.wait_for(self._io_pipeline(cmds), budget)
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
self._force_close()
|
||||||
|
raise ConnectionError(
|
||||||
|
f"流水线超时 {self.host} ({budget:.1f}s, {len(cmds)} 条)") from None
|
||||||
|
except (ConnectionError, OSError) as e:
|
||||||
|
self._force_close()
|
||||||
|
raise ConnectionError(f"流水线错误 {self.host}: {e}") from e
|
||||||
|
except (ConnectionError, OSError):
|
||||||
|
raise
|
||||||
|
return self._pipeline_result
|
||||||
|
|
||||||
|
async def _io_pipeline(self, cmds: Sequence[BaseCommand[T]]) -> None:
|
||||||
|
assert self._reader is not None and self._writer is not None
|
||||||
|
self._writer.write(b"".join(c.build_request() for c in cmds))
|
||||||
|
await self._writer.drain()
|
||||||
|
results: list[T] = []
|
||||||
|
for cmd in cmds:
|
||||||
|
header_buf = await self._recv_exact(HEADER_SIZE)
|
||||||
|
header = parse_header(header_buf)
|
||||||
|
raw_body = await self._recv_exact(header.zipsize)
|
||||||
|
results.append(cmd.parse_response(decompress_body(header, raw_body)))
|
||||||
|
self._pipeline_result: list[T] = results
|
||||||
|
|
||||||
async def _send_setup(self) -> None:
|
async def _send_setup(self) -> None:
|
||||||
"""按序发送3条握手命令并丢弃响应。"""
|
"""按序发送3条握手命令并丢弃响应。"""
|
||||||
assert self._reader is not None and self._writer is not None
|
assert self._reader is not None and self._writer is not None
|
||||||
|
|||||||
@@ -1,4 +1,11 @@
|
|||||||
"""EX扩展市场连接池(简化版:固定连接数,借还+坏连接重建+login自动)"""
|
"""EX扩展市场连接池(固定连接数,借还+坏连接重建+login自动)。
|
||||||
|
|
||||||
|
与标准池 pool.py 的背压语义对齐:acquire 在无空闲连接且无法扩容时,
|
||||||
|
基于 asyncio.Condition 排队等待 release 唤醒(默认 acquire_timeout=30s),
|
||||||
|
而不是快速失败。per_server_cap 限制单台服务器连接数;单服务器场景下
|
||||||
|
有效并发 = servers × per_server_cap(默认 1),通过 TdxData(ex_pool_cap=N)
|
||||||
|
可对同一 EX 主机开多连接。
|
||||||
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
@@ -9,6 +16,23 @@ from .ex_connection import ExTdxConnection
|
|||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
DEFAULT_ACQUIRE_TIMEOUT = 30.0
|
||||||
|
|
||||||
|
|
||||||
|
class ExPoolTimeoutError(ConnectionError):
|
||||||
|
"""acquire 等待超时(携带池状态快照,便于排障)。"""
|
||||||
|
|
||||||
|
def __init__(self, pool: "ExPool", timeout: float) -> None:
|
||||||
|
idle = len(pool._idle)
|
||||||
|
total = pool._total
|
||||||
|
servers = len(pool.servers)
|
||||||
|
cap = pool.per_server_cap
|
||||||
|
super().__init__(
|
||||||
|
f"EX连接池获取超时(等了 {timeout:.1f}s):"
|
||||||
|
f"idle={idle} total={total} servers={servers} per_server_cap={cap} "
|
||||||
|
f"有效并发={servers * cap}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class ExPool:
|
class ExPool:
|
||||||
"""EX通道连接池(港股/期货,端口7727,连接时自动login)。"""
|
"""EX通道连接池(港股/期货,端口7727,连接时自动login)。"""
|
||||||
@@ -16,6 +40,7 @@ class ExPool:
|
|||||||
def __init__(self, servers: list[str], *, timeout: float = 10.0,
|
def __init__(self, servers: list[str], *, timeout: float = 10.0,
|
||||||
min_size: int = 2, max_size: int = 6,
|
min_size: int = 2, max_size: int = 6,
|
||||||
per_server_cap: int = 1,
|
per_server_cap: int = 1,
|
||||||
|
acquire_timeout: float | None = None,
|
||||||
handshake: str = "login",
|
handshake: str = "login",
|
||||||
handshake_map: dict | None = None) -> None:
|
handshake_map: dict | None = None) -> None:
|
||||||
self.servers = servers
|
self.servers = servers
|
||||||
@@ -25,14 +50,17 @@ class ExPool:
|
|||||||
self.min_size = min_size
|
self.min_size = min_size
|
||||||
self.max_size = max_size
|
self.max_size = max_size
|
||||||
self.per_server_cap = max(1, per_server_cap) # 一台服务器最多几个连接(默认1=单连接)
|
self.per_server_cap = max(1, per_server_cap) # 一台服务器最多几个连接(默认1=单连接)
|
||||||
|
self.acquire_timeout = (
|
||||||
|
DEFAULT_ACQUIRE_TIMEOUT if acquire_timeout is None else acquire_timeout
|
||||||
|
)
|
||||||
self._idle: list[tuple[str, ExTdxConnection]] = []
|
self._idle: list[tuple[str, ExTdxConnection]] = []
|
||||||
self._server_count: dict[str, int] = {}
|
self._server_count: dict[str, int] = {}
|
||||||
self._total = 0
|
self._total = 0
|
||||||
self._closed = False
|
self._closed = False
|
||||||
self._lock = asyncio.Lock()
|
self._cond = asyncio.Condition()
|
||||||
|
|
||||||
async def start(self) -> None:
|
async def start(self) -> None:
|
||||||
async with self._lock:
|
async with self._cond:
|
||||||
for host in self.servers[: self.min_size]:
|
for host in self.servers[: self.min_size]:
|
||||||
if self._server_count.get(host, 0) >= self.per_server_cap:
|
if self._server_count.get(host, 0) >= self.per_server_cap:
|
||||||
continue
|
continue
|
||||||
@@ -47,7 +75,7 @@ class ExPool:
|
|||||||
|
|
||||||
async def close(self) -> None:
|
async def close(self) -> None:
|
||||||
self._closed = True
|
self._closed = True
|
||||||
async with self._lock:
|
async with self._cond:
|
||||||
for _, conn in self._idle:
|
for _, conn in self._idle:
|
||||||
try:
|
try:
|
||||||
await conn.close()
|
await conn.close()
|
||||||
@@ -56,48 +84,61 @@ class ExPool:
|
|||||||
self._idle.clear()
|
self._idle.clear()
|
||||||
self._total = 0
|
self._total = 0
|
||||||
self._server_count.clear()
|
self._server_count.clear()
|
||||||
|
self._cond.notify_all()
|
||||||
|
|
||||||
async def acquire(self) -> ExTdxConnection:
|
async def acquire(self, timeout: float | None = None) -> ExTdxConnection:
|
||||||
for _ in range(3):
|
"""借出一条连接;无空闲且无法扩容时排队等待 release 唤醒。
|
||||||
stale_conn = None
|
|
||||||
new_conn = None
|
timeout 默认取 self.acquire_timeout(30s);传 None 表示无限等待。
|
||||||
new_host = None
|
"""
|
||||||
async with self._lock:
|
if timeout is None:
|
||||||
|
timeout = self.acquire_timeout
|
||||||
|
deadline = None if timeout is None else time.monotonic() + timeout
|
||||||
|
async with self._cond:
|
||||||
|
while True:
|
||||||
|
# 1. 优先借用空闲连接(过期则丢弃,槽位在锁内回收)
|
||||||
if self._idle:
|
if self._idle:
|
||||||
_, conn = self._idle.pop()
|
_, conn = self._idle.pop()
|
||||||
fresh = time.monotonic() - conn.last_used < 30.0
|
fresh = time.monotonic() - conn.last_used < 30.0
|
||||||
if conn.alive and fresh:
|
if conn.alive and fresh:
|
||||||
return conn
|
return conn
|
||||||
# 连接空闲过久(login会话可能失效)→ 丢弃重连
|
# 连接空闲过久(login会话可能失效)→ 丢弃,槽位让给扩容
|
||||||
self._total = max(0, self._total - 1)
|
self._total = max(0, self._total - 1)
|
||||||
self._server_count[conn.host] = max(0, self._server_count.get(conn.host, 0) - 1)
|
self._server_count[conn.host] = max(0, self._server_count.get(conn.host, 0) - 1)
|
||||||
stale_conn = conn
|
self._schedule_close(conn)
|
||||||
# 扩容:在锁内预留槽位,锁外执行网络IO
|
|
||||||
elif self._total < self.max_size and self.servers:
|
# 2. 可扩容 → 锁内预留槽位,锁外建连
|
||||||
|
new_conn = None
|
||||||
|
new_host = None
|
||||||
|
if self._total < self.max_size and self.servers:
|
||||||
host = self._next_available_server()
|
host = self._next_available_server()
|
||||||
if host:
|
if host is not None:
|
||||||
self._total += 1
|
self._total += 1
|
||||||
self._server_count[host] = self._server_count.get(host, 0) + 1
|
self._server_count[host] = self._server_count.get(host, 0) + 1
|
||||||
new_conn = ExTdxConnection(host, 7727, self.timeout, self.handshake_map.get(host, self.handshake))
|
new_conn = ExTdxConnection(host, 7727, self.timeout, self.handshake_map.get(host, self.handshake))
|
||||||
new_host = host
|
new_host = host
|
||||||
# 锁外:关闭过期连接、建立新连接
|
|
||||||
if stale_conn is not None:
|
|
||||||
try:
|
|
||||||
await stale_conn.close()
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
if new_conn is not None:
|
if new_conn is not None:
|
||||||
|
# 锁外网络 IO:连接失败回滚槽位后继续等待
|
||||||
try:
|
try:
|
||||||
await new_conn.connect()
|
await new_conn.connect()
|
||||||
return new_conn
|
return new_conn
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
log.warning("EX扩容失败 %s: %s", new_host, str(e)[:60])
|
log.warning("EX扩容失败 %s: %s", new_host, str(e)[:60])
|
||||||
# 连接失败,回滚预留槽位
|
async with self._cond:
|
||||||
async with self._lock:
|
|
||||||
self._total = max(0, self._total - 1)
|
self._total = max(0, self._total - 1)
|
||||||
self._server_count[new_host] = max(0, self._server_count.get(new_host, 0) - 1)
|
self._server_count[new_host] = max(0, self._server_count.get(new_host, 0) - 1)
|
||||||
await asyncio.sleep(0.05)
|
|
||||||
raise ConnectionError("EX连接池无可用连接")
|
# 3. 池满:排队等待 release 唤醒(或超时)
|
||||||
|
now = time.monotonic()
|
||||||
|
if deadline is not None and now >= deadline:
|
||||||
|
raise ExPoolTimeoutError(self, timeout)
|
||||||
|
try:
|
||||||
|
if deadline is None:
|
||||||
|
await self._cond.wait()
|
||||||
|
else:
|
||||||
|
await asyncio.wait_for(self._cond.wait(), timeout=deadline - now)
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
raise ExPoolTimeoutError(self, timeout)
|
||||||
|
|
||||||
def _next_available_server(self) -> str | None:
|
def _next_available_server(self) -> str | None:
|
||||||
for h in self.servers:
|
for h in self.servers:
|
||||||
@@ -105,15 +146,23 @@ class ExPool:
|
|||||||
return h
|
return h
|
||||||
return None
|
return None
|
||||||
|
|
||||||
async def release(self, conn: ExTdxConnection, *, broken: bool = False) -> None:
|
def _schedule_close(self, conn: ExTdxConnection) -> None:
|
||||||
if broken or not conn.alive:
|
async def _close():
|
||||||
async with self._lock:
|
|
||||||
self._total = max(0, self._total - 1)
|
|
||||||
self._server_count[conn.host] = max(0, self._server_count.get(conn.host, 0) - 1)
|
|
||||||
try:
|
try:
|
||||||
await conn.close()
|
await conn.close()
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
return
|
try:
|
||||||
async with self._lock:
|
asyncio.get_running_loop().create_task(_close())
|
||||||
|
except RuntimeError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
async def release(self, conn: ExTdxConnection, *, broken: bool = False) -> None:
|
||||||
|
async with self._cond:
|
||||||
|
if broken or not conn.alive:
|
||||||
|
self._total = max(0, self._total - 1)
|
||||||
|
self._server_count[conn.host] = max(0, self._server_count.get(conn.host, 0) - 1)
|
||||||
|
self._schedule_close(conn)
|
||||||
|
else:
|
||||||
self._idle.append((conn.host, conn))
|
self._idle.append((conn.host, conn))
|
||||||
|
self._cond.notify_all()
|
||||||
@@ -6,8 +6,9 @@
|
|||||||
- 动态:支持运行时 add() 新服务器;支持后台全量测速入库(当日首次启动)。
|
- 动态:支持运行时 add() 新服务器;支持后台全量测速入库(当日首次启动)。
|
||||||
- 单连接约束(核心需求):per_server_cap 默认 1 —— 一台服务器只允许建立 1 个连接,
|
- 单连接约束(核心需求):per_server_cap 默认 1 —— 一台服务器只允许建立 1 个连接,
|
||||||
不允许多开,避免对单一券商服务器造成并发压力被限流/封禁。
|
不允许多开,避免对单一券商服务器造成并发压力被限流/封禁。
|
||||||
- 后台测速(核心需求):当日首次启动时,在后台对全量候选服务器测速,按
|
- 后台测速(核心需求):当日首次启动时,在后台对**全量候选**服务器测速(覆盖标准/MAC/EX
|
||||||
「排名前 FAST_POOL_TOP_N 或 延迟 < FAST_POOL_MAX_LATENCY」筛选入库,结果写回 servers.json。
|
三个协议,而不是只测当前池内节点),结果缓存到用户级缓存目录;当天已有缓存则跳过、不再刷新。
|
||||||
|
全程不阻塞 start()、不打印、不抛错(对用户无感)。详见 pool/probe_cache.py。
|
||||||
- 弹性:空闲连接超时后由 reaper 缩容回 min_size;支持 stats 统计。
|
- 弹性:空闲连接超时后由 reaper 缩容回 min_size;支持 stats 统计。
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -19,7 +20,7 @@ import time
|
|||||||
|
|
||||||
from .connection import TdxConnection
|
from .connection import TdxConnection
|
||||||
from .health import HealthEngine
|
from .health import HealthEngine
|
||||||
from .probe_cache import get_probe_cache
|
from .probe_cache import schedule_probe
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -58,6 +59,11 @@ class PooledConnection:
|
|||||||
self.last_used = time.monotonic()
|
self.last_used = time.monotonic()
|
||||||
return await self.conn.execute(cmd)
|
return await self.conn.execute(cmd)
|
||||||
|
|
||||||
|
async def execute_pipeline(self, cmds, *, timeout: float | None = None):
|
||||||
|
"""在同一条连接上流水线执行多条命令(见 TdxConnection.execute_pipeline)。"""
|
||||||
|
self.last_used = time.monotonic()
|
||||||
|
return await self.conn.execute_pipeline(cmds, timeout=timeout)
|
||||||
|
|
||||||
async def close(self) -> None:
|
async def close(self) -> None:
|
||||||
self.alive = False
|
self.alive = False
|
||||||
try:
|
try:
|
||||||
@@ -101,6 +107,7 @@ class DynamicPool:
|
|||||||
self._lock = asyncio.Lock()
|
self._lock = asyncio.Lock()
|
||||||
self._cond = asyncio.Condition(self._lock)
|
self._cond = asyncio.Condition(self._lock)
|
||||||
self._reaper_task: asyncio.Task | None = None
|
self._reaper_task: asyncio.Task | None = None
|
||||||
|
self._probe_task: asyncio.Task | None = None
|
||||||
|
|
||||||
# ---------- 生命周期 ----------
|
# ---------- 生命周期 ----------
|
||||||
async def start(self) -> None:
|
async def start(self) -> None:
|
||||||
@@ -108,13 +115,11 @@ class DynamicPool:
|
|||||||
return
|
return
|
||||||
self._started = True
|
self._started = True
|
||||||
|
|
||||||
# 需求①:当日首次启动 → 后台全量测速入库(不阻塞主流程)
|
# 需求①:当日首次启动 → 后台全量测速(不阻塞主流程;当天已有缓存则自动跳过)
|
||||||
|
# 候选清单由 probe_cache 自行枚举全量主机(本池的 servers 只是其中一个子集,
|
||||||
|
# 不能直接当作测速输入,否则只会重测已在池内的少数节点、其余记录永久过期)。
|
||||||
if self.auto_probe:
|
if self.auto_probe:
|
||||||
hosts = list(self.servers)
|
self._probe_task = schedule_probe()
|
||||||
asyncio.create_task(
|
|
||||||
get_probe_cache().ensure_probe(hosts),
|
|
||||||
name="pytdxdata-daily-probe",
|
|
||||||
)
|
|
||||||
|
|
||||||
targets = self._pick_servers(self.min_size)
|
targets = self._pick_servers(self.min_size)
|
||||||
for host in targets:
|
for host in targets:
|
||||||
@@ -175,6 +180,15 @@ class DynamicPool:
|
|||||||
if self._reaper_task is not None:
|
if self._reaper_task is not None:
|
||||||
self._reaper_task.cancel()
|
self._reaper_task.cancel()
|
||||||
self._reaper_task = None
|
self._reaper_task = None
|
||||||
|
# 后台测速仍可能持有 socket;取消并等它退场,避免事件循环关闭时冒出
|
||||||
|
# "Task was destroyed but it is pending"(那会破坏"无感")
|
||||||
|
if self._probe_task is not None:
|
||||||
|
self._probe_task.cancel()
|
||||||
|
try:
|
||||||
|
await self._probe_task
|
||||||
|
except (asyncio.CancelledError, Exception):
|
||||||
|
pass
|
||||||
|
self._probe_task = None
|
||||||
async with self._lock:
|
async with self._lock:
|
||||||
for pc in list(self._conns):
|
for pc in list(self._conns):
|
||||||
try:
|
try:
|
||||||
@@ -264,6 +278,46 @@ class DynamicPool:
|
|||||||
]
|
]
|
||||||
return random.choice(alive) if alive else None
|
return random.choice(alive) if alive else None
|
||||||
|
|
||||||
|
async def warm(self, n: int) -> None:
|
||||||
|
"""并发预建连接至 ``n`` 条(最多到 ``max_size``)。
|
||||||
|
|
||||||
|
:meth:`acquire` 建连时持有池锁,**一次只能建一条**——22 路并发首次
|
||||||
|
调用会排队 22×握手时间(实测全市场首拉多花 5~6s)。大批量请求前先调
|
||||||
|
本方法把连接并发建好,首拉只多付一次握手。
|
||||||
|
|
||||||
|
失败静默忽略:缺的连接后续由 ``acquire`` 按需补齐。
|
||||||
|
"""
|
||||||
|
if self._closed:
|
||||||
|
return
|
||||||
|
need = min(n, self.max_size) - self._total
|
||||||
|
if need <= 0:
|
||||||
|
return
|
||||||
|
|
||||||
|
# 用局部计数挑 host,避免并发建连时反复选中同一台
|
||||||
|
counts = dict(self._server_count)
|
||||||
|
ranked = self.health.rank(self.servers)
|
||||||
|
hosts: list[str] = []
|
||||||
|
for _ in range(need):
|
||||||
|
host = next((h for h in ranked
|
||||||
|
if not self.health.in_cooldown(h)
|
||||||
|
and counts.get(h, 0) < self.per_server_cap), None)
|
||||||
|
if host is None:
|
||||||
|
break
|
||||||
|
hosts.append(host)
|
||||||
|
counts[host] = counts.get(host, 0) + 1
|
||||||
|
if not hosts:
|
||||||
|
return
|
||||||
|
|
||||||
|
results = await asyncio.gather(
|
||||||
|
*(self._new_conn(h) for h in hosts), return_exceptions=True)
|
||||||
|
async with self._lock:
|
||||||
|
for pc in results:
|
||||||
|
if isinstance(pc, PooledConnection):
|
||||||
|
self._total += 1
|
||||||
|
self.stats.total = self._total
|
||||||
|
if not self._closed and pc.alive:
|
||||||
|
self._idle.append(pc)
|
||||||
|
|
||||||
async def release(self, pc: PooledConnection, *, broken: bool = False) -> None:
|
async def release(self, pc: PooledConnection, *, broken: bool = False) -> None:
|
||||||
if self.stats.borrowed > 0:
|
if self.stats.borrowed > 0:
|
||||||
self.stats.borrowed -= 1
|
self.stats.borrowed -= 1
|
||||||
|
|||||||
+272
-112
@@ -1,162 +1,322 @@
|
|||||||
"""当日首次启动的全量服务器测速与入库缓存。
|
"""当日首次启动的全量服务器测速与缓存(对用户完全无感)。
|
||||||
|
|
||||||
设计要点:
|
行为
|
||||||
- 仅在"当日首次启动"时触发一次后台全量测速(按本地日期判断是否已探测)。
|
----
|
||||||
- 测速采用真实握手连通(TdxConnection.test),比单纯 ping 更能反映通达信可用性。
|
- **当日首次启动**触发一次后台全量测速;当天已有缓存则直接复用,**不再刷新**。
|
||||||
- 入库规则:延迟排名前 FAST_POOL_TOP_N(10) **或** 延迟 < FAST_POOL_MAX_LATENCY(0.2s) 的服务器
|
- 测速覆盖**全部候选**(包内清单 ∪ 代码常量 ∪ env 覆盖,见
|
||||||
才标记为 status="ok" 并写入 servers.json,其余降权(保留但不优先)。
|
:func:`pytdxdata.config.all_probe_candidates`),而不是只测当前已在池里的少数节点 ——
|
||||||
- 结果持久化到 servers.json,供 config / 连接池直接使用。
|
否则"上次慢、这次快"的节点永远捞不回来。
|
||||||
|
- 用**真实业务命令**判定可达(标准: GetSecurityBarsCmd / MAC: MacKlineCmd /
|
||||||
|
EX: GetExInstrumentBarsCmd)并要求返回非空数据,而不是只做握手回显:
|
||||||
|
存在"握手正常但 K线空响应"的坏节点(见 ``config.py`` 注释),只有真数据能测出来。
|
||||||
|
- 结果写入**用户级缓存目录**(见 :func:`pytdxdata.config.probe_cache_file`),
|
||||||
|
不写安装目录:site-packages 在容器 / 系统级安装 / pipx 下可能只读,
|
||||||
|
多用户共用一份安装时还会互相覆盖。
|
||||||
|
- 结果被 :func:`pytdxdata.config.get_standard_hosts` 等按**实测延迟升序**采用。
|
||||||
|
|
||||||
|
无感保证
|
||||||
|
--------
|
||||||
|
- 不打印任何东西;日志一律 ``debug`` 级(默认不输出)。
|
||||||
|
- 不抛异常(取消除外);失败只落 debug 日志,下次启动自然重试。
|
||||||
|
- 由连接池以后台 task 方式调用,不阻塞 ``start()`` 与任何用户请求。
|
||||||
|
|
||||||
|
一致性与并发
|
||||||
|
------------
|
||||||
|
- 进程内单例 + ``asyncio.Lock``:标准 / MAC / EX 三个池共用一次真正测速。
|
||||||
|
- 原子写入(tmp + ``os.replace``):进程被杀或多进程并发都不会读到半截 JSON。
|
||||||
|
- 全部不可达时**不写缓存**,避免把"全挂"状态缓存一整天。
|
||||||
|
|
||||||
|
开关
|
||||||
|
----
|
||||||
|
- ``PYPYTDX_DATA_DISABLE_PROBE=1`` 关闭(CI / 测试用,保证确定性)。
|
||||||
|
- ``PYPYTDX_DATA_CACHE_DIR`` 覆盖缓存目录(测试指向临时目录)。
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
|
import os
|
||||||
import time
|
import time
|
||||||
|
from collections import defaultdict
|
||||||
|
from collections.abc import Awaitable, Callable
|
||||||
from datetime import date
|
from datetime import date
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from ..config import _SERVERS_FILE
|
from ..config import (
|
||||||
|
PROBE_CACHE_VERSION,
|
||||||
|
all_probe_candidates,
|
||||||
|
get_ex_handshakes,
|
||||||
|
is_valid_host,
|
||||||
|
load_probe_cache,
|
||||||
|
probe_cache_file,
|
||||||
|
probe_disabled,
|
||||||
|
)
|
||||||
|
from ..models import Adjust, KlinePeriod
|
||||||
|
from ..protocol.commands.ex_proto import GetExInstrumentBarsCmd
|
||||||
|
from ..protocol.commands.kline import GetSecurityBarsCmd
|
||||||
|
from ..protocol.commands.mac import MacKlineCmd
|
||||||
|
from .connection import TdxConnection
|
||||||
|
from .ex_connection import ExTdxConnection
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
# 入池阈值
|
# 入池阈值(仅用于缓存里的 fast 诊断标记;运行时按实测延迟整体排序,不做硬切)
|
||||||
FAST_POOL_TOP_N = 10 # 延迟排名前 N
|
FAST_POOL_TOP_N = 10
|
||||||
FAST_POOL_MAX_LATENCY = 0.2 # 延迟 < 200ms(秒)也可入池
|
FAST_POOL_MAX_LATENCY = 0.2
|
||||||
|
# 探针参数
|
||||||
PROBE_PORT = 7709
|
PROBE_PORT = 7709
|
||||||
|
EX_PORT = 7727
|
||||||
PROBE_TIMEOUT = 3.0
|
PROBE_TIMEOUT = 3.0
|
||||||
PROBE_CONCURRENCY = 24
|
PROBE_CONCURRENCY = 12
|
||||||
|
PROBE_BUDGET = 60.0 # 整轮测速硬上限(秒),防止极端网络下长期占着后台
|
||||||
|
|
||||||
|
ProbeFn = Callable[[str, float], Awaitable[float | None]]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 三个协议各自的真实数据探针 ----------
|
||||||
|
async def _probe_standard(host: str, timeout: float) -> float | None:
|
||||||
|
"""标准通道 7709:取 600000 日K,非空才算可达。"""
|
||||||
|
conn = TdxConnection(host, PROBE_PORT, timeout)
|
||||||
|
t0 = time.monotonic()
|
||||||
|
try:
|
||||||
|
await conn.connect()
|
||||||
|
bars = await conn.execute(GetSecurityBarsCmd(1, "600000", KlinePeriod.DAY, 0, 100))
|
||||||
|
return time.monotonic() - t0 if bars else None
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
finally:
|
||||||
|
await conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
async def _probe_mac(host: str, timeout: float) -> float | None:
|
||||||
|
"""MAC 通道 7709:MAC 帧与标准帧不同,用 MacKlineCmd 取日K。"""
|
||||||
|
conn = TdxConnection(host, PROBE_PORT, timeout)
|
||||||
|
t0 = time.monotonic()
|
||||||
|
try:
|
||||||
|
await conn.connect()
|
||||||
|
bars = await conn.execute(
|
||||||
|
MacKlineCmd(1, "600000", KlinePeriod.DAY, Adjust.NONE, 0, 100)
|
||||||
|
)
|
||||||
|
return time.monotonic() - t0 if bars else None
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
finally:
|
||||||
|
await conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
async def _probe_ex(host: str, timeout: float) -> float | None:
|
||||||
|
"""EX 扩展市场 7727:取港股 00700,握手模式按主机取(login / setup)。"""
|
||||||
|
conn = ExTdxConnection(host, EX_PORT, timeout, get_ex_handshakes().get(host, "setup"))
|
||||||
|
t0 = time.monotonic()
|
||||||
|
try:
|
||||||
|
await conn.connect()
|
||||||
|
bars = await conn.execute(GetExInstrumentBarsCmd(31, "00700", 0, 0, 5))
|
||||||
|
return time.monotonic() - t0 if bars else None
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
finally:
|
||||||
|
await conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
_DEFAULT_PROBERS: dict[str, ProbeFn] = {
|
||||||
|
"standard": _probe_standard,
|
||||||
|
"mac": _probe_mac,
|
||||||
|
"ex": _probe_ex,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
class ProbeCache:
|
class ProbeCache:
|
||||||
"""管理当日测速缓存与 servers.json 入库。
|
"""当日全量测速的进程内协调器。
|
||||||
|
|
||||||
一台服务器不论协议(standard/mac/ex),测速结果共享:
|
一台主机不论申报几个协议,都按协议分别探测;缓存里记录
|
||||||
只要任一端口可达即视为该主机可用,并以最快端口延迟作为延迟基准。
|
``{host: {protocol: 延迟或 None}}``,``None`` 表示该协议本次不可达。
|
||||||
|
|
||||||
|
参数(后 5 个主要供测试注入):
|
||||||
|
cache_file: 显式指定缓存文件;``None`` 时按 config 的平台规则解析。
|
||||||
|
probers: 协议 → 探针函数;默认使用真实网络探针。
|
||||||
|
today_fn: 返回 ``YYYY-MM-DD``;测试可注入假日期来验证跨日重测。
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, servers_file: Path | None = None) -> None:
|
def __init__(self, cache_file: Path | str | None = None, *,
|
||||||
self._file = servers_file or _SERVERS_FILE
|
timeout: float = PROBE_TIMEOUT,
|
||||||
|
concurrency: int = PROBE_CONCURRENCY,
|
||||||
|
budget: float = PROBE_BUDGET,
|
||||||
|
probers: dict[str, ProbeFn] | None = None,
|
||||||
|
today_fn: Callable[[], str] | None = None) -> None:
|
||||||
|
self._file = Path(cache_file) if cache_file is not None else None
|
||||||
|
self._timeout = timeout
|
||||||
|
self._concurrency = concurrency
|
||||||
|
self._budget = budget
|
||||||
|
self._probers = dict(probers or _DEFAULT_PROBERS)
|
||||||
|
self._today_fn = today_fn or (lambda: date.today().isoformat())
|
||||||
self._lock = asyncio.Lock()
|
self._lock = asyncio.Lock()
|
||||||
self._last_probe_date: str | None = None
|
self._handled = False # 本进程是否已处理过今日测速(含"已有缓存,跳过")
|
||||||
self._probed_today = False
|
self._task: asyncio.Task | None = None
|
||||||
|
|
||||||
|
# ---------- 路径与状态 ----------
|
||||||
|
def cache_file(self) -> Path:
|
||||||
|
"""缓存文件路径(构造时显式传入的优先,否则按平台规则解析)。"""
|
||||||
|
return self._file if self._file is not None else probe_cache_file()
|
||||||
|
|
||||||
|
def today(self) -> str:
|
||||||
|
return self._today_fn()
|
||||||
|
|
||||||
|
def read_cache(self) -> dict | None:
|
||||||
|
"""读取当日缓存;``None`` 表示"今天还没测过"。"""
|
||||||
|
return load_probe_cache(self.today(), self.cache_file())
|
||||||
|
|
||||||
# ---- 公开 API ----
|
|
||||||
def needs_probe_today(self) -> bool:
|
def needs_probe_today(self) -> bool:
|
||||||
"""是否有必要在今日执行一次全量测速(文件不存在 / 日期不符 / 无 ok 项)。"""
|
"""今日是否仍需测速(缓存缺失 / 非今日 / 损坏 / 版本不符 / 无任何可用项)。"""
|
||||||
if not self._file.exists():
|
if probe_disabled():
|
||||||
return True
|
|
||||||
try:
|
|
||||||
cfg = json.loads(self._file.read_text(encoding="utf-8"))
|
|
||||||
except (OSError, ValueError):
|
|
||||||
return True
|
|
||||||
checked = str(cfg.get("checked_date", ""))
|
|
||||||
today = date.today().isoformat()
|
|
||||||
if checked != today:
|
|
||||||
return True
|
|
||||||
servers = cfg.get("servers", [])
|
|
||||||
if not any(s.get("status") == "ok" for s in servers):
|
|
||||||
return True
|
|
||||||
return False
|
return False
|
||||||
|
return self.read_cache() is None
|
||||||
|
|
||||||
async def ensure_probe(self, hosts: list[str]) -> None:
|
def reset(self) -> None:
|
||||||
"""若今日尚未测速,则后台执行一次全量测速并入库(幂等)。"""
|
"""清除"本进程已处理"标记(仅供测试复用单例)。"""
|
||||||
|
self._handled = False
|
||||||
|
|
||||||
|
# ---------- 主入口 ----------
|
||||||
|
async def ensure_probe(self, hosts: list[str] | None = None) -> None:
|
||||||
|
"""幂等入口:当日尚未测速则测一次;当天已有缓存则立即返回。
|
||||||
|
|
||||||
|
- 不阻塞调用方:正常由连接池以后台 task 调用。
|
||||||
|
- 绝不抛异常(``CancelledError`` 除外):测速 / 写盘失败只落 debug 日志。
|
||||||
|
- ``hosts``:为兼容旧签名保留,作为**额外的标准通道候选**并入,
|
||||||
|
且必须通过 :func:`~pytdxdata.config.is_valid_host` 校验(挡住 mock 脏数据)。
|
||||||
|
"""
|
||||||
async with self._lock:
|
async with self._lock:
|
||||||
if self._probed_today:
|
if self._handled:
|
||||||
return
|
return
|
||||||
|
self._handled = True # 先占位:并发调用方不必排队等网络
|
||||||
if not self.needs_probe_today():
|
if not self.needs_probe_today():
|
||||||
self._probed_today = True
|
log.debug("当日测速缓存已存在,跳过测速: %s", self.cache_file())
|
||||||
return
|
return
|
||||||
self._probed_today = True
|
|
||||||
|
|
||||||
# 在锁外执行(耗时网络 IO),允许并发进入的调用方共享本次结果
|
|
||||||
try:
|
try:
|
||||||
await self._run_full_probe(hosts)
|
payload = await self._run_probe(hosts)
|
||||||
except Exception as e: # 测速失败不能阻塞主流程
|
if payload is not None:
|
||||||
log.warning("后台全量测速失败,沿用既有清单: %s", str(e)[:80])
|
self._write(payload)
|
||||||
|
except asyncio.CancelledError:
|
||||||
|
raise
|
||||||
|
except Exception as exc: # 测速绝不能影响主流程
|
||||||
|
log.debug("后台测速失败,沿用既有清单: %s", str(exc)[:120])
|
||||||
|
|
||||||
# ---- 内部实现 ----
|
# ---------- 内部实现 ----------
|
||||||
async def _run_full_probe(self, hosts: list[str]) -> None:
|
async def _run_probe(self, extra_hosts: list[str] | None = None) -> dict | None:
|
||||||
from .connection import TdxConnection
|
"""全量并发测速,返回待写入的缓存 payload;无任何可达主机时返回 None。"""
|
||||||
|
candidates = all_probe_candidates()
|
||||||
|
if extra_hosts:
|
||||||
|
extra = [h.strip() for h in extra_hosts
|
||||||
|
if isinstance(h, str) and is_valid_host(h)]
|
||||||
|
if extra:
|
||||||
|
candidates["standard"] = list(dict.fromkeys(candidates["standard"] + extra))
|
||||||
|
jobs = [(proto, ip) for proto, ips in candidates.items() for ip in ips]
|
||||||
|
if not jobs:
|
||||||
|
log.debug("无测速候选,跳过")
|
||||||
|
return None
|
||||||
|
|
||||||
log.info("开始当日全量服务器测速 (%d 台)...", len(hosts))
|
log.debug("开始当日全量测速: %d 台主机 / %d 个协议探针",
|
||||||
sem = asyncio.Semaphore(PROBE_CONCURRENCY)
|
len({ip for _, ip in jobs}), len(jobs))
|
||||||
results: dict[str, float | None] = {}
|
sem = asyncio.Semaphore(self._concurrency)
|
||||||
|
# {host: {protocol: 延迟或 None}}
|
||||||
|
results: dict[str, dict[str, float | None]] = defaultdict(dict)
|
||||||
|
|
||||||
async def _probe_one(host: str) -> None:
|
async def _one(protocol: str, host: str) -> None:
|
||||||
async with sem:
|
async with sem:
|
||||||
try:
|
try:
|
||||||
# 使用连接自带的 ping():建连+握手+发1条命令测往返延迟
|
lat = await asyncio.wait_for(
|
||||||
conn = TdxConnection(host, PROBE_PORT, PROBE_TIMEOUT)
|
self._probers[protocol](host, self._timeout),
|
||||||
dt = await asyncio.wait_for(conn.ping(), timeout=PROBE_TIMEOUT + 1.0)
|
timeout=self._timeout + 2.0,
|
||||||
results[host] = dt # 失败时为 None
|
)
|
||||||
except Exception:
|
except Exception:
|
||||||
results[host] = None
|
lat = None
|
||||||
|
results[host][protocol] = lat
|
||||||
|
|
||||||
await asyncio.gather(*(_probe_one(h) for h in hosts))
|
t0 = time.monotonic()
|
||||||
|
|
||||||
ok_hosts = {h: lt for h, lt in results.items() if lt is not None}
|
|
||||||
# 按延迟升序排名
|
|
||||||
ranked = sorted(ok_hosts.items(), key=lambda kv: kv[1])
|
|
||||||
fast = set()
|
|
||||||
for i, (h, lt) in enumerate(ranked):
|
|
||||||
if i < FAST_POOL_TOP_N or lt < FAST_POOL_MAX_LATENCY:
|
|
||||||
fast.add(h)
|
|
||||||
|
|
||||||
log.info("测速完成: 可用 %d 台, 入池 %d 台 (前%d或<%dms)",
|
|
||||||
len(ok_hosts), len(fast), FAST_POOL_TOP_N,
|
|
||||||
int(FAST_POOL_MAX_LATENCY * 1000))
|
|
||||||
self._write_results(results, fast)
|
|
||||||
|
|
||||||
def _write_results(self, results: dict[str, float | None], fast: set[str]) -> None:
|
|
||||||
try:
|
try:
|
||||||
cfg = json.loads(self._file.read_text(encoding="utf-8"))
|
await asyncio.wait_for(
|
||||||
except (OSError, ValueError):
|
asyncio.gather(*(_one(p, ip) for p, ip in jobs)),
|
||||||
cfg = {"servers": []}
|
timeout=self._budget,
|
||||||
servers = cfg.get("servers", [])
|
|
||||||
by_ip = {s.get("ip"): s for s in servers}
|
|
||||||
|
|
||||||
today = date.today().isoformat()
|
|
||||||
now = time.strftime("%Y-%m-%d %H:%M:%S")
|
|
||||||
checked_any = False
|
|
||||||
for host, lt in results.items():
|
|
||||||
entry = by_ip.get(host)
|
|
||||||
if entry is None:
|
|
||||||
entry = {"ip": host, "name": host, "protocols": ["standard"]}
|
|
||||||
servers.append(entry)
|
|
||||||
by_ip[host] = entry
|
|
||||||
if lt is not None:
|
|
||||||
checked_any = True
|
|
||||||
entry["status"] = "ok" if host in fast else "slow"
|
|
||||||
entry["latency"] = round(lt, 3)
|
|
||||||
entry["fast_pool"] = host in fast
|
|
||||||
else:
|
|
||||||
# 探测不可达:不降级已有 ok(避免误杀),仅标记本次不可达
|
|
||||||
entry.setdefault("status", "unknown")
|
|
||||||
entry["latency"] = None
|
|
||||||
entry["fast_pool"] = False
|
|
||||||
entry["checked_date"] = today
|
|
||||||
entry["checked_at"] = now
|
|
||||||
|
|
||||||
# 若有 ok 服务器写入可入池日期;否则保留今日以免反复重试
|
|
||||||
cfg["checked_date"] = today
|
|
||||||
cfg["note"] = (
|
|
||||||
f"当日全量测速入库({today}): 入池={len(fast)}台 "
|
|
||||||
f"(排名前{FAST_POOL_TOP_N}或延迟<{int(FAST_POOL_MAX_LATENCY*1000)}ms)"
|
|
||||||
)
|
)
|
||||||
if not checked_any:
|
except TimeoutError:
|
||||||
# 全部不可达时不写 checked_date,下次启动重试
|
# 超时:采用已完成的部分结果(部分可用也强于完全没有)
|
||||||
cfg.pop("checked_date", None)
|
log.debug("测速整体超过 %.0fs 预算,采用已完成的部分结果", self._budget)
|
||||||
|
except Exception as exc:
|
||||||
|
log.debug("测速过程异常: %s", str(exc)[:120])
|
||||||
|
elapsed = time.monotonic() - t0
|
||||||
|
|
||||||
self._file.write_text(
|
responsive = {
|
||||||
json.dumps(cfg, ensure_ascii=False, indent=2), encoding="utf-8"
|
ip: {p: lat for p, lat in per.items() if lat is not None}
|
||||||
|
for ip, per in results.items()
|
||||||
|
}
|
||||||
|
responsive = {ip: per for ip, per in responsive.items() if per}
|
||||||
|
if not responsive:
|
||||||
|
log.debug("测速结束: 无任何可达主机,不写缓存(下次启动重试)")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _ordered(protocol: str) -> list[str]:
|
||||||
|
rows = [(per[protocol], ip) for ip, per in responsive.items()
|
||||||
|
if protocol in per]
|
||||||
|
rows.sort(key=lambda kv: (kv[0], kv[1]))
|
||||||
|
return [ip for _, ip in rows]
|
||||||
|
|
||||||
|
hosts_by_proto = {p: _ordered(p) for p in ("standard", "mac", "ex")}
|
||||||
|
overall = sorted(
|
||||||
|
(min(per.values()), ip) for ip, per in responsive.items()
|
||||||
)
|
)
|
||||||
log.info("测速结果已写入 %s", self._file)
|
fast = {ip for i, (lat, ip) in enumerate(overall)
|
||||||
|
if i < FAST_POOL_TOP_N or lat < FAST_POOL_MAX_LATENCY}
|
||||||
|
|
||||||
|
log.debug("测速完成: 候选 %d 台, 可达 %d 台, fast %d 台, 耗时 %.2fs",
|
||||||
|
len({ip for _, ip in jobs}), len(responsive), len(fast), elapsed)
|
||||||
|
return {
|
||||||
|
"version": PROBE_CACHE_VERSION,
|
||||||
|
"date": self.today(),
|
||||||
|
"probed_at": time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||||
|
"elapsed": round(elapsed, 3),
|
||||||
|
# 每个协议的完整探测结果(含 None = 本次不可达),便于事后诊断
|
||||||
|
"results": {ip: dict(per) for ip, per in results.items() if per},
|
||||||
|
# 按延迟升序的可达主机(运行时即按此顺序取用)
|
||||||
|
"hosts": hosts_by_proto,
|
||||||
|
# 诊断用:符合"前N名或<200ms"的可达主机
|
||||||
|
"fast": {p: [h for h in ips if h in fast]
|
||||||
|
for p, ips in hosts_by_proto.items()},
|
||||||
|
}
|
||||||
|
|
||||||
|
def _write(self, payload: dict) -> None:
|
||||||
|
"""原子写入:先写临时文件再 ``os.replace``,避免读到半截 JSON。"""
|
||||||
|
target = self.cache_file()
|
||||||
|
tmp = target.with_name(f"{target.name}.tmp{os.getpid()}")
|
||||||
|
try:
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
tmp.write_text(json.dumps(payload, ensure_ascii=False, indent=2),
|
||||||
|
encoding="utf-8")
|
||||||
|
os.replace(tmp, target)
|
||||||
|
log.debug("当日测速结果已缓存: %s", target)
|
||||||
|
except OSError as exc:
|
||||||
|
log.debug("测速结果写入失败(忽略): %s", str(exc)[:120])
|
||||||
|
try:
|
||||||
|
tmp.unlink(missing_ok=True)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
# 进程级单例(跨池共享,确保当日仅测一次)
|
# 进程级单例(标准 / MAC / EX 三池共享,确保当日只真正测一次)
|
||||||
_probe_cache = ProbeCache()
|
_probe_cache = ProbeCache()
|
||||||
|
|
||||||
|
|
||||||
def get_probe_cache() -> ProbeCache:
|
def get_probe_cache() -> ProbeCache:
|
||||||
return _probe_cache
|
return _probe_cache
|
||||||
|
|
||||||
|
|
||||||
|
def schedule_probe(hosts: list[str] | None = None) -> asyncio.Task | None:
|
||||||
|
"""[连接池用] 把当日测速挂到后台并返回 Task,供调用方持有 / 取消。
|
||||||
|
|
||||||
|
返回 ``None`` 表示当前没有运行中的事件循环(此时无法起后台任务)。
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
return asyncio.create_task(
|
||||||
|
get_probe_cache().ensure_probe(hosts), name="pytdxdata-daily-probe"
|
||||||
|
)
|
||||||
|
except RuntimeError:
|
||||||
|
return None
|
||||||
@@ -12,18 +12,153 @@ from .base import BaseCommand
|
|||||||
MAX_QUOTES = 80
|
MAX_QUOTES = 80
|
||||||
|
|
||||||
|
|
||||||
|
# ====================================================================== #
|
||||||
|
# 权威小数位表(由证券清单 SecurityInfo.decimal_point 灌入)
|
||||||
|
# ====================================================================== #
|
||||||
|
# 查找顺序:**权威表 → 静态前缀规则**。权威表来自 ``get_universe`` 返回的
|
||||||
|
# ``SecurityInfo.decimal_point``(协议自带,见 protocol/commands/security_list.py),
|
||||||
|
# 由 ``pytdxdata.decimal_sync`` 在后台每日同步一次;未同步时静默退回静态规则,
|
||||||
|
# 行为与 0.6.5 完全一致。
|
||||||
|
_DECIMAL_PREFIX: dict[tuple[int, str], int] = {}
|
||||||
|
_DECIMAL_CODE: dict[tuple[int, str], int] = {}
|
||||||
|
_DECIMAL_MIXED: set[tuple[int, str]] = set()
|
||||||
|
_DECIMAL_SOURCE: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
def install_decimal_map(
|
||||||
|
items: "list[tuple[int, str, int]] | tuple[tuple[int, str, int], ...]",
|
||||||
|
*,
|
||||||
|
source: str = "",
|
||||||
|
) -> dict[str, int]:
|
||||||
|
"""用证券清单灌入权威小数位表。
|
||||||
|
|
||||||
|
``items`` 为 ``(market, code, decimal_point)`` 序列(直接来自
|
||||||
|
``SecurityInfo``)。按**三位代码前缀**归并成一个极小的表(全量 5 万只标的
|
||||||
|
只产生数百个前缀);若某个前缀出现**不同**小数位,则该前缀退化为逐代码查表,
|
||||||
|
只把这些例外代码存进 ``_DECIMAL_CODE``——这样即使将来交易所新增代码段,
|
||||||
|
也不会因为前缀规则过时而算错价格。
|
||||||
|
|
||||||
|
返回统计:``{"prefixes", "codes", "mixed_prefixes", "stock"}``。
|
||||||
|
"""
|
||||||
|
global _DECIMAL_SOURCE
|
||||||
|
votes: dict[tuple[int, str], dict[int, int]] = {}
|
||||||
|
per_code: dict[tuple[int, str], int] = {}
|
||||||
|
for market, code, dp in items:
|
||||||
|
c = str(code or "").strip().rstrip("\x00")
|
||||||
|
if len(c) < 3:
|
||||||
|
continue
|
||||||
|
m = int(market)
|
||||||
|
if m not in (0, 1):
|
||||||
|
continue
|
||||||
|
d = int(dp)
|
||||||
|
if d <= 0:
|
||||||
|
continue
|
||||||
|
per_code[(m, c)] = d
|
||||||
|
key = (m, c[:3])
|
||||||
|
bucket = votes.setdefault(key, {})
|
||||||
|
bucket[d] = bucket.get(d, 0) + 1
|
||||||
|
|
||||||
|
_DECIMAL_PREFIX.clear()
|
||||||
|
_DECIMAL_CODE.clear()
|
||||||
|
_DECIMAL_MIXED.clear()
|
||||||
|
for key, bucket in votes.items():
|
||||||
|
if len(bucket) == 1:
|
||||||
|
_DECIMAL_PREFIX[key] = next(iter(bucket))
|
||||||
|
else:
|
||||||
|
_DECIMAL_MIXED.add(key)
|
||||||
|
for (m, c), d in per_code.items():
|
||||||
|
if (m, c[:3]) in _DECIMAL_MIXED:
|
||||||
|
_DECIMAL_CODE[(m, c)] = d
|
||||||
|
_DECIMAL_SOURCE = source or "universe"
|
||||||
|
return decimal_map_stats()
|
||||||
|
|
||||||
|
|
||||||
|
def clear_decimal_map() -> None:
|
||||||
|
"""清空权威表(回到纯静态前缀规则;测试与排障用)。"""
|
||||||
|
global _DECIMAL_SOURCE
|
||||||
|
_DECIMAL_PREFIX.clear()
|
||||||
|
_DECIMAL_CODE.clear()
|
||||||
|
_DECIMAL_MIXED.clear()
|
||||||
|
_DECIMAL_SOURCE = ""
|
||||||
|
|
||||||
|
|
||||||
|
def decimal_map_stats() -> dict[str, int]:
|
||||||
|
"""权威表状态:已覆盖前缀数 / 例外代码数 / 混合前缀数 / 总条目数。"""
|
||||||
|
return {
|
||||||
|
"prefixes": len(_DECIMAL_PREFIX),
|
||||||
|
"codes": len(_DECIMAL_CODE),
|
||||||
|
"mixed_prefixes": len(_DECIMAL_MIXED),
|
||||||
|
"stock": len(_DECIMAL_PREFIX) + len(_DECIMAL_CODE),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def decimal_map_source() -> str:
|
||||||
|
"""权威表来源标记(``universe`` / ``cache`` / ``""``=未加载)。"""
|
||||||
|
return _DECIMAL_SOURCE
|
||||||
|
|
||||||
|
|
||||||
|
def _decimal_digits_static(market: int, code: str) -> int:
|
||||||
|
"""静态前缀规则(**兜底**,权威表缺失时使用)。"""
|
||||||
|
if market == 1: # 沪市
|
||||||
|
if code.startswith("5"):
|
||||||
|
return 3 # 基金/ETF/LOF(50/51/52/56/58 系)
|
||||||
|
if code.startswith("900"):
|
||||||
|
return 3 # B股
|
||||||
|
if code.startswith(("000", "36", "88", "99", "7", "6")):
|
||||||
|
return 2 # 指数/板块/优先股/配售/股票(60/601/603/605/688)
|
||||||
|
return 4 # 债券(01x/1xx/2xx)、回购(204xxx)、ABS
|
||||||
|
if market == 0: # 深市
|
||||||
|
if code.startswith(("080", "158", "159", "160", "161", "162", "163",
|
||||||
|
"164", "165", "166", "167", "168", "169",
|
||||||
|
"180", "181")):
|
||||||
|
return 3 # 基金/ETF/LOF/配债
|
||||||
|
if code.startswith(("000", "001", "002", "003", "070", "140",
|
||||||
|
"200", "201", "300", "301", "302", "395", "399")):
|
||||||
|
return 2 # 股票/指数/B股/优先股
|
||||||
|
return 4 # 债券(10x-14x/19x)、回购(131xxx)、ABS(50x/52x/56x)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
|
||||||
def price_decimal_digits(market: int, code: str) -> int:
|
def price_decimal_digits(market: int, code: str) -> int:
|
||||||
"""推断报价有效小数位:股票2位(/100)、指数/ETF/基金/转债/国债3位(/1000)。"""
|
"""报价小数位:**协议原始整数除以 ``10**该值``** 才是真实价格。
|
||||||
|
|
||||||
|
查找顺序:
|
||||||
|
|
||||||
|
1. **权威表**(:func:`install_decimal_map` 灌入)—— 来自证券清单里协议自带的
|
||||||
|
``SecurityInfo.decimal_point``,由 ``pytdxdata.decimal_sync`` 每日后台同步一次。
|
||||||
|
2. **静态前缀规则**(:func:`_decimal_digits_static`)—— 权威表未就绪时兜底。
|
||||||
|
四档:股票/指数/板块 ``2``、基金/ETF/B股 ``3``、债券/回购 ``4``。
|
||||||
|
|
||||||
|
为什么还要静态规则:``get_quotes`` 解析是**纯同步、无网络**的路径,解析那一刻
|
||||||
|
可能还没同步过证券清单;而现网首次安装/离线环境也必须能算出价格。
|
||||||
|
|
||||||
|
权威表与静态规则的一致性:2026-10-22 拉取全量 51,982 只(沪 27,520 + 深 24,462)
|
||||||
|
逐条比对,静态规则与 ``decimal_point`` **一一吻合、零例外**——因为全量清单里
|
||||||
|
没有任何一个三位前缀出现混合小数位。因此两者并存不会产生口径冲突。
|
||||||
|
|
||||||
|
历史修正(两次都是同一类错误:把某段前缀想当然地归错档):
|
||||||
|
|
||||||
|
- **2026-10-21**:沪市 ``000``/``8`` 被判成 3 位 → 上证指数/上证50/科创50/
|
||||||
|
板块指数**价格少 10 倍**(3713 读成 371)。它们实际是 2 位。
|
||||||
|
- **2026-10-22**:债券与回购全线偏差 —— 沪市 ``010/018/019/020/1xx/2xx``
|
||||||
|
与深市 ``10x/11x/12x/13x/14x/19x/50x/52x/56x`` 实际是 **4 位**,原先
|
||||||
|
沪市落到 2 位(**少 100 倍**)、深市落到 3 位(**少 10 倍**)。实测例:
|
||||||
|
``sh019742`` 110.228 读成 11022.8、``sh204001`` 1.385 读成 138.5、
|
||||||
|
``sz131810`` 1.38 读成 13.8。同时沪市 B 股 ``900xxx`` 是 3 位(原先 2 位)。
|
||||||
|
|
||||||
|
实证方法:拿 ``pre_close``(快照口径)除以上一交易日**日线收盘**(K线口径,
|
||||||
|
走 ``/1000`` 独立路径),比值即标度误差 —— 1.0 正确、10 少 10 倍、100 少 100 倍。
|
||||||
|
"""
|
||||||
code = (code or "").strip().rstrip("\x00")
|
code = (code or "").strip().rstrip("\x00")
|
||||||
if market == 1: # SH
|
m = int(market)
|
||||||
if code.startswith(("5", "000", "8")):
|
if m in (0, 1):
|
||||||
return 3
|
hit = _DECIMAL_PREFIX.get((m, code[:3]))
|
||||||
return 2
|
if hit is not None:
|
||||||
if market == 0: # SZ
|
return hit
|
||||||
if code.startswith("1"):
|
hit = _DECIMAL_CODE.get((m, code))
|
||||||
return 3
|
if hit is not None:
|
||||||
return 2
|
return hit
|
||||||
return 2
|
return _decimal_digits_static(m, code)
|
||||||
|
|
||||||
|
|
||||||
def format_server_time(raw: int) -> str:
|
def format_server_time(raw: int) -> str:
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import struct
|
|||||||
|
|
||||||
from ..._binary import unpack_from
|
from ..._binary import unpack_from
|
||||||
from ...models import TransactionRecord
|
from ...models import TransactionRecord
|
||||||
|
from ...symbols import has_order_count
|
||||||
from ..datetime_ import get_time
|
from ..datetime_ import get_time
|
||||||
from ..mac_frame import build_mac_request
|
from ..mac_frame import build_mac_request
|
||||||
from ..price import get_price
|
from ..price import get_price
|
||||||
@@ -32,6 +33,8 @@ class GetTransactionDataCmd(BaseCommand[list[TransactionRecord]]):
|
|||||||
pos = 2
|
pos = 2
|
||||||
last_price = 0
|
last_price = 0
|
||||||
records: list[TransactionRecord] = []
|
records: list[TransactionRecord] = []
|
||||||
|
code = self.code.decode("utf-8")
|
||||||
|
has_tc = has_order_count(self.market, code)
|
||||||
for _ in range(num):
|
for _ in range(num):
|
||||||
hour, minute, pos = get_time(body, pos)
|
hour, minute, pos = get_time(body, pos)
|
||||||
price_diff, pos = get_price(body, pos)
|
price_diff, pos = get_price(body, pos)
|
||||||
@@ -41,10 +44,12 @@ class GetTransactionDataCmd(BaseCommand[list[TransactionRecord]]):
|
|||||||
_unknown, pos = get_price(body, pos)
|
_unknown, pos = get_price(body, pos)
|
||||||
last_price += price_diff
|
last_price += price_diff
|
||||||
records.append(TransactionRecord(
|
records.append(TransactionRecord(
|
||||||
market=self.market, code=self.code.decode("utf-8"),
|
market=self.market, code=code,
|
||||||
time=f"{hour:02d}:{minute:02d}:00",
|
time=f"{hour:02d}:{minute:02d}:00",
|
||||||
price=last_price / 100.0, vol=float(vol),
|
price=last_price / 100.0, vol=float(vol),
|
||||||
trade_count=int(num_orders), bs_flag=bs_flag,
|
# 指数/ETF/LOF 该字段是「价格×1000」占位,不是笔数
|
||||||
|
trade_count=int(num_orders) if has_tc else 0,
|
||||||
|
bs_flag=bs_flag, trade_count_available=has_tc,
|
||||||
))
|
))
|
||||||
return records
|
return records
|
||||||
|
|
||||||
@@ -68,6 +73,7 @@ class GetHistoryTransactionDataCmd(BaseCommand[list[TransactionRecord]]):
|
|||||||
pos = 6 # 2(num) + 4(skip)
|
pos = 6 # 2(num) + 4(skip)
|
||||||
last_price = 0
|
last_price = 0
|
||||||
records: list[TransactionRecord] = []
|
records: list[TransactionRecord] = []
|
||||||
|
code = self.code.decode("utf-8")
|
||||||
for _ in range(num):
|
for _ in range(num):
|
||||||
hour, minute, pos = get_time(body, pos)
|
hour, minute, pos = get_time(body, pos)
|
||||||
price_diff, pos = get_price(body, pos)
|
price_diff, pos = get_price(body, pos)
|
||||||
@@ -76,10 +82,11 @@ class GetHistoryTransactionDataCmd(BaseCommand[list[TransactionRecord]]):
|
|||||||
_unknown, pos = get_price(body, pos)
|
_unknown, pos = get_price(body, pos)
|
||||||
last_price += price_diff
|
last_price += price_diff
|
||||||
records.append(TransactionRecord(
|
records.append(TransactionRecord(
|
||||||
market=self.market, code=self.code.decode("utf-8"),
|
market=self.market, code=code,
|
||||||
time=f"{hour:02d}:{minute:02d}:00",
|
time=f"{hour:02d}:{minute:02d}:00",
|
||||||
price=last_price / 100.0, vol=float(vol),
|
price=last_price / 100.0, vol=float(vol),
|
||||||
trade_count=0, bs_flag=bs_flag, # 历史无笔数
|
trade_count=0, bs_flag=bs_flag,
|
||||||
|
trade_count_available=False, # 历史通道整条链路都没有笔数字段
|
||||||
))
|
))
|
||||||
return records
|
return records
|
||||||
|
|
||||||
@@ -108,11 +115,15 @@ class MacTransactionCmd(BaseCommand[list[TransactionRecord]]):
|
|||||||
def parse_response(self, body: bytes) -> list[TransactionRecord]:
|
def parse_response(self, body: bytes) -> list[TransactionRecord]:
|
||||||
(count,) = unpack_from("<H", body, 29, "mac transaction count")
|
(count,) = unpack_from("<H", body, 29, "mac transaction count")
|
||||||
records: list[TransactionRecord] = []
|
records: list[TransactionRecord] = []
|
||||||
|
has_tc = has_order_count(self.market, self.code)
|
||||||
for i in range(count):
|
for i in range(count):
|
||||||
offset = 39 + i * 18
|
offset = 39 + i * 18
|
||||||
try:
|
try:
|
||||||
|
# trade_count 必须按**有符号** int32 解析:集合竞价段(bs_flag=8)
|
||||||
|
# 该字段实为负值,按 uint32 会读出 42 亿量级的脏数据。
|
||||||
|
# 实测(2026-10-21 sh600000):09:15:03 原始 96fcffff = -874。
|
||||||
time_sec, price, volume, trade_count, bs_flag = unpack_from(
|
time_sec, price, volume, trade_count, bs_flag = unpack_from(
|
||||||
"<IfIIH", body, offset, "mac transaction item"
|
"<IfIiH", body, offset, "mac transaction item"
|
||||||
)
|
)
|
||||||
except ValueError:
|
except ValueError:
|
||||||
break
|
break
|
||||||
@@ -121,6 +132,8 @@ class MacTransactionCmd(BaseCommand[list[TransactionRecord]]):
|
|||||||
market=self.market, code=self.code,
|
market=self.market, code=self.code,
|
||||||
time=f"{h:02d}:{m:02d}:{s:02d}",
|
time=f"{h:02d}:{m:02d}:{s:02d}",
|
||||||
price=price, vol=float(volume),
|
price=price, vol=float(volume),
|
||||||
trade_count=int(trade_count), bs_flag=bs_flag,
|
# 指数/ETF/LOF 该字段是「价格×1000」占位,不是笔数
|
||||||
|
trade_count=int(trade_count) if has_tc else 0,
|
||||||
|
bs_flag=bs_flag, trade_count_available=has_tc,
|
||||||
))
|
))
|
||||||
return records
|
return records
|
||||||
@@ -27,6 +27,7 @@ class PageSpec:
|
|||||||
ordered: bool = True # 按 start 升序拼接
|
ordered: bool = True # 按 start 升序拼接
|
||||||
min_page: int = 100 # 规避标准K线 count<100 空响应坑
|
min_page: int = 100 # 规避标准K线 count<100 空响应坑
|
||||||
start_offset: int = 0 # 起始偏移(从该偏移开始分页,避免拉取历史全量)
|
start_offset: int = 0 # 起始偏移(从该偏移开始分页,避免拉取历史全量)
|
||||||
|
newest_first: bool = False # 服务端 start 从最新往回数(0x122F 逐笔即如此)
|
||||||
|
|
||||||
|
|
||||||
@dataclass(slots=True)
|
@dataclass(slots=True)
|
||||||
@@ -94,7 +95,7 @@ class Paginator:
|
|||||||
if s.pc is not None:
|
if s.pc is not None:
|
||||||
pc, s.pc = s.pc, None
|
pc, s.pc = s.pc, None
|
||||||
await self.pool.release(pc)
|
await self.pool.release(pc)
|
||||||
return self._assemble(out, spec.ordered)
|
return self._assemble(out, spec.ordered, spec.newest_first)
|
||||||
|
|
||||||
def _plan_starts(self, spec: PageSpec) -> list[tuple[int, int]]:
|
def _plan_starts(self, spec: PageSpec) -> list[tuple[int, int]]:
|
||||||
"""切页:返回 [(start, count), ...]。"""
|
"""切页:返回 [(start, count), ...]。"""
|
||||||
@@ -158,9 +159,12 @@ class Paginator:
|
|||||||
return None, False
|
return None, False
|
||||||
return None, False
|
return None, False
|
||||||
|
|
||||||
def _assemble(self, results: list[PageResult], ordered: bool) -> list[Any]:
|
def _assemble(self, results: list[PageResult], ordered: bool,
|
||||||
|
newest_first: bool = False) -> list[Any]:
|
||||||
|
# newest_first=True:start=0 是最新一页,start 越大越旧 → 反向拼才对得上时间序。
|
||||||
|
# 单页内部服务端已按时间升序,所以只翻页序,不翻页内元素。
|
||||||
if ordered:
|
if ordered:
|
||||||
results.sort(key=lambda r: r.start)
|
results.sort(key=lambda r: r.start, reverse=newest_first)
|
||||||
out: list[Any] = []
|
out: list[Any] = []
|
||||||
for r in results:
|
for r in results:
|
||||||
out.extend(r.items)
|
out.extend(r.items)
|
||||||
|
|||||||
+113
-39
@@ -3,28 +3,43 @@
|
|||||||
读 servers.json → 并发探测每台可用性与延时(标准/MAC用7709、EX用7727)→ 写回状态。
|
读 servers.json → 并发探测每台可用性与延时(标准/MAC用7709、EX用7727)→ 写回状态。
|
||||||
|
|
||||||
用法:
|
用法:
|
||||||
python -m pytdxdata.server_probe # 全量探测
|
python -m pytdxdata.server_probe # 按申报协议全量探测
|
||||||
|
python -m pytdxdata.server_probe --discover # 额外测【每种协议】并据实修正 protocols
|
||||||
python -m pytdxdata.server_probe --group ex # 只测EX
|
python -m pytdxdata.server_probe --group ex # 只测EX
|
||||||
python -m pytdxdata.server_probe --refresh # 重新探测(status=ok的)
|
python -m pytdxdata.server_probe --refresh # 只重测status=ok的
|
||||||
|
|
||||||
|
写回的字段
|
||||||
|
----------
|
||||||
|
``status`` / ``latency`` / ``checked_at`` / ``checked_date`` / ``fast_pool``
|
||||||
|
以及顶层 ``checked_date`` / ``note``。其中 ``fast_pool`` 与运行时缓存里的 ``fast``
|
||||||
|
用同一套规则(整体延迟前 10 名,或延迟 < 200ms),供"还没有当日缓存时"的冷启动排序。
|
||||||
|
|
||||||
|
``--discover`` 会顺带剔除 ip 非法的脏记录(如历史遗留的 ``"s1"``)。
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import asyncio
|
import asyncio
|
||||||
import json
|
import json
|
||||||
import sys
|
|
||||||
import time
|
import time
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
from .config import is_valid_host
|
||||||
|
from .models import Adjust, KlinePeriod
|
||||||
from .pool.connection import TdxConnection
|
from .pool.connection import TdxConnection
|
||||||
from .pool.ex_connection import ExTdxConnection
|
from .pool.ex_connection import ExTdxConnection
|
||||||
|
from .protocol.commands.ex_proto import GetExInstrumentBarsCmd
|
||||||
from .protocol.commands.kline import GetSecurityBarsCmd
|
from .protocol.commands.kline import GetSecurityBarsCmd
|
||||||
from .protocol.commands.mac import MacKlineCmd
|
from .protocol.commands.mac import MacKlineCmd
|
||||||
from .protocol.commands.ex_proto import GetExInstrumentBarsCmd
|
|
||||||
from .models import KlinePeriod, Adjust
|
|
||||||
|
|
||||||
SERVERS_FILE = Path(__file__).resolve().parent / "servers.json"
|
SERVERS_FILE = Path(__file__).resolve().parent / "servers.json"
|
||||||
|
|
||||||
|
# 与 pool/probe_cache.py 保持同一套入池规则
|
||||||
|
FAST_POOL_TOP_N = 10
|
||||||
|
FAST_POOL_MAX_LATENCY = 0.2
|
||||||
|
|
||||||
|
ALL_PROTOCOLS = ("standard", "mac", "ex")
|
||||||
|
|
||||||
|
|
||||||
def load_servers() -> dict:
|
def load_servers() -> dict:
|
||||||
if not SERVERS_FILE.exists():
|
if not SERVERS_FILE.exists():
|
||||||
@@ -41,7 +56,7 @@ async def probe_standard(ip: str, timeout: float = 5.0) -> tuple[bool, float]:
|
|||||||
t0 = time.monotonic()
|
t0 = time.monotonic()
|
||||||
try:
|
try:
|
||||||
await conn.connect()
|
await conn.connect()
|
||||||
bars = await conn.execute(GetSecurityBarsCmd(1, "600000", 5, 0, 100))
|
bars = await conn.execute(GetSecurityBarsCmd(1, "600000", KlinePeriod.DAY, 0, 100))
|
||||||
return (bool(bars), time.monotonic() - t0)
|
return (bool(bars), time.monotonic() - t0)
|
||||||
except Exception:
|
except Exception:
|
||||||
return (False, time.monotonic() - t0)
|
return (False, time.monotonic() - t0)
|
||||||
@@ -75,25 +90,51 @@ async def probe_ex(ip: str, handshake: str, timeout: float = 5.0) -> tuple[bool,
|
|||||||
await conn.close()
|
await conn.close()
|
||||||
|
|
||||||
|
|
||||||
async def probe_one(server: dict, timeout: float) -> dict:
|
async def probe_proto(proto: str, ip: str, handshake: str, timeout: float) -> tuple[bool, float]:
|
||||||
ip = server["ip"]
|
"""按协议分派到具体探针。"""
|
||||||
protocols = server.get("protocols", [])
|
|
||||||
best_lat = None
|
|
||||||
ok = False
|
|
||||||
# 逐协议探测(一台多协议: standard/mac同端口可复用连接——简化逐协议测)
|
|
||||||
for proto in protocols:
|
|
||||||
if proto == "ex":
|
if proto == "ex":
|
||||||
r, lat = await probe_ex(ip, server.get("handshake", "setup"), timeout)
|
return await probe_ex(ip, handshake, timeout)
|
||||||
elif proto == "mac":
|
if proto == "mac":
|
||||||
r, lat = await probe_mac(ip, timeout)
|
return await probe_mac(ip, timeout)
|
||||||
|
return await probe_standard(ip, timeout)
|
||||||
|
|
||||||
|
|
||||||
|
async def probe_one(server: dict, timeout: float, discover: bool = False) -> dict:
|
||||||
|
ip = server["ip"]
|
||||||
|
declared = server.get("protocols", [])
|
||||||
|
handshake = server.get("handshake") or "setup"
|
||||||
|
# discover: 三种协议全测(EX 先按申报握手,失败再试另一种);
|
||||||
|
# 默认: 只测申报的协议(保持原行为,避免误改人工裁剪的清单)
|
||||||
|
targets = list(ALL_PROTOCOLS) if discover else list(declared)
|
||||||
|
|
||||||
|
per: dict[str, float | None] = {}
|
||||||
|
for proto in targets:
|
||||||
|
if proto == "ex":
|
||||||
|
ok, lat = await probe_ex(ip, handshake, timeout)
|
||||||
|
if not ok and discover:
|
||||||
|
# 握手模式申报错的节点:换一种再试(login <-> setup)
|
||||||
|
alt = "setup" if handshake == "login" else "login"
|
||||||
|
ok, lat = await probe_ex(ip, alt, timeout)
|
||||||
else:
|
else:
|
||||||
r, lat = await probe_standard(ip, timeout)
|
ok, lat = await probe_proto(proto, ip, handshake, timeout)
|
||||||
if r:
|
if not ok and discover:
|
||||||
ok = True
|
# discover 会据结果**收窄 protocols**,单次抖动就会误删协议 → 失败重试一次
|
||||||
best_lat = lat if best_lat is None else min(best_lat, lat)
|
ok, lat = await probe_proto(proto, ip, handshake, timeout)
|
||||||
server["status"] = "ok" if ok else "fail"
|
if ok:
|
||||||
server["latency"] = round(best_lat, 3) if best_lat is not None else None
|
per[proto] = lat
|
||||||
server["checked_at"] = time.strftime("%Y-%m-%d %H:%M:%S")
|
|
||||||
|
if discover and per:
|
||||||
|
# 据实测结果修正申报协议(保序:standard / mac / ex)
|
||||||
|
server["protocols"] = [p for p in ALL_PROTOCOLS if p in per]
|
||||||
|
|
||||||
|
server["status"] = "ok" if per else "fail"
|
||||||
|
server["latency"] = round(min(per.values()), 3) if per else None
|
||||||
|
now = time.strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
server["checked_at"] = now
|
||||||
|
server["checked_date"] = now[:10]
|
||||||
|
server.setdefault("fast_pool", False)
|
||||||
|
# 供表格打印用,不落盘
|
||||||
|
server["_per_proto"] = {p: (round(v, 3) if v is not None else None) for p, v in per.items()}
|
||||||
return server
|
return server
|
||||||
|
|
||||||
|
|
||||||
@@ -102,10 +143,21 @@ async def main() -> None:
|
|||||||
ap.add_argument("--group", choices=["standard", "mac", "ex"], default=None)
|
ap.add_argument("--group", choices=["standard", "mac", "ex"], default=None)
|
||||||
ap.add_argument("--timeout", type=float, default=5.0)
|
ap.add_argument("--timeout", type=float, default=5.0)
|
||||||
ap.add_argument("--refresh", action="store_true", help="只重测status=ok的")
|
ap.add_argument("--refresh", action="store_true", help="只重测status=ok的")
|
||||||
|
ap.add_argument("--discover", action="store_true",
|
||||||
|
help="三种协议全测并据实修正 protocols(同时剔除 ip 非法的脏记录)")
|
||||||
args = ap.parse_args()
|
args = ap.parse_args()
|
||||||
|
|
||||||
cfg = load_servers()
|
cfg = load_servers()
|
||||||
servers = cfg["servers"]
|
all_servers = cfg["servers"]
|
||||||
|
|
||||||
|
# --- 清理:ip 不合法的脏记录(历史遗留,如 "s1") ---
|
||||||
|
junk = [s.get("ip") for s in all_servers if not is_valid_host(s.get("ip"))]
|
||||||
|
if junk:
|
||||||
|
cfg["servers"] = [s for s in all_servers if is_valid_host(s.get("ip"))]
|
||||||
|
all_servers = cfg["servers"]
|
||||||
|
print("已剔除 ip 非法记录: %s" % ", ".join(map(str, junk)))
|
||||||
|
|
||||||
|
servers = all_servers
|
||||||
if args.group:
|
if args.group:
|
||||||
servers = [s for s in servers if args.group in s.get("protocols", [])]
|
servers = [s for s in servers if args.group in s.get("protocols", [])]
|
||||||
if args.refresh:
|
if args.refresh:
|
||||||
@@ -114,33 +166,55 @@ async def main() -> None:
|
|||||||
print("没有待探测的服务器")
|
print("没有待探测的服务器")
|
||||||
return
|
return
|
||||||
|
|
||||||
print("=" * 90)
|
print("=" * 96)
|
||||||
print("探测 %d 台服务器 (标准/MAC:7709, EX:7727)..." % len(servers))
|
print("探测 %d 台服务器 (标准/MAC:7709, EX:7727)%s..." % (
|
||||||
print("=" * 90)
|
len(servers), " [discover: 全协议]" if args.discover else ""))
|
||||||
|
print("=" * 96)
|
||||||
t0 = time.time()
|
t0 = time.time()
|
||||||
results = await asyncio.gather(*[probe_one(s, args.timeout) for s in servers])
|
results = await asyncio.gather(
|
||||||
|
*[probe_one(s, args.timeout, args.discover) for s in servers]
|
||||||
|
)
|
||||||
elapsed = time.time() - t0
|
elapsed = time.time() - t0
|
||||||
|
|
||||||
# 写回
|
# --- 入池标记:与运行时缓存同一规则(前N名 或 <200ms) ---
|
||||||
updated = {s["ip"]: s for s in results}
|
ok_list = [s for s in results if s["status"] == "ok"]
|
||||||
|
ok_list.sort(key=lambda s: s["latency"])
|
||||||
|
fast_ips = {s["ip"] for i, s in enumerate(ok_list)
|
||||||
|
if i < FAST_POOL_TOP_N or s["latency"] < FAST_POOL_MAX_LATENCY}
|
||||||
|
for s in ok_list:
|
||||||
|
s["fast_pool"] = s["ip"] in fast_ips
|
||||||
|
fail_list = [s for s in results if s["status"] == "fail"]
|
||||||
|
for s in fail_list:
|
||||||
|
s["fast_pool"] = False
|
||||||
|
|
||||||
|
# --- 写回(剔除内部诊断字段) ---
|
||||||
|
today = time.strftime("%Y-%m-%d")
|
||||||
|
updated = {s["ip"]: {k: v for k, v in s.items() if not k.startswith("_")} for s in results}
|
||||||
for s in cfg["servers"]:
|
for s in cfg["servers"]:
|
||||||
if s["ip"] in updated:
|
if s["ip"] in updated:
|
||||||
s.update(updated[s["ip"]])
|
s.update(updated[s["ip"]])
|
||||||
|
cfg["checked_date"] = today
|
||||||
|
cfg["version"] = today # 清单基线日期(无代码读取,仅标注本次复测时间)
|
||||||
|
cfg["note"] = "离线探针复测(%s %s): 可达=%d/%d, 入池fast_pool=%d台" % (
|
||||||
|
today, time.strftime("%H:%M"), len(ok_list), len(results), len(fast_ips))
|
||||||
save_servers(cfg)
|
save_servers(cfg)
|
||||||
|
|
||||||
ok_list = [s for s in results if s["status"] == "ok"]
|
print("\n可用: %d/%d (%.1fs) 入池: %d台" % (len(ok_list), len(results), elapsed, len(fast_ips)))
|
||||||
ok_list.sort(key=lambda s: s["latency"])
|
print("-" * 96)
|
||||||
fail_list = [s for s in results if s["status"] == "fail"]
|
|
||||||
print("\n可用: %d/%d (%.1fs)" % (len(ok_list), len(results), elapsed))
|
|
||||||
print("-" * 90)
|
|
||||||
for s in ok_list:
|
for s in ok_list:
|
||||||
print(" [OK] %-16s %-14s %-28s %.3fs %s" % (
|
per = s.get("_per_proto", {})
|
||||||
s["ip"], "/".join(s.get("protocols", [])), s["name"],
|
detail = " ".join(
|
||||||
s["latency"], s.get("handshake") or ""))
|
"%s=%s" % (p, ("%.3fs" % v) if v is not None else "-")
|
||||||
|
for p, v in per.items()
|
||||||
|
) or "/".join(s.get("protocols", []))
|
||||||
|
print(" [OK] %-16s %-24s %-24s %.3fs %s%s" % (
|
||||||
|
s["ip"], "/".join(s.get("protocols", [])), s["name"], s["latency"],
|
||||||
|
"★" if s["fast_pool"] else " ", detail))
|
||||||
if fail_list:
|
if fail_list:
|
||||||
print("\n不可用: %d台" % len(fail_list))
|
print("\n不可用: %d台" % len(fail_list))
|
||||||
for s in fail_list:
|
for s in fail_list:
|
||||||
print(" [X] %-16s %s" % (s["ip"], "/".join(s.get("protocols", []))))
|
print(" [X] %-16s %-24s %s" % (
|
||||||
|
s["ip"], "/".join(s.get("protocols", [])), s["name"]))
|
||||||
print("\n已写回 %s" % SERVERS_FILE)
|
print("\n已写回 %s" % SERVERS_FILE)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+291
-148
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"version": "2026-08-12",
|
"version": "2026-10-16",
|
||||||
"note": "当日全量测速入库(2026-09-08): 入池=9台 (排名前10或延迟<200ms)",
|
"note": "离线探针复测(2026-10-16 13:46): 可达=30/36, 入池fast_pool=10台",
|
||||||
"servers": [
|
"servers": [
|
||||||
{
|
{
|
||||||
"ip": "139.9.52.158",
|
"ip": "139.9.52.158",
|
||||||
@@ -11,11 +11,15 @@
|
|||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": null,
|
"latency": 0.172,
|
||||||
"checked_at": "2026-08-21 11:23:14",
|
"checked_at": "2026-10-16 13:46:17",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": true,
|
||||||
"checked_date": "2026-08-21"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"standard": 0.172,
|
||||||
|
"mac": 0.183
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "103.251.85.94",
|
"ip": "103.251.85.94",
|
||||||
@@ -25,42 +29,50 @@
|
|||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 0.481,
|
"latency": 0.298,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:18",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": true,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"standard": 0.316,
|
||||||
|
"mac": 0.298
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "58.67.221.146",
|
"ip": "58.67.221.146",
|
||||||
"name": "中信短线宝21",
|
"name": "中信短线宝21",
|
||||||
"protocols": [
|
"protocols": [
|
||||||
"standard",
|
|
||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": 0.254,
|
"latency": 0.236,
|
||||||
"checked_at": "2026-09-08 10:05:12",
|
"checked_at": "2026-10-16 13:46:23",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": true,
|
"fast_pool": true,
|
||||||
"checked_date": "2026-09-08"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 0.236
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "59.36.5.11",
|
"ip": "59.36.5.11",
|
||||||
"name": "安信",
|
"name": "安信",
|
||||||
"protocols": [
|
"protocols": [
|
||||||
"standard",
|
|
||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": 0.254,
|
"latency": 0.213,
|
||||||
"checked_at": "2026-09-08 10:05:12",
|
"checked_at": "2026-10-16 13:46:17",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": true,
|
"fast_pool": true,
|
||||||
"checked_date": "2026-09-08"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 0.213
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "103.221.142.82",
|
"ip": "103.221.142.82",
|
||||||
@@ -71,12 +83,17 @@
|
|||||||
"ex"
|
"ex"
|
||||||
],
|
],
|
||||||
"handshake": "setup",
|
"handshake": "setup",
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 0.487,
|
"latency": 0.316,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:13",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"standard": 0.316,
|
||||||
|
"mac": 0.329,
|
||||||
|
"ex": 0.539
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "218.106.92.182",
|
"ip": "218.106.92.182",
|
||||||
@@ -85,27 +102,30 @@
|
|||||||
"standard"
|
"standard"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "fail",
|
||||||
"latency": 0.371,
|
"latency": null,
|
||||||
"checked_at": "2026-09-08 10:05:12",
|
"checked_at": "2026-10-16 13:46:34",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": true,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-09-08"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "60.12.136.251",
|
"ip": "60.12.136.251",
|
||||||
"name": "中信短线宝22",
|
"name": "中信短线宝22",
|
||||||
"protocols": [
|
"protocols": [
|
||||||
"standard",
|
|
||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": 0.37,
|
"latency": 0.682,
|
||||||
"checked_at": "2026-09-08 10:05:12",
|
"checked_at": "2026-10-16 13:46:24",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": true,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-09-08"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 0.682
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "218.106.92.183",
|
"ip": "218.106.92.183",
|
||||||
@@ -114,42 +134,47 @@
|
|||||||
"standard"
|
"standard"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "fail",
|
||||||
"latency": 0.377,
|
"latency": null,
|
||||||
"checked_at": "2026-09-08 10:05:12",
|
"checked_at": "2026-10-16 13:46:34",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": true,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-09-08"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "60.12.136.250",
|
"ip": "60.12.136.250",
|
||||||
"name": "中信短线宝26",
|
"name": "中信短线宝26",
|
||||||
"protocols": [
|
"protocols": [
|
||||||
"standard",
|
|
||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": 0.373,
|
"latency": 0.607,
|
||||||
"checked_at": "2026-09-08 10:05:12",
|
"checked_at": "2026-10-16 13:46:23",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": true,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-09-08"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 0.607
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "115.238.90.170",
|
"ip": "115.238.90.170",
|
||||||
"name": "中信短线宝23",
|
"name": "中信短线宝23",
|
||||||
"protocols": [
|
"protocols": [
|
||||||
"standard",
|
|
||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 0.482,
|
"latency": 1.264,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:24",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 1.264
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "220.178.55.86",
|
"ip": "220.178.55.86",
|
||||||
@@ -158,12 +183,13 @@
|
|||||||
"standard"
|
"standard"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "fail",
|
||||||
"latency": 0.479,
|
"latency": null,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:34",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "220.178.55.71",
|
"ip": "220.178.55.71",
|
||||||
@@ -172,87 +198,98 @@
|
|||||||
"standard"
|
"standard"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "fail",
|
||||||
"latency": 0.47,
|
"latency": null,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:34",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "180.153.18.170",
|
"ip": "180.153.18.170",
|
||||||
"name": "中信短线宝1",
|
"name": "中信短线宝1",
|
||||||
"protocols": [
|
"protocols": [
|
||||||
"standard",
|
|
||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 0.438,
|
"latency": 0.612,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:23",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 0.612
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "115.238.90.165",
|
"ip": "115.238.90.165",
|
||||||
"name": "杭州J4",
|
"name": "杭州J4",
|
||||||
"protocols": [
|
"protocols": [
|
||||||
"standard",
|
|
||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 0.476,
|
"latency": 0.613,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:23",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 0.613
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "60.191.117.167",
|
"ip": "60.191.117.167",
|
||||||
"name": "杭州J1",
|
"name": "杭州J1",
|
||||||
"protocols": [
|
"protocols": [
|
||||||
"standard",
|
|
||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": 0.362,
|
"latency": 0.624,
|
||||||
"checked_at": "2026-09-08 10:05:12",
|
"checked_at": "2026-10-16 13:46:23",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": true,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-09-08"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 0.624
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "218.75.126.9",
|
"ip": "218.75.126.9",
|
||||||
"name": "杭州J3",
|
"name": "杭州J3",
|
||||||
"protocols": [
|
"protocols": [
|
||||||
"standard",
|
|
||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 0.481,
|
"latency": 0.592,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:23",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 0.592
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "115.238.56.198",
|
"ip": "115.238.56.198",
|
||||||
"name": "杭州J2",
|
"name": "杭州J2",
|
||||||
"protocols": [
|
"protocols": [
|
||||||
"standard",
|
|
||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": 0.362,
|
"latency": 0.624,
|
||||||
"checked_at": "2026-09-08 10:05:12",
|
"checked_at": "2026-10-16 13:46:23",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": true,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-09-08"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 0.624
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "117.34.114.16",
|
"ip": "117.34.114.16",
|
||||||
@@ -262,12 +299,16 @@
|
|||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 1.575,
|
"latency": 0.306,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:18",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"standard": 0.352,
|
||||||
|
"mac": 0.306
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "159.75.55.232",
|
"ip": "159.75.55.232",
|
||||||
@@ -276,26 +317,32 @@
|
|||||||
"standard"
|
"standard"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "fail",
|
||||||
"latency": 0.244,
|
"latency": null,
|
||||||
"checked_at": "2026-09-08 10:05:12",
|
"checked_at": "2026-10-16 13:46:18",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": true,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-09-08"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "117.34.114.17",
|
"ip": "117.34.114.17",
|
||||||
"name": "国泰君安",
|
"name": "国泰君安",
|
||||||
"protocols": [
|
"protocols": [
|
||||||
"standard"
|
"standard",
|
||||||
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 1.573,
|
"latency": 0.307,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:18",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"standard": 0.35,
|
||||||
|
"mac": 0.307
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "117.34.114.14",
|
"ip": "117.34.114.14",
|
||||||
@@ -305,12 +352,16 @@
|
|||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 0.508,
|
"latency": 0.395,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:18",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"standard": 0.395,
|
||||||
|
"mac": 0.675
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "117.34.114.15",
|
"ip": "117.34.114.15",
|
||||||
@@ -320,12 +371,16 @@
|
|||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 0.636,
|
"latency": 0.345,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:18",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"standard": 0.345,
|
||||||
|
"mac": 0.726
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "117.34.114.13",
|
"ip": "117.34.114.13",
|
||||||
@@ -335,12 +390,16 @@
|
|||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 0.608,
|
"latency": 0.326,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:18",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"standard": 0.326,
|
||||||
|
"mac": 0.743
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "117.34.114.27",
|
"ip": "117.34.114.27",
|
||||||
@@ -350,12 +409,16 @@
|
|||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 1.138,
|
"latency": 0.394,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:18",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"standard": 0.394,
|
||||||
|
"mac": 0.78
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "117.34.114.18",
|
"ip": "117.34.114.18",
|
||||||
@@ -365,12 +428,16 @@
|
|||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 0.969,
|
"latency": 0.327,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:18",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"standard": 0.327,
|
||||||
|
"mac": 0.328
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "117.34.114.20",
|
"ip": "117.34.114.20",
|
||||||
@@ -380,12 +447,16 @@
|
|||||||
"mac"
|
"mac"
|
||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "slow",
|
"status": "ok",
|
||||||
"latency": 0.526,
|
"latency": 0.348,
|
||||||
"checked_at": "2026-08-19 15:37:35",
|
"checked_at": "2026-10-16 13:46:18",
|
||||||
"source": "probe-2026-08-12",
|
"source": "probe-2026-08-12",
|
||||||
"fast_pool": false,
|
"fast_pool": false,
|
||||||
"checked_date": "2026-08-19"
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"standard": 0.348,
|
||||||
|
"mac": 0.74
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "117.34.114.30",
|
"ip": "117.34.114.30",
|
||||||
@@ -396,8 +467,11 @@
|
|||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "fail",
|
"status": "fail",
|
||||||
"latency": null,
|
"latency": null,
|
||||||
"checked_at": "2026-08-12 18:36:21",
|
"checked_at": "2026-10-16 13:46:37",
|
||||||
"source": "probe-2026-08-12"
|
"source": "probe-2026-08-12",
|
||||||
|
"checked_date": "2026-10-16",
|
||||||
|
"fast_pool": false,
|
||||||
|
"_per_proto": {}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "121.37.207.165",
|
"ip": "121.37.207.165",
|
||||||
@@ -407,9 +481,14 @@
|
|||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": 0.249,
|
"latency": 0.726,
|
||||||
"checked_at": "2026-08-12 18:36:17",
|
"checked_at": "2026-10-16 13:46:18",
|
||||||
"source": "probe-2026-08-12"
|
"source": "probe-2026-08-12",
|
||||||
|
"checked_date": "2026-10-16",
|
||||||
|
"fast_pool": false,
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 0.726
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "121.36.248.138",
|
"ip": "121.36.248.138",
|
||||||
@@ -419,9 +498,14 @@
|
|||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": 0.436,
|
"latency": 0.624,
|
||||||
"checked_at": "2026-08-12 18:36:17",
|
"checked_at": "2026-10-16 13:46:18",
|
||||||
"source": "probe-2026-08-12"
|
"source": "probe-2026-08-12",
|
||||||
|
"checked_date": "2026-10-16",
|
||||||
|
"fast_pool": false,
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 0.624
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "123.60.47.136",
|
"ip": "123.60.47.136",
|
||||||
@@ -431,9 +515,14 @@
|
|||||||
],
|
],
|
||||||
"handshake": null,
|
"handshake": null,
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": 0.476,
|
"latency": 0.56,
|
||||||
"checked_at": "2026-08-12 18:36:17",
|
"checked_at": "2026-10-16 13:46:30",
|
||||||
"source": "probe-2026-08-12"
|
"source": "probe-2026-08-12",
|
||||||
|
"checked_date": "2026-10-16",
|
||||||
|
"fast_pool": false,
|
||||||
|
"_per_proto": {
|
||||||
|
"mac": 0.56
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "116.205.135.205",
|
"ip": "116.205.135.205",
|
||||||
@@ -443,9 +532,14 @@
|
|||||||
],
|
],
|
||||||
"handshake": "login",
|
"handshake": "login",
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": 0.192,
|
"latency": 0.097,
|
||||||
"checked_at": "2026-08-12 18:36:17",
|
"checked_at": "2026-10-16 13:46:21",
|
||||||
"source": "probe-2026-08-12"
|
"source": "probe-2026-08-12",
|
||||||
|
"checked_date": "2026-10-16",
|
||||||
|
"fast_pool": true,
|
||||||
|
"_per_proto": {
|
||||||
|
"ex": 0.097
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "121.37.232.167",
|
"ip": "121.37.232.167",
|
||||||
@@ -455,9 +549,14 @@
|
|||||||
],
|
],
|
||||||
"handshake": "login",
|
"handshake": "login",
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": 0.194,
|
"latency": 0.097,
|
||||||
"checked_at": "2026-08-12 18:36:17",
|
"checked_at": "2026-10-16 13:46:21",
|
||||||
"source": "probe-2026-08-12"
|
"source": "probe-2026-08-12",
|
||||||
|
"checked_date": "2026-10-16",
|
||||||
|
"fast_pool": true,
|
||||||
|
"_per_proto": {
|
||||||
|
"ex": 0.097
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "116.205.143.214",
|
"ip": "116.205.143.214",
|
||||||
@@ -467,22 +566,66 @@
|
|||||||
],
|
],
|
||||||
"handshake": "setup",
|
"handshake": "setup",
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"latency": 0.192,
|
"latency": 0.103,
|
||||||
"checked_at": "2026-08-12 18:36:17",
|
"checked_at": "2026-10-16 13:46:21",
|
||||||
"source": "probe-2026-08-12"
|
"source": "probe-2026-08-12",
|
||||||
|
"checked_date": "2026-10-16",
|
||||||
|
"fast_pool": true,
|
||||||
|
"_per_proto": {
|
||||||
|
"ex": 0.103
|
||||||
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"ip": "s1",
|
"ip": "112.74.214.43",
|
||||||
"name": "s1",
|
"name": "扩展市场深圳双线1",
|
||||||
"protocols": [
|
"protocols": [
|
||||||
"standard"
|
"ex"
|
||||||
],
|
],
|
||||||
"status": "unknown",
|
"handshake": "login",
|
||||||
"latency": null,
|
"status": "ok",
|
||||||
"fast_pool": false,
|
"latency": 0.095,
|
||||||
"checked_date": "2026-08-19",
|
"checked_at": "2026-10-16 13:46:21",
|
||||||
"checked_at": "2026-08-19 15:34:45"
|
"source": "connect.cfg",
|
||||||
|
"fast_pool": true,
|
||||||
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"ex": 0.095
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"ip": "120.25.218.6",
|
||||||
|
"name": "扩展市场深圳双线2",
|
||||||
|
"protocols": [
|
||||||
|
"ex"
|
||||||
|
],
|
||||||
|
"handshake": "login",
|
||||||
|
"status": "ok",
|
||||||
|
"latency": 0.101,
|
||||||
|
"checked_at": "2026-10-16 13:46:21",
|
||||||
|
"source": "connect.cfg",
|
||||||
|
"fast_pool": true,
|
||||||
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"ex": 0.101
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"ip": "47.102.108.214",
|
||||||
|
"name": "扩展市场上海双线2",
|
||||||
|
"protocols": [
|
||||||
|
"ex"
|
||||||
|
],
|
||||||
|
"handshake": "login",
|
||||||
|
"status": "ok",
|
||||||
|
"latency": 0.149,
|
||||||
|
"checked_at": "2026-10-16 13:46:22",
|
||||||
|
"source": "connect.cfg",
|
||||||
|
"fast_pool": true,
|
||||||
|
"checked_date": "2026-10-16",
|
||||||
|
"_per_proto": {
|
||||||
|
"ex": 0.149
|
||||||
|
}
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"checked_date": "2026-09-08"
|
"checked_date": "2026-10-16"
|
||||||
}
|
}
|
||||||
@@ -238,3 +238,53 @@ def is_cn_index(market: int, code: str) -> bool:
|
|||||||
or (code.startswith("000") and int(market) == int(Market.SH))
|
or (code.startswith("000") and int(market) == int(Market.SH))
|
||||||
or (code.startswith("399") and int(market) == int(Market.SZ))
|
or (code.startswith("399") and int(market) == int(Market.SZ))
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def has_order_count(market: int, code: str) -> bool:
|
||||||
|
"""该标的的逐笔 ``trade_count`` 是否是**真实成交笔数**。
|
||||||
|
|
||||||
|
通达信对**指数**和**多数场内基金**不提供笔数,但那 4 字节里仍塞了值:
|
||||||
|
|
||||||
|
- 指数:``当时均价 × 1000``(见 :class:`~pytdxdata.models.TransactionRecord`)
|
||||||
|
- ETF / LOF:``价格 × 1000``;货币 ETF(如 ``sh511990``)恒为 ``0``
|
||||||
|
|
||||||
|
直接当笔数用会得到 3994 这种荒谬值(实测 ``sh510310``)。
|
||||||
|
|
||||||
|
实测覆盖(2026-10-22,按代码前缀各取样 1~7 只):
|
||||||
|
|
||||||
|
================== ========================== ==========
|
||||||
|
品种 前缀 笔数可用
|
||||||
|
================== ========================== ==========
|
||||||
|
沪股票/科创 600/601/603/605/688/689 是
|
||||||
|
沪 LOF 501/502/506 是
|
||||||
|
沪 ETF/LOF 51/52/56/58 **否**
|
||||||
|
沪指数 000/88 否
|
||||||
|
沪可转债 110/111/113/118 是
|
||||||
|
沪 EB(可交换债) 132 是
|
||||||
|
沪逆回购 204 是
|
||||||
|
深股票 000/001/002/003/300/301 是
|
||||||
|
深 ETF 159 **否**
|
||||||
|
深 LOF/基金 15/16/17/18 **否**
|
||||||
|
深指数 399 否
|
||||||
|
深可转债 123/127/128 是
|
||||||
|
================== ========================== ==========
|
||||||
|
|
||||||
|
``501x/502x/506x``(沪市 LOF)实测**有**真实笔数,所以不能用「沪市 5 开头
|
||||||
|
一律当基金」的粗规则;必须按上面这组前缀精确判定。
|
||||||
|
|
||||||
|
可转债(沪 110/111/113/118/132、深 123/127/128)**走的是兜底分支**——
|
||||||
|
它们既不在沪市 ``51/52/56/58`` 里,也不在深市 ``159/15/16/17/18`` 里,
|
||||||
|
所以两个分支都落到「是」。这不是巧合而是设计:兜底按「**排除法**」认定
|
||||||
|
基金/指数,其余一律当普通品种。实测验证(2026-10-22,16 只当日 + 8 只历史日):
|
||||||
|
笔数合计与日线量级吻合(``sh111000`` 3631 条切片 / 20443 笔,
|
||||||
|
``sz128136`` 1939 条 / 14593 笔),且 ``Σvol`` 与日线 ``vol`` 比值 **1.000000**。
|
||||||
|
"""
|
||||||
|
code = str(code)
|
||||||
|
if is_cn_index(market, code):
|
||||||
|
return False
|
||||||
|
m = int(market)
|
||||||
|
if m == int(Market.SH):
|
||||||
|
return code[:2] not in ("51", "52", "56", "58")
|
||||||
|
if m == int(Market.SZ):
|
||||||
|
return not (code[:3] == "159" or code[:2] in ("15", "16", "17", "18"))
|
||||||
|
return True
|
||||||
@@ -156,12 +156,15 @@ def _dispatch(cmd) -> object:
|
|||||||
return [{"minutes": 570, "price": 10.0, "vol": 100.0, "direction": 0,
|
return [{"minutes": 570, "price": 10.0, "vol": 100.0, "direction": 0,
|
||||||
"zengcang": 1.0}]
|
"zengcang": 1.0}]
|
||||||
if n == "MacTickChartCmd":
|
if n == "MacTickChartCmd":
|
||||||
return [MinuteBar(price=10.0, vol=100.0, hour=9, minute=30)]
|
return [MinuteBar(price=p, vol=100.0, hour=9, minute=30 + i)
|
||||||
|
for i, p in enumerate((10.0, 10.1, 10.2))]
|
||||||
if n == "SymbolTickChartCmd":
|
if n == "SymbolTickChartCmd":
|
||||||
return [{"minutes": 570, "price": 10.0, "vol": 100.0, "avg": 10.0,
|
return [{"minutes": 570, "price": 10.0, "vol": 100.0, "avg": 10.0,
|
||||||
"momentum": 0.1}]
|
"momentum": 0.1}]
|
||||||
if n == "MacChartSamplingCmd":
|
if n == "MacChartSamplingCmd":
|
||||||
return [10.0, 10.1, 10.2]
|
# 刻意与 MacTickChartCmd 的 mock 数据不同:A股由分时派生、EX 用 0x254D,
|
||||||
|
# 靠返回值即可区分实际走了哪条命令(BUG-1 回归用)。
|
||||||
|
return [99.0, 98.0]
|
||||||
# ---- 公司信息 ----
|
# ---- 公司信息 ----
|
||||||
if n == "MacSymbolInfoCmd":
|
if n == "MacSymbolInfoCmd":
|
||||||
return SymbolSnapshot(
|
return SymbolSnapshot(
|
||||||
@@ -270,6 +273,13 @@ def _make_financial_zip() -> bytes:
|
|||||||
return buf.getvalue()
|
return buf.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
def _make_financial_dat() -> bytes:
|
||||||
|
"""未压缩 .dat(服务器实际提供的形式)。"""
|
||||||
|
entry = struct.pack("<6sBL", b"600000", 1, 31)
|
||||||
|
header = struct.pack("<1hI1H3L", 0, 20260331, 1, 0, 8, 0)
|
||||||
|
return header + entry + struct.pack("<2f", 1.0, 2.0)
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# fixture
|
# fixture
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
@@ -281,8 +291,7 @@ async def td():
|
|||||||
mock.patch("pytdxdata.api.get_ex_hosts", return_value=["e1"]), \
|
mock.patch("pytdxdata.api.get_ex_hosts", return_value=["e1"]), \
|
||||||
mock.patch("pytdxdata.pool.pool.TdxConnection", new=_MockConn), \
|
mock.patch("pytdxdata.pool.pool.TdxConnection", new=_MockConn), \
|
||||||
mock.patch("pytdxdata.pool.ex_pool.ExTdxConnection", new=_MockConn), \
|
mock.patch("pytdxdata.pool.ex_pool.ExTdxConnection", new=_MockConn), \
|
||||||
mock.patch("pytdxdata.pool.pool.get_probe_cache",
|
mock.patch("pytdxdata.pool.pool.schedule_probe", return_value=None):
|
||||||
return_value=mock.MagicMock(ensure_probe=mock.AsyncMock())):
|
|
||||||
t = TdxData(cache_dir=None, pool_min=1, pool_max=2)
|
t = TdxData(cache_dir=None, pool_min=1, pool_max=2)
|
||||||
_use("start")
|
_use("start")
|
||||||
await t.start()
|
await t.start()
|
||||||
@@ -485,6 +494,69 @@ async def test_parsed_files(td):
|
|||||||
assert len(recs) == 1 and recs[0].report_date == 20260331
|
assert len(recs) == 1 and recs[0].report_date == 20260331
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# 六之二、BUG 回归(真实联网实测发现,见 tests 说明)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
async def test_financial_dat_fallback(td):
|
||||||
|
"""BUG-2 回归:索引写 ``.zip``,服务器只给未压缩 ``.dat`` 时必须能回退取到。
|
||||||
|
|
||||||
|
服务器对**不存在的**文件名不报错,而是回 0~1 字节空壳;旧实现会把空壳直接
|
||||||
|
丢给 ``zipfile`` → ``BadZipFile``。现在要求按候选名回退到 ``tdxfin/*.dat``。
|
||||||
|
"""
|
||||||
|
calls: list[str] = []
|
||||||
|
|
||||||
|
async def fake_get_file(name: str) -> bytes:
|
||||||
|
calls.append(name)
|
||||||
|
if name == "tdxfin/gpcw20260331.zip":
|
||||||
|
return b"" # 服务器没有 .zip
|
||||||
|
if name == "gpcw20260331.zip":
|
||||||
|
return b"\x00" # 裸名被路由到 MAC 通道 → 1 字节空壳
|
||||||
|
if name == "tdxfin/gpcw20260331.dat":
|
||||||
|
return _make_financial_dat()
|
||||||
|
return b""
|
||||||
|
|
||||||
|
with mock.patch.object(td, "get_file", new=fake_get_file):
|
||||||
|
recs = await td.get_financial_records_parsed("gpcw20260331.zip")
|
||||||
|
|
||||||
|
assert "tdxfin/gpcw20260331.dat" in calls, "必须回退尝试未压缩 .dat"
|
||||||
|
assert len(recs) == 1 and recs[0].report_date == 20260331
|
||||||
|
assert recs[0].code == "600000"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_financial_records_alias_not_silent(td):
|
||||||
|
"""BUG-2 回归:废弃别名取不到有效财报数据时应返回 ``b""``,而非 0~1 字节空壳。"""
|
||||||
|
import warnings
|
||||||
|
|
||||||
|
async def empty_get_file(name: str) -> bytes:
|
||||||
|
return b"\x00"
|
||||||
|
|
||||||
|
with mock.patch.object(td, "get_file", new=empty_get_file), warnings.catch_warnings():
|
||||||
|
warnings.simplefilter("ignore", DeprecationWarning)
|
||||||
|
got = await td.get_financial_records("gpcw20260331.zip")
|
||||||
|
assert got == b""
|
||||||
|
|
||||||
|
|
||||||
|
async def test_sampling_derives_from_tick_chart(td):
|
||||||
|
"""BUG-1 回归:A股 ``sampling=True`` 必须由分时(0x122D)派生。
|
||||||
|
|
||||||
|
``MacChartSamplingCmd``(0x254D) 是 **EX 协议**命令,7709 主机对它一律不响应
|
||||||
|
(实测 3 台各 10s 超时,叠加连接池重试 ≈24s 才报错)。两条路径的 mock 数据刻意
|
||||||
|
不同,以便从结果区分实际走的是哪条命令。
|
||||||
|
"""
|
||||||
|
# A股:得到分时派生值(10.0/10.1/10.2),而非 0x254D 的 99.0/98.0
|
||||||
|
a_share = await td.get_minutes("sh600000", sampling=True)
|
||||||
|
a_share_hist = await td.get_minutes("sh600000", date=20260811, sampling=True)
|
||||||
|
assert [b.price for b in a_share] == [10.0, 10.1, 10.2]
|
||||||
|
assert [b.price for b in a_share_hist] == [10.0, 10.1, 10.2]
|
||||||
|
assert all(b.vol == 0.0 and b.avg_price == 0 for b in a_share), "采样点无量/均价"
|
||||||
|
assert [b.fields["i"] for b in a_share] == [0.0, 1.0, 2.0], "点位序号在 fields['i']"
|
||||||
|
|
||||||
|
# EX:仍走 0x254D
|
||||||
|
ex = await td.get_minutes("hk00700", sampling=True)
|
||||||
|
assert [b.price for b in ex] == [99.0, 98.0]
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# 七、废弃别名(33 个)
|
# 七、废弃别名(33 个)
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
@@ -544,7 +616,11 @@ async def test_aliases_files(td):
|
|||||||
with warnings.catch_warnings():
|
with warnings.catch_warnings():
|
||||||
warnings.simplefilter("ignore", DeprecationWarning)
|
warnings.simplefilter("ignore", DeprecationWarning)
|
||||||
assert await td.get_financial_file_list() == ["abcd"]
|
assert await td.get_financial_file_list() == ["abcd"]
|
||||||
assert await td.get_financial_records("tdxfin/gpcw.txt") == b"abcd"
|
with mock.patch.object(td, "get_file",
|
||||||
|
new=mock.AsyncMock(
|
||||||
|
return_value=_make_financial_dat())):
|
||||||
|
assert await td.get_financial_records(
|
||||||
|
"gpcw20260331.zip") == _make_financial_dat()
|
||||||
assert isinstance(await td.get_company_info_category(1, "600000"), list)
|
assert isinstance(await td.get_company_info_category(1, "600000"), list)
|
||||||
assert isinstance(await td.get_company_info_content(
|
assert isinstance(await td.get_company_info_content(
|
||||||
1, "600000", "f.txt"), str)
|
1, "600000", "f.txt"), str)
|
||||||
@@ -579,7 +655,7 @@ async def test_aliases_goods(td):
|
|||||||
assert await td.get_goods_count(31) == 1 # 只有 1 条 market=31
|
assert await td.get_goods_count(31) == 1 # 只有 1 条 market=31
|
||||||
assert isinstance(await td.get_goods_quotes([(31, "00700")]), list)
|
assert isinstance(await td.get_goods_quotes([(31, "00700")]), list)
|
||||||
assert isinstance(await td.get_goods_quotes_list(31), list)
|
assert isinstance(await td.get_goods_quotes_list(31), list)
|
||||||
assert await td.get_goods_chart_sampling(31, "00700") == [10.0, 10.1, 10.2]
|
assert await td.get_goods_chart_sampling(31, "00700") == [99.0, 98.0]
|
||||||
assert isinstance(await td.get_goods_tick_chart(31, "00700"), list)
|
assert isinstance(await td.get_goods_tick_chart(31, "00700"), list)
|
||||||
assert isinstance(await td.get_goods_transaction(31, "00700"), list)
|
assert isinstance(await td.get_goods_transaction(31, "00700"), list)
|
||||||
assert isinstance(await td.get_goods_transaction_all(31, "00700"), list)
|
assert isinstance(await td.get_goods_transaction_all(31, "00700"), list)
|
||||||
|
|||||||
+140
-2
@@ -6,14 +6,19 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import io
|
import io
|
||||||
import struct
|
import struct
|
||||||
import unittest.mock as mock
|
|
||||||
import zipfile
|
import zipfile
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from pytdxdata.api import TdxData
|
from pytdxdata.api import TdxData
|
||||||
from pytdxdata.codec.block import infer_block_category, parse_block_dat
|
from pytdxdata.codec.block import infer_block_category, parse_block_dat
|
||||||
from pytdxdata.codec.financial import parse_financial_dat, parse_financial_file_list
|
from pytdxdata.codec.financial import (
|
||||||
|
extract_gpcw_dat,
|
||||||
|
looks_like_gpcw_dat,
|
||||||
|
parse_financial_dat,
|
||||||
|
parse_financial_file_list,
|
||||||
|
)
|
||||||
from pytdxdata.codec.industry import parse_tdxhy_cfg
|
from pytdxdata.codec.industry import parse_tdxhy_cfg
|
||||||
from pytdxdata.models import FinancialFileInfo, FinancialRecord, IndustryInfo, TdxBlock
|
from pytdxdata.models import FinancialFileInfo, FinancialRecord, IndustryInfo, TdxBlock
|
||||||
|
|
||||||
@@ -317,3 +322,136 @@ async def test_api_get_financial_records_parsed_no_date_in_name():
|
|||||||
new=mock.AsyncMock(return_value=zip_bytes)):
|
new=mock.AsyncMock(return_value=zip_bytes)):
|
||||||
recs = await td.get_financial_records_parsed("gpcw.zip")
|
recs = await td.get_financial_records_parsed("gpcw.zip")
|
||||||
assert recs[0].report_date == 20260331
|
assert recs[0].report_date == 20260331
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# looks_like_gpcw_dat / extract_gpcw_dat(BUG-2 修复的判据层)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_looks_like_gpcw_dat_accepts_real_dat():
|
||||||
|
assert looks_like_gpcw_dat(_valid_two_record_dat()) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_looks_like_gpcw_dat_rejects_short_shell():
|
||||||
|
"""服务器对不存在的文件名回的空壳(0~1 字节)必须判否。"""
|
||||||
|
assert looks_like_gpcw_dat(b"") is False
|
||||||
|
assert looks_like_gpcw_dat(b"\x00") is False
|
||||||
|
assert looks_like_gpcw_dat(b"abcd") is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_looks_like_gpcw_dat_rejects_zero_header():
|
||||||
|
"""长度够但 header 全零(max_count=0 / report_size=0)也要判否。"""
|
||||||
|
assert looks_like_gpcw_dat(b"\x00" * 64) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_looks_like_gpcw_dat_rejects_report_size_not_multiple_of_4():
|
||||||
|
data = _make_dat_raw([(b"600000", 1, 31)], struct.pack("<2f", 1.0, 2.0),
|
||||||
|
report_size=6)
|
||||||
|
assert looks_like_gpcw_dat(data) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_looks_like_gpcw_dat_rejects_truncated_index():
|
||||||
|
"""header 声称的条目数超出实际长度 -> 判否。"""
|
||||||
|
data = _make_dat_raw([(b"600000", 1, 31)], struct.pack("<2f", 1.0, 2.0),
|
||||||
|
max_count=100)
|
||||||
|
assert looks_like_gpcw_dat(data) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_gpcw_dat_from_uncompressed():
|
||||||
|
dat = _valid_two_record_dat()
|
||||||
|
assert extract_gpcw_dat(dat) == dat
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_gpcw_dat_from_zip():
|
||||||
|
dat = _valid_two_record_dat()
|
||||||
|
assert extract_gpcw_dat(_make_financial_zip(dat)) == dat
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_gpcw_dat_empty_and_garbage():
|
||||||
|
assert extract_gpcw_dat(b"") == b""
|
||||||
|
assert extract_gpcw_dat(b"\x00") == b""
|
||||||
|
assert extract_gpcw_dat(b"not a financial file") == b""
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_gpcw_dat_zip_without_dat_member():
|
||||||
|
assert extract_gpcw_dat(_make_financial_zip(b"x", inner_name="readme.txt")) == b""
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_gpcw_dat_zip_with_invalid_inner_dat():
|
||||||
|
"""zip 里虽有 .dat,但内容不是有效财报数据 -> 空。"""
|
||||||
|
assert extract_gpcw_dat(_make_financial_zip(b"\x00" * 4)) == b""
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_gpcw_dat_broken_zip():
|
||||||
|
"""以 PK 开头但不是合法 zip -> 空,不抛异常。"""
|
||||||
|
assert extract_gpcw_dat(b"PK\x03\x04broken-not-a-zip") == b""
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# TdxData._fetch_financial_dat 候选回退(BUG-2 回归)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fetch_financial_dat_falls_back_to_uncompressed_dat():
|
||||||
|
"""索引名 ``gpcw*.zip`` 取不到时,必须回退到服务器实际的 ``tdxfin/*.dat``。"""
|
||||||
|
td = _new_tdxdata()
|
||||||
|
dat = _valid_two_record_dat()
|
||||||
|
seen: list[str] = []
|
||||||
|
|
||||||
|
async def fake(name: str) -> bytes:
|
||||||
|
seen.append(name)
|
||||||
|
if name == "tdxfin/gpcw20260331.dat":
|
||||||
|
return dat
|
||||||
|
if name == "gpcw20260331.zip":
|
||||||
|
return b"\x00" # 裸名 -> MAC 路由 -> 1 字节空壳
|
||||||
|
return b"" # 带前缀的 .zip 不存在
|
||||||
|
|
||||||
|
with mock.patch.object(td, "get_file", new=fake):
|
||||||
|
got = await td._fetch_financial_dat("gpcw20260331.zip")
|
||||||
|
|
||||||
|
assert got == dat
|
||||||
|
assert seen == ["tdxfin/gpcw20260331.zip", "tdxfin/gpcw20260331.dat"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fetch_financial_dat_prefers_declared_zip_when_present():
|
||||||
|
"""服务器若恢复提供 .zip,应优先用索引声明的那个名字。"""
|
||||||
|
td = _new_tdxdata()
|
||||||
|
dat = _valid_two_record_dat()
|
||||||
|
seen: list[str] = []
|
||||||
|
|
||||||
|
async def fake(name: str) -> bytes:
|
||||||
|
seen.append(name)
|
||||||
|
return _make_financial_zip(dat)
|
||||||
|
|
||||||
|
with mock.patch.object(td, "get_file", new=fake):
|
||||||
|
got = await td._fetch_financial_dat("tdxfin/gpcw20260331.zip")
|
||||||
|
|
||||||
|
assert got == dat
|
||||||
|
assert seen == ["tdxfin/gpcw20260331.zip"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fetch_financial_dat_skips_raising_candidate():
|
||||||
|
"""某个候选取回时抛异常,不应中断,继续试后面的候选。"""
|
||||||
|
td = _new_tdxdata()
|
||||||
|
dat = _valid_two_record_dat()
|
||||||
|
|
||||||
|
async def fake(name: str) -> bytes:
|
||||||
|
if name.endswith(".zip"):
|
||||||
|
raise ConnectionError("模拟网络错误")
|
||||||
|
return dat
|
||||||
|
|
||||||
|
with mock.patch.object(td, "get_file", new=fake):
|
||||||
|
assert await td._fetch_financial_dat("gpcw20260331.zip") == dat
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fetch_financial_dat_all_candidates_fail_returns_empty():
|
||||||
|
td = _new_tdxdata()
|
||||||
|
|
||||||
|
async def fake(name: str) -> bytes:
|
||||||
|
return b""
|
||||||
|
|
||||||
|
with mock.patch.object(td, "get_file", new=fake):
|
||||||
|
assert await td._fetch_financial_dat("gpcw20260331.zip") == b""
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
"""ExPool 背压/并发语义回归(Issue #1:并发超限 0.15s 秒失败)。
|
||||||
|
|
||||||
|
用 mock 连接模拟:单服务器 + per_server_cap=1 时,
|
||||||
|
- 并发 acquire 超过有效并发 → 排队等待而不是快速失败;
|
||||||
|
- release 唤醒等待者;
|
||||||
|
- acquire 超时异常带池状态;
|
||||||
|
- ex_pool_cap 提升同主机并发。
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import time
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from pytdxdata.pool.ex_pool import ExPool, ExPoolTimeoutError
|
||||||
|
|
||||||
|
|
||||||
|
class MockConn:
|
||||||
|
"""Minimal stand-in for ExTdxConnection."""
|
||||||
|
|
||||||
|
last_created = 0
|
||||||
|
|
||||||
|
def __init__(self, host: str, port: int, timeout: float, handshake: str = "login"):
|
||||||
|
self.host = host
|
||||||
|
self.port = port
|
||||||
|
self.alive = True
|
||||||
|
self.last_used = time.monotonic()
|
||||||
|
self.closed = False
|
||||||
|
MockConn.last_created += 1
|
||||||
|
|
||||||
|
async def connect(self) -> None:
|
||||||
|
await asyncio.sleep(0) # instant "connect"
|
||||||
|
|
||||||
|
async def close(self) -> None:
|
||||||
|
self.closed = True
|
||||||
|
self.alive = False
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def patch_conn(monkeypatch):
|
||||||
|
monkeypatch.setattr("pytdxdata.pool.ex_pool.ExTdxConnection", MockConn)
|
||||||
|
MockConn.last_created = 0
|
||||||
|
|
||||||
|
|
||||||
|
async def test_acquire_queues_instead_of_fast_fail(patch_conn):
|
||||||
|
"""并发 > 有效并发时,后续 acquire 等待 release,而非 0.15s 抛错。"""
|
||||||
|
pool = ExPool(["e1"], min_size=1, max_size=6, per_server_cap=1)
|
||||||
|
await pool.start()
|
||||||
|
c1 = await pool.acquire() # 唯一连接被占用
|
||||||
|
t0 = time.monotonic()
|
||||||
|
|
||||||
|
async def holder():
|
||||||
|
await asyncio.sleep(0.5) # 持有 0.5s 后归还
|
||||||
|
await pool.release(c1)
|
||||||
|
|
||||||
|
asyncio.create_task(holder())
|
||||||
|
c2 = await pool.acquire() # 应在 ~0.5s 后拿到,而不是 0.15s 抛错
|
||||||
|
waited = time.monotonic() - t0
|
||||||
|
assert waited >= 0.4, f"acquire 没有等待释放(仅 {waited:.2f}s)"
|
||||||
|
assert c2 is c1 # 归还的是同一条连接
|
||||||
|
|
||||||
|
|
||||||
|
async def test_acquire_timeout_carries_pool_state(patch_conn):
|
||||||
|
"""超时异常信息带池状态(idle/total/servers/cap)。"""
|
||||||
|
pool = ExPool(["e1"], min_size=1, max_size=6, per_server_cap=1, acquire_timeout=0.3)
|
||||||
|
await pool.start()
|
||||||
|
c1 = await pool.acquire()
|
||||||
|
with pytest.raises(ExPoolTimeoutError) as ei:
|
||||||
|
await pool.acquire()
|
||||||
|
msg = str(ei.value)
|
||||||
|
assert "total=1" in msg and "per_server_cap=1" in msg and "servers=1" in msg
|
||||||
|
await pool.release(c1)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_ex_pool_cap_allows_parallel_conns_same_host(patch_conn):
|
||||||
|
"""per_server_cap=2 时单主机可并发持有 2 条连接。"""
|
||||||
|
pool = ExPool(["e1"], min_size=1, max_size=6, per_server_cap=2)
|
||||||
|
await pool.start()
|
||||||
|
got = await asyncio.gather(pool.acquire(), pool.acquire())
|
||||||
|
assert len(got) == 2 and got[0] is not got[1]
|
||||||
|
assert MockConn.last_created == 2
|
||||||
|
for c in got:
|
||||||
|
await pool.release(c)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_concurrent_acquire_all_succeed(patch_conn):
|
||||||
|
"""Issue 场景:单服务器 + cap=2,6 个并发任务应全部拿到连接(排队)而非部分失败。"""
|
||||||
|
pool = ExPool(["e1"], min_size=2, max_size=6, per_server_cap=2)
|
||||||
|
await pool.start()
|
||||||
|
|
||||||
|
async def task(i):
|
||||||
|
c = await pool.acquire()
|
||||||
|
await asyncio.sleep(0.1) # 模拟一次 1m 分页拉取
|
||||||
|
await pool.release(c)
|
||||||
|
return i
|
||||||
|
|
||||||
|
results = await asyncio.gather(*[task(i) for i in range(6)])
|
||||||
|
assert sorted(results) == list(range(6)) # 全部成功,零失败
|
||||||
|
|
||||||
|
|
||||||
|
async def test_release_notifies_waiters(patch_conn):
|
||||||
|
"""broken 归还(连接死亡)也要唤醒等待者(槽位回收后可扩容)。"""
|
||||||
|
pool = ExPool(["e1"], min_size=1, max_size=1, per_server_cap=1)
|
||||||
|
await pool.start()
|
||||||
|
c1 = await pool.acquire()
|
||||||
|
c1.alive = False
|
||||||
|
|
||||||
|
async def broken_release():
|
||||||
|
await asyncio.sleep(0.2)
|
||||||
|
await pool.release(c1, broken=True)
|
||||||
|
|
||||||
|
asyncio.create_task(broken_release())
|
||||||
|
c2 = await pool.acquire() # 槽位被回收→扩容新建
|
||||||
|
assert c2 is not c1
|
||||||
|
await pool.release(c2)
|
||||||
@@ -0,0 +1,365 @@
|
|||||||
|
"""当日测速缓存单元测试(注入假探针与假日期,不连真实网络)
|
||||||
|
|
||||||
|
覆盖点:
|
||||||
|
- 当日首次启动测一次;当天已有缓存则**不刷新**(探针一次都不发起)
|
||||||
|
- 跨日 / 缓存损坏 / 版本不符 → 重新测速
|
||||||
|
- 结果按实测延迟升序被 config 采用
|
||||||
|
- **绝不改写包内 servers.json**,缓存落在用户级缓存目录
|
||||||
|
- 全部不可达 → 不写缓存(下次启动重试)
|
||||||
|
- 并发调用只真正测一次(单飞)
|
||||||
|
- 非法主机名(测试 mock 混入的 "s1" 等)被挡在测速之外
|
||||||
|
- 连接池 start() 不阻塞、close() 会取消后台测速
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
from datetime import date, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from pytdxdata import config
|
||||||
|
from pytdxdata.pool.probe_cache import ProbeCache
|
||||||
|
|
||||||
|
TODAY = date.today().isoformat()
|
||||||
|
YESTERDAY = (date.today() - timedelta(days=1)).isoformat()
|
||||||
|
PROTOS = ("standard", "mac", "ex")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def isolated_cache(tmp_path, monkeypatch):
|
||||||
|
"""把用户级缓存目录指到临时目录,并复位所有相关环境变量。"""
|
||||||
|
monkeypatch.setenv("PYPYTDX_DATA_CACHE_DIR", str(tmp_path))
|
||||||
|
monkeypatch.delenv("PYPYTDX_DATA_DISABLE_PROBE", raising=False)
|
||||||
|
for var in ("PYPYTDX_DATA_STANDARD_HOSTS", "PYPYTDX_DATA_MAC_HOSTS",
|
||||||
|
"PYPYTDX_DATA_EX_HOSTS"):
|
||||||
|
monkeypatch.delenv(var, raising=False)
|
||||||
|
return tmp_path
|
||||||
|
|
||||||
|
|
||||||
|
def fake_probers(latencies: dict[str, dict[str, float | None]], calls: list):
|
||||||
|
"""构造假探针;latencies = {协议: {主机: 延迟或 None}}。"""
|
||||||
|
def _make(protocol: str):
|
||||||
|
async def _probe(host: str, timeout: float) -> float | None:
|
||||||
|
calls.append((protocol, host))
|
||||||
|
return latencies.get(protocol, {}).get(host)
|
||||||
|
return _probe
|
||||||
|
return {p: _make(p) for p in PROTOS}
|
||||||
|
|
||||||
|
|
||||||
|
def patch_candidates(monkeypatch, mapping: dict[str, list[str]]) -> None:
|
||||||
|
"""把"全量候选"换成受控清单,避免依赖真实服务器清单。"""
|
||||||
|
monkeypatch.setattr("pytdxdata.pool.probe_cache.all_probe_candidates",
|
||||||
|
lambda: mapping)
|
||||||
|
|
||||||
|
|
||||||
|
def write_cache(path: Path, day: str, results: dict, version: int = 1) -> None:
|
||||||
|
path.write_text(json.dumps({"version": version, "date": day, "results": results}),
|
||||||
|
encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 是否该测速 ----------
|
||||||
|
|
||||||
|
def test_needs_probe_when_no_cache(tmp_path):
|
||||||
|
pc = ProbeCache(tmp_path / "c.json", today_fn=lambda: TODAY)
|
||||||
|
assert pc.needs_probe_today() is True
|
||||||
|
assert pc.read_cache() is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_needs_probe_false_for_today_cache(tmp_path):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
write_cache(cache, TODAY, {"1.1.1.1": {"standard": 0.05}})
|
||||||
|
pc = ProbeCache(cache, today_fn=lambda: TODAY)
|
||||||
|
assert pc.needs_probe_today() is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_stale_cache_needs_probe(tmp_path):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
write_cache(cache, YESTERDAY, {"1.1.1.1": {"standard": 0.05}})
|
||||||
|
pc = ProbeCache(cache, today_fn=lambda: TODAY)
|
||||||
|
assert pc.needs_probe_today() is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_version_mismatch_needs_probe(tmp_path):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
write_cache(cache, TODAY, {"1.1.1.1": {"standard": 0.05}}, version=999)
|
||||||
|
pc = ProbeCache(cache, today_fn=lambda: TODAY)
|
||||||
|
assert pc.needs_probe_today() is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("content", ["{not json", "[]", '{"version":1}', ""])
|
||||||
|
def test_broken_cache_is_ignored_not_crash(tmp_path, content):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
cache.write_text(content, encoding="utf-8")
|
||||||
|
pc = ProbeCache(cache, today_fn=lambda: TODAY)
|
||||||
|
assert pc.read_cache() is None
|
||||||
|
assert pc.needs_probe_today() is True
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 核心行为:当天已有缓存就不刷新 ----------
|
||||||
|
|
||||||
|
async def test_today_cache_is_not_refreshed(tmp_path, monkeypatch):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
write_cache(cache, TODAY, {"1.1.1.1": {"standard": 0.05}})
|
||||||
|
before = cache.read_text(encoding="utf-8")
|
||||||
|
calls: list = []
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
|
||||||
|
pc = ProbeCache(cache, probers=fake_probers({}, calls), today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
|
||||||
|
assert calls == [] # 未发起任何测速
|
||||||
|
assert cache.read_text(encoding="utf-8") == before # 缓存内容原封不动
|
||||||
|
|
||||||
|
|
||||||
|
async def test_stale_cache_is_refreshed(tmp_path, monkeypatch):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
write_cache(cache, YESTERDAY, {"9.9.9.9": {"standard": 9.0}})
|
||||||
|
calls: list = []
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
|
||||||
|
pc = ProbeCache(cache,
|
||||||
|
probers=fake_probers({"standard": {"1.1.1.1": 0.02}}, calls),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
|
||||||
|
assert calls == [("standard", "1.1.1.1")]
|
||||||
|
data = json.loads(cache.read_text(encoding="utf-8"))
|
||||||
|
assert data["date"] == TODAY and data["version"] == config.PROBE_CACHE_VERSION
|
||||||
|
assert data["results"] == {"1.1.1.1": {"standard": 0.02}}
|
||||||
|
assert data["hosts"]["standard"] == ["1.1.1.1"]
|
||||||
|
assert "9.9.9.9" not in data["results"] # 旧结果被整体替换,不残留
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 缓存写入 ----------
|
||||||
|
|
||||||
|
async def test_write_is_atomic_and_leaves_no_tmp(tmp_path, monkeypatch):
|
||||||
|
cache = tmp_path / "sub" / "c.json" # 目录不存在也要能建
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
pc = ProbeCache(cache,
|
||||||
|
probers=fake_probers({"standard": {"1.1.1.1": 0.03}}, []),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
|
||||||
|
assert cache.exists()
|
||||||
|
assert list(cache.parent.glob("*.tmp*")) == []
|
||||||
|
assert json.loads(cache.read_text(encoding="utf-8"))["probed_at"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_all_unreachable_writes_nothing(tmp_path, monkeypatch):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
pc = ProbeCache(cache, probers=fake_probers({}, []), today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
|
||||||
|
assert not cache.exists() # 不把"全挂"缓存一整天
|
||||||
|
assert pc.needs_probe_today() is True
|
||||||
|
|
||||||
|
|
||||||
|
async def test_never_touches_package_servers_json(tmp_path, monkeypatch):
|
||||||
|
"""最关键的一条:测速**绝不能**改写随包分发的 servers.json。"""
|
||||||
|
servers_json = config._SERVERS_FILE
|
||||||
|
before_bytes = servers_json.read_bytes()
|
||||||
|
before_mtime = servers_json.stat().st_mtime_ns
|
||||||
|
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
pc = ProbeCache(probers=fake_probers({"standard": {"1.1.1.1": 0.04}}, []),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
|
||||||
|
assert servers_json.read_bytes() == before_bytes
|
||||||
|
assert servers_json.stat().st_mtime_ns == before_mtime
|
||||||
|
assert pc.cache_file() == tmp_path / "hosts-today.json" # 落在用户缓存目录
|
||||||
|
assert Path(config.probe_cache_file()).parent == tmp_path
|
||||||
|
|
||||||
|
|
||||||
|
async def test_results_are_recorded_per_protocol(tmp_path, monkeypatch):
|
||||||
|
cache = tmp_path / "c.json"
|
||||||
|
patch_candidates(monkeypatch, {
|
||||||
|
"standard": ["1.1.1.1", "2.2.2.2"],
|
||||||
|
"mac": ["1.1.1.1", "2.2.2.2"],
|
||||||
|
"ex": ["3.3.3.3"],
|
||||||
|
})
|
||||||
|
lats = {"standard": {"1.1.1.1": 0.30, "2.2.2.2": 0.10},
|
||||||
|
"mac": {"1.1.1.1": 0.20}, # 2.2.2.2 的 mac 故意不给 → 应记为 None
|
||||||
|
"ex": {"3.3.3.3": 0.05}}
|
||||||
|
pc = ProbeCache(cache, probers=fake_probers(lats, []), today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
|
||||||
|
data = json.loads(cache.read_text(encoding="utf-8"))
|
||||||
|
assert data["hosts"]["standard"] == ["2.2.2.2", "1.1.1.1"] # 按延迟升序
|
||||||
|
assert data["hosts"]["mac"] == ["1.1.1.1"] # 不可达的 2.2.2.2 被排除
|
||||||
|
assert data["hosts"]["ex"] == ["3.3.3.3"]
|
||||||
|
# 2.2.2.2 的 mac 探针返回 None,也要如实记录(便于事后诊断)
|
||||||
|
assert data["results"]["2.2.2.2"]["mac"] is None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 单飞 / 开关 / 脏数据 ----------
|
||||||
|
|
||||||
|
async def test_concurrent_ensure_probe_runs_once(tmp_path, monkeypatch):
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1", "2.2.2.2"], "mac": [], "ex": []})
|
||||||
|
calls: list = []
|
||||||
|
pc = ProbeCache(tmp_path / "c.json",
|
||||||
|
probers=fake_probers({"standard": {"1.1.1.1": 0.01,
|
||||||
|
"2.2.2.2": 0.02}}, calls),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
await asyncio.gather(pc.ensure_probe(), pc.ensure_probe(), pc.ensure_probe())
|
||||||
|
|
||||||
|
assert sorted(calls) == [("standard", "1.1.1.1"), ("standard", "2.2.2.2")]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_disabled_env_skips_everything(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("PYPYTDX_DATA_DISABLE_PROBE", "1")
|
||||||
|
calls: list = []
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
pc = ProbeCache(tmp_path / "c.json",
|
||||||
|
probers=fake_probers({"standard": {"1.1.1.1": 0.01}}, calls),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
|
||||||
|
assert pc.needs_probe_today() is False
|
||||||
|
await pc.ensure_probe()
|
||||||
|
assert calls == []
|
||||||
|
assert not (tmp_path / "c.json").exists()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_illegal_hosts_are_filtered_out(tmp_path, monkeypatch):
|
||||||
|
"""测试 mock 常传入 "s1"/"m1" 这类伪主机名,必须挡在缓存之外。"""
|
||||||
|
patch_candidates(monkeypatch, {"standard": [], "mac": [], "ex": []})
|
||||||
|
calls: list = []
|
||||||
|
pc = ProbeCache(tmp_path / "c.json",
|
||||||
|
probers=fake_probers({"standard": {"1.1.1.1": 0.01}}, calls),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe(["s1", "m1", "1.1.1.1", ""])
|
||||||
|
|
||||||
|
assert calls == [("standard", "1.1.1.1")]
|
||||||
|
data = json.loads((tmp_path / "c.json").read_text(encoding="utf-8"))
|
||||||
|
assert "s1" not in data["results"] and "m1" not in data["results"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_reset_allows_recheck(tmp_path, monkeypatch):
|
||||||
|
patch_candidates(monkeypatch, {"standard": ["1.1.1.1"], "mac": [], "ex": []})
|
||||||
|
calls: list = []
|
||||||
|
pc = ProbeCache(tmp_path / "c.json",
|
||||||
|
probers=fake_probers({"standard": {"1.1.1.1": 0.01}}, calls),
|
||||||
|
today_fn=lambda: TODAY)
|
||||||
|
await pc.ensure_probe()
|
||||||
|
await pc.ensure_probe()
|
||||||
|
assert len(calls) == 1 # 第二次被单飞挡掉
|
||||||
|
|
||||||
|
pc.reset()
|
||||||
|
await pc.ensure_probe()
|
||||||
|
assert len(calls) == 1 # 重置后当天已有缓存,仍不重测
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- config 侧:采用顺序与兜底 ----------
|
||||||
|
|
||||||
|
def test_config_getters_prefer_cache_and_order_by_latency(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("PYPYTDX_DATA_CACHE_DIR", str(tmp_path))
|
||||||
|
(tmp_path / "hosts-today.json").write_text(json.dumps({
|
||||||
|
"version": config.PROBE_CACHE_VERSION, "date": TODAY,
|
||||||
|
"results": {
|
||||||
|
"3.3.3.3": {"standard": 0.9},
|
||||||
|
"1.1.1.1": {"standard": 0.05},
|
||||||
|
"2.2.2.2": {"standard": 0.4, "mac": 0.1},
|
||||||
|
"9.9.9.9": {"ex": 0.2},
|
||||||
|
},
|
||||||
|
}), encoding="utf-8")
|
||||||
|
|
||||||
|
assert config.get_standard_hosts() == ["1.1.1.1", "2.2.2.2", "3.3.3.3"]
|
||||||
|
assert config.get_mac_hosts() == ["2.2.2.2"]
|
||||||
|
assert config.get_ex_hosts() == ["9.9.9.9"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_config_getters_fall_back_without_cache(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("PYPYTDX_DATA_CACHE_DIR", str(tmp_path))
|
||||||
|
assert config.get_standard_hosts() == config._load_ok_hosts("standard", fast_only=True)
|
||||||
|
assert config.get_mac_hosts() == config._load_ok_hosts("mac", fast_only=True)
|
||||||
|
assert config.get_ex_hosts() == config._load_ok_hosts("ex")
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_override_beats_cache(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("PYPYTDX_DATA_CACHE_DIR", str(tmp_path))
|
||||||
|
(tmp_path / "hosts-today.json").write_text(json.dumps({
|
||||||
|
"version": config.PROBE_CACHE_VERSION, "date": TODAY,
|
||||||
|
"results": {"1.1.1.1": {"standard": 0.05}},
|
||||||
|
}), encoding="utf-8")
|
||||||
|
monkeypatch.setenv("PYPYTDX_DATA_STANDARD_HOSTS", "8.8.8.8")
|
||||||
|
assert config.get_standard_hosts() == ["8.8.8.8"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_all_probe_candidates_covers_full_set_not_filtered():
|
||||||
|
cand = config.all_probe_candidates()
|
||||||
|
assert set(config.STANDARD_HOSTS) <= set(cand["standard"])
|
||||||
|
assert set(config.MAC_HOSTS) <= set(cand["mac"])
|
||||||
|
assert set(config.EX_HOSTS) <= set(cand["ex"])
|
||||||
|
assert "117.34.114.17" in cand["standard"] # 未申报 MAC 的节点也要被测到
|
||||||
|
assert "s1" not in cand["standard"] # 包内清单里的脏数据被挡
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(("host", "expected"), [
|
||||||
|
("1.2.3.4", True),
|
||||||
|
("117.34.114.17", True),
|
||||||
|
("a.b", True),
|
||||||
|
("good-host.com", True),
|
||||||
|
("s1", False),
|
||||||
|
("m1", False),
|
||||||
|
("", False),
|
||||||
|
(" ", False),
|
||||||
|
("-bad.com", False),
|
||||||
|
("bad-.com", False),
|
||||||
|
("has space.com", False),
|
||||||
|
(123, False),
|
||||||
|
(None, False),
|
||||||
|
])
|
||||||
|
def test_is_valid_host(host, expected):
|
||||||
|
assert config.is_valid_host(host) is expected
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 连接池集成 ----------
|
||||||
|
|
||||||
|
async def test_pool_start_does_not_block_and_close_cancels_probe(monkeypatch):
|
||||||
|
from pytdxdata.pool import pool as pool_mod
|
||||||
|
|
||||||
|
cancel_seen = asyncio.Event()
|
||||||
|
created: list[asyncio.Task] = []
|
||||||
|
|
||||||
|
async def _hang(hosts=None):
|
||||||
|
try:
|
||||||
|
await asyncio.sleep(30) # 模拟耗时测速
|
||||||
|
except asyncio.CancelledError:
|
||||||
|
cancel_seen.set()
|
||||||
|
raise
|
||||||
|
|
||||||
|
def _fake_schedule(hosts=None):
|
||||||
|
task = asyncio.create_task(_hang())
|
||||||
|
created.append(task)
|
||||||
|
return task
|
||||||
|
|
||||||
|
monkeypatch.setattr(pool_mod, "schedule_probe", _fake_schedule)
|
||||||
|
|
||||||
|
p = pool_mod.DynamicPool([], min_size=1, max_size=1)
|
||||||
|
t0 = time.monotonic()
|
||||||
|
await p.start()
|
||||||
|
assert time.monotonic() - t0 < 1.0 # start() 不等测速
|
||||||
|
assert p._probe_task is created[0]
|
||||||
|
|
||||||
|
await asyncio.sleep(0) # 让测速任务真正跑起来再关闭
|
||||||
|
await p.close()
|
||||||
|
assert p._probe_task is None
|
||||||
|
assert created[0].cancelled()
|
||||||
|
assert cancel_seen.is_set()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_pool_auto_probe_false_schedules_nothing(monkeypatch):
|
||||||
|
from pytdxdata.pool import pool as pool_mod
|
||||||
|
|
||||||
|
def _boom(hosts=None): # pragma: no cover - 不应被调用
|
||||||
|
raise AssertionError("auto_probe=False 时不应调度测速")
|
||||||
|
|
||||||
|
monkeypatch.setattr(pool_mod, "schedule_probe", _boom)
|
||||||
|
p = pool_mod.DynamicPool([], min_size=1, max_size=1, auto_probe=False)
|
||||||
|
await p.start()
|
||||||
|
assert p._probe_task is None
|
||||||
|
await p.close()
|
||||||
+114
-4
@@ -2,17 +2,20 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import struct
|
import struct
|
||||||
|
from collections import Counter
|
||||||
|
from datetime import date, datetime
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from pytdxdata.protocol.frame import FrameHeader, decompress_body, parse_header
|
from pytdxdata.models import Adjust, KlinePeriod, SecurityBar
|
||||||
from pytdxdata.protocol.price import get_price, put_price, decode_ohlc_diffs
|
|
||||||
from pytdxdata.protocol.volume import decode_volume
|
|
||||||
from pytdxdata.protocol.commands.ex_proto import GetExInstrumentBarsCmd
|
from pytdxdata.protocol.commands.ex_proto import GetExInstrumentBarsCmd
|
||||||
from pytdxdata.protocol.commands.kline import GetSecurityBarsCmd
|
from pytdxdata.protocol.commands.kline import GetSecurityBarsCmd
|
||||||
|
from pytdxdata.protocol.commands.mac import MacKlineCmd
|
||||||
from pytdxdata.protocol.commands.minute import GetHistoryMinuteTimeDataCmd
|
from pytdxdata.protocol.commands.minute import GetHistoryMinuteTimeDataCmd
|
||||||
from pytdxdata.protocol.commands.transaction import MacTransactionCmd
|
from pytdxdata.protocol.commands.transaction import MacTransactionCmd
|
||||||
|
from pytdxdata.protocol.frame import FrameHeader, decompress_body, parse_header
|
||||||
|
from pytdxdata.protocol.price import decode_ohlc_diffs, get_price, put_price
|
||||||
|
from pytdxdata.protocol.volume import decode_volume
|
||||||
|
|
||||||
# ---------- 变长有符号整数 ----------
|
# ---------- 变长有符号整数 ----------
|
||||||
|
|
||||||
@@ -176,3 +179,110 @@ def test_mac_transaction_parse():
|
|||||||
assert recs[0].trade_count == 3
|
assert recs[0].trade_count == 3
|
||||||
assert recs[0].bs_flag == 1
|
assert recs[0].bs_flag == 1
|
||||||
assert recs[1].trade_count == 1
|
assert recs[1].trade_count == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 跨时区时间戳归一化(美股 EX 通道) ----------
|
||||||
|
#
|
||||||
|
# 通达信 EX 通道对美股返回的时间戳把两个时区拼在一起:日期字段是**美东交易日**、
|
||||||
|
# 时分字段是**北京时间**。直拼出的 datetime 会把同一交易日的下午半场(北京
|
||||||
|
# 00:00~04:00)排到上午半场(北京 21:30~23:59)之前,单日 390 根全部错位。
|
||||||
|
|
||||||
|
def _bar(market: int, y: int, mo: int, d: int, h: int, mi: int) -> SecurityBar:
|
||||||
|
return SecurityBar(market=market, code="X", open=1.0, high=1.0, low=1.0,
|
||||||
|
close=1.0, vol=1.0, amount=1.0,
|
||||||
|
year=y, month=mo, day=d, hour=h, minute=mi)
|
||||||
|
|
||||||
|
|
||||||
|
def test_us_evening_segment_keeps_date():
|
||||||
|
# 北京 21:31 ~ 23:59 与美东交易日属同一自然日,日期不动
|
||||||
|
b = _bar(74, 2026, 10, 12, 21, 31)
|
||||||
|
assert b.datetime == datetime(2026, 10, 12, 21, 31)
|
||||||
|
assert b.trading_date == "2026-10-12"
|
||||||
|
|
||||||
|
|
||||||
|
def test_us_midnight_segment_rolls_to_next_day():
|
||||||
|
# 北京次日 00:00 ~ 04:00 实际是美东 12:00~16:00,日期要 +1 天
|
||||||
|
b = _bar(74, 2026, 10, 12, 4, 0)
|
||||||
|
assert b.datetime == datetime(2026, 10, 13, 4, 0)
|
||||||
|
assert (b.hour, b.minute) == (4, 0) # 时分不参与归一化
|
||||||
|
assert b.trading_date == "2026-10-12" # 仍归属美东 10-12 交易日
|
||||||
|
|
||||||
|
|
||||||
|
def test_us_split_hour_boundary():
|
||||||
|
assert _bar(74, 2026, 10, 12, 11, 59).day == 13
|
||||||
|
assert _bar(74, 2026, 10, 12, 12, 0).day == 12
|
||||||
|
|
||||||
|
|
||||||
|
def test_us_month_and_year_rollover():
|
||||||
|
assert _bar(74, 2026, 10, 31, 4, 0).datetime == datetime(2026, 11, 1, 4, 0)
|
||||||
|
assert _bar(74, 2026, 12, 31, 4, 0).datetime == datetime(2027, 1, 1, 4, 0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_cn_and_hk_bars_untouched():
|
||||||
|
cn = _bar(1, 2026, 10, 12, 9, 31) # A股/标准通道
|
||||||
|
assert (cn.year, cn.month, cn.day) == (2026, 10, 12)
|
||||||
|
assert cn.trading_date == "2026-10-12"
|
||||||
|
hk = _bar(31, 2026, 10, 12, 4, 0) # 港股同为 UTC+8,凌晨段不应 +1
|
||||||
|
assert (hk.year, hk.month, hk.day) == (2026, 10, 12)
|
||||||
|
assert hk.trading_date == "2026-10-12"
|
||||||
|
|
||||||
|
|
||||||
|
def test_us_full_session_is_monotonic_and_single_trading_day():
|
||||||
|
# 实测口径:北京 21:31~23:59(149 根)+ 次日 00:00~04:00(241 根)= 390 根
|
||||||
|
mins = list(range(21 * 60 + 31, 24 * 60)) + list(range(4 * 60 + 1))
|
||||||
|
assert len(mins) == 390
|
||||||
|
bars = [_bar(74, 2026, 10, 12, m // 60, m % 60) for m in mins]
|
||||||
|
bars.sort(key=lambda b: b.datetime)
|
||||||
|
dts = [b.datetime for b in bars]
|
||||||
|
assert dts == sorted(dts) # 归一化后严格单调
|
||||||
|
assert dts[0] == datetime(2026, 10, 12, 21, 31)
|
||||||
|
assert dts[-1] == datetime(2026, 10, 13, 4, 0)
|
||||||
|
# 全部归属于同一个美东交易日
|
||||||
|
assert {b.trading_date for b in bars} == {"2026-10-12"}
|
||||||
|
# 但横跨两个北京自然日(这正是不能用 datetime.date() 聚合的原因)
|
||||||
|
by_day = Counter(b.datetime.date() for b in bars)
|
||||||
|
assert by_day[date(2026, 10, 12)] == 149
|
||||||
|
assert by_day[date(2026, 10, 13)] == 241
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- MAC 通道 K线(0x122E)同样归一化 ----------
|
||||||
|
|
||||||
|
def _mac_bars_body(records: list[tuple[int, int]]) -> bytes:
|
||||||
|
"""构造 0x122E 响应体:33B头(count@24) + 每条36B(4B ymd + 4B time秒 + 7float)。"""
|
||||||
|
body = bytearray(33 + len(records) * 36)
|
||||||
|
struct.pack_into("<HBHI", body, 24, 7, 1, len(records), 0)
|
||||||
|
for i, (ymd, tsec) in enumerate(records):
|
||||||
|
struct.pack_into("<II7f", body, 33 + i * 36,
|
||||||
|
ymd, tsec, 10.0, 10.5, 9.5, 10.2, 0.0, 1000.0, 0.0)
|
||||||
|
return bytes(body)
|
||||||
|
|
||||||
|
|
||||||
|
def test_mac_kline_us_bar_normalized():
|
||||||
|
# _probe_bar(日期区间二分依赖它)对美股走的就是这条命令
|
||||||
|
cmd = MacKlineCmd(74, "AAPL", KlinePeriod.MIN_1, Adjust.NONE, 0, 5)
|
||||||
|
bars = cmd.parse_response(_mac_bars_body([(20261012, 4 * 3600)])) # 北京 04:00
|
||||||
|
assert len(bars) == 1
|
||||||
|
assert bars[0].datetime == datetime(2026, 10, 13, 4, 0)
|
||||||
|
assert bars[0].trading_date == "2026-10-12"
|
||||||
|
|
||||||
|
|
||||||
|
def test_mac_kline_hk_bar_untouched():
|
||||||
|
cmd = MacKlineCmd(31, "00700", KlinePeriod.MIN_1, Adjust.NONE, 0, 5)
|
||||||
|
bars = cmd.parse_response(_mac_bars_body([(20260729, 13 * 3600 + 40 * 60)]))
|
||||||
|
assert (bars[0].datetime) == datetime(2026, 7, 29, 13, 40)
|
||||||
|
assert bars[0].trading_date == "2026-07-29"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- 排序:EX 返回顺序按时间正序(切片取「最新 N 根」依赖它) ----------
|
||||||
|
|
||||||
|
def test_finish_bars_sorts_us_session_chronologically():
|
||||||
|
from pytdxdata.api import TdxData
|
||||||
|
from pytdxdata.symbols import parse_symbols
|
||||||
|
|
||||||
|
sym = parse_symbols("usAAPL")[0]
|
||||||
|
# 故意按错误的顺序给(凌晨段在前)
|
||||||
|
out = TdxData._finish_bars([_bar(74, 2026, 10, 12, 4, 0),
|
||||||
|
_bar(74, 2026, 10, 12, 21, 31)], sym)
|
||||||
|
assert [b.datetime for b in out] == [datetime(2026, 10, 12, 21, 31),
|
||||||
|
datetime(2026, 10, 13, 4, 0)]
|
||||||
|
assert {b.symbol for b in out} == {"usAAPL"}
|
||||||
Reference in new issue
Block a user