Compare commits
6
Commits
2cf3657f4b
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7c4541a315 | ||
|
|
1cd8f692a1 | ||
|
|
dea39db441 | ||
|
|
c9722943e2 | ||
|
|
8c25feca18 | ||
|
|
04e0c57569 |
No files matched your search
+246
-1
@@ -2,6 +2,248 @@
|
||||
|
||||
本项目遵循 [语义化版本](https://semver.org/lang/zh-CN/)。
|
||||
|
||||
## [0.7.2] - 2026-10-30
|
||||
|
||||
### 可靠性
|
||||
|
||||
- **服务器中途掉线不再拖垮整批任务**。
|
||||
|
||||
此前三种调度细节会把「一台机器坏了」放大成「整批卡住」:
|
||||
|
||||
| 问题 | 后果 | 现在 |
|
||||
|---|---|---|
|
||||
| `acquire()` 在**池锁内**做 TCP 握手 + 探针 | 一台坏机握手卡满 `timeout`,所有并发协程一起停住 | 锁内只预留槽位,握手在锁外做 |
|
||||
| 扩容失败分支 `continue` 且无截止检查 | 全节点失联时无限重试,永远不抛 `PoolTimeoutError` | 每台主机同一次 `acquire` 内只试一遍,试完立刻抛错 |
|
||||
| 全灭后上层逐只重试 | 6,000 只 × 16 台 × 握手的空转 | 池级熔断 2 秒,熔断期内直接失败;有连接成功即解除 |
|
||||
|
||||
实测(16 台替身节点、6,000 只 × 240 根 1 分钟 K 线、跑起来 0.9 秒后 6 台变黑洞):
|
||||
|
||||
| 场景 | 0.7.1 | 0.7.2 |
|
||||
|---|---|---|
|
||||
| 无节点故障 | 1.79 s | 1.80 s |
|
||||
| 2/16 中途失联 | 2.28 s | **1.91 s** |
|
||||
| 6/16 中途失联 | 2.97 s | **2.30 s** |
|
||||
| **16/16 中途失联** | **>60 s 未返回** | **2.37 s** |
|
||||
|
||||
- **批量 K 线失败批次从「事后串行补」改为「立刻回队」**。
|
||||
|
||||
流水线中途失败时,原来把整批攒到 `gather` 结束再逐只重试 —— 每只都要重付一次 RTT,
|
||||
几百只就是十几秒长尾。现在失败批次直接 `extendleft` 回队首,由**其它健康连接**马上接手
|
||||
(每只最多回队 2 次,超限后落进尾部并发兜底)。尾部兜底也从串行改成信号量并发。
|
||||
|
||||
- **分页器不再因为一条连接坏掉而丢掉它那份页**。
|
||||
|
||||
老实现按 `jobs[wid::k]` 静态切片,一个 worker 退出,它那一整份页就没人做了。现在
|
||||
worker 退出时把剩余页放回共享 `leftover`,还活着的 worker 会接着做;全部 worker 退出后
|
||||
还有剩页,再借一条新连接补完(最多两轮)。
|
||||
|
||||
- **EX 池同样改为锁外建连**,并给握手失败的主机加 30 秒拉黑(`ExPool.FAIL_COOLDOWN`),
|
||||
避免每次 `acquire` 都重付一遍握手超时。
|
||||
|
||||
### 修复
|
||||
|
||||
- **修复 `get_quotes` 的 `server_time`(行情快照时间)解码错误**。
|
||||
|
||||
原实现按「小时 + 百万分之一小时」(`divmod(raw, 1_000_000)`)解码,但协议实际是
|
||||
**十进制位拼接**:
|
||||
|
||||
```
|
||||
整数 = HH*10**(4+d) + MM*10**(2+d) + SS*10**d + 小数
|
||||
d = 该品种价格小数位(股票 2 / ETF·LOF·B股 3 / 国债·回购 4)
|
||||
```
|
||||
|
||||
后果:股票 13:29:59 显示成 `11:17:57`(基准错、速率也只有实际的 53%),ETF/债券
|
||||
直接给出小时 > 23 的非法值。
|
||||
|
||||
| 标的 | raw | 0.7.1 显示 | 现在 |
|
||||
|---|---|---|---|
|
||||
| `sh600000`(dp=2) | `13224002` | `13:13:26.407` | **`13:22:40.02`** |
|
||||
| `sh510300`(dp=3) | `132245903` | `132:14:45.250` | **`13:22:45.903`** |
|
||||
| `sh204001`(dp=4) | `1310228050` | `1310:13:40.980` | **`13:10:22.8050`** |
|
||||
|
||||
- 小数位数跟着品种价格精度走,输出 `HH:MM:SS.ff`(2~4 位,兼容旧文档的 `HH:MM:SS.mmm`)。
|
||||
- 服务端怪癖:**SS 落在 50~59 秒时写成 90~99**(编码值 +40 秒)。未修正时该区间的滞后
|
||||
会从稳定的 10.6 秒突变成 -29.4 秒;现在 `SS >= 60` 一律减 40 秒还原。
|
||||
- **指数(`sh000001` / `sz399001` / `sh000300` …)该字段返回空串**:协议里它对指数不是
|
||||
时钟(同一时刻 `000001` 解出 `12:89`、`399001` 解出 `12:15`,互相矛盾)。此前会给出一个
|
||||
看着合理、其实是垃圾的时间。要指数时间请用 `get_minutes` / K 线。
|
||||
- 实测验收(2026-10-30 盘中,同服务器连续采样 3 分钟):`server_time` 与本地时钟的滞后
|
||||
稳定在 10.5~10.9 秒;股票 / ETF / B股 / 国债 / 回购全覆盖,见 `tests/test_quote_time.py`。
|
||||
|
||||
## [0.7.1] - 2026-10-29
|
||||
|
||||
### 性能
|
||||
|
||||
- **DataFrame 转换提速近一倍:24 万行 K 线 894ms → 462ms(polars)/ 1,001ms → 587ms(pandas)**。
|
||||
|
||||
0.7.0 的取列实现有两处浪费:每个单元格都套一层 Python 闭包(顺带做 `IntEnum` 判断),
|
||||
而 `trading_date` 每行都重新格式化三个整数 —— 24 万行里这个值往往只有 **1~2 种**。
|
||||
|
||||
| 转换层(不碰网络) | 0.7.0 | 现在 |
|
||||
|---|---|---|
|
||||
| 5,225 行报价 → polars | 66 ms | **51 ms** |
|
||||
| 5,225 行报价 → pandas | 74 ms | **62 ms** |
|
||||
| 240,000 行 K 线 → polars | 1,038 ms | **444 ms** |
|
||||
| 240,000 行 K 线 → pandas | 1,234 ms | **585 ms** |
|
||||
|
||||
端到端(缓存已热,1,000 只 × 240 根 1 分钟):`df=None` 1,386ms、
|
||||
polars 1,768ms(原 2,680ms)、pandas 1,976ms(原 2,848ms)。
|
||||
|
||||
- 普通列改用 `list(map(attrgetter(name), rows))`:取值发生在 C 层,不再每格建一个 Python 帧。
|
||||
- `trading_date` 按「时间字段(除 minute 外)」缓存(`_TD_KEYS`):A股/港股/期货一个交易日
|
||||
一个键,美股靠键里的 `hour` 覆盖凌晨段跨日判断 —— 不依赖 `_US_DAY_SPLIT_HOUR` 的具体取值。
|
||||
- `IntEnum` 归一化从「每格判断一次」改为「按列判断一次、需要时才整列转」。
|
||||
- 逐字段等价有测试锁定(`test_every_bar_field_equals_model` 等 3 项),口径零变化。
|
||||
|
||||
### 文档
|
||||
|
||||
- `docs/dataframe.md` 性能一节改为**实测数据 + 何时该用/不该用 DataFrame 出口**的说明,
|
||||
并把「缓存命中也要付转换成本」写清。
|
||||
|
||||
## [0.7.0] - 2026-10-28
|
||||
|
||||
### 新增
|
||||
|
||||
- **DataFrame 直出:`TdxData(df="polars")` / `TdxData(df="pandas")`**。
|
||||
|
||||
表格型接口不再只能返回 `list[dataclass]`,构造时声明一次后端即可直接拿到 DataFrame:
|
||||
|
||||
```python
|
||||
async with TdxData(df="polars") as td:
|
||||
df = await td.get_bars("sh600000", KlinePeriod.DAY, count=240)
|
||||
```
|
||||
|
||||
- 覆盖 **42 个**返回表格数据的方法(33 个 `list[dataclass]`、2 个 `list[dict]`、
|
||||
2 个批量 `dict[str, list[...]]`、`get_indicators` 与 `get_index_trade_stats(_multi)`),
|
||||
另含 deprecated 别名(`get_kline` / `get_transactions` / `get_goods_*` 等)。
|
||||
- **核心包仍然零依赖**:pandas / polars 是可选 extras ——
|
||||
`pip install "pytdxdata[polars]"` / `[pandas]` / `[dataframe]`。
|
||||
- `df=None`(默认)**行为与 0.6.x 完全一致**,且装饰器直接透传,零额外开销。
|
||||
- 取值非法抛 `ValueError` 并列出可用值;后端未安装抛 `FrameBackendError`(继承 `ImportError`,
|
||||
`e.name` 为缺的包名,消息里带安装命令)——**在构造时就失败**,不会拖到第一次查询。
|
||||
|
||||
- **列口径**(三条约定,一次定死):
|
||||
|
||||
| 约定 | 说明 |
|
||||
|---|---|
|
||||
| 时间列替换 | `SecurityBar`/`IndexBar`/`MinuteBar` 的 `year/month/day/hour/minute` 不再单列输出,统一给 `datetime`(polars `Datetime('us')` / pandas `datetime64[ns]`)与 `trading_date`;`XdxrRecord` 给 `date` |
|
||||
| 嵌套定型 | `SecurityQuote.bid`/`ask` 展开成 `bid1_price`…`ask5_vol` 共 10 列(缺档 `None`);`fields` 字典 JSON 化成字符串列(空字典给 `None`);`TdxBlock.codes` 保持列表 |
|
||||
| 容器展开 | `get_indicators` → bars + 各指标一列(长度不足补 `None`);`get_index_trade_stats(_multi)` → `minutes` 展开并补 `index`/`date` 常量列;`get_kline_batch`/`get_minute_batch` → 各标的纵向拼接成一张表 |
|
||||
|
||||
- **嵌套调用不二次转换**:`get_kline_batch` 这类方法内部还会调 `get_bars`,
|
||||
用 `ContextVar`(按 task 隔离)记录层次,内层一律返回原始 `list`,只有最外层那次才转 DataFrame。
|
||||
并发调用互不干扰。
|
||||
|
||||
### 性能
|
||||
|
||||
- **按日期区间取 K 线:右端贴最新时改走「倒推」,1 分钟 K 线 4.2s → 82ms(51 倍)**。
|
||||
|
||||
原先 `start_date` / `end_date` 一律走「探总根数 → 二分定位偏移」。1 分钟 K 线历史总量
|
||||
二十万根量级,等于 6 次翻倍探测 + 2 次方向探测 + **两次各 ~18 次**跳转,共 36+ 次串行往返。
|
||||
而「取今天」「增量到最新」这类占绝对多数的调用,右端本来就贴着最新,根本不需要二分。
|
||||
|
||||
| 场景(1 分钟,冷启动) | 优化前 | 优化后 |
|
||||
|---|---|---|
|
||||
| 单只「取今天」 | 4,208 ms | **82 ms** |
|
||||
| 单只「2 天前 ~ now」 | 2,211 ms | **75 ms** |
|
||||
| 单只「取最近 3 天」 | 2,000 ms+ | **239 ms** |
|
||||
| 单只「历史中段(5~3 天前)」 | 1,512 ms | 285 ms |
|
||||
| 全市场 5,225 只「取今天」 | — | 32.1 s(1,140,699 根,**精确当天口径**) |
|
||||
|
||||
做法:右端离最新不超过 `_BACKTRACK_GRACE`(10 天,覆盖长假)时,直接
|
||||
`start=0, count=N` 拿最新 N 根再裁剪 —— N 按时间跨度估算(`_BARS_PER_DAY`),
|
||||
一天内一次就命中,不够才翻倍。历史中段仍走二分,结果完全一致。
|
||||
|
||||
偏移方向是**通道级属性**(与具体标的无关),首次探测后缓存在实例上:
|
||||
全市场 5,225 只批量调用只付一次探测成本,不会每只探两遍。
|
||||
|
||||
- **多标的 K 线改「多连接 × 单连接流水线」抓取,全市场 1 分钟 K 线 43.6s → 5s 级
|
||||
(盘中增量 1.5s)**。
|
||||
|
||||
单只 K 线是「一请求一标的」,5,000 只就是 5,000 次往返。实测单连接串行 **33ms/只
|
||||
≈ 恰好一个 RTT** —— 瓶颈是**往返次数,不是带宽**(此前「单 IP 限流 1.5MB/s」的
|
||||
结论是浅流水下的假象:同一台服务器 16 连接 × 深度 8 实测能跑到 5.2 MB/s)。
|
||||
|
||||
做法:同一连接连续下发 `pipeline` 条请求(协议帧自描述长度且服务器**按序应答**,
|
||||
无需等前一个回来),再叠加多台服务器并行(每台 1 连接,尊重 `per_server_cap`)。
|
||||
|
||||
| 场景(全市场 5,225 只,1 分钟) | 优化前(逐只 `concurrency=6`) | 优化后 |
|
||||
|---|---|---|
|
||||
| 取当天全天(240 根) | 43.6 s | **5~8 s** |
|
||||
| 盘中 10:00(35 根) | 15 s+ | **~2 s** |
|
||||
| 盘中增量(最新 2 根) | 12 s+ | **~1.5 s** |
|
||||
|
||||
- 流水深度默认**自适应**(`pipeline=None`):按单请求字节数把在途量控制在 ~60KB,
|
||||
夹在 8~32 之间 —— 取 240 根用深度 8,只取最新 2 根用深度 32。
|
||||
实测过深反而更慢(24 台 depth=64 比 depth=8 慢一倍,服务器会攒够一批才回)。
|
||||
- 服务器**只取最快的 12 台**:池里 24 台全用反而被慢节点拖长尾
|
||||
(`count=2` 全市场:24 台 2.95s / 12 台 1.45s)。
|
||||
- **盘中只拉已走过的根数**:`start_date=今天` 时按交易时段折算(10:00 → 30 根,
|
||||
而不是 240 根),数据量直接差 8 倍。收盘后回到一整天的量级。
|
||||
- 标准通道(日线不复权)同样走批量:全市场日线 240 根 49~72s → **~10s**。
|
||||
- 流水线中途失败时该批**交回逐只重试**(流水线失败意味着响应流已错配、连接作废),
|
||||
连接全挂时队列里剩下的标的也一并回退 —— 不会静默少数据(新增测试覆盖)。
|
||||
- `pipeline=1` 可关闭批量路径,行为与旧版逐只路径一致。
|
||||
|
||||
- **修正「右端是否贴最新」的时间戳比较**:`0x122E` 返回的 `hour/minute` 是**该周期的
|
||||
结束时刻**,最新一根可能比 `datetime.now()` 略大(14:29 那根此刻还在走)。
|
||||
原先没有容差,`end_date=datetime.now()` 这类最自然的写法会被误判成「历史中段」而走慢路径。
|
||||
|
||||
### 修复
|
||||
|
||||
- **EX 通道 K 线的 `amount` 恒为 0.0,且末位字段(结算价)被解包丢弃**。
|
||||
|
||||
`0x23FF` 一条记录 32B = 4B 时间戳 + **28B 行情**,这 28B 只有**一个** 4B 槽位可复用,
|
||||
语义按市场分两类(全市场实测,用 `成交额 / (close × vol)` 逐条校验):
|
||||
|
||||
| 市场 | 槽位 5 | 槽位 6 | 槽位 7 | 校验 |
|
||||
|---|---|---|---|---|
|
||||
| 港股主板 31 / 港股基金 49 | **成交额**(f) | 成交量(手=100 股) | 未知 | 额/(c·vol) = **100.0** |
|
||||
| 港股创业板 48 / 新三板 44 / 美股 74 | **成交额**(f) | 成交量(股) | 0 | 额/(c·vol) ≈ **1.0** |
|
||||
| 期货 / 期权(7/8/9/47…) | 持仓量(I) | 成交量 | **结算价**(f) | 与 CFFEX 官网日结逐日相符 |
|
||||
|
||||
改动前:`amount` 被硬编码成 `0.0`、第 7 个 float 用 `_` 丢弃 —— 于是港股/美股/新三板
|
||||
**明明帧内有成交额却全是 0**,期货期权的结算价也拿不到(用户只能换源)。
|
||||
|
||||
- 新增 `SecurityBar.settle`(结算价,`float | None`)。仅 EX 期货/期权**日线**有值,
|
||||
分钟线该槽位恒 0 已归一化为 `None`;其它通道恒 `None`。
|
||||
- 权益类市场的 `amount` 回填真实成交额。⚠️ 港股主板/基金的 `vol` 单位是**手(=100 股)**,
|
||||
`amount/vol` 是均价的 **100 倍**。
|
||||
- 期货/期权的 `amount` **仍是 0.0** —— 帧内确实没有成交额,这是协议限制,已在
|
||||
`SecurityBar` 与文档中写清,避免使用者再按文档误判为调用方 bug。
|
||||
- 港股指数 27 / 中证指数 62 该槽位量级对不上(既非成交额也非结算价),保守保持 `0.0`。
|
||||
- `SecurityBar.__setstate__` 兜底旧磁盘缓存:缓存键没有 schema 版本,升级后读旧 pickle
|
||||
会缺 `settle` 槽位,默认反序列化不补默认值 → 直接 `AttributeError`,已容错。
|
||||
EX K 线缓存键额外加 `@v2` 标记,让 `amount` 恒 0 的旧缓存自然失效。
|
||||
|
||||
### 兼容性
|
||||
|
||||
- 无破坏性变更。不传 `df` 时所有方法签名、返回类型、行为与 0.6.9 完全相同。
|
||||
- 空结果返回**列名齐全的空表**(列由模型定义推出),dtype 由后端自行推断。
|
||||
- `SecurityBar` 新增可选字段 `settle`(默认 `None`):K 线 DataFrame 因此多一列 `settle`,
|
||||
位置参数构造的老代码不受影响。
|
||||
|
||||
### 升级
|
||||
|
||||
```bash
|
||||
pip install -U pytdxdata # 只要 dataclass
|
||||
pip install -U "pytdxdata[polars]" # 想直出 DataFrame
|
||||
```
|
||||
|
||||
- **磁盘缓存不需要手动清**:`SecurityBar` 的 `settle` 由 `__setstate__` 兜底补默认值,
|
||||
旧缓存照常可读;EX K 线缓存键已带 `@v2`,`amount` 恒 0 的旧记录会自动失效重取。
|
||||
- 新参数 `pipeline`(`get_bars`)默认自适应,**不传就是最优**;只有想显式退回旧的逐只路径时才传 `1`。
|
||||
|
||||
|
||||
### 文档
|
||||
|
||||
- 新增 [DataFrame 输出](https://openbot-coder.github.io/pytdxdata/dataframe/) 专页,
|
||||
`API_REFERENCE.md`(构造参数表补 `df` 与 `ex_pool_cap`、`pool_max` 默认值修正为 24)、
|
||||
`SKILL.md`、`llms.txt`、`README.md`、`mkdocs.yml` 导航同步。
|
||||
- 全站同步 0.7.0 口径:首页特性表与导航卡片、安装页(extras 与构造参数表补
|
||||
`df` / `std_pool_cap` / `ex_pool_cap`)、场景示例的「转成 pandas」改为推荐 `df=` 开关、
|
||||
数据模型页说明 dataclass 与 DataFrame 是同一份数据、README「已知限制」补 EX 成交额口径。
|
||||
|
||||
## [0.6.9] - 2026-10-23
|
||||
|
||||
### 修复
|
||||
@@ -679,7 +921,10 @@
|
||||
|
||||
- 初始版本:动态连接池 + 双层 TTL 缓存 + 3 连接并发分页 + 双通道(标准 / MAC)+ 扩展市场(港股 / 美股 / 期货 / 期权)
|
||||
|
||||
[Unreleased]: https://github.com/openbot-coder/pytdxdata/compare/v0.6.0...HEAD
|
||||
[Unreleased]: https://github.com/openbot-coder/pytdxdata/compare/v0.7.2...HEAD
|
||||
[0.7.2]: https://github.com/openbot-coder/pytdxdata/compare/v0.7.1...v0.7.2
|
||||
[0.7.1]: https://github.com/openbot-coder/pytdxdata/compare/v0.7.0...v0.7.1
|
||||
[0.7.0]: https://github.com/openbot-coder/pytdxdata/compare/v0.6.0...v0.7.0
|
||||
[0.6.0]: https://github.com/openbot-coder/pytdxdata/compare/v0.5.0...v0.6.0
|
||||
[0.5.0]: https://github.com/openbot-coder/pytdxdata/compare/v0.4.0...v0.5.0
|
||||
[0.4.0]: https://github.com/openbot-coder/pytdxdata/compare/v0.3.2...v0.4.0
|
||||
|
||||
@@ -47,14 +47,17 @@ async def main():
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
返回的是标准库 `dataclass` 组成的 `list`(库本身不依赖 pandas)。要 DataFrame 一行转:
|
||||
默认返回的是标准库 `dataclass` 组成的 `list`(核心库不依赖 pandas)。需要 DataFrame 就换个后端构造:
|
||||
|
||||
```python
|
||||
from dataclasses import asdict
|
||||
import pandas as pd
|
||||
df = pd.DataFrame([asdict(b) for b in bars])
|
||||
async with TdxData(df="polars") as td: # 或 df="pandas";需 pip install "pytdxdata[polars]"
|
||||
df = await td.get_bars("sh600000", KlinePeriod.DAY, count=240)
|
||||
# 列:market code open high low close vol amount datetime trading_date symbol settle
|
||||
```
|
||||
|
||||
时间字段会被**合并成一列 `datetime`**(不再有 year/month/day/hour/minute),五档报价展开成
|
||||
`bid1_price`…`ask5_vol`,详见 [DataFrame 输出](https://openbot-coder.github.io/pytdxdata/dataframe/)。
|
||||
|
||||
## 特性
|
||||
|
||||
| 特性 | 说明 |
|
||||
@@ -63,7 +66,8 @@ df = pd.DataFrame([asdict(b) for b in bars])
|
||||
| 🗄️ 双层 TTL 缓存 | 内存 LRU + 磁盘 SQLite(WAL),TTL 分级;single-flight 并发同键去重;缓存**已解析记录**(重复请求 0 网络 0 解析) |
|
||||
| ⚡ 并发分页 | 单请求拆 N 页(显式 start 偏移 → 无状态可并行)并行拉取、按序拼接、失败换连接重试 |
|
||||
| 🔀 双通道整合 | 标准通道(经典行情)+ MAC 通道(逐笔 / 复权 / 1 分钟 K 线),按命令自动选路 |
|
||||
| 📦 零重型依赖 | 纯 asyncio + stdlib(socket / zlib / struct / sqlite3) |
|
||||
| 📦 零重型依赖 | 纯 asyncio + stdlib(socket / zlib / struct / sqlite3),`dependencies` 为空 |
|
||||
| 🐼 DataFrame 直出 | `TdxData(df="polars")` / `df="pandas"` 让 42 个表格型接口直接返回 DataFrame;时间列自动合并成 `datetime`,五档报价自动展开。后端是可选 extras,核心包仍零依赖 |
|
||||
| 📊 全市场覆盖 | A股、港股、美股、期货、期权(46 个扩展市场) |
|
||||
| 📄 服务器文件解析 | 板块 `block_*.dat` / 行业 `tdxhy.cfg` / 财报 `gpcw*` **下载即解析**成 dataclass(索引名与服务器实际文件名不一致时自动纠正) |
|
||||
| 🤖 AI 助手友好 | 内置 [`SKILL.md`](https://github.com/openbot-coder/pytdxdata/blob/main/SKILL.md) 与 [`llms.txt`](https://openbot-coder.github.io/pytdxdata/llms.txt) |
|
||||
@@ -114,6 +118,10 @@ uv run mkdocs serve # 本地预览文档站
|
||||
- 行情来自通达信公开服务器,**15 分钟延时**;需实时数据请使用券商行情
|
||||
- 标准通道 K 线 `count < 100` 部分服务器返回空 → 分页器已强制 `min_page=100` 规避
|
||||
- 港股 / 美股期权在通达信协议内无数据
|
||||
- **EX 通道(港股 / 美股 / 期货 / 期权)的 K 线没有统一成交额**:`0x23FF` 帧内只有一个可复用槽位,
|
||||
权益类市场(港股主板/基金/创业板、新三板、美股)该槽位是成交额,已回填;**期货 / 期权该槽位是持仓量,
|
||||
所以它们的 `SecurityBar.amount` 恒为 `0.0`**(协议限制,不是 bug)。期货/期权的结算价在 `SecurityBar.settle`。
|
||||
⚠️ 港股主板/基金的 `vol` 单位是「手」(=100 股),`amount/vol` 是均价的 100 倍
|
||||
- **K 线时间戳统一为北京时间**(v0.6.2 起)。美股原始报文把「美东交易日日期」与「北京时间时分」
|
||||
拼在一起,库已自动归一化,`bar.datetime` 可直接排序 / 过滤;但**美股一个交易日横跨两个北京
|
||||
自然日**,按交易日聚合请用 `SecurityBar.trading_date`,不要用 `datetime.date()`
|
||||
|
||||
@@ -52,6 +52,7 @@ df = pd.DataFrame([asdict(b) for b in bars])
|
||||
|
||||
```bash
|
||||
pip install pytdxdata # 或 uv add pytdxdata
|
||||
pip install "pytdxdata[polars]" # 需要 DataFrame 输出时才装(可选 extras)
|
||||
```
|
||||
|
||||
## 可运行模板
|
||||
@@ -82,12 +83,20 @@ async def main():
|
||||
# 分时(MAC 通道,今日或历史)
|
||||
minutes = await td.get_minutes(["sz000001"])
|
||||
|
||||
# 批量 K 线:多股票自动并发,池子会分配到不同服务器
|
||||
# 批量 K 线:≥16 只自动走「多连接 × 单连接流水线」,池子把标的摊到多台服务器
|
||||
batch = await td.get_bars(
|
||||
["sz000001", "sh600000"],
|
||||
KlinePeriod.DAY, count=240,
|
||||
) # 平铺返回,每条带 .symbol 字段
|
||||
|
||||
# 全市场盘中实时:只拉已经走完的根数(10:00 → 30 根),5,200 只 1~2 秒
|
||||
from datetime import datetime
|
||||
live = await td.get_bars(
|
||||
[s.symbol for s in await td.get_universe("sh")],
|
||||
KlinePeriod.MIN_1, start_date=datetime.now(),
|
||||
)
|
||||
# pipeline=1 可关闭批量路径,退回逐只并发(结果一致,只是慢)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
@@ -182,6 +191,28 @@ await td.get_bars("cffex:IFL0", KlinePeriod.DAY, count=700) # 期货合约
|
||||
- **小数位表不用手动管,但可以手动刷**。`start()` 会检查当日缓存并后台同步一次(`SecurityInfo.decimal_point` 权威值);要强制刷新用 `await td.sync_decimal_points()`,看同步状态用 `await td.decimal_point_status()`。`PYPYTDX_DATA_DISABLE_DECIMAL_SYNC=1` 可关闭后台同步。
|
||||
- 板块文件(`block_*.dat`)/ 行业(`tdxhy.cfg`)/ 财报(`gpcw*.zip`)是**服务器上的静态文件**,解析后是普通 dataclass,**落 JSON 后完全脱网可用**——适合做离线快照。
|
||||
|
||||
## DataFrame 输出(v0.7+)
|
||||
|
||||
表格型方法可以直出 DataFrame,**唯一开关是构造参数 `df`**,不要在调用处传 `df=True`(没有这个参数):
|
||||
|
||||
```python
|
||||
async with TdxData(df="polars") as td: # 或 df="pandas"
|
||||
df = await td.get_bars("sh600000", KlinePeriod.DAY, count=240)
|
||||
```
|
||||
|
||||
| 事项 | 说明 |
|
||||
|---|---|
|
||||
| 安装 | `pip install "pytdxdata[polars]"` / `[pandas]` / `[dataframe]`;核心包仍零依赖 |
|
||||
| `df=None`(默认) | 行为与 0.6.x 完全一致,返回 `list[dataclass]` |
|
||||
| 报错 | `df` 取值非法 → `ValueError`;包没装 → `FrameBackendError`(启动即失败,不拖到第一次查询) |
|
||||
| 时间列 | `year/month/day/hour/minute` **被 `datetime` 替换**,另带 `trading_date`;`XdxrRecord` 给 `date` |
|
||||
| 嵌套 | `bid`/`ask` 展开成 `bid1_price`…`ask5_vol` 共 10 列;`fields` 字典 JSON 化成字符串列;`codes` 保持列表 |
|
||||
| 特殊容器 | `get_indicators` → bars+指标合并;`get_index_trade_stats(_multi)` → `minutes` 展开 + `index`/`date` 列;`get_kline_batch` / `get_minute_batch` → 纵向拼接成一张表 |
|
||||
| 不受影响 | `bytes` / `int` / `dict` / 单对象 / `tuple` / `list[float]` 类返回原样(`get_file`、`get_security_count`、`get_server_info`、`get_price_limits` 等) |
|
||||
| 空结果 | 列名齐全的空表;dtype 只由后端推断 |
|
||||
|
||||
完整口径见 [DataFrame 输出](https://openbot-coder.github.io/pytdxdata/dataframe/)。
|
||||
|
||||
## 陷阱清单
|
||||
|
||||
| 坑 | 后果 | 正确做法 |
|
||||
@@ -200,6 +231,8 @@ await td.get_bars("cffex:IFL0", KlinePeriod.DAY, count=700) # 期货合约
|
||||
| 拿指数逐笔的 `trade_count` 当笔数 | 拿到的是「均价×1000」,数值荒谬 | 用 `get_index_trade_stats(index)` 由成分累加 |
|
||||
| 拿 ETF/LOF 逐笔的 `trade_count` 当笔数 | 拿到的是「价格×1000」(`sh510310` 显示 3994 笔) | 看 `trade_count_available`,或 `symbols.has_order_count()` |
|
||||
| 拿深市 `399xxx` 的 `amount` 校验成分合计 | 比值 0.43,误以为数据错 | 深市指数 amount 是**全板块**口径(`sz399001`=深市全部 5081 亿),不是样本股 |
|
||||
| 在期货/期权 K 线上取 `amount` | 恒 0.0,误以为解析坏了 | EX `0x23FF` 帧内**没有成交额**(协议限制);结算价看 `bar.settle`,成交额改用 `get_quotes` |
|
||||
| 用港股主板的 `amount / vol` 当均价 | 差 **100 倍** | 港股主板/基金的 `vol` 单位是**手(=100 股)**,要 `amount/vol/100`;港股创业板/美股才是股 |
|
||||
| 依赖实时性做信号 | 数据是 15 分钟前的 | 换券商实时源 |
|
||||
| 手动改包内 `servers.json` / 写死 IP | 容器只读、升级被覆盖、快节点捞不回来 | 日常靠内置的**当日测速缓存**自动选路;要强制指定就用 `PYPYTDX_DATA_{STANDARD,MAC,EX}_HOSTS` 环境变量 |
|
||||
|
||||
|
||||
+44
-8
@@ -1,6 +1,6 @@
|
||||
# pytdxdata 对外接口清单
|
||||
|
||||
> 以 `src/pytdxdata` 源码为准整理(v0.6.0)。分四类:Python API / 数据模型 / 辅助模块 / CLI。
|
||||
> 以 `src/pytdxdata` 源码为准整理(v0.7.2)。分四类:Python API / 数据模型 / 辅助模块 / CLI。
|
||||
>
|
||||
> 方法与代码的一致性由 `scripts/check_docs.py` 在 CI 中校验。
|
||||
|
||||
@@ -28,8 +28,9 @@ from pytdxdata import (
|
||||
| `cache_dir` | None | 磁盘缓存目录;None=仅内存缓存 |
|
||||
| `default_adjust` | `Adjust.NONE` | 默认复权方式 |
|
||||
| `max_inflight` | 64 | single-flight 并发上限 |
|
||||
| `pool_min` / `pool_max` | 3 / 12 | 动态池上下限 |
|
||||
| `mac_pool_cap` | 4 | 单台 MAC 服务器连接上限 |
|
||||
| `pool_min` / `pool_max` | 3 / 24 | 动态池上下限 |
|
||||
| `mac_pool_cap` / `std_pool_cap` / `ex_pool_cap` | 4 / 4 / 2 | 单台服务器连接上限 |
|
||||
| `df` | None | **DataFrame 后端**:`None`=返回 `list[dataclass]`(0.6.x 行为)、`"polars"`、`"pandas"`。需先装 extra:`pip install pytdxdata[polars]`,详见 [DataFrame 输出](dataframe.md) |
|
||||
|
||||
生命周期:
|
||||
|
||||
@@ -41,6 +42,10 @@ from pytdxdata import (
|
||||
|
||||
### 2.1 统一接口(v0.6+ 推荐){#unified}
|
||||
|
||||
> **DataFrame(v0.7+)**:本节的表格型方法在 `TdxData(df="polars")` / `df="pandas"` 时
|
||||
> 直接返回 DataFrame;`df=None`(默认)返回下表的 `list[...]`。列口径、时间列替换规则、
|
||||
> 不受影响的接口清单见 [DataFrame 输出](dataframe.md)。
|
||||
|
||||
以下方法在 v0.6 新增,标的一律用字符串(`sz000001` / `hk00700` / `usAAPL` / `cffex:IFL0`)。
|
||||
|
||||
#### 标的清单
|
||||
@@ -64,7 +69,7 @@ from pytdxdata import (
|
||||
|
||||
| 方法 | 返回 | 说明 |
|
||||
|------|------|------|
|
||||
| `get_bars(symbols, period=DAY, *, start=0, count=800, adjust=None, start_date=None, end_date=None, concurrency=6)` | `list[SecurityBar]` | 跨市场 K 线(自动判指数、复权走 MAC、并发分页、日期区间二分) |
|
||||
| `get_bars(symbols, period=DAY, *, start=0, count=800, adjust=None, start_date=None, end_date=None, concurrency=6, pipeline=None)` | `list[SecurityBar]` | 跨市场 K 线(自动判指数、复权走 MAC、并发分页;日期区间贴最新走倒推、历史中段走二分)。**≥16 只自动走多连接流水线**:`pipeline` 为每条连接的在途请求数(默认按单请求字节自适应 8~32,传 `1` 关闭),全市场 1 分钟 K 线取当天 43.6s → 5s 级 |
|
||||
| `get_indicators(symbols, indicators, *, period=DAY, count=30, adjust=None, params=None)` | `list[IndicatorSet]` | K 线+技术指标(MACD/KDJ/RSI/BOLL/MA/EMA) |
|
||||
|
||||
#### 逐笔与分时
|
||||
@@ -91,7 +96,7 @@ from pytdxdata import (
|
||||
|
||||
| 方法 | 返回 | 说明 |
|
||||
|------|------|------|
|
||||
| `get_kline(market, code, period, *, start=0, count=800, adjust=None, start_date=None, end_date=None)` | `list[SecurityBar]` | 主接口;count>页自动并发分页;复权走 MAC;支持日期区间(二分定位) |
|
||||
| `get_kline(market, code, period, *, start=0, count=800, adjust=None, start_date=None, end_date=None)` | `list[SecurityBar]` | 主接口;count>页自动并发分页;复权走 MAC;支持日期区间(贴最新倒推 / 中段二分) |
|
||||
| `get_kline_since(market, code, period, since, adjust=None)` | `list[SecurityBar]` | 增量便捷方法(≥since,含当天) |
|
||||
| `get_kline_batch(stocks, period, *, start=0, count=800, adjust=None, concurrency=6)` | `dict[str, list[SecurityBar]]` | 多股票并发,返回 {code: bars} |
|
||||
| `get_index_kline(market, code, period, *, start=0, count=800)` | `list[SecurityBar]` | 指数K线(含涨跌家数字段) |
|
||||
@@ -199,7 +204,7 @@ from pytdxdata import (
|
||||
|
||||
| 模型 | 关键字段 |
|
||||
|------|---------|
|
||||
| `SecurityBar` | market/code/open/high/low/close/vol/amount/year/month/day/hour/minute + `datetime` 属性 |
|
||||
| `SecurityBar` | market/code/open/high/low/close/vol/amount/year/month/day/hour/minute/symbol/**settle** + `datetime` 属性(`settle` 结算价仅 EX 期货/期权日线有值,其余为 `None`) |
|
||||
| `IndexBar` | 继承 `SecurityBar`(指数K线) |
|
||||
| `SecurityQuote` | price/pre_close/OHLC/vol/amount/s_vol/b_vol/`bid[5]`/`ask[5]`/server_time/trading_status/cur_vol/rise_speed/limit_up/limit_down/decimal_point/open_amount |
|
||||
| `SecurityInfo` | market/code/name/volunit/decimal_point/pre_close |
|
||||
@@ -223,6 +228,18 @@ from pytdxdata import (
|
||||
|
||||
> 注:`__init__` 仅导出 `Market/ExMarket/KlinePeriod/Adjust` 与 6 个核心模型;`BoardInfo/MemberQuote/AuctionItem` 等需 `from pytdxdata.models import ...`。
|
||||
|
||||
**`SecurityQuote.server_time`(行情快照时间,0.7.2 修正)**
|
||||
|
||||
- 格式 `HH:MM:SS.ff`,小数位数 = 该品种**价格小数位**:股票 2、ETF/LOF/B股 3、国债·回购 4。
|
||||
例 `13:22:40.02` / `13:22:45.903` / `13:10:22.8050`。
|
||||
- 它比本地时钟**慢 5~15 秒**(服务器快照滞后),超低流动性品种(货币 ETF、国债)可能差几分钟。
|
||||
判断「行情有没有更新」请看这个值有没有变,别看调用时刻 —— 盘口本身约 3 秒刷新一次。
|
||||
- **指数(`sh000001` / `sz399001` / `sh000300` …)恒为空串**:该字段对指数不是时钟
|
||||
(同一时刻 `000001` 解出 `12:89`、`399001` 解出 `12:15`,协议内没有可用信息)。
|
||||
要指数时间请用 `get_minutes` / K 线。
|
||||
- ⚠️ 0.7.1 及之前按「小时 + 百万分之一小时」解码是**错的**:股票 `11:29:59` 会显示成
|
||||
`11:17:57`,ETF/债券会显示成 `113:00:21` 这种非法小时。
|
||||
|
||||
**量纲对照(跨品种,2026-10-22 实测)**
|
||||
|
||||
同名字段在不同品种上单位不同。最典型的坑是用 `amount / vol` 算均价——它在可转债上会**错 10 倍**:
|
||||
@@ -232,8 +249,18 @@ from pytdxdata import (
|
||||
| 股票 / ETF / LOF | 手(= 100 股) | 股 | **0.01** | 均价 ✓ |
|
||||
| 可转债(110/111/113/118/123/127/128) | 手(= 10 张) | **手** | **1.0** | 均价 **×10** ⚠️ |
|
||||
| 指数 | 百元(成交额/100) | 手 | — | 无意义(`amount` 是全板块口径) |
|
||||
| 港股主板 31 / 港股基金 49 | — | **手(= 100 股)** | — | 均价 **×100** ⚠️ |
|
||||
| 港股创业板 48 / 新三板 44 / 美股 74 | — | 股 | — | 均价 ✓ |
|
||||
| 期货 / 期权(EX 全市场) | — | 手/张 | — | **`amount` 恒 0.0**(帧内无成交额,见下) |
|
||||
|
||||
算均价:股票 `amount / vol`;可转债 `amount / vol / 10`;跨品种统一用 `Σ(price×vol) / Σvol`。
|
||||
算均价:股票 `amount / vol`;可转债 `amount / vol / 10`;港股主板/基金 `amount / vol / 100`;跨品种统一用 `Σ(price×vol) / Σvol`。
|
||||
|
||||
> ⚠️ **EX 通道(0x23FF)的成交额不是全市场都有的**。一条记录 32B = 4B 时间 + 28B 行情,
|
||||
> 28B 里只有**一个** 4B 槽位可复用,语义按市场分两类:
|
||||
> 权益类(港股主板/创业板/基金、新三板、美股)该槽位是**成交额**,`amount` 正常;
|
||||
> 期货期权类该槽位是**结算价**(体现在 `SecurityBar.settle`,日线才有值),
|
||||
> `amount` 恒 **0.0** —— 这是协议限制,不是解析漏读,要成交额请改用 `get_quotes` 或其它源。
|
||||
> 港股指数(27)/ 中证指数(62)该槽位语义未明,同样保守返回 `0.0`。
|
||||
|
||||
> 可转债的价格标度、分时、逐笔已全量校验(16 只当日 + 8 只历史日):分时末价 / 快照现价、`Σvol / 日线vol`、逐笔 VWAP vs `amount/(vol×10)` 比值均为 **1.000000**。
|
||||
|
||||
@@ -320,4 +347,13 @@ CLI 开关:`--discover`(三种协议全测并据实修正 `protocols`,同
|
||||
| 逐笔成交 / 分时 | MAC |
|
||||
| 列表 / 数量 / 除权 / 财务 | 标准 |
|
||||
| 板块 / 竞价 / 异动 / 快照 / 资金流 / 服务器 / 分类报价 | MAC |
|
||||
| 港股 / 美股 / 期货 / 期权 | EX(端口 7727) |
|
||||
| 港股 / 美股 / 期货 / 期权 | EX(端口 7727,`0x23FF`) |
|
||||
|
||||
> EX 通道的 K 线(`0x23FF`)字段口径与 A 股不同,见「量纲对照」下的注:
|
||||
> 权益类市场(港股/美股/新三板)有 `amount`;期货期权有 `settle`(结算价)但**没有** `amount`。
|
||||
|
||||
**多标的 K 线(`get_bars` 传 ≥16 只)额外走「批量流水线」**:按上表选池后,
|
||||
多条连接并行、每条连接一次下发 `pipeline` 条请求(默认自适应 8~32),把 N 次往返
|
||||
折叠成 `N/pipeline` 次。只用实测最快的 12 台服务器(池里 24 台全用会被慢节点拖长尾)。
|
||||
`start_date=今天` 时只拉已走过的根数(10:00 → 30 根),盘中实时刷新因此只要 1~2 秒。
|
||||
流水线失败时该批自动交回逐只重试,不会丢数据。
|
||||
+1
-1
@@ -26,7 +26,7 @@ curl -o ~/.workbuddy/skills/pytdxdata/SKILL.md \
|
||||
## 里面有什么
|
||||
|
||||
- **三条必守约定** —— 唯一入口是 `TdxData` 且必须 `async with`;标的是字符串(如 `sh600000`)
|
||||
而不是 `(market, code)` 元组;返回值是 `list[dataclass]` 而不是 DataFrame
|
||||
而不是 `(market, code)` 元组;默认返回 `list[dataclass]`(要 DataFrame 得用 `TdxData(df="polars")` 显式声明后端)
|
||||
- **枚举取值表** —— `Market` / `KlinePeriod` / `Adjust` / `ExMarket` 的准确取值
|
||||
- **可运行模板** —— K 线、复权、报价、逐笔、分时、批量,全部是能直接跑的代码
|
||||
- **「想做什么 → 用哪个方法」决策表** —— 18 个常见需求的直接映射
|
||||
|
||||
+10
-2
@@ -2,8 +2,16 @@
|
||||
|
||||
全部数据模型位于 `pytdxdata.models`,统一为 `@dataclass(slots=True)`。
|
||||
|
||||
选择 dataclass 而不是 pandas DataFrame 是刻意的:**零依赖 + 零转换开销**,
|
||||
需要 DataFrame 时你自己转(一行的事,见 [场景示例 → 转成 pandas](../cookbook.md#to-pandas))。
|
||||
默认(`df=None`)所有接口返回这些 dataclass —— **零依赖 + 零转换开销**。
|
||||
不想要 dataclass 就直接开 DataFrame 出口,不用自己 `asdict` 再建表:
|
||||
|
||||
```python
|
||||
async with TdxData(df="polars") as td: # 或 df="pandas"
|
||||
df = await td.get_bars("sh600000", KlinePeriod.DAY, count=240)
|
||||
```
|
||||
|
||||
列口径(时间列合并、五档展开、`fields` JSON 化)见 [DataFrame 输出](../dataframe.md);
|
||||
两者是同一份数据,只是容器不同,**量纲与语义完全一致**。
|
||||
|
||||
## 导入方式
|
||||
|
||||
|
||||
+12
-1
@@ -196,7 +196,18 @@ async def ex_markets():
|
||||
|
||||
## 转成 pandas {#to-pandas}
|
||||
|
||||
包本身不带 pandas 依赖,但转换是一行的事:
|
||||
**0.7.0 起不用自己转了** —— 构造时声明一次后端,表格型接口直接返回 DataFrame:
|
||||
|
||||
```python
|
||||
async with TdxData(df="pandas") as td: # 或 df="polars"
|
||||
df = await td.get_bars("sh600000", KlinePeriod.DAY, count=800)
|
||||
df = df.set_index("datetime") # 时间列已经合并好了
|
||||
```
|
||||
|
||||
需要 `pip install "pytdxdata[pandas]"`(或 `[polars]` / `[dataframe]`),细节见
|
||||
[DataFrame 输出](dataframe.md)。
|
||||
|
||||
如果只想对**某一处**手动转、不想受影响全局,仍可用 dataclass 自己拼:
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
# DataFrame 输出(pandas / polars)
|
||||
|
||||
0.7.0 起,`TdxData` 可以**直接把表格型接口的返回值变成 DataFrame**,不用再自己 `[asdict(x) for x in rows]`。
|
||||
|
||||
设计上只有一条开关,就是构造参数 `df`:
|
||||
|
||||
```python
|
||||
from pytdxdata import TdxData, KlinePeriod
|
||||
|
||||
# 默认:df=None,行为和 0.6.x 完全一致,返回 list[dataclass]
|
||||
async with TdxData() as td:
|
||||
bars = await td.get_bars("sh600000", KlinePeriod.DAY, count=240) # list[SecurityBar]
|
||||
|
||||
# 要 DataFrame 就在构造时声明一次后端
|
||||
async with TdxData(df="polars") as td:
|
||||
df = await td.get_bars("sh600000", KlinePeriod.DAY, count=240) # polars.DataFrame
|
||||
|
||||
async with TdxData(df="pandas") as td:
|
||||
df = await td.get_bars("sh600000", KlinePeriod.DAY, count=240) # pandas.DataFrame
|
||||
```
|
||||
|
||||
## 安装
|
||||
|
||||
pandas / polars 都是**可选依赖**——`pytdxdata` 的 `dependencies` 仍然是空的,不想用 DataFrame 的人一点额外开销都不用付。
|
||||
|
||||
```bash
|
||||
pip install "pytdxdata[polars]" # 只要 polars
|
||||
pip install "pytdxdata[pandas]" # 只要 pandas
|
||||
pip install "pytdxdata[dataframe]" # 两个都要
|
||||
```
|
||||
|
||||
没装就 `TdxData(df="polars")`,会在**构造时**直接报错,而不是等到第一次查询:
|
||||
|
||||
```
|
||||
FrameBackendError: 未安装 polars,无法输出 DataFrame。请执行:pip install pytdxdata[polars]
|
||||
```
|
||||
|
||||
`df` 取值非法(比如 `df="numpy"`)抛 `ValueError`,并列出可用值。
|
||||
|
||||
## 三个口径约定
|
||||
|
||||
### 1. 时间列:`datetime` 替换拆开的年月日时分
|
||||
|
||||
原始模型为了解析性能把时间拆成 `year/month/day/hour/minute`。DataFrame 里再这么摆,用的人第一件事就是自己拼时间戳——所以直接给一列 `datetime`:
|
||||
|
||||
| 模型 | 原始字段 | DataFrame 里的列 |
|
||||
|---|---|---|
|
||||
| `SecurityBar` / `IndexBar` | `year month day hour minute` | `datetime`(+ `trading_date`) |
|
||||
| `MinuteBar` | `date hour minute day_offset` | `datetime`(+ `trading_date`) |
|
||||
| `XdxrRecord` | `year month day` | `date` |
|
||||
|
||||
- `datetime` 的 dtype:polars 是 `Datetime('us')`,pandas 是 `datetime64[ns]`。
|
||||
- `trading_date` 是 `YYYY-MM-DD` 字符串,**按交易日聚合要用它**:美股一个交易日横跨两个北京自然日(21:30→次日 04:00),凌晨段的 `datetime` 属于次日,但 `trading_date` 归属前一交易日。
|
||||
- 除权记录的 `date` 是「日」不是「时刻」,值为当日 00:00,仍用 datetime 类型以便排序比较。
|
||||
- `TransactionRecord.time` 只有 `HH:MM:SS`、**协议里没有日期**,所以逐笔不生成 `datetime`,`time` 原样保留。
|
||||
|
||||
### 2. 嵌套字段:能定型的定型,不能定型的 JSON 化
|
||||
|
||||
| 字段 | DataFrame 里的形态 |
|
||||
|---|---|
|
||||
| `SecurityQuote.bid` / `ask`(五档) | 展开成 `bid1_price` `bid1_vol` … `ask5_price` `ask5_vol` 共 **10 列**,缺档为 `None` |
|
||||
| `fields`(A股字段位 / EX 额外字段) | **JSON 字符串**列;空字典给 `None`(不会看到一列 `"{}"`) |
|
||||
| `TdxBlock.codes`(成分股列表) | 保持列表(polars 为 `List(Str)`,pandas 为 object),需要时 `.explode()` |
|
||||
|
||||
`fields` 之所以不展开:键集随市场和通道变化,展开会让列数失控。
|
||||
|
||||
### 3. 特殊容器会被展开成一张表
|
||||
|
||||
| 接口 | 原始返回 | DataFrame 形态 |
|
||||
|---|---|---|
|
||||
| `get_indicators` | `list[IndicatorSet]` | bars 列 + 每个指标一列,按长度对齐(不足补 `None`),`symbol` 取自 `IndicatorSet.symbol` |
|
||||
| `get_index_trade_stats` | `IndexTradeStats` | `minutes` 展开成行,并补 `index` / `date` 两个常量列 |
|
||||
| `get_index_trade_stats_multi` | `list[IndexTradeStats]` | 同上,多个指数纵向拼接 |
|
||||
| `get_kline_batch` / `get_minute_batch` | `dict[标的, list[...]]` | 各标的分块纵向拼接成一张表(`symbol`/`code` 列本来就在行里) |
|
||||
| `get_board_ranking` / `get_board_change_ranking` | `list[dict]` | 按首次出现顺序取键并集作为列名 |
|
||||
|
||||
## 哪些接口不受影响
|
||||
|
||||
`df` 只作用于**表格型返回**。下面这些原样返回,不转 DataFrame:
|
||||
|
||||
| 返回类型 | 接口 |
|
||||
|---|---|
|
||||
| `bytes` | `get_file` `get_block_info` `get_report_file` `download_file` `get_financial_records` |
|
||||
| `int` | `get_security_count` `get_goods_count` |
|
||||
| `dict` | `decimal_point_status` `get_board_summary` `get_industry_map` |
|
||||
| 单个对象 | `get_server_info` `get_market_stat` `get_symbol_info` |
|
||||
| `tuple` | `get_price_limits` `get_kline_offset` |
|
||||
| `list[float]` / `list[str]` | `get_chart_sampling` `get_goods_chart_sampling` `get_financial_file_list` |
|
||||
|
||||
理由是单行表没有意义,而 `bytes`/`int` 根本不是表格数据。
|
||||
|
||||
## 空结果
|
||||
|
||||
查不到数据时返回的是**列名齐全的空 DataFrame**(列由模型定义推出,不依赖是否拿到行):
|
||||
|
||||
```python
|
||||
async with TdxData(df="polars") as td:
|
||||
df = await td.get_xdxr("sz000001") # 假设当天无除权
|
||||
df.height, df.columns # 0, ['market', 'code', 'date', ...]
|
||||
```
|
||||
|
||||
⚠️ 空表只保证**列名**正确,dtype 由后端自行推断(polars 可能全为 `Null`,pandas 为 object)。
|
||||
|
||||
## 语义提醒
|
||||
|
||||
DataFrame 只是换了个容器,**下面这些口径不会因为转成 DataFrame 而变化**:
|
||||
|
||||
- 股票/ETF 的 `vol` 是「手」,日线是「股」;可转债 `vol` 手=10 张,`amount/vol` 是均价 **×10**;指数逐笔 `vol` 是百元。
|
||||
- 指数和 ETF/LOF 的 `trade_count` 不是笔数(见 `TransactionRecord.trade_count_available`)。
|
||||
- 深市 `399xxx` 的 `amount` 是全板块口径,不要拿去校验成分合计。
|
||||
- K 线的 `settle`(结算价)列只有 **EX 通道的期货/期权日线**有值,其余行全为 `None`;
|
||||
同一批 EX 标的中,期货/期权的 `amount` 恒 `0.0`(`0x23FF` 帧内没有成交额),
|
||||
港股主板/基金的 `vol` 单位是「手」(`amount/vol` 是均价的 100 倍)。
|
||||
|
||||
细节见 [数据模型](api/models.md) 与 [常见问题](faq.md)。
|
||||
|
||||
## 性能
|
||||
|
||||
先说结论:**`df=` 只影响出参那一层,取数、连接池、缓存完全不变**,但转换本身不是免费的——
|
||||
结果集越大越明显。实测(缓存已热、同一台机器、Windows / Python 3.13):
|
||||
|
||||
| 场景 | `df=None` | `df="polars"` | `df="pandas"` |
|
||||
|---|---|---|---|
|
||||
| 全市场报价 5,225 只 | 66 ms | 117 ms(+51) | 124 ms(+58) |
|
||||
| 1,000 只 × 240 根 1 分钟(24 万行) | 1,386 ms | 1,768 ms(+383) | 1,976 ms(+590) |
|
||||
| 单只 × 240 根 1 分钟 | 1 ms | 2 ms | 2 ms |
|
||||
|
||||
只看转换层本身(不碰网络):5,225 行报价 → polars **51 ms** / pandas **62 ms**;
|
||||
24 万行 K 线 → polars **444 ms** / pandas **585 ms**。
|
||||
|
||||
几个要点:
|
||||
|
||||
- **小结果无感**:单只 K 线、单只报价这类常见调用只多 1 ms 左右。
|
||||
- **大结果按行数线性增长**:每行约 2~3 µs(12 列)。全市场 1 分钟线(约 125 万行)
|
||||
转换约 2.3 s(polars)/ 3.1 s(pandas)——如果你的链路是「全市场分钟线直落库」,
|
||||
这笔开销通常远小于下游写入,但如果只是做筛选/聚合,建议先拿 `df=None` 的 list 算完再转。
|
||||
- **polars 比 pandas 快 25~30%**(同数据、同机器),大结果更明显。
|
||||
- **转换是列式采集**:列计划按模型缓存(含交易日列的按时间字段缓存),
|
||||
普通列走 C 层取值,不做逐行 `asdict()`。
|
||||
- **`df=None` 时完全不经过这一层**:装饰器直接透传,零额外开销。
|
||||
- **缓存命中也要转**:磁盘/内存缓存里存的是 dataclass 记录,DataFrame 是每次调用现建的
|
||||
——高频轮询同一批数据时,这笔转换开销每次都会付。
|
||||
+21
-1
@@ -35,6 +35,23 @@
|
||||
|
||||
安装后即可使用 `tdx` 命令行工具(见 [CLI 手册](cli.md))。
|
||||
|
||||
### 可选:DataFrame 后端(0.7.0+)
|
||||
|
||||
想让表格型接口**直接返回 DataFrame**,需要额外装一个后端。核心包依旧零依赖,不装就完全不受影响:
|
||||
|
||||
```bash
|
||||
pip install "pytdxdata[polars]" # 只要 polars
|
||||
pip install "pytdxdata[pandas]" # 只要 pandas
|
||||
pip install "pytdxdata[dataframe]" # 两个都要
|
||||
```
|
||||
|
||||
装上之后在构造时声明一次即可,详见 [DataFrame 输出](dataframe.md):
|
||||
|
||||
```python
|
||||
async with TdxData(df="polars") as td: # 或 df="pandas"
|
||||
df = await td.get_bars("sh600000", KlinePeriod.DAY, count=240)
|
||||
```
|
||||
|
||||
### 可选:uvloop 加速
|
||||
|
||||
Linux / macOS 上可选用 `uvloop` 替换默认事件循环(Windows 不支持):
|
||||
@@ -63,8 +80,11 @@ asyncio.run(main())
|
||||
| `cache_dir` | `None` | 磁盘缓存目录;**`None` = 仅内存缓存**(进程退出即失效) |
|
||||
| `default_adjust` | `Adjust.NONE` | 默认复权方式,可被单次调用覆盖 |
|
||||
| `max_inflight` | `64` | single-flight 并发上限(同键请求合并) |
|
||||
| `pool_min` / `pool_max` | `3` / `12` | 动态连接池上下限 |
|
||||
| `pool_min` / `pool_max` | `3` / `24` | 动态连接池上下限 |
|
||||
| `mac_pool_cap` | `4` | 单台 MAC 服务器连接数上限 |
|
||||
| `std_pool_cap` | `4` | 单台标准服务器连接数上限 |
|
||||
| `ex_pool_cap` | `2` | 单台 EX 服务器连接数上限(EX 主机较少且较慢,压太高收益为负) |
|
||||
| `df` | `None` | 表格型接口的返回形态:`None` = `list[dataclass]`(与 0.6.x 一致);`"polars"` / `"pandas"` = 直接返回 DataFrame。见 [DataFrame 输出](dataframe.md) |
|
||||
|
||||
### 典型配置
|
||||
|
||||
|
||||
+13
-3
@@ -35,8 +35,9 @@ asyncio.run(main())
|
||||
| ⚡ **并发分页** | 把大请求拆成显式 `start` 偏移的页,多连接并行拉取、按序拼接、失败换连接重试 |
|
||||
| 🔀 **双通道整合** | 标准通道(经典行情)+ MAC 通道(逐笔 / 复权 / 1分钟K线),按命令自动选路 |
|
||||
| 📄 **服务器文件解析** | 板块 `block_*.dat` / 行业 `tdxhy.cfg` / 历史财报 `gpcw*` **下载即解析**成 dataclass(纯 stdlib),无需自己拆二进制;文件名与索引不一致时自动纠正 |
|
||||
| 🚀 **批量下载** | `get_bars` / `get_minutes` 传标的列表即多股票并发,连接池自动分配到不同服务器 |
|
||||
| 📦 **零重型依赖** | 返回 `list[dataclass]`,pandas / polars 由调用方按需转换 |
|
||||
| 🚀 **批量下载** | `get_bars` / `get_minutes` 传标的列表即多股票并发,连接池自动分配到不同服务器;K 线走「多连接 × 单连接流水线」,全市场 5,225 只 1 分钟线取当天 **5~8 秒**(盘中增量 1~2 秒) |
|
||||
| 🐼 **DataFrame 直出** | `TdxData(df="polars")` / `df="pandas"` 让 42 个表格型接口直接返回 DataFrame;时间列自动合并成 `datetime`,五档报价自动展开。后端是可选 extras,**核心包仍零依赖** |
|
||||
| 📦 **零重型依赖** | 默认返回 `list[dataclass]`,行为与 0.6.x 完全一致,转不转 DataFrame 由你决定 |
|
||||
|
||||
## 从这里开始
|
||||
|
||||
@@ -66,11 +67,19 @@ asyncio.run(main())
|
||||
|
||||
[:octicons-arrow-right-24: Cookbook](cookbook.md)
|
||||
|
||||
- **直接要 DataFrame**
|
||||
|
||||
---
|
||||
|
||||
`TdxData(df="polars")` 一行开关,42 个表格型接口直接出表,不用自己 `asdict`。
|
||||
|
||||
[:octicons-arrow-right-24: DataFrame 输出](dataframe.md)
|
||||
|
||||
- **查接口签名**
|
||||
|
||||
---
|
||||
|
||||
28 个统一方法(+ 33 个 deprecated 别名),从 docstring 实时生成,不会过期。
|
||||
35 个统一方法(+ 33 个 deprecated 别名),从 docstring 实时生成,不会过期。
|
||||
|
||||
[:octicons-arrow-right-24: API 参考](api/index.md)
|
||||
|
||||
@@ -122,6 +131,7 @@ asyncio.run(main())
|
||||
- [安装与配置](getting-started.md)
|
||||
- [场景示例](cookbook.md)
|
||||
- [命令行工具](cli.md)
|
||||
- [DataFrame 输出](dataframe.md)
|
||||
|
||||
- **参考**
|
||||
- [接口总览](api/index.md)
|
||||
|
||||
@@ -81,6 +81,30 @@ results = await asyncio.gather(*[
|
||||
|
||||
EWMA(指数加权移动平均)延迟让选路对「最近变慢」敏感,而不是被历史均值拖住。
|
||||
|
||||
### 单台服务器中途掉线
|
||||
|
||||
「掉线」和「慢」是两回事:慢只是多等一会儿,掉线会把整批任务卡住。0.7.2 起有四道闸:
|
||||
|
||||
| 机制 | 作用 |
|
||||
|---|---|
|
||||
| **锁外建连** | `acquire` 锁内只预留槽位,TCP 握手 + 探针在锁外做。否则一台主机握手卡满 `timeout` 期间,所有并发协程的 `acquire` / `release` 全被池锁挡住 |
|
||||
| **本轮不重试同一台** | 一次 `acquire` 里握手失败的主机记进 `avoid`,不再重复试;每台都试完立刻抛 `PoolTimeoutError` |
|
||||
| **池级熔断** | 全主机失败后进入 `BLACKOUT_COOLDOWN`(2 秒)黑洞期,期内 `acquire` 直接失败、不再握手;任一连接成功即解除 |
|
||||
| **批次回队 / 份额再分配** | 批量 K 线流水线失败的那一批立刻回队首由健康连接接手;分页器让活着的 worker 接手坏 worker 剩下的页 |
|
||||
|
||||
实测 16 台替身节点、6,000 只 × 240 根 1 分钟 K 线、跑起来 0.9 秒后集体黑洞:
|
||||
|
||||
| 场景 | 0.7.1 | 0.7.2 |
|
||||
|---|---|---|
|
||||
| 无节点故障 | 1.79 s | 1.80 s |
|
||||
| 2/16 中途失联 | 2.28 s | 1.91 s |
|
||||
| 6/16 中途失联 | 2.97 s | 2.30 s |
|
||||
| 16/16 中途失联 | **> 60 s 未返回** | **2.37 s** |
|
||||
|
||||
全灭时的行为分两种:**一开始就没有可用连接** → `get_bars` 抛 `PoolTimeoutError`(不再静默
|
||||
返回空列表);**跑到一半全灭** → 返回已经取到的部分(不抛错、不挂死)。熔断只在**全部主机**
|
||||
都握手失败时触发,池子满了排队等待不受影响。
|
||||
|
||||
### MAC 通道的单台连接上限
|
||||
|
||||
`mac_pool_cap` 默认 `4`,与标准池的 `std_pool_cap` 一致——这是**对端单台连接限制**,
|
||||
|
||||
+25
-1
@@ -45,7 +45,13 @@ asyncio.run(main())
|
||||
标的一律用字符串;以下均为 v0.6 统一接口,跨市场自动选路。
|
||||
|
||||
- **标的清单**:`get_universe`(A股 + 港股 + 美股 + 期货/期权)。⚠️ 北交所(`market="bj"`)清单**协议不支持**,返回空列表(`0x053E` 对 market=2 无响应,27 台服务器全测);但北交所**标的本身可用** —— 报价/K线/逐笔直接传 `bj920992` 这类代码正常
|
||||
- **K 线**:`get_bars`(自动并发分页、支持复权与日期区间、多标的并发)
|
||||
- **K 线**:`get_bars`(自动并发分页、支持复权与日期区间、多标的并发)。传 ≥16 只时自动走
|
||||
**多连接流水线**(`pipeline` 默认自适应 8~32,传 `1` 关闭),全市场 5,200 只 1 分钟 K 线
|
||||
取当天 43.6s → 5s 级;`start_date=今天` 盘中只拉已走过的根数(10:00 → 30 根),实时刷新 1~2 秒
|
||||
- ⚠️ **EX 通道(港股/美股/期货/期权)的 `amount` 不是全有的**:`0x23FF` 帧内只有一个 4B 槽位可复用,
|
||||
权益类(港股主板 31/创业板 48/基金 49、新三板 44、美股 74)装**成交额**(`amount` 正常),
|
||||
期货期权装**结算价**(见 `SecurityBar.settle`,日线才有值)→ 它们 `amount` 恒 **0.0**,
|
||||
是协议限制而非解析漏读;港股主板/基金的 `vol` 单位是**手(=100 股)**,`amount/vol` 是均价的 100 倍
|
||||
- **技术指标**:`get_indicators`(K线 + MACD/KDJ/RSI/BOLL/MA/EMA)
|
||||
- **报价**:`get_quotes`(五档,>80 只自动切批;**单连接流水线**默认开启,全市场 5,200 只 A股 ~0.33s)、`get_snapshot`(个股特征)、`get_market_stat`
|
||||
- 价格标度:协议给的是整数,真实价 = `整数 / 10**小数位`。小数位由 `price_decimal_digits` 决定,
|
||||
@@ -63,6 +69,24 @@ asyncio.run(main())
|
||||
> 旧的分通道/市场方法(`get_kline` / `get_transactions` / `get_goods_*` 等共 33 个)保留为
|
||||
> deprecated 别名,1.0 移除;完整对照见[接口重构审查](https://openbot-coder.github.io/pytdxdata/refactor-audit/)与 CHANGELOG。
|
||||
|
||||
## DataFrame 输出(v0.7+)
|
||||
|
||||
表格型接口可以直出 pandas / polars DataFrame,开关只有构造参数 `df`:
|
||||
|
||||
```python
|
||||
from pytdxdata import TdxData
|
||||
|
||||
async with TdxData(df="polars") as td: # 或 df="pandas";df=None(默认)= list[dataclass]
|
||||
df = await td.get_bars("sh600000", count=240)
|
||||
```
|
||||
|
||||
- 需先装 extra:`pip install "pytdxdata[polars]"` / `[pandas]` / `[dataframe]`。**核心包仍零依赖**。
|
||||
- `df=None` 时行为与 0.6.x 完全一致(装饰器直接透传,零开销);`df` 非法抛 `ValueError`,包没装抛 `FrameBackendError`(带安装命令)。
|
||||
- 列口径:**时间列替换**拆开的 `year/month/day/hour/minute` → `datetime`(+ `trading_date`);`bid`/`ask` 五档展开成 10 列;`fields` 字典 JSON 化成字符串列(空字典给 `None`);`TdxBlock.codes` 保持列表。
|
||||
- `get_indicators` → bars + 指标合并成一张表;`get_index_trade_stats(_multi)` → `minutes` 展开并补 `index`/`date` 常量列;`get_kline_batch`/`get_minute_batch` → 各标的纵向拼接成一张表。
|
||||
- **不受影响**:`bytes`/`int`/`dict`/单对象/`tuple`/`list[float]`/`list[str]` 类接口原样返回(如 `get_file`、`get_security_count`、`get_server_info`)。
|
||||
- 空结果返回**列名齐全的空表**,但 dtype 只由后端推断。
|
||||
|
||||
## 服务器与配置
|
||||
|
||||
服务器主机来源有一条优先级链(各协议独立):环境变量 → **当日测速缓存** → 包内清单
|
||||
|
||||
@@ -47,6 +47,7 @@ nav:
|
||||
- 安装与配置: getting-started.md
|
||||
- 场景示例: cookbook.md
|
||||
- 命令行工具: cli.md
|
||||
- DataFrame 输出: dataframe.md
|
||||
- API 参考:
|
||||
- 接口总览: api/index.md
|
||||
- TdxData 方法: api/tdxdata.md
|
||||
|
||||
+7
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "pytdxdata"
|
||||
version = "0.6.9"
|
||||
version = "0.7.3"
|
||||
description = "通达信(TongDaXin)行情数据客户端:动态连接池 + 双层TTL缓存 + 3连接并发分页,纯asyncio零运行时依赖,支持A股/港股/美股/期货/期权"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
@@ -31,6 +31,9 @@ Documentation = "https://openbot-coder.github.io/pytdxdata/"
|
||||
|
||||
[project.optional-dependencies]
|
||||
uvloop = ["uvloop>=0.21"]
|
||||
pandas = ["pandas>=2.0"]
|
||||
polars = ["polars>=1.0"]
|
||||
dataframe = ["pandas>=2.0", "polars>=1.0"]
|
||||
dev = ["pytest>=8.0", "pytest-asyncio>=0.24", "ruff>=0.6"]
|
||||
|
||||
[project.scripts]
|
||||
@@ -52,6 +55,9 @@ testpaths = ["tests"]
|
||||
dev = [
|
||||
"pytest>=9.1.1",
|
||||
"pytest-asyncio>=1.4.0",
|
||||
# DataFrame 后端的双实现都要在本地/CI 里被真实覆盖,不能只在 extras 里声明
|
||||
"pandas>=2.0",
|
||||
"polars>=1.0",
|
||||
]
|
||||
docs = [
|
||||
"mkdocs-material>=9.5",
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
"""坏节点隔离压测:N 台服务器,其中几台**跑到一半突然死掉**。
|
||||
|
||||
关注两个指标:
|
||||
1. **总耗时** —— 死节点不能把整批任务拖长(老实现会因为持锁建连 + 串行补
|
||||
重试把长尾放大成倍)。
|
||||
2. **数据完整度** —— 死节点不能造成缺数据。
|
||||
|
||||
用法::
|
||||
|
||||
python scripts/bench_failover.py --syms 3000 --hosts 16 --dead 5,9
|
||||
|
||||
输出一行 JSON,便于对比改前/改后。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import time
|
||||
|
||||
import pytdxdata.api as api
|
||||
import pytdxdata.pool.pool as poolmod
|
||||
from pytdxdata import TdxData
|
||||
from pytdxdata.models import KlinePeriod, SecurityBar
|
||||
|
||||
NOW = time.monotonic()
|
||||
PIPE_LATENCY = 0.012 # 一次流水线往返(模拟公网 12ms)
|
||||
CONNECT_LATENCY = 0.05 # 正常主机 TCP 握手 + 握手命令
|
||||
DEAD_HANG = 0.8 # 死节点:请求发出去没响应,最终超时
|
||||
CONNECT_HANG = 0.0 # 黑洞握手:连上了但握手永不完(0=直接拒绝)
|
||||
|
||||
DEAD_AT = 0.0 # 死节点从第几秒开始不响应
|
||||
|
||||
|
||||
class FakeConn:
|
||||
dead: set[int] = set()
|
||||
calls = 0
|
||||
pipes = 0
|
||||
dead_pipe_hits = 0 # 打到死节点的流水线次数(含"半开等待")
|
||||
|
||||
def __init__(self, host, port=7709, timeout=3.0):
|
||||
self.host = host
|
||||
self.idx = int(host[1:])
|
||||
self._alive = True
|
||||
|
||||
@property
|
||||
def alive(self) -> bool:
|
||||
return self._alive
|
||||
|
||||
async def connect(self) -> None:
|
||||
if self._is_dead() and CONNECT_HANG:
|
||||
# 黑洞节点:TCP 连上但握手包永远不回 —— 真实宕机最常见的样子
|
||||
await asyncio.sleep(CONNECT_HANG)
|
||||
raise TimeoutError(f"{self.host} 握手超时")
|
||||
await asyncio.sleep(CONNECT_LATENCY)
|
||||
if self._is_dead():
|
||||
raise ConnectionRefusedError(f"{self.host} 已宕机")
|
||||
|
||||
def _is_dead(self) -> bool:
|
||||
return self.idx in type(self).dead and time.monotonic() - NOW >= DEAD_AT
|
||||
|
||||
async def execute(self, cmd):
|
||||
await asyncio.sleep(PIPE_LATENCY)
|
||||
if self._is_dead():
|
||||
raise ConnectionError(f"{self.host} 无响应")
|
||||
type(self).calls += 1
|
||||
return self._rows(cmd)
|
||||
|
||||
async def execute_pipeline(self, cmds, **kwargs):
|
||||
await asyncio.sleep(PIPE_LATENCY)
|
||||
if self._is_dead():
|
||||
type(self).dead_pipe_hits += 1
|
||||
await asyncio.sleep(DEAD_HANG) # 半开连接:等到超时才报错
|
||||
raise ConnectionError(f"{self.host} 无响应")
|
||||
type(self).pipes += 1
|
||||
return [self._rows(c) for c in cmds]
|
||||
|
||||
async def close(self) -> None:
|
||||
self._alive = False
|
||||
|
||||
@staticmethod
|
||||
def _rows(cmd):
|
||||
if "MacKlineCmd" not in type(cmd).__name__:
|
||||
return []
|
||||
bar = SecurityBar(
|
||||
market=1, code="600000", open=10.0, high=10.1, low=9.9, close=10.0,
|
||||
vol=1000, amount=10000, year=2026, month=1, day=1, hour=9, minute=30)
|
||||
return [bar] * int(getattr(cmd, "count", 1)) # 同一个对象复用,便于控内存
|
||||
|
||||
|
||||
async def run(n_syms: int, n_hosts: int, dead: set[int]) -> dict:
|
||||
global NOW
|
||||
NOW = time.monotonic()
|
||||
FakeConn.dead = dead
|
||||
FakeConn.calls = 0
|
||||
FakeConn.pipes = 0
|
||||
FakeConn.dead_pipe_hits = 0
|
||||
hosts = [f"h{i}" for i in range(n_hosts)]
|
||||
api.get_mac_hosts = lambda: hosts
|
||||
api.get_standard_hosts = lambda: hosts # 别让标准池去碰真实公网主机
|
||||
api.get_ex_hosts = lambda: []
|
||||
poolmod.schedule_probe = lambda: None # 关掉后台全量测速
|
||||
poolmod.TdxConnection = FakeConn # 池内部按名字取,必须在建池之前替身
|
||||
# cap=1:一台主机一条连接,坏一台就少一条链路(最坏情况)
|
||||
td = TdxData(cache_dir=None, pool_min=4, pool_max=24, mac_pool_cap=1)
|
||||
await td.start()
|
||||
syms = [f"sh60{i:04d}" for i in range(n_syms)]
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
bars = await td.get_bars(syms, KlinePeriod.MIN_1, count=240)
|
||||
finally:
|
||||
elapsed = time.monotonic() - t0
|
||||
await td.close()
|
||||
return {
|
||||
"syms": n_syms,
|
||||
"hosts": n_hosts,
|
||||
"dead": sorted(dead),
|
||||
"elapsed_s": round(elapsed, 3),
|
||||
"bars": len(bars),
|
||||
"expect_bars": n_syms * 240,
|
||||
"complete": len(bars) == n_syms * 240,
|
||||
"pipeline_calls": FakeConn.pipes,
|
||||
"dead_pipe_hits": FakeConn.dead_pipe_hits,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--syms", type=int, default=3000)
|
||||
ap.add_argument("--hosts", type=int, default=16)
|
||||
ap.add_argument("--dead", default="5,9",
|
||||
help="中途死掉的主机下标,逗号分隔;空串=全健康")
|
||||
ap.add_argument("--die-at", type=float, default=0.25,
|
||||
help="开跑后第几秒这些主机开始不响应(0=一开始就死)")
|
||||
ap.add_argument("--connect-hang", type=float, default=0.0,
|
||||
help="死节点的握手卡顿秒数(>0 模拟黑洞节点,0=直接拒绝)")
|
||||
args = ap.parse_args()
|
||||
dead = {int(x) for x in args.dead.split(",") if x.strip()}
|
||||
global DEAD_AT, CONNECT_HANG
|
||||
DEAD_AT = args.die_at
|
||||
CONNECT_HANG = args.connect_hang
|
||||
print(json.dumps(asyncio.run(run(args.syms, args.hosts, dead)),
|
||||
ensure_ascii=False))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -18,7 +18,7 @@ from .models import (
|
||||
XdxrRecord,
|
||||
)
|
||||
|
||||
__version__ = "0.6.9"
|
||||
__version__ = "0.7.2"
|
||||
__all__ = [
|
||||
"TdxData",
|
||||
"Market",
|
||||
|
||||
+460
-20
@@ -27,7 +27,7 @@ import time
|
||||
import warnings
|
||||
from collections import deque
|
||||
from collections.abc import Callable
|
||||
from datetime import date as _date, datetime
|
||||
from datetime import date as _date, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Sequence
|
||||
|
||||
@@ -38,6 +38,7 @@ from .cache.manager import CacheManager
|
||||
from .cache.memory import MemoryCache
|
||||
from .codec.payload import pack_payload, unpack_payload
|
||||
from .config import EX_HANDSHAKES, get_ex_hosts, get_mac_hosts, get_standard_hosts
|
||||
from .frames import framed, get_backend
|
||||
from .models import (
|
||||
Adjust,
|
||||
AuctionItem,
|
||||
@@ -145,6 +146,54 @@ _KLINE_MAX_BARS: dict[KlinePeriod, int] = {
|
||||
KlinePeriod.MONTH: 500,
|
||||
}
|
||||
|
||||
# 每交易日大约多少根(倒推时按时间跨度估要拉多少,估小了会多跑一轮)
|
||||
_BARS_PER_DAY: dict[KlinePeriod, float] = {
|
||||
KlinePeriod.MIN_1: 240,
|
||||
KlinePeriod.MIN_5: 48,
|
||||
KlinePeriod.MIN_15: 16,
|
||||
KlinePeriod.MIN_30: 8,
|
||||
KlinePeriod.MIN_60: 4,
|
||||
KlinePeriod.DAY: 1,
|
||||
KlinePeriod.WEEK: 0.2,
|
||||
KlinePeriod.MONTH: 0.05,
|
||||
}
|
||||
_BACKTRACK_MIN_BARS = 240 # 倒推起始根数:一天 1 分钟正好 240,一次命中
|
||||
_BACKTRACK_MAX_BARS = 20_000 # 单次别拉太多;超了回落到二分
|
||||
_BACKTRACK_ROUNDS = 5 # 最多放大 5 轮(240→7680)
|
||||
_BACKTRACK_GRACE = timedelta(days=10) # 右端离最新多远还算「贴着最新」(覆盖长假)
|
||||
|
||||
# 一根K线跨多长时间。K线时间戳标的是**该周期的结束时刻**,所以最新一根可能
|
||||
# 比 datetime.now() 略大(14:29 那根此刻还在走)——判断「右端贴最新」要给这个容差。
|
||||
_BAR_SPAN: dict[KlinePeriod, timedelta] = {
|
||||
KlinePeriod.MIN_1: timedelta(minutes=1),
|
||||
KlinePeriod.MIN_5: timedelta(minutes=5),
|
||||
KlinePeriod.MIN_15: timedelta(minutes=15),
|
||||
KlinePeriod.MIN_30: timedelta(minutes=30),
|
||||
KlinePeriod.MIN_60: timedelta(hours=1),
|
||||
KlinePeriod.DAY: timedelta(days=1),
|
||||
KlinePeriod.WEEK: timedelta(days=7),
|
||||
KlinePeriod.MONTH: timedelta(days=31),
|
||||
}
|
||||
|
||||
# ---- K线批量流水线(get_bars 多标的路径)----
|
||||
# 单只K线是「一请求一标的」,全市场 5,000 只就是 5,000 次往返。实测单连接
|
||||
# 串行 33ms/只 ≈ 1 个 RTT,**瓶颈是往返不是带宽**;同连接连续下发 pipe 条请求
|
||||
# (协议帧自描述 + 服务器按序应答)能把往返折叠掉。实测 24 台 × 8 流水:
|
||||
# 全市场 1 分钟K线取当天 43.6s → 5.1s。
|
||||
_BULK_MIN_SYMS = 16 # 低于此数量走逐只路径(流水线收益抵不过批量开销)
|
||||
_BULK_PIPE_MIN = 8 # 自适应深度下限
|
||||
_BULK_PIPE_MAX = 32 # 自适应深度上限(再深服务器会攒够一批才回,反而慢)
|
||||
_BULK_MAX_COUNT = 700 # MAC 单请求返回上限;超过只能逐只分页
|
||||
# 最多用几台服务器:池里 24 台全用反而更慢 —— 慢节点会成为长尾(最长的那台
|
||||
# 拖住整轮),实测 count=2 全市场 24 台 2.95s / 只取最快 12 台 1.45s。
|
||||
_BULK_MAX_CONNS = 12
|
||||
# 流水线失败的一批最多回队几次。回队 = 交给**其它健康连接**立刻接手;
|
||||
# 上限保证全节点失联时不会无限打转,超限后落到尾部并发兜底。
|
||||
_BULK_MAX_RETRY = 2
|
||||
# 交易时段(分钟):上午 9:30-11:30、下午 13:00-15:00,共 240 分钟
|
||||
_AM_OPEN, _AM_CLOSE = 9 * 60 + 30, 11 * 60 + 30
|
||||
_PM_OPEN, _PM_CLOSE = 13 * 60, 15 * 60
|
||||
|
||||
# 自定义字段报价(0x122B/0x122C)基础字段位:昨收/开/高/低/现价/量/量比/额
|
||||
_QUOTE_BITS = [0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07]
|
||||
_QUOTE_PAGE = 80 # 0x122B/0x122C 单次上限 80 只
|
||||
@@ -166,6 +215,65 @@ _list_unsupported_warned: set[int] = set() # 同一市场只 WARNING 一次
|
||||
_HK_MARKETS = frozenset({27, 31, 48, 49, 71, 98})
|
||||
|
||||
|
||||
def _elapsed_trading_minutes(now: datetime) -> int:
|
||||
"""今天到现在为止已过去的**交易分钟数**(0~240)。
|
||||
|
||||
只按钟点算,不区分节假日(非交易日时区间裁剪会把这些根全滤掉,不影响正确性);
|
||||
盘中调用是常态,10:00 只需 30 根而不是 240 根 —— 数据量直接差 8 倍。
|
||||
"""
|
||||
m = now.hour * 60 + now.minute
|
||||
if m < _AM_OPEN:
|
||||
return 0
|
||||
if m <= _AM_CLOSE:
|
||||
return m - _AM_OPEN
|
||||
if m < _PM_OPEN:
|
||||
return 120 # 午休:上午已走完
|
||||
if m <= _PM_CLOSE:
|
||||
return 120 + (m - _PM_OPEN)
|
||||
return 240 # 收盘后
|
||||
|
||||
|
||||
def _bars_for_range(period: KlinePeriod, start_dt: datetime,
|
||||
now: datetime) -> int:
|
||||
"""估算「从 ``start_dt`` 到最新」需要拉多少根(倒推初值)。"""
|
||||
per_day = _BARS_PER_DAY.get(period, 1)
|
||||
if start_dt.date() == now.date():
|
||||
# 当天:按已过交易分钟数折算,别把一整天都拉下来(10:00 只需 30 根而非 240)
|
||||
frac = _elapsed_trading_minutes(now) / 240
|
||||
return max(1, int(per_day * frac) + 5)
|
||||
days = max(0.0, (now - start_dt).total_seconds() / 86400.0)
|
||||
return max(1, int(days * per_day * 1.6))
|
||||
|
||||
|
||||
def _auto_pipeline(count: int, pipeline: int | None) -> int:
|
||||
"""流水深度:默认按「在途字节数 ≈60KB」自适应,夹在 8~32。
|
||||
|
||||
轮数 = 请求数 / (连接数 × 深度),深度越深往返折叠得越彻底;但请求太深时
|
||||
服务器会把整批处理完才回,实测 24 台 depth=64 反而比 8 慢一倍(5.79s vs 2.95s)。
|
||||
每根的应答固定 32 字节,所以「取 240 根」天然只用得着浅流水,
|
||||
「只取最新 2 根」则要用满深流水。
|
||||
"""
|
||||
if pipeline is not None:
|
||||
return max(2, int(pipeline))
|
||||
per_req = max(1, count * 32 + 33)
|
||||
return max(_BULK_PIPE_MIN, min(_BULK_PIPE_MAX, 60_000 // per_req))
|
||||
|
||||
|
||||
def _norm_range(start_date, end_date) -> tuple[datetime | None, datetime | None]:
|
||||
"""日期参数归一化:``date`` → 当日 00:00 / 23:59:59.999999。"""
|
||||
start_dt = None
|
||||
if start_date is not None:
|
||||
start_dt = (start_date if isinstance(start_date, datetime)
|
||||
else datetime.combine(start_date, datetime.min.time()))
|
||||
end_dt = None
|
||||
if end_date is not None:
|
||||
end_dt = (end_date if isinstance(end_date, datetime)
|
||||
else datetime.combine(end_date, datetime.max.time()))
|
||||
if end_dt.time() == datetime.min.time():
|
||||
end_dt = end_dt.replace(hour=23, minute=59, second=59, microsecond=999999)
|
||||
return start_dt, end_dt
|
||||
|
||||
|
||||
def _fields_tag(bits: Sequence[int] | None) -> str:
|
||||
"""字段位集合 -> 缓存键片段。"""
|
||||
if not bits:
|
||||
@@ -249,7 +357,19 @@ class TdxData:
|
||||
mac_pool_cap: int = 4,
|
||||
std_pool_cap: int = 4,
|
||||
ex_pool_cap: int = 2,
|
||||
df: str | None = None,
|
||||
) -> None:
|
||||
# df=None(默认)保持 0.6.x 的行为:表格型方法一律返回 list[dataclass]。
|
||||
# df="polars" / df="pandas" 时这些方法直接返回对应后端的 DataFrame。
|
||||
# 这里早失败一次:名字非法抛 ValueError,包没装抛 FrameBackendError(带安装命令)。
|
||||
if df is not None:
|
||||
get_backend(df)
|
||||
self._df_backend = df
|
||||
|
||||
# K线偏移方向是**通道级属性**(不是每只标的的属性),首次探测后实例内复用:
|
||||
# 全市场 5,000 只各探 2 次往返 = 上万次请求,纯属浪费。
|
||||
self._kline_dir_cache: dict[str, bool] = {}
|
||||
|
||||
self.port = port
|
||||
self.timeout = timeout
|
||||
self.default_adjust = default_adjust
|
||||
@@ -326,6 +446,7 @@ class TdxData:
|
||||
# 一、标的清单
|
||||
# ================================================================== #
|
||||
|
||||
@framed
|
||||
async def get_universe(self, market: str | None = None) -> list[SecurityInfo]:
|
||||
"""标的清单(A股证券列表 + 扩展市场商品列表),一次覆盖所有市场。
|
||||
|
||||
@@ -472,6 +593,7 @@ class TdxData:
|
||||
# 二、报价
|
||||
# ================================================================== #
|
||||
|
||||
@framed
|
||||
async def get_quotes(self, symbols: str | Sequence[str], *,
|
||||
fields: Sequence[int] | None = None,
|
||||
concurrency: int | None = None,
|
||||
@@ -649,6 +771,7 @@ class TdxData:
|
||||
await self._std_pool.release(pc)
|
||||
return rows
|
||||
|
||||
@framed
|
||||
async def get_snapshot(self, symbols: str | Sequence[str]) -> list[SymbolSnapshot]:
|
||||
"""个股特征快照(当日/换手/活跃度等,仅 A股)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -668,6 +791,7 @@ class TdxData:
|
||||
out.append(rec)
|
||||
return out
|
||||
|
||||
@framed
|
||||
async def get_auction(self, symbols: str | Sequence[str]) -> list[AuctionItem]:
|
||||
"""集合竞价(仅 A股)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -680,6 +804,7 @@ class TdxData:
|
||||
out.extend(items)
|
||||
return out
|
||||
|
||||
@framed
|
||||
async def get_capital_flow(self, symbols: str | Sequence[str]) -> list[CapitalFlow]:
|
||||
"""个股资金流向(仅 A股)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -696,6 +821,7 @@ class TdxData:
|
||||
# 三、K线
|
||||
# ================================================================== #
|
||||
|
||||
@framed
|
||||
async def get_bars(
|
||||
self,
|
||||
symbols: str | Sequence[str],
|
||||
@@ -707,6 +833,7 @@ class TdxData:
|
||||
start_date: datetime | None = None,
|
||||
end_date: datetime | None = None,
|
||||
concurrency: int = 6,
|
||||
pipeline: int | None = None,
|
||||
) -> list[SecurityBar]:
|
||||
"""K线(多标的并发,自动判指数、自动复权选路、自动跨市场)。
|
||||
|
||||
@@ -718,12 +845,30 @@ class TdxData:
|
||||
adjust: 复权方式(默认用实例 ``default_adjust``);``QFQ``/``HFQ`` 自动走 MAC 通道。
|
||||
start_date / end_date: 按日期区间取(与 start/count 互斥,优先日期过滤)。
|
||||
只给 ``start_date`` 即增量更新语义(取该日期之后的全部K线)。
|
||||
concurrency: 逐只路径的并发数(仅在对小批量/标准通道生效时使用)。
|
||||
pipeline: **批量路径**每条连接同时在途的请求数。``None``(默认)按单请求
|
||||
字节数自适应(约 60KB 在途,夹在 8~32);传 ``1`` 关闭流水线走逐只。
|
||||
批量路径把 N 次往返折叠成 ``N/pipeline`` 次 —— 全市场 5,000 只取当天
|
||||
1 分钟K线从 43.6s 降到 5s 级。
|
||||
|
||||
返回:
|
||||
``list[SecurityBar]``,按入参标的顺序拼接,每根带 ``symbol``;组内按时间正序。
|
||||
|
||||
说明:
|
||||
批量路径只在 A股/指数(非扩展市场)且区间贴着最新时启用;历史中段
|
||||
仍需逐只二分定位,会回落到逐只路径(结果一致,只是慢一些)。
|
||||
"""
|
||||
period = KlinePeriod(int(period))
|
||||
syms = parse_symbols(symbols)
|
||||
|
||||
# 批量路径:一次请求一标的,逐只走就是 N 次往返;多连接流水线能把往返压掉
|
||||
if len(syms) >= _BULK_MIN_SYMS and pipeline != 1:
|
||||
bulk = await self._bars_bulk(
|
||||
syms, period, start=start, count=count, adjust=adjust,
|
||||
start_date=start_date, end_date=end_date, pipeline=pipeline)
|
||||
if bulk is not None:
|
||||
return bulk
|
||||
|
||||
sem = asyncio.Semaphore(max(1, concurrency))
|
||||
|
||||
async def one(sym: Symbol) -> list[SecurityBar]:
|
||||
@@ -735,6 +880,161 @@ class TdxData:
|
||||
groups = await asyncio.gather(*(one(s) for s in syms))
|
||||
return [b for g in groups for b in g]
|
||||
|
||||
async def _bars_bulk(
|
||||
self, syms: Sequence[Symbol], period: KlinePeriod, *,
|
||||
start: int, count: int, adjust: Adjust | None,
|
||||
start_date: datetime | None, end_date: datetime | None,
|
||||
pipeline: int | None,
|
||||
) -> list[SecurityBar] | None:
|
||||
"""多标的批量取K线(多连接流水线)。返回 ``None`` = 不适用,调用方回退逐只。
|
||||
|
||||
不适用只有两种情况:含扩展市场标的、需要历史中段二分定位。
|
||||
"""
|
||||
if any(s.is_ex for s in syms):
|
||||
return None
|
||||
adj = self.default_adjust if adjust is None else adjust
|
||||
pool, page = self._bulk_channel(period, adj)
|
||||
|
||||
def make_cmd(s: Symbol, o: int, c: int):
|
||||
if pool is self._mac_pool:
|
||||
return MacKlineCmd(s.market, s.code, period, adj, o, c)
|
||||
if is_cn_index(s.market, s.code):
|
||||
return GetIndexBarsCmd(s.market, s.code, period, o, c)
|
||||
return GetSecurityBarsCmd(s.market, s.code, period, o, c)
|
||||
|
||||
if start_date is None and end_date is None:
|
||||
pipe = _auto_pipeline(min(count, page), pipeline)
|
||||
got = await self._bulk_bars(pool, make_cmd, syms, start, count, page, pipe,
|
||||
period=period, adj=adj)
|
||||
return [b for s in syms for b in (got.get(s.text) or [])]
|
||||
|
||||
start_dt, end_dt = _norm_range(start_date, end_date)
|
||||
now = datetime.now()
|
||||
if end_dt is not None and end_dt + _BACKTRACK_GRACE < now:
|
||||
return None # 历史中段:得先探总根数再二分,批量帮不上
|
||||
if start_dt is None:
|
||||
need = _BACKTRACK_MIN_BARS
|
||||
else:
|
||||
if start_dt > now + _BAR_SPAN.get(period, timedelta(days=1)):
|
||||
return [] # 整段区间在未来
|
||||
need = _bars_for_range(period, start_dt, now)
|
||||
# 偏移方向是通道级属性:探一次(结果按通道缓存,全市场只付一次代价)
|
||||
if not await self._kline_newest_at_zero(syms[0], period, adj):
|
||||
return None
|
||||
|
||||
pipe = _auto_pipeline(min(need, page), pipeline)
|
||||
got = await self._bulk_bars(pool, make_cmd, syms, 0, need, page, pipe,
|
||||
period=period, adj=adj)
|
||||
out: list[SecurityBar] = []
|
||||
for s in syms:
|
||||
bars = got.get(s.text) or []
|
||||
if start_dt is not None:
|
||||
bars = [b for b in bars if b.datetime >= start_dt]
|
||||
if end_dt is not None:
|
||||
bars = [b for b in bars if b.datetime <= end_dt]
|
||||
out.extend(bars)
|
||||
return out
|
||||
|
||||
def _bulk_channel(self, period: KlinePeriod, adj: Adjust):
|
||||
"""批量路径用哪个池 / 单请求最多几根。"""
|
||||
if route_for("kline", adjust=adj, period=period) == "mac":
|
||||
return self._mac_pool, _BULK_MAX_COUNT
|
||||
return self._std_pool, KLINE_PAGE
|
||||
|
||||
async def _bulk_bars(
|
||||
self, pool, make_cmd, syms: Sequence[Symbol],
|
||||
start: int, count: int, page: int, pipeline: int, *,
|
||||
period: KlinePeriod, adj: Adjust,
|
||||
) -> dict[str, list[SecurityBar]]:
|
||||
"""多连接 + 单连接流水线抓K线,按 ``symbol.text`` 归集。
|
||||
|
||||
为什么快:单只一请求,串行时候每只固定付一个 RTT(实测 33ms);同一连接
|
||||
连续发 ``pipeline`` 条请求,服务器按序应答,于是 ``pipeline`` 只只共付
|
||||
一个 RTT。再叠加多台服务器并行(每台 1 连接,尊重 per_server_cap),
|
||||
实测全市场 1 分钟K线取当天:43.6s → 5s 级。
|
||||
|
||||
单请求装不下 ``start+count`` 时按 ``page`` 切页,多页与其他标的的请求
|
||||
混在同一批里流水线发出去(不是逐只串行翻页)。
|
||||
"""
|
||||
plan: list[tuple[int, int]] = []
|
||||
off, remain = start, start + count
|
||||
while remain > 0:
|
||||
take = min(page, remain)
|
||||
plan.append((off, take))
|
||||
off += take
|
||||
remain -= take
|
||||
|
||||
tasks: deque[tuple[Symbol, int, int]] = deque(
|
||||
(s, o, c) for s in syms for o, c in plan)
|
||||
n_conn = min(len(tasks), pool.max_size, _BULK_MAX_CONNS,
|
||||
max(2, -(-len(tasks) // max(1, pipeline * 4))))
|
||||
await pool.warm(n_conn) # 并发建连;acquire 建连是串行的,先热好
|
||||
held = await pool.acquire_many(n_conn)
|
||||
if not held:
|
||||
return {}
|
||||
|
||||
by_text = {s.text: s for s in syms}
|
||||
acc: dict[str, list[SecurityBar]] = {}
|
||||
retry: dict[str, Symbol] = {}
|
||||
attempts: dict[str, int] = {}
|
||||
|
||||
async def worker(pc) -> None:
|
||||
broken = False
|
||||
try:
|
||||
while tasks:
|
||||
batch: list[tuple[Symbol, int, int]] = []
|
||||
while tasks and len(batch) < pipeline:
|
||||
batch.append(tasks.popleft())
|
||||
cmds = [make_cmd(s, o, c) for s, o, c in batch]
|
||||
try:
|
||||
res = await pc.execute_pipeline(cmds)
|
||||
except Exception:
|
||||
# 流水线中途失败 = 响应流已错配,连接作废。
|
||||
# 这一批**立刻放回队首**:其它健康连接会马上接手,
|
||||
# 不必等 gather 结束再串行补(老实现每条重试都要重新
|
||||
# 付一次 RTT,几百只就是十几秒的长尾)。
|
||||
broken = True
|
||||
again: list[tuple[Symbol, int, int]] = []
|
||||
for item in batch:
|
||||
s = item[0]
|
||||
n = attempts.get(s.text, 0)
|
||||
if n < _BULK_MAX_RETRY:
|
||||
attempts[s.text] = n + 1
|
||||
again.append(item)
|
||||
else:
|
||||
retry[s.text] = s
|
||||
if again:
|
||||
tasks.extendleft(reversed(again))
|
||||
return
|
||||
for (s, _, _), bars in zip(batch, res):
|
||||
acc.setdefault(s.text, []).extend(bars)
|
||||
finally:
|
||||
await pool.release(pc, broken=broken)
|
||||
|
||||
await asyncio.gather(*(worker(pc) for pc in held))
|
||||
# 连接全挂时 worker 会提前 return,队列里剩下的标的没人取 —— 一并交回重试,
|
||||
# 否则会**静默少数据**(实测 20 只只回 16 只)。
|
||||
while tasks:
|
||||
s, _, _ = tasks.popleft()
|
||||
retry.setdefault(s.text, s)
|
||||
out = {k: self._finish_bars(v, by_text[k]) for k, v in acc.items()}
|
||||
if retry:
|
||||
# 尾部兜底也并发做:坏节点把批次吐回来时可能有几十上百只,
|
||||
# 串行重试等于把「一台机器卡住」放大成「整批卡住」。
|
||||
sem = asyncio.Semaphore(max(2, min(len(retry), pool.max_size)))
|
||||
|
||||
async def rescue(text: str, s: Symbol) -> None:
|
||||
async with sem:
|
||||
try:
|
||||
out[text] = await self._bars_one(
|
||||
s, period, start=start, count=count, adjust=adj,
|
||||
start_date=None, end_date=None)
|
||||
except Exception:
|
||||
out.setdefault(text, [])
|
||||
|
||||
await asyncio.gather(*(rescue(t, s) for t, s in retry.items()))
|
||||
return out
|
||||
|
||||
async def _bars_one(
|
||||
self, sym: Symbol, period: KlinePeriod, *,
|
||||
start: int, count: int, adjust: Adjust | None,
|
||||
@@ -777,23 +1077,43 @@ class TdxData:
|
||||
async def _bars_ex(self, sym: Symbol, period: KlinePeriod, adj: Adjust,
|
||||
start: int, count: int) -> list[SecurityBar]:
|
||||
"""扩展市场K线(EX 0x23FF;category=周期编码,与 TDXParams 一致)。"""
|
||||
from .protocol.commands.ex_proto import GetExInstrumentBarsCmd
|
||||
from .protocol.commands.ex_proto import GetExInstrumentBarsCmd, _EX_AMOUNT_MARKETS
|
||||
cat = int(period)
|
||||
fetch_count = max(700, start + count)
|
||||
# extra 带的 @v3 是缓存 schema 标记:v3 起期货/期权回填估算 amount + position;
|
||||
# 旧版本写下的缓存里 amount 恒 0.0、position 丢失,必须让它们自然失效。
|
||||
key = CacheKey("kline", sym.market, sym.code, period.name, start,
|
||||
fetch_count, f"{adj.name}@{cat}")
|
||||
fetch_count, f"{adj.name}@{cat}@v3")
|
||||
# 期货/期权乘数(估算成交额用);未知品种乘数为 None → amount 保持 0.0
|
||||
from .ex_multiplier import find_multiplier
|
||||
from .symbols import _EX_PREFIX
|
||||
mult = None
|
||||
if sym.market not in _EX_AMOUNT_MARKETS:
|
||||
mult = find_multiplier(_EX_PREFIX.get(sym.market, ""), sym.code, sym.market)
|
||||
|
||||
async def fetch() -> bytes:
|
||||
rows = await self._fetch_paginated_ex(
|
||||
lambda s2, c2: GetExInstrumentBarsCmd(sym.market, sym.code, cat, s2, c2),
|
||||
PageSpec(page_size=700, total=fetch_count, min_page=100))
|
||||
bars = [SecurityBar(
|
||||
market=sym.market, code=sym.code,
|
||||
year=d["year"], month=d["month"], day=d["day"],
|
||||
hour=d.get("hour", 0), minute=d.get("minute", 0),
|
||||
open=d["open"], high=d["high"], low=d["low"], close=d["close"],
|
||||
vol=float(d.get("trade", 0) or 0), amount=0.0,
|
||||
) for d in rows]
|
||||
bars = []
|
||||
for d in rows:
|
||||
vol = float(d.get("trade", 0) or 0)
|
||||
amount = float(d.get("amount", 0.0) or 0.0)
|
||||
if amount == 0.0 and mult and vol:
|
||||
# 期货/期权:帧内无成交额,按收盘价估算(见 ex_multiplier 模块 docstring)
|
||||
amount = vol * float(d["close"]) * mult
|
||||
pos = d.get("position", 0) or 0
|
||||
bars.append(SecurityBar(
|
||||
market=sym.market, code=sym.code,
|
||||
year=d["year"], month=d["month"], day=d["day"],
|
||||
hour=d.get("hour", 0), minute=d.get("minute", 0),
|
||||
open=d["open"], high=d["high"], low=d["low"], close=d["close"],
|
||||
vol=vol,
|
||||
amount=amount,
|
||||
settle=d.get("settle"),
|
||||
# 权益类市场 position 恒 0 → None;期货/期权给真实持仓量
|
||||
position=float(pos) if pos else None,
|
||||
))
|
||||
return pack_payload(bars)
|
||||
|
||||
bars = self._finish_bars(unpack_payload(await self._cache.get_or_fetch(key, fetch)), sym)
|
||||
@@ -815,21 +1135,23 @@ class TdxData:
|
||||
start_date: datetime | None, end_date: datetime | None,
|
||||
adjust: Adjust | None,
|
||||
) -> list[SecurityBar]:
|
||||
"""按日期区间取K线:先二分定位偏移,再只拉目标区间。
|
||||
"""按日期区间取K线:贴最新走倒推,历史中段才二分定位。
|
||||
|
||||
关键坑:**偏移方向各通道不一致**(实测标准通道 offset 0 是最新一根、
|
||||
越大越老;MAC/EX 反过来),所以先探一次方向再二分,不做静态假设。
|
||||
|
||||
快慢两条路径(:meth:`_bars_backtrack` 优先):
|
||||
倒推 = 直接 ``count=N`` 拿最新 N 根再裁剪,1~3 次往返;
|
||||
二分 = 先探总根数再 log2 次跳转,长周期上是 36+ 次串行往返。
|
||||
「取今天」「增量到最新」都贴着右端,占了实际使用的绝大多数。
|
||||
"""
|
||||
# 归一化日期边界
|
||||
start_dt = (start_date if isinstance(start_date, datetime)
|
||||
else datetime.combine(start_date, datetime.min.time())) \
|
||||
if start_date is not None else None
|
||||
end_dt = (end_date if isinstance(end_date, datetime)
|
||||
else datetime.combine(end_date, datetime.max.time())) \
|
||||
if end_date is not None else None
|
||||
# end_date 传入 datetime 但时间为 00:00 时视为"当天结束"
|
||||
if end_dt is not None and end_dt.time() == datetime.min.time().replace(tzinfo=None):
|
||||
end_dt = end_dt.replace(hour=23, minute=59, second=59, microsecond=999999)
|
||||
start_dt, end_dt = _norm_range(start_date, end_date)
|
||||
|
||||
# 快路径:区间右端贴着最新 → 从最新倒推,1~3 次往返
|
||||
fast = await self._bars_backtrack(sym, period, adjust, start_dt, end_dt)
|
||||
if fast is not None:
|
||||
return fast
|
||||
|
||||
est = _KLINE_MAX_BARS.get(period, 10_000)
|
||||
total = await self._bar_total(sym, period, adjust, est)
|
||||
@@ -839,6 +1161,7 @@ class TdxData:
|
||||
|
||||
if desc:
|
||||
# offset 越大越老:[0,k) 全都 >= start_date,[j,∞) 全都 <= end_date
|
||||
# 不做并行二分:两个协程同时抢连接会触发服务器侧限速,实测比串行更慢
|
||||
k = total
|
||||
if start_dt is not None:
|
||||
k = await self._bisect_offset(
|
||||
@@ -870,6 +1193,87 @@ class TdxData:
|
||||
bars = [b for b in bars if b.datetime <= end_dt]
|
||||
return bars
|
||||
|
||||
async def _bars_backtrack(
|
||||
self, sym: Symbol, period: KlinePeriod, adjust: Adjust | None,
|
||||
start_dt: datetime | None, end_dt: datetime | None,
|
||||
) -> list[SecurityBar] | None:
|
||||
"""从最新一根往前倒推目标区间(区间右端贴着最新时适用)。
|
||||
|
||||
返回 ``None`` 表示**不适用**(偏移方向相反 / 要的是历史中间段 /
|
||||
区间太长放大不划算),调用方应回落到二分路径;返回列表表示已算完
|
||||
(**空列表**也是有效结果,代表区间内无数据)。
|
||||
|
||||
为什么快:二分要先 ``_bar_total`` 翻倍探总根数(长周期 6 次)再跳
|
||||
``log2(总根数)`` 次,1 分钟 K 线约 36 次串行往返 = 4 秒级;
|
||||
倒推只是「拿最新 N 根」,一次就能覆盖一天。
|
||||
|
||||
**标准通道不探测方向**:``_bars_one`` 的 ``count`` 路径本来就靠
|
||||
「start=0 即最新端」这个协议事实,这里沿用同一假定,省掉 2 次往返
|
||||
——全市场 5,000 只时每只省 2 次,是几十秒的量级。
|
||||
只有 MAC/EX 才真去探(它们 offset 方向相反)。
|
||||
"""
|
||||
adj = self.default_adjust if adjust is None else adjust
|
||||
span = _BAR_SPAN.get(period, timedelta(days=1))
|
||||
now = datetime.now()
|
||||
|
||||
if not await self._kline_newest_at_zero(sym, period, adj):
|
||||
return None # offset 0 是最老端 → 倒推不成立,交给二分
|
||||
|
||||
if start_dt is not None and start_dt > now + span:
|
||||
return [] # 整段区间都在未来
|
||||
# 右端够不着最新(留 _BACKTRACK_GRACE 余量覆盖周末/长假)→ 走二分
|
||||
if end_dt is not None and end_dt + _BACKTRACK_GRACE < now:
|
||||
return None
|
||||
ref = now
|
||||
|
||||
# 初值按时间跨度估根数:一天内 1 次就够,长区间也基本 1 次到位
|
||||
count = _BACKTRACK_MIN_BARS
|
||||
if start_dt is not None:
|
||||
need = _bars_for_range(period, start_dt, ref)
|
||||
# 当天(盘中)实际只要已走完的那几十根,别沿用一整天 240 根
|
||||
count = max(need, 8) if start_dt.date() == ref.date() else max(count, need)
|
||||
count = min(count, _BACKTRACK_MAX_BARS)
|
||||
|
||||
bars: list[SecurityBar] = []
|
||||
for _ in range(_BACKTRACK_ROUNDS):
|
||||
bars = await self.get_bars(sym.text, period, start=0, count=count,
|
||||
adjust=adjust)
|
||||
if not bars:
|
||||
return []
|
||||
first = bars[0].datetime
|
||||
# 覆盖到 start 即止;跨到自己之前的那一天也算够(K线连续,中间不会缺)
|
||||
if start_dt is None or first <= start_dt or first.date() < start_dt.date():
|
||||
break
|
||||
if count >= _BACKTRACK_MAX_BARS:
|
||||
return None
|
||||
count = min(count * 2, _BACKTRACK_MAX_BARS)
|
||||
else:
|
||||
return None # 放大到上限仍不够 → 交给二分
|
||||
|
||||
if start_dt is not None:
|
||||
bars = [b for b in bars if b.datetime >= start_dt]
|
||||
if end_dt is not None:
|
||||
bars = [b for b in bars if b.datetime <= end_dt]
|
||||
return bars
|
||||
|
||||
async def _kline_newest_at_zero(self, sym: Symbol, period: KlinePeriod,
|
||||
adjust: Adjust) -> bool:
|
||||
"""K 线的 ``offset=0`` 是否就是**最新**一根(True 才能倒推)。
|
||||
|
||||
偏移方向是通道级属性,与具体标的无关,首次探测后按通道名缓存在实例上。
|
||||
"""
|
||||
key = "ex" if sym.is_ex else route_for("kline", adjust=adjust, period=period)
|
||||
cached = self._kline_dir_cache.get(key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
newest = await self._probe_bar(sym, period, adjust, 0)
|
||||
if newest is None:
|
||||
return False # 探不到就别猜,交给二分去处理空数据
|
||||
nxt = await self._probe_bar(sym, period, adjust, 1)
|
||||
desc = nxt is None or nxt.datetime < newest.datetime
|
||||
self._kline_dir_cache[key] = desc
|
||||
return desc
|
||||
|
||||
async def _bar_total(self, sym: Symbol, period: KlinePeriod,
|
||||
adjust: Adjust | None, est: int) -> int:
|
||||
"""指数扩张找K线总根数上界(探测到越界为止)。"""
|
||||
@@ -937,6 +1341,7 @@ class TdxData:
|
||||
return None
|
||||
return bars[0] if bars else None
|
||||
|
||||
@framed
|
||||
async def get_indicators(
|
||||
self, symbols: str | Sequence[str], indicators: Sequence[str], *,
|
||||
period: KlinePeriod = KlinePeriod.DAY, count: int = 30,
|
||||
@@ -975,6 +1380,7 @@ class TdxData:
|
||||
# 四、逐笔成交 / 分时
|
||||
# ================================================================== #
|
||||
|
||||
@framed
|
||||
async def get_ticks(
|
||||
self, symbols: str | Sequence[str], *,
|
||||
date: int | None = None, start: int = 0, count: int | None = None,
|
||||
@@ -1156,6 +1562,7 @@ class TdxData:
|
||||
trade_count_available=False, # 扩展市场逐笔不带笔数字段
|
||||
)
|
||||
|
||||
@framed
|
||||
async def get_minutes(
|
||||
self, symbols: str | Sequence[str], *,
|
||||
date: int | None = None, sampling: bool = False, concurrency: int = 6,
|
||||
@@ -1236,6 +1643,7 @@ class TdxData:
|
||||
# 五、公司信息
|
||||
# ================================================================== #
|
||||
|
||||
@framed
|
||||
async def get_xdxr(self, symbols: str | Sequence[str]) -> list[XdxrRecord]:
|
||||
"""除权除息历史,仅 A股。标的用字符串(``"sz000001"``)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -1254,6 +1662,7 @@ class TdxData:
|
||||
out.extend(recs)
|
||||
return out
|
||||
|
||||
@framed
|
||||
async def get_finance(self, symbols: str | Sequence[str]) -> list[FinanceRecord]:
|
||||
"""财务快照(仅 A股)。无数据的标的会被跳过(原单标的接口返回 None)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -1324,6 +1733,7 @@ class TdxData:
|
||||
stat.limit_down_count = int(q6.open * 10)
|
||||
return stat
|
||||
|
||||
@framed
|
||||
async def get_unusual(self, market: str = "all", *, start: int = 0,
|
||||
count: int = 600) -> list[UnusualItem]:
|
||||
"""市场异动(仅 A股;``market="all"`` = 深市 + 沪市)。"""
|
||||
@@ -1359,6 +1769,7 @@ class TdxData:
|
||||
f"未知板块类别: {kind}(可选: {'/'.join(table)} 或数字)")
|
||||
return table[key]
|
||||
|
||||
@framed
|
||||
async def get_boards(self, kind: str = "all", *, count: int = 10000) -> list[BoardInfo]:
|
||||
"""板块列表(``kind``: ``all``/``industry``/``concept``/``style``,也接受数字)。
|
||||
|
||||
@@ -1380,6 +1791,7 @@ class TdxData:
|
||||
b.symbol = format_symbol(b.market, b.code)
|
||||
return items
|
||||
|
||||
@framed
|
||||
async def get_board_members(self, board: str, count: int = 100000,
|
||||
sort_type: int = 0, *,
|
||||
fields: Sequence[int] | None = None
|
||||
@@ -1405,6 +1817,7 @@ class TdxData:
|
||||
break
|
||||
return items
|
||||
|
||||
@framed
|
||||
async def get_board_of(self, symbols: str | Sequence[str]) -> list[BelongBoard]:
|
||||
"""个股所属板块(仅 A股;``symbol`` 记录被查询的个股)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -1433,11 +1846,13 @@ class TdxData:
|
||||
"down_count": down,
|
||||
}
|
||||
|
||||
@framed
|
||||
async def get_board_ranking(self, kind: str = "industry", *,
|
||||
top_n: int = 20) -> list[dict]:
|
||||
"""板块当日涨跌幅排行(基于板块报价的现价/昨收)。"""
|
||||
return self._rank_boards(await self.get_boards(kind, count=200), top_n)
|
||||
|
||||
@framed
|
||||
async def get_board_change_ranking(self, kind: str = "industry", *,
|
||||
days: int = 20, top_n: int = 20) -> list[dict]:
|
||||
"""板块 N 日涨跌幅排行(基于板块指数K线)。"""
|
||||
@@ -1522,6 +1937,7 @@ class TdxData:
|
||||
return 2, s
|
||||
raise ValueError(f"无法判定指数市场: {index!r}(不是 A股指数代码段)")
|
||||
|
||||
@framed
|
||||
async def get_index_members(self, index: str, *, count: int = 100000,
|
||||
fields: Sequence[int] | None = None
|
||||
) -> list[SecurityQuote]:
|
||||
@@ -1570,6 +1986,7 @@ class TdxData:
|
||||
break
|
||||
return items[:count]
|
||||
|
||||
@framed
|
||||
async def get_index_trade_stats(
|
||||
self,
|
||||
index: str,
|
||||
@@ -1694,6 +2111,7 @@ class TdxData:
|
||||
elapsed=time.perf_counter() - started,
|
||||
)
|
||||
|
||||
@framed
|
||||
async def get_index_trade_stats_multi(
|
||||
self,
|
||||
index: str,
|
||||
@@ -1774,6 +2192,7 @@ class TdxData:
|
||||
break
|
||||
return bytes(data)
|
||||
|
||||
@framed
|
||||
async def get_block_parsed(self, filename: str) -> list[TdxBlock]:
|
||||
"""板块文件 -> ``list[TdxBlock]``(板块名/类别/成分股)。
|
||||
|
||||
@@ -1788,11 +2207,13 @@ class TdxData:
|
||||
from .codec.industry import parse_tdxhy_cfg
|
||||
return parse_tdxhy_cfg(await self.get_file("tdxhy.cfg"))
|
||||
|
||||
@framed
|
||||
async def get_financial_file_infos(self) -> list[FinancialFileInfo]:
|
||||
"""tdxfin/gpcw.txt -> ``list[FinancialFileInfo]``(文件名/MD5/大小)。"""
|
||||
from .codec.financial import parse_financial_file_list
|
||||
return parse_financial_file_list(await self.get_file("tdxfin/gpcw.txt"))
|
||||
|
||||
@framed
|
||||
async def get_financial_records_parsed(self, filename: str) -> list[FinancialRecord]:
|
||||
"""财报数据解析 -> ``list[FinancialRecord]``。
|
||||
|
||||
@@ -1986,6 +2407,7 @@ class TdxData:
|
||||
# 九、旧接口(deprecated 别名;入参沿用旧口径,1.0 移除)
|
||||
# ================================================================== #
|
||||
|
||||
@framed
|
||||
async def get_kline(self, market, code, period, *, start: int = 0,
|
||||
count: int = 800, adjust: Adjust | None = None,
|
||||
start_date: datetime | None = None,
|
||||
@@ -1996,6 +2418,7 @@ class TdxData:
|
||||
start=start, count=count, adjust=adjust,
|
||||
start_date=start_date, end_date=end_date)
|
||||
|
||||
@framed
|
||||
async def get_kline_since(self, market, code, period, since: datetime,
|
||||
adjust: Adjust | None = None) -> list[SecurityBar]:
|
||||
"""已废弃 -> :meth:`get_bars`(``start_date=since``)。"""
|
||||
@@ -2003,6 +2426,7 @@ class TdxData:
|
||||
return await self.get_bars(format_symbol(int(market), code), period,
|
||||
start_date=since, adjust=adjust)
|
||||
|
||||
@framed
|
||||
async def get_kline_batch(self, stocks, period, *, start: int = 0,
|
||||
count: int = 800, adjust: Adjust | None = None,
|
||||
concurrency: int = 6) -> dict[str, list[SecurityBar]]:
|
||||
@@ -2016,6 +2440,7 @@ class TdxData:
|
||||
out.setdefault(b.code, []).append(b)
|
||||
return out
|
||||
|
||||
@framed
|
||||
async def get_index_kline(self, market, code, period, *, start: int = 0,
|
||||
count: int = 800) -> list[SecurityBar]:
|
||||
"""已废弃 -> :meth:`get_bars`(指数自动识别,无需单独接口)。"""
|
||||
@@ -2036,12 +2461,14 @@ class TdxData:
|
||||
one = sets[0]
|
||||
return {"bars": one.bars, "indicators": one.indicators}
|
||||
|
||||
@framed
|
||||
async def get_stock_quotes(self, stocks, bits=None) -> list[SecurityQuote]:
|
||||
"""已废弃 -> :meth:`get_quotes`(``fields=bits``)。"""
|
||||
_deprecated("get_stock_quotes", "get_quotes")
|
||||
return await self.get_quotes([format_symbol(int(m), c) for m, c in stocks],
|
||||
fields=bits)
|
||||
|
||||
@framed
|
||||
async def get_stock_quotes_list(self, category: int, *, start: int = 0,
|
||||
count: int = 80, sort_type: int = 0
|
||||
) -> list[SecurityQuote]:
|
||||
@@ -2050,6 +2477,7 @@ class TdxData:
|
||||
return await self._board_members_raw(category, start=start, count=count,
|
||||
sort_type=sort_type)
|
||||
|
||||
@framed
|
||||
async def get_transactions(self, market, code, *, date: int | None = None,
|
||||
start: int = 0, count: int = 2000
|
||||
) -> list[TransactionRecord]:
|
||||
@@ -2058,12 +2486,14 @@ class TdxData:
|
||||
return await self.get_ticks(format_symbol(int(market), code), date=date,
|
||||
start=start, count=count)
|
||||
|
||||
@framed
|
||||
async def get_minute(self, market, code, *, date: int | None = None
|
||||
) -> list[MinuteBar]:
|
||||
"""已废弃 -> :meth:`get_minutes`。"""
|
||||
_deprecated("get_minute", "get_minutes")
|
||||
return await self.get_minutes(format_symbol(int(market), code), date=date)
|
||||
|
||||
@framed
|
||||
async def get_minute_batch(self, stocks, *, date: int | None = None,
|
||||
concurrency: int = 6) -> dict[str, list[MinuteBar]]:
|
||||
"""已废弃 -> :meth:`get_minutes`(返回值由 ``{code: bars}`` 变为平铺列表)。"""
|
||||
@@ -2086,6 +2516,7 @@ class TdxData:
|
||||
_deprecated("get_security_count", "get_universe")
|
||||
return await self._security_count(int(market))
|
||||
|
||||
@framed
|
||||
async def get_security_list(self, market, *, start: int = 0,
|
||||
count: int | None = None) -> list[SecurityInfo]:
|
||||
"""已废弃 -> :meth:`get_universe`(本地切片 ``[start:start+count]``)。"""
|
||||
@@ -2093,6 +2524,7 @@ class TdxData:
|
||||
items = await self.get_universe(market_name(int(market)))
|
||||
return items[start:] if count is None else items[start:start + count]
|
||||
|
||||
@framed
|
||||
async def get_security_list_all(self, market) -> list[SecurityInfo]:
|
||||
"""已废弃 -> :meth:`get_universe`。"""
|
||||
_deprecated("get_security_list_all", "get_universe")
|
||||
@@ -2146,12 +2578,14 @@ class TdxData:
|
||||
_deprecated("download_file", "get_file")
|
||||
return await self.get_file(filename)
|
||||
|
||||
@framed
|
||||
async def get_board_list(self, board_type: int | str = 0,
|
||||
count: int = 10000) -> list[BoardInfo]:
|
||||
"""已废弃 -> :meth:`get_boards`。"""
|
||||
_deprecated("get_board_list", "get_boards")
|
||||
return await self.get_boards(board_type, count=count)
|
||||
|
||||
@framed
|
||||
async def get_belong_board(self, market, code) -> list[BelongBoard]:
|
||||
"""已废弃 -> :meth:`get_board_of`。"""
|
||||
_deprecated("get_belong_board", "get_board_of")
|
||||
@@ -2163,6 +2597,7 @@ class TdxData:
|
||||
items = await self.get_snapshot(format_symbol(int(market), code))
|
||||
return items[0] if items else None
|
||||
|
||||
@framed
|
||||
async def get_goods_list(self, market, start: int = 0,
|
||||
count: int = 100) -> list[SecurityInfo]:
|
||||
"""已废弃 -> :meth:`get_universe`(扩展到 ``SecurityInfo`` 统一模型)。"""
|
||||
@@ -2177,12 +2612,14 @@ class TdxData:
|
||||
return len(await self._ex_all())
|
||||
return len(await self._universe_ex(int(market)))
|
||||
|
||||
@framed
|
||||
async def get_goods_quotes(self, stocks, bits=None) -> list[SecurityQuote]:
|
||||
"""已废弃 -> :meth:`get_quotes`。"""
|
||||
_deprecated("get_goods_quotes", "get_quotes")
|
||||
return await self.get_quotes([format_symbol(int(m), c) for m, c in stocks],
|
||||
fields=bits)
|
||||
|
||||
@framed
|
||||
async def get_goods_quotes_list(self, market, count: int = 80
|
||||
) -> list[SecurityQuote]:
|
||||
"""已废弃 -> :meth:`get_quotes`(或 ``get_universe`` 取清单后自选)。"""
|
||||
@@ -2198,6 +2635,7 @@ class TdxData:
|
||||
bars = await self.get_minutes(format_symbol(int(market), code), sampling=True)
|
||||
return [b.price for b in bars]
|
||||
|
||||
@framed
|
||||
async def get_goods_tick_chart(self, market, code,
|
||||
ymd: int = 0) -> list[MinuteBar]:
|
||||
"""已废弃 -> :meth:`get_minutes`(扩展到统一分时模型)。"""
|
||||
@@ -2205,6 +2643,7 @@ class TdxData:
|
||||
return await self.get_minutes(format_symbol(int(market), code),
|
||||
date=ymd or None)
|
||||
|
||||
@framed
|
||||
async def get_goods_transaction(self, market, code, *, ymd: int = 0,
|
||||
start: int = 0, count: int = 1800
|
||||
) -> list[TransactionRecord]:
|
||||
@@ -2213,6 +2652,7 @@ class TdxData:
|
||||
return await self.get_ticks(format_symbol(int(market), code),
|
||||
date=ymd or None, start=start, count=count)
|
||||
|
||||
@framed
|
||||
async def get_goods_transaction_all(self, market, code,
|
||||
ymd: int = 0) -> list[TransactionRecord]:
|
||||
"""已废弃 -> :meth:`get_ticks`(``count=None`` 即全量翻页)。"""
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
"""EX 期货/期权合约乘数表(估算成交额用)。
|
||||
|
||||
0x23FF 帧内没有成交额槽位(见 ex_proto.py 注释)。要给期货/期权 K 线一个
|
||||
可用的 ``amount``,只能客户端自算。本表覆盖主流境内品种,键为**品种代码
|
||||
小写字母前缀**(从合约代码提取:``RB2611`` → ``rb``,``IFL0`` → ``if``)。
|
||||
|
||||
⚠️ 口径:``amount = vol × close × multiplier`` —— 按**收盘价**估算,
|
||||
不是 Σ(逐笔成交价×手数×乘数) 的精确值,日线级别误差通常 <2%(收盘价
|
||||
成交集中假设)。对多数策略足够;要精确成交额请用逐笔聚合。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
# 中金所(乘数:点×元)
|
||||
_CFFEX = {
|
||||
"if": 200, "ih": 300, "ic": 200, "im": 200,
|
||||
"ifl0": 200, "ihl0": 300, "icl0": 200, "iml0": 200,
|
||||
"tf": 10_000, "t": 10_000, "ts": 10_000, "tl": 10_000, "tp": 10_000,
|
||||
"mo": 50, "ho": 50, "io": 100,
|
||||
}
|
||||
# 上期所
|
||||
_SHF = {
|
||||
"rb": 10, "hc": 10, "wr": 10, "cu": 5, "al": 5, "zn": 5, "pb": 5,
|
||||
"ni": 1, "sn": 1, "ss": 5, "ao": 20, "au": 100, "ag": 15,
|
||||
"fu": 10, "bu": 10, "ru": 10, "br": 5, "sp": 10, "lu": 10, "nr": 10,
|
||||
"sc": 1000, "bc": 5, "ec": 50, "sh": 1_000_000, "dii": 100,
|
||||
}
|
||||
# 大商所
|
||||
_DL = {
|
||||
"a": 10, "b": 10, "m": 10, "y": 10, "p": 10, "c": 10, "cs": 20,
|
||||
"i": 100, "j": 100, "jm": 60, "jd": 5, "fb": 10, "bb": 500,
|
||||
"l": 5, "v": 5, "pp": 5, "eg": 10, "eb": 5, "pg": 20, "rr": 10,
|
||||
"lh": 16, "px": 10, "lg": 50,
|
||||
}
|
||||
# 郑商所
|
||||
_ZZ = {
|
||||
"sr": 10, "cf": 5, "cy": 5, "ap": 10, "cj": 5, "pj": 10,
|
||||
"ta": 5, "ma": 10, "fg": 20, "sa": 20, "ur": 20, "sf": 5, "sm": 5,
|
||||
"oi": 10, "rm": 10, "zc": 100, "wh": 20, "pm": 50, "ri": 20, "jr": 20,
|
||||
"lr": 20, "pf": 5, "sh": 30, "pk": 5, "cft": 5,
|
||||
}
|
||||
# 广期所
|
||||
_GZ = {"si": 5, "lc": 20, "ps": 5}
|
||||
|
||||
_MARKET_TABLES = {
|
||||
"cffex": _CFFEX,
|
||||
"shf": _SHF,
|
||||
"dl": _DL,
|
||||
"zz": _ZZ,
|
||||
"gz": _GZ,
|
||||
}
|
||||
|
||||
# 期权乘数:金融期权按标的;商品期权 = 标的期货乘数(同一品种字母前缀命中即可)。
|
||||
# 沪深股票期权(50ETF/300ETF/500ETF/科创50ETF/创业板ETF)乘数 10000。
|
||||
_STOCK_OPTION_MULTIPLIER = 10_000
|
||||
|
||||
|
||||
def product_code(code: str) -> str:
|
||||
"""合约代码 -> 品种小写前缀(``RB2611``→``rb``、``IFL0``→``ifl0``、``m2501``→``m``)。
|
||||
|
||||
规则:取前缀直到首个数字;连续/主力后缀(L0、M0 等)保留在品种键里,
|
||||
表中给主力键与普通键都录入(见 _CFFEX 的 IFL0/IF 双键)。
|
||||
"""
|
||||
s = code.strip().lower()
|
||||
out = []
|
||||
for ch in s:
|
||||
if ch.isdigit():
|
||||
break
|
||||
out.append(ch)
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def find_multiplier(ex_prefix: str, code: str, market: int | None = None) -> float | None:
|
||||
"""(EX 市场短前缀, 合约代码[, 市场号]) -> 合约乘数;未知返回 None。
|
||||
|
||||
期权市场按**市场号**判断(IntEnum 名含 OPTION):股票期权乘数 10000,
|
||||
商品期权(ZZ/DL/CFFEX 期权市场号 4/5/7)回落到对应商品品种表。
|
||||
"""
|
||||
if market is not None:
|
||||
name = _OPTION_MARKET_NAMES.get(int(market))
|
||||
if name == "stock":
|
||||
return _STOCK_OPTION_MULTIPLIER
|
||||
if name == "commodity":
|
||||
ex_prefix = {"ZZ_FUTURES_OPTION": "zz", "DL_FUTURES_OPTION": "dl",
|
||||
"CFFEX_OPTION": "cffex", "GZ_OPTIONS": "gz"}.get(name, ex_prefix)
|
||||
s = ex_prefix.lower()
|
||||
if s.endswith("_option") or s in ("opt", "option"):
|
||||
return _STOCK_OPTION_MULTIPLIER
|
||||
table = _MARKET_TABLES.get(s)
|
||||
if table is None:
|
||||
return None
|
||||
prod = product_code(code)
|
||||
if prod in table:
|
||||
return float(table[prod])
|
||||
# 主力/连续后缀剥离后再试(IFL0 → IF)
|
||||
stripped = prod.rstrip("lmo0")
|
||||
return table.get(stripped)
|
||||
|
||||
|
||||
_OPTION_MARKET_NAMES = {
|
||||
4: "commodity", # ZZ_FUTURES_OPTION
|
||||
5: "commodity", # DL_FUTURES_OPTION
|
||||
7: "commodity", # CFFEX_OPTION
|
||||
67: "commodity", # GZ_OPTIONS
|
||||
8: "stock", # SH_STOCK_OPTION
|
||||
9: "stock", # SZ_STOCK_OPTION
|
||||
}
|
||||
@@ -0,0 +1,522 @@
|
||||
"""DataFrame 转换层(0.7.0):把模型列表转成 pandas / polars 的 DataFrame。
|
||||
|
||||
为什么单独一层而不是改各方法的返回:
|
||||
|
||||
- **零运行时依赖**是项目的硬约束(``dependencies = []``)。pandas / polars 只在
|
||||
真正用到 ``TdxData(df=...)`` 时才导入,没装就走不到这里。
|
||||
- 转换口径(哪些列留、时间怎么摆、嵌套字段怎么展开)只写一份,40 个返回表格型
|
||||
数据的方法共用,不会各自漂移。
|
||||
|
||||
三条口径约定(0.7.0 定稿):
|
||||
|
||||
1. **时间列替换原始列**:K线/分时的 ``year/month/day/hour/minute`` 不再单列输出,
|
||||
统一给一列 ``datetime``(pandas 为 ``datetime64[ns]``,polars 为 ``Datetime('us')``),
|
||||
另带 ``trading_date``(``YYYY-MM-DD`` 字符串,美股跨日口径见模型注释)。
|
||||
2. **嵌套字段定型**:``SecurityQuote.bid/ask`` 展开成 ``bid1_price``…``ask5_vol`` 共
|
||||
10 列(缺失为 ``None``);``fields`` 这类不定长字典序列化成 JSON 字符串列,
|
||||
空字典给 ``None`` 而不是 ``"{}"``;``TdxBlock.codes`` 保持列表。
|
||||
3. **嵌套调用不二次转换**:``get_kline_batch`` 这类方法内部还会调 ``get_bars``,
|
||||
用 :data:`_FRAME_DEPTH`(``ContextVar``,按 task 隔离)标记层次,内层一律返回
|
||||
原始 ``list``,只有最外层那次才转 DataFrame。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import json
|
||||
import re
|
||||
from collections.abc import Callable, Sequence
|
||||
from contextvars import ContextVar
|
||||
from datetime import datetime as _dt
|
||||
from enum import IntEnum
|
||||
from operator import attrgetter
|
||||
from typing import Any, NamedTuple
|
||||
|
||||
from . import models as _models
|
||||
from .models import (
|
||||
IndexTradeStats,
|
||||
IndicatorSet,
|
||||
MinuteBar,
|
||||
MinuteTradeStat,
|
||||
SecurityBar,
|
||||
SecurityQuote,
|
||||
TransactionRecord,
|
||||
XdxrRecord,
|
||||
)
|
||||
|
||||
__all__ = ["BACKENDS", "FrameBackendError", "framed", "to_frame"]
|
||||
|
||||
BACKENDS: tuple[str, ...] = ("polars", "pandas")
|
||||
"""支持的后端名,也是 ``TdxData(df=...)`` 的合法取值。"""
|
||||
|
||||
|
||||
class FrameBackendError(ImportError):
|
||||
"""指定的 DataFrame 后端未安装。
|
||||
|
||||
继承 :class:`ImportError` 是刻意的:捕获姿势和缺包一致,
|
||||
且 ``e.name`` 就是缺的那个包名。
|
||||
"""
|
||||
|
||||
def __init__(self, backend: str) -> None:
|
||||
self.backend = backend
|
||||
super().__init__(
|
||||
f"未安装 {backend},无法输出 DataFrame。请执行:pip install pytdxdata[{backend}]",
|
||||
name=backend,
|
||||
)
|
||||
|
||||
|
||||
# 转换层次计数:>0 表示当前正处于某次公开方法的执行过程中(内层),此时不转换。
|
||||
_FRAME_DEPTH: ContextVar[int] = ContextVar("pytdxdata_frame_depth", default=0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 后端
|
||||
|
||||
|
||||
class _Backend:
|
||||
"""后端适配:只封装「造表」和「拼表」两个动作。"""
|
||||
|
||||
name: str = ""
|
||||
|
||||
def frame(self, cols: dict[str, list]) -> Any: # pragma: no cover - 抽象
|
||||
raise NotImplementedError
|
||||
|
||||
def concat(self, frames: Sequence[Any]) -> Any: # pragma: no cover - 抽象
|
||||
raise NotImplementedError
|
||||
|
||||
def empty(self, cols: Sequence[str]) -> Any:
|
||||
return self.frame({name: [] for name in cols})
|
||||
|
||||
|
||||
class _PolarsBackend(_Backend):
|
||||
name = "polars"
|
||||
|
||||
def __init__(self) -> None:
|
||||
try:
|
||||
import polars as _pl
|
||||
except ImportError as exc: # pragma: no cover - 依赖环境
|
||||
raise FrameBackendError("polars") from exc
|
||||
self._pl = _pl
|
||||
|
||||
def frame(self, cols: dict[str, list]) -> Any:
|
||||
# 全空时必须显式给列名,否则 polars 会造出 0 列的表
|
||||
return self._pl.DataFrame(cols)
|
||||
|
||||
def concat(self, frames: Sequence[Any]) -> Any:
|
||||
frames = [f for f in frames if f is not None and f.height > 0]
|
||||
if not frames:
|
||||
return None
|
||||
if len(frames) == 1:
|
||||
return frames[0]
|
||||
# vertical_relaxed:批量接口各标的分块建表,dtype 可能略有差异(如 Null 列)
|
||||
return self._pl.concat(frames, how="vertical_relaxed")
|
||||
|
||||
|
||||
class _PandasBackend(_Backend):
|
||||
name = "pandas"
|
||||
|
||||
def __init__(self) -> None:
|
||||
try:
|
||||
import pandas as _pd
|
||||
except ImportError as exc: # pragma: no cover - 依赖环境
|
||||
raise FrameBackendError("pandas") from exc
|
||||
self._pd = _pd
|
||||
|
||||
def frame(self, cols: dict[str, list]) -> Any:
|
||||
return self._pd.DataFrame(cols)
|
||||
|
||||
def concat(self, frames: Sequence[Any]) -> Any:
|
||||
frames = [f for f in frames if f is not None and len(f) > 0]
|
||||
if not frames:
|
||||
return None
|
||||
if len(frames) == 1:
|
||||
return frames[0]
|
||||
return self._pd.concat(frames, ignore_index=True)
|
||||
|
||||
|
||||
_BACKEND_CACHE: dict[str, _Backend] = {}
|
||||
|
||||
|
||||
def get_backend(name: str) -> _Backend:
|
||||
"""取后端实例(进程内缓存,重复取不会重复 import)。
|
||||
|
||||
:raises ValueError: 名字不在 :data:`BACKENDS` 里。
|
||||
:raises FrameBackendError: 对应的包没装。
|
||||
"""
|
||||
if name not in BACKENDS:
|
||||
raise ValueError(f"未知的 DataFrame 后端 {name!r},可选:{', '.join(BACKENDS)}")
|
||||
be = _BACKEND_CACHE.get(name)
|
||||
if be is None:
|
||||
be = _PolarsBackend() if name == "polars" else _PandasBackend()
|
||||
_BACKEND_CACHE[name] = be
|
||||
return be
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 列计划
|
||||
|
||||
|
||||
def _as_int(value: Any) -> Any:
|
||||
"""``IntEnum`` 转纯 ``int``:polars 对 IntEnum 的类型推断不可靠,转掉最稳。"""
|
||||
if isinstance(value, IntEnum):
|
||||
return int(value)
|
||||
return value
|
||||
|
||||
|
||||
def _json_or_none(value: Any) -> str | None:
|
||||
"""不定长字典序列化成 JSON 列;空字典给 ``None``(别让用户看到一列 ``"{}"``)。"""
|
||||
if not value:
|
||||
return None
|
||||
return json.dumps(value, ensure_ascii=False, separators=(",", ":"))
|
||||
|
||||
|
||||
def _json_getter(name: str) -> Callable[[Any], Any]:
|
||||
def _get(row: Any) -> Any:
|
||||
return _json_or_none(getattr(row, name))
|
||||
|
||||
return _get
|
||||
|
||||
|
||||
def _level_getter(side: str, index: int, pos: int) -> Callable[[Any], Any]:
|
||||
"""``bid``/``ask`` 第 ``index`` 档的价格(pos=0)或数量(pos=1)。"""
|
||||
|
||||
def _get(row: Any) -> Any:
|
||||
levels = getattr(row, side)
|
||||
if len(levels) <= index:
|
||||
return None
|
||||
return levels[index][pos]
|
||||
|
||||
return _get
|
||||
|
||||
|
||||
def _moment(row: SecurityBar | MinuteBar) -> _dt:
|
||||
return row.datetime
|
||||
|
||||
|
||||
def _trading_date(row: SecurityBar | MinuteBar) -> str:
|
||||
return row.trading_date
|
||||
|
||||
|
||||
def _xdxr_day(row: XdxrRecord) -> _dt:
|
||||
# 除权日是「日」不是「时刻」,但 datetime64 / Datetime 才是可排序、可比较的列类型。
|
||||
# 全项目时间口径都是「不带 tzinfo 的北京时间」,这里保持一致。
|
||||
return _dt(row.year, row.month, row.day) # noqa: DTZ001
|
||||
|
||||
|
||||
#: 模型 -> 需要丢弃的原始字段(时间拆解字段,由增强列替代)
|
||||
_DROPS: dict[type, tuple[str, ...]] = {
|
||||
SecurityBar: ("year", "month", "day", "hour", "minute"),
|
||||
MinuteBar: ("hour", "minute", "date", "day_offset"),
|
||||
XdxrRecord: ("year", "month", "day"),
|
||||
SecurityQuote: ("bid", "ask"),
|
||||
}
|
||||
|
||||
#: 模型 -> 在「第一个被丢弃字段的位置」插入的增强列
|
||||
_EXTRAS: dict[type, tuple[tuple[str, Callable[[Any], Any]], ...]] = {
|
||||
SecurityBar: (("datetime", _moment), ("trading_date", _trading_date)),
|
||||
MinuteBar: (("datetime", _moment), ("trading_date", _trading_date)),
|
||||
XdxrRecord: (("date", _xdxr_day),),
|
||||
SecurityQuote: tuple(
|
||||
(f"{side}{level + 1}_{part}", _level_getter(side, level, pos))
|
||||
for side in ("bid", "ask")
|
||||
for level in range(5)
|
||||
for pos, part in ((0, "price"), (1, "vol"))
|
||||
),
|
||||
}
|
||||
|
||||
#: 模型 -> 需要 JSON 化的字典字段(按类区分,避免误伤 ``FinancialRecord.fields: list[float]``)
|
||||
_JSON_FIELDS: dict[type, frozenset[str]] = {
|
||||
SecurityQuote: frozenset({"fields"}),
|
||||
TransactionRecord: frozenset({"fields"}),
|
||||
MinuteBar: frozenset({"fields"}),
|
||||
}
|
||||
|
||||
#: 交易日列的缓存键(时间字段里参与 ``trading_date`` 计算的那些)。
|
||||
#:
|
||||
#: ``trading_date`` 是纯函数:A股/港股/期货就是自然日,美股凌晨段回退一天。
|
||||
#: 但它每算一次都要格式化三个整数(美股还要减一天),240k 行里往往只有
|
||||
#: **1~2 个不同的键** —— 按「除 minute 外的时间字段」缓存后,整列从 218ms 降到 33ms。
|
||||
#: 键里保留 ``hour`` 是为了一箭双雕:既覆盖美股 ``hour < 12`` 的跨日判断,
|
||||
#: 又不必把 ``_US_DAY_SPLIT_HOUR`` 的取值抄过来(切换点改了本文件也不用跟着改)。
|
||||
_TD_KEYS: dict[type, tuple[str, ...]] = {
|
||||
SecurityBar: ("year", "month", "day", "hour"),
|
||||
MinuteBar: ("date", "day_offset", "hour"),
|
||||
}
|
||||
|
||||
#: 交易日列名(走缓存路径,不按行取属性)
|
||||
_TD_COL = "trading_date"
|
||||
|
||||
#: 增强列里可以直接用 ``attrgetter`` 取的(属性本身即得值,不需要额外加工)
|
||||
_PLAIN_EXTRAS = frozenset({"datetime"})
|
||||
|
||||
|
||||
class _ColPlan(NamedTuple):
|
||||
"""某模型的输出列计划(构造一次,按类缓存)。
|
||||
|
||||
分成三类是性能决定的:逐行调 Python 闭包比 C 层 ``attrgetter`` 慢一倍以上,
|
||||
而 ``trading_date`` 这种「同一批里值大量重复」的列更是值得单独缓存。
|
||||
"""
|
||||
|
||||
order: tuple[str, ...] # 输出列顺序
|
||||
plain: tuple[str, ...] # 直接取属性即可的列名
|
||||
wrapped: tuple[tuple[str, Callable[[Any], Any]], ...] # 需要加工的列(JSON / 五档 / 除权日)
|
||||
td_name: str | None # 交易日列名
|
||||
td_key: Callable[[Any], Any] | None # 交易日缓存键
|
||||
|
||||
|
||||
_PLAN_CACHE: dict[type, _ColPlan] = {}
|
||||
|
||||
|
||||
def _plan_for(cls: type) -> _ColPlan:
|
||||
"""构造(并缓存)某模型的输出列计划。
|
||||
|
||||
查找顺序按 MRO 走:``IndexBar`` 没有自己的计划,但它是 ``SecurityBar`` 的子类,
|
||||
直接复用父类计划即可(字段完全相同)。
|
||||
"""
|
||||
cached = _PLAN_CACHE.get(cls)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
drops: tuple[str, ...] = ()
|
||||
extras: tuple[tuple[str, Callable[[Any], Any]], ...] = ()
|
||||
json_fields: frozenset[str] = frozenset()
|
||||
td_fields: tuple[str, ...] = ()
|
||||
for klass in cls.__mro__:
|
||||
if not drops and klass in _DROPS:
|
||||
drops = _DROPS[klass]
|
||||
if not extras and klass in _EXTRAS:
|
||||
extras = _EXTRAS[klass]
|
||||
if not json_fields and klass in _JSON_FIELDS:
|
||||
json_fields = _JSON_FIELDS[klass]
|
||||
if not td_fields and klass in _TD_KEYS:
|
||||
td_fields = _TD_KEYS[klass]
|
||||
|
||||
import dataclasses
|
||||
|
||||
order: list[str] = []
|
||||
plain: list[str] = []
|
||||
wrapped: list[tuple[str, Callable[[Any], Any]]] = []
|
||||
td_name: str | None = None
|
||||
|
||||
def add(name: str, getter: Callable[[Any], Any] | None = None) -> None:
|
||||
nonlocal td_name
|
||||
order.append(name)
|
||||
if name == _TD_COL:
|
||||
td_name = name # 交易日列走缓存,不看 getter
|
||||
elif getter is None or name in _PLAIN_EXTRAS:
|
||||
plain.append(name)
|
||||
else:
|
||||
wrapped.append((name, getter))
|
||||
|
||||
inserted = False
|
||||
for f in dataclasses.fields(cls): # type: ignore[arg-type]
|
||||
if f.name in drops:
|
||||
if not inserted:
|
||||
for name, getter in extras:
|
||||
add(name, getter)
|
||||
inserted = True
|
||||
continue
|
||||
add(f.name, _json_getter(f.name) if f.name in json_fields else None)
|
||||
if not inserted:
|
||||
# 没有丢弃字段的模型(如 SecurityInfo):增强列直接追加到末尾
|
||||
for name, getter in extras:
|
||||
add(name, getter)
|
||||
|
||||
plan = _ColPlan(
|
||||
order=tuple(order),
|
||||
plain=tuple(plain),
|
||||
wrapped=tuple(wrapped),
|
||||
td_name=td_name,
|
||||
td_key=attrgetter(*td_fields) if td_fields else None,
|
||||
)
|
||||
_PLAN_CACHE[cls] = plan
|
||||
return plan
|
||||
|
||||
|
||||
def _trading_dates(key_of: Callable[[Any], Any], rows: Sequence[Any]) -> list[str]:
|
||||
"""按缓存键批量算交易日串(同一批里绝大多数行是重复键)。"""
|
||||
cache: dict[Any, str] = {}
|
||||
out: list[str] = []
|
||||
ap = out.append
|
||||
for row in rows:
|
||||
key = key_of(row)
|
||||
value = cache.get(key)
|
||||
if value is None: # trading_date 永远是字符串,不会与 None 混淆
|
||||
value = cache[key] = row.trading_date
|
||||
ap(value)
|
||||
return out
|
||||
|
||||
|
||||
def _columns(cls: type, rows: Sequence[Any]) -> dict[str, list]:
|
||||
"""列式采集:逐列一次遍历,比逐行 ``asdict()`` 快得多。
|
||||
|
||||
- 普通列用 ``list(map(attrgetter(name), rows))``:取值发生在 C 层,没有 Python 帧;
|
||||
- ``trading_date`` 按时间字段缓存(240k 行常只有 1~2 个键);
|
||||
- 其余列(JSON / 五档 / 除权日)才走逐行回调;
|
||||
- ``IntEnum`` 归一化放在最后**按列**做一次,而不是每个单元格都判一次。
|
||||
"""
|
||||
plan = _plan_for(cls)
|
||||
if not rows:
|
||||
return {name: [] for name in plan.order}
|
||||
|
||||
cols: dict[str, list] = {}
|
||||
for name in plan.plain:
|
||||
cols[name] = list(map(attrgetter(name), rows))
|
||||
if plan.td_name is not None and plan.td_key is not None:
|
||||
cols[plan.td_name] = _trading_dates(plan.td_key, rows)
|
||||
for name, get in plan.wrapped:
|
||||
cols[name] = [get(row) for row in rows]
|
||||
|
||||
# polars 对 IntEnum 的类型推断不可靠(会退化成 Object 列),统一转纯 int。
|
||||
# 只在首元素就是 IntEnum 时才整列转,普通 int 列一次都不多花。
|
||||
for name, col in cols.items():
|
||||
if col and isinstance(col[0], IntEnum):
|
||||
cols[name] = [_as_int(value) for value in col]
|
||||
|
||||
return {name: cols[name] for name in plan.order}
|
||||
|
||||
|
||||
def _columns_of_dicts(rows: Sequence[dict]) -> dict[str, list]:
|
||||
"""``list[dict]``(板块排行这类字段不固定的返回)按首次出现顺序并集列名。"""
|
||||
keys: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for row in rows:
|
||||
for key in row:
|
||||
if key not in seen:
|
||||
seen.add(key)
|
||||
keys.append(key)
|
||||
return {key: [row.get(key) for row in rows] for key in keys}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 容器分发
|
||||
|
||||
|
||||
def _model_by_name(name: str | None) -> type | None:
|
||||
if not name:
|
||||
return None
|
||||
obj = getattr(_models, name, None)
|
||||
return obj if isinstance(obj, type) else None
|
||||
|
||||
|
||||
def _resolve_model(data: Any, model: str | None) -> type | None:
|
||||
"""推断元素类型:非空列表看首元素;空列表只能靠调用方给的类型提示。"""
|
||||
if isinstance(data, list) and data:
|
||||
return type(data[0])
|
||||
return _model_by_name(model)
|
||||
|
||||
|
||||
def _frames_of_rows(be: _Backend, cls: type | None, rows: Sequence[Any]) -> Any:
|
||||
if cls is not None and rows and hasattr(cls, "__dataclass_fields__"):
|
||||
return be.frame(_columns(cls, rows))
|
||||
if rows and isinstance(rows[0], dict):
|
||||
return be.frame(_columns_of_dicts(rows))
|
||||
return be.empty(())
|
||||
|
||||
|
||||
def _from_indicator_sets(be: _Backend, items: Sequence[IndicatorSet]) -> Any:
|
||||
frames = []
|
||||
for item in items:
|
||||
cols = _columns(SecurityBar, item.bars)
|
||||
size = len(item.bars)
|
||||
if item.symbol:
|
||||
cols["symbol"] = [item.symbol] * size
|
||||
for name, series in item.indicators.items():
|
||||
values = list(series[:size])
|
||||
if len(values) < size:
|
||||
values.extend([None] * (size - len(values)))
|
||||
cols[name] = values
|
||||
frames.append(be.frame(cols))
|
||||
merged = be.concat(frames)
|
||||
return merged if merged is not None else be.frame(_columns(SecurityBar, []))
|
||||
|
||||
|
||||
def _from_index_trade_stats(be: _Backend, items: Sequence[IndexTradeStats]) -> Any:
|
||||
frames = []
|
||||
for stat in items:
|
||||
cols = _columns(MinuteTradeStat, stat.minutes)
|
||||
size = len(stat.minutes)
|
||||
# 元信息补成常量列:一张表就能看出是哪个指数、哪一天
|
||||
cols = {"index": [stat.index] * size, "date": [stat.date] * size, **cols}
|
||||
frames.append(be.frame(cols))
|
||||
merged = be.concat(frames)
|
||||
return merged if merged is not None else be.frame(_columns(MinuteTradeStat, []))
|
||||
|
||||
|
||||
def to_frame(data: Any, *, backend: str = "polars", model: str | None = None) -> Any:
|
||||
"""把模型返回值转成 DataFrame。
|
||||
|
||||
:param data: ``list[dataclass]`` / ``list[dict]`` / ``dict[str, list[dataclass]]``
|
||||
/ ``list[IndicatorSet]`` / ``IndexTradeStats`` / ``list[IndexTradeStats]``。
|
||||
:param backend: ``"polars"`` 或 ``"pandas"``。
|
||||
:param model: 元素类型的名字(如 ``"SecurityBar"``);只在 ``data`` 为空、
|
||||
无法从元素推断类型时使用,由 :func:`framed` 从返回注解里自动填。
|
||||
"""
|
||||
be = get_backend(backend)
|
||||
|
||||
if isinstance(data, IndexTradeStats):
|
||||
return _from_index_trade_stats(be, [data])
|
||||
if isinstance(data, IndicatorSet):
|
||||
return _from_indicator_sets(be, [data])
|
||||
|
||||
if isinstance(data, dict):
|
||||
# 批量接口:{标的: [行]} → 拼成一张表(symbol/code 列本来就在行里)
|
||||
frames = []
|
||||
for rows in data.values():
|
||||
if isinstance(rows, list) and rows:
|
||||
frames.append(_frames_of_rows(be, type(rows[0]), rows))
|
||||
merged = be.concat(frames)
|
||||
if merged is not None:
|
||||
return merged
|
||||
cls = _model_by_name(model)
|
||||
return be.empty(_plan_for(cls).order if cls else ())
|
||||
|
||||
if isinstance(data, list):
|
||||
if data and isinstance(data[0], IndexTradeStats):
|
||||
return _from_index_trade_stats(be, data)
|
||||
if data and isinstance(data[0], IndicatorSet):
|
||||
return _from_indicator_sets(be, data)
|
||||
cls = _resolve_model(data, model)
|
||||
if data and isinstance(data[0], dict):
|
||||
return _frames_of_rows(be, None, data)
|
||||
if cls is not None and data and hasattr(cls, "__dataclass_fields__"):
|
||||
return _frames_of_rows(be, cls, data)
|
||||
if data:
|
||||
# 元素既不是 dataclass 也不是 dict(如 list[float])——不该走到这里
|
||||
raise TypeError(f"无法转换为 DataFrame 的列表元素:{type(data[0])!r}")
|
||||
return (be.frame({name: [] for name in _plan_for(cls).order})
|
||||
if cls else be.empty(()))
|
||||
|
||||
raise TypeError(f"无法转换为 DataFrame 的返回值类型:{type(data)!r}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 接线
|
||||
|
||||
|
||||
_RETURN_MODEL = re.compile(r"list\[(\w+)\]")
|
||||
|
||||
|
||||
def framed(method: Callable[..., Any]) -> Callable[..., Any]:
|
||||
"""给 ``TdxData`` 的表格型方法套一层「按需转 DataFrame」。
|
||||
|
||||
- ``self._df_backend is None``(默认)时零开销,原样返回 ``list[dataclass]``。
|
||||
- 处于某次公开方法执行过程中(:data:`_FRAME_DEPTH` > 0)时不转换,
|
||||
保证批量/别名方法内部拿到的仍是原始 ``list``。
|
||||
"""
|
||||
model: str | None = None
|
||||
annotation = getattr(method, "__annotations__", {}).get("return")
|
||||
if isinstance(annotation, str):
|
||||
matched = _RETURN_MODEL.search(annotation)
|
||||
if matched:
|
||||
model = matched.group(1)
|
||||
|
||||
@functools.wraps(method)
|
||||
async def wrapper(self: Any, *args: Any, **kwargs: Any) -> Any:
|
||||
backend = getattr(self, "_df_backend", None)
|
||||
if backend is None or _FRAME_DEPTH.get() > 0:
|
||||
return await method(self, *args, **kwargs)
|
||||
|
||||
token = _FRAME_DEPTH.set(1)
|
||||
try:
|
||||
result = await method(self, *args, **kwargs)
|
||||
finally:
|
||||
_FRAME_DEPTH.reset(token)
|
||||
return to_frame(result, backend=backend, model=model)
|
||||
|
||||
return wrapper
|
||||
@@ -103,6 +103,10 @@ _US_EX_MARKETS: frozenset[int] = frozenset({int(ExMarket.US_STOCK)})
|
||||
_US_DAY_SPLIT_HOUR = 12
|
||||
|
||||
|
||||
# SecurityBar 上「后加的」可选字段:旧磁盘缓存反序列化时要补默认值(见 __setstate__)
|
||||
_BAR_LATE_FIELDS: tuple[str, ...] = ("settle",)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SecurityBar:
|
||||
"""K线记录(价格单位:元)。跨市场统一模型(A股/港股/美股/期货共用)。
|
||||
@@ -110,6 +114,10 @@ class SecurityBar:
|
||||
时间戳口径**统一为北京时间**:A股/港股/期货原始即为北京时间;美股原始是
|
||||
「日期=美东交易日 + 时分=北京时间」的混合值,构造时已还原为真实北京时间
|
||||
(见 ``__post_init__``)。该 bar 归属的交易日见 :attr:`trading_date`。
|
||||
|
||||
``amount`` / ``settle`` 的可取性**按通道**:A股标准/MAC 通道两者都有;
|
||||
EX 通道(0x23FF)一个 32B 记录里只有**一个** 4B 槽位可复用,权益类市场用它
|
||||
装成交额、期货期权类用它装结算价,两者不可能同时拿到 —— 详见 :attr:`amount`。
|
||||
"""
|
||||
market: int
|
||||
code: str
|
||||
@@ -117,14 +125,39 @@ class SecurityBar:
|
||||
high: float
|
||||
low: float
|
||||
close: float
|
||||
vol: float # 股票=股 / 转债=手(1 手=10 张,与 vipdoc 一致)
|
||||
amount: float # 元。⚠️ amount/vol 只在股票上等于均价;转债还要再 /10(手=10张)
|
||||
# vol 单位按品种:A股/ETF/美股=股;转债=手(1手=10张);⚠️港股主板/基金=手(1手=100股)
|
||||
vol: float
|
||||
# 元。⚠️ amount/vol 只在 A股上等于均价;转债还要再 /10(手=10张)、港股再 /100(手=100股)。
|
||||
# ⚠️ EX 通道的期货/期权(中金所、上期所、郑商所、大商所及各期权)协议帧内**没有成交额**,
|
||||
# 恒为 0.0 —— 要成交额请改用 EX 报价(``get_quotes``)或其它数据源。
|
||||
amount: float
|
||||
year: int
|
||||
month: int
|
||||
day: int
|
||||
hour: int = 0
|
||||
minute: int = 0
|
||||
symbol: str = "" # 规范化标的串(sz000001 / hk00700),由 API 层填充
|
||||
# 结算价。仅 EX 通道的**期货/期权**有(日线才有值,分钟线为 None);其余通道恒 None。
|
||||
settle: float | None = None
|
||||
# 持仓量(手)。仅 EX 通道的**期货/期权**有(0x23FF 帧槽位 5);其余通道恒 None。
|
||||
# 权益类市场(港股/美股等)无此概念,归一化为 None。
|
||||
position: float | None = None
|
||||
|
||||
def __setstate__(self, state: Any) -> None:
|
||||
"""反序列化兜底:早期版本写下的磁盘缓存里没有后加的可选字段(如 ``settle``)。
|
||||
|
||||
``@dataclass(slots=True)`` 的默认反序列化只把 pickle 里存在的槽位塞回去,
|
||||
缺键的槽位**不会**被赋默认值 —— 直接读 ``bar.settle`` 会抛 ``AttributeError``。
|
||||
缓存键里没有 schema 版本号,所以这里必须容错,否则升级后读旧缓存即崩。
|
||||
"""
|
||||
if isinstance(state, tuple) and len(state) == 2: # slots 形式的 (dict, slots)
|
||||
data = dict(state[1] or {})
|
||||
else:
|
||||
data = dict(state or {})
|
||||
for name in _BAR_LATE_FIELDS:
|
||||
data.setdefault(name, None)
|
||||
for key, value in data.items():
|
||||
object.__setattr__(self, key, value)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
"""美股(EX 通道)时间戳归一化:还原成真实北京时间。
|
||||
@@ -186,6 +219,10 @@ class SecurityQuote:
|
||||
A股(标准通道):五档 ``bid``/``ask`` + 买卖量 + 涨跌停价齐全,``name`` 为空。
|
||||
A股(``fields=[...]`` 走 MAC 0x122B):``name`` 与 ``fields`` 齐全,五档为空。
|
||||
扩展市场(港股/美股/期货):``name`` + 基础价量在属性上,其余字段位在 ``fields``。
|
||||
|
||||
``server_time`` 是**行情快照时间**:``HH:MM:SS.ff``,小数位数等于该品种价格小数位
|
||||
(股票 2 / ETF·LOF·B股 3 / 国债·回购 4),通常比本地时钟慢 5~15 秒。**指数恒为空串**
|
||||
(协议里这个字段对指数不是时钟);判断行情是否更新看它有没有变,别看调用时刻。
|
||||
"""
|
||||
market: int
|
||||
code: str
|
||||
|
||||
@@ -37,6 +37,9 @@ class ExPoolTimeoutError(ConnectionError):
|
||||
class ExPool:
|
||||
"""EX通道连接池(港股/期货,端口7727,连接时自动login)。"""
|
||||
|
||||
FAIL_COOLDOWN = 30.0
|
||||
"""握手失败后拉黑该主机的秒数(EX 节点少,冷却比标准池短)。"""
|
||||
|
||||
def __init__(self, servers: list[str], *, timeout: float = 10.0,
|
||||
min_size: int = 2, max_size: int = 6,
|
||||
per_server_cap: int = 1,
|
||||
@@ -58,6 +61,9 @@ class ExPool:
|
||||
self._total = 0
|
||||
self._closed = False
|
||||
self._cond = asyncio.Condition()
|
||||
# 死节点短期拉黑:一台 EX 主机握手失败后,后续 acquire 直接绕开它,
|
||||
# 而不是每次都再付一遍 ``self.timeout`` 的握手超时。
|
||||
self._fail_until: dict[str, float] = {}
|
||||
|
||||
async def start(self) -> None:
|
||||
async with self._cond:
|
||||
@@ -72,6 +78,7 @@ class ExPool:
|
||||
self._server_count[host] = self._server_count.get(host, 0) + 1
|
||||
except Exception as e:
|
||||
log.warning("EX连接失败 %s: %s", host, str(e)[:60])
|
||||
self._fail_until[host] = time.monotonic() + self.FAIL_COOLDOWN
|
||||
|
||||
async def close(self) -> None:
|
||||
self._closed = True
|
||||
@@ -94,8 +101,11 @@ class ExPool:
|
||||
if timeout is None:
|
||||
timeout = self.acquire_timeout
|
||||
deadline = None if timeout is None else time.monotonic() + timeout
|
||||
async with self._cond:
|
||||
while True:
|
||||
avoid: set[str] = set() # 本次 acquire 里已经握手失败、别再试的主机
|
||||
while True:
|
||||
new_conn = None
|
||||
new_host = None
|
||||
async with self._cond:
|
||||
# 1. 优先借用空闲连接(过期则丢弃,槽位在锁内回收)
|
||||
if self._idle:
|
||||
_, conn = self._idle.pop()
|
||||
@@ -107,44 +117,65 @@ class ExPool:
|
||||
self._server_count[conn.host] = max(0, self._server_count.get(conn.host, 0) - 1)
|
||||
self._schedule_close(conn)
|
||||
|
||||
# 2. 可扩容 → 锁内预留槽位,锁外建连
|
||||
new_conn = None
|
||||
new_host = None
|
||||
# 2. 可扩容 → 锁内预留槽位,锁外建连(不能持锁做网络 IO)
|
||||
if self._total < self.max_size and self.servers:
|
||||
host = self._next_available_server()
|
||||
host = self._next_available_server(avoid)
|
||||
if host is not None:
|
||||
self._total += 1
|
||||
self._server_count[host] = self._server_count.get(host, 0) + 1
|
||||
new_conn = ExTdxConnection(host, 7727, self.timeout, self.handshake_map.get(host, self.handshake))
|
||||
new_conn = ExTdxConnection(
|
||||
host, 7727, self.timeout,
|
||||
self.handshake_map.get(host, self.handshake))
|
||||
new_host = host
|
||||
if new_conn is not None:
|
||||
# 锁外网络 IO:连接失败回滚槽位后继续等待
|
||||
elif avoid and all(h in avoid for h in self.servers):
|
||||
# 所有主机本次都试过了、全失败:等下去也不会有结果
|
||||
raise ExPoolTimeoutError(self, timeout)
|
||||
|
||||
# 3. 没抢到槽位 → 在锁内排队等 release 唤醒(waiter 必须持锁)
|
||||
if new_conn is None:
|
||||
now = time.monotonic()
|
||||
if deadline is not None and now >= deadline:
|
||||
raise ExPoolTimeoutError(self, timeout)
|
||||
try:
|
||||
await new_conn.connect()
|
||||
return new_conn
|
||||
except Exception as e:
|
||||
log.warning("EX扩容失败 %s: %s", new_host, str(e)[:60])
|
||||
async with self._cond:
|
||||
self._total = max(0, self._total - 1)
|
||||
self._server_count[new_host] = max(0, self._server_count.get(new_host, 0) - 1)
|
||||
if deadline is None:
|
||||
await self._cond.wait()
|
||||
else:
|
||||
await asyncio.wait_for(self._cond.wait(), timeout=deadline - now)
|
||||
except asyncio.TimeoutError:
|
||||
raise ExPoolTimeoutError(self, timeout) from None
|
||||
continue
|
||||
|
||||
# 3. 池满:排队等待 release 唤醒(或超时)
|
||||
now = time.monotonic()
|
||||
if deadline is not None and now >= deadline:
|
||||
raise ExPoolTimeoutError(self, timeout)
|
||||
try:
|
||||
if deadline is None:
|
||||
await self._cond.wait()
|
||||
else:
|
||||
await asyncio.wait_for(self._cond.wait(), timeout=deadline - now)
|
||||
except asyncio.TimeoutError:
|
||||
raise ExPoolTimeoutError(self, timeout)
|
||||
# ---- 锁外建连 ----
|
||||
try:
|
||||
await new_conn.connect()
|
||||
async with self._cond:
|
||||
self._fail_until.pop(new_host, None)
|
||||
return new_conn
|
||||
except Exception as e:
|
||||
log.warning("EX扩容失败 %s: %s", new_host, str(e)[:60])
|
||||
avoid.add(new_host) # type: ignore[arg-type]
|
||||
async with self._cond:
|
||||
self._total = max(0, self._total - 1)
|
||||
self._server_count[new_host] = max(0, self._server_count.get(new_host, 0) - 1)
|
||||
self._fail_until[new_host] = time.monotonic() + self.FAIL_COOLDOWN
|
||||
self._cond.notify_all()
|
||||
continue
|
||||
|
||||
def _next_available_server(self) -> str | None:
|
||||
def _next_available_server(self, avoid: set[str] | None = None) -> str | None:
|
||||
"""挑一台**未达并发上限**的主机:优先未冷却的,冷却中的只在别无选择时用。"""
|
||||
avoid = avoid or set()
|
||||
fallback: str | None = None
|
||||
now = time.monotonic()
|
||||
for h in self.servers:
|
||||
if self._server_count.get(h, 0) < self.per_server_cap:
|
||||
if h in avoid:
|
||||
continue
|
||||
if self._server_count.get(h, 0) >= self.per_server_cap:
|
||||
continue
|
||||
if now >= self._fail_until.get(h, 0.0):
|
||||
return h
|
||||
return None
|
||||
if fallback is None:
|
||||
fallback = h
|
||||
return fallback
|
||||
|
||||
def _schedule_close(self, conn: ExTdxConnection) -> None:
|
||||
async def _close():
|
||||
|
||||
+127
-43
@@ -75,6 +75,15 @@ class PooledConnection:
|
||||
class DynamicPool:
|
||||
"""动态连接池:按服务器维度统计连接数,每台服务器最多 per_server_cap 个连接。"""
|
||||
|
||||
BLACKOUT_COOLDOWN = 2.0
|
||||
"""全主机失败后的熔断时长(秒)。
|
||||
|
||||
全部服务器同时不可用时,每次 :meth:`acquire` 都会把每台主机握手一遍再放弃。
|
||||
上层是「按标的逐只重试」的循环,6000 只标的 × 16 台主机 × 握手耗时 = 几分钟
|
||||
的空转。熔断期内直接抛 :class:`PoolTimeoutError`,让调用方快速走完并返回,
|
||||
冷却结束再试一次探测全灭是否恢复。
|
||||
"""
|
||||
|
||||
def __init__(self, servers: list[str], *, port: int = DEFAULT_PORT,
|
||||
timeout: float = 10.0, min_size: int = 4, max_size: int = 12,
|
||||
per_server_cap: int = 1,
|
||||
@@ -104,6 +113,7 @@ class DynamicPool:
|
||||
self._total = 0
|
||||
self._closed = False
|
||||
self._started = False
|
||||
self._blackout_until = 0.0 # 全灭熔断截止时刻(monotonic)
|
||||
self._lock = asyncio.Lock()
|
||||
self._cond = asyncio.Condition(self._lock)
|
||||
self._reaper_task: asyncio.Task | None = None
|
||||
@@ -127,10 +137,12 @@ class DynamicPool:
|
||||
break
|
||||
try:
|
||||
pc = await self._new_conn(host)
|
||||
self._idle.append(pc)
|
||||
self._total += 1
|
||||
except Exception as e:
|
||||
log.warning("初始连接失败 %s: %s", host, str(e)[:60])
|
||||
continue
|
||||
async with self._lock:
|
||||
self._register_locked(pc)
|
||||
self._idle.append(pc)
|
||||
|
||||
if self._total == 0:
|
||||
log.warning("连接池初始连接全部失败,将在使用时按需重试")
|
||||
@@ -160,6 +172,12 @@ class DynamicPool:
|
||||
return out
|
||||
|
||||
async def _new_conn(self, host: str) -> PooledConnection:
|
||||
"""建立一条连接(**纯 IO,不改任何计数**)。
|
||||
|
||||
计数由调用方在锁内用 :meth:`_register_locked` / :meth:`_reserve_locked`
|
||||
单独完成——建连必须能在池锁之外进行,否则一台坏服务器的握手超时
|
||||
会把所有并发协程一起冻住。
|
||||
"""
|
||||
conn = TdxConnection(host, self.port, self.timeout)
|
||||
await conn.connect()
|
||||
if self.probe_factory is not None:
|
||||
@@ -168,12 +186,28 @@ class DynamicPool:
|
||||
except Exception:
|
||||
await conn.close()
|
||||
raise
|
||||
pc = PooledConnection(conn, host)
|
||||
return PooledConnection(conn, host)
|
||||
|
||||
def _register_locked(self, pc: PooledConnection) -> None:
|
||||
"""登记一条已建好的连接(调用方需持锁)。"""
|
||||
self._conns.add(pc)
|
||||
self._server_count[host] = self._server_count.get(host, 0) + 1
|
||||
self._server_count[pc.host] = self._server_count.get(pc.host, 0) + 1
|
||||
self._total += 1
|
||||
self.stats.total = self._total
|
||||
self.stats.created += 1
|
||||
self.stats.total = self._total + 1
|
||||
return pc
|
||||
|
||||
def _reserve_locked(self, host: str) -> None:
|
||||
"""锁内**预留**一个槽位(真正建连在锁外做)。"""
|
||||
self._total += 1
|
||||
self._server_count[host] = self._server_count.get(host, 0) + 1
|
||||
self.stats.total = self._total
|
||||
|
||||
def _unreserve_locked(self, host: str) -> None:
|
||||
"""撤销预留(建连失败时回滚计数)。"""
|
||||
self._total = max(0, self._total - 1)
|
||||
if self._server_count.get(host, 0) > 0:
|
||||
self._server_count[host] -= 1
|
||||
self.stats.total = self._total
|
||||
|
||||
async def close(self) -> None:
|
||||
self._closed = True
|
||||
@@ -208,10 +242,19 @@ class DynamicPool:
|
||||
单连接约束下,池大小 = 服务器数 × per_server_cap。当并发请求超过池大小时,
|
||||
超额协程在此**排队等待**(基于 Condition),待其他协程 release 后被唤醒,
|
||||
而非立即超时放弃。默认 timeout=30s 防止在坏服务器场景下永久挂起。
|
||||
|
||||
**建连在池锁之外**:锁内只预留槽位,TCP 握手 + 探针放到锁外做。否则
|
||||
一台坏服务器握手超时(最长 ``self.timeout``)期间,所有其它协程的
|
||||
acquire/release 全部被这个锁挡住——5 台坏节点能把整批任务拖慢十几秒。
|
||||
"""
|
||||
deadline = None if timeout is None else (time.monotonic() + timeout)
|
||||
async with self._cond:
|
||||
while True:
|
||||
avoid: set[str] = set() # 本次 acquire 里已握手失败的主机,别再重复试
|
||||
while True:
|
||||
if time.monotonic() < self._blackout_until:
|
||||
raise PoolTimeoutError(
|
||||
"所有服务器均不可用(熔断中,稍后自动重试)")
|
||||
host = ""
|
||||
async with self._cond:
|
||||
now = time.monotonic()
|
||||
# 1. 优先借用空闲连接(跳过冷却中的服务器,避免反复选中坏节点)
|
||||
while self._idle:
|
||||
@@ -222,35 +265,65 @@ class DynamicPool:
|
||||
return pc
|
||||
self._drop_locked(pc)
|
||||
|
||||
# 2. 可扩容(未达上限且有可用服务器)→ 新建连接
|
||||
if self._total < self.max_size:
|
||||
host = self._next_available_server()
|
||||
if host is None:
|
||||
# 兜底:全部服务器都在冷却中,而池子未满 —— 此时等 release
|
||||
# 永远等不到(没人持有),只能挑最健康的一台顶着冷却重试
|
||||
host = self._next_available_server(ignore_cooldown=True)
|
||||
if host:
|
||||
try:
|
||||
pc = await self._new_conn(host)
|
||||
self._total += 1
|
||||
self.stats.borrowed += 1
|
||||
return pc
|
||||
except Exception as e:
|
||||
log.warning("扩容连接失败 %s: %s", host, str(e)[:60])
|
||||
self.health.record_failure(host)
|
||||
# 循环重试:可能换下一台服务器
|
||||
continue
|
||||
|
||||
# 3. 池满:排队等待 release 唤醒(或超时)
|
||||
# 2. 全坏场景下这里必须能退出:扩容失败分支会 continue,
|
||||
# 没有这一行就会无限重试、永远不会抛 PoolTimeoutError。
|
||||
if deadline is not None and now >= deadline:
|
||||
raise PoolTimeoutError("连接池获取超时(连接数已达上限且无空闲)")
|
||||
try:
|
||||
if deadline is None:
|
||||
await self._cond.wait()
|
||||
else:
|
||||
await asyncio.wait_for(self._cond.wait(), timeout=deadline - now)
|
||||
except asyncio.TimeoutError:
|
||||
raise PoolTimeoutError("连接池获取超时(连接数已达上限且无空闲)")
|
||||
|
||||
# 3. 可扩容 → 锁内**只预留槽位**
|
||||
if self._total < self.max_size:
|
||||
host = self._next_available_server(avoid=avoid) or ""
|
||||
if not host:
|
||||
# 兜底:全部服务器都在冷却中,而池子未满 —— 此时等 release
|
||||
# 永远等不到(没人持有),只能挑最健康的一台顶着冷却重试
|
||||
host = self._next_available_server(
|
||||
ignore_cooldown=True, avoid=avoid) or ""
|
||||
if host:
|
||||
self._reserve_locked(host)
|
||||
elif avoid and all(h in avoid for h in self.servers):
|
||||
# 本轮每台主机都握手失败过了 → 再等下去也不会有连接。
|
||||
# 同时熔断一小段时间,避免上层「逐只重试」把握手放大成
|
||||
# 成千上万次(实测 6000 标的能空转 60s+)。
|
||||
self._blackout_until = now + self.BLACKOUT_COOLDOWN
|
||||
raise PoolTimeoutError(
|
||||
f"所有服务器均不可用({len(avoid)} 台均已失败)")
|
||||
|
||||
if not host:
|
||||
# 4. 池满/无可用服务器:排队等 release 唤醒(或超时)
|
||||
remaining = None if deadline is None else deadline - now
|
||||
try:
|
||||
if remaining is None:
|
||||
await self._cond.wait()
|
||||
else:
|
||||
await asyncio.wait_for(self._cond.wait(), timeout=remaining)
|
||||
except asyncio.TimeoutError:
|
||||
raise PoolTimeoutError(
|
||||
"连接池获取超时(连接数已达上限且无空闲)") from None
|
||||
continue
|
||||
|
||||
# ---- 锁外建连:失败只影响本次请求,不冻结其它协程 ----
|
||||
try:
|
||||
pc = await self._new_conn(host)
|
||||
except Exception as e:
|
||||
avoid.add(host)
|
||||
async with self._cond:
|
||||
self._unreserve_locked(host)
|
||||
self._cond.notify_all() # 槽位已回滚,唤醒排队者
|
||||
self.health.record_failure(host)
|
||||
log.warning("扩容连接失败 %s: %s", host, str(e)[:60])
|
||||
continue
|
||||
if self._closed:
|
||||
await pc.close()
|
||||
async with self._cond:
|
||||
self._unreserve_locked(host)
|
||||
self._cond.notify_all()
|
||||
raise PoolTimeoutError("连接池已关闭")
|
||||
# 槽位已在锁内预留,这里只补登记,不重复计数
|
||||
self._conns.add(pc)
|
||||
self.stats.created += 1
|
||||
self.stats.borrowed += 1
|
||||
self._blackout_until = 0.0 # 有连接成功 → 立刻解除熔断
|
||||
return pc
|
||||
|
||||
async def acquire_many(self, k: int) -> list[PooledConnection]:
|
||||
"""批量借出 k 个连接(分页并发用),超额请求排队等待不丢。"""
|
||||
@@ -262,8 +335,11 @@ class DynamicPool:
|
||||
break
|
||||
return out
|
||||
|
||||
def _next_available_server(self, *, ignore_cooldown: bool = False) -> str | None:
|
||||
ranked = self.health.rank(self.servers)
|
||||
def _next_available_server(self, *, ignore_cooldown: bool = False,
|
||||
avoid: set[str] | None = None) -> str | None:
|
||||
"""挑一台主机(``avoid`` 里的跳过,用于同一次 acquire 内不重复试错)。"""
|
||||
avoid = avoid or set()
|
||||
ranked = self.health.rank([h for h in self.servers if h not in avoid])
|
||||
# 1. 优先选未冷却且未达并发上限的服务器
|
||||
for h in ranked:
|
||||
if self.health.in_cooldown(h) and not ignore_cooldown:
|
||||
@@ -311,12 +387,20 @@ class DynamicPool:
|
||||
results = await asyncio.gather(
|
||||
*(self._new_conn(h) for h in hosts), return_exceptions=True)
|
||||
async with self._lock:
|
||||
for pc in results:
|
||||
if isinstance(pc, PooledConnection):
|
||||
self._total += 1
|
||||
self.stats.total = self._total
|
||||
if not self._closed and pc.alive:
|
||||
self._idle.append(pc)
|
||||
for host, res in zip(hosts, results):
|
||||
if not isinstance(res, PooledConnection):
|
||||
# 建连失败 → 记一次失败,后续 acquire 会把它排到队尾
|
||||
self.health.record_failure(host)
|
||||
continue
|
||||
pc: PooledConnection = res
|
||||
if self._closed or not pc.alive or self._total >= self.max_size:
|
||||
# 建好却发现池已满/已关闭:必须关掉,否则连接泄漏,
|
||||
# 而且 _total 会被虚增(老实现就在这儿漏计数)
|
||||
asyncio.create_task(self._safe_close(pc))
|
||||
continue
|
||||
self._register_locked(pc)
|
||||
self._idle.append(pc)
|
||||
self._cond.notify_all()
|
||||
|
||||
async def release(self, pc: PooledConnection, *, broken: bool = False) -> None:
|
||||
if self.stats.borrowed > 0:
|
||||
|
||||
@@ -4,9 +4,24 @@ from __future__ import annotations
|
||||
import struct
|
||||
|
||||
from ..._binary import unpack_from
|
||||
from ...models import ExMarket
|
||||
from ..datetime_ import get_datetime
|
||||
from .base import BaseCommand
|
||||
|
||||
# 0x23FF 的「槽位 5 是成交额」的市场集合(其余市场该槽位是持仓量、槽位 7 是结算价)。
|
||||
# 实测依据(2026-10-28,日线,逐条校验 成交额/(close×vol)):
|
||||
# 31 港股主板 = 100.0(vol 单位手,1手=100股)、49 港股基金 = 100.3、
|
||||
# 48 港股创业板 ≈1.0、44 新三板 ≈0.99、74 美股 ≈1.0(vol 单位股)。
|
||||
# ⚠️ 27 港股指数 / 62 中证指数 该槽位既非成交额也非结算价(量级对不上),故**不在**表内,
|
||||
# 保持 amount=0.0 的原始行为,避免给出可疑数值。
|
||||
_EX_AMOUNT_MARKETS: frozenset[int] = frozenset({
|
||||
int(ExMarket.HK_MAIN_BOARD),
|
||||
int(ExMarket.HK_GEM),
|
||||
int(ExMarket.HK_FUND),
|
||||
int(ExMarket.NEEQ),
|
||||
int(ExMarket.US_STOCK),
|
||||
})
|
||||
|
||||
# pytdx EX setup 握手命令(完整84B: 12B帧头+72B body, 0x2454)—— 与MAC EX login同ID不同body
|
||||
EX_SETUP_CMD = bytes.fromhex("0101486500015200520054241f32c6e5d53dfb411f32c6e5d53dfb411f32c6e5d53dfb411f32c6e5d53dfb411f32c6e5d53dfb411f32c6e5d53dfb411f32c6e5d53dfb411f32c6e5d53dfb41cce16dffd5ba3fb8cbc57a054f7748ea")
|
||||
|
||||
@@ -57,7 +72,24 @@ class GetMarketsCmd(BaseCommand[list[dict]]):
|
||||
|
||||
|
||||
class GetExInstrumentBarsCmd(BaseCommand[list[dict]]):
|
||||
"""EX商品K线(0x23FF, 32B/条: 4B时间戳+28B行情)。"""
|
||||
"""EX商品K线(0x23FF, 32B/条: 4B时间戳+28B行情)。
|
||||
|
||||
后 28B 的字段语义**按市场分两类**(2026-10-28 全市场实测,见 ``_EX_AMOUNT_MARKETS``):
|
||||
|
||||
* 权益类(港股主板/创业板/基金、新三板、美股)::
|
||||
|
||||
open(4) high(4) low(4) close(4) 成交额(4f) 成交量(4I) 保留(4f)
|
||||
|
||||
成交额可用;校验 ``成交额 / (close × 成交量)`` = 100.0(港股主板/基金,
|
||||
``vol`` 单位是「手」=100 股)或 ≈1.0(港股创业板/新三板/美股,``vol`` 单位是股)。
|
||||
|
||||
* 期货期权类(中金所/上期所/郑商所/大商所及各期权)::
|
||||
|
||||
open(4) high(4) low(4) close(4) 持仓量(4I) 成交量(4I) 结算价(4f)
|
||||
|
||||
**协议帧内没有成交额**(``amount`` 恒 0.0,不是解析漏读);末位 float 是结算价,
|
||||
日线有值、分钟线为 0(``settle`` 归一化成 ``None``)。
|
||||
"""
|
||||
|
||||
def __init__(self, market: int, code: str, category: int = 0,
|
||||
start: int = 0, count: int = 700) -> None:
|
||||
@@ -76,6 +108,7 @@ class GetExInstrumentBarsCmd(BaseCommand[list[dict]]):
|
||||
if len(body) < 20:
|
||||
return []
|
||||
(ret_count,) = unpack_from("<H", body, 18, "ex bars count")
|
||||
amount_market = self.market in _EX_AMOUNT_MARKETS
|
||||
items: list[dict] = []
|
||||
off = 20
|
||||
for _ in range(ret_count):
|
||||
@@ -83,14 +116,23 @@ class GetExInstrumentBarsCmd(BaseCommand[list[dict]]):
|
||||
break
|
||||
# 时间4字节按category分格式:分钟级=2B zipday+2B tminutes;日线及以上=YYYYMMDD
|
||||
year, month, day, hour, minute, _ = get_datetime(self.category, body, off)
|
||||
open_, high, low, close, position, trade, _ = struct.unpack(
|
||||
"<ffffIIf", body[off + 4: off + 32])
|
||||
items.append({
|
||||
raw = body[off + 4: off + 32]
|
||||
open_, high, low, close = struct.unpack("<ffff", raw[:16])
|
||||
item = {
|
||||
"year": year, "month": month, "day": day,
|
||||
"hour": hour, "minute": minute,
|
||||
"open": open_, "high": high, "low": low, "close": close,
|
||||
"position": position, "trade": trade,
|
||||
})
|
||||
}
|
||||
if amount_market:
|
||||
# 权益类:槽位 5 = 成交额(float),槽位 7 语义未知(保留不用)
|
||||
amount, trade, _reserved = struct.unpack("<fIf", raw[16:28])
|
||||
item.update(amount=amount, trade=trade, position=0, settle=None)
|
||||
else:
|
||||
# 期货/期权:槽位 5 = 持仓量,槽位 7 = 结算价(分钟线为 0 → None)
|
||||
position, trade, settle = struct.unpack("<IIf", raw[16:28])
|
||||
item.update(amount=0.0, trade=trade, position=position,
|
||||
settle=settle or None)
|
||||
items.append(item)
|
||||
off += 32
|
||||
return items
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ import struct
|
||||
|
||||
from ..._binary import unpack_from
|
||||
from ...models import SecurityQuote
|
||||
from ...symbols import is_cn_index
|
||||
from ..price import get_price
|
||||
from ..volume import get_volume
|
||||
from .base import BaseCommand
|
||||
@@ -161,13 +162,49 @@ def price_decimal_digits(market: int, code: str) -> int:
|
||||
return _decimal_digits_static(m, code)
|
||||
|
||||
|
||||
def format_server_time(raw: int) -> str:
|
||||
"""服务器时间编码:小时 + 百万分之一小时。14999212 -> 14:59:57.163"""
|
||||
hours, fractional_hour = divmod(raw, 1_000_000)
|
||||
total_millis = fractional_hour * 3600 // 1000
|
||||
minutes, remainder = divmod(total_millis, 60_000)
|
||||
seconds, millis = divmod(remainder, 1000)
|
||||
return f"{hours:02d}:{minutes:02d}:{seconds:02d}.{millis:03d}"
|
||||
_SERVER_TIME_SKEW = 40
|
||||
"""服务端在「50~59 秒」段会写成「90~99」秒,整数值固定多 40×10**d。"""
|
||||
|
||||
|
||||
def format_server_time(raw: int, digits: int = 2) -> str:
|
||||
"""报价快照时间解码:十进制数字拼接 ``HH MM SS`` + ``digits`` 位小数。
|
||||
|
||||
编码(2026-10-30 盘中实测确定)::
|
||||
|
||||
整数 = HH*10**(4+d) + MM*10**(2+d) + SS*10**d + 小数
|
||||
d = 该品种**价格小数位**(股票 2 / ETF·LOF·B股 3 / 国债·回购 4)
|
||||
|
||||
实测样例:``13224002``(d=2)→ ``13:22:40.02``;``132245903``(d=3)→
|
||||
``13:22:45.903``;``1310228050``(d=4)→ ``13:10:22.8050``。
|
||||
|
||||
历史错误(0.8.0 修正):此前按「小时 + 百万分之一小时」解码 —— 用
|
||||
``divmod(raw, 1_000_000)`` 取小时、余数按每秒 3600/1e6 折算。股票会把
|
||||
11:29:59 解成 **11:17:57**(基准错了、速率也只有真实时钟的 53%),
|
||||
ETF / 债券更会解出 ``113:00:21.222`` 这种小时 > 23 的非法值。
|
||||
|
||||
⚠️ 服务端怪癖:**SS 落在 50~59 秒时写成 90~99**(编码值 +40 秒)。实测
|
||||
3 分钟连续采样,未修正时该区间的滞后会从稳定的 10.6 秒突然变成 -29.4 秒
|
||||
(正好差 40)。因此这里对 ``SS >= 60`` 一律减 40 秒还原。
|
||||
|
||||
⚠️ 指数(``sh000001`` / ``sz399001`` …)该字段**不是**这只时钟:解出来
|
||||
MM 常 > 59、且不同指数互相矛盾(同一时刻 000001 是 12:89、399001 是 12:15),
|
||||
协议里没有给出可用信息。这类返回值一律为空串,不要编造时间;要指数时间
|
||||
请用 :meth:`~pytdxdata.api.TdxData.get_minutes` 或 K 线。
|
||||
"""
|
||||
if raw <= 0:
|
||||
return ""
|
||||
d = digits if 0 <= digits <= 6 else 2
|
||||
scale = 10 ** d
|
||||
frac = raw % scale
|
||||
rest = raw // scale
|
||||
seconds = rest % 100
|
||||
minutes = (rest // 100) % 100
|
||||
hours = rest // 10000
|
||||
if seconds >= 60:
|
||||
seconds -= _SERVER_TIME_SKEW
|
||||
if hours > 23 or minutes > 59 or not 0 <= seconds <= 59:
|
||||
return ""
|
||||
return f"{hours:02d}:{minutes:02d}:{seconds:02d}.{frac:0{d}d}"
|
||||
|
||||
|
||||
class GetSecurityQuotesCmd(BaseCommand[list[SecurityQuote]]):
|
||||
@@ -235,13 +272,14 @@ class GetSecurityQuotesCmd(BaseCommand[list[SecurityQuote]]):
|
||||
pos += 4
|
||||
|
||||
code = code_b.decode("utf-8").rstrip("\x00")
|
||||
divisor = 10 ** price_decimal_digits(market_b, code)
|
||||
dp = price_decimal_digits(market_b, code)
|
||||
divisor = 10 ** dp
|
||||
p = price_raw / divisor
|
||||
results.append(SecurityQuote(
|
||||
market=market_b, code=code, price=p,
|
||||
cur_vol=float(cur_vol),
|
||||
rise_speed=_rise_speed / 100.0,
|
||||
decimal_point=price_decimal_digits(market_b, code),
|
||||
decimal_point=dp,
|
||||
open_amount=float(_unknown_3),
|
||||
pre_close=(price_raw + last_close_diff) / divisor,
|
||||
open=(price_raw + open_diff) / divisor,
|
||||
@@ -250,8 +288,8 @@ class GetSecurityQuotesCmd(BaseCommand[list[SecurityQuote]]):
|
||||
vol=float(vol), amount=amount, s_vol=float(s_vol), b_vol=float(b_vol),
|
||||
bid=[((price_raw + d) / divisor, v) for d, v in bid],
|
||||
ask=[((price_raw + d) / divisor, v) for d, v in ask],
|
||||
server_time=format_server_time(unknown_0),
|
||||
server_time="" if is_cn_index(market_b, code)
|
||||
else format_server_time(unknown_0, dp),
|
||||
trading_status=trading_status,
|
||||
))
|
||||
return results
|
||||
return results
|
||||
@@ -10,6 +10,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections import deque
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Callable
|
||||
|
||||
@@ -81,15 +82,53 @@ class Paginator:
|
||||
stop = asyncio.Event()
|
||||
out: list[PageResult] = []
|
||||
out_lock = asyncio.Lock()
|
||||
# 静态切片保证「每台连接都干活」(并发摊开),另备一个共享的
|
||||
# ``leftover``:某个 worker 的连接坏掉退出时,它**没做完的页**会
|
||||
# 放回这里,由还活着的 worker 接着做。老实现没有这层——一个
|
||||
# worker 挂掉,它那一整份页就没人做了,要么整轮报错、要么静默缺页。
|
||||
leftover: deque[tuple[int, int]] = deque()
|
||||
q_lock = asyncio.Lock()
|
||||
k = len(slots)
|
||||
workers = [
|
||||
asyncio.create_task(
|
||||
self._worker(slots[wid], jobs[wid::k], cmd_factory,
|
||||
stop, out, out_lock, spec)
|
||||
self._worker(slots[wid], jobs[wid::k], leftover, q_lock,
|
||||
cmd_factory, stop, out, out_lock, spec)
|
||||
)
|
||||
for wid in range(k)
|
||||
]
|
||||
await asyncio.gather(*workers)
|
||||
# 收尾补页:所有 worker 都退出后 leftover 里可能还剩页(连接全坏时会
|
||||
# 发生)。再借一条**新连接**顺序把它们补完,最多两轮——否则 skip 的页
|
||||
# 就变成静默缺数据(strict=False 时最危险:不报错、只是少几页)。
|
||||
rounds = 0
|
||||
while leftover and not stop.is_set() and rounds < 2:
|
||||
rounds += 1
|
||||
try:
|
||||
pc = await self.pool.acquire(timeout=5.0)
|
||||
except PoolTimeoutError:
|
||||
break
|
||||
slot = _Slot(pc)
|
||||
progressed = False
|
||||
try:
|
||||
while leftover and not stop.is_set():
|
||||
start, count = leftover[0] # 成功后才出队,失败可重试
|
||||
items, ok = await self._fetch_one(
|
||||
slot, cmd_factory(start, count), count)
|
||||
if not ok:
|
||||
break
|
||||
leftover.popleft()
|
||||
progressed = True
|
||||
async with out_lock:
|
||||
out.append(PageResult(start=start, count=count, items=items))
|
||||
if spec.stop_when_short and len(items) < count:
|
||||
stop.set()
|
||||
finally:
|
||||
if slot.pc is not None:
|
||||
await self.pool.release(slot.pc)
|
||||
if not progressed:
|
||||
break
|
||||
if leftover:
|
||||
log.warning("分页调度:%d 页最终未取到(strict=False 留空)", len(leftover))
|
||||
finally:
|
||||
for s in slots:
|
||||
if s.pc is not None:
|
||||
@@ -117,21 +156,40 @@ class Paginator:
|
||||
return jobs
|
||||
|
||||
async def _worker(
|
||||
self, slot: _Slot, jobs: list[tuple[int, int]],
|
||||
self, slot: _Slot,
|
||||
jobs: list[tuple[int, int]],
|
||||
leftover: "deque[tuple[int, int]]",
|
||||
q_lock: asyncio.Lock,
|
||||
cmd_factory: Callable[[int, int], Any], stop: asyncio.Event,
|
||||
out: list[PageResult], out_lock: asyncio.Lock, spec: PageSpec,
|
||||
) -> None:
|
||||
for start, count in jobs:
|
||||
if stop.is_set() or slot.pc is None:
|
||||
return
|
||||
items, ok = await self._fetch_one(slot, cmd_factory(start, count), count)
|
||||
if not ok:
|
||||
return # 换连接失败 → 本 worker 退出(该页留空)
|
||||
async with out_lock:
|
||||
out.append(PageResult(start=start, count=count, items=items))
|
||||
if spec.stop_when_short and len(items) < count:
|
||||
stop.set()
|
||||
return
|
||||
i = 0
|
||||
try:
|
||||
while True:
|
||||
if i < len(jobs):
|
||||
start, count = jobs[i]
|
||||
i += 1
|
||||
else:
|
||||
# 自己的份额做完了 → 看看有没有别人掉下的活
|
||||
async with q_lock:
|
||||
if stop.is_set() or not leftover:
|
||||
return
|
||||
start, count = leftover.popleft()
|
||||
if slot.pc is None or stop.is_set():
|
||||
return
|
||||
items, ok = await self._fetch_one(slot, cmd_factory(start, count), count)
|
||||
if not ok:
|
||||
return # 换连接失败 → 本 worker 退出(剩余页在 finally 里交还)
|
||||
async with out_lock:
|
||||
out.append(PageResult(start=start, count=count, items=items))
|
||||
if spec.stop_when_short and len(items) < count:
|
||||
stop.set()
|
||||
return
|
||||
finally:
|
||||
rest = jobs[i:]
|
||||
if rest:
|
||||
async with q_lock:
|
||||
leftover.extendleft(reversed(rest))
|
||||
|
||||
async def _fetch_one(self, slot: _Slot, cmd, page_size: int):
|
||||
"""单页拉取,失败换连接重试。
|
||||
|
||||
+19
-10
@@ -1,8 +1,13 @@
|
||||
"""测试全局隔离。
|
||||
|
||||
默认关掉**权威小数位表的后台同步** —— 否则任何调用 ``TdxData.start()`` 的用例都会
|
||||
顺手发起证券清单请求(26 + 25 个分页),测试会变慢且依赖网络。
|
||||
需要验证同步行为本身的用例(``test_decimal_sync.py``)自己 ``delenv`` 后再测。
|
||||
默认关掉两个**后台网络行为**:
|
||||
- 权威小数位表同步:否则任何调用 ``TdxData.start()`` 的用例都会顺手发起证券
|
||||
清单请求(26 + 25 个分页),测试变慢且依赖网络。
|
||||
- 当日服务器测速:后台 task 会真的去连主机取K线,会污染「往返次数」这类
|
||||
断言(实测同一个用例的 ``calls`` 在 3/4 之间随机跳),也会拖慢整体。
|
||||
|
||||
需要验证这两个行为本身的用例(``test_decimal_sync.py`` / ``test_probe_cache.py``)
|
||||
自己 ``delenv`` 后再测。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -10,13 +15,17 @@ import os
|
||||
|
||||
import pytest
|
||||
|
||||
_ENV_KEYS = ("PYPYTDX_DATA_DISABLE_DECIMAL_SYNC", "PYPYTDX_DATA_DISABLE_PROBE")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True, scope="session")
|
||||
def _disable_decimal_sync():
|
||||
old = os.environ.get("PYPYTDX_DATA_DISABLE_DECIMAL_SYNC")
|
||||
os.environ["PYPYTDX_DATA_DISABLE_DECIMAL_SYNC"] = "1"
|
||||
def _disable_background_network():
|
||||
old = {k: os.environ.get(k) for k in _ENV_KEYS}
|
||||
for k in _ENV_KEYS:
|
||||
os.environ[k] = "1"
|
||||
yield
|
||||
if old is None:
|
||||
os.environ.pop("PYPYTDX_DATA_DISABLE_DECIMAL_SYNC", None)
|
||||
else:
|
||||
os.environ["PYPYTDX_DATA_DISABLE_DECIMAL_SYNC"] = old
|
||||
for k, v in old.items():
|
||||
if v is None:
|
||||
os.environ.pop(k, None)
|
||||
else:
|
||||
os.environ[k] = v
|
||||
@@ -0,0 +1,234 @@
|
||||
"""K线批量流水线路径(多连接 × 单连接流水线)。
|
||||
|
||||
背景:单只K线一请求一标的,全市场 5,000 只就是 5,000 次往返。串行时每只固定付
|
||||
一个 RTT(实测 33ms),实测 24 台 × 8 流水能把全市场 1 分钟K线取当天从 43.6s
|
||||
压到 5s 级。这里用替身验证「路径选择」与「结果口径」,不依赖网络。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
|
||||
import pytest
|
||||
|
||||
from pytdxdata import TdxData
|
||||
from pytdxdata.api import _auto_pipeline, _bars_for_range, _elapsed_trading_minutes
|
||||
from pytdxdata.models import KlinePeriod, SecurityBar
|
||||
|
||||
# 替身的时间锚点必须跟着**真实 now** 走:API 内部用 ``datetime.now()`` 判断「是不是今天」,
|
||||
# 用固定日期会在 CI(UTC 时区,与本地差 8 小时)上漂到别的时段分支,断言时绿时红。
|
||||
NEWEST = datetime.datetime.now().replace(microsecond=0) # noqa: DTZ001
|
||||
|
||||
|
||||
def _bar(when: datetime.datetime, close: float = 10.0) -> SecurityBar:
|
||||
return SecurityBar(
|
||||
market=1, code="600000", open=10.0, high=10.1, low=9.9, close=close,
|
||||
vol=1000, amount=10000, year=when.year, month=when.month, day=when.day,
|
||||
hour=when.hour, minute=when.minute)
|
||||
|
||||
|
||||
class _MacFakeConn:
|
||||
"""MAC 通道语义替身:**offset 0 = 最新一根**,偏移越大越老。"""
|
||||
|
||||
seen: list[tuple[str, int, int, bool]] = [] # (命令名, start, count, 是否流水线)
|
||||
fail_pipeline = False
|
||||
|
||||
def __init__(self, host="m1", port=7709, timeout=10.0):
|
||||
self.host = host
|
||||
self._alive = True
|
||||
|
||||
@property
|
||||
def alive(self) -> bool:
|
||||
return self._alive
|
||||
|
||||
async def connect(self) -> None:
|
||||
pass
|
||||
|
||||
async def close(self) -> None:
|
||||
self._alive = False
|
||||
|
||||
def _rows(self, cmd, *, record: bool = True):
|
||||
if record:
|
||||
type(self).seen.append(
|
||||
(type(cmd).__name__, getattr(cmd, "start", 0),
|
||||
getattr(cmd, "count", 0), False))
|
||||
if "MacKlineCmd" not in type(cmd).__name__:
|
||||
return []
|
||||
out = []
|
||||
for off in range(cmd.start, cmd.start + cmd.count):
|
||||
if off > 20_000:
|
||||
continue
|
||||
when = NEWEST - datetime.timedelta(minutes=off)
|
||||
out.append(_bar(when, 10.0 + (20_000 - off) * 0.001))
|
||||
return out
|
||||
|
||||
async def execute(self, cmd):
|
||||
return self._rows(cmd)
|
||||
|
||||
async def execute_pipeline(self, cmds, **kwargs):
|
||||
if type(self).fail_pipeline:
|
||||
raise ConnectionError("流水线炸了")
|
||||
for c in cmds:
|
||||
type(self).seen.append(
|
||||
(type(c).__name__, getattr(c, "start", 0), getattr(c, "count", 0), True))
|
||||
return [self._rows(c, record=False) for c in cmds]
|
||||
|
||||
|
||||
def _syms(n: int) -> list[str]:
|
||||
"""n 只沪市股票代码(互不相同,避免按 symbol 归并时互相覆盖)。"""
|
||||
return [f"sh60{i:04d}" for i in range(n)]
|
||||
|
||||
|
||||
async def _run(n: int, period=KlinePeriod.MIN_1, **kw):
|
||||
_MacFakeConn.seen = []
|
||||
_MacFakeConn.fail_pipeline = False
|
||||
with pytest.MonkeyPatch.context() as mp:
|
||||
mp.setattr("pytdxdata.api.get_mac_hosts", lambda: ["m1", "m2"])
|
||||
mp.setattr("pytdxdata.pool.pool.TdxConnection",
|
||||
lambda h, p=7709, t=10.0: _MacFakeConn(h))
|
||||
td = TdxData(cache_dir=None, pool_min=1, pool_max=4)
|
||||
await td.start()
|
||||
try:
|
||||
bars = await td.get_bars(_syms(n), period, **kw)
|
||||
finally:
|
||||
await td.close()
|
||||
return bars
|
||||
|
||||
|
||||
def _data_calls() -> list[tuple[str, int, int, bool]]:
|
||||
"""只看真正的数据请求(排除建连探针:count=100 那条)。"""
|
||||
return [c for c in _MacFakeConn.seen if c[2] != 100]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bulk_path_uses_pipeline_for_many_symbols():
|
||||
bars = await _run(20, count=240)
|
||||
|
||||
data = _data_calls()
|
||||
assert data, "应该有数据请求"
|
||||
assert all(via_pipe for *_, via_pipe in data), "20 只必须走流水线"
|
||||
assert len(data) == 20
|
||||
assert len(bars) == 20 * 240
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_small_batch_falls_back_to_single():
|
||||
"""低于阈值走逐只:不经过流水线,结果不变。"""
|
||||
bars = await _run(5, count=240)
|
||||
|
||||
data = _data_calls()
|
||||
assert data and not any(via_pipe for *_, via_pipe in data)
|
||||
assert len(bars) == 5 * 240
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pipeline_1_disables_bulk():
|
||||
bars = await _run(20, count=240, pipeline=1)
|
||||
|
||||
data = _data_calls()
|
||||
assert data and not any(via_pipe for *_, via_pipe in data)
|
||||
assert len(bars) == 20 * 240
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_result_order_follows_input_order():
|
||||
"""返回按入参标的顺序拼接(组内升序)。"""
|
||||
bars = await _run(20, count=30)
|
||||
|
||||
seen: list[str] = []
|
||||
for b in bars:
|
||||
if not seen or seen[-1] != b.symbol:
|
||||
seen.append(b.symbol)
|
||||
assert seen == _syms(20)
|
||||
# 组内按时间正序
|
||||
head = [b.datetime for b in bars if b.symbol == _syms(20)[0]]
|
||||
assert head == sorted(head)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_long_count_is_paged_inside_pipeline():
|
||||
"""count 超过单请求上限时按页切分,且页也走流水线(不是逐只串行翻页)。"""
|
||||
bars = await _run(20, count=1600, pipeline=8)
|
||||
|
||||
data = _data_calls()
|
||||
assert all(via_pipe for *_, via_pipe in data)
|
||||
assert len(data) == 20 * 3 # 700 + 700 + 200
|
||||
assert {c[2] for c in data} == {700, 200}
|
||||
assert len(bars) == 20 * 1600
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_pipeline_failure_falls_back_to_single_per_symbol():
|
||||
"""流水线失败 → 该批交回逐只重试,结果不丢。"""
|
||||
_MacFakeConn.seen = []
|
||||
_MacFakeConn.fail_pipeline = True
|
||||
with pytest.MonkeyPatch.context() as mp:
|
||||
mp.setattr("pytdxdata.api.get_mac_hosts", lambda: ["m1", "m2"])
|
||||
mp.setattr("pytdxdata.pool.pool.TdxConnection",
|
||||
lambda h, p=7709, t=10.0: _MacFakeConn(h))
|
||||
td = TdxData(cache_dir=None, pool_min=1, pool_max=4)
|
||||
await td.start()
|
||||
try:
|
||||
bars = await td.get_bars(_syms(20), KlinePeriod.MIN_1, count=240)
|
||||
finally:
|
||||
await td.close()
|
||||
assert len(bars) == 20 * 240
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dated_range_bulk_trims_to_start_date():
|
||||
"""日期区间走批量时,区间左端仍然精确裁剪。"""
|
||||
start = NEWEST - datetime.timedelta(minutes=30)
|
||||
bars = await _run(20, start_date=start)
|
||||
|
||||
assert bars
|
||||
assert all(b.datetime >= start for b in bars)
|
||||
assert all(b.symbol in set(_syms(20)) for b in bars)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dated_future_range_returns_empty_without_requests():
|
||||
future = NEWEST + datetime.timedelta(days=5)
|
||||
bars = await _run(20, start_date=future)
|
||||
|
||||
assert bars == []
|
||||
|
||||
|
||||
# ---------------- 纯函数 ----------------
|
||||
def test_elapsed_trading_minutes():
|
||||
def at(h, m):
|
||||
return datetime.datetime(2026, 10, 28, h, m) # noqa: DTZ001
|
||||
|
||||
assert _elapsed_trading_minutes(at(9, 0)) == 0
|
||||
assert _elapsed_trading_minutes(at(9, 30)) == 0
|
||||
assert _elapsed_trading_minutes(at(10, 0)) == 30
|
||||
assert _elapsed_trading_minutes(at(11, 45)) == 120 # 午休
|
||||
assert _elapsed_trading_minutes(at(13, 30)) == 150
|
||||
assert _elapsed_trading_minutes(at(15, 0)) == 240
|
||||
assert _elapsed_trading_minutes(at(20, 0)) == 240 # 收盘后
|
||||
|
||||
|
||||
def test_bars_for_range_shrinks_intraday():
|
||||
"""盘中只需已走完的根数,不是一整天 240 根。"""
|
||||
morning = datetime.datetime(2026, 10, 28, 10, 0) # noqa: DTZ001
|
||||
midnight = datetime.datetime(2026, 10, 28, 0, 0) # noqa: DTZ001
|
||||
# 10:00 走了 30 分钟 → 30 + 余量,远小于 240
|
||||
assert 30 <= _bars_for_range(KlinePeriod.MIN_1, midnight, morning) <= 40
|
||||
# 收盘后回到一整天的量级
|
||||
close = datetime.datetime(2026, 10, 28, 16, 0) # noqa: DTZ001
|
||||
assert _bars_for_range(KlinePeriod.MIN_1, midnight, close) >= 245
|
||||
|
||||
|
||||
def test_bars_for_range_past_days():
|
||||
now = datetime.datetime(2026, 10, 28, 16, 0) # noqa: DTZ001
|
||||
start = now - datetime.timedelta(days=5)
|
||||
n = _bars_for_range(KlinePeriod.DAY, start, now)
|
||||
assert 5 <= n <= 12 # 5 个自然日 × 1.6 余量
|
||||
|
||||
|
||||
def test_auto_pipeline():
|
||||
# 每根 32B:小 count 用深流水,大 count 用浅流水(在途字节数恒定)
|
||||
assert _auto_pipeline(2, None) == 32
|
||||
assert _auto_pipeline(240, None) == 8
|
||||
assert _auto_pipeline(240, 4) == 4 # 显式指定优先
|
||||
assert _auto_pipeline(2, 1) == 2 # 夹到最小值 2
|
||||
@@ -0,0 +1,220 @@
|
||||
"""EX 0x23FF K线的字段语义:``amount`` / ``settle`` 按市场分别回填。
|
||||
|
||||
背景(2026-10-28 实测):0x23FF 每条 32B = 4B 时间戳 + **28B 行情**,这 28B 只有
|
||||
**一个** 4B 槽位可复用,语义随市场变化 ——
|
||||
|
||||
* 权益类(港股主板/创业板/基金、新三板、美股):该槽位是**成交额**,
|
||||
校验 ``成交额 / (close × vol)`` ≈ 100(港股主板/基金,vol 单位是手)或 ≈ 1(其余,vol 是股);
|
||||
* 期货期权类:该槽位是**结算价**,成交额在帧内根本不存在(恒 0.0)。
|
||||
|
||||
改之前 ``amount`` 被硬编码成 0.0、末位 float 直接丢弃,于是港股/美股拿不到成交额、
|
||||
期货期权拿不到结算价。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import pickle
|
||||
import struct
|
||||
|
||||
import pytest
|
||||
|
||||
from pytdxdata.api import TdxData
|
||||
from pytdxdata.models import SecurityBar
|
||||
from pytdxdata.protocol.commands.ex_proto import (
|
||||
_EX_AMOUNT_MARKETS,
|
||||
GetExInstrumentBarsCmd,
|
||||
)
|
||||
from pytdxdata.symbols import parse_symbols
|
||||
|
||||
DAY = 20261028
|
||||
|
||||
|
||||
def _f2u(value: float) -> int:
|
||||
"""float32 -> 其位模式的 uint32(用来拼「同一槽位两种解释」的报文)。"""
|
||||
return struct.unpack("<I", struct.pack("<f", value))[0]
|
||||
|
||||
|
||||
def _ex_body(rows: list[tuple[int, ...]], category: int = 4) -> bytes:
|
||||
"""拼 0x23FF 响应体:20B 头(count@18)+ N × 32B(时间4B + 7×4B)。"""
|
||||
body = bytearray(20 + 32 * len(rows))
|
||||
struct.pack_into("<H", body, 18, len(rows))
|
||||
for i, words in enumerate(rows):
|
||||
off = 20 + i * 32
|
||||
struct.pack_into("<I", body, off, DAY)
|
||||
struct.pack_into("<7I", body, off + 4, *words)
|
||||
return bytes(body)
|
||||
|
||||
|
||||
def _equity_words(close: float, amount: float, vol: int) -> tuple[int, ...]:
|
||||
"""权益类布局:OHLC + 成交额(float) + 成交量(uint) + 保留(float)。"""
|
||||
return (_f2u(close), _f2u(close), _f2u(close), _f2u(close),
|
||||
_f2u(amount), vol, _f2u(0.0))
|
||||
|
||||
|
||||
def _derivative_words(close: float, position: int, vol: int,
|
||||
settle: float) -> tuple[int, ...]:
|
||||
"""期货/期权布局:OHLC + 持仓量(uint) + 成交量(uint) + 结算价(float)。"""
|
||||
return (_f2u(close), _f2u(close), _f2u(close), _f2u(close),
|
||||
position, vol, _f2u(settle))
|
||||
|
||||
|
||||
# ---------- 解析层 ----------
|
||||
|
||||
@pytest.mark.parametrize("market", [31, 48, 49, 44, 74])
|
||||
def test_equity_markets_read_amount_from_fifth_slot(market: int) -> None:
|
||||
cmd = GetExInstrumentBarsCmd(market, "00700", 4, 0, 1)
|
||||
rows = cmd.parse_response(_ex_body([_equity_words(10.2, 2_040_000.0, 2000)]))
|
||||
|
||||
assert len(rows) == 1
|
||||
assert rows[0]["amount"] == pytest.approx(2_040_000.0)
|
||||
assert rows[0]["trade"] == 2000 # 成交量不受影响
|
||||
assert rows[0]["position"] == 0 # 权益类无持仓量概念
|
||||
assert rows[0]["settle"] is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("market", [7, 8, 9, 47])
|
||||
def test_derivative_markets_read_settle_from_seventh_slot(market: int) -> None:
|
||||
cmd = GetExInstrumentBarsCmd(market, "IFL0", 4, 0, 1)
|
||||
rows = cmd.parse_response(_ex_body([_derivative_words(4021.0, 49_772, 25_392, 4047.8)]))
|
||||
|
||||
assert rows[0]["amount"] == 0.0 # 帧内没有成交额,不是解析漏读
|
||||
assert rows[0]["position"] == 49_772
|
||||
assert rows[0]["trade"] == 25_392
|
||||
assert rows[0]["settle"] == pytest.approx(4047.8)
|
||||
|
||||
|
||||
def test_intraday_settle_zero_normalized_to_none() -> None:
|
||||
"""结算价是日终量:分钟线该槽位恒 0,归一化成 None 而不是 0.0。"""
|
||||
cmd = GetExInstrumentBarsCmd(47, "IFL0", 7, 0, 1)
|
||||
rows = cmd.parse_response(_ex_body([_derivative_words(4054.6, 49_490, 260, 0.0)], 7))
|
||||
|
||||
assert rows[0]["settle"] is None
|
||||
|
||||
|
||||
def test_index_markets_kept_out_of_amount_table() -> None:
|
||||
"""港股指数(27)/中证指数(62) 该槽位既非成交额也非结算价 —— 不能猜,保持 0.0。"""
|
||||
assert 27 not in _EX_AMOUNT_MARKETS
|
||||
assert 62 not in _EX_AMOUNT_MARKETS
|
||||
cmd = GetExInstrumentBarsCmd(27, "HSI", 4, 0, 1)
|
||||
rows = cmd.parse_response(_ex_body([_equity_words(23041.32, 178_348.0, 17_835)]))
|
||||
assert rows[0]["amount"] == 0.0
|
||||
|
||||
|
||||
# ---------- API 层透传 ----------
|
||||
|
||||
def _fake_api(rows_for: dict[int, list[dict]]) -> TdxData:
|
||||
td = TdxData(cache_dir=None, pool_min=1, pool_max=2)
|
||||
seen: list[tuple[int, str]] = []
|
||||
|
||||
async def fake(make_cmd, spec): # noqa: ANN001 — 与真实签名同形
|
||||
cmd = make_cmd(0, spec.page_size)
|
||||
seen.append((cmd.market, cmd.code))
|
||||
return rows_for[cmd.market]
|
||||
|
||||
td._fetch_paginated_ex = fake # type: ignore[method-assign]
|
||||
td._seen_ex = seen # type: ignore[attr-defined]
|
||||
return td
|
||||
|
||||
|
||||
def _row(market: int, code: str, **kw) -> dict:
|
||||
base = {"year": 2026, "month": 10, "day": 28, "hour": 15, "minute": 0,
|
||||
"open": 10.0, "high": 10.5, "low": 9.5, "close": 10.2,
|
||||
"amount": 0.0, "position": 0, "trade": 2000, "settle": None}
|
||||
base.update(kw)
|
||||
return base
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bars_ex_backfills_amount_for_hk() -> None:
|
||||
td = _fake_api({31: [_row(31, "00700", amount=2_040_000.0)]})
|
||||
bars = await td.get_bars("hk00700", count=1)
|
||||
|
||||
assert len(bars) == 1
|
||||
assert bars[0].amount == pytest.approx(2_040_000.0)
|
||||
assert bars[0].vol == 2000
|
||||
assert bars[0].settle is None
|
||||
assert bars[0].position is None # 权益类无持仓量
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bars_ex_backfills_settle_for_futures() -> None:
|
||||
td = _fake_api({47: [_row(47, "IFL0", position=49_772, trade=25_392, settle=4047.8)]})
|
||||
bars = await td.get_bars("cffex_futures:IFL0", count=1)
|
||||
|
||||
assert bars[0].settle == pytest.approx(4047.8)
|
||||
# v3 起期货按收盘价估算成交额:close 10.2 × vol 25392 × IF 乘数 200
|
||||
assert bars[0].amount == pytest.approx(10.2 * 25392 * 200)
|
||||
assert bars[0].position == 49_772.0
|
||||
|
||||
|
||||
# ---------- 旧磁盘缓存兼容 ----------
|
||||
|
||||
def test_legacy_pickle_without_settle_still_loads() -> None:
|
||||
"""升级前写下的缓存没有 settle 槽位;默认反序列化不会补默认值,读它必崩。"""
|
||||
bar = SecurityBar(market=1, code="600000", open=1.0, high=2.0, low=0.5,
|
||||
close=1.5, vol=100.0, amount=150.0,
|
||||
year=2026, month=10, day=28, settle=12.3)
|
||||
_, slots = bar.__getstate__()
|
||||
legacy = dict(slots)
|
||||
legacy.pop("settle")
|
||||
|
||||
revived = SecurityBar.__new__(SecurityBar)
|
||||
revived.__setstate__((None, legacy))
|
||||
|
||||
assert revived.settle is None
|
||||
assert revived.close == 1.5
|
||||
|
||||
|
||||
def test_new_pickle_roundtrip_keeps_settle() -> None:
|
||||
bar = SecurityBar(market=47, code="IFL0", open=4021.0, high=4060.0, low=4010.0,
|
||||
close=4054.6, vol=25_618.0, amount=0.0,
|
||||
year=2026, month=10, day=28, settle=4052.6)
|
||||
|
||||
assert pickle.loads(pickle.dumps(bar)).settle == pytest.approx(4052.6)
|
||||
|
||||
|
||||
def test_ex_symbols_route_to_ex_path() -> None:
|
||||
"""守住前提:上面对 31/47 的断言确实走的是 EX 通道。"""
|
||||
assert parse_symbols(["hk00700"])[0].is_ex
|
||||
assert parse_symbols(["cffex_futures:IFL0"])[0].is_ex
|
||||
|
||||
|
||||
# ---------- v3:期权持仓量 + 乘数表 ----------
|
||||
|
||||
def test_multiplier_known_products() -> None:
|
||||
from pytdxdata.ex_multiplier import find_multiplier
|
||||
assert find_multiplier("cffex", "IFL0") == 200
|
||||
assert find_multiplier("shf", "rb2611") == 10
|
||||
assert find_multiplier("shf", "au2611") == 100
|
||||
assert find_multiplier("dl", "i2501") == 100
|
||||
assert find_multiplier("zz", "AP0") == 10
|
||||
|
||||
|
||||
def test_multiplier_unknown_returns_none() -> None:
|
||||
from pytdxdata.ex_multiplier import find_multiplier
|
||||
assert find_multiplier("shf", "xyz999") is None
|
||||
assert find_multiplier("nosuch", "rb2611") is None
|
||||
|
||||
|
||||
def test_stock_option_multiplier() -> None:
|
||||
from pytdxdata.ex_multiplier import find_multiplier
|
||||
assert find_multiplier("sh_stock_option", "10011425") == 10_000
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bars_ex_option_position_backfilled() -> None:
|
||||
"""期权 K 线持仓量透传(用户主诉):协议槽位 5 已解析,v3 起到 SecurityBar。"""
|
||||
td = _fake_api({8: [_row(8, "10011425", position=123_456, trade=6373, settle=0.0887)]})
|
||||
bars = await td.get_bars("sh_stock_option:10011425", count=1)
|
||||
|
||||
assert bars[0].position == 123_456.0
|
||||
# 股票期权乘数 10000:close 10.2 × vol 6373 × 10000
|
||||
assert bars[0].amount == pytest.approx(10.2 * 6373 * 10_000)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bars_ex_unknown_multiplier_keeps_amount_zero() -> None:
|
||||
"""乘数表没有的品种:宁可不给 amount 也不编造。"""
|
||||
td = _fake_api({30: [_row(30, "ZZZ2601", position=10, trade=5, settle=1.0)]})
|
||||
bars = await td.get_bars("shf:ZZZ2601", count=1)
|
||||
assert bars[0].amount == 0.0
|
||||
assert bars[0].position == 10.0
|
||||
@@ -0,0 +1,329 @@
|
||||
"""坏节点隔离回归:**一台服务器中途死掉,不能拖慢整批任务**。
|
||||
|
||||
三件事必须成立(缺一个就会把「单机故障」放大成「整批卡死」):
|
||||
|
||||
1. **建连不能持池锁** —— 死节点握手要等满 ``timeout``,锁内做会把所有并发
|
||||
协程一起冻住。
|
||||
2. **全节点失联要能退出** —— 扩容失败分支 ``continue`` 会无限重试,
|
||||
没有截止检查就永远不抛 ``PoolTimeoutError``。
|
||||
3. **失败批次立刻回队 / 分页份额交给活着的 worker** —— 否则要么串行重试
|
||||
把长尾放大,要么那一整份数据没人做(静默缺页)。
|
||||
|
||||
替身全部在进程内,不连真实网络。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import datetime
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from pytdxdata import TdxData
|
||||
from pytdxdata.models import KlinePeriod, SecurityBar
|
||||
from pytdxdata.pool.pool import DynamicPool, PoolTimeoutError
|
||||
from pytdxdata.scheduler.paginator import PageSpec, Paginator
|
||||
|
||||
NEWEST = datetime.datetime.now().replace(microsecond=0) # noqa: DTZ001
|
||||
|
||||
|
||||
# ====================================================================== #
|
||||
# 一、建连不占池锁 + 全灭可退出
|
||||
# ====================================================================== #
|
||||
|
||||
class _HostConn:
|
||||
"""按 host 决定握手行为的替身连接。
|
||||
|
||||
``slow_hosts`` 里的主机握手会「卡住」``delay`` 秒后失败(模拟 SYN 被丢弃);
|
||||
``dead`` 为真时连握手都不做,立刻拒绝。
|
||||
"""
|
||||
|
||||
slow_hosts: dict[str, float] = {}
|
||||
dead: bool = False
|
||||
connects: list[str] = []
|
||||
|
||||
def __init__(self, host, port=7709, timeout=10.0):
|
||||
self.host = host
|
||||
self._alive = True
|
||||
type(self).connects.append(host)
|
||||
|
||||
@property
|
||||
def alive(self) -> bool:
|
||||
return self._alive
|
||||
|
||||
async def connect(self) -> None:
|
||||
delay = type(self).slow_hosts.get(self.host)
|
||||
if type(self).dead:
|
||||
raise ConnectionRefusedError(f"{self.host} 拒绝连接")
|
||||
if delay is not None:
|
||||
await asyncio.sleep(delay)
|
||||
raise ConnectionRefusedError(f"{self.host} 握手超时")
|
||||
|
||||
async def execute(self, cmd):
|
||||
return "ok"
|
||||
|
||||
async def close(self) -> None:
|
||||
self._alive = False
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def conns(monkeypatch):
|
||||
_HostConn.slow_hosts = {}
|
||||
_HostConn.dead = False
|
||||
_HostConn.connects = []
|
||||
monkeypatch.setattr("pytdxdata.pool.pool.TdxConnection", _HostConn)
|
||||
return _HostConn
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_slow_connect_does_not_freeze_other_borrowers(conns):
|
||||
"""一台主机握手卡 0.5s 时,另一个协程借连接不能跟着等。
|
||||
|
||||
老实现把 ``_new_conn`` 放在池锁里:所有 acquire/release 都被这 0.5s 挡住,
|
||||
几台坏节点叠加就是十几秒的全局停顿。
|
||||
"""
|
||||
conns.slow_hosts = {"dead": 0.5}
|
||||
pool = DynamicPool(["dead", "live"], min_size=0, max_size=2,
|
||||
per_server_cap=1, auto_probe=False)
|
||||
await pool.start()
|
||||
|
||||
slow = asyncio.create_task(pool.acquire()) # 抢到 "dead",开始卡握手
|
||||
await asyncio.sleep(0.05) # 确保它已进入 _new_conn
|
||||
|
||||
t0 = time.monotonic()
|
||||
fast_pc = await pool.acquire() # 此时应立刻拿到 "live"
|
||||
elapsed = time.monotonic() - t0
|
||||
assert elapsed < 0.25, f"被坏节点的握手阻塞了 {elapsed:.2f}s"
|
||||
assert fast_pc.host == "live"
|
||||
|
||||
slow_pc = await slow # 换到 "live" 后成功
|
||||
assert slow_pc.host == "live"
|
||||
for pc in (fast_pc, slow_pc):
|
||||
await pool.release(pc)
|
||||
await pool.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_acquire_gives_up_when_every_host_is_dead(conns):
|
||||
"""全节点失联时必须抛 PoolTimeoutError,而不是无限重试。"""
|
||||
conns.dead = True
|
||||
pool = DynamicPool(["d1", "d2"], min_size=0, max_size=4,
|
||||
per_server_cap=1, auto_probe=False)
|
||||
await pool.start()
|
||||
|
||||
t0 = time.monotonic()
|
||||
with pytest.raises(PoolTimeoutError):
|
||||
# 外层 wait_for 只是防止老实现挂死把测试卡住
|
||||
await asyncio.wait_for(pool.acquire(timeout=0.3), timeout=5.0)
|
||||
assert time.monotonic() - t0 < 2.0
|
||||
await pool.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_blackout_stops_retrying_every_host_per_call(conns):
|
||||
"""全灭后进入熔断期:后续 acquire 立刻失败,不再把每台主机握手一遍。
|
||||
|
||||
没有熔断时,上层「按标的逐只重试」的循环会把 16 台 × 6000 只放大成
|
||||
9 万次握手(实测 6000 只跑 60s 都跑不完)。
|
||||
"""
|
||||
conns.dead = True
|
||||
pool = DynamicPool(["d1", "d2", "d3"], min_size=0, max_size=8,
|
||||
per_server_cap=1, auto_probe=False)
|
||||
await pool.start()
|
||||
tried = len(conns.connects) # start() 的 min_size 预热(可能已试过 1 台)
|
||||
|
||||
with pytest.raises(PoolTimeoutError):
|
||||
await asyncio.wait_for(pool.acquire(timeout=1.0), timeout=5.0)
|
||||
round1 = len(conns.connects) - tried
|
||||
assert round1 == 3, f"首轮应恰好试完 3 台,实际 {round1}"
|
||||
tried = len(conns.connects)
|
||||
|
||||
# 熔断期内:不再产生任何建连尝试,且立即抛错
|
||||
t0 = time.monotonic()
|
||||
for _ in range(50):
|
||||
with pytest.raises(PoolTimeoutError):
|
||||
await pool.acquire()
|
||||
assert time.monotonic() - t0 < 0.2
|
||||
assert len(conns.connects) == tried, "熔断期内不该再握手"
|
||||
|
||||
# 熔断到期后恢复探测
|
||||
pool._blackout_until = 0.0
|
||||
with pytest.raises(PoolTimeoutError):
|
||||
await pool.acquire()
|
||||
assert len(conns.connects) > tried
|
||||
await pool.close()
|
||||
|
||||
|
||||
# ====================================================================== #
|
||||
# 二、K 线批量:死节点只损失在途那一批,且不重复打扰
|
||||
# ====================================================================== #
|
||||
|
||||
class _FleetConn:
|
||||
"""MAC 通道替身:``m1`` 在成功服务 ``m1_life`` 次后「宕机」。"""
|
||||
|
||||
m1_life = 2
|
||||
m1_calls = 0
|
||||
m2_calls = 0
|
||||
|
||||
def __init__(self, host="m1", port=7709, timeout=10.0):
|
||||
self.host = host
|
||||
self._alive = True
|
||||
|
||||
@property
|
||||
def alive(self) -> bool:
|
||||
return self._alive
|
||||
|
||||
async def connect(self) -> None:
|
||||
pass
|
||||
|
||||
async def close(self) -> None:
|
||||
self._alive = False
|
||||
|
||||
def _dead(self) -> bool:
|
||||
cls = type(self)
|
||||
if self.host != "m1":
|
||||
cls.m2_calls += 1
|
||||
return False
|
||||
cls.m1_calls += 1
|
||||
return cls.m1_calls > cls.m1_life
|
||||
|
||||
def _rows(self, cmd):
|
||||
if "MacKlineCmd" not in type(cmd).__name__:
|
||||
return []
|
||||
out = []
|
||||
for off in range(cmd.start, cmd.start + cmd.count):
|
||||
when = NEWEST - datetime.timedelta(minutes=off)
|
||||
out.append(SecurityBar(
|
||||
market=1, code="600000", open=10.0, high=10.1, low=9.9,
|
||||
close=10.0, vol=1000, amount=10000, year=when.year,
|
||||
month=when.month, day=when.day, hour=when.hour,
|
||||
minute=when.minute))
|
||||
return out
|
||||
|
||||
async def execute(self, cmd):
|
||||
if self._dead():
|
||||
raise ConnectionError("m1 已宕机")
|
||||
return self._rows(cmd)
|
||||
|
||||
async def execute_pipeline(self, cmds, **kwargs):
|
||||
if self._dead():
|
||||
raise ConnectionError("m1 已宕机")
|
||||
return [self._rows(c) for c in cmds]
|
||||
|
||||
|
||||
def _syms(n: int) -> list[str]:
|
||||
return [f"sh60{i:04d}" for i in range(n)]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_bulk_bars_recovers_from_midrun_host_death(monkeypatch):
|
||||
"""30 只K线,m1 服务 2 次后宕机:数据一根不少,且不反复打扰死节点。"""
|
||||
_FleetConn.m1_life = 2
|
||||
_FleetConn.m1_calls = 0
|
||||
_FleetConn.m2_calls = 0
|
||||
monkeypatch.setattr("pytdxdata.api.get_mac_hosts", lambda: ["m1", "m2"])
|
||||
monkeypatch.setattr("pytdxdata.pool.pool.TdxConnection",
|
||||
lambda h, p=7709, t=3.0: _FleetConn(h))
|
||||
|
||||
td = TdxData(cache_dir=None, pool_min=1, pool_max=4)
|
||||
await td.start()
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
bars = await td.get_bars(_syms(30), KlinePeriod.MIN_1, count=240)
|
||||
finally:
|
||||
elapsed = time.monotonic() - t0
|
||||
await td.close()
|
||||
|
||||
assert len(bars) == 30 * 240, "死节点不能造成缺数据"
|
||||
# 死节点只该被「在途那一批」碰到,之后靠冷却隔离,不该被反复重试
|
||||
assert _FleetConn.m1_calls <= 2 * 4, f"死节点被打了 {_FleetConn.m1_calls} 次"
|
||||
assert _FleetConn.m2_calls > 0
|
||||
assert elapsed < 5.0, f"单台服务器宕机把整批拖到 {elapsed:.2f}s"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_all_hosts_dead_fails_fast(monkeypatch):
|
||||
"""全节点失联:**快速报错**,不能挂死,也不能每只都付满一次 30s 池超时。
|
||||
|
||||
老实现里扩容失败分支 ``continue`` 且没有截止检查 → ``acquire`` 无限重试,
|
||||
十万次建连把事件循环饿死(协程连取消都投递不进去)。
|
||||
"""
|
||||
_FleetConn.m1_life = 0
|
||||
_FleetConn.m1_calls = 0
|
||||
_FleetConn.m2_calls = 0
|
||||
monkeypatch.setattr("pytdxdata.api.get_mac_hosts", lambda: ["m1"])
|
||||
monkeypatch.setattr("pytdxdata.pool.pool.TdxConnection",
|
||||
lambda h, p=7709, t=3.0: _FleetConn(h))
|
||||
|
||||
td = TdxData(cache_dir=None, pool_min=1, pool_max=2)
|
||||
await td.start()
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
with pytest.raises(PoolTimeoutError):
|
||||
await asyncio.wait_for(
|
||||
td.get_bars(_syms(10), KlinePeriod.MIN_1, count=240), timeout=15.0)
|
||||
finally:
|
||||
elapsed = time.monotonic() - t0
|
||||
await td.close()
|
||||
assert elapsed < 5.0, f"全灭场景花了 {elapsed:.2f}s 才报错"
|
||||
# 每台主机只该试一次,不该疯狂重连
|
||||
assert _FleetConn.m1_calls < 40, f"重复建连 {_FleetConn.m1_calls} 次"
|
||||
|
||||
|
||||
# ====================================================================== #
|
||||
# 三、分页器的份额再分配
|
||||
# ====================================================================== #
|
||||
|
||||
class _PageConn:
|
||||
def __init__(self, die_after: int | None = None, page_size: int = 100,
|
||||
total: int = 600):
|
||||
self.die_after = die_after
|
||||
self.served = 0
|
||||
self.page_size = page_size
|
||||
self.total = total
|
||||
|
||||
async def execute(self, cmd):
|
||||
start, count = cmd
|
||||
if self.die_after is not None and self.served >= self.die_after:
|
||||
raise ConnectionError("连接已断")
|
||||
self.served += 1
|
||||
return list(range(start, min(start + count, self.total)))
|
||||
|
||||
|
||||
class _PagePool:
|
||||
def __init__(self, conns):
|
||||
self.conns = conns
|
||||
self._i = 0
|
||||
self.released = []
|
||||
|
||||
async def acquire_many(self, n, *, timeout=None):
|
||||
return self.conns[:n]
|
||||
|
||||
async def acquire(self, *, timeout=None):
|
||||
pc = self.conns[self._i % len(self.conns)]
|
||||
self._i += 1
|
||||
return pc
|
||||
|
||||
async def release(self, pc, *, broken=False):
|
||||
self.released.append(broken)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_paginator_redistributes_pages_of_dead_worker():
|
||||
"""1 号连接拉到第 2 页就断:它剩下的页要由其它连接补完,不能整份丢掉。"""
|
||||
conns = [_PageConn(die_after=2), _PageConn(), _PageConn()]
|
||||
pag = Paginator(_PagePool(conns), concurrency=3) # type: ignore[arg-type]
|
||||
spec = PageSpec(page_size=100, total=600) # 6 页,每 worker 2 页
|
||||
got = await pag.fetch_all(lambda s, c: (s, c), spec)
|
||||
assert len(got) == 600, f"丢了 {600 - len(got)} 条"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_paginator_still_distributes_work_across_conns():
|
||||
"""份额再分配不能退化成「一条连接干完所有页」。"""
|
||||
conns = [_PageConn() for _ in range(3)]
|
||||
pag = Paginator(_PagePool(conns), concurrency=3) # type: ignore[arg-type]
|
||||
spec = PageSpec(page_size=100, total=600)
|
||||
got = await pag.fetch_all(lambda s, c: (s, c), spec)
|
||||
assert len(got) == 600
|
||||
assert all(c.served >= 1 for c in conns), "每台连接都该分到活"
|
||||
@@ -0,0 +1,383 @@
|
||||
"""DataFrame 转换层测试(0.7.0)。
|
||||
|
||||
覆盖两条独立路径:
|
||||
|
||||
1. **转换口径**:直接调 :func:`to_frame`,喂合成模型对象,断言列名/列序/取值/dtype。
|
||||
不碰网络,双后端各跑一遍。
|
||||
2. **接线**:``TdxData(df=...)`` 的取值校验、装饰器的「内层不二次转换」语义。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
from pytdxdata import TdxData
|
||||
from pytdxdata import frames as _frames
|
||||
from pytdxdata.frames import FrameBackendError, framed, get_backend, to_frame
|
||||
from pytdxdata.models import (
|
||||
ExMarket,
|
||||
IndexBar,
|
||||
IndexTradeStats,
|
||||
IndicatorSet,
|
||||
Market,
|
||||
MinuteBar,
|
||||
MinuteTradeStat,
|
||||
SecurityBar,
|
||||
SecurityQuote,
|
||||
TdxBlock,
|
||||
XdxrRecord,
|
||||
)
|
||||
|
||||
BACKENDS = ("polars", "pandas")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 工具
|
||||
|
||||
|
||||
def bar(**kw) -> SecurityBar:
|
||||
base = {
|
||||
"market": int(Market.SH), "code": "600000", "open": 10.0, "high": 11.0,
|
||||
"low": 9.5, "close": 10.5, "vol": 1000.0, "amount": 10500.0,
|
||||
"year": 2026, "month": 10, "day": 27, "hour": 0, "minute": 0,
|
||||
"symbol": "sh600000",
|
||||
}
|
||||
base.update(kw)
|
||||
return SecurityBar(**base)
|
||||
|
||||
|
||||
def rows(df, backend: str) -> int:
|
||||
return len(df) if backend == "pandas" else df.height
|
||||
|
||||
|
||||
def cols(df, backend: str) -> list[str]:
|
||||
return list(df.columns)
|
||||
|
||||
|
||||
def cell(df, backend: str, name: str, index: int = 0):
|
||||
if backend == "pandas":
|
||||
return df[name].iloc[index]
|
||||
# polars 2.x:Series[0] 会返回 Series 而不是标量,统一走 to_list()
|
||||
return df.get_column(name).to_list()[index]
|
||||
|
||||
|
||||
def missing(value) -> bool:
|
||||
"""pandas 的数值列会把 None 变成 NaN,两者都算「缺」。"""
|
||||
return value is None or (isinstance(value, float) and math.isnan(value))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 后端
|
||||
|
||||
|
||||
def test_backend_instances_are_cached():
|
||||
assert get_backend("polars") is get_backend("polars")
|
||||
assert get_backend("pandas") is get_backend("pandas")
|
||||
|
||||
|
||||
def test_unknown_backend_raises_value_error():
|
||||
with pytest.raises(ValueError, match="未知的 DataFrame 后端"):
|
||||
get_backend("numpy")
|
||||
|
||||
|
||||
def test_missing_backend_raises_with_install_hint(monkeypatch):
|
||||
monkeypatch.setitem(sys.modules, "polars", None) # 让 import polars 抛 ImportError
|
||||
monkeypatch.delitem(_frames._BACKEND_CACHE, "polars", raising=False)
|
||||
with pytest.raises(FrameBackendError) as excinfo:
|
||||
get_backend("polars")
|
||||
assert isinstance(excinfo.value, ImportError)
|
||||
assert excinfo.value.name == "polars"
|
||||
assert "pip install pytdxdata[polars]" in str(excinfo.value)
|
||||
monkeypatch.delitem(_frames._BACKEND_CACHE, "polars", raising=False)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 转换口径
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_bar_replaces_split_time_with_datetime(backend):
|
||||
df = to_frame([bar(), bar(minute=30)], backend=backend)
|
||||
assert cols(df, backend) == [
|
||||
"market", "code", "open", "high", "low", "close", "vol", "amount",
|
||||
"datetime", "trading_date", "symbol", "settle", "position",
|
||||
]
|
||||
assert rows(df, backend) == 2
|
||||
assert cell(df, backend, "market") == 1
|
||||
assert cell(df, backend, "trading_date") == "2026-10-27"
|
||||
stamp = cell(df, backend, "datetime", 1)
|
||||
assert stamp.year == 2026 and stamp.hour == 0 and stamp.minute == 30
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_int_enum_column_is_plain_int(backend):
|
||||
df = to_frame([bar(market=Market.SZ)], backend=backend)
|
||||
value = cell(df, backend, "market")
|
||||
assert value == 0
|
||||
assert not isinstance(value, Market)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_empty_list_falls_back_to_annotation_hint(backend):
|
||||
df = to_frame([], backend=backend, model="SecurityBar")
|
||||
assert rows(df, backend) == 0
|
||||
assert cols(df, backend) == [
|
||||
"market", "code", "open", "high", "low", "close", "vol", "amount",
|
||||
"datetime", "trading_date", "symbol", "settle", "position",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_quote_expands_bid_ask_levels(backend):
|
||||
quote = SecurityQuote(
|
||||
market=1, code="600000", price=10.5, pre_close=10.0, open=10.1, high=10.6,
|
||||
low=9.9, vol=100.0, amount=1000.0, s_vol=40.0, b_vol=60.0,
|
||||
bid=[(10.49, 100.0), (10.48, 200.0)], ask=[(10.51, 300.0)],
|
||||
symbol="sh600000", fields={"0x7": 1.5},
|
||||
)
|
||||
df = to_frame([quote], backend=backend)
|
||||
out = cols(df, backend)
|
||||
assert "bid1_price" in out and "ask5_vol" in out
|
||||
assert "bid" not in out and "ask" not in out
|
||||
assert cell(df, backend, "bid2_vol") == 200.0
|
||||
assert missing(cell(df, backend, "bid3_price")) # 只有两档,缺档给 None
|
||||
assert cell(df, backend, "fields") == '{"0x7":1.5}'
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_empty_dict_field_becomes_none(backend):
|
||||
df = to_frame([SecurityQuote(
|
||||
market=1, code="600000", price=1.0, pre_close=1.0, open=1.0, high=1.0,
|
||||
low=1.0, vol=1.0, amount=1.0, s_vol=0.0, b_vol=0.0,
|
||||
)], backend=backend)
|
||||
assert missing(cell(df, backend, "fields"))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_minute_uses_real_clock(backend):
|
||||
late = MinuteBar(price=1.0, vol=10.0, avg_price=1.0, hour=27, minute=59,
|
||||
symbol="usAAPL", market=74, code="AAPL", date=20261026)
|
||||
df = to_frame([late], backend=backend)
|
||||
assert cols(df, backend) == [
|
||||
"price", "vol", "avg_price", "datetime", "trading_date",
|
||||
"symbol", "market", "code", "fields",
|
||||
]
|
||||
stamp = cell(df, backend, "datetime")
|
||||
assert (stamp.year, stamp.month, stamp.day, stamp.hour, stamp.minute) == (
|
||||
2026, 10, 27, 3, 59,
|
||||
)
|
||||
assert cell(df, backend, "trading_date") == "2026-10-26"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_xdxr_date_column(backend):
|
||||
row = XdxrRecord(market=1, code="600000", year=2026, month=6, day=12,
|
||||
category=1, name="分红", fenhong=0.5, symbol="sh600000")
|
||||
df = to_frame([row], backend=backend)
|
||||
assert cols(df, backend)[:3] == ["market", "code", "date"]
|
||||
stamp = cell(df, backend, "date")
|
||||
assert (stamp.year, stamp.month, stamp.day) == (2026, 6, 12)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_tdx_block_keeps_codes_list(backend):
|
||||
df = to_frame([TdxBlock(name="银行", category=0, count=2, codes=["600000", "600036"])],
|
||||
backend=backend)
|
||||
assert cols(df, backend) == ["name", "category", "count", "codes"]
|
||||
assert cell(df, backend, "codes") == ["600000", "600036"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_indicator_set_merges_bars_and_indicators(backend):
|
||||
item = IndicatorSet(
|
||||
symbol="sh600000",
|
||||
bars=[bar(), bar(minute=1), bar(minute=2)],
|
||||
indicators={"ma5": [1.0, 2.0, 3.0], "boll": [None, None, 9.0]},
|
||||
)
|
||||
df = to_frame([item], backend=backend)
|
||||
assert rows(df, backend) == 3
|
||||
assert cell(df, backend, "symbol") == "sh600000"
|
||||
assert cell(df, backend, "ma5", 2) == 3.0
|
||||
assert missing(cell(df, backend, "boll"))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_indicator_series_shorter_than_bars_is_padded(backend):
|
||||
item = IndicatorSet(symbol="sh600000", bars=[bar(), bar(minute=1)],
|
||||
indicators={"ma5": [7.0]})
|
||||
df = to_frame([item], backend=backend)
|
||||
assert rows(df, backend) == 2
|
||||
assert cell(df, backend, "ma5") == 7.0
|
||||
assert missing(cell(df, backend, "ma5", 1))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_index_trade_stats_single_and_multi(backend):
|
||||
stat = IndexTradeStats(
|
||||
index="sh000300", date=20261027, member_count=300, active_members=299,
|
||||
trade_count_members=299,
|
||||
minutes=[MinuteTradeStat(time="09:31", volume=10.0, trade_count=5, amount=1000.0)],
|
||||
)
|
||||
single = to_frame(stat, backend=backend)
|
||||
assert cols(single, backend) == ["index", "date", "time", "volume", "trade_count", "amount"]
|
||||
assert cell(single, backend, "index") == "sh000300"
|
||||
|
||||
multi = to_frame([stat, stat], backend=backend)
|
||||
assert rows(multi, backend) == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_batch_dict_is_concatenated(backend):
|
||||
data = {
|
||||
"600000": [bar(code="600000"), bar(code="600000", minute=1)],
|
||||
"600036": [bar(code="600036", symbol="sh600036")],
|
||||
}
|
||||
df = to_frame(data, backend=backend)
|
||||
assert rows(df, backend) == 3
|
||||
assert cols(df, backend)[:2] == ["market", "code"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_batch_dict_all_empty_returns_typed_empty(backend):
|
||||
df = to_frame({"600000": [], "600036": []}, backend=backend, model="SecurityBar")
|
||||
assert rows(df, backend) == 0
|
||||
assert "datetime" in cols(df, backend)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_list_of_dicts_uses_key_union(backend):
|
||||
df = to_frame([{"name": "银行", "rise": 1.2}, {"name": "券商", "amount": 5.0}],
|
||||
backend=backend)
|
||||
assert cols(df, backend) == ["name", "rise", "amount"]
|
||||
assert missing(cell(df, backend, "amount", 0))
|
||||
|
||||
|
||||
def test_unsupported_payload_raises_type_error():
|
||||
with pytest.raises(TypeError, match="无法转换为 DataFrame 的返回值类型"):
|
||||
to_frame(42, backend="polars")
|
||||
with pytest.raises(TypeError, match="无法转换为 DataFrame 的列表元素"):
|
||||
to_frame([1.0, 2.0], backend="polars")
|
||||
|
||||
|
||||
def test_subclass_reuses_parent_plan():
|
||||
df = to_frame([IndexBar(market=1, code="000001", open=1.0, high=1.0, low=1.0,
|
||||
close=1.0, vol=1.0, amount=1.0, year=2026, month=1, day=5)],
|
||||
backend="polars")
|
||||
assert "datetime" in df.columns
|
||||
assert "year" not in df.columns
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 接线
|
||||
|
||||
|
||||
def test_tdxdata_default_keeps_legacy_behaviour():
|
||||
td = TdxData()
|
||||
assert td._df_backend is None
|
||||
|
||||
|
||||
def test_tdxdata_rejects_unknown_backend():
|
||||
with pytest.raises(ValueError, match="未知的 DataFrame 后端"):
|
||||
TdxData(df="numpy")
|
||||
|
||||
|
||||
def test_tdxdata_accepts_supported_backend():
|
||||
td = TdxData(df="polars")
|
||||
assert td._df_backend == "polars"
|
||||
|
||||
|
||||
class _Source:
|
||||
"""最小替身:只验装饰器语义,不碰网络。"""
|
||||
|
||||
def __init__(self, backend: str | None) -> None:
|
||||
self._df_backend = backend
|
||||
|
||||
@framed
|
||||
async def single(self) -> list[SecurityBar]:
|
||||
return [bar()]
|
||||
|
||||
@framed
|
||||
async def nested(self) -> dict[str, list[SecurityBar]]:
|
||||
got = await self.single() # 内层必须是原始 list
|
||||
assert isinstance(got, list), f"内层被提前转换成了 {type(got)!r}"
|
||||
return {"600000": got}
|
||||
|
||||
|
||||
async def test_decorator_passthrough_when_no_backend():
|
||||
src = _Source(None)
|
||||
assert isinstance(await src.single(), list)
|
||||
|
||||
|
||||
async def test_decorator_converts_outermost_only():
|
||||
src = _Source("polars")
|
||||
out = await src.nested()
|
||||
assert cols(out, "polars")[0] == "market"
|
||||
assert rows(out, "polars") == 1
|
||||
|
||||
|
||||
def test_model_name_resolution():
|
||||
# 注解是 dict[str, list[SecurityBar]] 时正则也能取到 SecurityBar;
|
||||
# 但 "dict" 这类内建名在 models 里找不到,必须返回 None 而不是内建类型
|
||||
assert _frames._model_by_name("dict") is None
|
||||
assert _frames._model_by_name("SecurityBar") is SecurityBar
|
||||
|
||||
|
||||
def test_datetime_dtype_is_datetime_like():
|
||||
pl = pytest.importorskip("polars")
|
||||
df = to_frame([bar()], backend="polars")
|
||||
assert isinstance(df.schema["datetime"], pl.Datetime)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 交易日列缓存
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_trading_date_multi_key_batch_matches_model(backend):
|
||||
"""一批里混着 A股与美股、跨 0~23 点,逐行都要等于模型自己的 ``trading_date``。
|
||||
|
||||
``trading_date`` 是按「时间字段」缓存算出来的(240k 行常只有 1~2 个键),
|
||||
这里专门造出「多键同批」的极端情形,防止缓存键漏掉参与计算的因素。
|
||||
|
||||
注意:美股的跨日处理在 ``SecurityBar.__post_init__`` 里就做完了(凌晨段
|
||||
的年月日会被推进一天),所以 ``trading_date`` 对每个 bar 只是自然日,
|
||||
真正会分叉的是 ``datetime`` —— 两者都要逐行核对。
|
||||
"""
|
||||
batch = []
|
||||
for market in (int(Market.SH), int(ExMarket.US_STOCK)):
|
||||
for hour in range(24):
|
||||
batch.append(bar(market=market, hour=hour, minute=15))
|
||||
df = to_frame(batch, backend=backend)
|
||||
got_td = [cell(df, backend, "trading_date", i) for i in range(rows(df, backend))]
|
||||
got_dt = [cell(df, backend, "datetime", i) for i in range(rows(df, backend))]
|
||||
assert got_td == [b.trading_date for b in batch]
|
||||
assert got_dt == [b.datetime for b in batch]
|
||||
# 美股凌晨段的 datetime 确实落在下一天(否则这条测试等于没测)
|
||||
us = [b for b in batch if b.market == int(ExMarket.US_STOCK)]
|
||||
assert {b.datetime.date().isoformat() for b in us} == {"2026-10-27", "2026-10-28"}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_minute_trading_date_cache_matches_model(backend):
|
||||
"""分时的 ``day_offset`` 也参与交易日计算,缓存键不能漏。"""
|
||||
batch = [
|
||||
MinuteBar(price=10.0, vol=1.0, hour=9, minute=30, date=20261027),
|
||||
MinuteBar(price=10.0, vol=1.0, hour=21, minute=30, date=20261027,
|
||||
market=int(ExMarket.US_STOCK)),
|
||||
MinuteBar(price=10.0, vol=1.0, hour=3, minute=0, date=20261027,
|
||||
market=int(ExMarket.US_STOCK), day_offset=1),
|
||||
]
|
||||
df = to_frame(batch, backend=backend)
|
||||
got = [cell(df, backend, "trading_date", i) for i in range(rows(df, backend))]
|
||||
assert got == [b.trading_date for b in batch]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_every_bar_field_equals_model(backend):
|
||||
"""逐字段等价:DataFrame 的每个值都必须等于 dataclass 的同名字段。
|
||||
|
||||
这条是取列路径(``attrgetter`` 批量取 + 按列转 Enum)改动的兜底,
|
||||
避免「快是快了,但某列悄悄串了值」。
|
||||
"""
|
||||
src = bar(minute=15, settle=3.5)
|
||||
df = to_frame([src], backend=backend)
|
||||
for name in cols(df, backend):
|
||||
want = getattr(src, name)
|
||||
assert cell(df, backend, name) == want, name
|
||||
@@ -0,0 +1,166 @@
|
||||
"""K线「按日期区间」取数的路径选择:倒推快路径 vs 二分慢路径。
|
||||
|
||||
背景:二分要先探总根数(翻倍 6 次)再跳 log2(N) 次,1 分钟 K 线约 36 次串行
|
||||
往返 = 4 秒级;而「取今天」「增量到最新」这类贴着右端的请求,倒推只要 1~3 次。
|
||||
"""
|
||||
|
||||
import datetime
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
from pytdxdata import TdxData
|
||||
from pytdxdata.models import KlinePeriod, SecurityBar
|
||||
|
||||
NEWEST = datetime.datetime(2026, 10, 28, 15, 0) # noqa: DTZ001 — 全项目用裸北京时间
|
||||
TOTAL_BARS = 5_000
|
||||
|
||||
|
||||
def _bar(when: datetime.datetime, close: float) -> SecurityBar:
|
||||
return SecurityBar(
|
||||
market=1, code="600000", open=10.0, high=10.1, low=9.9, close=close,
|
||||
vol=1000, amount=10000, year=when.year, month=when.month, day=when.day,
|
||||
hour=when.hour, minute=when.minute)
|
||||
|
||||
|
||||
class _FakeConn:
|
||||
"""标准通道语义替身:**offset 0 = 最新一根**,偏移越大越老。"""
|
||||
|
||||
calls = 0
|
||||
|
||||
def __init__(self, host="s1", port=7709, timeout=10.0):
|
||||
self.host = host
|
||||
self.alive_flag = True
|
||||
|
||||
@property
|
||||
def alive(self):
|
||||
return self.alive_flag
|
||||
|
||||
async def connect(self):
|
||||
pass
|
||||
|
||||
async def close(self):
|
||||
self.alive_flag = False
|
||||
|
||||
async def execute_pipeline(self, cmds, **kwargs):
|
||||
return [await self.execute(c) for c in cmds]
|
||||
|
||||
async def execute(self, cmd):
|
||||
type(self).calls += 1
|
||||
if "GetSecurityBarsCmd" not in type(cmd).__name__:
|
||||
return []
|
||||
rows = []
|
||||
for off in range(cmd.start, min(cmd.start + cmd.count, TOTAL_BARS)):
|
||||
rows.append(_bar(NEWEST - datetime.timedelta(days=off),
|
||||
10.0 + (TOTAL_BARS - off) * 0.01))
|
||||
return rows
|
||||
|
||||
|
||||
class _AscendingConn(_FakeConn):
|
||||
"""方向相反的替身(MAC/EX 语义):offset 0 = **最老**一根。"""
|
||||
|
||||
async def execute(self, cmd):
|
||||
type(self).calls += 1
|
||||
if "GetSecurityBarsCmd" not in type(cmd).__name__:
|
||||
return []
|
||||
rows = []
|
||||
base = datetime.datetime(2020, 1, 1) # noqa: DTZ001
|
||||
for off in range(cmd.start, min(cmd.start + cmd.count, TOTAL_BARS)):
|
||||
rows.append(_bar(base + datetime.timedelta(days=off), 10.0 + off * 0.01))
|
||||
return rows
|
||||
|
||||
|
||||
def _patched(conn_cls):
|
||||
return (
|
||||
mock.patch("pytdxdata.api.get_standard_hosts", return_value=["s1"]),
|
||||
mock.patch("pytdxdata.api.get_mac_hosts", return_value=["m1"]),
|
||||
mock.patch("pytdxdata.pool.pool.TdxConnection",
|
||||
side_effect=lambda h, p=7709, t=10.0: conn_cls(h)),
|
||||
)
|
||||
|
||||
|
||||
async def _fetch(conn_cls, **kwargs):
|
||||
conn_cls.calls = 0
|
||||
p1, p2, p3 = _patched(conn_cls)
|
||||
with p1, p2, p3:
|
||||
td = TdxData(cache_dir=None, pool_min=1, pool_max=2)
|
||||
await td.start()
|
||||
try:
|
||||
bars = await td.get_bars("sh600000", KlinePeriod.DAY, **kwargs)
|
||||
finally:
|
||||
await td.close()
|
||||
return bars, conn_cls.calls
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_recent_range_uses_backtrack():
|
||||
"""右端贴最新 → 倒推,往返次数是个位数(不是二分那几十次)。"""
|
||||
bars, calls = await _fetch(
|
||||
_FakeConn, start_date=NEWEST - datetime.timedelta(days=5), end_date=NEWEST)
|
||||
|
||||
# 返回按时间升序 → 偏移由老到新(5,4,...,0)
|
||||
assert [b.datetime.date() for b in bars] == [
|
||||
(NEWEST - datetime.timedelta(days=i)).date() for i in range(5, -1, -1)]
|
||||
assert calls <= 4, f"倒推路径应 1~3 次往返,实际 {calls}"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_end_before_newest_within_one_bar_still_backtracks():
|
||||
"""end 比最新一根略早(K线时间戳是周期结束时刻)仍算贴最新。"""
|
||||
end = NEWEST - datetime.timedelta(minutes=30)
|
||||
bars, calls = await _fetch(
|
||||
_FakeConn, start_date=NEWEST - datetime.timedelta(days=3), end_date=end)
|
||||
|
||||
# offset 0 那根(15:00)晚于 end,被裁掉
|
||||
assert bars[-1].datetime.date() == (NEWEST - datetime.timedelta(days=1)).date()
|
||||
assert calls <= 4
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_middle_range_falls_back_to_bisect():
|
||||
"""历史中段倒推够不着左边 → 回落二分,结果仍要正确。"""
|
||||
bars, calls = await _fetch(
|
||||
_FakeConn,
|
||||
start_date=NEWEST - datetime.timedelta(days=100),
|
||||
end_date=NEWEST - datetime.timedelta(days=90))
|
||||
|
||||
assert [b.datetime.date() for b in bars] == [
|
||||
(NEWEST - datetime.timedelta(days=i)).date() for i in range(100, 89, -1)]
|
||||
assert calls > 10, f"中段应走二分(几十次往返),实际只有 {calls}"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_future_range_returns_empty_quickly():
|
||||
# 用相对 now 的未来,别用固定日期:固定日期会随运行日漂进 _BACKTRACK_GRACE
|
||||
# (默认 1 天容差覆盖长假),倒推就成立、会多取一轮,断言随日期时绿时红。
|
||||
future = datetime.datetime.now() + datetime.timedelta(days=5) # noqa: DTZ001
|
||||
bars, calls = await _fetch(_FakeConn, start_date=future)
|
||||
|
||||
assert bars == []
|
||||
assert calls <= 3
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ascending_channel_skips_backtrack():
|
||||
"""offset 0 是最老(MAC/EX 语义)时倒推不成立,必须走二分且结果正确。"""
|
||||
base = datetime.datetime(2020, 1, 1) # noqa: DTZ001
|
||||
bars, calls = await _fetch(
|
||||
_AscendingConn,
|
||||
start_date=base + datetime.timedelta(days=99),
|
||||
end_date=base + datetime.timedelta(days=109))
|
||||
|
||||
assert [b.datetime.date() for b in bars] == [
|
||||
(base + datetime.timedelta(days=i)).date() for i in range(99, 110)]
|
||||
assert calls > 10
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_backtrack_matches_plain_count_result():
|
||||
"""倒推裁剪结果 == 直接取一大段再裁剪(两条路径口径必须一致)。"""
|
||||
start = NEWEST - datetime.timedelta(days=7)
|
||||
by_date, _ = await _fetch(_FakeConn, start_date=start, end_date=NEWEST)
|
||||
whole, _ = await _fetch(_FakeConn, count=60)
|
||||
|
||||
expect = [b for b in whole if b.datetime >= start]
|
||||
assert [(b.datetime, b.close) for b in by_date] == \
|
||||
[(b.datetime, b.close) for b in expect]
|
||||
@@ -0,0 +1,106 @@
|
||||
"""报价快照时间字段(0x053E 第 6 个变长字段)解码口径。
|
||||
|
||||
2026-10-30 修正前按「小时 + 百万分之一小时」解码,股票 11:29:59 会解成
|
||||
11:17:57、ETF 会解出 `113:00:21` 这种非法小时。这里把实测口径锁死:
|
||||
十进制位拼接 `HH MM SS` + 价格小数位,SS 的 50~59 段服务端写成 90~99。
|
||||
"""
|
||||
import struct
|
||||
|
||||
from pytdxdata.protocol.commands.quotes import (
|
||||
GetSecurityQuotesCmd,
|
||||
format_server_time,
|
||||
price_decimal_digits,
|
||||
)
|
||||
from pytdxdata.protocol.price import put_price
|
||||
|
||||
# ---------- 纯解码 ----------
|
||||
|
||||
def test_stock_two_digits():
|
||||
# 13:22:40.02(股票价格 2 位小数 → 时间带 2 位小数)
|
||||
assert format_server_time(13224002, 2) == "13:22:40.02"
|
||||
|
||||
|
||||
def test_etf_three_digits():
|
||||
# 13:22:45.903(ETF 3 位小数)
|
||||
assert format_server_time(132245903, 3) == "13:22:45.903"
|
||||
|
||||
|
||||
def test_bond_four_digits():
|
||||
# 13:10:22.8050(国债 / 回购 4 位小数)
|
||||
assert format_server_time(1310228050, 4) == "13:10:22.8050"
|
||||
|
||||
|
||||
def test_seconds_50_59_written_as_90_99():
|
||||
# 服务端把 52 秒写成 92 → 必须还原成 13:17:52.01
|
||||
assert format_server_time(13179201, 2) == "13:17:52.01"
|
||||
assert format_server_time(13179501, 2) == "13:17:55.01"
|
||||
assert format_server_time(13179801, 2) == "13:17:58.01"
|
||||
# 90 以下原样(不要误伤 51、59 这类合法秒)
|
||||
assert format_server_time(13175101, 2) == "13:17:51.01"
|
||||
assert format_server_time(13175901, 2) == "13:17:59.01"
|
||||
|
||||
|
||||
def test_index_and_garbage_return_empty():
|
||||
# 指数该字段不是时钟(同一时刻 000001 解出 12:89 → MM 越界)
|
||||
assert format_server_time(12867393, 2) == ""
|
||||
assert format_server_time(0, 2) == ""
|
||||
# 小时越界
|
||||
assert format_server_time(113005895, 2) == ""
|
||||
|
||||
|
||||
def test_digits_follow_decimal_point():
|
||||
assert price_decimal_digits(1, "600000") == 2
|
||||
assert price_decimal_digits(1, "510300") == 3
|
||||
assert price_decimal_digits(1, "019547") == 4
|
||||
|
||||
|
||||
# ---------- 整条报文 ----------
|
||||
|
||||
def _quote_record(market: int, code: str, price_raw: int, time_raw: int,
|
||||
vol: int = 1000, rise: int = 0) -> bytes:
|
||||
"""按 0x053E 记录布局拼一条报价(金额字段填 0,不参与本节断言)。"""
|
||||
out = bytearray(struct.pack("<B6sH", market, code.encode(), 0))
|
||||
for v in (price_raw, 0, 0, 0, 0): # price / lastclose / open / high / low
|
||||
out += put_price(v)
|
||||
out += put_price(time_raw) # ← 快照时间
|
||||
out += put_price(-price_raw) # unknown1(多数标的 = -price)
|
||||
out += put_price(vol) # vol
|
||||
out += put_price(1) # cur_vol
|
||||
out += struct.pack("<I", 0) # amount(4 字节自定义浮点)
|
||||
for v in (vol // 2, vol // 2, 0, 0): # s_vol / b_vol / unknown2 / unknown3
|
||||
out += put_price(v)
|
||||
for _ in range(5): # 五档:价差 0,量 1
|
||||
for v in (0, 0, 1, 1):
|
||||
out += put_price(v)
|
||||
out += struct.pack("<H", 0) # trading_status
|
||||
for _ in range(4):
|
||||
out += put_price(0)
|
||||
out += struct.pack("<hH", rise, 0) # rise_speed / active2
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def _body(records: list[bytes]) -> bytes:
|
||||
# 前 2 字节是服务端魔数(解析时跳过),随后是 2 字节条数
|
||||
return b"\x00\x00" + struct.pack("<H", len(records)) + b"".join(records)
|
||||
|
||||
|
||||
def test_parse_response_sets_server_time():
|
||||
body = _body([
|
||||
_quote_record(1, "600000", 996, 13224002),
|
||||
_quote_record(1, "510300", 4105, 132245903),
|
||||
])
|
||||
quotes = GetSecurityQuotesCmd([(1, "600000"), (1, "510300")]).parse_response(body)
|
||||
assert [q.server_time for q in quotes] == ["13:22:40.02", "13:22:45.903"]
|
||||
# 价格口径未受影响
|
||||
assert [round(q.price, 3) for q in quotes] == [9.96, 4.105]
|
||||
assert [q.decimal_point for q in quotes] == [2, 3]
|
||||
|
||||
|
||||
def test_parse_response_blank_time_for_index():
|
||||
body = _body([
|
||||
_quote_record(1, "000001", 363808, 12867393),
|
||||
_quote_record(1, "600000", 996, 13179201),
|
||||
])
|
||||
quotes = GetSecurityQuotesCmd([(1, "000001"), (1, "600000")]).parse_response(body)
|
||||
assert quotes[0].server_time == ""
|
||||
assert quotes[1].server_time == "13:17:52.01"
|
||||
Reference in new issue
Block a user