3 Commits
Author SHA1 Message Date
openbot-coder 7c4541a315 fix(ex): 期货/期权K线 amount 恒 0 + 补齐持仓量 position 字段 (#17)
- 0x23FF 帧内无成交额槽位(协议语义,非解析遗漏),期货/期权 amount
  按 close×vol×合约乘数 估算(新增 ex_multiplier 乘数表,覆盖
  cffex/shf/dl/zz/gz 主流品种 + 股票期权 10000;未知品种保持 0.0 不编造)
- position(持仓量)协议层早已解析但 api 层丢弃,v3 起透传至
  SecurityBar.position(权益类市场无此概念 → None)
- 缓存 schema v2→v3:旧缓存 amount 恒 0/无 position,自然失效重建
- 真机验证:rb2611/IFL0/au2611/上交所期权 10011425 日线
  amount/settle/position 全部有值
2026-11-14 14:16:10 +08:00
openbotandopenbot-coder 1cd8f692a1 fix(pool): 服务器中途掉线不再拖垮整批任务 + 发布 0.7.2 (#16)
* fix(pool): 服务器中途掉线不再拖垮整批任务

三种调度细节会把「一台机器坏了」放大成「整批卡住」:

1. acquire() 在池锁内做 TCP 握手 + 探针 —— 一台主机握手卡满 timeout,
   所有并发协程的 acquire/release 一起被挡。改为锁内只预留槽位、
   握手在锁外做,失败时回滚计数。
2. 扩容失败分支 continue 且无截止检查 —— 全节点失联时无限重试,
   永远不抛 PoolTimeoutError。改为同一次 acquire 内每台主机只试一遍,
   试完立刻抛错;再加池级熔断(BLACKOUT_COOLDOWN=2s),期内直接失败,
   任一连接成功即解除。否则上层「按标的逐只重试」会把 16×6000 次握手
   放大成 60s+ 的空转。
3. 批量 K 线失败批次攒到 gather 结束再串行补 —— 每只重付一次 RTT。
   改为 extendleft 回队首由健康连接立刻接手(每只最多 2 次),
   尾部兜底也改成信号量并发。分页器同理:worker 退出时把剩余页放回
   共享 leftover,活着的 worker 接手,全部退出后仍有剩页则借新连接补完。

EX 池同步改为锁外建连,并给握手失败主机加 30s 拉黑。

实测(16 台替身节点、6000 只 × 240 根 1 分钟 K 线、0.9s 后变黑洞):
  无故障   1.79s → 1.80s
  2/16 死   2.28s → 1.91s
  6/16 死   2.97s → 2.30s
  16/16 死  >60s 未返回 → 2.37s

新增 tests/test_failover.py(7 项,含熔断期不再握手)与
scripts/bench_failover.py(对比压测)。

* chore(release): 0.7.2

- server_time 解码修正(0.7.1 及之前按「小时 + 百万分之一小时」解析是错的)
- 服务器中途掉线的四项隔离机制(锁外建连 / 本轮不重试 / 池级熔断 / 批次回队)
- 版本号四处对齐:pyproject.toml、__init__.py、uv.lock、CHANGELOG

---------

Co-authored-by: openbot-coder <openbot-coder@users.noreply.github.com>
2026-10-30 16:57:29 +08:00
openbotandopenbot-coder dea39db441 fix(quotes): 修正 server_time 解码口径(十进制位拼接 HHMMSS + 价格小数位) (#15)
Co-authored-by: openbot-coder <openbot-coder@users.noreply.github.com>
2026-10-30 13:38:51 +08:00
18 changed files with 1191 additions and 96 deletions

No files matched your search

+70 -1
View File
@@ -2,6 +2,74 @@
本项目遵循 [语义化版本](https://semver.org/lang/zh-CN/)。 本项目遵循 [语义化版本](https://semver.org/lang/zh-CN/)。
## [0.7.2] - 2026-10-30
### 可靠性
- **服务器中途掉线不再拖垮整批任务**。
此前三种调度细节会把「一台机器坏了」放大成「整批卡住」:
| 问题 | 后果 | 现在 |
|---|---|---|
| `acquire()` 在**池锁内**做 TCP 握手 + 探针 | 一台坏机握手卡满 `timeout`,所有并发协程一起停住 | 锁内只预留槽位,握手在锁外做 |
| 扩容失败分支 `continue` 且无截止检查 | 全节点失联时无限重试,永远不抛 `PoolTimeoutError` | 每台主机同一次 `acquire` 内只试一遍,试完立刻抛错 |
| 全灭后上层逐只重试 | 6,000 只 × 16 台 × 握手的空转 | 池级熔断 2 秒,熔断期内直接失败;有连接成功即解除 |
实测(16 台替身节点、6,000 只 × 240 根 1 分钟 K 线、跑起来 0.9 秒后 6 台变黑洞):
| 场景 | 0.7.1 | 0.7.2 |
|---|---|---|
| 无节点故障 | 1.79 s | 1.80 s |
| 2/16 中途失联 | 2.28 s | **1.91 s** |
| 6/16 中途失联 | 2.97 s | **2.30 s** |
| **16/16 中途失联** | **>60 s 未返回** | **2.37 s** |
- **批量 K 线失败批次从「事后串行补」改为「立刻回队」**。
流水线中途失败时,原来把整批攒到 `gather` 结束再逐只重试 —— 每只都要重付一次 RTT,
几百只就是十几秒长尾。现在失败批次直接 `extendleft` 回队首,由**其它健康连接**马上接手
(每只最多回队 2 次,超限后落进尾部并发兜底)。尾部兜底也从串行改成信号量并发。
- **分页器不再因为一条连接坏掉而丢掉它那份页**。
老实现按 `jobs[wid::k]` 静态切片,一个 worker 退出,它那一整份页就没人做了。现在
worker 退出时把剩余页放回共享 `leftover`,还活着的 worker 会接着做;全部 worker 退出后
还有剩页,再借一条新连接补完(最多两轮)。
- **EX 池同样改为锁外建连**,并给握手失败的主机加 30 秒拉黑(`ExPool.FAIL_COOLDOWN`),
避免每次 `acquire` 都重付一遍握手超时。
### 修复
- **修复 `get_quotes` 的 `server_time`(行情快照时间)解码错误**。
原实现按「小时 + 百万分之一小时」(`divmod(raw, 1_000_000)`)解码,但协议实际是
**十进制位拼接**:
```
整数 = HH*10**(4+d) + MM*10**(2+d) + SS*10**d + 小数
d = 该品种价格小数位(股票 2 / ETF·LOF·B股 3 / 国债·回购 4)
```
后果:股票 13:29:59 显示成 `11:17:57`(基准错、速率也只有实际的 53%),ETF/债券
直接给出小时 > 23 的非法值。
| 标的 | raw | 0.7.1 显示 | 现在 |
|---|---|---|---|
| `sh600000`(dp=2) | `13224002` | `13:13:26.407` | **`13:22:40.02`** |
| `sh510300`(dp=3) | `132245903` | `132:14:45.250` | **`13:22:45.903`** |
| `sh204001`(dp=4) | `1310228050` | `1310:13:40.980` | **`13:10:22.8050`** |
- 小数位数跟着品种价格精度走,输出 `HH:MM:SS.ff`(2~4 位,兼容旧文档的 `HH:MM:SS.mmm`)。
- 服务端怪癖:**SS 落在 50~59 秒时写成 90~99**(编码值 +40 秒)。未修正时该区间的滞后
会从稳定的 10.6 秒突变成 -29.4 秒;现在 `SS >= 60` 一律减 40 秒还原。
- **指数(`sh000001` / `sz399001` / `sh000300` …)该字段返回空串**:协议里它对指数不是
时钟(同一时刻 `000001` 解出 `12:89`、`399001` 解出 `12:15`,互相矛盾)。此前会给出一个
看着合理、其实是垃圾的时间。要指数时间请用 `get_minutes` / K 线。
- 实测验收(2026-10-30 盘中,同服务器连续采样 3 分钟):`server_time` 与本地时钟的滞后
稳定在 10.5~10.9 秒;股票 / ETF / B股 / 国债 / 回购全覆盖,见 `tests/test_quote_time.py`。
## [0.7.1] - 2026-10-29 ## [0.7.1] - 2026-10-29
### 性能 ### 性能
@@ -853,7 +921,8 @@ pip install -U "pytdxdata[polars]" # 想直出 DataFrame
- 初始版本:动态连接池 + 双层 TTL 缓存 + 3 连接并发分页 + 双通道(标准 / MAC)+ 扩展市场(港股 / 美股 / 期货 / 期权) - 初始版本:动态连接池 + 双层 TTL 缓存 + 3 连接并发分页 + 双通道(标准 / MAC)+ 扩展市场(港股 / 美股 / 期货 / 期权)
[Unreleased]: https://github.com/openbot-coder/pytdxdata/compare/v0.7.1...HEAD [Unreleased]: https://github.com/openbot-coder/pytdxdata/compare/v0.7.2...HEAD
[0.7.2]: https://github.com/openbot-coder/pytdxdata/compare/v0.7.1...v0.7.2
[0.7.1]: https://github.com/openbot-coder/pytdxdata/compare/v0.7.0...v0.7.1 [0.7.1]: https://github.com/openbot-coder/pytdxdata/compare/v0.7.0...v0.7.1
[0.7.0]: https://github.com/openbot-coder/pytdxdata/compare/v0.6.0...v0.7.0 [0.7.0]: https://github.com/openbot-coder/pytdxdata/compare/v0.6.0...v0.7.0
[0.6.0]: https://github.com/openbot-coder/pytdxdata/compare/v0.5.0...v0.6.0 [0.6.0]: https://github.com/openbot-coder/pytdxdata/compare/v0.5.0...v0.6.0
+13 -1
View File
@@ -1,6 +1,6 @@
# pytdxdata 对外接口清单 # pytdxdata 对外接口清单
> 以 `src/pytdxdata` 源码为准整理(v0.7.1)。分四类:Python API / 数据模型 / 辅助模块 / CLI。 > 以 `src/pytdxdata` 源码为准整理(v0.7.2)。分四类:Python API / 数据模型 / 辅助模块 / CLI。
> >
> 方法与代码的一致性由 `scripts/check_docs.py` 在 CI 中校验。 > 方法与代码的一致性由 `scripts/check_docs.py` 在 CI 中校验。
@@ -228,6 +228,18 @@ from pytdxdata import (
> 注:`__init__` 仅导出 `Market/ExMarket/KlinePeriod/Adjust` 与 6 个核心模型;`BoardInfo/MemberQuote/AuctionItem` 等需 `from pytdxdata.models import ...`。 > 注:`__init__` 仅导出 `Market/ExMarket/KlinePeriod/Adjust` 与 6 个核心模型;`BoardInfo/MemberQuote/AuctionItem` 等需 `from pytdxdata.models import ...`。
**`SecurityQuote.server_time`(行情快照时间,0.7.2 修正)**
- 格式 `HH:MM:SS.ff`,小数位数 = 该品种**价格小数位**:股票 2、ETF/LOF/B股 3、国债·回购 4。
例 `13:22:40.02` / `13:22:45.903` / `13:10:22.8050`。
- 它比本地时钟**慢 5~15 秒**(服务器快照滞后),超低流动性品种(货币 ETF、国债)可能差几分钟。
判断「行情有没有更新」请看这个值有没有变,别看调用时刻 —— 盘口本身约 3 秒刷新一次。
- **指数(`sh000001` / `sz399001` / `sh000300` …)恒为空串**:该字段对指数不是时钟
(同一时刻 `000001` 解出 `12:89`、`399001` 解出 `12:15`,协议内没有可用信息)。
要指数时间请用 `get_minutes` / K 线。
- ⚠️ 0.7.1 及之前按「小时 + 百万分之一小时」解码是**错的**:股票 `11:29:59` 会显示成
`11:17:57`,ETF/债券会显示成 `113:00:21` 这种非法小时。
**量纲对照(跨品种,2026-10-22 实测)** **量纲对照(跨品种,2026-10-22 实测)**
同名字段在不同品种上单位不同。最典型的坑是用 `amount / vol` 算均价——它在可转债上会**错 10 倍**: 同名字段在不同品种上单位不同。最典型的坑是用 `amount / vol` 算均价——它在可转债上会**错 10 倍**:
+24
View File
@@ -81,6 +81,30 @@ results = await asyncio.gather(*[
EWMA(指数加权移动平均)延迟让选路对「最近变慢」敏感,而不是被历史均值拖住。 EWMA(指数加权移动平均)延迟让选路对「最近变慢」敏感,而不是被历史均值拖住。
### 单台服务器中途掉线
「掉线」和「慢」是两回事:慢只是多等一会儿,掉线会把整批任务卡住。0.7.2 起有四道闸:
| 机制 | 作用 |
|---|---|
| **锁外建连** | `acquire` 锁内只预留槽位,TCP 握手 + 探针在锁外做。否则一台主机握手卡满 `timeout` 期间,所有并发协程的 `acquire` / `release` 全被池锁挡住 |
| **本轮不重试同一台** | 一次 `acquire` 里握手失败的主机记进 `avoid`,不再重复试;每台都试完立刻抛 `PoolTimeoutError` |
| **池级熔断** | 全主机失败后进入 `BLACKOUT_COOLDOWN`(2 秒)黑洞期,期内 `acquire` 直接失败、不再握手;任一连接成功即解除 |
| **批次回队 / 份额再分配** | 批量 K 线流水线失败的那一批立刻回队首由健康连接接手;分页器让活着的 worker 接手坏 worker 剩下的页 |
实测 16 台替身节点、6,000 只 × 240 根 1 分钟 K 线、跑起来 0.9 秒后集体黑洞:
| 场景 | 0.7.1 | 0.7.2 |
|---|---|---|
| 无节点故障 | 1.79 s | 1.80 s |
| 2/16 中途失联 | 2.28 s | 1.91 s |
| 6/16 中途失联 | 2.97 s | 2.30 s |
| 16/16 中途失联 | **> 60 s 未返回** | **2.37 s** |
全灭时的行为分两种:**一开始就没有可用连接** → `get_bars` 抛 `PoolTimeoutError`(不再静默
返回空列表);**跑到一半全灭** → 返回已经取到的部分(不抛错、不挂死)。熔断只在**全部主机**
都握手失败时触发,池子满了排队等待不受影响。
### MAC 通道的单台连接上限 ### MAC 通道的单台连接上限
`mac_pool_cap` 默认 `4`,与标准池的 `std_pool_cap` 一致——这是**对端单台连接限制**, `mac_pool_cap` 默认 `4`,与标准池的 `std_pool_cap` 一致——这是**对端单台连接限制**,
+1 -1
View File
@@ -1,6 +1,6 @@
[project] [project]
name = "pytdxdata" name = "pytdxdata"
version = "0.7.1" version = "0.7.3"
description = "通达信(TongDaXin)行情数据客户端:动态连接池 + 双层TTL缓存 + 3连接并发分页,纯asyncio零运行时依赖,支持A股/港股/美股/期货/期权" description = "通达信(TongDaXin)行情数据客户端:动态连接池 + 双层TTL缓存 + 3连接并发分页,纯asyncio零运行时依赖,支持A股/港股/美股/期货/期权"
readme = "README.md" readme = "README.md"
requires-python = ">=3.11" requires-python = ">=3.11"
+147
View File
@@ -0,0 +1,147 @@
"""坏节点隔离压测:N 台服务器,其中几台**跑到一半突然死掉**。
关注两个指标:
1. **总耗时** —— 死节点不能把整批任务拖长(老实现会因为持锁建连 + 串行补
重试把长尾放大成倍)。
2. **数据完整度** —— 死节点不能造成缺数据。
用法::
python scripts/bench_failover.py --syms 3000 --hosts 16 --dead 5,9
输出一行 JSON,便于对比改前/改后。
"""
from __future__ import annotations
import argparse
import asyncio
import json
import time
import pytdxdata.api as api
import pytdxdata.pool.pool as poolmod
from pytdxdata import TdxData
from pytdxdata.models import KlinePeriod, SecurityBar
NOW = time.monotonic()
PIPE_LATENCY = 0.012 # 一次流水线往返(模拟公网 12ms)
CONNECT_LATENCY = 0.05 # 正常主机 TCP 握手 + 握手命令
DEAD_HANG = 0.8 # 死节点:请求发出去没响应,最终超时
CONNECT_HANG = 0.0 # 黑洞握手:连上了但握手永不完(0=直接拒绝)
DEAD_AT = 0.0 # 死节点从第几秒开始不响应
class FakeConn:
dead: set[int] = set()
calls = 0
pipes = 0
dead_pipe_hits = 0 # 打到死节点的流水线次数(含"半开等待")
def __init__(self, host, port=7709, timeout=3.0):
self.host = host
self.idx = int(host[1:])
self._alive = True
@property
def alive(self) -> bool:
return self._alive
async def connect(self) -> None:
if self._is_dead() and CONNECT_HANG:
# 黑洞节点:TCP 连上但握手包永远不回 —— 真实宕机最常见的样子
await asyncio.sleep(CONNECT_HANG)
raise TimeoutError(f"{self.host} 握手超时")
await asyncio.sleep(CONNECT_LATENCY)
if self._is_dead():
raise ConnectionRefusedError(f"{self.host} 已宕机")
def _is_dead(self) -> bool:
return self.idx in type(self).dead and time.monotonic() - NOW >= DEAD_AT
async def execute(self, cmd):
await asyncio.sleep(PIPE_LATENCY)
if self._is_dead():
raise ConnectionError(f"{self.host} 无响应")
type(self).calls += 1
return self._rows(cmd)
async def execute_pipeline(self, cmds, **kwargs):
await asyncio.sleep(PIPE_LATENCY)
if self._is_dead():
type(self).dead_pipe_hits += 1
await asyncio.sleep(DEAD_HANG) # 半开连接:等到超时才报错
raise ConnectionError(f"{self.host} 无响应")
type(self).pipes += 1
return [self._rows(c) for c in cmds]
async def close(self) -> None:
self._alive = False
@staticmethod
def _rows(cmd):
if "MacKlineCmd" not in type(cmd).__name__:
return []
bar = SecurityBar(
market=1, code="600000", open=10.0, high=10.1, low=9.9, close=10.0,
vol=1000, amount=10000, year=2026, month=1, day=1, hour=9, minute=30)
return [bar] * int(getattr(cmd, "count", 1)) # 同一个对象复用,便于控内存
async def run(n_syms: int, n_hosts: int, dead: set[int]) -> dict:
global NOW
NOW = time.monotonic()
FakeConn.dead = dead
FakeConn.calls = 0
FakeConn.pipes = 0
FakeConn.dead_pipe_hits = 0
hosts = [f"h{i}" for i in range(n_hosts)]
api.get_mac_hosts = lambda: hosts
api.get_standard_hosts = lambda: hosts # 别让标准池去碰真实公网主机
api.get_ex_hosts = lambda: []
poolmod.schedule_probe = lambda: None # 关掉后台全量测速
poolmod.TdxConnection = FakeConn # 池内部按名字取,必须在建池之前替身
# cap=1:一台主机一条连接,坏一台就少一条链路(最坏情况)
td = TdxData(cache_dir=None, pool_min=4, pool_max=24, mac_pool_cap=1)
await td.start()
syms = [f"sh60{i:04d}" for i in range(n_syms)]
t0 = time.monotonic()
try:
bars = await td.get_bars(syms, KlinePeriod.MIN_1, count=240)
finally:
elapsed = time.monotonic() - t0
await td.close()
return {
"syms": n_syms,
"hosts": n_hosts,
"dead": sorted(dead),
"elapsed_s": round(elapsed, 3),
"bars": len(bars),
"expect_bars": n_syms * 240,
"complete": len(bars) == n_syms * 240,
"pipeline_calls": FakeConn.pipes,
"dead_pipe_hits": FakeConn.dead_pipe_hits,
}
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--syms", type=int, default=3000)
ap.add_argument("--hosts", type=int, default=16)
ap.add_argument("--dead", default="5,9",
help="中途死掉的主机下标,逗号分隔;空串=全健康")
ap.add_argument("--die-at", type=float, default=0.25,
help="开跑后第几秒这些主机开始不响应(0=一开始就死)")
ap.add_argument("--connect-hang", type=float, default=0.0,
help="死节点的握手卡顿秒数(>0 模拟黑洞节点,0=直接拒绝)")
args = ap.parse_args()
dead = {int(x) for x in args.dead.split(",") if x.strip()}
global DEAD_AT, CONNECT_HANG
DEAD_AT = args.die_at
CONNECT_HANG = args.connect_hang
print(json.dumps(asyncio.run(run(args.syms, args.hosts, dead)),
ensure_ascii=False))
if __name__ == "__main__":
main()
+1 -1
View File
@@ -18,7 +18,7 @@ from .models import (
XdxrRecord, XdxrRecord,
) )
__version__ = "0.7.1" __version__ = "0.7.2"
__all__ = [ __all__ = [
"TdxData", "TdxData",
"Market", "Market",
+52 -13
View File
@@ -187,6 +187,9 @@ _BULK_MAX_COUNT = 700 # MAC 单请求返回上限;超过只能逐只分页
# 最多用几台服务器:池里 24 台全用反而更慢 —— 慢节点会成为长尾(最长的那台 # 最多用几台服务器:池里 24 台全用反而更慢 —— 慢节点会成为长尾(最长的那台
# 拖住整轮),实测 count=2 全市场 24 台 2.95s / 只取最快 12 台 1.45s。 # 拖住整轮),实测 count=2 全市场 24 台 2.95s / 只取最快 12 台 1.45s。
_BULK_MAX_CONNS = 12 _BULK_MAX_CONNS = 12
# 流水线失败的一批最多回队几次。回队 = 交给**其它健康连接**立刻接手;
# 上限保证全节点失联时不会无限打转,超限后落到尾部并发兜底。
_BULK_MAX_RETRY = 2
# 交易时段(分钟):上午 9:30-11:30、下午 13:00-15:00,共 240 分钟 # 交易时段(分钟):上午 9:30-11:30、下午 13:00-15:00,共 240 分钟
_AM_OPEN, _AM_CLOSE = 9 * 60 + 30, 11 * 60 + 30 _AM_OPEN, _AM_CLOSE = 9 * 60 + 30, 11 * 60 + 30
_PM_OPEN, _PM_CLOSE = 13 * 60, 15 * 60 _PM_OPEN, _PM_CLOSE = 13 * 60, 15 * 60
@@ -973,6 +976,7 @@ class TdxData:
by_text = {s.text: s for s in syms} by_text = {s.text: s for s in syms}
acc: dict[str, list[SecurityBar]] = {} acc: dict[str, list[SecurityBar]] = {}
retry: dict[str, Symbol] = {} retry: dict[str, Symbol] = {}
attempts: dict[str, int] = {}
async def worker(pc) -> None: async def worker(pc) -> None:
broken = False broken = False
@@ -985,10 +989,22 @@ class TdxData:
try: try:
res = await pc.execute_pipeline(cmds) res = await pc.execute_pipeline(cmds)
except Exception: except Exception:
# 流水线中途失败 = 响应流已错配,连接作废、整批交回逐只重试 # 流水线中途失败 = 响应流已错配,连接作废。
# 这一批**立刻放回队首**:其它健康连接会马上接手,
# 不必等 gather 结束再串行补(老实现每条重试都要重新
# 付一次 RTT,几百只就是十几秒的长尾)。
broken = True broken = True
for s, _, _ in batch: again: list[tuple[Symbol, int, int]] = []
for item in batch:
s = item[0]
n = attempts.get(s.text, 0)
if n < _BULK_MAX_RETRY:
attempts[s.text] = n + 1
again.append(item)
else:
retry[s.text] = s retry[s.text] = s
if again:
tasks.extendleft(reversed(again))
return return
for (s, _, _), bars in zip(batch, res): for (s, _, _), bars in zip(batch, res):
acc.setdefault(s.text, []).extend(bars) acc.setdefault(s.text, []).extend(bars)
@@ -996,19 +1012,27 @@ class TdxData:
await pool.release(pc, broken=broken) await pool.release(pc, broken=broken)
await asyncio.gather(*(worker(pc) for pc in held)) await asyncio.gather(*(worker(pc) for pc in held))
# 连接全挂时 worker 会提前 return,队列里剩下的标的没人取 —— 一并交回逐只, # 连接全挂时 worker 会提前 return,队列里剩下的标的没人取 —— 一并交回重试,
# 否则会**静默少数据**(实测 20 只只回 16 只)。 # 否则会**静默少数据**(实测 20 只只回 16 只)。
while tasks: while tasks:
s, _, _ = tasks.popleft() s, _, _ = tasks.popleft()
retry.setdefault(s.text, s) retry.setdefault(s.text, s)
out = {k: self._finish_bars(v, by_text[k]) for k, v in acc.items()} out = {k: self._finish_bars(v, by_text[k]) for k, v in acc.items()}
for text, s in retry.items(): if retry:
# 尾部兜底也并发做:坏节点把批次吐回来时可能有几十上百只,
# 串行重试等于把「一台机器卡住」放大成「整批卡住」。
sem = asyncio.Semaphore(max(2, min(len(retry), pool.max_size)))
async def rescue(text: str, s: Symbol) -> None:
async with sem:
try: try:
out[text] = await self._bars_one( out[text] = await self._bars_one(
s, period, start=start, count=count, adjust=adj, s, period, start=start, count=count, adjust=adj,
start_date=None, end_date=None) start_date=None, end_date=None)
except Exception: except Exception:
out.setdefault(text, []) out.setdefault(text, [])
await asyncio.gather(*(rescue(t, s) for t, s in retry.items()))
return out return out
async def _bars_one( async def _bars_one(
@@ -1053,28 +1077,43 @@ class TdxData:
async def _bars_ex(self, sym: Symbol, period: KlinePeriod, adj: Adjust, async def _bars_ex(self, sym: Symbol, period: KlinePeriod, adj: Adjust,
start: int, count: int) -> list[SecurityBar]: start: int, count: int) -> list[SecurityBar]:
"""扩展市场K线(EX 0x23FF;category=周期编码,与 TDXParams 一致)。""" """扩展市场K线(EX 0x23FF;category=周期编码,与 TDXParams 一致)。"""
from .protocol.commands.ex_proto import GetExInstrumentBarsCmd from .protocol.commands.ex_proto import GetExInstrumentBarsCmd, _EX_AMOUNT_MARKETS
cat = int(period) cat = int(period)
fetch_count = max(700, start + count) fetch_count = max(700, start + count)
# extra 带的 @v2 是缓存 schema 标记:v2 起 EX K线开始回填 amount/settle, # extra 带的 @v3 是缓存 schema 标记:v3 起期货/期权回填估算 amount + position;
# 旧版本写下的缓存里 amount 恒 0.0,必须让它们自然失效。 # 旧版本写下的缓存里 amount 恒 0.0、position 丢失,必须让它们自然失效。
key = CacheKey("kline", sym.market, sym.code, period.name, start, key = CacheKey("kline", sym.market, sym.code, period.name, start,
fetch_count, f"{adj.name}@{cat}@v2") fetch_count, f"{adj.name}@{cat}@v3")
# 期货/期权乘数(估算成交额用);未知品种乘数为 None → amount 保持 0.0
from .ex_multiplier import find_multiplier
from .symbols import _EX_PREFIX
mult = None
if sym.market not in _EX_AMOUNT_MARKETS:
mult = find_multiplier(_EX_PREFIX.get(sym.market, ""), sym.code, sym.market)
async def fetch() -> bytes: async def fetch() -> bytes:
rows = await self._fetch_paginated_ex( rows = await self._fetch_paginated_ex(
lambda s2, c2: GetExInstrumentBarsCmd(sym.market, sym.code, cat, s2, c2), lambda s2, c2: GetExInstrumentBarsCmd(sym.market, sym.code, cat, s2, c2),
PageSpec(page_size=700, total=fetch_count, min_page=100)) PageSpec(page_size=700, total=fetch_count, min_page=100))
bars = [SecurityBar( bars = []
for d in rows:
vol = float(d.get("trade", 0) or 0)
amount = float(d.get("amount", 0.0) or 0.0)
if amount == 0.0 and mult and vol:
# 期货/期权:帧内无成交额,按收盘价估算(见 ex_multiplier 模块 docstring)
amount = vol * float(d["close"]) * mult
pos = d.get("position", 0) or 0
bars.append(SecurityBar(
market=sym.market, code=sym.code, market=sym.market, code=sym.code,
year=d["year"], month=d["month"], day=d["day"], year=d["year"], month=d["month"], day=d["day"],
hour=d.get("hour", 0), minute=d.get("minute", 0), hour=d.get("hour", 0), minute=d.get("minute", 0),
open=d["open"], high=d["high"], low=d["low"], close=d["close"], open=d["open"], high=d["high"], low=d["low"], close=d["close"],
vol=float(d.get("trade", 0) or 0), vol=vol,
# 权益类(港股/美股/新三板)真有成交额;期货期权的帧里没有,恒 0.0 amount=amount,
amount=float(d.get("amount", 0.0) or 0.0),
settle=d.get("settle"), settle=d.get("settle"),
) for d in rows] # 权益类市场 position 恒 0 → None;期货/期权给真实持仓量
position=float(pos) if pos else None,
))
return pack_payload(bars) return pack_payload(bars)
bars = self._finish_bars(unpack_payload(await self._cache.get_or_fetch(key, fetch)), sym) bars = self._finish_bars(unpack_payload(await self._cache.get_or_fetch(key, fetch)), sym)
+106
View File
@@ -0,0 +1,106 @@
"""EX 期货/期权合约乘数表(估算成交额用)。
0x23FF 帧内没有成交额槽位(见 ex_proto.py 注释)。要给期货/期权 K 线一个
可用的 ``amount``,只能客户端自算。本表覆盖主流境内品种,键为**品种代码
小写字母前缀**(从合约代码提取:``RB2611`` → ``rb``,``IFL0`` → ``if``)。
⚠️ 口径:``amount = vol × close × multiplier`` —— 按**收盘价**估算,
不是 Σ(逐笔成交价×手数×乘数) 的精确值,日线级别误差通常 <2%(收盘价
成交集中假设)。对多数策略足够;要精确成交额请用逐笔聚合。
"""
from __future__ import annotations
# 中金所(乘数:点×元)
_CFFEX = {
"if": 200, "ih": 300, "ic": 200, "im": 200,
"ifl0": 200, "ihl0": 300, "icl0": 200, "iml0": 200,
"tf": 10_000, "t": 10_000, "ts": 10_000, "tl": 10_000, "tp": 10_000,
"mo": 50, "ho": 50, "io": 100,
}
# 上期所
_SHF = {
"rb": 10, "hc": 10, "wr": 10, "cu": 5, "al": 5, "zn": 5, "pb": 5,
"ni": 1, "sn": 1, "ss": 5, "ao": 20, "au": 100, "ag": 15,
"fu": 10, "bu": 10, "ru": 10, "br": 5, "sp": 10, "lu": 10, "nr": 10,
"sc": 1000, "bc": 5, "ec": 50, "sh": 1_000_000, "dii": 100,
}
# 大商所
_DL = {
"a": 10, "b": 10, "m": 10, "y": 10, "p": 10, "c": 10, "cs": 20,
"i": 100, "j": 100, "jm": 60, "jd": 5, "fb": 10, "bb": 500,
"l": 5, "v": 5, "pp": 5, "eg": 10, "eb": 5, "pg": 20, "rr": 10,
"lh": 16, "px": 10, "lg": 50,
}
# 郑商所
_ZZ = {
"sr": 10, "cf": 5, "cy": 5, "ap": 10, "cj": 5, "pj": 10,
"ta": 5, "ma": 10, "fg": 20, "sa": 20, "ur": 20, "sf": 5, "sm": 5,
"oi": 10, "rm": 10, "zc": 100, "wh": 20, "pm": 50, "ri": 20, "jr": 20,
"lr": 20, "pf": 5, "sh": 30, "pk": 5, "cft": 5,
}
# 广期所
_GZ = {"si": 5, "lc": 20, "ps": 5}
_MARKET_TABLES = {
"cffex": _CFFEX,
"shf": _SHF,
"dl": _DL,
"zz": _ZZ,
"gz": _GZ,
}
# 期权乘数:金融期权按标的;商品期权 = 标的期货乘数(同一品种字母前缀命中即可)。
# 沪深股票期权(50ETF/300ETF/500ETF/科创50ETF/创业板ETF)乘数 10000。
_STOCK_OPTION_MULTIPLIER = 10_000
def product_code(code: str) -> str:
"""合约代码 -> 品种小写前缀(``RB2611``→``rb``、``IFL0``→``ifl0``、``m2501``→``m``)。
规则:取前缀直到首个数字;连续/主力后缀(L0、M0 等)保留在品种键里,
表中给主力键与普通键都录入(见 _CFFEX 的 IFL0/IF 双键)。
"""
s = code.strip().lower()
out = []
for ch in s:
if ch.isdigit():
break
out.append(ch)
return "".join(out)
def find_multiplier(ex_prefix: str, code: str, market: int | None = None) -> float | None:
"""(EX 市场短前缀, 合约代码[, 市场号]) -> 合约乘数;未知返回 None。
期权市场按**市场号**判断(IntEnum 名含 OPTION):股票期权乘数 10000,
商品期权(ZZ/DL/CFFEX 期权市场号 4/5/7)回落到对应商品品种表。
"""
if market is not None:
name = _OPTION_MARKET_NAMES.get(int(market))
if name == "stock":
return _STOCK_OPTION_MULTIPLIER
if name == "commodity":
ex_prefix = {"ZZ_FUTURES_OPTION": "zz", "DL_FUTURES_OPTION": "dl",
"CFFEX_OPTION": "cffex", "GZ_OPTIONS": "gz"}.get(name, ex_prefix)
s = ex_prefix.lower()
if s.endswith("_option") or s in ("opt", "option"):
return _STOCK_OPTION_MULTIPLIER
table = _MARKET_TABLES.get(s)
if table is None:
return None
prod = product_code(code)
if prod in table:
return float(table[prod])
# 主力/连续后缀剥离后再试(IFL0 → IF)
stripped = prod.rstrip("lmo0")
return table.get(stripped)
_OPTION_MARKET_NAMES = {
4: "commodity", # ZZ_FUTURES_OPTION
5: "commodity", # DL_FUTURES_OPTION
7: "commodity", # CFFEX_OPTION
67: "commodity", # GZ_OPTIONS
8: "stock", # SH_STOCK_OPTION
9: "stock", # SZ_STOCK_OPTION
}
+7
View File
@@ -139,6 +139,9 @@ class SecurityBar:
symbol: str = "" # 规范化标的串(sz000001 / hk00700),由 API 层填充 symbol: str = "" # 规范化标的串(sz000001 / hk00700),由 API 层填充
# 结算价。仅 EX 通道的**期货/期权**有(日线才有值,分钟线为 None);其余通道恒 None。 # 结算价。仅 EX 通道的**期货/期权**有(日线才有值,分钟线为 None);其余通道恒 None。
settle: float | None = None settle: float | None = None
# 持仓量(手)。仅 EX 通道的**期货/期权**有(0x23FF 帧槽位 5);其余通道恒 None。
# 权益类市场(港股/美股等)无此概念,归一化为 None。
position: float | None = None
def __setstate__(self, state: Any) -> None: def __setstate__(self, state: Any) -> None:
"""反序列化兜底:早期版本写下的磁盘缓存里没有后加的可选字段(如 ``settle``)。 """反序列化兜底:早期版本写下的磁盘缓存里没有后加的可选字段(如 ``settle``)。
@@ -216,6 +219,10 @@ class SecurityQuote:
A股(标准通道):五档 ``bid``/``ask`` + 买卖量 + 涨跌停价齐全,``name`` 为空。 A股(标准通道):五档 ``bid``/``ask`` + 买卖量 + 涨跌停价齐全,``name`` 为空。
A股(``fields=[...]`` 走 MAC 0x122B):``name`` 与 ``fields`` 齐全,五档为空。 A股(``fields=[...]`` 走 MAC 0x122B):``name`` 与 ``fields`` 齐全,五档为空。
扩展市场(港股/美股/期货):``name`` + 基础价量在属性上,其余字段位在 ``fields``。 扩展市场(港股/美股/期货):``name`` + 基础价量在属性上,其余字段位在 ``fields``。
``server_time`` 是**行情快照时间**:``HH:MM:SS.ff``,小数位数等于该品种价格小数位
(股票 2 / ETF·LOF·B股 3 / 国债·回购 4),通常比本地时钟慢 5~15 秒。**指数恒为空串**
(协议里这个字段对指数不是时钟);判断行情是否更新看它有没有变,别看调用时刻。
""" """
market: int market: int
code: str code: str
+52 -21
View File
@@ -37,6 +37,9 @@ class ExPoolTimeoutError(ConnectionError):
class ExPool: class ExPool:
"""EX通道连接池(港股/期货,端口7727,连接时自动login)。""" """EX通道连接池(港股/期货,端口7727,连接时自动login)。"""
FAIL_COOLDOWN = 30.0
"""握手失败后拉黑该主机的秒数(EX 节点少,冷却比标准池短)。"""
def __init__(self, servers: list[str], *, timeout: float = 10.0, def __init__(self, servers: list[str], *, timeout: float = 10.0,
min_size: int = 2, max_size: int = 6, min_size: int = 2, max_size: int = 6,
per_server_cap: int = 1, per_server_cap: int = 1,
@@ -58,6 +61,9 @@ class ExPool:
self._total = 0 self._total = 0
self._closed = False self._closed = False
self._cond = asyncio.Condition() self._cond = asyncio.Condition()
# 死节点短期拉黑:一台 EX 主机握手失败后,后续 acquire 直接绕开它,
# 而不是每次都再付一遍 ``self.timeout`` 的握手超时。
self._fail_until: dict[str, float] = {}
async def start(self) -> None: async def start(self) -> None:
async with self._cond: async with self._cond:
@@ -72,6 +78,7 @@ class ExPool:
self._server_count[host] = self._server_count.get(host, 0) + 1 self._server_count[host] = self._server_count.get(host, 0) + 1
except Exception as e: except Exception as e:
log.warning("EX连接失败 %s: %s", host, str(e)[:60]) log.warning("EX连接失败 %s: %s", host, str(e)[:60])
self._fail_until[host] = time.monotonic() + self.FAIL_COOLDOWN
async def close(self) -> None: async def close(self) -> None:
self._closed = True self._closed = True
@@ -94,8 +101,11 @@ class ExPool:
if timeout is None: if timeout is None:
timeout = self.acquire_timeout timeout = self.acquire_timeout
deadline = None if timeout is None else time.monotonic() + timeout deadline = None if timeout is None else time.monotonic() + timeout
async with self._cond: avoid: set[str] = set() # 本次 acquire 里已经握手失败、别再试的主机
while True: while True:
new_conn = None
new_host = None
async with self._cond:
# 1. 优先借用空闲连接(过期则丢弃,槽位在锁内回收) # 1. 优先借用空闲连接(过期则丢弃,槽位在锁内回收)
if self._idle: if self._idle:
_, conn = self._idle.pop() _, conn = self._idle.pop()
@@ -107,28 +117,22 @@ class ExPool:
self._server_count[conn.host] = max(0, self._server_count.get(conn.host, 0) - 1) self._server_count[conn.host] = max(0, self._server_count.get(conn.host, 0) - 1)
self._schedule_close(conn) self._schedule_close(conn)
# 2. 可扩容 → 锁内预留槽位,锁外建连 # 2. 可扩容 → 锁内预留槽位,锁外建连(不能持锁做网络 IO)
new_conn = None
new_host = None
if self._total < self.max_size and self.servers: if self._total < self.max_size and self.servers:
host = self._next_available_server() host = self._next_available_server(avoid)
if host is not None: if host is not None:
self._total += 1 self._total += 1
self._server_count[host] = self._server_count.get(host, 0) + 1 self._server_count[host] = self._server_count.get(host, 0) + 1
new_conn = ExTdxConnection(host, 7727, self.timeout, self.handshake_map.get(host, self.handshake)) new_conn = ExTdxConnection(
host, 7727, self.timeout,
self.handshake_map.get(host, self.handshake))
new_host = host new_host = host
if new_conn is not None: elif avoid and all(h in avoid for h in self.servers):
# 锁外网络 IO:连接失败回滚槽位后继续等待 # 所有主机本次都试过了、全失败:等下去也不会有结果
try: raise ExPoolTimeoutError(self, timeout)
await new_conn.connect()
return new_conn
except Exception as e:
log.warning("EX扩容失败 %s: %s", new_host, str(e)[:60])
async with self._cond:
self._total = max(0, self._total - 1)
self._server_count[new_host] = max(0, self._server_count.get(new_host, 0) - 1)
# 3. 池满:排队等待 release 唤醒(或超时) # 3. 没抢到槽位 → 在锁内排队等 release 唤醒(waiter 必须持锁)
if new_conn is None:
now = time.monotonic() now = time.monotonic()
if deadline is not None and now >= deadline: if deadline is not None and now >= deadline:
raise ExPoolTimeoutError(self, timeout) raise ExPoolTimeoutError(self, timeout)
@@ -138,13 +142,40 @@ class ExPool:
else: else:
await asyncio.wait_for(self._cond.wait(), timeout=deadline - now) await asyncio.wait_for(self._cond.wait(), timeout=deadline - now)
except asyncio.TimeoutError: except asyncio.TimeoutError:
raise ExPoolTimeoutError(self, timeout) raise ExPoolTimeoutError(self, timeout) from None
continue
def _next_available_server(self) -> str | None: # ---- 锁外建连 ----
try:
await new_conn.connect()
async with self._cond:
self._fail_until.pop(new_host, None)
return new_conn
except Exception as e:
log.warning("EX扩容失败 %s: %s", new_host, str(e)[:60])
avoid.add(new_host) # type: ignore[arg-type]
async with self._cond:
self._total = max(0, self._total - 1)
self._server_count[new_host] = max(0, self._server_count.get(new_host, 0) - 1)
self._fail_until[new_host] = time.monotonic() + self.FAIL_COOLDOWN
self._cond.notify_all()
continue
def _next_available_server(self, avoid: set[str] | None = None) -> str | None:
"""挑一台**未达并发上限**的主机:优先未冷却的,冷却中的只在别无选择时用。"""
avoid = avoid or set()
fallback: str | None = None
now = time.monotonic()
for h in self.servers: for h in self.servers:
if self._server_count.get(h, 0) < self.per_server_cap: if h in avoid:
continue
if self._server_count.get(h, 0) >= self.per_server_cap:
continue
if now >= self._fail_until.get(h, 0.0):
return h return h
return None if fallback is None:
fallback = h
return fallback
def _schedule_close(self, conn: ExTdxConnection) -> None: def _schedule_close(self, conn: ExTdxConnection) -> None:
async def _close(): async def _close():
+121 -37
View File
@@ -75,6 +75,15 @@ class PooledConnection:
class DynamicPool: class DynamicPool:
"""动态连接池:按服务器维度统计连接数,每台服务器最多 per_server_cap 个连接。""" """动态连接池:按服务器维度统计连接数,每台服务器最多 per_server_cap 个连接。"""
BLACKOUT_COOLDOWN = 2.0
"""全主机失败后的熔断时长(秒)。
全部服务器同时不可用时,每次 :meth:`acquire` 都会把每台主机握手一遍再放弃。
上层是「按标的逐只重试」的循环,6000 只标的 × 16 台主机 × 握手耗时 = 几分钟
的空转。熔断期内直接抛 :class:`PoolTimeoutError`,让调用方快速走完并返回,
冷却结束再试一次探测全灭是否恢复。
"""
def __init__(self, servers: list[str], *, port: int = DEFAULT_PORT, def __init__(self, servers: list[str], *, port: int = DEFAULT_PORT,
timeout: float = 10.0, min_size: int = 4, max_size: int = 12, timeout: float = 10.0, min_size: int = 4, max_size: int = 12,
per_server_cap: int = 1, per_server_cap: int = 1,
@@ -104,6 +113,7 @@ class DynamicPool:
self._total = 0 self._total = 0
self._closed = False self._closed = False
self._started = False self._started = False
self._blackout_until = 0.0 # 全灭熔断截止时刻(monotonic)
self._lock = asyncio.Lock() self._lock = asyncio.Lock()
self._cond = asyncio.Condition(self._lock) self._cond = asyncio.Condition(self._lock)
self._reaper_task: asyncio.Task | None = None self._reaper_task: asyncio.Task | None = None
@@ -127,10 +137,12 @@ class DynamicPool:
break break
try: try:
pc = await self._new_conn(host) pc = await self._new_conn(host)
self._idle.append(pc)
self._total += 1
except Exception as e: except Exception as e:
log.warning("初始连接失败 %s: %s", host, str(e)[:60]) log.warning("初始连接失败 %s: %s", host, str(e)[:60])
continue
async with self._lock:
self._register_locked(pc)
self._idle.append(pc)
if self._total == 0: if self._total == 0:
log.warning("连接池初始连接全部失败,将在使用时按需重试") log.warning("连接池初始连接全部失败,将在使用时按需重试")
@@ -160,6 +172,12 @@ class DynamicPool:
return out return out
async def _new_conn(self, host: str) -> PooledConnection: async def _new_conn(self, host: str) -> PooledConnection:
"""建立一条连接(**纯 IO,不改任何计数**)。
计数由调用方在锁内用 :meth:`_register_locked` / :meth:`_reserve_locked`
单独完成——建连必须能在池锁之外进行,否则一台坏服务器的握手超时
会把所有并发协程一起冻住。
"""
conn = TdxConnection(host, self.port, self.timeout) conn = TdxConnection(host, self.port, self.timeout)
await conn.connect() await conn.connect()
if self.probe_factory is not None: if self.probe_factory is not None:
@@ -168,12 +186,28 @@ class DynamicPool:
except Exception: except Exception:
await conn.close() await conn.close()
raise raise
pc = PooledConnection(conn, host) return PooledConnection(conn, host)
def _register_locked(self, pc: PooledConnection) -> None:
"""登记一条已建好的连接(调用方需持锁)。"""
self._conns.add(pc) self._conns.add(pc)
self._server_count[host] = self._server_count.get(host, 0) + 1 self._server_count[pc.host] = self._server_count.get(pc.host, 0) + 1
self._total += 1
self.stats.total = self._total
self.stats.created += 1 self.stats.created += 1
self.stats.total = self._total + 1
return pc def _reserve_locked(self, host: str) -> None:
"""锁内**预留**一个槽位(真正建连在锁外做)。"""
self._total += 1
self._server_count[host] = self._server_count.get(host, 0) + 1
self.stats.total = self._total
def _unreserve_locked(self, host: str) -> None:
"""撤销预留(建连失败时回滚计数)。"""
self._total = max(0, self._total - 1)
if self._server_count.get(host, 0) > 0:
self._server_count[host] -= 1
self.stats.total = self._total
async def close(self) -> None: async def close(self) -> None:
self._closed = True self._closed = True
@@ -208,10 +242,19 @@ class DynamicPool:
单连接约束下,池大小 = 服务器数 × per_server_cap。当并发请求超过池大小时, 单连接约束下,池大小 = 服务器数 × per_server_cap。当并发请求超过池大小时,
超额协程在此**排队等待**(基于 Condition),待其他协程 release 后被唤醒, 超额协程在此**排队等待**(基于 Condition),待其他协程 release 后被唤醒,
而非立即超时放弃。默认 timeout=30s 防止在坏服务器场景下永久挂起。 而非立即超时放弃。默认 timeout=30s 防止在坏服务器场景下永久挂起。
**建连在池锁之外**:锁内只预留槽位,TCP 握手 + 探针放到锁外做。否则
一台坏服务器握手超时(最长 ``self.timeout``)期间,所有其它协程的
acquire/release 全部被这个锁挡住——5 台坏节点能把整批任务拖慢十几秒。
""" """
deadline = None if timeout is None else (time.monotonic() + timeout) deadline = None if timeout is None else (time.monotonic() + timeout)
async with self._cond: avoid: set[str] = set() # 本次 acquire 里已握手失败的主机,别再重复试
while True: while True:
if time.monotonic() < self._blackout_until:
raise PoolTimeoutError(
"所有服务器均不可用(熔断中,稍后自动重试)")
host = ""
async with self._cond:
now = time.monotonic() now = time.monotonic()
# 1. 优先借用空闲连接(跳过冷却中的服务器,避免反复选中坏节点) # 1. 优先借用空闲连接(跳过冷却中的服务器,避免反复选中坏节点)
while self._idle: while self._idle:
@@ -222,35 +265,65 @@ class DynamicPool:
return pc return pc
self._drop_locked(pc) self._drop_locked(pc)
# 2. 可扩容(未达上限且有可用服务器)→ 新建连接 # 2. 全坏场景下这里必须能退出:扩容失败分支会 continue,
if self._total < self.max_size: # 没有这一行就会无限重试、永远不会抛 PoolTimeoutError。
host = self._next_available_server()
if host is None:
# 兜底:全部服务器都在冷却中,而池子未满 —— 此时等 release
# 永远等不到(没人持有),只能挑最健康的一台顶着冷却重试
host = self._next_available_server(ignore_cooldown=True)
if host:
try:
pc = await self._new_conn(host)
self._total += 1
self.stats.borrowed += 1
return pc
except Exception as e:
log.warning("扩容连接失败 %s: %s", host, str(e)[:60])
self.health.record_failure(host)
# 循环重试:可能换下一台服务器
continue
# 3. 池满:排队等待 release 唤醒(或超时)
if deadline is not None and now >= deadline: if deadline is not None and now >= deadline:
raise PoolTimeoutError("连接池获取超时(连接数已达上限且无空闲)") raise PoolTimeoutError("连接池获取超时(连接数已达上限且无空闲)")
# 3. 可扩容 → 锁内**只预留槽位**
if self._total < self.max_size:
host = self._next_available_server(avoid=avoid) or ""
if not host:
# 兜底:全部服务器都在冷却中,而池子未满 —— 此时等 release
# 永远等不到(没人持有),只能挑最健康的一台顶着冷却重试
host = self._next_available_server(
ignore_cooldown=True, avoid=avoid) or ""
if host:
self._reserve_locked(host)
elif avoid and all(h in avoid for h in self.servers):
# 本轮每台主机都握手失败过了 → 再等下去也不会有连接。
# 同时熔断一小段时间,避免上层「逐只重试」把握手放大成
# 成千上万次(实测 6000 标的能空转 60s+)。
self._blackout_until = now + self.BLACKOUT_COOLDOWN
raise PoolTimeoutError(
f"所有服务器均不可用({len(avoid)} 台均已失败)")
if not host:
# 4. 池满/无可用服务器:排队等 release 唤醒(或超时)
remaining = None if deadline is None else deadline - now
try: try:
if deadline is None: if remaining is None:
await self._cond.wait() await self._cond.wait()
else: else:
await asyncio.wait_for(self._cond.wait(), timeout=deadline - now) await asyncio.wait_for(self._cond.wait(), timeout=remaining)
except asyncio.TimeoutError: except asyncio.TimeoutError:
raise PoolTimeoutError("连接池获取超时(连接数已达上限且无空闲)") raise PoolTimeoutError(
"连接池获取超时(连接数已达上限且无空闲)") from None
continue
# ---- 锁外建连:失败只影响本次请求,不冻结其它协程 ----
try:
pc = await self._new_conn(host)
except Exception as e:
avoid.add(host)
async with self._cond:
self._unreserve_locked(host)
self._cond.notify_all() # 槽位已回滚,唤醒排队者
self.health.record_failure(host)
log.warning("扩容连接失败 %s: %s", host, str(e)[:60])
continue
if self._closed:
await pc.close()
async with self._cond:
self._unreserve_locked(host)
self._cond.notify_all()
raise PoolTimeoutError("连接池已关闭")
# 槽位已在锁内预留,这里只补登记,不重复计数
self._conns.add(pc)
self.stats.created += 1
self.stats.borrowed += 1
self._blackout_until = 0.0 # 有连接成功 → 立刻解除熔断
return pc
async def acquire_many(self, k: int) -> list[PooledConnection]: async def acquire_many(self, k: int) -> list[PooledConnection]:
"""批量借出 k 个连接(分页并发用),超额请求排队等待不丢。""" """批量借出 k 个连接(分页并发用),超额请求排队等待不丢。"""
@@ -262,8 +335,11 @@ class DynamicPool:
break break
return out return out
def _next_available_server(self, *, ignore_cooldown: bool = False) -> str | None: def _next_available_server(self, *, ignore_cooldown: bool = False,
ranked = self.health.rank(self.servers) avoid: set[str] | None = None) -> str | None:
"""挑一台主机(``avoid`` 里的跳过,用于同一次 acquire 内不重复试错)。"""
avoid = avoid or set()
ranked = self.health.rank([h for h in self.servers if h not in avoid])
# 1. 优先选未冷却且未达并发上限的服务器 # 1. 优先选未冷却且未达并发上限的服务器
for h in ranked: for h in ranked:
if self.health.in_cooldown(h) and not ignore_cooldown: if self.health.in_cooldown(h) and not ignore_cooldown:
@@ -311,12 +387,20 @@ class DynamicPool:
results = await asyncio.gather( results = await asyncio.gather(
*(self._new_conn(h) for h in hosts), return_exceptions=True) *(self._new_conn(h) for h in hosts), return_exceptions=True)
async with self._lock: async with self._lock:
for pc in results: for host, res in zip(hosts, results):
if isinstance(pc, PooledConnection): if not isinstance(res, PooledConnection):
self._total += 1 # 建连失败 → 记一次失败,后续 acquire 会把它排到队尾
self.stats.total = self._total self.health.record_failure(host)
if not self._closed and pc.alive: continue
pc: PooledConnection = res
if self._closed or not pc.alive or self._total >= self.max_size:
# 建好却发现池已满/已关闭:必须关掉,否则连接泄漏,
# 而且 _total 会被虚增(老实现就在这儿漏计数)
asyncio.create_task(self._safe_close(pc))
continue
self._register_locked(pc)
self._idle.append(pc) self._idle.append(pc)
self._cond.notify_all()
async def release(self, pc: PooledConnection, *, broken: bool = False) -> None: async def release(self, pc: PooledConnection, *, broken: bool = False) -> None:
if self.stats.borrowed > 0: if self.stats.borrowed > 0:
+49 -11
View File
@@ -5,6 +5,7 @@ import struct
from ..._binary import unpack_from from ..._binary import unpack_from
from ...models import SecurityQuote from ...models import SecurityQuote
from ...symbols import is_cn_index
from ..price import get_price from ..price import get_price
from ..volume import get_volume from ..volume import get_volume
from .base import BaseCommand from .base import BaseCommand
@@ -161,13 +162,49 @@ def price_decimal_digits(market: int, code: str) -> int:
return _decimal_digits_static(m, code) return _decimal_digits_static(m, code)
def format_server_time(raw: int) -> str: _SERVER_TIME_SKEW = 40
"""服务器时间编码:小时 + 百万分之一小时。14999212 -> 14:59:57.163""" """服务端在「50~59 秒」段会写成「90~99」秒,整数值固定多 40×10**d。"""
hours, fractional_hour = divmod(raw, 1_000_000)
total_millis = fractional_hour * 3600 // 1000
minutes, remainder = divmod(total_millis, 60_000) def format_server_time(raw: int, digits: int = 2) -> str:
seconds, millis = divmod(remainder, 1000) """报价快照时间解码:十进制数字拼接 ``HH MM SS`` + ``digits`` 位小数。
return f"{hours:02d}:{minutes:02d}:{seconds:02d}.{millis:03d}"
编码(2026-10-30 盘中实测确定)::
整数 = HH*10**(4+d) + MM*10**(2+d) + SS*10**d + 小数
d = 该品种**价格小数位**(股票 2 / ETF·LOF·B股 3 / 国债·回购 4)
实测样例:``13224002``(d=2)→ ``13:22:40.02``;``132245903``(d=3)→
``13:22:45.903``;``1310228050``(d=4)→ ``13:10:22.8050``。
历史错误(0.8.0 修正):此前按「小时 + 百万分之一小时」解码 —— 用
``divmod(raw, 1_000_000)`` 取小时、余数按每秒 3600/1e6 折算。股票会把
11:29:59 解成 **11:17:57**(基准错了、速率也只有真实时钟的 53%),
ETF / 债券更会解出 ``113:00:21.222`` 这种小时 > 23 的非法值。
⚠️ 服务端怪癖:**SS 落在 50~59 秒时写成 90~99**(编码值 +40 秒)。实测
3 分钟连续采样,未修正时该区间的滞后会从稳定的 10.6 秒突然变成 -29.4 秒
(正好差 40)。因此这里对 ``SS >= 60`` 一律减 40 秒还原。
⚠️ 指数(``sh000001`` / ``sz399001`` …)该字段**不是**这只时钟:解出来
MM 常 > 59、且不同指数互相矛盾(同一时刻 000001 是 12:89、399001 是 12:15),
协议里没有给出可用信息。这类返回值一律为空串,不要编造时间;要指数时间
请用 :meth:`~pytdxdata.api.TdxData.get_minutes` 或 K 线。
"""
if raw <= 0:
return ""
d = digits if 0 <= digits <= 6 else 2
scale = 10 ** d
frac = raw % scale
rest = raw // scale
seconds = rest % 100
minutes = (rest // 100) % 100
hours = rest // 10000
if seconds >= 60:
seconds -= _SERVER_TIME_SKEW
if hours > 23 or minutes > 59 or not 0 <= seconds <= 59:
return ""
return f"{hours:02d}:{minutes:02d}:{seconds:02d}.{frac:0{d}d}"
class GetSecurityQuotesCmd(BaseCommand[list[SecurityQuote]]): class GetSecurityQuotesCmd(BaseCommand[list[SecurityQuote]]):
@@ -235,13 +272,14 @@ class GetSecurityQuotesCmd(BaseCommand[list[SecurityQuote]]):
pos += 4 pos += 4
code = code_b.decode("utf-8").rstrip("\x00") code = code_b.decode("utf-8").rstrip("\x00")
divisor = 10 ** price_decimal_digits(market_b, code) dp = price_decimal_digits(market_b, code)
divisor = 10 ** dp
p = price_raw / divisor p = price_raw / divisor
results.append(SecurityQuote( results.append(SecurityQuote(
market=market_b, code=code, price=p, market=market_b, code=code, price=p,
cur_vol=float(cur_vol), cur_vol=float(cur_vol),
rise_speed=_rise_speed / 100.0, rise_speed=_rise_speed / 100.0,
decimal_point=price_decimal_digits(market_b, code), decimal_point=dp,
open_amount=float(_unknown_3), open_amount=float(_unknown_3),
pre_close=(price_raw + last_close_diff) / divisor, pre_close=(price_raw + last_close_diff) / divisor,
open=(price_raw + open_diff) / divisor, open=(price_raw + open_diff) / divisor,
@@ -250,8 +288,8 @@ class GetSecurityQuotesCmd(BaseCommand[list[SecurityQuote]]):
vol=float(vol), amount=amount, s_vol=float(s_vol), b_vol=float(b_vol), vol=float(vol), amount=amount, s_vol=float(s_vol), b_vol=float(b_vol),
bid=[((price_raw + d) / divisor, v) for d, v in bid], bid=[((price_raw + d) / divisor, v) for d, v in bid],
ask=[((price_raw + d) / divisor, v) for d, v in ask], ask=[((price_raw + d) / divisor, v) for d, v in ask],
server_time=format_server_time(unknown_0), server_time="" if is_cn_index(market_b, code)
else format_server_time(unknown_0, dp),
trading_status=trading_status, trading_status=trading_status,
)) ))
return results return results
return results
+64 -6
View File
@@ -10,6 +10,7 @@ from __future__ import annotations
import asyncio import asyncio
import logging import logging
from collections import deque
from dataclasses import dataclass from dataclasses import dataclass
from typing import Any, Callable from typing import Any, Callable
@@ -81,15 +82,53 @@ class Paginator:
stop = asyncio.Event() stop = asyncio.Event()
out: list[PageResult] = [] out: list[PageResult] = []
out_lock = asyncio.Lock() out_lock = asyncio.Lock()
# 静态切片保证「每台连接都干活」(并发摊开),另备一个共享的
# ``leftover``:某个 worker 的连接坏掉退出时,它**没做完的页**会
# 放回这里,由还活着的 worker 接着做。老实现没有这层——一个
# worker 挂掉,它那一整份页就没人做了,要么整轮报错、要么静默缺页。
leftover: deque[tuple[int, int]] = deque()
q_lock = asyncio.Lock()
k = len(slots) k = len(slots)
workers = [ workers = [
asyncio.create_task( asyncio.create_task(
self._worker(slots[wid], jobs[wid::k], cmd_factory, self._worker(slots[wid], jobs[wid::k], leftover, q_lock,
stop, out, out_lock, spec) cmd_factory, stop, out, out_lock, spec)
) )
for wid in range(k) for wid in range(k)
] ]
await asyncio.gather(*workers) await asyncio.gather(*workers)
# 收尾补页:所有 worker 都退出后 leftover 里可能还剩页(连接全坏时会
# 发生)。再借一条**新连接**顺序把它们补完,最多两轮——否则 skip 的页
# 就变成静默缺数据(strict=False 时最危险:不报错、只是少几页)。
rounds = 0
while leftover and not stop.is_set() and rounds < 2:
rounds += 1
try:
pc = await self.pool.acquire(timeout=5.0)
except PoolTimeoutError:
break
slot = _Slot(pc)
progressed = False
try:
while leftover and not stop.is_set():
start, count = leftover[0] # 成功后才出队,失败可重试
items, ok = await self._fetch_one(
slot, cmd_factory(start, count), count)
if not ok:
break
leftover.popleft()
progressed = True
async with out_lock:
out.append(PageResult(start=start, count=count, items=items))
if spec.stop_when_short and len(items) < count:
stop.set()
finally:
if slot.pc is not None:
await self.pool.release(slot.pc)
if not progressed:
break
if leftover:
log.warning("分页调度:%d 页最终未取到(strict=False 留空)", len(leftover))
finally: finally:
for s in slots: for s in slots:
if s.pc is not None: if s.pc is not None:
@@ -117,21 +156,40 @@ class Paginator:
return jobs return jobs
async def _worker( async def _worker(
self, slot: _Slot, jobs: list[tuple[int, int]], self, slot: _Slot,
jobs: list[tuple[int, int]],
leftover: "deque[tuple[int, int]]",
q_lock: asyncio.Lock,
cmd_factory: Callable[[int, int], Any], stop: asyncio.Event, cmd_factory: Callable[[int, int], Any], stop: asyncio.Event,
out: list[PageResult], out_lock: asyncio.Lock, spec: PageSpec, out: list[PageResult], out_lock: asyncio.Lock, spec: PageSpec,
) -> None: ) -> None:
for start, count in jobs: i = 0
if stop.is_set() or slot.pc is None: try:
while True:
if i < len(jobs):
start, count = jobs[i]
i += 1
else:
# 自己的份额做完了 → 看看有没有别人掉下的活
async with q_lock:
if stop.is_set() or not leftover:
return
start, count = leftover.popleft()
if slot.pc is None or stop.is_set():
return return
items, ok = await self._fetch_one(slot, cmd_factory(start, count), count) items, ok = await self._fetch_one(slot, cmd_factory(start, count), count)
if not ok: if not ok:
return # 换连接失败 → 本 worker 退出(该页留空) return # 换连接失败 → 本 worker 退出(剩余页在 finally 里交还)
async with out_lock: async with out_lock:
out.append(PageResult(start=start, count=count, items=items)) out.append(PageResult(start=start, count=count, items=items))
if spec.stop_when_short and len(items) < count: if spec.stop_when_short and len(items) < count:
stop.set() stop.set()
return return
finally:
rest = jobs[i:]
if rest:
async with q_lock:
leftover.extendleft(reversed(rest))
async def _fetch_one(self, slot: _Slot, cmd, page_size: int): async def _fetch_one(self, slot: _Slot, cmd, page_size: int):
"""单页拉取,失败换连接重试。 """单页拉取,失败换连接重试。
+46 -1
View File
@@ -132,6 +132,7 @@ async def test_bars_ex_backfills_amount_for_hk() -> None:
assert bars[0].amount == pytest.approx(2_040_000.0) assert bars[0].amount == pytest.approx(2_040_000.0)
assert bars[0].vol == 2000 assert bars[0].vol == 2000
assert bars[0].settle is None assert bars[0].settle is None
assert bars[0].position is None # 权益类无持仓量
@pytest.mark.asyncio @pytest.mark.asyncio
@@ -140,7 +141,9 @@ async def test_bars_ex_backfills_settle_for_futures() -> None:
bars = await td.get_bars("cffex_futures:IFL0", count=1) bars = await td.get_bars("cffex_futures:IFL0", count=1)
assert bars[0].settle == pytest.approx(4047.8) assert bars[0].settle == pytest.approx(4047.8)
assert bars[0].amount == 0.0 # v3 起期货按收盘价估算成交额:close 10.2 × vol 25392 × IF 乘数 200
assert bars[0].amount == pytest.approx(10.2 * 25392 * 200)
assert bars[0].position == 49_772.0
# ---------- 旧磁盘缓存兼容 ---------- # ---------- 旧磁盘缓存兼容 ----------
@@ -173,3 +176,45 @@ def test_ex_symbols_route_to_ex_path() -> None:
"""守住前提:上面对 31/47 的断言确实走的是 EX 通道。""" """守住前提:上面对 31/47 的断言确实走的是 EX 通道。"""
assert parse_symbols(["hk00700"])[0].is_ex assert parse_symbols(["hk00700"])[0].is_ex
assert parse_symbols(["cffex_futures:IFL0"])[0].is_ex assert parse_symbols(["cffex_futures:IFL0"])[0].is_ex
# ---------- v3:期权持仓量 + 乘数表 ----------
def test_multiplier_known_products() -> None:
from pytdxdata.ex_multiplier import find_multiplier
assert find_multiplier("cffex", "IFL0") == 200
assert find_multiplier("shf", "rb2611") == 10
assert find_multiplier("shf", "au2611") == 100
assert find_multiplier("dl", "i2501") == 100
assert find_multiplier("zz", "AP0") == 10
def test_multiplier_unknown_returns_none() -> None:
from pytdxdata.ex_multiplier import find_multiplier
assert find_multiplier("shf", "xyz999") is None
assert find_multiplier("nosuch", "rb2611") is None
def test_stock_option_multiplier() -> None:
from pytdxdata.ex_multiplier import find_multiplier
assert find_multiplier("sh_stock_option", "10011425") == 10_000
@pytest.mark.asyncio
async def test_bars_ex_option_position_backfilled() -> None:
"""期权 K 线持仓量透传(用户主诉):协议槽位 5 已解析,v3 起到 SecurityBar。"""
td = _fake_api({8: [_row(8, "10011425", position=123_456, trade=6373, settle=0.0887)]})
bars = await td.get_bars("sh_stock_option:10011425", count=1)
assert bars[0].position == 123_456.0
# 股票期权乘数 10000:close 10.2 × vol 6373 × 10000
assert bars[0].amount == pytest.approx(10.2 * 6373 * 10_000)
@pytest.mark.asyncio
async def test_bars_ex_unknown_multiplier_keeps_amount_zero() -> None:
"""乘数表没有的品种:宁可不给 amount 也不编造。"""
td = _fake_api({30: [_row(30, "ZZZ2601", position=10, trade=5, settle=1.0)]})
bars = await td.get_bars("shf:ZZZ2601", count=1)
assert bars[0].amount == 0.0
assert bars[0].position == 10.0
+329
View File
@@ -0,0 +1,329 @@
"""坏节点隔离回归:**一台服务器中途死掉,不能拖慢整批任务**。
三件事必须成立(缺一个就会把「单机故障」放大成「整批卡死」):
1. **建连不能持池锁** —— 死节点握手要等满 ``timeout``,锁内做会把所有并发
协程一起冻住。
2. **全节点失联要能退出** —— 扩容失败分支 ``continue`` 会无限重试,
没有截止检查就永远不抛 ``PoolTimeoutError``。
3. **失败批次立刻回队 / 分页份额交给活着的 worker** —— 否则要么串行重试
把长尾放大,要么那一整份数据没人做(静默缺页)。
替身全部在进程内,不连真实网络。
"""
from __future__ import annotations
import asyncio
import datetime
import time
import pytest
from pytdxdata import TdxData
from pytdxdata.models import KlinePeriod, SecurityBar
from pytdxdata.pool.pool import DynamicPool, PoolTimeoutError
from pytdxdata.scheduler.paginator import PageSpec, Paginator
NEWEST = datetime.datetime.now().replace(microsecond=0) # noqa: DTZ001
# ====================================================================== #
# 一、建连不占池锁 + 全灭可退出
# ====================================================================== #
class _HostConn:
"""按 host 决定握手行为的替身连接。
``slow_hosts`` 里的主机握手会「卡住」``delay`` 秒后失败(模拟 SYN 被丢弃);
``dead`` 为真时连握手都不做,立刻拒绝。
"""
slow_hosts: dict[str, float] = {}
dead: bool = False
connects: list[str] = []
def __init__(self, host, port=7709, timeout=10.0):
self.host = host
self._alive = True
type(self).connects.append(host)
@property
def alive(self) -> bool:
return self._alive
async def connect(self) -> None:
delay = type(self).slow_hosts.get(self.host)
if type(self).dead:
raise ConnectionRefusedError(f"{self.host} 拒绝连接")
if delay is not None:
await asyncio.sleep(delay)
raise ConnectionRefusedError(f"{self.host} 握手超时")
async def execute(self, cmd):
return "ok"
async def close(self) -> None:
self._alive = False
@pytest.fixture
def conns(monkeypatch):
_HostConn.slow_hosts = {}
_HostConn.dead = False
_HostConn.connects = []
monkeypatch.setattr("pytdxdata.pool.pool.TdxConnection", _HostConn)
return _HostConn
@pytest.mark.asyncio
async def test_slow_connect_does_not_freeze_other_borrowers(conns):
"""一台主机握手卡 0.5s 时,另一个协程借连接不能跟着等。
老实现把 ``_new_conn`` 放在池锁里:所有 acquire/release 都被这 0.5s 挡住,
几台坏节点叠加就是十几秒的全局停顿。
"""
conns.slow_hosts = {"dead": 0.5}
pool = DynamicPool(["dead", "live"], min_size=0, max_size=2,
per_server_cap=1, auto_probe=False)
await pool.start()
slow = asyncio.create_task(pool.acquire()) # 抢到 "dead",开始卡握手
await asyncio.sleep(0.05) # 确保它已进入 _new_conn
t0 = time.monotonic()
fast_pc = await pool.acquire() # 此时应立刻拿到 "live"
elapsed = time.monotonic() - t0
assert elapsed < 0.25, f"被坏节点的握手阻塞了 {elapsed:.2f}s"
assert fast_pc.host == "live"
slow_pc = await slow # 换到 "live" 后成功
assert slow_pc.host == "live"
for pc in (fast_pc, slow_pc):
await pool.release(pc)
await pool.close()
@pytest.mark.asyncio
async def test_acquire_gives_up_when_every_host_is_dead(conns):
"""全节点失联时必须抛 PoolTimeoutError,而不是无限重试。"""
conns.dead = True
pool = DynamicPool(["d1", "d2"], min_size=0, max_size=4,
per_server_cap=1, auto_probe=False)
await pool.start()
t0 = time.monotonic()
with pytest.raises(PoolTimeoutError):
# 外层 wait_for 只是防止老实现挂死把测试卡住
await asyncio.wait_for(pool.acquire(timeout=0.3), timeout=5.0)
assert time.monotonic() - t0 < 2.0
await pool.close()
@pytest.mark.asyncio
async def test_blackout_stops_retrying_every_host_per_call(conns):
"""全灭后进入熔断期:后续 acquire 立刻失败,不再把每台主机握手一遍。
没有熔断时,上层「按标的逐只重试」的循环会把 16 台 × 6000 只放大成
9 万次握手(实测 6000 只跑 60s 都跑不完)。
"""
conns.dead = True
pool = DynamicPool(["d1", "d2", "d3"], min_size=0, max_size=8,
per_server_cap=1, auto_probe=False)
await pool.start()
tried = len(conns.connects) # start() 的 min_size 预热(可能已试过 1 台)
with pytest.raises(PoolTimeoutError):
await asyncio.wait_for(pool.acquire(timeout=1.0), timeout=5.0)
round1 = len(conns.connects) - tried
assert round1 == 3, f"首轮应恰好试完 3 台,实际 {round1}"
tried = len(conns.connects)
# 熔断期内:不再产生任何建连尝试,且立即抛错
t0 = time.monotonic()
for _ in range(50):
with pytest.raises(PoolTimeoutError):
await pool.acquire()
assert time.monotonic() - t0 < 0.2
assert len(conns.connects) == tried, "熔断期内不该再握手"
# 熔断到期后恢复探测
pool._blackout_until = 0.0
with pytest.raises(PoolTimeoutError):
await pool.acquire()
assert len(conns.connects) > tried
await pool.close()
# ====================================================================== #
# 二、K 线批量:死节点只损失在途那一批,且不重复打扰
# ====================================================================== #
class _FleetConn:
"""MAC 通道替身:``m1`` 在成功服务 ``m1_life`` 次后「宕机」。"""
m1_life = 2
m1_calls = 0
m2_calls = 0
def __init__(self, host="m1", port=7709, timeout=10.0):
self.host = host
self._alive = True
@property
def alive(self) -> bool:
return self._alive
async def connect(self) -> None:
pass
async def close(self) -> None:
self._alive = False
def _dead(self) -> bool:
cls = type(self)
if self.host != "m1":
cls.m2_calls += 1
return False
cls.m1_calls += 1
return cls.m1_calls > cls.m1_life
def _rows(self, cmd):
if "MacKlineCmd" not in type(cmd).__name__:
return []
out = []
for off in range(cmd.start, cmd.start + cmd.count):
when = NEWEST - datetime.timedelta(minutes=off)
out.append(SecurityBar(
market=1, code="600000", open=10.0, high=10.1, low=9.9,
close=10.0, vol=1000, amount=10000, year=when.year,
month=when.month, day=when.day, hour=when.hour,
minute=when.minute))
return out
async def execute(self, cmd):
if self._dead():
raise ConnectionError("m1 已宕机")
return self._rows(cmd)
async def execute_pipeline(self, cmds, **kwargs):
if self._dead():
raise ConnectionError("m1 已宕机")
return [self._rows(c) for c in cmds]
def _syms(n: int) -> list[str]:
return [f"sh60{i:04d}" for i in range(n)]
@pytest.mark.asyncio
async def test_bulk_bars_recovers_from_midrun_host_death(monkeypatch):
"""30 只K线,m1 服务 2 次后宕机:数据一根不少,且不反复打扰死节点。"""
_FleetConn.m1_life = 2
_FleetConn.m1_calls = 0
_FleetConn.m2_calls = 0
monkeypatch.setattr("pytdxdata.api.get_mac_hosts", lambda: ["m1", "m2"])
monkeypatch.setattr("pytdxdata.pool.pool.TdxConnection",
lambda h, p=7709, t=3.0: _FleetConn(h))
td = TdxData(cache_dir=None, pool_min=1, pool_max=4)
await td.start()
t0 = time.monotonic()
try:
bars = await td.get_bars(_syms(30), KlinePeriod.MIN_1, count=240)
finally:
elapsed = time.monotonic() - t0
await td.close()
assert len(bars) == 30 * 240, "死节点不能造成缺数据"
# 死节点只该被「在途那一批」碰到,之后靠冷却隔离,不该被反复重试
assert _FleetConn.m1_calls <= 2 * 4, f"死节点被打了 {_FleetConn.m1_calls} 次"
assert _FleetConn.m2_calls > 0
assert elapsed < 5.0, f"单台服务器宕机把整批拖到 {elapsed:.2f}s"
@pytest.mark.asyncio
async def test_all_hosts_dead_fails_fast(monkeypatch):
"""全节点失联:**快速报错**,不能挂死,也不能每只都付满一次 30s 池超时。
老实现里扩容失败分支 ``continue`` 且没有截止检查 → ``acquire`` 无限重试,
十万次建连把事件循环饿死(协程连取消都投递不进去)。
"""
_FleetConn.m1_life = 0
_FleetConn.m1_calls = 0
_FleetConn.m2_calls = 0
monkeypatch.setattr("pytdxdata.api.get_mac_hosts", lambda: ["m1"])
monkeypatch.setattr("pytdxdata.pool.pool.TdxConnection",
lambda h, p=7709, t=3.0: _FleetConn(h))
td = TdxData(cache_dir=None, pool_min=1, pool_max=2)
await td.start()
t0 = time.monotonic()
try:
with pytest.raises(PoolTimeoutError):
await asyncio.wait_for(
td.get_bars(_syms(10), KlinePeriod.MIN_1, count=240), timeout=15.0)
finally:
elapsed = time.monotonic() - t0
await td.close()
assert elapsed < 5.0, f"全灭场景花了 {elapsed:.2f}s 才报错"
# 每台主机只该试一次,不该疯狂重连
assert _FleetConn.m1_calls < 40, f"重复建连 {_FleetConn.m1_calls} 次"
# ====================================================================== #
# 三、分页器的份额再分配
# ====================================================================== #
class _PageConn:
def __init__(self, die_after: int | None = None, page_size: int = 100,
total: int = 600):
self.die_after = die_after
self.served = 0
self.page_size = page_size
self.total = total
async def execute(self, cmd):
start, count = cmd
if self.die_after is not None and self.served >= self.die_after:
raise ConnectionError("连接已断")
self.served += 1
return list(range(start, min(start + count, self.total)))
class _PagePool:
def __init__(self, conns):
self.conns = conns
self._i = 0
self.released = []
async def acquire_many(self, n, *, timeout=None):
return self.conns[:n]
async def acquire(self, *, timeout=None):
pc = self.conns[self._i % len(self.conns)]
self._i += 1
return pc
async def release(self, pc, *, broken=False):
self.released.append(broken)
@pytest.mark.asyncio
async def test_paginator_redistributes_pages_of_dead_worker():
"""1 号连接拉到第 2 页就断:它剩下的页要由其它连接补完,不能整份丢掉。"""
conns = [_PageConn(die_after=2), _PageConn(), _PageConn()]
pag = Paginator(_PagePool(conns), concurrency=3) # type: ignore[arg-type]
spec = PageSpec(page_size=100, total=600) # 6 页,每 worker 2 页
got = await pag.fetch_all(lambda s, c: (s, c), spec)
assert len(got) == 600, f"丢了 {600 - len(got)} 条"
@pytest.mark.asyncio
async def test_paginator_still_distributes_work_across_conns():
"""份额再分配不能退化成「一条连接干完所有页」。"""
conns = [_PageConn() for _ in range(3)]
pag = Paginator(_PagePool(conns), concurrency=3) # type: ignore[arg-type]
spec = PageSpec(page_size=100, total=600)
got = await pag.fetch_all(lambda s, c: (s, c), spec)
assert len(got) == 600
assert all(c.served >= 1 for c in conns), "每台连接都该分到活"
+2 -2
View File
@@ -99,7 +99,7 @@ def test_bar_replaces_split_time_with_datetime(backend):
df = to_frame([bar(), bar(minute=30)], backend=backend) df = to_frame([bar(), bar(minute=30)], backend=backend)
assert cols(df, backend) == [ assert cols(df, backend) == [
"market", "code", "open", "high", "low", "close", "vol", "amount", "market", "code", "open", "high", "low", "close", "vol", "amount",
"datetime", "trading_date", "symbol", "settle", "datetime", "trading_date", "symbol", "settle", "position",
] ]
assert rows(df, backend) == 2 assert rows(df, backend) == 2
assert cell(df, backend, "market") == 1 assert cell(df, backend, "market") == 1
@@ -122,7 +122,7 @@ def test_empty_list_falls_back_to_annotation_hint(backend):
assert rows(df, backend) == 0 assert rows(df, backend) == 0
assert cols(df, backend) == [ assert cols(df, backend) == [
"market", "code", "open", "high", "low", "close", "vol", "amount", "market", "code", "open", "high", "low", "close", "vol", "amount",
"datetime", "trading_date", "symbol", "settle", "datetime", "trading_date", "symbol", "settle", "position",
] ]
+106
View File
@@ -0,0 +1,106 @@
"""报价快照时间字段(0x053E 第 6 个变长字段)解码口径。
2026-10-30 修正前按「小时 + 百万分之一小时」解码,股票 11:29:59 会解成
11:17:57、ETF 会解出 `113:00:21` 这种非法小时。这里把实测口径锁死:
十进制位拼接 `HH MM SS` + 价格小数位,SS 的 50~59 段服务端写成 90~99。
"""
import struct
from pytdxdata.protocol.commands.quotes import (
GetSecurityQuotesCmd,
format_server_time,
price_decimal_digits,
)
from pytdxdata.protocol.price import put_price
# ---------- 纯解码 ----------
def test_stock_two_digits():
# 13:22:40.02(股票价格 2 位小数 → 时间带 2 位小数)
assert format_server_time(13224002, 2) == "13:22:40.02"
def test_etf_three_digits():
# 13:22:45.903(ETF 3 位小数)
assert format_server_time(132245903, 3) == "13:22:45.903"
def test_bond_four_digits():
# 13:10:22.8050(国债 / 回购 4 位小数)
assert format_server_time(1310228050, 4) == "13:10:22.8050"
def test_seconds_50_59_written_as_90_99():
# 服务端把 52 秒写成 92 → 必须还原成 13:17:52.01
assert format_server_time(13179201, 2) == "13:17:52.01"
assert format_server_time(13179501, 2) == "13:17:55.01"
assert format_server_time(13179801, 2) == "13:17:58.01"
# 90 以下原样(不要误伤 51、59 这类合法秒)
assert format_server_time(13175101, 2) == "13:17:51.01"
assert format_server_time(13175901, 2) == "13:17:59.01"
def test_index_and_garbage_return_empty():
# 指数该字段不是时钟(同一时刻 000001 解出 12:89 → MM 越界)
assert format_server_time(12867393, 2) == ""
assert format_server_time(0, 2) == ""
# 小时越界
assert format_server_time(113005895, 2) == ""
def test_digits_follow_decimal_point():
assert price_decimal_digits(1, "600000") == 2
assert price_decimal_digits(1, "510300") == 3
assert price_decimal_digits(1, "019547") == 4
# ---------- 整条报文 ----------
def _quote_record(market: int, code: str, price_raw: int, time_raw: int,
vol: int = 1000, rise: int = 0) -> bytes:
"""按 0x053E 记录布局拼一条报价(金额字段填 0,不参与本节断言)。"""
out = bytearray(struct.pack("<B6sH", market, code.encode(), 0))
for v in (price_raw, 0, 0, 0, 0): # price / lastclose / open / high / low
out += put_price(v)
out += put_price(time_raw) # ← 快照时间
out += put_price(-price_raw) # unknown1(多数标的 = -price)
out += put_price(vol) # vol
out += put_price(1) # cur_vol
out += struct.pack("<I", 0) # amount(4 字节自定义浮点)
for v in (vol // 2, vol // 2, 0, 0): # s_vol / b_vol / unknown2 / unknown3
out += put_price(v)
for _ in range(5): # 五档:价差 0,量 1
for v in (0, 0, 1, 1):
out += put_price(v)
out += struct.pack("<H", 0) # trading_status
for _ in range(4):
out += put_price(0)
out += struct.pack("<hH", rise, 0) # rise_speed / active2
return bytes(out)
def _body(records: list[bytes]) -> bytes:
# 前 2 字节是服务端魔数(解析时跳过),随后是 2 字节条数
return b"\x00\x00" + struct.pack("<H", len(records)) + b"".join(records)
def test_parse_response_sets_server_time():
body = _body([
_quote_record(1, "600000", 996, 13224002),
_quote_record(1, "510300", 4105, 132245903),
])
quotes = GetSecurityQuotesCmd([(1, "600000"), (1, "510300")]).parse_response(body)
assert [q.server_time for q in quotes] == ["13:22:40.02", "13:22:45.903"]
# 价格口径未受影响
assert [round(q.price, 3) for q in quotes] == [9.96, 4.105]
assert [q.decimal_point for q in quotes] == [2, 3]
def test_parse_response_blank_time_for_index():
body = _body([
_quote_record(1, "000001", 363808, 12867393),
_quote_record(1, "600000", 996, 13179201),
])
quotes = GetSecurityQuotesCmd([(1, "000001"), (1, "600000")]).parse_response(body)
assert quotes[0].server_time == ""
assert quotes[1].server_time == "13:17:52.01"
Generated
+1 -1
View File
@@ -810,7 +810,7 @@ wheels = [
[[package]] [[package]]
name = "pytdxdata" name = "pytdxdata"
version = "0.7.1" version = "0.7.2"
source = { editable = "." } source = { editable = "." }
[package.optional-dependencies] [package.optional-dependencies]