9 Commits
Author SHA1 Message Date
openbotandopenbot-coder ca24521d2c chore: bump version 0.6.4 (#5)
Co-authored-by: openbot-coder <openbot-coder@users.noreply.github.com>
2026-10-22 09:11:50 +08:00
openbotandopenbot-coder 8344ed6e6b fix: 修复沪市指数价格少10倍 + 逐笔竞价委托污染(Σvol 虚高/跨主机不稳定) (#3)
核对 sh000016 上证50 与 50 只成分股的 tick 汇总时,发现三个口径
(快照/分时/逐笔)互不一致,追查到 5 个缺陷。

1. 沪市指数价格整体少 10 倍
   price_decimal_digits 把沪市 000xxx(上证/上证50/沪深300/科创50)与
   8xxxxx(板块指数)判为 3 位小数,实际协议是 2 位。
   上证指数 3713 -> 371、上证50 2747 -> 274。
   实证 6 只指数 + 2 只板块指数全部 ×10,而 ETF/股票/科创板/深市全类 ×1 正确。
   改法:沪市仅 ETF(5开头) 保留 3 位,其余 2 位。

2. get_ticks 把 bs_flag=8 竞价委托当成成交
   该记录是 09:15~09:25 / 14:57~15:00 竞价段的撮合委托快照,
   vol 是累计值(可被后续覆盖),不是成交增量。
   - Σvol 虚高:sh600000 含 b8 = 668,693 vs 快照 462,562(1.4456);
     剔 b8 = 462,634(1.000156,精确一致)
   - 跨主机不稳定:干净主机 0 条、脏主机 135 条 -> 同一 start 落到不同真实成交
   改法:分页后统一剔除 b8。

3. A股逐笔改单连接串行游标(_ticks_cn_serial)
   并发分页各页来自不同主机,而 0x122F 的 start 是「从最新往回数」的相对锚点,
   主机间 b8 收录差异导致页边界既重叠(冗余 116 条)又漏条(13:03 附近丢 59 条)。
   逐笔单日仅 5~6 页,串行开销可忽略。

4. trade_count 有符号解析(前序)、返回列表时间序(前序)保留。

- sh000016 三口径闭合:快照 vol/amount 比值 1.00000061/0.99999997;
  指数逐笔 Σvol = 分时 Σvol = 1,045,584,467(完全相等)
- 6 只标的 Σvol/快照 vol 全部回到 1.0001~1.0007
- 跨主机公共时间窗逐条一致(5/5 标的,条数差 0)
- 测试 194 -> 217 passed(+20 小数位、+3 逐笔回归)

- TransactionRecord 类注释 + get_ticks docstring 明示指数语义:
  vol 单位「百元」(成交额/100)、trade_count 为「均价×1000」而非笔数
- API_REFERENCE / llms.txt 同步

Co-authored-by: openbot-coder <openbot-coder@users.noreply.github.com>
2026-10-22 09:06:40 +08:00
openbotandopenbot-coder dfd6fab3af fix: 补齐 v0.6.3 遗漏的版本号(__init__.py / uv.lock) (#4)
## 问题
v0.6.3 的 bump 提交只改了 pyproject.toml 一处,漏掉:
- src/pytdxdata/__init__.py 的 __version__
- uv.lock 的 pytdxdata 条目版本

后果(远端 main 已复现):
- CI 两个 job 全部失败:`uv sync --locked` 报
  "The lockfile at `uv.lock` needs to be updated, but `--locked` was provided"
- Docs / Release 同样失败
- v0.6.3 tag 已打,但 PyPI 上 0.6.3 仍是 404(发布没成功)

## 修复
补齐两处版本号,四处对齐:pyproject.toml / __init__.py / uv.lock / CHANGELOG.md

## 验证
- `uv sync --locked` 退出码 0
- `pytdxdata.__version__` = `importlib.metadata.version()` = 0.6.3

Co-authored-by: openbot-coder <openbot-coder@users.noreply.github.com>
2026-10-22 08:52:19 +08:00
openbot 0618062e54 chore: bump version 0.6.3 2026-10-18 16:12:51 +08:00
openbot 9c97fea71b fix(pool): ExPool acquire backpressure + ex_pool_cap (fixes #1)
- acquire(): replace 3x0.05s fast-fail with Condition-based queueing
  (default acquire_timeout=30s); timeout error carries pool state
  (idle/total/servers/per_server_cap)
- api: expose ex_pool_cap (default 2, aligned with std/mac pool caps);
  single-server scenario no longer silently capped at 1 concurrent
2026-10-18 16:10:15 +08:00
openbot-coder 2d62d82b51 feat: 新增3台EX扩展行情服务器(connect.cfg官方扩展市场,handshake=login),全量复测30/36可用 2026-10-16 13:53:19 +08:00
openbot-coder 7e72f70eb2 chore: 离线探针复测写回 servers.json(2026-10-16,可达25→27,117.34.114.13/.14 复活) 2026-10-16 12:51:12 +08:00
openbotandopenbot-coder b2f899df49 fix(models): 修复美股分钟K线 datetime 时区拼接错位,还原为北京时间 (#2)
通达信 EX 通道对美股返回的分钟级 K线把两个时区拼进同一个五元组:日期分量是
美东交易日(zipday),时分分量是北京时间(tminutes)。直拼出的 datetime 会把
同一交易日的下午半场排到上午半场之前——单日 390 根首末颠倒,且全部落在同一个
北京自然日里,导致排序 / count 切片 / 日期区间二分与过滤 / 增量水位线全部不可用。

在 SecurityBar.__post_init__ 统一归一化为北京时间(凌晨段日期 +1 天,含月末与
年末进位),EX 的两条 K线命令路径(_bars_ex 的 0x23FF 与二分所依赖的 MAC 0x122E)
因此共用同一口径,不会出现"取数按一套时间、二分按另一套时间"的错位。

新增两个只读属性:trading_date(归属交易日;美股一个交易日横跨两个北京自然日,
按交易日聚合必须用它,不能用 datetime.date())与 is_us。

实测(安装 wheel 0.6.2,非源码树):10-06 ~ 10-12 五个完整交易日各 390 根、
严格单调,首根 open / 末根 close 与日线逐位相同,成交量差 <= 1;A股 / 港股零回归。

- 版本 0.6.1 -> 0.6.2(pyproject / __init__ / uv.lock)
- tests/test_protocol.py 新增 9 项回归测试(分界点 / 月末年末进位 / 390 根整日单调性 / MAC 路径),全量 191 passed
- CHANGELOG / README / SKILL.md / docs(faq、llms.txt、api/models)同步口径说明

遗留(本版未含,待决策):EX 分时对美股返回的 MinuteBar.hour 是未取模的连续分钟数
(实测可达 27,即次日 03:xx),且 MinuteBar 无日期分量可承载跨日信息。

Co-authored-by: openbot-coder <openbot-coder@users.noreply.github.com>
2026-10-13 16:47:57 +08:00
openbot-coder 8193d560da Revert "fix(models): 修复美股分钟K线 datetime 时区拼接错位,还原为北京时间"
This reverts commit 5408903a4c.
2026-10-13 16:42:41 +08:00
16 changed files with 908 additions and 222 deletions

No files matched your search

+86
View File
@@ -2,6 +2,92 @@
本项目遵循 [语义化版本](https://semver.org/lang/zh-CN/)。
## [0.6.4] - 2026-10-22
### 修复
- **`get_ticks()` 返回列表不是时间序,每 `page_size` 处时间回跳一次**。
`0x122F`(A股逐笔)与 EX 逐笔通道的偏移量 `start` 都是**从最新一笔往回数**的:
实测 `sh600000`,`start=0` 返回 `14:07:48..15:20:16`(当天最新 1000 条),
`start=4000` 返回 `09:15:03..09:41:18`(当天最早)。而 `paginator` 按 `start`
升序拼接,于是拼出"新的块在前、旧的块在后"的倒序结构:
| | 修复前 | 修复后 |
|---|---|---|
| `sh600000` 时间轴 | 4 处逆序跳跃(index 999 / 1999 / 2999) | 严格升序 |
| `hk00700` 时间轴 | 21 处逆序跳跃(每 1800 条一次) | 严格升序 |
修复分两层:① `PageSpec.newest_first=True` 让 `paginator` 反页序拼装(页内本身升序,
只翻页序),这条能力保留给将来需要按"最新优先"锚点分页的命令;② A股逐笔最终改为
**单连接串行游标**(见下一条),从根上避开多主机锚点错位。
**`start` 参数语义同时明确**:它是"从最新往回数的偏移",不是"从最早往后数的偏移",
`docstring` 已补充。注意这修正的是**顺序**,数据本身无缺口、无重复(4304 条唯一时间戳)。
- **`TransactionRecord.trade_count` 在集合竞价段被读成 42 亿量级脏值**。
该字段在协议里是**有符号** `int32`,但解析格式串写的是无符号 `I`。集合竞价段
(`bs_flag=8`)真实值为负数,按 uint32 解出 `4294966422`:
```
原始 18 字节: 17820000 0ad71b41 d9000000 96fcffff 0800
修复前: trade_count = 4294966422
修复后: trade_count = -874 ← 0xfffffc96 的 int32 补码
```
实测 `sh600000` 全天 4304 条中有 **38 条**命中(占 0.88%),全部落在 09:15~09:25
集合竞价段;正常交易段(`bs_flag` 0/1/2/5)**无一条为负**。旧行为会让
`max()` / `mean()` 等统计直接失真(曾算出 `mean=37,920,247`)。
格式串由 `<IfIIH` 改为 `<IfIiH`。**注意负值语义尚未确证**(可能是撤单量/净委托),
目前只是如实保留符号,未做业务解释。
- **`get_quotes()` 沪市指数价格整体少 10 倍**。
`price_decimal_digits` 把沪市 `000`(如 000001/000016/000300/000688)与 `8`
开头(如板块指数 880001/880305)一律判成 3 位小数(除 1000),但协议原始整数
实际是 **2 位**(除 100)→ 上证指数 3713 被读成 371、上证50 2747 被读成 274。
实证(2026-10-21):沪市指数 6 只 + 板块 2 只全部 `分时中位价 / 快照价 = 10.0`,
而沪市 ETF(510300/588000/511010/501018)、股票(60/601)、科创板(688)、
深市指数/股票/创业板/ETF 全部 `= 1.0` 正确。修法是**把沪市 `000`/`8` 从 3 位改回 2 位**,
其余前缀(`5` ETF/基金、深市 `1` 基金转债)保持 3 位不变。回归测试覆盖 21 组代码。
- **`get_ticks()` 把集合竞价**委托**(`bs_flag=8`)当成成交计入,导致 Σvol 虚高且跨主机不稳定**。
`0x122F` 第 5 字段 `bs_flag=8` 的记录是 **09:15~09:25 / 14:57~15:00 竞价段的撮合委托
快照**,`vol` 是**累计值**(可被同段后续记录覆盖),根本不是成交增量。旧代码把它一并
返回,两个后果:
1. **Σvol 比真值虚高**:`sh600000` 全天 Σvol 含 b8 时 = 668,693,快照 vol = 462,562,
比值 1.4456;剔掉 b8 后 Σvol = 462,634,比值 **1.000156**(与快照精确一致)。
2. **跨主机不稳定**:干净主机不收录 b8(0 条)、脏主机收录 135 条。并发分页时各页
来自不同主机,同一 `start`(从最新往回数的相对锚点)在不同主机落到**不同的真实成交**
上 → 页边界既**重叠**(sh600000 冗余 116 条)又**漏条**(13:03 附近丢 59 条),
同一查询两次调用结果不同。
修复:① 分页拼装后**统一剔除 `bs_flag=8`**;② A股逐笔改用**单连接串行游标翻页**
(`_ticks_cn_serial`),彻底消除跨主机锚点错位。修复后单主机 1.0002、剔 b8 后跨主机
公共时间窗**逐条一致**。
- **文档化指数逐笔字段的量纲差异**(非 bug,是数据源固有语义)。
指数(如 `sh000016`)的 `vol` 单位是**百元**(成交额/100),`trade_count` 是**均价×1000**
(不是笔数),与个股完全不同。实测 `快照amount / Σtick.vol = 100.0002`、`tc/1000` 与分时
`avg_price` 差 < 1.1。已写进 `TransactionRecord` 类注释与 `get_ticks` docstring,
避免用户把指数 Σvol 直接和成分股成交量比(会误得 ~26 倍)。
### 说明
- 新增回归测试:价格小数位 ×20(19 组参数化 + 1 组去 NUL)、`get_ticks` 剔 b8 /
单连接串行 / 多页无重叠 ×3,测试总数 **194 → 217**。
## [0.6.3] - 2026-10-18
- fix(pool): ExPool acquire 背压等待(Condition 排队,默认 30s 超时)替代 3x0.05s 快速失败;超时异常带池状态
- feat(api): 新增 ex_pool_cap 参数(默认 2),单 EX 服务器场景不再静默退化为 1 并发(fixes #1)
## [0.6.2] - 2026-10-13
### 修复
+1 -1
View File
@@ -69,7 +69,7 @@ from pytdxdata import (
| 方法 | 返回 | 说明 |
|------|------|------|
| `get_ticks(symbols, *, date=None, start=0, count=None, concurrency=6)` | `list[TransactionRecord]` | 跨市场逐笔成交 |
| `get_ticks(symbols, *, date=None, start=0, count=None, concurrency=6)` | `list[TransactionRecord]` | 跨市场逐笔成交。返回按时间升序;`start` **从最新一笔往回数**;`vol`/`trade_count` 为约 3 秒切片的增量值(非累计)。**已剔除 `bs_flag=8` 竞价委托**(非成交,`vol` 为累计撮合量)。**指数**:`vol` 单位是百元(成交额/100)、`trade_count` 是均价×1000,均非股票语义 |
| `get_minutes(symbols, *, date=None, sampling=False, concurrency=6)` | `list[MinuteBar]` | 跨市场分时(sampling=True 取缩略采样:A股由 0x122D 派生,扩展市场 0x254D) |
#### 公司信息/文件/板块
+1 -1
View File
@@ -48,7 +48,7 @@ asyncio.run(main())
- **K 线**:`get_bars`(自动并发分页、支持复权与日期区间、多标的并发)
- **技术指标**:`get_indicators`(K线 + MACD/KDJ/RSI/BOLL/MA/EMA)
- **报价**:`get_quotes`(五档,>80 只自动切批)、`get_snapshot`(个股特征)、`get_market_stat`
- **逐笔 / 分时**:`get_ticks`(含 `trade_count`,支持历史日期)、`get_minutes`(`sampling=True` 取缩略采样)
- **逐笔 / 分时**:`get_ticks`(含 `trade_count`,支持历史日期,返回按时间升序;`vol`/`trade_count` 为约 3 秒切片增量而非累计;已剔除 `bs_flag=8` 竞价委托;**指数**的 `vol` 是百元、`trade_count` 是均价×1000)、`get_minutes`(`sampling=True` 取缩略采样)
- **财务 / 除权**:`get_finance`(最新快照)、`get_xdxr`、`get_price_limits`(本地计算涨跌停)
- **F10**:`get_f10`(公司资料目录 + 正文)
- **板块**:`get_boards`、`get_board_members`、`get_board_of`、`get_board_summary`、
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "pytdxdata"
version = "0.6.2"
version = "0.6.4"
description = "通达信(TongDaXin)行情数据客户端:动态连接池 + 双层TTL缓存 + 3连接并发分页,纯asyncio零运行时依赖,支持A股/港股/美股/期货/期权"
readme = "README.md"
requires-python = ">=3.11"
+1 -1
View File
@@ -16,7 +16,7 @@ from .models import (
XdxrRecord,
)
__version__ = "0.6.2"
__version__ = "0.6.4"
__all__ = [
"TdxData",
"Market",
+97 -22
View File
@@ -146,6 +146,8 @@ _EX_PAGE = 1000 # EX 商品列表单页
_TICK_PAGE = 1000 # MAC 逐笔单页
_EX_TICK_PAGE = 1800 # 港股逐笔单页
_ALL_PAGES_CAP = 50 # count=None(全量)时的翻页上限
# 集合竞价**委托**记录标记(0x122F 第 5 字段)。不是成交,见 _ticks_one 注释。
_BS_AUCTION_ORDER = 8
# 港股系市场:逐笔走 EX 专用 16B 协议(0x23FC/0x2406),其余扩展市场走 MAC 0x122F
_HK_MARKETS = frozenset({27, 31, 48, 49, 71, 98})
@@ -233,11 +235,13 @@ class TdxData:
pool_max: int = 12,
mac_pool_cap: int = 4,
std_pool_cap: int = 4,
ex_pool_cap: int = 2,
) -> None:
self.port = port
self.timeout = timeout
self.default_adjust = default_adjust
self.mac_pool_cap = mac_pool_cap
self.ex_pool_cap = ex_pool_cap
self._std_servers = list(standard_servers or get_standard_hosts())
self._mac_servers = list(mac_servers or get_mac_hosts())
self._ex_servers = list(ex_servers or get_ex_hosts())
@@ -262,8 +266,11 @@ class TdxData:
probe_factory=lambda: MacKlineCmd(1, "600000", KlinePeriod.DAY, Adjust.NONE, 0, 100),
)
# EX扩展市场池(港股/期货/美股,端口7727;MAC EX login + pytdx EX setup 混合)
# 单服务器场景(测速缓存收缩后仅 1 台)下,有效并发 = 服务器数 × ex_pool_cap,
# 默认 cap=2 保证至少 2 并发;acquire 有 Condition 背压等待,不再快速失败。
self._ex_pool = ExPool(self._ex_servers, timeout=timeout,
min_size=min(pool_min, 2), max_size=min(pool_max, 6),
per_server_cap=ex_pool_cap,
handshake_map=EX_HANDSHAKES)
disk = None
if self._cache_dir is not None:
@@ -790,12 +797,26 @@ class TdxData:
参数:
date: ``None``=当日,``YYYYMMDD``=历史某日。
start: 起始偏移。
start: 起始偏移,**从最新一笔往回数**(``0``=当天最后一条)。
``start=1000`` 表示跳过最近 1000 条、从更早的位置开始。
count: ``None``=尽量取全量(自动翻页,上限 50 页);给了就取这么多条。
返回:
``list[TransactionRecord]``,按入参标的顺序拼接(A股含 ``trade_count``,
港股为 HH:MM 时间 + ``fields["zengcang"]`` 增仓)。
``list[TransactionRecord]``,按入参标的顺序拼接,**每条标的内部按时间升序**。
注意服务端是按约 3 秒聚合的切片,``vol``/``trade_count`` 是切片增量
而非当日累计。
A股含 ``trade_count``,港股为 HH:MM 时间 + ``fields["zengcang"]`` 增仓。
**集合竞价委托(``bs_flag=8``)不会出现在结果里**:它们不是成交,``vol``
是累计撮合量(会被后续记录覆盖),计入会让 Σvol 比真值虚高 20%~160%,
且不同主机收录的条数不同 → 同一查询两次调用结果不一致。
**指数(``sh000016`` 等)的两个字段语义与个股不同**:
- ``vol`` 单位是**百元**(成交额/100),不是手。
- ``trade_count`` 是**当时均价 × 1000**,不是笔数。
详见 :class:`~pytdxdata.models.TransactionRecord` 的类注释。
"""
syms = parse_symbols(symbols)
sem = asyncio.Semaphore(max(1, concurrency))
@@ -816,18 +837,19 @@ class TdxData:
if sym.is_ex:
rows = await self._ticks_ex(sym, date=date, start=start, count=count)
else:
def make_cmd(s: int, c: int):
return MacTransactionCmd(sym.market, sym.code, date or 0, s, c)
spec = PageSpec(
page_size=_TICK_PAGE,
total=None if count is None else start + count,
min_page=100,
stop_when_short=True,
max_pages=_ALL_PAGES_CAP if count is None else None,
start_offset=start,
)
rows = await self._fetch_paginated_list("mac", make_cmd, spec)
rows = await self._ticks_cn_serial(sym, date=date, start=start,
count=count)
# ★ 过滤集合竞价**委托**记录(bs_flag=8)。必须在逐页拉取**全部结束之后**做:
# 分页靠 `len(rows) < page_size` 判终止,若在 parse_response 里过滤,
# 满页会被剥短成"短页" → 提前终止 → 丢数据。
# 实测(2026-10-21 sh600000):b8 是 09:15~09:25 / 14:57~15:00 竞价段的
# 撮合快照,vol 是**累计值**(可被后续记录覆盖),不是成交增量;
# 计入后 Σvol 比真值虚高 20%~160%,且**不同主机的 b8 集合不同**
# (干净主机 0 条、脏主机 135 条)→ 同一查询两次调用结果不一致。
# 剔除后:Σvol 与快照 vol 精确相等(比值 1.0000),且跨主机逐条一致。
rows = [r for r in rows if r.bs_flag != _BS_AUCTION_ORDER]
# 排序放在 pack 之前,让缓存直接存入正确顺序,避免每次命中都重排。
rows.sort(key=lambda r: r.time)
return pack_payload(rows)
records = unpack_payload(await self._cache.get_or_fetch(key, fetch))
@@ -835,6 +857,54 @@ class TdxData:
r.symbol = sym.text
return records
async def _ticks_cn_serial(self, sym: Symbol, *, date: int | None, start: int,
count: int | None) -> list[TransactionRecord]:
"""A股逐笔(0x122F)**单连接串行游标**翻页。
为什么不用并发分页器:``0x122F`` 的 ``start`` 是「从**最新一笔**往回数的偏移」,
是个**相对锚点**,不是稳定的绝对序号。而池里各主机对集合竞价委托(``bs_flag=8``)
的收录**不一致**(实测干净主机 0 条、脏主机 135 条),同一 ``start`` 在不同主机上
落到**不同的真实成交**上。并发分页 = 各页来自不同主机 → 页边界既**重叠**
(sh600000 冗余 116 条)又**漏条**(13:03 附近丢 59 条),Σvol 偏移 −1.2%。
逐笔单日仅 5~6 页(1000 条/页),串行开销可忽略、换来精确一致,
故这里固定**单连接**、按 ``start`` 递增(= 向更早方向)推进。
返回:按页序累加后再**整体翻转**,得到时间升序的完整列表。
"""
chunks: list[list[TransactionRecord]] = []
pos = start
remaining = count
pc = await self._mac_pool.acquire()
try:
for _ in range(_ALL_PAGES_CAP):
n = _TICK_PAGE if remaining is None else min(_TICK_PAGE, remaining)
if n <= 0:
break
rows = await pc.execute(
MacTransactionCmd(sym.market, sym.code, date or 0, pos, n))
if not rows:
break
chunks.append(list(rows))
pos += len(rows)
if remaining is not None:
remaining -= len(rows)
if remaining <= 0:
break
# 短页 = 已到最早一笔
if len(rows) < n:
break
except Exception:
await self._mac_pool.release(pc, broken=True)
raise
await self._mac_pool.release(pc)
out: list[TransactionRecord] = []
for chunk in reversed(chunks):
out.extend(chunk)
for r in out:
r.symbol = sym.text
return out
async def _ticks_ex(self, sym: Symbol, *, date: int | None, start: int,
count: int | None) -> list[TransactionRecord]:
from .protocol.commands.ex_ext import (
@@ -843,7 +913,7 @@ class TdxData:
)
hk = sym.market in _HK_MARKETS
page = _EX_TICK_PAGE if hk else _TICK_PAGE
out: list[TransactionRecord] = []
chunks: list[list[TransactionRecord]] = []
pos = start
remaining = count
for _ in range(_ALL_PAGES_CAP):
@@ -855,22 +925,27 @@ class TdxData:
if date else
GetExTransactionDataCmd(sym.market, sym.code, pos, n))
rows = await self._execute_ex(cmd)
out.extend(self._ex_tick(sym, r) for r in rows)
chunk = [self._ex_tick(sym, r) for r in rows]
else:
cmd = MacTransactionCmd(sym.market, sym.code, date or 0, pos, n)
rows = await self._execute_ex(cmd)
for r in rows:
r.symbol = sym.text
out.extend(rows)
if not rows:
chunk = list(rows)
if not chunk:
break
pos += len(rows)
# EX 通道的 pos 同样是"从最新往回数",逐页倒序累加 → 最后整体翻转。
chunks.append(chunk)
pos += len(chunk)
if remaining is not None:
remaining -= len(rows)
remaining -= len(chunk)
if remaining <= 0:
break
if len(rows) < n:
if len(chunk) < n:
break
out: list[TransactionRecord] = []
for chunk in reversed(chunks):
out.extend(chunk)
return out
@staticmethod
+24 -4
View File
@@ -228,14 +228,34 @@ class SecurityInfo:
@dataclass(slots=True)
class TransactionRecord:
"""逐笔成交(跨市场统一模型;A股 MAC 通道含 trade_count=成交笔数)"""
"""逐笔成交(跨市场统一模型;A股 MAC 通道含 trade_count=成交笔数)。
注意(实测 2026-10-21):
- 服务端是按**约 3 秒聚合**返回的切片,不是交易所原始逐笔成交。
- ``vol`` / ``trade_count`` 都是**该切片自身的增量**,不是当日累计值。
- ``trade_count`` 为**有符号**值:集合竞价段(``bs_flag=8``)会是负数
(实测 sh600000 09:15 段为 -874),不要对全表直接求 max/mean。
- 返回列表已按时间升序,但同秒多条时顺序不保证。
- ``bs_flag=8`` 的**竞价委托记录已被 ``get_ticks()`` 过滤掉**,不会出现在结果里。
**指数语义完全不同**(``is_cn_index`` 为真时,如 ``sh000016``):
- ``vol`` 单位是**百元**(= 成交额 / 100),不是「手」。实测
``快照amount / Σtick.vol = 100.0002``,且 ``Σtick.vol`` 与分时 Σvol **逐位相等**。
想和成分股成交量对比必须先换算,直接比会得到 ~26 倍的假比例。
- ``trade_count`` **不是笔数**,而是**当时均价 × 1000**(实测 tc/1000 与分时
``avg_price`` 的差 < 1.1)。指数没有「成交笔数」概念。
- ``bs_flag`` 语义不成立(实测返回 2,而模型只定义 0=卖 / 1=买)。
"""
market: int
code: str
time: str # HH:MM:SS(扩展市场为 HH:MM)
price: float
vol: float # 手(MAC逐笔)
trade_count: int # 成交笔数(MAC=逐笔记录字段;标准当日逐笔协议也有但需单独解析)
bs_flag: int # 1=主动买 0=主动卖
vol: float # 股票=手;指数=百元(成交额/100),见类注释
trade_count: int # 股票=笔数增量;指数=均价×1000,见类注释
bs_flag: int # 1=主动买 0=主动卖 2=中性 5=盘后固定价(指数语义不成立)
symbol: str = ""
fields: dict[str, float] = field(default_factory=dict) # 扩展市场额外字段(增仓等)
+89 -40
View File
@@ -1,4 +1,11 @@
"""EX扩展市场连接池(简化版:固定连接数,借还+坏连接重建+login自动)"""
"""EX扩展市场连接池(固定连接数,借还+坏连接重建+login自动)。
与标准池 pool.py 的背压语义对齐:acquire 在无空闲连接且无法扩容时,
基于 asyncio.Condition 排队等待 release 唤醒(默认 acquire_timeout=30s),
而不是快速失败。per_server_cap 限制单台服务器连接数;单服务器场景下
有效并发 = servers × per_server_cap(默认 1),通过 TdxData(ex_pool_cap=N)
可对同一 EX 主机开多连接。
"""
from __future__ import annotations
import asyncio
@@ -9,6 +16,23 @@ from .ex_connection import ExTdxConnection
log = logging.getLogger(__name__)
DEFAULT_ACQUIRE_TIMEOUT = 30.0
class ExPoolTimeoutError(ConnectionError):
"""acquire 等待超时(携带池状态快照,便于排障)。"""
def __init__(self, pool: "ExPool", timeout: float) -> None:
idle = len(pool._idle)
total = pool._total
servers = len(pool.servers)
cap = pool.per_server_cap
super().__init__(
f"EX连接池获取超时(等了 {timeout:.1f}s):"
f"idle={idle} total={total} servers={servers} per_server_cap={cap} "
f"有效并发={servers * cap}"
)
class ExPool:
"""EX通道连接池(港股/期货,端口7727,连接时自动login)。"""
@@ -16,6 +40,7 @@ class ExPool:
def __init__(self, servers: list[str], *, timeout: float = 10.0,
min_size: int = 2, max_size: int = 6,
per_server_cap: int = 1,
acquire_timeout: float | None = None,
handshake: str = "login",
handshake_map: dict | None = None) -> None:
self.servers = servers
@@ -25,14 +50,17 @@ class ExPool:
self.min_size = min_size
self.max_size = max_size
self.per_server_cap = max(1, per_server_cap) # 一台服务器最多几个连接(默认1=单连接)
self.acquire_timeout = (
DEFAULT_ACQUIRE_TIMEOUT if acquire_timeout is None else acquire_timeout
)
self._idle: list[tuple[str, ExTdxConnection]] = []
self._server_count: dict[str, int] = {}
self._total = 0
self._closed = False
self._lock = asyncio.Lock()
self._cond = asyncio.Condition()
async def start(self) -> None:
async with self._lock:
async with self._cond:
for host in self.servers[: self.min_size]:
if self._server_count.get(host, 0) >= self.per_server_cap:
continue
@@ -47,7 +75,7 @@ class ExPool:
async def close(self) -> None:
self._closed = True
async with self._lock:
async with self._cond:
for _, conn in self._idle:
try:
await conn.close()
@@ -56,48 +84,61 @@ class ExPool:
self._idle.clear()
self._total = 0
self._server_count.clear()
self._cond.notify_all()
async def acquire(self) -> ExTdxConnection:
for _ in range(3):
stale_conn = None
new_conn = None
new_host = None
async with self._lock:
async def acquire(self, timeout: float | None = None) -> ExTdxConnection:
"""借出一条连接;无空闲且无法扩容时排队等待 release 唤醒。
timeout 默认取 self.acquire_timeout(30s);传 None 表示无限等待。
"""
if timeout is None:
timeout = self.acquire_timeout
deadline = None if timeout is None else time.monotonic() + timeout
async with self._cond:
while True:
# 1. 优先借用空闲连接(过期则丢弃,槽位在锁内回收)
if self._idle:
_, conn = self._idle.pop()
fresh = time.monotonic() - conn.last_used < 30.0
if conn.alive and fresh:
return conn
# 连接空闲过久(login会话可能失效)→ 丢弃重连
# 连接空闲过久(login会话可能失效)→ 丢弃,槽位让给扩容
self._total = max(0, self._total - 1)
self._server_count[conn.host] = max(0, self._server_count.get(conn.host, 0) - 1)
stale_conn = conn
# 扩容:在锁内预留槽位,锁外执行网络IO
elif self._total < self.max_size and self.servers:
self._schedule_close(conn)
# 2. 可扩容 → 锁内预留槽位,锁外建连
new_conn = None
new_host = None
if self._total < self.max_size and self.servers:
host = self._next_available_server()
if host:
if host is not None:
self._total += 1
self._server_count[host] = self._server_count.get(host, 0) + 1
new_conn = ExTdxConnection(host, 7727, self.timeout, self.handshake_map.get(host, self.handshake))
new_host = host
# 锁外:关闭过期连接、建立新连接
if stale_conn is not None:
if new_conn is not None:
# 锁外网络 IO:连接失败回滚槽位后继续等待
try:
await new_conn.connect()
return new_conn
except Exception as e:
log.warning("EX扩容失败 %s: %s", new_host, str(e)[:60])
async with self._cond:
self._total = max(0, self._total - 1)
self._server_count[new_host] = max(0, self._server_count.get(new_host, 0) - 1)
# 3. 池满:排队等待 release 唤醒(或超时)
now = time.monotonic()
if deadline is not None and now >= deadline:
raise ExPoolTimeoutError(self, timeout)
try:
await stale_conn.close()
except Exception:
pass
if new_conn is not None:
try:
await new_conn.connect()
return new_conn
except Exception as e:
log.warning("EX扩容失败 %s: %s", new_host, str(e)[:60])
# 连接失败,回滚预留槽位
async with self._lock:
self._total = max(0, self._total - 1)
self._server_count[new_host] = max(0, self._server_count.get(new_host, 0) - 1)
await asyncio.sleep(0.05)
raise ConnectionError("EX连接池无可用连接")
if deadline is None:
await self._cond.wait()
else:
await asyncio.wait_for(self._cond.wait(), timeout=deadline - now)
except asyncio.TimeoutError:
raise ExPoolTimeoutError(self, timeout)
def _next_available_server(self) -> str | None:
for h in self.servers:
@@ -105,15 +146,23 @@ class ExPool:
return h
return None
async def release(self, conn: ExTdxConnection, *, broken: bool = False) -> None:
if broken or not conn.alive:
async with self._lock:
self._total = max(0, self._total - 1)
self._server_count[conn.host] = max(0, self._server_count.get(conn.host, 0) - 1)
def _schedule_close(self, conn: ExTdxConnection) -> None:
async def _close():
try:
await conn.close()
except Exception:
pass
return
async with self._lock:
self._idle.append((conn.host, conn))
try:
asyncio.get_running_loop().create_task(_close())
except RuntimeError:
pass
async def release(self, conn: ExTdxConnection, *, broken: bool = False) -> None:
async with self._cond:
if broken or not conn.alive:
self._total = max(0, self._total - 1)
self._server_count[conn.host] = max(0, self._server_count.get(conn.host, 0) - 1)
self._schedule_close(conn)
else:
self._idle.append((conn.host, conn))
self._cond.notify_all()
+13 -6
View File
@@ -13,16 +13,23 @@ MAX_QUOTES = 80
def price_decimal_digits(market: int, code: str) -> int:
"""推断报价有效小数位:股票2位(/100)、指数/ETF/基金/转债/国债3位(/1000)。"""
"""推断报价有效小数位:股票 2 位(/100)、ETF/基金 3 位(/1000)。
**沪市指数必须是 2 位**,此前把 ``000``/``8`` 一并判成 3 位,导致上证50、
上证指数、科创50 等**价格整体少 10 倍**。实证(2026-10-21):协议原始整数
与分时/逐笔价格比对,6 只沪市指数 + 2 只板块指数(880001/880305)全部是
``×10`` 错误,而 4 只沪市 ETF(510300/510500/588000/511010)、4 只沪市
股票、3 只科创板、3 只深市指数、深市股票/创业板/ETF 全部 ``×1`` 正确。
"""
code = (code or "").strip().rstrip("\x00")
if market == 1: # SH
if code.startswith(("5", "000", "8")):
return 3
return 2
if code.startswith("5"):
return 3 # ETF/基金
return 2 # 指数(000xxx/880xxx)、股票(60xxxx/688xxx)
if market == 0: # SZ
if code.startswith("1"):
return 3
return 2
return 3 # 基金/债券
return 2 # 指数(399xxx)、股票(00xxxx/30xxxx)
return 2
@@ -111,8 +111,11 @@ class MacTransactionCmd(BaseCommand[list[TransactionRecord]]):
for i in range(count):
offset = 39 + i * 18
try:
# trade_count 必须按**有符号** int32 解析:集合竞价段(bs_flag=8)
# 该字段实为负值,按 uint32 会读出 42 亿量级的脏数据。
# 实测(2026-10-21 sh600000):09:15:03 原始 96fcffff = -874。
time_sec, price, volume, trade_count, bs_flag = unpack_from(
"<IfIIH", body, offset, "mac transaction item"
"<IfIiH", body, offset, "mac transaction item"
)
except ValueError:
break
+7 -3
View File
@@ -27,6 +27,7 @@ class PageSpec:
ordered: bool = True # 按 start 升序拼接
min_page: int = 100 # 规避标准K线 count<100 空响应坑
start_offset: int = 0 # 起始偏移(从该偏移开始分页,避免拉取历史全量)
newest_first: bool = False # 服务端 start 从最新往回数(0x122F 逐笔即如此)
@dataclass(slots=True)
@@ -94,7 +95,7 @@ class Paginator:
if s.pc is not None:
pc, s.pc = s.pc, None
await self.pool.release(pc)
return self._assemble(out, spec.ordered)
return self._assemble(out, spec.ordered, spec.newest_first)
def _plan_starts(self, spec: PageSpec) -> list[tuple[int, int]]:
"""切页:返回 [(start, count), ...]。"""
@@ -158,9 +159,12 @@ class Paginator:
return None, False
return None, False
def _assemble(self, results: list[PageResult], ordered: bool) -> list[Any]:
def _assemble(self, results: list[PageResult], ordered: bool,
newest_first: bool = False) -> list[Any]:
# newest_first=True:start=0 是最新一页,start 越大越旧 → 反向拼才对得上时间序。
# 单页内部服务端已按时间升序,所以只翻页序,不翻页内元素。
if ordered:
results.sort(key=lambda r: r.start)
results.sort(key=lambda r: r.start, reverse=newest_first)
out: list[Any] = []
for r in results:
out.extend(r.items)
+197 -140
View File
@@ -1,6 +1,6 @@
{
"version": "2026-09-21",
"note": "离线探针复测(2026-09-21 19:03): 可达=25/33, 入池fast_pool=10台",
"version": "2026-10-16",
"note": "离线探针复测(2026-10-16 13:46): 可达=30/36, 入池fast_pool=10台",
"servers": [
{
"ip": "139.9.52.158",
@@ -11,14 +11,14 @@
],
"handshake": null,
"status": "ok",
"latency": 0.22,
"checked_at": "2026-09-21 19:03:04",
"latency": 0.172,
"checked_at": "2026-10-16 13:46:17",
"source": "probe-2026-08-12",
"fast_pool": true,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"standard": 0.226,
"mac": 0.22
"standard": 0.172,
"mac": 0.183
}
},
{
@@ -30,14 +30,14 @@
],
"handshake": null,
"status": "ok",
"latency": 0.366,
"checked_at": "2026-09-21 19:03:04",
"latency": 0.298,
"checked_at": "2026-10-16 13:46:18",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"fast_pool": true,
"checked_date": "2026-10-16",
"_per_proto": {
"standard": 0.396,
"mac": 0.366
"standard": 0.316,
"mac": 0.298
}
},
{
@@ -48,13 +48,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.257,
"checked_at": "2026-09-21 19:03:09",
"latency": 0.236,
"checked_at": "2026-10-16 13:46:23",
"source": "probe-2026-08-12",
"fast_pool": true,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"mac": 0.257
"mac": 0.236
}
},
{
@@ -65,13 +65,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.248,
"checked_at": "2026-09-21 19:03:04",
"latency": 0.213,
"checked_at": "2026-10-16 13:46:17",
"source": "probe-2026-08-12",
"fast_pool": true,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"mac": 0.248
"mac": 0.213
}
},
{
@@ -84,15 +84,15 @@
],
"handshake": "setup",
"status": "ok",
"latency": 0.221,
"checked_at": "2026-09-21 19:03:00",
"latency": 0.316,
"checked_at": "2026-10-16 13:46:13",
"source": "probe-2026-08-12",
"fast_pool": true,
"checked_date": "2026-09-21",
"fast_pool": false,
"checked_date": "2026-10-16",
"_per_proto": {
"standard": 0.399,
"mac": 0.414,
"ex": 0.221
"standard": 0.316,
"mac": 0.329,
"ex": 0.539
}
},
{
@@ -104,10 +104,10 @@
"handshake": null,
"status": "fail",
"latency": null,
"checked_at": "2026-09-21 19:03:20",
"checked_at": "2026-10-16 13:46:34",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {}
},
{
@@ -118,13 +118,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.363,
"checked_at": "2026-09-21 19:03:10",
"latency": 0.682,
"checked_at": "2026-10-16 13:46:24",
"source": "probe-2026-08-12",
"fast_pool": true,
"checked_date": "2026-09-21",
"fast_pool": false,
"checked_date": "2026-10-16",
"_per_proto": {
"mac": 0.363
"mac": 0.682
}
},
{
@@ -136,10 +136,10 @@
"handshake": null,
"status": "fail",
"latency": null,
"checked_at": "2026-09-21 19:03:20",
"checked_at": "2026-10-16 13:46:34",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {}
},
{
@@ -150,13 +150,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.344,
"checked_at": "2026-09-21 19:03:10",
"latency": 0.607,
"checked_at": "2026-10-16 13:46:23",
"source": "probe-2026-08-12",
"fast_pool": true,
"checked_date": "2026-09-21",
"fast_pool": false,
"checked_date": "2026-10-16",
"_per_proto": {
"mac": 0.344
"mac": 0.607
}
},
{
@@ -167,13 +167,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.404,
"checked_at": "2026-09-21 19:03:10",
"latency": 1.264,
"checked_at": "2026-10-16 13:46:24",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"mac": 0.404
"mac": 1.264
}
},
{
@@ -185,10 +185,10 @@
"handshake": null,
"status": "fail",
"latency": null,
"checked_at": "2026-09-21 19:03:24",
"checked_at": "2026-10-16 13:46:34",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {}
},
{
@@ -200,10 +200,10 @@
"handshake": null,
"status": "fail",
"latency": null,
"checked_at": "2026-09-21 19:03:21",
"checked_at": "2026-10-16 13:46:34",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {}
},
{
@@ -214,13 +214,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.376,
"checked_at": "2026-09-21 19:03:10",
"latency": 0.612,
"checked_at": "2026-10-16 13:46:23",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"mac": 0.376
"mac": 0.612
}
},
{
@@ -231,13 +231,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.377,
"checked_at": "2026-09-21 19:03:10",
"latency": 0.613,
"checked_at": "2026-10-16 13:46:23",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"mac": 0.377
"mac": 0.613
}
},
{
@@ -248,13 +248,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.401,
"checked_at": "2026-09-21 19:03:10",
"latency": 0.624,
"checked_at": "2026-10-16 13:46:23",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"mac": 0.401
"mac": 0.624
}
},
{
@@ -265,13 +265,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.42,
"checked_at": "2026-09-21 19:03:10",
"latency": 0.592,
"checked_at": "2026-10-16 13:46:23",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"mac": 0.42
"mac": 0.592
}
},
{
@@ -282,13 +282,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.401,
"checked_at": "2026-09-21 19:03:10",
"latency": 0.624,
"checked_at": "2026-10-16 13:46:23",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"mac": 0.401
"mac": 0.624
}
},
{
@@ -300,14 +300,14 @@
],
"handshake": null,
"status": "ok",
"latency": 0.485,
"checked_at": "2026-09-21 19:03:05",
"latency": 0.306,
"checked_at": "2026-10-16 13:46:18",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"standard": 0.485,
"mac": 0.605
"standard": 0.352,
"mac": 0.306
}
},
{
@@ -319,10 +319,10 @@
"handshake": null,
"status": "fail",
"latency": null,
"checked_at": "2026-09-21 19:03:04",
"checked_at": "2026-10-16 13:46:18",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {}
},
{
@@ -334,14 +334,14 @@
],
"handshake": null,
"status": "ok",
"latency": 0.444,
"checked_at": "2026-09-21 19:03:05",
"latency": 0.307,
"checked_at": "2026-10-16 13:46:18",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"standard": 0.444,
"mac": 0.903
"standard": 0.35,
"mac": 0.307
}
},
{
@@ -352,13 +352,16 @@
"mac"
],
"handshake": null,
"status": "fail",
"latency": null,
"checked_at": "2026-09-21 19:03:15",
"status": "ok",
"latency": 0.395,
"checked_at": "2026-10-16 13:46:18",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"_per_proto": {}
"checked_date": "2026-10-16",
"_per_proto": {
"standard": 0.395,
"mac": 0.675
}
},
{
"ip": "117.34.114.15",
@@ -369,14 +372,14 @@
],
"handshake": null,
"status": "ok",
"latency": 0.435,
"checked_at": "2026-09-21 19:03:05",
"latency": 0.345,
"checked_at": "2026-10-16 13:46:18",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"standard": 0.463,
"mac": 0.435
"standard": 0.345,
"mac": 0.726
}
},
{
@@ -387,13 +390,16 @@
"mac"
],
"handshake": null,
"status": "fail",
"latency": null,
"checked_at": "2026-09-21 19:03:14",
"status": "ok",
"latency": 0.326,
"checked_at": "2026-10-16 13:46:18",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"_per_proto": {}
"checked_date": "2026-10-16",
"_per_proto": {
"standard": 0.326,
"mac": 0.743
}
},
{
"ip": "117.34.114.27",
@@ -404,14 +410,14 @@
],
"handshake": null,
"status": "ok",
"latency": 0.482,
"checked_at": "2026-09-21 19:03:05",
"latency": 0.394,
"checked_at": "2026-10-16 13:46:18",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"standard": 0.482,
"mac": 0.579
"standard": 0.394,
"mac": 0.78
}
},
{
@@ -423,14 +429,14 @@
],
"handshake": null,
"status": "ok",
"latency": 0.609,
"checked_at": "2026-09-21 19:03:05",
"latency": 0.327,
"checked_at": "2026-10-16 13:46:18",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"standard": 0.946,
"mac": 0.609
"standard": 0.327,
"mac": 0.328
}
},
{
@@ -442,14 +448,14 @@
],
"handshake": null,
"status": "ok",
"latency": 0.46,
"checked_at": "2026-09-21 19:03:05",
"latency": 0.348,
"checked_at": "2026-10-16 13:46:18",
"source": "probe-2026-08-12",
"fast_pool": false,
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"_per_proto": {
"standard": 0.46,
"mac": 0.521
"standard": 0.348,
"mac": 0.74
}
},
{
@@ -461,9 +467,9 @@
"handshake": null,
"status": "fail",
"latency": null,
"checked_at": "2026-09-21 19:03:24",
"checked_at": "2026-10-16 13:46:37",
"source": "probe-2026-08-12",
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"fast_pool": false,
"_per_proto": {}
},
@@ -475,13 +481,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.271,
"checked_at": "2026-09-21 19:03:04",
"latency": 0.726,
"checked_at": "2026-10-16 13:46:18",
"source": "probe-2026-08-12",
"checked_date": "2026-09-21",
"fast_pool": true,
"checked_date": "2026-10-16",
"fast_pool": false,
"_per_proto": {
"mac": 0.271
"mac": 0.726
}
},
{
@@ -492,13 +498,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.375,
"checked_at": "2026-09-21 19:03:05",
"latency": 0.624,
"checked_at": "2026-10-16 13:46:18",
"source": "probe-2026-08-12",
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"fast_pool": false,
"_per_proto": {
"mac": 0.375
"mac": 0.624
}
},
{
@@ -509,13 +515,13 @@
],
"handshake": null,
"status": "ok",
"latency": 0.424,
"checked_at": "2026-09-21 19:03:05",
"latency": 0.56,
"checked_at": "2026-10-16 13:46:30",
"source": "probe-2026-08-12",
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"fast_pool": false,
"_per_proto": {
"mac": 0.424
"mac": 0.56
}
},
{
@@ -526,13 +532,13 @@
],
"handshake": "login",
"status": "ok",
"latency": 0.176,
"checked_at": "2026-09-21 19:03:08",
"latency": 0.097,
"checked_at": "2026-10-16 13:46:21",
"source": "probe-2026-08-12",
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"fast_pool": true,
"_per_proto": {
"ex": 0.176
"ex": 0.097
}
},
{
@@ -543,13 +549,13 @@
],
"handshake": "login",
"status": "ok",
"latency": 0.148,
"checked_at": "2026-09-21 19:03:08",
"latency": 0.097,
"checked_at": "2026-10-16 13:46:21",
"source": "probe-2026-08-12",
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"fast_pool": true,
"_per_proto": {
"ex": 0.148
"ex": 0.097
}
},
{
@@ -560,15 +566,66 @@
],
"handshake": "setup",
"status": "ok",
"latency": 0.166,
"checked_at": "2026-09-21 19:03:08",
"latency": 0.103,
"checked_at": "2026-10-16 13:46:21",
"source": "probe-2026-08-12",
"checked_date": "2026-09-21",
"checked_date": "2026-10-16",
"fast_pool": true,
"_per_proto": {
"ex": 0.166
"ex": 0.103
}
},
{
"ip": "112.74.214.43",
"name": "扩展市场深圳双线1",
"protocols": [
"ex"
],
"handshake": "login",
"status": "ok",
"latency": 0.095,
"checked_at": "2026-10-16 13:46:21",
"source": "connect.cfg",
"fast_pool": true,
"checked_date": "2026-10-16",
"_per_proto": {
"ex": 0.095
}
},
{
"ip": "120.25.218.6",
"name": "扩展市场深圳双线2",
"protocols": [
"ex"
],
"handshake": "login",
"status": "ok",
"latency": 0.101,
"checked_at": "2026-10-16 13:46:21",
"source": "connect.cfg",
"fast_pool": true,
"checked_date": "2026-10-16",
"_per_proto": {
"ex": 0.101
}
},
{
"ip": "47.102.108.214",
"name": "扩展市场上海双线2",
"protocols": [
"ex"
],
"handshake": "login",
"status": "ok",
"latency": 0.149,
"checked_at": "2026-10-16 13:46:22",
"source": "connect.cfg",
"fast_pool": true,
"checked_date": "2026-10-16",
"_per_proto": {
"ex": 0.149
}
}
],
"checked_date": "2026-09-21"
"checked_date": "2026-10-16"
}
+211 -1
View File
@@ -8,7 +8,13 @@ import unittest.mock as mock
import pytest
from pytdxdata.api import TdxData
from pytdxdata.models import KlinePeriod, Market, MinuteBar, SecurityBar
from pytdxdata.models import (
KlinePeriod,
Market,
MinuteBar,
SecurityBar,
TransactionRecord,
)
from pytdxdata.protocol.commands.mac_tick import MacTickChartCmd
from pytdxdata.scheduler.paginator import PageSpec, Paginator
@@ -243,6 +249,70 @@ async def test_paginator_start_offset_default_zero():
assert result[0].close == 10.0 # 从 offset 0 开始
# ---------- Paginator 逆序偏移(服务端 start 从最新往回数) ----------
class NewestFirstConn:
"""模拟 0x122F 真实行为:start=0 返回**最新**的一页,start 越大越旧。
实测依据(2026-10-21,sh600000):
start=0 -> 14:07:48 .. 15:20:16(当天最新 1000 条)
start=4000 -> 09:15:03 .. 09:41:18(当天最早)
每页内部按时间升序,页与页之间是倒序。
"""
def __init__(self, total: int, page_size: int):
self.total = total
self.page_size = page_size
self.executed: list[tuple[int, int]] = []
async def execute(self, cmd):
start, count = cmd
self.executed.append((start, count))
# 服务端语义:偏移 start 是从"最新"往回数,所以取倒数区间
end_from_oldest = self.total - start
begin_from_oldest = max(0, end_from_oldest - count)
return [_make_bar(i) for i in range(begin_from_oldest, end_from_oldest)]
@pytest.mark.asyncio
async def test_paginator_newest_first_offset_returns_chronological():
"""服务端 start 从最新往回数时,交付列表必须仍是时间升序(BUG 回归)。"""
conns = [NewestFirstConn(3000, 1000) for _ in range(3)]
pool = FakePool(conns)
pag = Paginator(pool, concurrency=3) # type: ignore[arg-type]
def factory(s, c):
return (s, c)
result = await pag.fetch_all(
factory, PageSpec(page_size=1000, total=3000, newest_first=True))
assert len(result) == 3000
# 时间升序:close 随 index 单调递增
closes = [b.close for b in result]
assert closes == sorted(closes), "返回列表必须按时间升序拼接"
assert result[0].close == pytest.approx(10.0)
assert result[-1].close == pytest.approx(10.0 + 2999 * 0.01)
@pytest.mark.asyncio
async def test_paginator_newest_first_respects_start_offset():
"""newest_first + start_offset:只取最近 N 条里的后段。"""
conns = [NewestFirstConn(3000, 1000) for _ in range(3)]
pool = FakePool(conns)
pag = Paginator(pool, concurrency=3) # type: ignore[arg-type]
def factory(s, c):
return (s, c)
result = await pag.fetch_all(
factory,
PageSpec(page_size=1000, total=1000, newest_first=True, start_offset=0))
assert len(result) == 1000
# 只取了最新的 1000 条 → 是 2000..2999 这一段
assert result[0].close == pytest.approx(10.0 + 2000 * 0.01)
assert result[-1].close == pytest.approx(10.0 + 2999 * 0.01)
# ---------- get_kline 日期过滤 ----------
class DateBarApiConn(FakeApiConn):
@@ -329,3 +399,143 @@ async def test_get_kline_since():
assert bars[-1].year == 2027
finally:
await td.close()
# ---------- A股逐笔:b8 过滤 + 单连接串行游标(回归) ----------
class TickApiConn(FakeApiConn):
"""模拟 0x122F:服务端 start 从**最新**往回数,且可注入 bs_flag=8 委托记录。"""
# 类级:[(time, price, vol, trade_count, bs_flag), ...],时间升序(最早在前)
rows: list[tuple] = []
calls: list[tuple[int, int]] = []
async def execute(self, cmd):
if "MacTransactionCmd" not in type(cmd).__name__:
return await super().execute(cmd)
TickApiConn.calls.append((cmd.start, cmd.count))
# 从最新往回数:start 偏移落在末尾,往前取 count 条
end = len(TickApiConn.rows) - cmd.start
begin = max(0, end - cmd.count)
chunk = TickApiConn.rows[begin:end]
return [TransactionRecord(market=1, code="600000", time=t, price=p,
vol=v, trade_count=tc, bs_flag=bs)
for (t, p, v, tc, bs) in chunk]
def _build_tick_rows(n_trade: int = 25, n_b8: int = 7) -> list[tuple]:
"""构造 n_trade 条正常成交 + n_b8 条竞价委托(穿插在中间)。"""
rows: list[tuple] = []
for i in range(n_trade):
rows.append((f"09:{30 + i // 60:02d}:{i % 60:02d}", 10.0 + i * 0.01,
100.0 + i, 3 + i, 1 if i % 2 else 0))
if i % 4 == 0 and len([r for r in rows if r[4] == 8]) < n_b8:
# 竞价委托:(time, price, vol, tc, bs=8),vol 是累计值、不计入成交
rows.append((f"09:15:{i:02d}", 10.0, 9999.0, -1, 8))
return rows
@pytest.mark.asyncio
async def test_get_ticks_filters_auction_orders():
"""bs_flag=8 是竞价**委托**快照,不是成交 → 必须从 get_ticks 结果中剔除。
实测(2026-10-21 sh600000):b8 的 vol 是累计撮合量(可被覆盖),
计入后 Σvol 比快照 vol 虚高 20%~160%,且不同主机的 b8 集合不同。
"""
TickApiConn.rows = _build_tick_rows(n_trade=25, n_b8=7)
TickApiConn.calls = []
assert sum(1 for r in TickApiConn.rows if r[4] == 8) == 7
with mock.patch("pytdxdata.api.get_standard_hosts", return_value=["s1"]), \
mock.patch("pytdxdata.api.get_mac_hosts", return_value=["m1"]), \
mock.patch("pytdxdata.pool.pool.TdxConnection",
side_effect=lambda h, p=7709, t=10.0: TickApiConn(h)):
td = TdxData(cache_dir=None, pool_min=1, pool_max=2)
await td.start()
try:
out = await td.get_ticks("sh600000")
finally:
await td.close()
assert out, "应返回成交记录"
assert all(r.bs_flag != 8 for r in out), "bs_flag=8 竞价委托必须被剔除"
assert len(out) == 25
assert sum(r.vol for r in out) == sum(100.0 + i for i in range(25))
@pytest.mark.asyncio
async def test_get_ticks_pages_on_single_connection():
"""翻页必须走**同一连接**:并发多主机会让页边界错位(重叠 + 漏条)。"""
TickApiConn.rows = _build_tick_rows(n_trade=25, n_b8=0)
TickApiConn.calls = []
with mock.patch("pytdxdata.api.get_standard_hosts", return_value=["s1"]), \
mock.patch("pytdxdata.api.get_mac_hosts", return_value=["m1", "m2", "m3"]), \
mock.patch("pytdxdata.pool.pool.TdxConnection",
side_effect=lambda h, p=7709, t=10.0: TickApiConn(h)):
td = TdxData(cache_dir=None, pool_min=1, pool_max=4)
await td.start()
try:
out = await td.get_ticks("sh600000")
finally:
await td.close()
# 单页装得下 → 只发一次请求
assert len(TickApiConn.calls) == 1
assert len(out) == 25
# 时间升序
times = [r.time for r in out]
assert times == sorted(times)
@pytest.mark.asyncio
async def test_get_ticks_multi_page_uses_one_connection_and_no_overlap():
"""多页(>_TICK_PAGE):同一连接、游标按 len(rows) 推进、页间不重叠不漏条。
回归:并发分页时各页来自不同主机,而 0x122F 的 start 是「从最新往回数」的
相对锚点 —— 各主机对 bs_flag=8 收录不同,同一 start 落到不同真实成交上,
结果页边界既重叠又漏条(实测 sh600000 冗余 116 条、13:03 附近丢 59 条)。
"""
from pytdxdata.api import _TICK_PAGE
n = _TICK_PAGE * 2 + 137 # 3 页,末页短页 → 触发 stop_when_short
TickApiConn.rows = [
(f"{9 + i // 3600:02d}:{i // 60 % 60:02d}:{i % 60:02d}",
10.0 + i * 0.001, 1.0 + i, 1 + i, 1)
for i in range(n)
]
TickApiConn.calls = []
with mock.patch("pytdxdata.api.get_standard_hosts", return_value=["s1"]), \
mock.patch("pytdxdata.api.get_mac_hosts", return_value=["m1", "m2", "m3"]), \
mock.patch("pytdxdata.pool.pool.TdxConnection",
side_effect=lambda h, p=7709, t=10.0: TickApiConn(h)):
td = TdxData(cache_dir=None, pool_min=1, pool_max=4)
await td.start()
try:
out = await td.get_ticks("sh600000")
finally:
await td.close()
# 恰好 3 页:0 / 1000 / 2000,游标按实际返回条数推进
assert TickApiConn.calls == [(0, _TICK_PAGE),
(_TICK_PAGE, _TICK_PAGE),
(_TICK_PAGE * 2, _TICK_PAGE)]
assert len(out) == n, f"必须无重叠、无漏条:期望 {n},实得 {len(out)}"
vols = [r.vol for r in out]
assert len(set(vols)) == n, "vol 全唯一(无重复记录)"
assert vols == sorted(vols), "时间升序"
# count 参数:游标逐页扣减
TickApiConn.calls = []
with mock.patch("pytdxdata.api.get_standard_hosts", return_value=["s1"]), \
mock.patch("pytdxdata.api.get_mac_hosts", return_value=["m1", "m2", "m3"]), \
mock.patch("pytdxdata.pool.pool.TdxConnection",
side_effect=lambda h, p=7709, t=10.0: TickApiConn(h)):
td = TdxData(cache_dir=None, pool_min=1, pool_max=4)
await td.start()
try:
out2 = await td.get_ticks("sh600000", count=_TICK_PAGE + 50)
finally:
await td.close()
assert len(out2) == _TICK_PAGE + 50
assert TickApiConn.calls[1][1] == 50, "末页只补足剩余 50 条"
+116
View File
@@ -0,0 +1,116 @@
"""ExPool 背压/并发语义回归(Issue #1:并发超限 0.15s 秒失败)。
用 mock 连接模拟:单服务器 + per_server_cap=1 时,
- 并发 acquire 超过有效并发 → 排队等待而不是快速失败;
- release 唤醒等待者;
- acquire 超时异常带池状态;
- ex_pool_cap 提升同主机并发。
"""
from __future__ import annotations
import asyncio
import time
import pytest
from pytdxdata.pool.ex_pool import ExPool, ExPoolTimeoutError
class MockConn:
"""Minimal stand-in for ExTdxConnection."""
last_created = 0
def __init__(self, host: str, port: int, timeout: float, handshake: str = "login"):
self.host = host
self.port = port
self.alive = True
self.last_used = time.monotonic()
self.closed = False
MockConn.last_created += 1
async def connect(self) -> None:
await asyncio.sleep(0) # instant "connect"
async def close(self) -> None:
self.closed = True
self.alive = False
@pytest.fixture
def patch_conn(monkeypatch):
monkeypatch.setattr("pytdxdata.pool.ex_pool.ExTdxConnection", MockConn)
MockConn.last_created = 0
async def test_acquire_queues_instead_of_fast_fail(patch_conn):
"""并发 > 有效并发时,后续 acquire 等待 release,而非 0.15s 抛错。"""
pool = ExPool(["e1"], min_size=1, max_size=6, per_server_cap=1)
await pool.start()
c1 = await pool.acquire() # 唯一连接被占用
t0 = time.monotonic()
async def holder():
await asyncio.sleep(0.5) # 持有 0.5s 后归还
await pool.release(c1)
asyncio.create_task(holder())
c2 = await pool.acquire() # 应在 ~0.5s 后拿到,而不是 0.15s 抛错
waited = time.monotonic() - t0
assert waited >= 0.4, f"acquire 没有等待释放(仅 {waited:.2f}s)"
assert c2 is c1 # 归还的是同一条连接
async def test_acquire_timeout_carries_pool_state(patch_conn):
"""超时异常信息带池状态(idle/total/servers/cap)。"""
pool = ExPool(["e1"], min_size=1, max_size=6, per_server_cap=1, acquire_timeout=0.3)
await pool.start()
c1 = await pool.acquire()
with pytest.raises(ExPoolTimeoutError) as ei:
await pool.acquire()
msg = str(ei.value)
assert "total=1" in msg and "per_server_cap=1" in msg and "servers=1" in msg
await pool.release(c1)
async def test_ex_pool_cap_allows_parallel_conns_same_host(patch_conn):
"""per_server_cap=2 时单主机可并发持有 2 条连接。"""
pool = ExPool(["e1"], min_size=1, max_size=6, per_server_cap=2)
await pool.start()
got = await asyncio.gather(pool.acquire(), pool.acquire())
assert len(got) == 2 and got[0] is not got[1]
assert MockConn.last_created == 2
for c in got:
await pool.release(c)
async def test_concurrent_acquire_all_succeed(patch_conn):
"""Issue 场景:单服务器 + cap=2,6 个并发任务应全部拿到连接(排队)而非部分失败。"""
pool = ExPool(["e1"], min_size=2, max_size=6, per_server_cap=2)
await pool.start()
async def task(i):
c = await pool.acquire()
await asyncio.sleep(0.1) # 模拟一次 1m 分页拉取
await pool.release(c)
return i
results = await asyncio.gather(*[task(i) for i in range(6)])
assert sorted(results) == list(range(6)) # 全部成功,零失败
async def test_release_notifies_waiters(patch_conn):
"""broken 归还(连接死亡)也要唤醒等待者(槽位回收后可扩容)。"""
pool = ExPool(["e1"], min_size=1, max_size=1, per_server_cap=1)
await pool.start()
c1 = await pool.acquire()
c1.alive = False
async def broken_release():
await asyncio.sleep(0.2)
await pool.release(c1, broken=True)
asyncio.create_task(broken_release())
c2 = await pool.acquire() # 槽位被回收→扩容新建
assert c2 is not c1
await pool.release(c2)
+59
View File
@@ -12,6 +12,7 @@ from pytdxdata.protocol.commands.ex_proto import GetExInstrumentBarsCmd
from pytdxdata.protocol.commands.kline import GetSecurityBarsCmd
from pytdxdata.protocol.commands.mac import MacKlineCmd
from pytdxdata.protocol.commands.minute import GetHistoryMinuteTimeDataCmd
from pytdxdata.protocol.commands.quotes import price_decimal_digits
from pytdxdata.protocol.commands.transaction import MacTransactionCmd
from pytdxdata.protocol.frame import FrameHeader, decompress_body, parse_header
from pytdxdata.protocol.price import decode_ohlc_diffs, get_price, put_price
@@ -181,6 +182,30 @@ def test_mac_transaction_parse():
assert recs[1].trade_count == 1
def test_mac_transaction_parse_negative_trade_count_auction():
"""集合竞价段 trade_count 是有符号 int32,负数不能被读成 42 亿(BUG 回归)。
实测(2026-10-21,sh600000 09:15:03):18 字节为
17820000 0ad71b41 d9000000 96fcffff 0800
最后四字节前的 96fcffff = int32 -874,旧解析按 uint32 得 4294966422。
"""
body = bytearray(39 + 2 * 18)
struct.pack_into("<H", body, 29, 2)
# 第1条:集合竞价,trade_count = -874
struct.pack_into("<IfIiH", body, 39, 33303, 9.74, 217, -874, 8)
# 第2条:正常交易,trade_count = 3
struct.pack_into("<IfIiH", body, 39 + 18, 34200, 9.75, 100, 3, 1)
cmd = MacTransactionCmd(1, "600000", 20261021, 0, 100)
recs = cmd.parse_response(bytes(body))
assert len(recs) == 2
assert recs[0].trade_count == -874, "负值必须保留符号,不能变成 4294966422"
assert recs[0].bs_flag == 8
assert recs[1].trade_count == 3
# 正常交易段的量级不受影响
assert max(r.trade_count for r in recs) == 3
# ---------- 跨时区时间戳归一化(美股 EX 通道) ----------
#
# 通达信 EX 通道对美股返回的时间戳把两个时区拼在一起:日期字段是**美东交易日**、
@@ -286,3 +311,37 @@ def test_finish_bars_sorts_us_session_chronologically():
assert [b.datetime for b in out] == [datetime(2026, 10, 12, 21, 31),
datetime(2026, 10, 13, 4, 0)]
assert {b.symbol for b in out} == {"usAAPL"}
# ---------- 报价小数位:沪市指数必须 2 位(回归:曾整体少 10 倍) ----------
@pytest.mark.parametrize("market,code,expect", [
# 沪市指数:回归 bug —— 曾被判 3 位,导致价格少 10 倍
(1, "000001", 2), # 上证指数
(1, "000016", 2), # 上证50
(1, "000300", 2), # 沪深300
(1, "000688", 2), # 科创50
(1, "880001", 2), # 板块指数(8 开头,曾被判 3 位)
(1, "880305", 2),
# 沪市 ETF/基金:仍是 3 位
(1, "510300", 3),
(1, "588000", 3),
(1, "511010", 3),
(1, "501018", 3),
# 沪市股票:2 位
(1, "600000", 2),
(1, "601398", 2),
(1, "688111", 2), # 科创板
# 深市
(0, "399001", 2), # 深证成指
(0, "399006", 2), # 创业板指
(0, "000001", 2), # 平安银行(不是指数!)
(0, "300750", 2),
(0, "159915", 3), # 深市 ETF
(0, "128035", 3), # 深市转债
])
def test_price_decimal_digits(market, code, expect):
assert price_decimal_digits(market, code) == expect
def test_price_decimal_digits_strips_nul():
assert price_decimal_digits(1, "000016\x00\x00") == 2
Generated
+1 -1
View File
@@ -535,7 +535,7 @@ wheels = [
[[package]]
name = "pytdxdata"
version = "0.6.2"
version = "0.6.4"
source = { editable = "." }
[package.optional-dependencies]