perf: 全市场报价改单连接流水线(3.5s → 0.33s)+ 修复基金笔数口径 (#9)
## 性能:get_quotes 全市场 5,227 只 A股 3.5s → 0.33s 协议硬上限 80 只/请求 → 必须切 66 片;老实现是纯串行 for 循环, 每片一次往返(≈60ms),3.5s 全花在等网络上。 发现通达信响应是自描述长度帧且服务器按序应答,因此同一连接可连发 多条请求只付一次往返:实测单连接流水 66 请求 526ms(串行 7,372ms)。 - TdxConnection.execute_pipeline(cmds):一次下发全部 → 按序读回;中途失败强制断连 - _quotes_std 改工作窃取调度(快台多干,避免被最慢节点拖累) - DynamicPool.warm(n):并发预建连接(acquire 在全局锁内一次一条,22 并发首拉会多花 5~6s) - CacheManager.get_or_fetch_obj:回源直接返回对象,省掉冷路径 pickle→unpickle 白烧 - pool_max 默认 12 → 24 调优实测 p50/max:每台 1 连接×深度 6 = 455ms;每台 2 连接×深度 3 = 325ms。 端到端(全默认):首次 783ms、稳态 300~343ms、热缓存 67ms。 正确性:用昨收价交叉比对 30 只零错配,流水线与逐片结果集合完全一致。 ## 修复:指数 / ETF / LOF 的 trade_count 不再伪装成笔数 通达信对指数和多数场内基金不提供笔数,那 4 字节里塞的是「价格 × 1000」 (指数是「均价 × 1000」)。旧版直接当笔数用,tdx transactions sh510310 会打印出「3994 笔」这种假数据。 - 新增 TransactionRecord.trade_count_available - 新增 symbols.has_order_count(market, code) 按品种判定 (沪市 501x/502x/506x 实测有真实笔数,不能按「5 开头=基金」一刀切) - 历史通道 0x013001 无笔数字段,同样标记不可用 ## 文档:可转债量纲口径 核查可转债分时/逐笔后确认数据本身无缺陷(16 只当日 + 8 只历史日, 分时末价/现价、Σvol/日线vol、逐笔 VWAP 比值均 1.000000), 但 vol 单位是「手」(=10 张)导致 amount/vol 是均价的 10 倍,对外文档缺失。 - SecurityBar / TransactionRecord / MinuteBar 量纲注释补全 - has_order_count 实测表补沪可转债 110/111/113/118、沪 EB 132、深可转债 123/127/128 - API_REFERENCE 新增跨品种量纲对照表;llms.txt / SKILL.md 同步 ## 版本 0.6.7 ## 测试 306 → 346 全通过(新增 tests/test_pipeline.py 11 项 + 基金笔数相关回归) Co-authored-by: openbot-coder <openbot-coder@users.noreply.github.com>
This commit is contained in:
1 parent
2c07400a3a
commit
8a78156145
17 files changed
+860
-61
No files matched your search
@@ -18,7 +18,7 @@ from .models import (
|
||||
XdxrRecord,
|
||||
)
|
||||
|
||||
__version__ = "0.6.6"
|
||||
__version__ = "0.6.7"
|
||||
__all__ = [
|
||||
"TdxData",
|
||||
"Market",
|
||||
|
||||
+158
-29
@@ -25,6 +25,7 @@ import logging
|
||||
import re
|
||||
import time
|
||||
import warnings
|
||||
from collections import deque
|
||||
from collections.abc import Callable
|
||||
from datetime import date as _date, datetime
|
||||
from pathlib import Path
|
||||
@@ -147,6 +148,7 @@ _KLINE_MAX_BARS: dict[KlinePeriod, int] = {
|
||||
# 自定义字段报价(0x122B/0x122C)基础字段位:昨收/开/高/低/现价/量/量比/额
|
||||
_QUOTE_BITS = [0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07]
|
||||
_QUOTE_PAGE = 80 # 0x122B/0x122C 单次上限 80 只
|
||||
_QUOTE_PIPE_CHUNK = 3 # 单连接流水线一次取几片(实测 3 片 + 每台 2 连接最稳)
|
||||
_EX_PAGE = 1000 # EX 商品列表单页
|
||||
_TICK_PAGE = 1000 # MAC 逐笔单页
|
||||
_EX_TICK_PAGE = 1800 # 港股逐笔单页
|
||||
@@ -237,7 +239,7 @@ class TdxData:
|
||||
default_adjust: Adjust = Adjust.NONE,
|
||||
max_inflight: int = 64,
|
||||
pool_min: int = 3,
|
||||
pool_max: int = 12,
|
||||
pool_max: int = 24,
|
||||
mac_pool_cap: int = 4,
|
||||
std_pool_cap: int = 4,
|
||||
ex_pool_cap: int = 2,
|
||||
@@ -442,7 +444,9 @@ class TdxData:
|
||||
# ================================================================== #
|
||||
|
||||
async def get_quotes(self, symbols: str | Sequence[str], *,
|
||||
fields: Sequence[int] | None = None
|
||||
fields: Sequence[int] | None = None,
|
||||
concurrency: int | None = None,
|
||||
pipeline: bool = True
|
||||
) -> list[SecurityQuote]:
|
||||
"""报价,一次可跨市场、跨通道混合查询。
|
||||
|
||||
@@ -452,6 +456,11 @@ class TdxData:
|
||||
fields: 可选自定义字段位(如 ``[0x00, 0x04, 0x38]``)。给了就走 MAC/EX 的
|
||||
0x122B 字段位通道(有 ``name``,无五档);不给时 A股走标准通道
|
||||
(**有五档 bid/ask、买卖量、涨跌停价,无 name**),扩展市场走 0x122B。
|
||||
concurrency: 连接并发数(默认取连接池上限)。协议硬上限 80 只/请求,
|
||||
全市场约 5,200 只 A股需切 66 片;调小可降低对服务器的压力。
|
||||
pipeline: 是否启用**单连接流水线**(默认开启)。开启后一条连接可一次发出
|
||||
多个分片、只付一次往返,全市场 5,200 只 A股由 ~4s 降到 **~0.3s**;
|
||||
个股或小批量不受影响。个别服务器不支持流水线时会自动降级为逐片。
|
||||
|
||||
返回:
|
||||
``list[SecurityQuote]``,按入参顺序排列(各记录带 ``symbol``)。
|
||||
@@ -462,50 +471,155 @@ class TdxData:
|
||||
bits = [int(b) for b in fields] if fields else None
|
||||
tag = "mac:" + _fields_tag(bits) if bits else "std"
|
||||
key = CacheKey.quote_key([s.as_pair() for s in syms], tag=tag)
|
||||
payload = await self._cache.get_or_fetch(
|
||||
key, lambda: self._fetch_quotes(syms, bits))
|
||||
quotes = unpack_payload(payload)
|
||||
quotes = await self._cache.get_or_fetch_obj(
|
||||
key,
|
||||
lambda: self._fetch_quotes(syms, bits, concurrency, pipeline),
|
||||
pack=pack_payload, unpack=unpack_payload)
|
||||
order = {s.as_pair(): i for i, s in enumerate(syms)}
|
||||
quotes.sort(key=lambda q: order.get((int(q.market), q.code), len(order)))
|
||||
return quotes
|
||||
|
||||
async def _fetch_quotes(self, syms: list[Symbol], bits: list[int] | None) -> bytes:
|
||||
async def _fetch_quotes(self, syms: list[Symbol], bits: list[int] | None,
|
||||
concurrency: int | None = None,
|
||||
pipeline: bool = True) -> list[SecurityQuote]:
|
||||
out: list[SecurityQuote] = []
|
||||
cn = [s for s in syms if not s.is_ex]
|
||||
ex = [s for s in syms if s.is_ex]
|
||||
if bits is None:
|
||||
if cn:
|
||||
out.extend(await self._quotes_std(cn))
|
||||
out.extend(await self._quotes_std(
|
||||
cn, concurrency=concurrency, pipeline=pipeline))
|
||||
if ex:
|
||||
rows = await self._execute_ex(
|
||||
MacSymbolQuotesCmd([s.as_pair() for s in ex], None))
|
||||
out.extend(_quote_from_member(m) for m in rows)
|
||||
else:
|
||||
for i in range(0, len(syms), _QUOTE_PAGE):
|
||||
batch = syms[i:i + _QUOTE_PAGE]
|
||||
pairs = [s.as_pair() for s in batch]
|
||||
cn_pairs = [p for p, s in zip(pairs, batch) if not s.is_ex]
|
||||
ex_pairs = [p for p, s in zip(pairs, batch) if s.is_ex]
|
||||
if cn_pairs:
|
||||
rows = await self._execute_mac(MacSymbolQuotesCmd(cn_pairs, bits))
|
||||
out.extend(_quote_from_member(m) for m in rows)
|
||||
if ex_pairs:
|
||||
rows = await self._execute_ex(MacSymbolQuotesCmd(ex_pairs, bits))
|
||||
out.extend(_quote_from_member(m) for m in rows)
|
||||
return pack_payload(out)
|
||||
batches = [syms[i:i + _QUOTE_PAGE]
|
||||
for i in range(0, len(syms), _QUOTE_PAGE)]
|
||||
sem = asyncio.Semaphore(self._quote_concurrency(concurrency))
|
||||
|
||||
async def _quotes_std(self, syms: list[Symbol]) -> list[SecurityQuote]:
|
||||
"""A股标准通道五档报价(>80 只自动切批)。"""
|
||||
out: list[SecurityQuote] = []
|
||||
for i in range(0, len(syms), MAX_QUOTES):
|
||||
batch = syms[i:i + MAX_QUOTES]
|
||||
cmd = GetSecurityQuotesCmd([s.as_pair() for s in batch])
|
||||
rows = await self._execute_standard(cmd)
|
||||
async def one(batch: list[Symbol]) -> list[SecurityQuote]:
|
||||
async with sem:
|
||||
pairs = [s.as_pair() for s in batch]
|
||||
cn_pairs = [p for p, s in zip(pairs, batch) if not s.is_ex]
|
||||
ex_pairs = [p for p, s in zip(pairs, batch) if s.is_ex]
|
||||
got: list[SecurityQuote] = []
|
||||
if cn_pairs:
|
||||
rows = await self._execute_mac(
|
||||
MacSymbolQuotesCmd(cn_pairs, bits))
|
||||
got.extend(_quote_from_member(m) for m in rows)
|
||||
if ex_pairs:
|
||||
rows = await self._execute_ex(
|
||||
MacSymbolQuotesCmd(ex_pairs, bits))
|
||||
got.extend(_quote_from_member(m) for m in rows)
|
||||
return got
|
||||
|
||||
if batches:
|
||||
for got in await asyncio.gather(*(one(b) for b in batches)):
|
||||
out.extend(got)
|
||||
return out
|
||||
|
||||
def _quote_concurrency(self, requested: int | None) -> int:
|
||||
"""连接并发数:显式指定优先,否则按「每台服务器 2 连接」估算。
|
||||
|
||||
实测(5,227 只 A股 / 66 片,11 台服务器):
|
||||
|
||||
========== ===== ====== ====== ======
|
||||
并发 每台连接 流水深度 p50 max
|
||||
========== ===== ====== ====== ======
|
||||
11 1 6 455ms 464ms
|
||||
12 1~2 3 457ms 523ms
|
||||
**22** **2** **3** **325ms** **334ms**
|
||||
33 3 2 328ms 345ms
|
||||
========== ===== ====== ====== ======
|
||||
|
||||
结论:每台 2 连接 × 流水 3 片最稳;再多连接只是把同一台的队列搬个位置,
|
||||
对整体无益反增抖动。池 ``per_server_cap`` 默认 4,2 连接仍留有余量。
|
||||
"""
|
||||
if requested is not None and requested > 0:
|
||||
return max(1, int(requested))
|
||||
n = len(self._std_pool.servers)
|
||||
return max(1, min(self._std_pool.max_size, (n or 1) * 2))
|
||||
|
||||
async def _quotes_std(self, syms: list[Symbol], *,
|
||||
concurrency: int | None = None,
|
||||
pipeline: bool = True) -> list[SecurityQuote]:
|
||||
"""A股标准通道五档报价(协议上限 80 只/请求,超出自动切批)。
|
||||
|
||||
切片多于 1 片时把各片**按轮转分给若干条连接**,每条连接内部用流水线
|
||||
一次发出(1 次往返拿多片),不再逐片往返。
|
||||
"""
|
||||
if not syms:
|
||||
return []
|
||||
|
||||
def stamp(rows: list[SecurityQuote]) -> list[SecurityQuote]:
|
||||
for q in rows:
|
||||
q.symbol = format_symbol(q.market, q.code)
|
||||
out.extend(rows)
|
||||
return rows
|
||||
|
||||
batches = [syms[i:i + MAX_QUOTES]
|
||||
for i in range(0, len(syms), MAX_QUOTES)]
|
||||
if len(batches) == 1:
|
||||
cmd = GetSecurityQuotesCmd([s.as_pair() for s in batches[0]])
|
||||
return stamp(await self._execute_standard(cmd))
|
||||
|
||||
conc = min(self._quote_concurrency(concurrency), len(batches))
|
||||
|
||||
# 首次大批量:先把连接并发建好,否则 acquire 会在池锁里串行握手
|
||||
# (22 路并发首次调用实测多花 5~6s)
|
||||
await self._std_pool.warm(conc)
|
||||
|
||||
if not pipeline:
|
||||
sem = asyncio.Semaphore(conc)
|
||||
|
||||
async def one(batch: list[Symbol]) -> list[SecurityQuote]:
|
||||
async with sem:
|
||||
cmd = GetSecurityQuotesCmd([s.as_pair() for s in batch])
|
||||
return stamp(await self._execute_standard(cmd))
|
||||
|
||||
groups = await asyncio.gather(*(one(b) for b in batches))
|
||||
return [q for g in groups for q in g]
|
||||
|
||||
# 工作窃取:连接做完一批就回来再取,快的多干、慢的少干。
|
||||
# 平均分配(每台固定几片)会被最慢的一台拖住 —— 实测 11 台平均分
|
||||
# 时 p50 在 400~580ms 之间抖,改窃取后波动收窄。
|
||||
queue = deque(batches)
|
||||
gate = asyncio.Lock()
|
||||
|
||||
async def worker() -> list[SecurityQuote]:
|
||||
got: list[SecurityQuote] = []
|
||||
while True:
|
||||
async with gate:
|
||||
if not queue:
|
||||
return got
|
||||
take = min(_QUOTE_PIPE_CHUNK, len(queue))
|
||||
chunk = [queue.popleft() for _ in range(take)]
|
||||
for rows in await self._execute_std_pipeline(chunk):
|
||||
got.extend(stamp(rows))
|
||||
|
||||
out: list[SecurityQuote] = []
|
||||
for part in await asyncio.gather(*(worker() for _ in range(conc))):
|
||||
out.extend(part)
|
||||
return out
|
||||
|
||||
async def _execute_std_pipeline(
|
||||
self, batches: list[list[Symbol]]
|
||||
) -> list[list[SecurityQuote]]:
|
||||
"""一条连接流水线执行多片报价;节点不支持流水线时降级为逐片。"""
|
||||
cmds = [GetSecurityQuotesCmd([s.as_pair() for s in b]) for b in batches]
|
||||
pc = await self._std_pool.acquire()
|
||||
try:
|
||||
rows = await pc.execute_pipeline(cmds)
|
||||
except Exception:
|
||||
# 流水线中断后响应流已错配,连接必须废弃;改用逐片(每条独立借还)
|
||||
await self._std_pool.release(pc, broken=True)
|
||||
fallback: list[list[SecurityQuote]] = []
|
||||
for cmd in cmds:
|
||||
fallback.append(await self._execute_standard(cmd))
|
||||
return fallback
|
||||
await self._std_pool.release(pc)
|
||||
return rows
|
||||
|
||||
async def get_snapshot(self, symbols: str | Sequence[str]) -> list[SymbolSnapshot]:
|
||||
"""个股特征快照(当日/换手/活跃度等,仅 A股)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -860,6 +974,10 @@ class TdxData:
|
||||
- ``vol`` 单位是**百元**(成交额/100),不是手。
|
||||
- ``trade_count`` 是**当时均价 × 1000**,不是笔数。
|
||||
|
||||
**ETF / LOF 同样没有笔数**:``trade_count`` 是「价格 × 1000」(货币 ETF 恒为 0),
|
||||
这类记录的 ``trade_count_available`` 为 ``False`` 且 ``trade_count`` 已置 0。
|
||||
判定规则见 :func:`~pytdxdata.symbols.has_order_count`。
|
||||
|
||||
详见 :class:`~pytdxdata.models.TransactionRecord` 的类注释。
|
||||
"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -1006,6 +1124,7 @@ class TdxData:
|
||||
trade_count=0,
|
||||
bs_flag=int(row.get("direction", 0) or 0),
|
||||
fields=fields,
|
||||
trade_count_available=False, # 扩展市场逐笔不带笔数字段
|
||||
)
|
||||
|
||||
async def get_minutes(
|
||||
@@ -1392,6 +1511,14 @@ class TdxData:
|
||||
|
||||
注意:返回的是**当前**成分,指数调仓后历史日期不适用。
|
||||
|
||||
⚠️ **综合指数的成员名单不完整**(实测 2026-10-22):``sh000001`` 只返回
|
||||
2230 只(2190 A + 40 B),而当日沪市 A 股实有 2318 只;缺的 128 只
|
||||
(多为近期科创板/主板新股)当日成交 282 亿,正好对上「指数日线 amount −
|
||||
成员日线 amount」的 297 亿(残差 4.9%)。因此
|
||||
:meth:`get_index_trade_stats` 对**综合指数**会系统性低估约 4.6%;
|
||||
样本指数 ``sh000016``(50 只)/ ``sh000300``(300 只)实测比值
|
||||
**1.000000**,不受影响。要精确汇总综指成交,请自行用全市场股票列表。
|
||||
|
||||
用法::
|
||||
|
||||
members = await td.get_index_members("000300")
|
||||
@@ -1508,10 +1635,12 @@ class TdxData:
|
||||
for t in ticks:
|
||||
hhmm = t.time[:5]
|
||||
vol_by_min[hhmm] = vol_by_min.get(hhmm, 0.0) + t.vol
|
||||
tc_by_min[hhmm] = tc_by_min.get(hhmm, 0) + t.trade_count
|
||||
# 成分里若混入 ETF/LOF(协议无笔数),不能把占位值累加进笔数
|
||||
if getattr(t, "trade_count_available", True):
|
||||
tc_by_min[hhmm] = tc_by_min.get(hhmm, 0) + t.trade_count
|
||||
amt_by_min[hhmm] = amt_by_min.get(hhmm, 0.0) + t.price * t.vol * 100.0
|
||||
active.add(t.symbol)
|
||||
if t.trade_count:
|
||||
if getattr(t, "trade_count_available", True) and t.trade_count:
|
||||
with_tc.add(t.symbol)
|
||||
done += len(batch)
|
||||
if on_progress is not None:
|
||||
|
||||
Vendored
+32
@@ -90,6 +90,38 @@ class CacheManager:
|
||||
# 确保清理锁(即使 fetcher 异常也清理,避免无限增长)
|
||||
self._locks.pop(ks, None)
|
||||
|
||||
async def get_or_fetch_obj(
|
||||
self,
|
||||
key: CacheKey,
|
||||
fetcher: Callable[[], Awaitable[Any]],
|
||||
*,
|
||||
pack: Callable[[Any], bytes],
|
||||
unpack: Callable[[bytes], Any],
|
||||
) -> Any:
|
||||
"""同 :meth:`get_or_fetch`,但**回源时直接返回新取到的对象**。
|
||||
|
||||
用于「一次取回上千条记录」的场景(如全市场报价):常规路径对冷数据
|
||||
会先 ``pack`` 再立刻 ``unpack`` 一次,纯属往返浪费 —— 5,200 条报价实测
|
||||
白烧约 90ms。这里命中缓存才解序列化,回源则把对象原样返回,只在写缓存时打包。
|
||||
"""
|
||||
ks = key.to_str()
|
||||
cached = await self.get(key)
|
||||
if cached is not None:
|
||||
return unpack(cached)
|
||||
|
||||
lock = self._locks.setdefault(ks, asyncio.Lock())
|
||||
try:
|
||||
async with lock:
|
||||
cached = await self.get(key)
|
||||
if cached is not None:
|
||||
return unpack(cached)
|
||||
async with self._semaphore:
|
||||
obj = await fetcher()
|
||||
await self.set(key, pack(obj))
|
||||
return obj
|
||||
finally:
|
||||
self._locks.pop(ks, None)
|
||||
|
||||
async def invalidate(self, category: str, *, market: int | None = None,
|
||||
code: str | None = None) -> int:
|
||||
"""按前缀批量失效,返回失效条数。"""
|
||||
|
||||
+63
-13
@@ -38,6 +38,7 @@ from .api import TdxData
|
||||
from dataclasses import asdict
|
||||
|
||||
from .models import Adjust, ExMarket, KlinePeriod, Market
|
||||
from .protocol.commands.quotes import price_decimal_digits
|
||||
|
||||
PERIODS = {
|
||||
"min1": KlinePeriod.MIN_1, "min5": KlinePeriod.MIN_5, "min15": KlinePeriod.MIN_15,
|
||||
@@ -107,6 +108,40 @@ def _pct(price: float, pre: float) -> str:
|
||||
return "%+.2f%%" % ((price / pre - 1) * 100)
|
||||
|
||||
|
||||
def _price_fmt(dp: int) -> str:
|
||||
"""按品种小数位生成右对齐格式串。
|
||||
|
||||
股票 2 位、**可转债/ETF 3 位**、债券/回购 4 位。固定写 ``%8.2f`` 会把
|
||||
可转债的 128.891 打成 128.89(少一位有效报价)。
|
||||
"""
|
||||
dp = max(0, min(int(dp or 2), 4))
|
||||
return "%%%d.%df" % (6 + dp, dp)
|
||||
|
||||
|
||||
# 可转债后缀(解码 4 位,但交易所报价步长 0.001 元,第 4 位恒为 0)
|
||||
_CB_PREFIX = {1: ("110", "111", "113", "118"), 0: ("123", "127", "128")}
|
||||
|
||||
|
||||
def _display_dp(market: int, code: str) -> int:
|
||||
"""报价**显示**小数位。
|
||||
|
||||
注意 ``price_decimal_digits`` 给的是**解码位数**(原始整数 / 10**dp),
|
||||
可转债是 4 位;但真实报价是 0.001 元的步长,第 4 位恒为 0,
|
||||
按 4 位打印只会多一个 ``0``(``128.8920``)。
|
||||
"""
|
||||
dp = price_decimal_digits(market, code)
|
||||
if str(code).startswith(_CB_PREFIX.get(int(market), ())):
|
||||
return 3
|
||||
return dp
|
||||
|
||||
|
||||
def _dp_of(sym) -> int:
|
||||
"""Symbol -> 价格小数位(非 A 股/指数一律 2 位)。"""
|
||||
if getattr(sym, "kind", "") == "cn":
|
||||
return _display_dp(sym.market, sym.code)
|
||||
return 2
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# 各子命令
|
||||
# ------------------------------------------------------------------ #
|
||||
@@ -120,24 +155,31 @@ async def cmd_quotes(td: TdxData, symbols: list[str]) -> None:
|
||||
def _print_quote(q) -> None:
|
||||
pct = (q.price / q.pre_close - 1) * 100 if q.pre_close else 0
|
||||
mkt = {31: "港股", 74: "美股"}.get(q.market, "")
|
||||
if q.market in (0, 1):
|
||||
dp = _display_dp(q.market, q.code)
|
||||
else:
|
||||
dp = getattr(q, "decimal_point", 2) or 2
|
||||
print("=" * 58)
|
||||
label = "%s %s" % (mkt, q.code) if mkt else "%s %s" % (q.code, q.server_time or datetime.now().strftime("%H:%M:%S"))
|
||||
print(" %s" % label)
|
||||
print(" 最新 %-10.2f 涨跌 %+8.2f (%+.2f%%)" % (q.price, q.price - q.pre_close, pct))
|
||||
print(" 今开 %-10.2f 最高 %.2f 最低 %.2f" % (q.open, q.high, q.low))
|
||||
print(" 昨收 %-10.2f 成交量 %s 成交额 %s" % (q.pre_close, _fmt_num(q.vol), _fmt_num(q.amount)))
|
||||
print((" 最新 %-10.*f 涨跌 %+8.*f (%+.2f%%)" % (dp, q.price, dp, q.price - q.pre_close, pct)))
|
||||
print((" 今开 %-10.*f 最高 %.*f 最低 %.*f" % (dp, q.open, dp, q.high, dp, q.low)))
|
||||
print((" 昨收 %-10.*f 成交量 %s 成交额 %s" % (dp, q.pre_close, _fmt_num(q.vol), _fmt_num(q.amount))))
|
||||
if q.bid and q.ask:
|
||||
print(" " + "-" * 56)
|
||||
pf = _price_fmt(dp)
|
||||
for i in range(max(len(q.bid), len(q.ask))):
|
||||
b = q.bid[i] if i < len(q.bid) else (0, 0)
|
||||
a = q.ask[i] if i < len(q.ask) else (0, 0)
|
||||
print(" 买%-1d %8.2f %10s | 卖%-1d %8.2f %10s" % (
|
||||
print((" 买%-1d " + pf + " %10s | 卖%-1d " + pf + " %10s") % (
|
||||
i + 1, b[0], _fmt_num(b[1]), i + 1, a[0], _fmt_num(a[1])))
|
||||
print()
|
||||
|
||||
|
||||
async def cmd_kline(td: TdxData, sym: str, period: KlinePeriod, count: int,
|
||||
start: int, adjust: Adjust) -> None:
|
||||
s = parse_symbol(sym)
|
||||
pf = _price_fmt(_dp_of(s))
|
||||
bars = await td.get_bars(sym, period, start=start, count=count, adjust=adjust)
|
||||
print("%s K线 (%s, %d根):" % (sym, period.name, len(bars)))
|
||||
print(" %-12s %8s %8s %8s %8s %12s" % ("日期", "开盘", "最高", "最低", "收盘", "成交量"))
|
||||
@@ -145,30 +187,38 @@ async def cmd_kline(td: TdxData, sym: str, period: KlinePeriod, count: int,
|
||||
dt = "%04d-%02d-%02d" % (b.year, b.month, b.day)
|
||||
if b.hour:
|
||||
dt += " %02d:%02d" % (b.hour, b.minute)
|
||||
print(" %-12s %8.2f %8.2f %8.2f %8.2f %12s" % (
|
||||
print((" %-12s " + " ".join([pf] * 4) + " %12s") % (
|
||||
dt, b.open, b.high, b.low, b.close, _fmt_num(b.vol)))
|
||||
|
||||
|
||||
async def cmd_transactions(td: TdxData, sym: str, count: int, date: int | None) -> None:
|
||||
s = parse_symbol(sym)
|
||||
tx = await td.get_transactions(Market(s.market), s.code, date=date, count=count)
|
||||
pf = _price_fmt(_dp_of(s))
|
||||
tx = await td.get_ticks(sym, date=date, count=count)
|
||||
print("%s 逐笔成交 (%d条):" % (sym, len(tx)))
|
||||
print(" 时间 价格 成交量 方向")
|
||||
for t in tx:
|
||||
bs = "买" if getattr(t, "bs_flag", 0) == 1 else "卖" if getattr(t, "bs_flag", 0) == 2 else "-"
|
||||
tc = getattr(t, "trade_count", 0)
|
||||
print(" %-10s %8.2f %10s %s%s" % (t.time, t.price, _fmt_num(t.vol), bs,
|
||||
" (%d笔)" % tc if tc else ""))
|
||||
if not getattr(t, "trade_count_available", True):
|
||||
tc = " (-)" # 指数/ETF/LOF:协议不提供笔数
|
||||
elif getattr(t, "trade_count", 0):
|
||||
tc = " (%d笔)" % t.trade_count
|
||||
else:
|
||||
tc = ""
|
||||
print((" %-10s " + pf + " %10s %s%s") % (
|
||||
t.time, t.price, _fmt_num(t.vol), bs, tc))
|
||||
|
||||
|
||||
async def cmd_minute(td: TdxData, sym: str, date: int | None) -> None:
|
||||
s = parse_symbol(sym)
|
||||
bars = await td.get_minute(Market(s.market), s.code, date=date)
|
||||
pf = _price_fmt(_dp_of(s))
|
||||
bars = await td.get_minutes(sym, date=date)
|
||||
print("%s 分时 (%d条):" % (sym, len(bars)))
|
||||
print(" 时间 价格 均价 成交量")
|
||||
for i, b in enumerate(bars):
|
||||
print(" %02d:%02d %8.2f %8.2f %10s" % (
|
||||
i // 60, i % 60, b.price, b.avg_price, _fmt_num(b.vol)))
|
||||
for b in bars:
|
||||
# 用 bar 自带时间:早期按序号推算会打成 00:00~03:59
|
||||
print((" %02d:%02d " + pf + " " + pf + " %10s") % (
|
||||
b.hour, b.minute, b.price, b.avg_price, _fmt_num(b.vol)))
|
||||
|
||||
|
||||
async def cmd_board(td: TdxData, args) -> None:
|
||||
|
||||
@@ -117,8 +117,8 @@ class SecurityBar:
|
||||
high: float
|
||||
low: float
|
||||
close: float
|
||||
vol: float # 股票=股 / 转债=手(与vipdoc一致)
|
||||
amount: float # 元
|
||||
vol: float # 股票=股 / 转债=手(1 手=10 张,与 vipdoc 一致)
|
||||
amount: float # 元。⚠️ amount/vol 只在股票上等于均价;转债还要再 /10(手=10张)
|
||||
year: int
|
||||
month: int
|
||||
day: int
|
||||
@@ -247,17 +247,42 @@ class TransactionRecord:
|
||||
- ``trade_count`` **不是笔数**,而是**当时均价 × 1000**(实测 tc/1000 与分时
|
||||
``avg_price`` 的差 < 1.1)。指数没有「成交笔数」概念。
|
||||
- ``bs_flag`` 语义不成立(实测返回 2,而模型只定义 0=卖 / 1=买)。
|
||||
|
||||
**场内基金同样没有笔数**(ETF / 多数 LOF,见
|
||||
:func:`~pytdxdata.symbols.has_order_count`):
|
||||
|
||||
- ``trade_count`` 是 ``价格 × 1000``(实测 ``sh510310`` 收盘 tc=3994,
|
||||
当时价 3.989 → 3989,偏差 σ≈2;货币 ETF ``sh511990`` 恒为 0)。
|
||||
- ``vol`` 仍是**手**,``bs_flag`` 正常。
|
||||
- 这类标的的 :attr:`trade_count_available` 为 ``False``,``trade_count`` 已被
|
||||
置 ``0``,不要拿去求和/求均值。
|
||||
|
||||
``trade_count_available=False`` 表示该品种**协议上就不带笔数**,
|
||||
与「本切片恰好 0 笔」不是一回事。
|
||||
|
||||
**可转债(110/111/113/118/123/127/128/132)三样都是真的**(实测 2026-10-22,
|
||||
16 只 A股可转债当日 + 8 只历史日全部自洽):
|
||||
|
||||
- ``vol`` 单位是**手**(1 手 = 10 张),与股票逐笔一致;``trade_count`` 是**真实成交笔数**
|
||||
(``sh111000`` 3631 条切片 / 20443 笔,可转债在
|
||||
:func:`~pytdxdata.symbols.has_order_count` 里判为**可用**)。
|
||||
- ``Σvol`` 与**日线** ``SecurityBar.vol`` **1:1 相等**(比值 1.000000)——
|
||||
这点和股票相反:股票逐笔是「手」、日线是「股」,Σ比值是 **0.01**。
|
||||
- ⚠️ 但**日线** ``amount / vol`` **不等于**均价,而是均价的 **10 倍**(``vol`` 的「手」= 10 张)。
|
||||
要算可转债均价须用 ``amount / vol / 10``;股票则是 ``amount / vol`` 直接就对。
|
||||
实测:``sz127025`` 逐笔 VWAP 105.9063 vs ``amount/vol`` 1059.0957,比值 10.0003。
|
||||
"""
|
||||
|
||||
market: int
|
||||
code: str
|
||||
time: str # HH:MM:SS(扩展市场为 HH:MM)
|
||||
price: float
|
||||
vol: float # 股票=手;指数=百元(成交额/100),见类注释
|
||||
trade_count: int # 股票=笔数增量;指数=均价×1000,见类注释
|
||||
vol: float # 股票/转债=手;指数=百元(成交额/100),见类注释
|
||||
trade_count: int # 股票=笔数增量;指数/基金=占位值,见类注释
|
||||
bs_flag: int # 1=主动买 0=主动卖 2=中性 5=盘后固定价(指数语义不成立)
|
||||
symbol: str = ""
|
||||
fields: dict[str, float] = field(default_factory=dict) # 扩展市场额外字段(增仓等)
|
||||
trade_count_available: bool = True # 指数/ETF/LOF 为 False(协议无笔数字段)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
@@ -268,7 +293,7 @@ class MinuteBar:
|
||||
:attr:`datetime` / :attr:`trading_date`,不要自己拿 ``hour`` 拼日期。
|
||||
"""
|
||||
price: float
|
||||
vol: float
|
||||
vol: float # 手(股票/转债同);Σ分时vol 与当日日线 vol 同口径
|
||||
avg_price: float = 0.0 # 均价(协议第2字段)
|
||||
hour: int = 0
|
||||
minute: int = 0
|
||||
|
||||
@@ -3,6 +3,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from collections.abc import Sequence
|
||||
from typing import Generic, TypeVar
|
||||
|
||||
from ..protocol.commands.base import BaseCommand
|
||||
@@ -22,6 +23,7 @@ class TdxConnection(Generic[T]):
|
||||
self._reader: asyncio.StreamReader | None = None
|
||||
self._writer: asyncio.StreamWriter | None = None
|
||||
self._last_result: T | None = None
|
||||
self._pipeline_result: list[T] = []
|
||||
|
||||
@property
|
||||
def alive(self) -> bool:
|
||||
@@ -107,6 +109,58 @@ class TdxConnection(Generic[T]):
|
||||
body = decompress_body(header, raw_body)
|
||||
self._last_result: T = cmd.parse_response(body)
|
||||
|
||||
async def execute_pipeline(
|
||||
self, cmds: Sequence[BaseCommand[T]], *, timeout: float | None = None
|
||||
) -> list[T]:
|
||||
"""单连接**流水线**:一次性发出全部请求,再按序读取全部响应。
|
||||
|
||||
通达信协议是自描述长度帧(16 字节帧头 + body),且服务器**按序**应答,
|
||||
因此可以在同一 TCP 连接上连续下发多条请求、省掉 ``(N-1)`` 次往返。
|
||||
实测 66 个「80 只报价」请求:逐条 7,372ms → 流水线 526ms(14 倍)。
|
||||
|
||||
与 :meth:`execute` 的关键差异:``execute`` 逐条严格配对,失败可小范围重试;
|
||||
流水线一旦中途失败,**响应流已错配**,剩余响应无法再与请求对齐,
|
||||
故此处强制断开连接(上层应换连接重试)。
|
||||
|
||||
参数:
|
||||
cmds: 同一连接的请求序列(通常为同一类命令的不同分片)。
|
||||
timeout: 整体超时(默认 ``self.timeout * 2``,流水线耗时随条数增长)。
|
||||
|
||||
返回:
|
||||
与 ``cmds`` **等长且同序**的结果列表。
|
||||
"""
|
||||
if not cmds:
|
||||
return []
|
||||
if not self.alive:
|
||||
raise ConnectionError(f"连接未建立或已关闭: {self.host}")
|
||||
assert self._reader is not None and self._writer is not None
|
||||
budget = timeout if timeout is not None else self.timeout * 2
|
||||
try:
|
||||
try:
|
||||
await asyncio.wait_for(self._io_pipeline(cmds), budget)
|
||||
except asyncio.TimeoutError:
|
||||
self._force_close()
|
||||
raise ConnectionError(
|
||||
f"流水线超时 {self.host} ({budget:.1f}s, {len(cmds)} 条)") from None
|
||||
except (ConnectionError, OSError) as e:
|
||||
self._force_close()
|
||||
raise ConnectionError(f"流水线错误 {self.host}: {e}") from e
|
||||
except (ConnectionError, OSError):
|
||||
raise
|
||||
return self._pipeline_result
|
||||
|
||||
async def _io_pipeline(self, cmds: Sequence[BaseCommand[T]]) -> None:
|
||||
assert self._reader is not None and self._writer is not None
|
||||
self._writer.write(b"".join(c.build_request() for c in cmds))
|
||||
await self._writer.drain()
|
||||
results: list[T] = []
|
||||
for cmd in cmds:
|
||||
header_buf = await self._recv_exact(HEADER_SIZE)
|
||||
header = parse_header(header_buf)
|
||||
raw_body = await self._recv_exact(header.zipsize)
|
||||
results.append(cmd.parse_response(decompress_body(header, raw_body)))
|
||||
self._pipeline_result: list[T] = results
|
||||
|
||||
async def _send_setup(self) -> None:
|
||||
"""按序发送3条握手命令并丢弃响应。"""
|
||||
assert self._reader is not None and self._writer is not None
|
||||
|
||||
@@ -59,6 +59,11 @@ class PooledConnection:
|
||||
self.last_used = time.monotonic()
|
||||
return await self.conn.execute(cmd)
|
||||
|
||||
async def execute_pipeline(self, cmds, *, timeout: float | None = None):
|
||||
"""在同一条连接上流水线执行多条命令(见 TdxConnection.execute_pipeline)。"""
|
||||
self.last_used = time.monotonic()
|
||||
return await self.conn.execute_pipeline(cmds, timeout=timeout)
|
||||
|
||||
async def close(self) -> None:
|
||||
self.alive = False
|
||||
try:
|
||||
@@ -273,6 +278,46 @@ class DynamicPool:
|
||||
]
|
||||
return random.choice(alive) if alive else None
|
||||
|
||||
async def warm(self, n: int) -> None:
|
||||
"""并发预建连接至 ``n`` 条(最多到 ``max_size``)。
|
||||
|
||||
:meth:`acquire` 建连时持有池锁,**一次只能建一条**——22 路并发首次
|
||||
调用会排队 22×握手时间(实测全市场首拉多花 5~6s)。大批量请求前先调
|
||||
本方法把连接并发建好,首拉只多付一次握手。
|
||||
|
||||
失败静默忽略:缺的连接后续由 ``acquire`` 按需补齐。
|
||||
"""
|
||||
if self._closed:
|
||||
return
|
||||
need = min(n, self.max_size) - self._total
|
||||
if need <= 0:
|
||||
return
|
||||
|
||||
# 用局部计数挑 host,避免并发建连时反复选中同一台
|
||||
counts = dict(self._server_count)
|
||||
ranked = self.health.rank(self.servers)
|
||||
hosts: list[str] = []
|
||||
for _ in range(need):
|
||||
host = next((h for h in ranked
|
||||
if not self.health.in_cooldown(h)
|
||||
and counts.get(h, 0) < self.per_server_cap), None)
|
||||
if host is None:
|
||||
break
|
||||
hosts.append(host)
|
||||
counts[host] = counts.get(host, 0) + 1
|
||||
if not hosts:
|
||||
return
|
||||
|
||||
results = await asyncio.gather(
|
||||
*(self._new_conn(h) for h in hosts), return_exceptions=True)
|
||||
async with self._lock:
|
||||
for pc in results:
|
||||
if isinstance(pc, PooledConnection):
|
||||
self._total += 1
|
||||
self.stats.total = self._total
|
||||
if not self._closed and pc.alive:
|
||||
self._idle.append(pc)
|
||||
|
||||
async def release(self, pc: PooledConnection, *, broken: bool = False) -> None:
|
||||
if self.stats.borrowed > 0:
|
||||
self.stats.borrowed -= 1
|
||||
|
||||
@@ -5,6 +5,7 @@ import struct
|
||||
|
||||
from ..._binary import unpack_from
|
||||
from ...models import TransactionRecord
|
||||
from ...symbols import has_order_count
|
||||
from ..datetime_ import get_time
|
||||
from ..mac_frame import build_mac_request
|
||||
from ..price import get_price
|
||||
@@ -32,6 +33,8 @@ class GetTransactionDataCmd(BaseCommand[list[TransactionRecord]]):
|
||||
pos = 2
|
||||
last_price = 0
|
||||
records: list[TransactionRecord] = []
|
||||
code = self.code.decode("utf-8")
|
||||
has_tc = has_order_count(self.market, code)
|
||||
for _ in range(num):
|
||||
hour, minute, pos = get_time(body, pos)
|
||||
price_diff, pos = get_price(body, pos)
|
||||
@@ -41,10 +44,12 @@ class GetTransactionDataCmd(BaseCommand[list[TransactionRecord]]):
|
||||
_unknown, pos = get_price(body, pos)
|
||||
last_price += price_diff
|
||||
records.append(TransactionRecord(
|
||||
market=self.market, code=self.code.decode("utf-8"),
|
||||
market=self.market, code=code,
|
||||
time=f"{hour:02d}:{minute:02d}:00",
|
||||
price=last_price / 100.0, vol=float(vol),
|
||||
trade_count=int(num_orders), bs_flag=bs_flag,
|
||||
# 指数/ETF/LOF 该字段是「价格×1000」占位,不是笔数
|
||||
trade_count=int(num_orders) if has_tc else 0,
|
||||
bs_flag=bs_flag, trade_count_available=has_tc,
|
||||
))
|
||||
return records
|
||||
|
||||
@@ -68,6 +73,7 @@ class GetHistoryTransactionDataCmd(BaseCommand[list[TransactionRecord]]):
|
||||
pos = 6 # 2(num) + 4(skip)
|
||||
last_price = 0
|
||||
records: list[TransactionRecord] = []
|
||||
code = self.code.decode("utf-8")
|
||||
for _ in range(num):
|
||||
hour, minute, pos = get_time(body, pos)
|
||||
price_diff, pos = get_price(body, pos)
|
||||
@@ -76,10 +82,11 @@ class GetHistoryTransactionDataCmd(BaseCommand[list[TransactionRecord]]):
|
||||
_unknown, pos = get_price(body, pos)
|
||||
last_price += price_diff
|
||||
records.append(TransactionRecord(
|
||||
market=self.market, code=self.code.decode("utf-8"),
|
||||
market=self.market, code=code,
|
||||
time=f"{hour:02d}:{minute:02d}:00",
|
||||
price=last_price / 100.0, vol=float(vol),
|
||||
trade_count=0, bs_flag=bs_flag, # 历史无笔数
|
||||
trade_count=0, bs_flag=bs_flag,
|
||||
trade_count_available=False, # 历史通道整条链路都没有笔数字段
|
||||
))
|
||||
return records
|
||||
|
||||
@@ -108,6 +115,7 @@ class MacTransactionCmd(BaseCommand[list[TransactionRecord]]):
|
||||
def parse_response(self, body: bytes) -> list[TransactionRecord]:
|
||||
(count,) = unpack_from("<H", body, 29, "mac transaction count")
|
||||
records: list[TransactionRecord] = []
|
||||
has_tc = has_order_count(self.market, self.code)
|
||||
for i in range(count):
|
||||
offset = 39 + i * 18
|
||||
try:
|
||||
@@ -124,6 +132,8 @@ class MacTransactionCmd(BaseCommand[list[TransactionRecord]]):
|
||||
market=self.market, code=self.code,
|
||||
time=f"{h:02d}:{m:02d}:{s:02d}",
|
||||
price=price, vol=float(volume),
|
||||
trade_count=int(trade_count), bs_flag=bs_flag,
|
||||
# 指数/ETF/LOF 该字段是「价格×1000」占位,不是笔数
|
||||
trade_count=int(trade_count) if has_tc else 0,
|
||||
bs_flag=bs_flag, trade_count_available=has_tc,
|
||||
))
|
||||
return records
|
||||
@@ -238,3 +238,53 @@ def is_cn_index(market: int, code: str) -> bool:
|
||||
or (code.startswith("000") and int(market) == int(Market.SH))
|
||||
or (code.startswith("399") and int(market) == int(Market.SZ))
|
||||
)
|
||||
|
||||
|
||||
def has_order_count(market: int, code: str) -> bool:
|
||||
"""该标的的逐笔 ``trade_count`` 是否是**真实成交笔数**。
|
||||
|
||||
通达信对**指数**和**多数场内基金**不提供笔数,但那 4 字节里仍塞了值:
|
||||
|
||||
- 指数:``当时均价 × 1000``(见 :class:`~pytdxdata.models.TransactionRecord`)
|
||||
- ETF / LOF:``价格 × 1000``;货币 ETF(如 ``sh511990``)恒为 ``0``
|
||||
|
||||
直接当笔数用会得到 3994 这种荒谬值(实测 ``sh510310``)。
|
||||
|
||||
实测覆盖(2026-10-22,按代码前缀各取样 1~7 只):
|
||||
|
||||
================== ========================== ==========
|
||||
品种 前缀 笔数可用
|
||||
================== ========================== ==========
|
||||
沪股票/科创 600/601/603/605/688/689 是
|
||||
沪 LOF 501/502/506 是
|
||||
沪 ETF/LOF 51/52/56/58 **否**
|
||||
沪指数 000/88 否
|
||||
沪可转债 110/111/113/118 是
|
||||
沪 EB(可交换债) 132 是
|
||||
沪逆回购 204 是
|
||||
深股票 000/001/002/003/300/301 是
|
||||
深 ETF 159 **否**
|
||||
深 LOF/基金 15/16/17/18 **否**
|
||||
深指数 399 否
|
||||
深可转债 123/127/128 是
|
||||
================== ========================== ==========
|
||||
|
||||
``501x/502x/506x``(沪市 LOF)实测**有**真实笔数,所以不能用「沪市 5 开头
|
||||
一律当基金」的粗规则;必须按上面这组前缀精确判定。
|
||||
|
||||
可转债(沪 110/111/113/118/132、深 123/127/128)**走的是兜底分支**——
|
||||
它们既不在沪市 ``51/52/56/58`` 里,也不在深市 ``159/15/16/17/18`` 里,
|
||||
所以两个分支都落到「是」。这不是巧合而是设计:兜底按「**排除法**」认定
|
||||
基金/指数,其余一律当普通品种。实测验证(2026-10-22,16 只当日 + 8 只历史日):
|
||||
笔数合计与日线量级吻合(``sh111000`` 3631 条切片 / 20443 笔,
|
||||
``sz128136`` 1939 条 / 14593 笔),且 ``Σvol`` 与日线 ``vol`` 比值 **1.000000**。
|
||||
"""
|
||||
code = str(code)
|
||||
if is_cn_index(market, code):
|
||||
return False
|
||||
m = int(market)
|
||||
if m == int(Market.SH):
|
||||
return code[:2] not in ("51", "52", "56", "58")
|
||||
if m == int(Market.SZ):
|
||||
return not (code[:3] == "159" or code[:2] in ("15", "16", "17", "18"))
|
||||
return True
|
||||
Reference in new issue
Block a user