feat: DataFrame 直出(df="polars"/"pandas")+ 0.7.0 (#12)
* feat: DataFrame 直出(df="polars"/"pandas")+ 版本 0.7.0 TdxData 新增构造参数 df: - None(默认)→ 行为与 0.6.x 完全一致,返回 list[dataclass] - "polars"/"pandas" → 42 个表格型接口直接返回 DataFrame 转换层 src/pytdxdata/frames.py: - 列计划按模型缓存,列式采集(不做逐行 asdict) - 时间列替换:year/month/day/hour/minute → datetime + trading_date;Xdxr 给 date - 嵌套定型:bid/ask 展开 10 列、fields 字典 JSON 化、codes 保持列表 - 容器展开:get_indicators 合并指标、get_index_trade_stats 展开 minutes、 get_kline_batch/get_minute_batch 纵向拼接 - ContextVar 层次标记:批量/别名方法内层不二次转换,并发互不干扰 - 后端惰性加载:缺包抛 FrameBackendError(带安装命令),构造时即失败 依赖:新增 extras pandas/polars/dataframe,核心包仍零依赖;dev 组加 pandas+polars 测试:tests/test_frames.py 40 项(双后端×双格式);全量 401 项通过 文档:新增 docs/dataframe.md,同步 API_REFERENCE/SKILL/llms.txt/README/mkdocs * perf(kline): 按日期取K线贴最新时走倒推,1分钟 4.2s → 82ms 问题:start_date/end_date 一律走「探总根数 + 二分定位」。1 分钟 K 线历史 二十万根量级 = 6 次翻倍探测 + 2 次方向探测 + 两次各 ~18 次跳转,共 36+ 次 串行往返。「取今天」这个最常见的调用要 4.2 秒。 改法(_bars_backtrack 快路径): - 右端离最新不超过 _BACKTRACK_GRACE(10 天) 时,直接 start=0, count=N 拿最新 N 根再裁剪;N 按时间跨度估(_BARS_PER_DAY),一天内一次命中, 不够才翻倍,放大到 _BACKTRACK_MAX_BARS 仍不够则回落二分 - 历史中段仍走二分,结果口径完全一致(已逐条比对基准) 两个关键细节: - 偏移方向是通道级属性而非标的属性,首次探测后缓存在实例上 (_kline_dir_cache)——全市场 5,225 只批量只付一次探测成本 - K线 hour/minute 是周期**结束**时刻,最新一根可能比 now() 略大(14:29 那根还在走),「右端贴最新」必须留周期容差,否则 end_date=now() 被误判 成历史中段 实测(1 分钟,冷启动): 取今天 4,208ms → 82ms 2天前~now 2,211ms → 75ms 最近3天 2,000ms+ → 239ms 历史中段 1,512ms → 285ms 全市场5225只取今天 32.1s(1,140,699 根,精确当天口径) 注:route_for 让 MIN_1 一律走 MAC(0x122E),所以按通道做方向缓存而不是按 「标准/扩展」写死分支。 测试:新增 tests/test_kline_date.py 6 项(倒推次数上界、结束时刻容差、 中段回落二分、未来区间、反向通道、与 count 路径口径一致);全量 407 项通过 * perf(kline): 全市场1分钟K线改多连接流水线抓取(43.6s → 5~7s,盘中 2s) 瓶颈其实是**往返次数**不是带宽:单只K线一请求一标的,串行 33ms/只 ≈ 1 个 RTT。 同一台服务器 16 连接 × 深度 8 实测能跑 5.2 MB/s(此前「单 IP 限流 1.5MB/s」 是浅流水下的假象)。 - get_bars 新增 pipeline 参数(None=按单请求字节自适应 8~32,1=关闭走逐只) - 多标的(≥16 只)走 _bulk_bars:多连接 work-stealing + 单连接 execute_pipeline - 只用实测最快的 12 台:池里 24 台全用反而被慢节点拖长尾 - 盘中按交易时段只拉已走过的根数(10:00 → 35 根,数据量差 7 倍) - 标准通道(日线不复权)同样走批量:全市场 49~72s → 13s - 修 pipeline 失败时队列剩余标的静默丢失(实测 20 只只回 16 只) - 修 test_future_range_returns_empty_quickly 时间炸弹(固定日期会漂进 _BACKTRACK_GRACE) - conftest 关闭后台测速 task,避免污染「往返次数」类断言 实测(5,225 只):count=2 → 2.7s;count=35 → 2.2s;count=240 → 6.9s。 逐根一致性 30/30 相同。新增 tests/test_bulk_kline.py 12 项,合计 419 项全绿。 * test: 批量K线测试的时间锚点改用真实 now(CI 是 UTC,固定日期会漂到别的时段分支) * fix(ex): EX 通道 K 线回填成交额与结算价(0x23FF 槽位按市场分两类) 报告:EX 通道 get_bars 返回的 SecurityBar.amount 恒为 0.0。 定位属实,但实际情况比报告更细 —— 32B 记录里那 28B 行情只有**一个** 4B 槽位 可复用,语义按市场分两类(全市场实测,用 成交额/(close×vol) 逐条校验): | 市场 | 槽位5 | 槽位6 | 槽位7 | |---|---|---|---| | 港股主板31/港股基金49 | 成交额(f) | 量(手=100股) | 未知 | | 港股创业板48/新三板44/美股74 | 成交额(f) | 量(股) | 0 | | 期货/期权 7/8/9/47… | 持仓量(I) | 量 | 结算价(f) | 改前 amount 硬编码 0.0、第 7 个 float 用 `_` 丢弃 —— 于是港股/美股/新三板 明明帧内有成交额却全为 0,期货期权的结算价也拿不到。 - 新增 SecurityBar.settle(结算价,float|None):仅 EX 期货/期权日线有值, 分钟线该槽位恒 0 已归一化为 None,其它通道恒 None。 - 权益类市场回填真实 amount;港股主板/基金的 vol 单位是手(=100股), amount/vol 是均价的 100 倍,已写进模型注释与量纲表。 - 期货/期权的 amount 仍是 0.0(帧内确实没有成交额,协议限制), 已在 SecurityBar / API_REFERENCE / llms.txt / SKILL.md 写清, 避免使用者再按旧文档判断成调用方 bug。 - 港股指数27/中证指数62 该槽位量级对不上(既非额也非结算价),保守保持 0.0。 - SecurityBar.__setstate__ 兜底旧磁盘缓存:缓存键无 schema 版本,升级后读旧 pickle 缺 settle 槽位,默认反序列化不补默认值 → AttributeError,已容错; EX K 线缓存键额外加 @v2,让 amount 恒 0 的旧缓存自然失效。 - K 线 DataFrame 因此多一列 settle(0.7.0 未发布,列清单同步更新)。 测试:tests/test_ex_bars_fields.py 16 项(解析层双布局、日内 settle 归零、 API 透传、旧 pickle 兼容);全套 435 项通过。 * docs: 0.7.0 全站口径同步(DataFrame 出口 + EX 成交额限制 + 构造参数表) - 首页:特性表补「DataFrame 直出」与 K 线批量性能,导航新增 DataFrame 卡片, 统一方法数按实测修正为 35(+33 别名 = 68 个公开方法) - 安装页:补 extras(polars/pandas/dataframe)与 df 用法;构造参数表补 df / std_pool_cap / ex_pool_cap,并把 pool_max 默认值从 12 修正为 24(实际值) - 场景示例:`转成 pandas` 一节改为先推 df= 开关,手动转换降级为局部场景 - 数据模型页:不再说「刻意不提供 DataFrame」,改为 dataclass 与 DataFrame 是同一份数据、两种容器 - AI 接入页:返回值约定补 `df=` 开关的说法 - README:K 线列注释补 settle;已知限制补 EX 通道成交额/结算价的协议口径 - CHANGELOG:新增「升级」小节(升级命令、旧缓存无需清理) --------- Co-authored-by: openbot-coder <openbot-coder@users.noreply.github.com>
This commit is contained in:
1 parent
2cf3657f4b
commit
04e0c57569
24 files changed
+3022
-496
No files matched your search
@@ -18,7 +18,7 @@ from .models import (
|
||||
XdxrRecord,
|
||||
)
|
||||
|
||||
__version__ = "0.6.9"
|
||||
__version__ = "0.7.0"
|
||||
__all__ = [
|
||||
"TdxData",
|
||||
"Market",
|
||||
|
||||
+414
-13
@@ -27,7 +27,7 @@ import time
|
||||
import warnings
|
||||
from collections import deque
|
||||
from collections.abc import Callable
|
||||
from datetime import date as _date, datetime
|
||||
from datetime import date as _date, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Sequence
|
||||
|
||||
@@ -38,6 +38,7 @@ from .cache.manager import CacheManager
|
||||
from .cache.memory import MemoryCache
|
||||
from .codec.payload import pack_payload, unpack_payload
|
||||
from .config import EX_HANDSHAKES, get_ex_hosts, get_mac_hosts, get_standard_hosts
|
||||
from .frames import framed, get_backend
|
||||
from .models import (
|
||||
Adjust,
|
||||
AuctionItem,
|
||||
@@ -145,6 +146,51 @@ _KLINE_MAX_BARS: dict[KlinePeriod, int] = {
|
||||
KlinePeriod.MONTH: 500,
|
||||
}
|
||||
|
||||
# 每交易日大约多少根(倒推时按时间跨度估要拉多少,估小了会多跑一轮)
|
||||
_BARS_PER_DAY: dict[KlinePeriod, float] = {
|
||||
KlinePeriod.MIN_1: 240,
|
||||
KlinePeriod.MIN_5: 48,
|
||||
KlinePeriod.MIN_15: 16,
|
||||
KlinePeriod.MIN_30: 8,
|
||||
KlinePeriod.MIN_60: 4,
|
||||
KlinePeriod.DAY: 1,
|
||||
KlinePeriod.WEEK: 0.2,
|
||||
KlinePeriod.MONTH: 0.05,
|
||||
}
|
||||
_BACKTRACK_MIN_BARS = 240 # 倒推起始根数:一天 1 分钟正好 240,一次命中
|
||||
_BACKTRACK_MAX_BARS = 20_000 # 单次别拉太多;超了回落到二分
|
||||
_BACKTRACK_ROUNDS = 5 # 最多放大 5 轮(240→7680)
|
||||
_BACKTRACK_GRACE = timedelta(days=10) # 右端离最新多远还算「贴着最新」(覆盖长假)
|
||||
|
||||
# 一根K线跨多长时间。K线时间戳标的是**该周期的结束时刻**,所以最新一根可能
|
||||
# 比 datetime.now() 略大(14:29 那根此刻还在走)——判断「右端贴最新」要给这个容差。
|
||||
_BAR_SPAN: dict[KlinePeriod, timedelta] = {
|
||||
KlinePeriod.MIN_1: timedelta(minutes=1),
|
||||
KlinePeriod.MIN_5: timedelta(minutes=5),
|
||||
KlinePeriod.MIN_15: timedelta(minutes=15),
|
||||
KlinePeriod.MIN_30: timedelta(minutes=30),
|
||||
KlinePeriod.MIN_60: timedelta(hours=1),
|
||||
KlinePeriod.DAY: timedelta(days=1),
|
||||
KlinePeriod.WEEK: timedelta(days=7),
|
||||
KlinePeriod.MONTH: timedelta(days=31),
|
||||
}
|
||||
|
||||
# ---- K线批量流水线(get_bars 多标的路径)----
|
||||
# 单只K线是「一请求一标的」,全市场 5,000 只就是 5,000 次往返。实测单连接
|
||||
# 串行 33ms/只 ≈ 1 个 RTT,**瓶颈是往返不是带宽**;同连接连续下发 pipe 条请求
|
||||
# (协议帧自描述 + 服务器按序应答)能把往返折叠掉。实测 24 台 × 8 流水:
|
||||
# 全市场 1 分钟K线取当天 43.6s → 5.1s。
|
||||
_BULK_MIN_SYMS = 16 # 低于此数量走逐只路径(流水线收益抵不过批量开销)
|
||||
_BULK_PIPE_MIN = 8 # 自适应深度下限
|
||||
_BULK_PIPE_MAX = 32 # 自适应深度上限(再深服务器会攒够一批才回,反而慢)
|
||||
_BULK_MAX_COUNT = 700 # MAC 单请求返回上限;超过只能逐只分页
|
||||
# 最多用几台服务器:池里 24 台全用反而更慢 —— 慢节点会成为长尾(最长的那台
|
||||
# 拖住整轮),实测 count=2 全市场 24 台 2.95s / 只取最快 12 台 1.45s。
|
||||
_BULK_MAX_CONNS = 12
|
||||
# 交易时段(分钟):上午 9:30-11:30、下午 13:00-15:00,共 240 分钟
|
||||
_AM_OPEN, _AM_CLOSE = 9 * 60 + 30, 11 * 60 + 30
|
||||
_PM_OPEN, _PM_CLOSE = 13 * 60, 15 * 60
|
||||
|
||||
# 自定义字段报价(0x122B/0x122C)基础字段位:昨收/开/高/低/现价/量/量比/额
|
||||
_QUOTE_BITS = [0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07]
|
||||
_QUOTE_PAGE = 80 # 0x122B/0x122C 单次上限 80 只
|
||||
@@ -166,6 +212,65 @@ _list_unsupported_warned: set[int] = set() # 同一市场只 WARNING 一次
|
||||
_HK_MARKETS = frozenset({27, 31, 48, 49, 71, 98})
|
||||
|
||||
|
||||
def _elapsed_trading_minutes(now: datetime) -> int:
|
||||
"""今天到现在为止已过去的**交易分钟数**(0~240)。
|
||||
|
||||
只按钟点算,不区分节假日(非交易日时区间裁剪会把这些根全滤掉,不影响正确性);
|
||||
盘中调用是常态,10:00 只需 30 根而不是 240 根 —— 数据量直接差 8 倍。
|
||||
"""
|
||||
m = now.hour * 60 + now.minute
|
||||
if m < _AM_OPEN:
|
||||
return 0
|
||||
if m <= _AM_CLOSE:
|
||||
return m - _AM_OPEN
|
||||
if m < _PM_OPEN:
|
||||
return 120 # 午休:上午已走完
|
||||
if m <= _PM_CLOSE:
|
||||
return 120 + (m - _PM_OPEN)
|
||||
return 240 # 收盘后
|
||||
|
||||
|
||||
def _bars_for_range(period: KlinePeriod, start_dt: datetime,
|
||||
now: datetime) -> int:
|
||||
"""估算「从 ``start_dt`` 到最新」需要拉多少根(倒推初值)。"""
|
||||
per_day = _BARS_PER_DAY.get(period, 1)
|
||||
if start_dt.date() == now.date():
|
||||
# 当天:按已过交易分钟数折算,别把一整天都拉下来(10:00 只需 30 根而非 240)
|
||||
frac = _elapsed_trading_minutes(now) / 240
|
||||
return max(1, int(per_day * frac) + 5)
|
||||
days = max(0.0, (now - start_dt).total_seconds() / 86400.0)
|
||||
return max(1, int(days * per_day * 1.6))
|
||||
|
||||
|
||||
def _auto_pipeline(count: int, pipeline: int | None) -> int:
|
||||
"""流水深度:默认按「在途字节数 ≈60KB」自适应,夹在 8~32。
|
||||
|
||||
轮数 = 请求数 / (连接数 × 深度),深度越深往返折叠得越彻底;但请求太深时
|
||||
服务器会把整批处理完才回,实测 24 台 depth=64 反而比 8 慢一倍(5.79s vs 2.95s)。
|
||||
每根的应答固定 32 字节,所以「取 240 根」天然只用得着浅流水,
|
||||
「只取最新 2 根」则要用满深流水。
|
||||
"""
|
||||
if pipeline is not None:
|
||||
return max(2, int(pipeline))
|
||||
per_req = max(1, count * 32 + 33)
|
||||
return max(_BULK_PIPE_MIN, min(_BULK_PIPE_MAX, 60_000 // per_req))
|
||||
|
||||
|
||||
def _norm_range(start_date, end_date) -> tuple[datetime | None, datetime | None]:
|
||||
"""日期参数归一化:``date`` → 当日 00:00 / 23:59:59.999999。"""
|
||||
start_dt = None
|
||||
if start_date is not None:
|
||||
start_dt = (start_date if isinstance(start_date, datetime)
|
||||
else datetime.combine(start_date, datetime.min.time()))
|
||||
end_dt = None
|
||||
if end_date is not None:
|
||||
end_dt = (end_date if isinstance(end_date, datetime)
|
||||
else datetime.combine(end_date, datetime.max.time()))
|
||||
if end_dt.time() == datetime.min.time():
|
||||
end_dt = end_dt.replace(hour=23, minute=59, second=59, microsecond=999999)
|
||||
return start_dt, end_dt
|
||||
|
||||
|
||||
def _fields_tag(bits: Sequence[int] | None) -> str:
|
||||
"""字段位集合 -> 缓存键片段。"""
|
||||
if not bits:
|
||||
@@ -249,7 +354,19 @@ class TdxData:
|
||||
mac_pool_cap: int = 4,
|
||||
std_pool_cap: int = 4,
|
||||
ex_pool_cap: int = 2,
|
||||
df: str | None = None,
|
||||
) -> None:
|
||||
# df=None(默认)保持 0.6.x 的行为:表格型方法一律返回 list[dataclass]。
|
||||
# df="polars" / df="pandas" 时这些方法直接返回对应后端的 DataFrame。
|
||||
# 这里早失败一次:名字非法抛 ValueError,包没装抛 FrameBackendError(带安装命令)。
|
||||
if df is not None:
|
||||
get_backend(df)
|
||||
self._df_backend = df
|
||||
|
||||
# K线偏移方向是**通道级属性**(不是每只标的的属性),首次探测后实例内复用:
|
||||
# 全市场 5,000 只各探 2 次往返 = 上万次请求,纯属浪费。
|
||||
self._kline_dir_cache: dict[str, bool] = {}
|
||||
|
||||
self.port = port
|
||||
self.timeout = timeout
|
||||
self.default_adjust = default_adjust
|
||||
@@ -326,6 +443,7 @@ class TdxData:
|
||||
# 一、标的清单
|
||||
# ================================================================== #
|
||||
|
||||
@framed
|
||||
async def get_universe(self, market: str | None = None) -> list[SecurityInfo]:
|
||||
"""标的清单(A股证券列表 + 扩展市场商品列表),一次覆盖所有市场。
|
||||
|
||||
@@ -472,6 +590,7 @@ class TdxData:
|
||||
# 二、报价
|
||||
# ================================================================== #
|
||||
|
||||
@framed
|
||||
async def get_quotes(self, symbols: str | Sequence[str], *,
|
||||
fields: Sequence[int] | None = None,
|
||||
concurrency: int | None = None,
|
||||
@@ -649,6 +768,7 @@ class TdxData:
|
||||
await self._std_pool.release(pc)
|
||||
return rows
|
||||
|
||||
@framed
|
||||
async def get_snapshot(self, symbols: str | Sequence[str]) -> list[SymbolSnapshot]:
|
||||
"""个股特征快照(当日/换手/活跃度等,仅 A股)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -668,6 +788,7 @@ class TdxData:
|
||||
out.append(rec)
|
||||
return out
|
||||
|
||||
@framed
|
||||
async def get_auction(self, symbols: str | Sequence[str]) -> list[AuctionItem]:
|
||||
"""集合竞价(仅 A股)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -680,6 +801,7 @@ class TdxData:
|
||||
out.extend(items)
|
||||
return out
|
||||
|
||||
@framed
|
||||
async def get_capital_flow(self, symbols: str | Sequence[str]) -> list[CapitalFlow]:
|
||||
"""个股资金流向(仅 A股)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -696,6 +818,7 @@ class TdxData:
|
||||
# 三、K线
|
||||
# ================================================================== #
|
||||
|
||||
@framed
|
||||
async def get_bars(
|
||||
self,
|
||||
symbols: str | Sequence[str],
|
||||
@@ -707,6 +830,7 @@ class TdxData:
|
||||
start_date: datetime | None = None,
|
||||
end_date: datetime | None = None,
|
||||
concurrency: int = 6,
|
||||
pipeline: int | None = None,
|
||||
) -> list[SecurityBar]:
|
||||
"""K线(多标的并发,自动判指数、自动复权选路、自动跨市场)。
|
||||
|
||||
@@ -718,12 +842,30 @@ class TdxData:
|
||||
adjust: 复权方式(默认用实例 ``default_adjust``);``QFQ``/``HFQ`` 自动走 MAC 通道。
|
||||
start_date / end_date: 按日期区间取(与 start/count 互斥,优先日期过滤)。
|
||||
只给 ``start_date`` 即增量更新语义(取该日期之后的全部K线)。
|
||||
concurrency: 逐只路径的并发数(仅在对小批量/标准通道生效时使用)。
|
||||
pipeline: **批量路径**每条连接同时在途的请求数。``None``(默认)按单请求
|
||||
字节数自适应(约 60KB 在途,夹在 8~32);传 ``1`` 关闭流水线走逐只。
|
||||
批量路径把 N 次往返折叠成 ``N/pipeline`` 次 —— 全市场 5,000 只取当天
|
||||
1 分钟K线从 43.6s 降到 5s 级。
|
||||
|
||||
返回:
|
||||
``list[SecurityBar]``,按入参标的顺序拼接,每根带 ``symbol``;组内按时间正序。
|
||||
|
||||
说明:
|
||||
批量路径只在 A股/指数(非扩展市场)且区间贴着最新时启用;历史中段
|
||||
仍需逐只二分定位,会回落到逐只路径(结果一致,只是慢一些)。
|
||||
"""
|
||||
period = KlinePeriod(int(period))
|
||||
syms = parse_symbols(symbols)
|
||||
|
||||
# 批量路径:一次请求一标的,逐只走就是 N 次往返;多连接流水线能把往返压掉
|
||||
if len(syms) >= _BULK_MIN_SYMS and pipeline != 1:
|
||||
bulk = await self._bars_bulk(
|
||||
syms, period, start=start, count=count, adjust=adjust,
|
||||
start_date=start_date, end_date=end_date, pipeline=pipeline)
|
||||
if bulk is not None:
|
||||
return bulk
|
||||
|
||||
sem = asyncio.Semaphore(max(1, concurrency))
|
||||
|
||||
async def one(sym: Symbol) -> list[SecurityBar]:
|
||||
@@ -735,6 +877,140 @@ class TdxData:
|
||||
groups = await asyncio.gather(*(one(s) for s in syms))
|
||||
return [b for g in groups for b in g]
|
||||
|
||||
async def _bars_bulk(
|
||||
self, syms: Sequence[Symbol], period: KlinePeriod, *,
|
||||
start: int, count: int, adjust: Adjust | None,
|
||||
start_date: datetime | None, end_date: datetime | None,
|
||||
pipeline: int | None,
|
||||
) -> list[SecurityBar] | None:
|
||||
"""多标的批量取K线(多连接流水线)。返回 ``None`` = 不适用,调用方回退逐只。
|
||||
|
||||
不适用只有两种情况:含扩展市场标的、需要历史中段二分定位。
|
||||
"""
|
||||
if any(s.is_ex for s in syms):
|
||||
return None
|
||||
adj = self.default_adjust if adjust is None else adjust
|
||||
pool, page = self._bulk_channel(period, adj)
|
||||
|
||||
def make_cmd(s: Symbol, o: int, c: int):
|
||||
if pool is self._mac_pool:
|
||||
return MacKlineCmd(s.market, s.code, period, adj, o, c)
|
||||
if is_cn_index(s.market, s.code):
|
||||
return GetIndexBarsCmd(s.market, s.code, period, o, c)
|
||||
return GetSecurityBarsCmd(s.market, s.code, period, o, c)
|
||||
|
||||
if start_date is None and end_date is None:
|
||||
pipe = _auto_pipeline(min(count, page), pipeline)
|
||||
got = await self._bulk_bars(pool, make_cmd, syms, start, count, page, pipe,
|
||||
period=period, adj=adj)
|
||||
return [b for s in syms for b in (got.get(s.text) or [])]
|
||||
|
||||
start_dt, end_dt = _norm_range(start_date, end_date)
|
||||
now = datetime.now()
|
||||
if end_dt is not None and end_dt + _BACKTRACK_GRACE < now:
|
||||
return None # 历史中段:得先探总根数再二分,批量帮不上
|
||||
if start_dt is None:
|
||||
need = _BACKTRACK_MIN_BARS
|
||||
else:
|
||||
if start_dt > now + _BAR_SPAN.get(period, timedelta(days=1)):
|
||||
return [] # 整段区间在未来
|
||||
need = _bars_for_range(period, start_dt, now)
|
||||
# 偏移方向是通道级属性:探一次(结果按通道缓存,全市场只付一次代价)
|
||||
if not await self._kline_newest_at_zero(syms[0], period, adj):
|
||||
return None
|
||||
|
||||
pipe = _auto_pipeline(min(need, page), pipeline)
|
||||
got = await self._bulk_bars(pool, make_cmd, syms, 0, need, page, pipe,
|
||||
period=period, adj=adj)
|
||||
out: list[SecurityBar] = []
|
||||
for s in syms:
|
||||
bars = got.get(s.text) or []
|
||||
if start_dt is not None:
|
||||
bars = [b for b in bars if b.datetime >= start_dt]
|
||||
if end_dt is not None:
|
||||
bars = [b for b in bars if b.datetime <= end_dt]
|
||||
out.extend(bars)
|
||||
return out
|
||||
|
||||
def _bulk_channel(self, period: KlinePeriod, adj: Adjust):
|
||||
"""批量路径用哪个池 / 单请求最多几根。"""
|
||||
if route_for("kline", adjust=adj, period=period) == "mac":
|
||||
return self._mac_pool, _BULK_MAX_COUNT
|
||||
return self._std_pool, KLINE_PAGE
|
||||
|
||||
async def _bulk_bars(
|
||||
self, pool, make_cmd, syms: Sequence[Symbol],
|
||||
start: int, count: int, page: int, pipeline: int, *,
|
||||
period: KlinePeriod, adj: Adjust,
|
||||
) -> dict[str, list[SecurityBar]]:
|
||||
"""多连接 + 单连接流水线抓K线,按 ``symbol.text`` 归集。
|
||||
|
||||
为什么快:单只一请求,串行时候每只固定付一个 RTT(实测 33ms);同一连接
|
||||
连续发 ``pipeline`` 条请求,服务器按序应答,于是 ``pipeline`` 只只共付
|
||||
一个 RTT。再叠加多台服务器并行(每台 1 连接,尊重 per_server_cap),
|
||||
实测全市场 1 分钟K线取当天:43.6s → 5s 级。
|
||||
|
||||
单请求装不下 ``start+count`` 时按 ``page`` 切页,多页与其他标的的请求
|
||||
混在同一批里流水线发出去(不是逐只串行翻页)。
|
||||
"""
|
||||
plan: list[tuple[int, int]] = []
|
||||
off, remain = start, start + count
|
||||
while remain > 0:
|
||||
take = min(page, remain)
|
||||
plan.append((off, take))
|
||||
off += take
|
||||
remain -= take
|
||||
|
||||
tasks: deque[tuple[Symbol, int, int]] = deque(
|
||||
(s, o, c) for s in syms for o, c in plan)
|
||||
n_conn = min(len(tasks), pool.max_size, _BULK_MAX_CONNS,
|
||||
max(2, -(-len(tasks) // max(1, pipeline * 4))))
|
||||
await pool.warm(n_conn) # 并发建连;acquire 建连是串行的,先热好
|
||||
held = await pool.acquire_many(n_conn)
|
||||
if not held:
|
||||
return {}
|
||||
|
||||
by_text = {s.text: s for s in syms}
|
||||
acc: dict[str, list[SecurityBar]] = {}
|
||||
retry: dict[str, Symbol] = {}
|
||||
|
||||
async def worker(pc) -> None:
|
||||
broken = False
|
||||
try:
|
||||
while tasks:
|
||||
batch: list[tuple[Symbol, int, int]] = []
|
||||
while tasks and len(batch) < pipeline:
|
||||
batch.append(tasks.popleft())
|
||||
cmds = [make_cmd(s, o, c) for s, o, c in batch]
|
||||
try:
|
||||
res = await pc.execute_pipeline(cmds)
|
||||
except Exception:
|
||||
# 流水线中途失败 = 响应流已错配,连接作废、整批交回逐只重试
|
||||
broken = True
|
||||
for s, _, _ in batch:
|
||||
retry[s.text] = s
|
||||
return
|
||||
for (s, _, _), bars in zip(batch, res):
|
||||
acc.setdefault(s.text, []).extend(bars)
|
||||
finally:
|
||||
await pool.release(pc, broken=broken)
|
||||
|
||||
await asyncio.gather(*(worker(pc) for pc in held))
|
||||
# 连接全挂时 worker 会提前 return,队列里剩下的标的没人取 —— 一并交回逐只,
|
||||
# 否则会**静默少数据**(实测 20 只只回 16 只)。
|
||||
while tasks:
|
||||
s, _, _ = tasks.popleft()
|
||||
retry.setdefault(s.text, s)
|
||||
out = {k: self._finish_bars(v, by_text[k]) for k, v in acc.items()}
|
||||
for text, s in retry.items():
|
||||
try:
|
||||
out[text] = await self._bars_one(
|
||||
s, period, start=start, count=count, adjust=adj,
|
||||
start_date=None, end_date=None)
|
||||
except Exception:
|
||||
out.setdefault(text, [])
|
||||
return out
|
||||
|
||||
async def _bars_one(
|
||||
self, sym: Symbol, period: KlinePeriod, *,
|
||||
start: int, count: int, adjust: Adjust | None,
|
||||
@@ -780,8 +1056,10 @@ class TdxData:
|
||||
from .protocol.commands.ex_proto import GetExInstrumentBarsCmd
|
||||
cat = int(period)
|
||||
fetch_count = max(700, start + count)
|
||||
# extra 带的 @v2 是缓存 schema 标记:v2 起 EX K线开始回填 amount/settle,
|
||||
# 旧版本写下的缓存里 amount 恒 0.0,必须让它们自然失效。
|
||||
key = CacheKey("kline", sym.market, sym.code, period.name, start,
|
||||
fetch_count, f"{adj.name}@{cat}")
|
||||
fetch_count, f"{adj.name}@{cat}@v2")
|
||||
|
||||
async def fetch() -> bytes:
|
||||
rows = await self._fetch_paginated_ex(
|
||||
@@ -792,7 +1070,10 @@ class TdxData:
|
||||
year=d["year"], month=d["month"], day=d["day"],
|
||||
hour=d.get("hour", 0), minute=d.get("minute", 0),
|
||||
open=d["open"], high=d["high"], low=d["low"], close=d["close"],
|
||||
vol=float(d.get("trade", 0) or 0), amount=0.0,
|
||||
vol=float(d.get("trade", 0) or 0),
|
||||
# 权益类(港股/美股/新三板)真有成交额;期货期权的帧里没有,恒 0.0
|
||||
amount=float(d.get("amount", 0.0) or 0.0),
|
||||
settle=d.get("settle"),
|
||||
) for d in rows]
|
||||
return pack_payload(bars)
|
||||
|
||||
@@ -815,21 +1096,23 @@ class TdxData:
|
||||
start_date: datetime | None, end_date: datetime | None,
|
||||
adjust: Adjust | None,
|
||||
) -> list[SecurityBar]:
|
||||
"""按日期区间取K线:先二分定位偏移,再只拉目标区间。
|
||||
"""按日期区间取K线:贴最新走倒推,历史中段才二分定位。
|
||||
|
||||
关键坑:**偏移方向各通道不一致**(实测标准通道 offset 0 是最新一根、
|
||||
越大越老;MAC/EX 反过来),所以先探一次方向再二分,不做静态假设。
|
||||
|
||||
快慢两条路径(:meth:`_bars_backtrack` 优先):
|
||||
倒推 = 直接 ``count=N`` 拿最新 N 根再裁剪,1~3 次往返;
|
||||
二分 = 先探总根数再 log2 次跳转,长周期上是 36+ 次串行往返。
|
||||
「取今天」「增量到最新」都贴着右端,占了实际使用的绝大多数。
|
||||
"""
|
||||
# 归一化日期边界
|
||||
start_dt = (start_date if isinstance(start_date, datetime)
|
||||
else datetime.combine(start_date, datetime.min.time())) \
|
||||
if start_date is not None else None
|
||||
end_dt = (end_date if isinstance(end_date, datetime)
|
||||
else datetime.combine(end_date, datetime.max.time())) \
|
||||
if end_date is not None else None
|
||||
# end_date 传入 datetime 但时间为 00:00 时视为"当天结束"
|
||||
if end_dt is not None and end_dt.time() == datetime.min.time().replace(tzinfo=None):
|
||||
end_dt = end_dt.replace(hour=23, minute=59, second=59, microsecond=999999)
|
||||
start_dt, end_dt = _norm_range(start_date, end_date)
|
||||
|
||||
# 快路径:区间右端贴着最新 → 从最新倒推,1~3 次往返
|
||||
fast = await self._bars_backtrack(sym, period, adjust, start_dt, end_dt)
|
||||
if fast is not None:
|
||||
return fast
|
||||
|
||||
est = _KLINE_MAX_BARS.get(period, 10_000)
|
||||
total = await self._bar_total(sym, period, adjust, est)
|
||||
@@ -839,6 +1122,7 @@ class TdxData:
|
||||
|
||||
if desc:
|
||||
# offset 越大越老:[0,k) 全都 >= start_date,[j,∞) 全都 <= end_date
|
||||
# 不做并行二分:两个协程同时抢连接会触发服务器侧限速,实测比串行更慢
|
||||
k = total
|
||||
if start_dt is not None:
|
||||
k = await self._bisect_offset(
|
||||
@@ -870,6 +1154,87 @@ class TdxData:
|
||||
bars = [b for b in bars if b.datetime <= end_dt]
|
||||
return bars
|
||||
|
||||
async def _bars_backtrack(
|
||||
self, sym: Symbol, period: KlinePeriod, adjust: Adjust | None,
|
||||
start_dt: datetime | None, end_dt: datetime | None,
|
||||
) -> list[SecurityBar] | None:
|
||||
"""从最新一根往前倒推目标区间(区间右端贴着最新时适用)。
|
||||
|
||||
返回 ``None`` 表示**不适用**(偏移方向相反 / 要的是历史中间段 /
|
||||
区间太长放大不划算),调用方应回落到二分路径;返回列表表示已算完
|
||||
(**空列表**也是有效结果,代表区间内无数据)。
|
||||
|
||||
为什么快:二分要先 ``_bar_total`` 翻倍探总根数(长周期 6 次)再跳
|
||||
``log2(总根数)`` 次,1 分钟 K 线约 36 次串行往返 = 4 秒级;
|
||||
倒推只是「拿最新 N 根」,一次就能覆盖一天。
|
||||
|
||||
**标准通道不探测方向**:``_bars_one`` 的 ``count`` 路径本来就靠
|
||||
「start=0 即最新端」这个协议事实,这里沿用同一假定,省掉 2 次往返
|
||||
——全市场 5,000 只时每只省 2 次,是几十秒的量级。
|
||||
只有 MAC/EX 才真去探(它们 offset 方向相反)。
|
||||
"""
|
||||
adj = self.default_adjust if adjust is None else adjust
|
||||
span = _BAR_SPAN.get(period, timedelta(days=1))
|
||||
now = datetime.now()
|
||||
|
||||
if not await self._kline_newest_at_zero(sym, period, adj):
|
||||
return None # offset 0 是最老端 → 倒推不成立,交给二分
|
||||
|
||||
if start_dt is not None and start_dt > now + span:
|
||||
return [] # 整段区间都在未来
|
||||
# 右端够不着最新(留 _BACKTRACK_GRACE 余量覆盖周末/长假)→ 走二分
|
||||
if end_dt is not None and end_dt + _BACKTRACK_GRACE < now:
|
||||
return None
|
||||
ref = now
|
||||
|
||||
# 初值按时间跨度估根数:一天内 1 次就够,长区间也基本 1 次到位
|
||||
count = _BACKTRACK_MIN_BARS
|
||||
if start_dt is not None:
|
||||
need = _bars_for_range(period, start_dt, ref)
|
||||
# 当天(盘中)实际只要已走完的那几十根,别沿用一整天 240 根
|
||||
count = max(need, 8) if start_dt.date() == ref.date() else max(count, need)
|
||||
count = min(count, _BACKTRACK_MAX_BARS)
|
||||
|
||||
bars: list[SecurityBar] = []
|
||||
for _ in range(_BACKTRACK_ROUNDS):
|
||||
bars = await self.get_bars(sym.text, period, start=0, count=count,
|
||||
adjust=adjust)
|
||||
if not bars:
|
||||
return []
|
||||
first = bars[0].datetime
|
||||
# 覆盖到 start 即止;跨到自己之前的那一天也算够(K线连续,中间不会缺)
|
||||
if start_dt is None or first <= start_dt or first.date() < start_dt.date():
|
||||
break
|
||||
if count >= _BACKTRACK_MAX_BARS:
|
||||
return None
|
||||
count = min(count * 2, _BACKTRACK_MAX_BARS)
|
||||
else:
|
||||
return None # 放大到上限仍不够 → 交给二分
|
||||
|
||||
if start_dt is not None:
|
||||
bars = [b for b in bars if b.datetime >= start_dt]
|
||||
if end_dt is not None:
|
||||
bars = [b for b in bars if b.datetime <= end_dt]
|
||||
return bars
|
||||
|
||||
async def _kline_newest_at_zero(self, sym: Symbol, period: KlinePeriod,
|
||||
adjust: Adjust) -> bool:
|
||||
"""K 线的 ``offset=0`` 是否就是**最新**一根(True 才能倒推)。
|
||||
|
||||
偏移方向是通道级属性,与具体标的无关,首次探测后按通道名缓存在实例上。
|
||||
"""
|
||||
key = "ex" if sym.is_ex else route_for("kline", adjust=adjust, period=period)
|
||||
cached = self._kline_dir_cache.get(key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
newest = await self._probe_bar(sym, period, adjust, 0)
|
||||
if newest is None:
|
||||
return False # 探不到就别猜,交给二分去处理空数据
|
||||
nxt = await self._probe_bar(sym, period, adjust, 1)
|
||||
desc = nxt is None or nxt.datetime < newest.datetime
|
||||
self._kline_dir_cache[key] = desc
|
||||
return desc
|
||||
|
||||
async def _bar_total(self, sym: Symbol, period: KlinePeriod,
|
||||
adjust: Adjust | None, est: int) -> int:
|
||||
"""指数扩张找K线总根数上界(探测到越界为止)。"""
|
||||
@@ -937,6 +1302,7 @@ class TdxData:
|
||||
return None
|
||||
return bars[0] if bars else None
|
||||
|
||||
@framed
|
||||
async def get_indicators(
|
||||
self, symbols: str | Sequence[str], indicators: Sequence[str], *,
|
||||
period: KlinePeriod = KlinePeriod.DAY, count: int = 30,
|
||||
@@ -975,6 +1341,7 @@ class TdxData:
|
||||
# 四、逐笔成交 / 分时
|
||||
# ================================================================== #
|
||||
|
||||
@framed
|
||||
async def get_ticks(
|
||||
self, symbols: str | Sequence[str], *,
|
||||
date: int | None = None, start: int = 0, count: int | None = None,
|
||||
@@ -1156,6 +1523,7 @@ class TdxData:
|
||||
trade_count_available=False, # 扩展市场逐笔不带笔数字段
|
||||
)
|
||||
|
||||
@framed
|
||||
async def get_minutes(
|
||||
self, symbols: str | Sequence[str], *,
|
||||
date: int | None = None, sampling: bool = False, concurrency: int = 6,
|
||||
@@ -1236,6 +1604,7 @@ class TdxData:
|
||||
# 五、公司信息
|
||||
# ================================================================== #
|
||||
|
||||
@framed
|
||||
async def get_xdxr(self, symbols: str | Sequence[str]) -> list[XdxrRecord]:
|
||||
"""除权除息历史,仅 A股。标的用字符串(``"sz000001"``)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -1254,6 +1623,7 @@ class TdxData:
|
||||
out.extend(recs)
|
||||
return out
|
||||
|
||||
@framed
|
||||
async def get_finance(self, symbols: str | Sequence[str]) -> list[FinanceRecord]:
|
||||
"""财务快照(仅 A股)。无数据的标的会被跳过(原单标的接口返回 None)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -1324,6 +1694,7 @@ class TdxData:
|
||||
stat.limit_down_count = int(q6.open * 10)
|
||||
return stat
|
||||
|
||||
@framed
|
||||
async def get_unusual(self, market: str = "all", *, start: int = 0,
|
||||
count: int = 600) -> list[UnusualItem]:
|
||||
"""市场异动(仅 A股;``market="all"`` = 深市 + 沪市)。"""
|
||||
@@ -1359,6 +1730,7 @@ class TdxData:
|
||||
f"未知板块类别: {kind}(可选: {'/'.join(table)} 或数字)")
|
||||
return table[key]
|
||||
|
||||
@framed
|
||||
async def get_boards(self, kind: str = "all", *, count: int = 10000) -> list[BoardInfo]:
|
||||
"""板块列表(``kind``: ``all``/``industry``/``concept``/``style``,也接受数字)。
|
||||
|
||||
@@ -1380,6 +1752,7 @@ class TdxData:
|
||||
b.symbol = format_symbol(b.market, b.code)
|
||||
return items
|
||||
|
||||
@framed
|
||||
async def get_board_members(self, board: str, count: int = 100000,
|
||||
sort_type: int = 0, *,
|
||||
fields: Sequence[int] | None = None
|
||||
@@ -1405,6 +1778,7 @@ class TdxData:
|
||||
break
|
||||
return items
|
||||
|
||||
@framed
|
||||
async def get_board_of(self, symbols: str | Sequence[str]) -> list[BelongBoard]:
|
||||
"""个股所属板块(仅 A股;``symbol`` 记录被查询的个股)。"""
|
||||
syms = parse_symbols(symbols)
|
||||
@@ -1433,11 +1807,13 @@ class TdxData:
|
||||
"down_count": down,
|
||||
}
|
||||
|
||||
@framed
|
||||
async def get_board_ranking(self, kind: str = "industry", *,
|
||||
top_n: int = 20) -> list[dict]:
|
||||
"""板块当日涨跌幅排行(基于板块报价的现价/昨收)。"""
|
||||
return self._rank_boards(await self.get_boards(kind, count=200), top_n)
|
||||
|
||||
@framed
|
||||
async def get_board_change_ranking(self, kind: str = "industry", *,
|
||||
days: int = 20, top_n: int = 20) -> list[dict]:
|
||||
"""板块 N 日涨跌幅排行(基于板块指数K线)。"""
|
||||
@@ -1522,6 +1898,7 @@ class TdxData:
|
||||
return 2, s
|
||||
raise ValueError(f"无法判定指数市场: {index!r}(不是 A股指数代码段)")
|
||||
|
||||
@framed
|
||||
async def get_index_members(self, index: str, *, count: int = 100000,
|
||||
fields: Sequence[int] | None = None
|
||||
) -> list[SecurityQuote]:
|
||||
@@ -1570,6 +1947,7 @@ class TdxData:
|
||||
break
|
||||
return items[:count]
|
||||
|
||||
@framed
|
||||
async def get_index_trade_stats(
|
||||
self,
|
||||
index: str,
|
||||
@@ -1694,6 +2072,7 @@ class TdxData:
|
||||
elapsed=time.perf_counter() - started,
|
||||
)
|
||||
|
||||
@framed
|
||||
async def get_index_trade_stats_multi(
|
||||
self,
|
||||
index: str,
|
||||
@@ -1774,6 +2153,7 @@ class TdxData:
|
||||
break
|
||||
return bytes(data)
|
||||
|
||||
@framed
|
||||
async def get_block_parsed(self, filename: str) -> list[TdxBlock]:
|
||||
"""板块文件 -> ``list[TdxBlock]``(板块名/类别/成分股)。
|
||||
|
||||
@@ -1788,11 +2168,13 @@ class TdxData:
|
||||
from .codec.industry import parse_tdxhy_cfg
|
||||
return parse_tdxhy_cfg(await self.get_file("tdxhy.cfg"))
|
||||
|
||||
@framed
|
||||
async def get_financial_file_infos(self) -> list[FinancialFileInfo]:
|
||||
"""tdxfin/gpcw.txt -> ``list[FinancialFileInfo]``(文件名/MD5/大小)。"""
|
||||
from .codec.financial import parse_financial_file_list
|
||||
return parse_financial_file_list(await self.get_file("tdxfin/gpcw.txt"))
|
||||
|
||||
@framed
|
||||
async def get_financial_records_parsed(self, filename: str) -> list[FinancialRecord]:
|
||||
"""财报数据解析 -> ``list[FinancialRecord]``。
|
||||
|
||||
@@ -1986,6 +2368,7 @@ class TdxData:
|
||||
# 九、旧接口(deprecated 别名;入参沿用旧口径,1.0 移除)
|
||||
# ================================================================== #
|
||||
|
||||
@framed
|
||||
async def get_kline(self, market, code, period, *, start: int = 0,
|
||||
count: int = 800, adjust: Adjust | None = None,
|
||||
start_date: datetime | None = None,
|
||||
@@ -1996,6 +2379,7 @@ class TdxData:
|
||||
start=start, count=count, adjust=adjust,
|
||||
start_date=start_date, end_date=end_date)
|
||||
|
||||
@framed
|
||||
async def get_kline_since(self, market, code, period, since: datetime,
|
||||
adjust: Adjust | None = None) -> list[SecurityBar]:
|
||||
"""已废弃 -> :meth:`get_bars`(``start_date=since``)。"""
|
||||
@@ -2003,6 +2387,7 @@ class TdxData:
|
||||
return await self.get_bars(format_symbol(int(market), code), period,
|
||||
start_date=since, adjust=adjust)
|
||||
|
||||
@framed
|
||||
async def get_kline_batch(self, stocks, period, *, start: int = 0,
|
||||
count: int = 800, adjust: Adjust | None = None,
|
||||
concurrency: int = 6) -> dict[str, list[SecurityBar]]:
|
||||
@@ -2016,6 +2401,7 @@ class TdxData:
|
||||
out.setdefault(b.code, []).append(b)
|
||||
return out
|
||||
|
||||
@framed
|
||||
async def get_index_kline(self, market, code, period, *, start: int = 0,
|
||||
count: int = 800) -> list[SecurityBar]:
|
||||
"""已废弃 -> :meth:`get_bars`(指数自动识别,无需单独接口)。"""
|
||||
@@ -2036,12 +2422,14 @@ class TdxData:
|
||||
one = sets[0]
|
||||
return {"bars": one.bars, "indicators": one.indicators}
|
||||
|
||||
@framed
|
||||
async def get_stock_quotes(self, stocks, bits=None) -> list[SecurityQuote]:
|
||||
"""已废弃 -> :meth:`get_quotes`(``fields=bits``)。"""
|
||||
_deprecated("get_stock_quotes", "get_quotes")
|
||||
return await self.get_quotes([format_symbol(int(m), c) for m, c in stocks],
|
||||
fields=bits)
|
||||
|
||||
@framed
|
||||
async def get_stock_quotes_list(self, category: int, *, start: int = 0,
|
||||
count: int = 80, sort_type: int = 0
|
||||
) -> list[SecurityQuote]:
|
||||
@@ -2050,6 +2438,7 @@ class TdxData:
|
||||
return await self._board_members_raw(category, start=start, count=count,
|
||||
sort_type=sort_type)
|
||||
|
||||
@framed
|
||||
async def get_transactions(self, market, code, *, date: int | None = None,
|
||||
start: int = 0, count: int = 2000
|
||||
) -> list[TransactionRecord]:
|
||||
@@ -2058,12 +2447,14 @@ class TdxData:
|
||||
return await self.get_ticks(format_symbol(int(market), code), date=date,
|
||||
start=start, count=count)
|
||||
|
||||
@framed
|
||||
async def get_minute(self, market, code, *, date: int | None = None
|
||||
) -> list[MinuteBar]:
|
||||
"""已废弃 -> :meth:`get_minutes`。"""
|
||||
_deprecated("get_minute", "get_minutes")
|
||||
return await self.get_minutes(format_symbol(int(market), code), date=date)
|
||||
|
||||
@framed
|
||||
async def get_minute_batch(self, stocks, *, date: int | None = None,
|
||||
concurrency: int = 6) -> dict[str, list[MinuteBar]]:
|
||||
"""已废弃 -> :meth:`get_minutes`(返回值由 ``{code: bars}`` 变为平铺列表)。"""
|
||||
@@ -2086,6 +2477,7 @@ class TdxData:
|
||||
_deprecated("get_security_count", "get_universe")
|
||||
return await self._security_count(int(market))
|
||||
|
||||
@framed
|
||||
async def get_security_list(self, market, *, start: int = 0,
|
||||
count: int | None = None) -> list[SecurityInfo]:
|
||||
"""已废弃 -> :meth:`get_universe`(本地切片 ``[start:start+count]``)。"""
|
||||
@@ -2093,6 +2485,7 @@ class TdxData:
|
||||
items = await self.get_universe(market_name(int(market)))
|
||||
return items[start:] if count is None else items[start:start + count]
|
||||
|
||||
@framed
|
||||
async def get_security_list_all(self, market) -> list[SecurityInfo]:
|
||||
"""已废弃 -> :meth:`get_universe`。"""
|
||||
_deprecated("get_security_list_all", "get_universe")
|
||||
@@ -2146,12 +2539,14 @@ class TdxData:
|
||||
_deprecated("download_file", "get_file")
|
||||
return await self.get_file(filename)
|
||||
|
||||
@framed
|
||||
async def get_board_list(self, board_type: int | str = 0,
|
||||
count: int = 10000) -> list[BoardInfo]:
|
||||
"""已废弃 -> :meth:`get_boards`。"""
|
||||
_deprecated("get_board_list", "get_boards")
|
||||
return await self.get_boards(board_type, count=count)
|
||||
|
||||
@framed
|
||||
async def get_belong_board(self, market, code) -> list[BelongBoard]:
|
||||
"""已废弃 -> :meth:`get_board_of`。"""
|
||||
_deprecated("get_belong_board", "get_board_of")
|
||||
@@ -2163,6 +2558,7 @@ class TdxData:
|
||||
items = await self.get_snapshot(format_symbol(int(market), code))
|
||||
return items[0] if items else None
|
||||
|
||||
@framed
|
||||
async def get_goods_list(self, market, start: int = 0,
|
||||
count: int = 100) -> list[SecurityInfo]:
|
||||
"""已废弃 -> :meth:`get_universe`(扩展到 ``SecurityInfo`` 统一模型)。"""
|
||||
@@ -2177,12 +2573,14 @@ class TdxData:
|
||||
return len(await self._ex_all())
|
||||
return len(await self._universe_ex(int(market)))
|
||||
|
||||
@framed
|
||||
async def get_goods_quotes(self, stocks, bits=None) -> list[SecurityQuote]:
|
||||
"""已废弃 -> :meth:`get_quotes`。"""
|
||||
_deprecated("get_goods_quotes", "get_quotes")
|
||||
return await self.get_quotes([format_symbol(int(m), c) for m, c in stocks],
|
||||
fields=bits)
|
||||
|
||||
@framed
|
||||
async def get_goods_quotes_list(self, market, count: int = 80
|
||||
) -> list[SecurityQuote]:
|
||||
"""已废弃 -> :meth:`get_quotes`(或 ``get_universe`` 取清单后自选)。"""
|
||||
@@ -2198,6 +2596,7 @@ class TdxData:
|
||||
bars = await self.get_minutes(format_symbol(int(market), code), sampling=True)
|
||||
return [b.price for b in bars]
|
||||
|
||||
@framed
|
||||
async def get_goods_tick_chart(self, market, code,
|
||||
ymd: int = 0) -> list[MinuteBar]:
|
||||
"""已废弃 -> :meth:`get_minutes`(扩展到统一分时模型)。"""
|
||||
@@ -2205,6 +2604,7 @@ class TdxData:
|
||||
return await self.get_minutes(format_symbol(int(market), code),
|
||||
date=ymd or None)
|
||||
|
||||
@framed
|
||||
async def get_goods_transaction(self, market, code, *, ymd: int = 0,
|
||||
start: int = 0, count: int = 1800
|
||||
) -> list[TransactionRecord]:
|
||||
@@ -2213,6 +2613,7 @@ class TdxData:
|
||||
return await self.get_ticks(format_symbol(int(market), code),
|
||||
date=ymd or None, start=start, count=count)
|
||||
|
||||
@framed
|
||||
async def get_goods_transaction_all(self, market, code,
|
||||
ymd: int = 0) -> list[TransactionRecord]:
|
||||
"""已废弃 -> :meth:`get_ticks`(``count=None`` 即全量翻页)。"""
|
||||
|
||||
@@ -0,0 +1,431 @@
|
||||
"""DataFrame 转换层(0.7.0):把模型列表转成 pandas / polars 的 DataFrame。
|
||||
|
||||
为什么单独一层而不是改各方法的返回:
|
||||
|
||||
- **零运行时依赖**是项目的硬约束(``dependencies = []``)。pandas / polars 只在
|
||||
真正用到 ``TdxData(df=...)`` 时才导入,没装就走不到这里。
|
||||
- 转换口径(哪些列留、时间怎么摆、嵌套字段怎么展开)只写一份,40 个返回表格型
|
||||
数据的方法共用,不会各自漂移。
|
||||
|
||||
三条口径约定(0.7.0 定稿):
|
||||
|
||||
1. **时间列替换原始列**:K线/分时的 ``year/month/day/hour/minute`` 不再单列输出,
|
||||
统一给一列 ``datetime``(pandas 为 ``datetime64[ns]``,polars 为 ``Datetime('us')``),
|
||||
另带 ``trading_date``(``YYYY-MM-DD`` 字符串,美股跨日口径见模型注释)。
|
||||
2. **嵌套字段定型**:``SecurityQuote.bid/ask`` 展开成 ``bid1_price``…``ask5_vol`` 共
|
||||
10 列(缺失为 ``None``);``fields`` 这类不定长字典序列化成 JSON 字符串列,
|
||||
空字典给 ``None`` 而不是 ``"{}"``;``TdxBlock.codes`` 保持列表。
|
||||
3. **嵌套调用不二次转换**:``get_kline_batch`` 这类方法内部还会调 ``get_bars``,
|
||||
用 :data:`_FRAME_DEPTH`(``ContextVar``,按 task 隔离)标记层次,内层一律返回
|
||||
原始 ``list``,只有最外层那次才转 DataFrame。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import json
|
||||
import re
|
||||
from collections.abc import Callable, Sequence
|
||||
from contextvars import ContextVar
|
||||
from datetime import datetime as _dt
|
||||
from enum import IntEnum
|
||||
from typing import Any
|
||||
|
||||
from . import models as _models
|
||||
from .models import (
|
||||
IndexTradeStats,
|
||||
IndicatorSet,
|
||||
MinuteBar,
|
||||
MinuteTradeStat,
|
||||
SecurityBar,
|
||||
SecurityQuote,
|
||||
TransactionRecord,
|
||||
XdxrRecord,
|
||||
)
|
||||
|
||||
__all__ = ["BACKENDS", "FrameBackendError", "framed", "to_frame"]
|
||||
|
||||
BACKENDS: tuple[str, ...] = ("polars", "pandas")
|
||||
"""支持的后端名,也是 ``TdxData(df=...)`` 的合法取值。"""
|
||||
|
||||
|
||||
class FrameBackendError(ImportError):
|
||||
"""指定的 DataFrame 后端未安装。
|
||||
|
||||
继承 :class:`ImportError` 是刻意的:捕获姿势和缺包一致,
|
||||
且 ``e.name`` 就是缺的那个包名。
|
||||
"""
|
||||
|
||||
def __init__(self, backend: str) -> None:
|
||||
self.backend = backend
|
||||
super().__init__(
|
||||
f"未安装 {backend},无法输出 DataFrame。请执行:pip install pytdxdata[{backend}]",
|
||||
name=backend,
|
||||
)
|
||||
|
||||
|
||||
# 转换层次计数:>0 表示当前正处于某次公开方法的执行过程中(内层),此时不转换。
|
||||
_FRAME_DEPTH: ContextVar[int] = ContextVar("pytdxdata_frame_depth", default=0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 后端
|
||||
|
||||
|
||||
class _Backend:
|
||||
"""后端适配:只封装「造表」和「拼表」两个动作。"""
|
||||
|
||||
name: str = ""
|
||||
|
||||
def frame(self, cols: dict[str, list]) -> Any: # pragma: no cover - 抽象
|
||||
raise NotImplementedError
|
||||
|
||||
def concat(self, frames: Sequence[Any]) -> Any: # pragma: no cover - 抽象
|
||||
raise NotImplementedError
|
||||
|
||||
def empty(self, cols: Sequence[str]) -> Any:
|
||||
return self.frame({name: [] for name in cols})
|
||||
|
||||
|
||||
class _PolarsBackend(_Backend):
|
||||
name = "polars"
|
||||
|
||||
def __init__(self) -> None:
|
||||
try:
|
||||
import polars as _pl
|
||||
except ImportError as exc: # pragma: no cover - 依赖环境
|
||||
raise FrameBackendError("polars") from exc
|
||||
self._pl = _pl
|
||||
|
||||
def frame(self, cols: dict[str, list]) -> Any:
|
||||
# 全空时必须显式给列名,否则 polars 会造出 0 列的表
|
||||
return self._pl.DataFrame(cols)
|
||||
|
||||
def concat(self, frames: Sequence[Any]) -> Any:
|
||||
frames = [f for f in frames if f is not None and f.height > 0]
|
||||
if not frames:
|
||||
return None
|
||||
if len(frames) == 1:
|
||||
return frames[0]
|
||||
# vertical_relaxed:批量接口各标的分块建表,dtype 可能略有差异(如 Null 列)
|
||||
return self._pl.concat(frames, how="vertical_relaxed")
|
||||
|
||||
|
||||
class _PandasBackend(_Backend):
|
||||
name = "pandas"
|
||||
|
||||
def __init__(self) -> None:
|
||||
try:
|
||||
import pandas as _pd
|
||||
except ImportError as exc: # pragma: no cover - 依赖环境
|
||||
raise FrameBackendError("pandas") from exc
|
||||
self._pd = _pd
|
||||
|
||||
def frame(self, cols: dict[str, list]) -> Any:
|
||||
return self._pd.DataFrame(cols)
|
||||
|
||||
def concat(self, frames: Sequence[Any]) -> Any:
|
||||
frames = [f for f in frames if f is not None and len(f) > 0]
|
||||
if not frames:
|
||||
return None
|
||||
if len(frames) == 1:
|
||||
return frames[0]
|
||||
return self._pd.concat(frames, ignore_index=True)
|
||||
|
||||
|
||||
_BACKEND_CACHE: dict[str, _Backend] = {}
|
||||
|
||||
|
||||
def get_backend(name: str) -> _Backend:
|
||||
"""取后端实例(进程内缓存,重复取不会重复 import)。
|
||||
|
||||
:raises ValueError: 名字不在 :data:`BACKENDS` 里。
|
||||
:raises FrameBackendError: 对应的包没装。
|
||||
"""
|
||||
if name not in BACKENDS:
|
||||
raise ValueError(f"未知的 DataFrame 后端 {name!r},可选:{', '.join(BACKENDS)}")
|
||||
be = _BACKEND_CACHE.get(name)
|
||||
if be is None:
|
||||
be = _PolarsBackend() if name == "polars" else _PandasBackend()
|
||||
_BACKEND_CACHE[name] = be
|
||||
return be
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 列计划
|
||||
|
||||
|
||||
def _as_int(value: Any) -> Any:
|
||||
"""``IntEnum`` 转纯 ``int``:polars 对 IntEnum 的类型推断不可靠,转掉最稳。"""
|
||||
if isinstance(value, IntEnum):
|
||||
return int(value)
|
||||
return value
|
||||
|
||||
|
||||
def _json_or_none(value: Any) -> str | None:
|
||||
"""不定长字典序列化成 JSON 列;空字典给 ``None``(别让用户看到一列 ``"{}"``)。"""
|
||||
if not value:
|
||||
return None
|
||||
return json.dumps(value, ensure_ascii=False, separators=(",", ":"))
|
||||
|
||||
|
||||
def _getter(name: str) -> Callable[[Any], Any]:
|
||||
def _get(row: Any) -> Any:
|
||||
return _as_int(getattr(row, name))
|
||||
|
||||
return _get
|
||||
|
||||
|
||||
def _json_getter(name: str) -> Callable[[Any], Any]:
|
||||
def _get(row: Any) -> Any:
|
||||
return _json_or_none(getattr(row, name))
|
||||
|
||||
return _get
|
||||
|
||||
|
||||
def _level_getter(side: str, index: int, pos: int) -> Callable[[Any], Any]:
|
||||
"""``bid``/``ask`` 第 ``index`` 档的价格(pos=0)或数量(pos=1)。"""
|
||||
|
||||
def _get(row: Any) -> Any:
|
||||
levels = getattr(row, side)
|
||||
if len(levels) <= index:
|
||||
return None
|
||||
return levels[index][pos]
|
||||
|
||||
return _get
|
||||
|
||||
|
||||
def _moment(row: SecurityBar | MinuteBar) -> _dt:
|
||||
return row.datetime
|
||||
|
||||
|
||||
def _trading_date(row: SecurityBar | MinuteBar) -> str:
|
||||
return row.trading_date
|
||||
|
||||
|
||||
def _xdxr_day(row: XdxrRecord) -> _dt:
|
||||
# 除权日是「日」不是「时刻」,但 datetime64 / Datetime 才是可排序、可比较的列类型。
|
||||
# 全项目时间口径都是「不带 tzinfo 的北京时间」,这里保持一致。
|
||||
return _dt(row.year, row.month, row.day) # noqa: DTZ001
|
||||
|
||||
|
||||
#: 模型 -> 需要丢弃的原始字段(时间拆解字段,由增强列替代)
|
||||
_DROPS: dict[type, tuple[str, ...]] = {
|
||||
SecurityBar: ("year", "month", "day", "hour", "minute"),
|
||||
MinuteBar: ("hour", "minute", "date", "day_offset"),
|
||||
XdxrRecord: ("year", "month", "day"),
|
||||
SecurityQuote: ("bid", "ask"),
|
||||
}
|
||||
|
||||
#: 模型 -> 在「第一个被丢弃字段的位置」插入的增强列
|
||||
_EXTRAS: dict[type, tuple[tuple[str, Callable[[Any], Any]], ...]] = {
|
||||
SecurityBar: (("datetime", _moment), ("trading_date", _trading_date)),
|
||||
MinuteBar: (("datetime", _moment), ("trading_date", _trading_date)),
|
||||
XdxrRecord: (("date", _xdxr_day),),
|
||||
SecurityQuote: tuple(
|
||||
(f"{side}{level + 1}_{part}", _level_getter(side, level, pos))
|
||||
for side in ("bid", "ask")
|
||||
for level in range(5)
|
||||
for pos, part in ((0, "price"), (1, "vol"))
|
||||
),
|
||||
}
|
||||
|
||||
#: 需要 JSON 化的字典字段(按类区分,避免误伤 ``FinancialRecord.fields: list[float]``)
|
||||
_JSON_FIELDS: dict[type, frozenset[str]] = {
|
||||
SecurityQuote: frozenset({"fields"}),
|
||||
TransactionRecord: frozenset({"fields"}),
|
||||
MinuteBar: frozenset({"fields"}),
|
||||
}
|
||||
|
||||
_PLAN_CACHE: dict[type, tuple[tuple[str, Callable[[Any], Any]], ...]] = {}
|
||||
|
||||
|
||||
def _plan_for(cls: type) -> tuple[tuple[str, Callable[[Any], Any]], ...]:
|
||||
"""构造(并缓存)某模型的输出列计划。
|
||||
|
||||
查找顺序按 MRO 走:``IndexBar`` 没有自己的计划,但它是 ``SecurityBar`` 的子类,
|
||||
直接复用父类计划即可(字段完全相同)。
|
||||
"""
|
||||
cached = _PLAN_CACHE.get(cls)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
drops: tuple[str, ...] = ()
|
||||
extras: tuple[tuple[str, Callable[[Any], Any]], ...] = ()
|
||||
json_fields: frozenset[str] = frozenset()
|
||||
for klass in cls.__mro__:
|
||||
if not drops and klass in _DROPS:
|
||||
drops = _DROPS[klass]
|
||||
if not extras and klass in _EXTRAS:
|
||||
extras = _EXTRAS[klass]
|
||||
if not json_fields and klass in _JSON_FIELDS:
|
||||
json_fields = _JSON_FIELDS[klass]
|
||||
|
||||
import dataclasses
|
||||
|
||||
plan: list[tuple[str, Callable[[Any], Any]]] = []
|
||||
inserted = False
|
||||
for f in dataclasses.fields(cls): # type: ignore[arg-type]
|
||||
if f.name in drops:
|
||||
if not inserted:
|
||||
plan.extend(extras)
|
||||
inserted = True
|
||||
continue
|
||||
getter = _json_getter(f.name) if f.name in json_fields else _getter(f.name)
|
||||
plan.append((f.name, getter))
|
||||
if not inserted:
|
||||
# 没有丢弃字段的模型(如 SecurityInfo):增强列直接追加到末尾
|
||||
plan.extend(extras)
|
||||
|
||||
_PLAN_CACHE[cls] = tuple(plan)
|
||||
return _PLAN_CACHE[cls]
|
||||
|
||||
|
||||
def _columns(cls: type, rows: Sequence[Any]) -> dict[str, list]:
|
||||
"""列式采集:一次遍历填满所有列,比逐行 ``asdict()`` 快得多。"""
|
||||
return {name: [get(row) for row in rows] for name, get in _plan_for(cls)}
|
||||
|
||||
|
||||
def _columns_of_dicts(rows: Sequence[dict]) -> dict[str, list]:
|
||||
"""``list[dict]``(板块排行这类字段不固定的返回)按首次出现顺序并集列名。"""
|
||||
keys: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for row in rows:
|
||||
for key in row:
|
||||
if key not in seen:
|
||||
seen.add(key)
|
||||
keys.append(key)
|
||||
return {key: [row.get(key) for row in rows] for key in keys}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 容器分发
|
||||
|
||||
|
||||
def _model_by_name(name: str | None) -> type | None:
|
||||
if not name:
|
||||
return None
|
||||
obj = getattr(_models, name, None)
|
||||
return obj if isinstance(obj, type) else None
|
||||
|
||||
|
||||
def _resolve_model(data: Any, model: str | None) -> type | None:
|
||||
"""推断元素类型:非空列表看首元素;空列表只能靠调用方给的类型提示。"""
|
||||
if isinstance(data, list) and data:
|
||||
return type(data[0])
|
||||
return _model_by_name(model)
|
||||
|
||||
|
||||
def _frames_of_rows(be: _Backend, cls: type | None, rows: Sequence[Any]) -> Any:
|
||||
if cls is not None and rows and hasattr(cls, "__dataclass_fields__"):
|
||||
return be.frame(_columns(cls, rows))
|
||||
if rows and isinstance(rows[0], dict):
|
||||
return be.frame(_columns_of_dicts(rows))
|
||||
return be.empty(())
|
||||
|
||||
|
||||
def _from_indicator_sets(be: _Backend, items: Sequence[IndicatorSet]) -> Any:
|
||||
frames = []
|
||||
for item in items:
|
||||
cols = _columns(SecurityBar, item.bars)
|
||||
size = len(item.bars)
|
||||
if item.symbol:
|
||||
cols["symbol"] = [item.symbol] * size
|
||||
for name, series in item.indicators.items():
|
||||
values = list(series[:size])
|
||||
if len(values) < size:
|
||||
values.extend([None] * (size - len(values)))
|
||||
cols[name] = values
|
||||
frames.append(be.frame(cols))
|
||||
merged = be.concat(frames)
|
||||
return merged if merged is not None else be.frame(_columns(SecurityBar, []))
|
||||
|
||||
|
||||
def _from_index_trade_stats(be: _Backend, items: Sequence[IndexTradeStats]) -> Any:
|
||||
frames = []
|
||||
for stat in items:
|
||||
cols = _columns(MinuteTradeStat, stat.minutes)
|
||||
size = len(stat.minutes)
|
||||
# 元信息补成常量列:一张表就能看出是哪个指数、哪一天
|
||||
cols = {"index": [stat.index] * size, "date": [stat.date] * size, **cols}
|
||||
frames.append(be.frame(cols))
|
||||
merged = be.concat(frames)
|
||||
return merged if merged is not None else be.frame(_columns(MinuteTradeStat, []))
|
||||
|
||||
|
||||
def to_frame(data: Any, *, backend: str = "polars", model: str | None = None) -> Any:
|
||||
"""把模型返回值转成 DataFrame。
|
||||
|
||||
:param data: ``list[dataclass]`` / ``list[dict]`` / ``dict[str, list[dataclass]]``
|
||||
/ ``list[IndicatorSet]`` / ``IndexTradeStats`` / ``list[IndexTradeStats]``。
|
||||
:param backend: ``"polars"`` 或 ``"pandas"``。
|
||||
:param model: 元素类型的名字(如 ``"SecurityBar"``);只在 ``data`` 为空、
|
||||
无法从元素推断类型时使用,由 :func:`framed` 从返回注解里自动填。
|
||||
"""
|
||||
be = get_backend(backend)
|
||||
|
||||
if isinstance(data, IndexTradeStats):
|
||||
return _from_index_trade_stats(be, [data])
|
||||
if isinstance(data, IndicatorSet):
|
||||
return _from_indicator_sets(be, [data])
|
||||
|
||||
if isinstance(data, dict):
|
||||
# 批量接口:{标的: [行]} → 拼成一张表(symbol/code 列本来就在行里)
|
||||
frames = []
|
||||
for rows in data.values():
|
||||
if isinstance(rows, list) and rows:
|
||||
frames.append(_frames_of_rows(be, type(rows[0]), rows))
|
||||
merged = be.concat(frames)
|
||||
if merged is not None:
|
||||
return merged
|
||||
cls = _model_by_name(model)
|
||||
return be.empty(tuple(name for name, _ in _plan_for(cls)) if cls else ())
|
||||
|
||||
if isinstance(data, list):
|
||||
if data and isinstance(data[0], IndexTradeStats):
|
||||
return _from_index_trade_stats(be, data)
|
||||
if data and isinstance(data[0], IndicatorSet):
|
||||
return _from_indicator_sets(be, data)
|
||||
cls = _resolve_model(data, model)
|
||||
if data and isinstance(data[0], dict):
|
||||
return _frames_of_rows(be, None, data)
|
||||
if cls is not None and data and hasattr(cls, "__dataclass_fields__"):
|
||||
return _frames_of_rows(be, cls, data)
|
||||
if data:
|
||||
# 元素既不是 dataclass 也不是 dict(如 list[float])——不该走到这里
|
||||
raise TypeError(f"无法转换为 DataFrame 的列表元素:{type(data[0])!r}")
|
||||
return be.frame({name: [] for name, _ in _plan_for(cls)}) if cls else be.empty(())
|
||||
|
||||
raise TypeError(f"无法转换为 DataFrame 的返回值类型:{type(data)!r}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 接线
|
||||
|
||||
|
||||
_RETURN_MODEL = re.compile(r"list\[(\w+)\]")
|
||||
|
||||
|
||||
def framed(method: Callable[..., Any]) -> Callable[..., Any]:
|
||||
"""给 ``TdxData`` 的表格型方法套一层「按需转 DataFrame」。
|
||||
|
||||
- ``self._df_backend is None``(默认)时零开销,原样返回 ``list[dataclass]``。
|
||||
- 处于某次公开方法执行过程中(:data:`_FRAME_DEPTH` > 0)时不转换,
|
||||
保证批量/别名方法内部拿到的仍是原始 ``list``。
|
||||
"""
|
||||
model: str | None = None
|
||||
annotation = getattr(method, "__annotations__", {}).get("return")
|
||||
if isinstance(annotation, str):
|
||||
matched = _RETURN_MODEL.search(annotation)
|
||||
if matched:
|
||||
model = matched.group(1)
|
||||
|
||||
@functools.wraps(method)
|
||||
async def wrapper(self: Any, *args: Any, **kwargs: Any) -> Any:
|
||||
backend = getattr(self, "_df_backend", None)
|
||||
if backend is None or _FRAME_DEPTH.get() > 0:
|
||||
return await method(self, *args, **kwargs)
|
||||
|
||||
token = _FRAME_DEPTH.set(1)
|
||||
try:
|
||||
result = await method(self, *args, **kwargs)
|
||||
finally:
|
||||
_FRAME_DEPTH.reset(token)
|
||||
return to_frame(result, backend=backend, model=model)
|
||||
|
||||
return wrapper
|
||||
@@ -103,6 +103,10 @@ _US_EX_MARKETS: frozenset[int] = frozenset({int(ExMarket.US_STOCK)})
|
||||
_US_DAY_SPLIT_HOUR = 12
|
||||
|
||||
|
||||
# SecurityBar 上「后加的」可选字段:旧磁盘缓存反序列化时要补默认值(见 __setstate__)
|
||||
_BAR_LATE_FIELDS: tuple[str, ...] = ("settle",)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SecurityBar:
|
||||
"""K线记录(价格单位:元)。跨市场统一模型(A股/港股/美股/期货共用)。
|
||||
@@ -110,6 +114,10 @@ class SecurityBar:
|
||||
时间戳口径**统一为北京时间**:A股/港股/期货原始即为北京时间;美股原始是
|
||||
「日期=美东交易日 + 时分=北京时间」的混合值,构造时已还原为真实北京时间
|
||||
(见 ``__post_init__``)。该 bar 归属的交易日见 :attr:`trading_date`。
|
||||
|
||||
``amount`` / ``settle`` 的可取性**按通道**:A股标准/MAC 通道两者都有;
|
||||
EX 通道(0x23FF)一个 32B 记录里只有**一个** 4B 槽位可复用,权益类市场用它
|
||||
装成交额、期货期权类用它装结算价,两者不可能同时拿到 —— 详见 :attr:`amount`。
|
||||
"""
|
||||
market: int
|
||||
code: str
|
||||
@@ -117,14 +125,36 @@ class SecurityBar:
|
||||
high: float
|
||||
low: float
|
||||
close: float
|
||||
vol: float # 股票=股 / 转债=手(1 手=10 张,与 vipdoc 一致)
|
||||
amount: float # 元。⚠️ amount/vol 只在股票上等于均价;转债还要再 /10(手=10张)
|
||||
# vol 单位按品种:A股/ETF/美股=股;转债=手(1手=10张);⚠️港股主板/基金=手(1手=100股)
|
||||
vol: float
|
||||
# 元。⚠️ amount/vol 只在 A股上等于均价;转债还要再 /10(手=10张)、港股再 /100(手=100股)。
|
||||
# ⚠️ EX 通道的期货/期权(中金所、上期所、郑商所、大商所及各期权)协议帧内**没有成交额**,
|
||||
# 恒为 0.0 —— 要成交额请改用 EX 报价(``get_quotes``)或其它数据源。
|
||||
amount: float
|
||||
year: int
|
||||
month: int
|
||||
day: int
|
||||
hour: int = 0
|
||||
minute: int = 0
|
||||
symbol: str = "" # 规范化标的串(sz000001 / hk00700),由 API 层填充
|
||||
# 结算价。仅 EX 通道的**期货/期权**有(日线才有值,分钟线为 None);其余通道恒 None。
|
||||
settle: float | None = None
|
||||
|
||||
def __setstate__(self, state: Any) -> None:
|
||||
"""反序列化兜底:早期版本写下的磁盘缓存里没有后加的可选字段(如 ``settle``)。
|
||||
|
||||
``@dataclass(slots=True)`` 的默认反序列化只把 pickle 里存在的槽位塞回去,
|
||||
缺键的槽位**不会**被赋默认值 —— 直接读 ``bar.settle`` 会抛 ``AttributeError``。
|
||||
缓存键里没有 schema 版本号,所以这里必须容错,否则升级后读旧缓存即崩。
|
||||
"""
|
||||
if isinstance(state, tuple) and len(state) == 2: # slots 形式的 (dict, slots)
|
||||
data = dict(state[1] or {})
|
||||
else:
|
||||
data = dict(state or {})
|
||||
for name in _BAR_LATE_FIELDS:
|
||||
data.setdefault(name, None)
|
||||
for key, value in data.items():
|
||||
object.__setattr__(self, key, value)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
"""美股(EX 通道)时间戳归一化:还原成真实北京时间。
|
||||
|
||||
@@ -4,9 +4,24 @@ from __future__ import annotations
|
||||
import struct
|
||||
|
||||
from ..._binary import unpack_from
|
||||
from ...models import ExMarket
|
||||
from ..datetime_ import get_datetime
|
||||
from .base import BaseCommand
|
||||
|
||||
# 0x23FF 的「槽位 5 是成交额」的市场集合(其余市场该槽位是持仓量、槽位 7 是结算价)。
|
||||
# 实测依据(2026-10-28,日线,逐条校验 成交额/(close×vol)):
|
||||
# 31 港股主板 = 100.0(vol 单位手,1手=100股)、49 港股基金 = 100.3、
|
||||
# 48 港股创业板 ≈1.0、44 新三板 ≈0.99、74 美股 ≈1.0(vol 单位股)。
|
||||
# ⚠️ 27 港股指数 / 62 中证指数 该槽位既非成交额也非结算价(量级对不上),故**不在**表内,
|
||||
# 保持 amount=0.0 的原始行为,避免给出可疑数值。
|
||||
_EX_AMOUNT_MARKETS: frozenset[int] = frozenset({
|
||||
int(ExMarket.HK_MAIN_BOARD),
|
||||
int(ExMarket.HK_GEM),
|
||||
int(ExMarket.HK_FUND),
|
||||
int(ExMarket.NEEQ),
|
||||
int(ExMarket.US_STOCK),
|
||||
})
|
||||
|
||||
# pytdx EX setup 握手命令(完整84B: 12B帧头+72B body, 0x2454)—— 与MAC EX login同ID不同body
|
||||
EX_SETUP_CMD = bytes.fromhex("0101486500015200520054241f32c6e5d53dfb411f32c6e5d53dfb411f32c6e5d53dfb411f32c6e5d53dfb411f32c6e5d53dfb411f32c6e5d53dfb411f32c6e5d53dfb411f32c6e5d53dfb41cce16dffd5ba3fb8cbc57a054f7748ea")
|
||||
|
||||
@@ -57,7 +72,24 @@ class GetMarketsCmd(BaseCommand[list[dict]]):
|
||||
|
||||
|
||||
class GetExInstrumentBarsCmd(BaseCommand[list[dict]]):
|
||||
"""EX商品K线(0x23FF, 32B/条: 4B时间戳+28B行情)。"""
|
||||
"""EX商品K线(0x23FF, 32B/条: 4B时间戳+28B行情)。
|
||||
|
||||
后 28B 的字段语义**按市场分两类**(2026-10-28 全市场实测,见 ``_EX_AMOUNT_MARKETS``):
|
||||
|
||||
* 权益类(港股主板/创业板/基金、新三板、美股)::
|
||||
|
||||
open(4) high(4) low(4) close(4) 成交额(4f) 成交量(4I) 保留(4f)
|
||||
|
||||
成交额可用;校验 ``成交额 / (close × 成交量)`` = 100.0(港股主板/基金,
|
||||
``vol`` 单位是「手」=100 股)或 ≈1.0(港股创业板/新三板/美股,``vol`` 单位是股)。
|
||||
|
||||
* 期货期权类(中金所/上期所/郑商所/大商所及各期权)::
|
||||
|
||||
open(4) high(4) low(4) close(4) 持仓量(4I) 成交量(4I) 结算价(4f)
|
||||
|
||||
**协议帧内没有成交额**(``amount`` 恒 0.0,不是解析漏读);末位 float 是结算价,
|
||||
日线有值、分钟线为 0(``settle`` 归一化成 ``None``)。
|
||||
"""
|
||||
|
||||
def __init__(self, market: int, code: str, category: int = 0,
|
||||
start: int = 0, count: int = 700) -> None:
|
||||
@@ -76,6 +108,7 @@ class GetExInstrumentBarsCmd(BaseCommand[list[dict]]):
|
||||
if len(body) < 20:
|
||||
return []
|
||||
(ret_count,) = unpack_from("<H", body, 18, "ex bars count")
|
||||
amount_market = self.market in _EX_AMOUNT_MARKETS
|
||||
items: list[dict] = []
|
||||
off = 20
|
||||
for _ in range(ret_count):
|
||||
@@ -83,14 +116,23 @@ class GetExInstrumentBarsCmd(BaseCommand[list[dict]]):
|
||||
break
|
||||
# 时间4字节按category分格式:分钟级=2B zipday+2B tminutes;日线及以上=YYYYMMDD
|
||||
year, month, day, hour, minute, _ = get_datetime(self.category, body, off)
|
||||
open_, high, low, close, position, trade, _ = struct.unpack(
|
||||
"<ffffIIf", body[off + 4: off + 32])
|
||||
items.append({
|
||||
raw = body[off + 4: off + 32]
|
||||
open_, high, low, close = struct.unpack("<ffff", raw[:16])
|
||||
item = {
|
||||
"year": year, "month": month, "day": day,
|
||||
"hour": hour, "minute": minute,
|
||||
"open": open_, "high": high, "low": low, "close": close,
|
||||
"position": position, "trade": trade,
|
||||
})
|
||||
}
|
||||
if amount_market:
|
||||
# 权益类:槽位 5 = 成交额(float),槽位 7 语义未知(保留不用)
|
||||
amount, trade, _reserved = struct.unpack("<fIf", raw[16:28])
|
||||
item.update(amount=amount, trade=trade, position=0, settle=None)
|
||||
else:
|
||||
# 期货/期权:槽位 5 = 持仓量,槽位 7 = 结算价(分钟线为 0 → None)
|
||||
position, trade, settle = struct.unpack("<IIf", raw[16:28])
|
||||
item.update(amount=0.0, trade=trade, position=position,
|
||||
settle=settle or None)
|
||||
items.append(item)
|
||||
off += 32
|
||||
return items
|
||||
|
||||
|
||||
Reference in new issue
Block a user