* feat: DataFrame 直出(df="polars"/"pandas")+ 版本 0.7.0 TdxData 新增构造参数 df: - None(默认)→ 行为与 0.6.x 完全一致,返回 list[dataclass] - "polars"/"pandas" → 42 个表格型接口直接返回 DataFrame 转换层 src/pytdxdata/frames.py: - 列计划按模型缓存,列式采集(不做逐行 asdict) - 时间列替换:year/month/day/hour/minute → datetime + trading_date;Xdxr 给 date - 嵌套定型:bid/ask 展开 10 列、fields 字典 JSON 化、codes 保持列表 - 容器展开:get_indicators 合并指标、get_index_trade_stats 展开 minutes、 get_kline_batch/get_minute_batch 纵向拼接 - ContextVar 层次标记:批量/别名方法内层不二次转换,并发互不干扰 - 后端惰性加载:缺包抛 FrameBackendError(带安装命令),构造时即失败 依赖:新增 extras pandas/polars/dataframe,核心包仍零依赖;dev 组加 pandas+polars 测试:tests/test_frames.py 40 项(双后端×双格式);全量 401 项通过 文档:新增 docs/dataframe.md,同步 API_REFERENCE/SKILL/llms.txt/README/mkdocs * perf(kline): 按日期取K线贴最新时走倒推,1分钟 4.2s → 82ms 问题:start_date/end_date 一律走「探总根数 + 二分定位」。1 分钟 K 线历史 二十万根量级 = 6 次翻倍探测 + 2 次方向探测 + 两次各 ~18 次跳转,共 36+ 次 串行往返。「取今天」这个最常见的调用要 4.2 秒。 改法(_bars_backtrack 快路径): - 右端离最新不超过 _BACKTRACK_GRACE(10 天) 时,直接 start=0, count=N 拿最新 N 根再裁剪;N 按时间跨度估(_BARS_PER_DAY),一天内一次命中, 不够才翻倍,放大到 _BACKTRACK_MAX_BARS 仍不够则回落二分 - 历史中段仍走二分,结果口径完全一致(已逐条比对基准) 两个关键细节: - 偏移方向是通道级属性而非标的属性,首次探测后缓存在实例上 (_kline_dir_cache)——全市场 5,225 只批量只付一次探测成本 - K线 hour/minute 是周期**结束**时刻,最新一根可能比 now() 略大(14:29 那根还在走),「右端贴最新」必须留周期容差,否则 end_date=now() 被误判 成历史中段 实测(1 分钟,冷启动): 取今天 4,208ms → 82ms 2天前~now 2,211ms → 75ms 最近3天 2,000ms+ → 239ms 历史中段 1,512ms → 285ms 全市场5225只取今天 32.1s(1,140,699 根,精确当天口径) 注:route_for 让 MIN_1 一律走 MAC(0x122E),所以按通道做方向缓存而不是按 「标准/扩展」写死分支。 测试:新增 tests/test_kline_date.py 6 项(倒推次数上界、结束时刻容差、 中段回落二分、未来区间、反向通道、与 count 路径口径一致);全量 407 项通过 * perf(kline): 全市场1分钟K线改多连接流水线抓取(43.6s → 5~7s,盘中 2s) 瓶颈其实是**往返次数**不是带宽:单只K线一请求一标的,串行 33ms/只 ≈ 1 个 RTT。 同一台服务器 16 连接 × 深度 8 实测能跑 5.2 MB/s(此前「单 IP 限流 1.5MB/s」 是浅流水下的假象)。 - get_bars 新增 pipeline 参数(None=按单请求字节自适应 8~32,1=关闭走逐只) - 多标的(≥16 只)走 _bulk_bars:多连接 work-stealing + 单连接 execute_pipeline - 只用实测最快的 12 台:池里 24 台全用反而被慢节点拖长尾 - 盘中按交易时段只拉已走过的根数(10:00 → 35 根,数据量差 7 倍) - 标准通道(日线不复权)同样走批量:全市场 49~72s → 13s - 修 pipeline 失败时队列剩余标的静默丢失(实测 20 只只回 16 只) - 修 test_future_range_returns_empty_quickly 时间炸弹(固定日期会漂进 _BACKTRACK_GRACE) - conftest 关闭后台测速 task,避免污染「往返次数」类断言 实测(5,225 只):count=2 → 2.7s;count=35 → 2.2s;count=240 → 6.9s。 逐根一致性 30/30 相同。新增 tests/test_bulk_kline.py 12 项,合计 419 项全绿。 * test: 批量K线测试的时间锚点改用真实 now(CI 是 UTC,固定日期会漂到别的时段分支) * fix(ex): EX 通道 K 线回填成交额与结算价(0x23FF 槽位按市场分两类) 报告:EX 通道 get_bars 返回的 SecurityBar.amount 恒为 0.0。 定位属实,但实际情况比报告更细 —— 32B 记录里那 28B 行情只有**一个** 4B 槽位 可复用,语义按市场分两类(全市场实测,用 成交额/(close×vol) 逐条校验): | 市场 | 槽位5 | 槽位6 | 槽位7 | |---|---|---|---| | 港股主板31/港股基金49 | 成交额(f) | 量(手=100股) | 未知 | | 港股创业板48/新三板44/美股74 | 成交额(f) | 量(股) | 0 | | 期货/期权 7/8/9/47… | 持仓量(I) | 量 | 结算价(f) | 改前 amount 硬编码 0.0、第 7 个 float 用 `_` 丢弃 —— 于是港股/美股/新三板 明明帧内有成交额却全为 0,期货期权的结算价也拿不到。 - 新增 SecurityBar.settle(结算价,float|None):仅 EX 期货/期权日线有值, 分钟线该槽位恒 0 已归一化为 None,其它通道恒 None。 - 权益类市场回填真实 amount;港股主板/基金的 vol 单位是手(=100股), amount/vol 是均价的 100 倍,已写进模型注释与量纲表。 - 期货/期权的 amount 仍是 0.0(帧内确实没有成交额,协议限制), 已在 SecurityBar / API_REFERENCE / llms.txt / SKILL.md 写清, 避免使用者再按旧文档判断成调用方 bug。 - 港股指数27/中证指数62 该槽位量级对不上(既非额也非结算价),保守保持 0.0。 - SecurityBar.__setstate__ 兜底旧磁盘缓存:缓存键无 schema 版本,升级后读旧 pickle 缺 settle 槽位,默认反序列化不补默认值 → AttributeError,已容错; EX K 线缓存键额外加 @v2,让 amount 恒 0 的旧缓存自然失效。 - K 线 DataFrame 因此多一列 settle(0.7.0 未发布,列清单同步更新)。 测试:tests/test_ex_bars_fields.py 16 项(解析层双布局、日内 settle 归零、 API 透传、旧 pickle 兼容);全套 435 项通过。 * docs: 0.7.0 全站口径同步(DataFrame 出口 + EX 成交额限制 + 构造参数表) - 首页:特性表补「DataFrame 直出」与 K 线批量性能,导航新增 DataFrame 卡片, 统一方法数按实测修正为 35(+33 别名 = 68 个公开方法) - 安装页:补 extras(polars/pandas/dataframe)与 df 用法;构造参数表补 df / std_pool_cap / ex_pool_cap,并把 pool_max 默认值从 12 修正为 24(实际值) - 场景示例:`转成 pandas` 一节改为先推 df= 开关,手动转换降级为局部场景 - 数据模型页:不再说「刻意不提供 DataFrame」,改为 dataclass 与 DataFrame 是同一份数据、两种容器 - AI 接入页:返回值约定补 `df=` 开关的说法 - README:K 线列注释补 settle;已知限制补 EX 通道成交额/结算价的协议口径 - CHANGELOG:新增「升级」小节(升级命令、旧缓存无需清理) --------- Co-authored-by: openbot-coder <openbot-coder@users.noreply.github.com>
326 lines
11 KiB
Python
326 lines
11 KiB
Python
"""DataFrame 转换层测试(0.7.0)。
|
||
|
||
覆盖两条独立路径:
|
||
|
||
1. **转换口径**:直接调 :func:`to_frame`,喂合成模型对象,断言列名/列序/取值/dtype。
|
||
不碰网络,双后端各跑一遍。
|
||
2. **接线**:``TdxData(df=...)`` 的取值校验、装饰器的「内层不二次转换」语义。
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import math
|
||
import sys
|
||
|
||
import pytest
|
||
|
||
from pytdxdata import TdxData
|
||
from pytdxdata import frames as _frames
|
||
from pytdxdata.frames import FrameBackendError, framed, get_backend, to_frame
|
||
from pytdxdata.models import (
|
||
IndexBar,
|
||
IndexTradeStats,
|
||
IndicatorSet,
|
||
Market,
|
||
MinuteBar,
|
||
MinuteTradeStat,
|
||
SecurityBar,
|
||
SecurityQuote,
|
||
TdxBlock,
|
||
XdxrRecord,
|
||
)
|
||
|
||
BACKENDS = ("polars", "pandas")
|
||
|
||
|
||
# ---------------------------------------------------------------- 工具
|
||
|
||
|
||
def bar(**kw) -> SecurityBar:
|
||
base = {
|
||
"market": int(Market.SH), "code": "600000", "open": 10.0, "high": 11.0,
|
||
"low": 9.5, "close": 10.5, "vol": 1000.0, "amount": 10500.0,
|
||
"year": 2026, "month": 10, "day": 27, "hour": 0, "minute": 0,
|
||
"symbol": "sh600000",
|
||
}
|
||
base.update(kw)
|
||
return SecurityBar(**base)
|
||
|
||
|
||
def rows(df, backend: str) -> int:
|
||
return len(df) if backend == "pandas" else df.height
|
||
|
||
|
||
def cols(df, backend: str) -> list[str]:
|
||
return list(df.columns)
|
||
|
||
|
||
def cell(df, backend: str, name: str, index: int = 0):
|
||
if backend == "pandas":
|
||
return df[name].iloc[index]
|
||
# polars 2.x:Series[0] 会返回 Series 而不是标量,统一走 to_list()
|
||
return df.get_column(name).to_list()[index]
|
||
|
||
|
||
def missing(value) -> bool:
|
||
"""pandas 的数值列会把 None 变成 NaN,两者都算「缺」。"""
|
||
return value is None or (isinstance(value, float) and math.isnan(value))
|
||
|
||
|
||
# ---------------------------------------------------------------- 后端
|
||
|
||
|
||
def test_backend_instances_are_cached():
|
||
assert get_backend("polars") is get_backend("polars")
|
||
assert get_backend("pandas") is get_backend("pandas")
|
||
|
||
|
||
def test_unknown_backend_raises_value_error():
|
||
with pytest.raises(ValueError, match="未知的 DataFrame 后端"):
|
||
get_backend("numpy")
|
||
|
||
|
||
def test_missing_backend_raises_with_install_hint(monkeypatch):
|
||
monkeypatch.setitem(sys.modules, "polars", None) # 让 import polars 抛 ImportError
|
||
monkeypatch.delitem(_frames._BACKEND_CACHE, "polars", raising=False)
|
||
with pytest.raises(FrameBackendError) as excinfo:
|
||
get_backend("polars")
|
||
assert isinstance(excinfo.value, ImportError)
|
||
assert excinfo.value.name == "polars"
|
||
assert "pip install pytdxdata[polars]" in str(excinfo.value)
|
||
monkeypatch.delitem(_frames._BACKEND_CACHE, "polars", raising=False)
|
||
|
||
|
||
# ---------------------------------------------------------------- 转换口径
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_bar_replaces_split_time_with_datetime(backend):
|
||
df = to_frame([bar(), bar(minute=30)], backend=backend)
|
||
assert cols(df, backend) == [
|
||
"market", "code", "open", "high", "low", "close", "vol", "amount",
|
||
"datetime", "trading_date", "symbol", "settle",
|
||
]
|
||
assert rows(df, backend) == 2
|
||
assert cell(df, backend, "market") == 1
|
||
assert cell(df, backend, "trading_date") == "2026-10-27"
|
||
stamp = cell(df, backend, "datetime", 1)
|
||
assert stamp.year == 2026 and stamp.hour == 0 and stamp.minute == 30
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_int_enum_column_is_plain_int(backend):
|
||
df = to_frame([bar(market=Market.SZ)], backend=backend)
|
||
value = cell(df, backend, "market")
|
||
assert value == 0
|
||
assert not isinstance(value, Market)
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_empty_list_falls_back_to_annotation_hint(backend):
|
||
df = to_frame([], backend=backend, model="SecurityBar")
|
||
assert rows(df, backend) == 0
|
||
assert cols(df, backend) == [
|
||
"market", "code", "open", "high", "low", "close", "vol", "amount",
|
||
"datetime", "trading_date", "symbol", "settle",
|
||
]
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_quote_expands_bid_ask_levels(backend):
|
||
quote = SecurityQuote(
|
||
market=1, code="600000", price=10.5, pre_close=10.0, open=10.1, high=10.6,
|
||
low=9.9, vol=100.0, amount=1000.0, s_vol=40.0, b_vol=60.0,
|
||
bid=[(10.49, 100.0), (10.48, 200.0)], ask=[(10.51, 300.0)],
|
||
symbol="sh600000", fields={"0x7": 1.5},
|
||
)
|
||
df = to_frame([quote], backend=backend)
|
||
out = cols(df, backend)
|
||
assert "bid1_price" in out and "ask5_vol" in out
|
||
assert "bid" not in out and "ask" not in out
|
||
assert cell(df, backend, "bid2_vol") == 200.0
|
||
assert missing(cell(df, backend, "bid3_price")) # 只有两档,缺档给 None
|
||
assert cell(df, backend, "fields") == '{"0x7":1.5}'
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_empty_dict_field_becomes_none(backend):
|
||
df = to_frame([SecurityQuote(
|
||
market=1, code="600000", price=1.0, pre_close=1.0, open=1.0, high=1.0,
|
||
low=1.0, vol=1.0, amount=1.0, s_vol=0.0, b_vol=0.0,
|
||
)], backend=backend)
|
||
assert missing(cell(df, backend, "fields"))
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_minute_uses_real_clock(backend):
|
||
late = MinuteBar(price=1.0, vol=10.0, avg_price=1.0, hour=27, minute=59,
|
||
symbol="usAAPL", market=74, code="AAPL", date=20261026)
|
||
df = to_frame([late], backend=backend)
|
||
assert cols(df, backend) == [
|
||
"price", "vol", "avg_price", "datetime", "trading_date",
|
||
"symbol", "market", "code", "fields",
|
||
]
|
||
stamp = cell(df, backend, "datetime")
|
||
assert (stamp.year, stamp.month, stamp.day, stamp.hour, stamp.minute) == (
|
||
2026, 10, 27, 3, 59,
|
||
)
|
||
assert cell(df, backend, "trading_date") == "2026-10-26"
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_xdxr_date_column(backend):
|
||
row = XdxrRecord(market=1, code="600000", year=2026, month=6, day=12,
|
||
category=1, name="分红", fenhong=0.5, symbol="sh600000")
|
||
df = to_frame([row], backend=backend)
|
||
assert cols(df, backend)[:3] == ["market", "code", "date"]
|
||
stamp = cell(df, backend, "date")
|
||
assert (stamp.year, stamp.month, stamp.day) == (2026, 6, 12)
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_tdx_block_keeps_codes_list(backend):
|
||
df = to_frame([TdxBlock(name="银行", category=0, count=2, codes=["600000", "600036"])],
|
||
backend=backend)
|
||
assert cols(df, backend) == ["name", "category", "count", "codes"]
|
||
assert cell(df, backend, "codes") == ["600000", "600036"]
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_indicator_set_merges_bars_and_indicators(backend):
|
||
item = IndicatorSet(
|
||
symbol="sh600000",
|
||
bars=[bar(), bar(minute=1), bar(minute=2)],
|
||
indicators={"ma5": [1.0, 2.0, 3.0], "boll": [None, None, 9.0]},
|
||
)
|
||
df = to_frame([item], backend=backend)
|
||
assert rows(df, backend) == 3
|
||
assert cell(df, backend, "symbol") == "sh600000"
|
||
assert cell(df, backend, "ma5", 2) == 3.0
|
||
assert missing(cell(df, backend, "boll"))
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_indicator_series_shorter_than_bars_is_padded(backend):
|
||
item = IndicatorSet(symbol="sh600000", bars=[bar(), bar(minute=1)],
|
||
indicators={"ma5": [7.0]})
|
||
df = to_frame([item], backend=backend)
|
||
assert rows(df, backend) == 2
|
||
assert cell(df, backend, "ma5") == 7.0
|
||
assert missing(cell(df, backend, "ma5", 1))
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_index_trade_stats_single_and_multi(backend):
|
||
stat = IndexTradeStats(
|
||
index="sh000300", date=20261027, member_count=300, active_members=299,
|
||
trade_count_members=299,
|
||
minutes=[MinuteTradeStat(time="09:31", volume=10.0, trade_count=5, amount=1000.0)],
|
||
)
|
||
single = to_frame(stat, backend=backend)
|
||
assert cols(single, backend) == ["index", "date", "time", "volume", "trade_count", "amount"]
|
||
assert cell(single, backend, "index") == "sh000300"
|
||
|
||
multi = to_frame([stat, stat], backend=backend)
|
||
assert rows(multi, backend) == 2
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_batch_dict_is_concatenated(backend):
|
||
data = {
|
||
"600000": [bar(code="600000"), bar(code="600000", minute=1)],
|
||
"600036": [bar(code="600036", symbol="sh600036")],
|
||
}
|
||
df = to_frame(data, backend=backend)
|
||
assert rows(df, backend) == 3
|
||
assert cols(df, backend)[:2] == ["market", "code"]
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_batch_dict_all_empty_returns_typed_empty(backend):
|
||
df = to_frame({"600000": [], "600036": []}, backend=backend, model="SecurityBar")
|
||
assert rows(df, backend) == 0
|
||
assert "datetime" in cols(df, backend)
|
||
|
||
|
||
@pytest.mark.parametrize("backend", BACKENDS)
|
||
def test_list_of_dicts_uses_key_union(backend):
|
||
df = to_frame([{"name": "银行", "rise": 1.2}, {"name": "券商", "amount": 5.0}],
|
||
backend=backend)
|
||
assert cols(df, backend) == ["name", "rise", "amount"]
|
||
assert missing(cell(df, backend, "amount", 0))
|
||
|
||
|
||
def test_unsupported_payload_raises_type_error():
|
||
with pytest.raises(TypeError, match="无法转换为 DataFrame 的返回值类型"):
|
||
to_frame(42, backend="polars")
|
||
with pytest.raises(TypeError, match="无法转换为 DataFrame 的列表元素"):
|
||
to_frame([1.0, 2.0], backend="polars")
|
||
|
||
|
||
def test_subclass_reuses_parent_plan():
|
||
df = to_frame([IndexBar(market=1, code="000001", open=1.0, high=1.0, low=1.0,
|
||
close=1.0, vol=1.0, amount=1.0, year=2026, month=1, day=5)],
|
||
backend="polars")
|
||
assert "datetime" in df.columns
|
||
assert "year" not in df.columns
|
||
|
||
|
||
# ---------------------------------------------------------------- 接线
|
||
|
||
|
||
def test_tdxdata_default_keeps_legacy_behaviour():
|
||
td = TdxData()
|
||
assert td._df_backend is None
|
||
|
||
|
||
def test_tdxdata_rejects_unknown_backend():
|
||
with pytest.raises(ValueError, match="未知的 DataFrame 后端"):
|
||
TdxData(df="numpy")
|
||
|
||
|
||
def test_tdxdata_accepts_supported_backend():
|
||
td = TdxData(df="polars")
|
||
assert td._df_backend == "polars"
|
||
|
||
|
||
class _Source:
|
||
"""最小替身:只验装饰器语义,不碰网络。"""
|
||
|
||
def __init__(self, backend: str | None) -> None:
|
||
self._df_backend = backend
|
||
|
||
@framed
|
||
async def single(self) -> list[SecurityBar]:
|
||
return [bar()]
|
||
|
||
@framed
|
||
async def nested(self) -> dict[str, list[SecurityBar]]:
|
||
got = await self.single() # 内层必须是原始 list
|
||
assert isinstance(got, list), f"内层被提前转换成了 {type(got)!r}"
|
||
return {"600000": got}
|
||
|
||
|
||
async def test_decorator_passthrough_when_no_backend():
|
||
src = _Source(None)
|
||
assert isinstance(await src.single(), list)
|
||
|
||
|
||
async def test_decorator_converts_outermost_only():
|
||
src = _Source("polars")
|
||
out = await src.nested()
|
||
assert cols(out, "polars")[0] == "market"
|
||
assert rows(out, "polars") == 1
|
||
|
||
|
||
def test_model_name_resolution():
|
||
# 注解是 dict[str, list[SecurityBar]] 时正则也能取到 SecurityBar;
|
||
# 但 "dict" 这类内建名在 models 里找不到,必须返回 None 而不是内建类型
|
||
assert _frames._model_by_name("dict") is None
|
||
assert _frames._model_by_name("SecurityBar") is SecurityBar
|
||
|
||
|
||
def test_datetime_dtype_is_datetime_like():
|
||
pl = pytest.importorskip("polars")
|
||
df = to_frame([bar()], backend="polars")
|
||
assert isinstance(df.schema["datetime"], pl.Datetime)
|