Files
pytdxdata/tests/test_frames.py
T
openbotandopenbot-coder 04e0c57569 feat: DataFrame 直出(df="polars"/"pandas")+ 0.7.0 (#12)
* feat: DataFrame 直出(df="polars"/"pandas")+ 版本 0.7.0

TdxData 新增构造参数 df:
- None(默认)→ 行为与 0.6.x 完全一致,返回 list[dataclass]
- "polars"/"pandas" → 42 个表格型接口直接返回 DataFrame

转换层 src/pytdxdata/frames.py:
- 列计划按模型缓存,列式采集(不做逐行 asdict)
- 时间列替换:year/month/day/hour/minute → datetime + trading_date;Xdxr 给 date
- 嵌套定型:bid/ask 展开 10 列、fields 字典 JSON 化、codes 保持列表
- 容器展开:get_indicators 合并指标、get_index_trade_stats 展开 minutes、
  get_kline_batch/get_minute_batch 纵向拼接
- ContextVar 层次标记:批量/别名方法内层不二次转换,并发互不干扰
- 后端惰性加载:缺包抛 FrameBackendError(带安装命令),构造时即失败

依赖:新增 extras pandas/polars/dataframe,核心包仍零依赖;dev 组加 pandas+polars
测试:tests/test_frames.py 40 项(双后端×双格式);全量 401 项通过
文档:新增 docs/dataframe.md,同步 API_REFERENCE/SKILL/llms.txt/README/mkdocs

* perf(kline): 按日期取K线贴最新时走倒推,1分钟 4.2s → 82ms

问题:start_date/end_date 一律走「探总根数 + 二分定位」。1 分钟 K 线历史
二十万根量级 = 6 次翻倍探测 + 2 次方向探测 + 两次各 ~18 次跳转,共 36+ 次
串行往返。「取今天」这个最常见的调用要 4.2 秒。

改法(_bars_backtrack 快路径):
- 右端离最新不超过 _BACKTRACK_GRACE(10 天) 时,直接 start=0, count=N
  拿最新 N 根再裁剪;N 按时间跨度估(_BARS_PER_DAY),一天内一次命中,
  不够才翻倍,放大到 _BACKTRACK_MAX_BARS 仍不够则回落二分
- 历史中段仍走二分,结果口径完全一致(已逐条比对基准)

两个关键细节:
- 偏移方向是通道级属性而非标的属性,首次探测后缓存在实例上
  (_kline_dir_cache)——全市场 5,225 只批量只付一次探测成本
- K线 hour/minute 是周期**结束**时刻,最新一根可能比 now() 略大(14:29
  那根还在走),「右端贴最新」必须留周期容差,否则 end_date=now() 被误判
  成历史中段

实测(1 分钟,冷启动):
  取今天        4,208ms → 82ms
  2天前~now     2,211ms → 75ms
  最近3天       2,000ms+ → 239ms
  历史中段      1,512ms → 285ms
  全市场5225只取今天 32.1s(1,140,699 根,精确当天口径)

注:route_for 让 MIN_1 一律走 MAC(0x122E),所以按通道做方向缓存而不是按
「标准/扩展」写死分支。

测试:新增 tests/test_kline_date.py 6 项(倒推次数上界、结束时刻容差、
中段回落二分、未来区间、反向通道、与 count 路径口径一致);全量 407 项通过

* perf(kline): 全市场1分钟K线改多连接流水线抓取(43.6s → 5~7s,盘中 2s)

瓶颈其实是**往返次数**不是带宽:单只K线一请求一标的,串行 33ms/只 ≈ 1 个 RTT。
同一台服务器 16 连接 × 深度 8 实测能跑 5.2 MB/s(此前「单 IP 限流 1.5MB/s」
是浅流水下的假象)。

- get_bars 新增 pipeline 参数(None=按单请求字节自适应 8~32,1=关闭走逐只)
- 多标的(≥16 只)走 _bulk_bars:多连接 work-stealing + 单连接 execute_pipeline
- 只用实测最快的 12 台:池里 24 台全用反而被慢节点拖长尾
- 盘中按交易时段只拉已走过的根数(10:00 → 35 根,数据量差 7 倍)
- 标准通道(日线不复权)同样走批量:全市场 49~72s → 13s
- 修 pipeline 失败时队列剩余标的静默丢失(实测 20 只只回 16 只)
- 修 test_future_range_returns_empty_quickly 时间炸弹(固定日期会漂进 _BACKTRACK_GRACE)
- conftest 关闭后台测速 task,避免污染「往返次数」类断言

实测(5,225 只):count=2 → 2.7s;count=35 → 2.2s;count=240 → 6.9s。
逐根一致性 30/30 相同。新增 tests/test_bulk_kline.py 12 项,合计 419 项全绿。

* test: 批量K线测试的时间锚点改用真实 now(CI 是 UTC,固定日期会漂到别的时段分支)

* fix(ex): EX 通道 K 线回填成交额与结算价(0x23FF 槽位按市场分两类)

报告:EX 通道 get_bars 返回的 SecurityBar.amount 恒为 0.0。
定位属实,但实际情况比报告更细 —— 32B 记录里那 28B 行情只有**一个** 4B 槽位
可复用,语义按市场分两类(全市场实测,用 成交额/(close×vol) 逐条校验):

| 市场 | 槽位5 | 槽位6 | 槽位7 |
|---|---|---|---|
| 港股主板31/港股基金49 | 成交额(f) | 量(手=100股) | 未知 |
| 港股创业板48/新三板44/美股74 | 成交额(f) | 量(股) | 0 |
| 期货/期权 7/8/9/47… | 持仓量(I) | 量 | 结算价(f) |

改前 amount 硬编码 0.0、第 7 个 float 用 `_` 丢弃 —— 于是港股/美股/新三板
明明帧内有成交额却全为 0,期货期权的结算价也拿不到。

- 新增 SecurityBar.settle(结算价,float|None):仅 EX 期货/期权日线有值,
  分钟线该槽位恒 0 已归一化为 None,其它通道恒 None。
- 权益类市场回填真实 amount;港股主板/基金的 vol 单位是手(=100股),
  amount/vol 是均价的 100 倍,已写进模型注释与量纲表。
- 期货/期权的 amount 仍是 0.0(帧内确实没有成交额,协议限制),
  已在 SecurityBar / API_REFERENCE / llms.txt / SKILL.md 写清,
  避免使用者再按旧文档判断成调用方 bug。
- 港股指数27/中证指数62 该槽位量级对不上(既非额也非结算价),保守保持 0.0。
- SecurityBar.__setstate__ 兜底旧磁盘缓存:缓存键无 schema 版本,升级后读旧
  pickle 缺 settle 槽位,默认反序列化不补默认值 → AttributeError,已容错;
  EX K 线缓存键额外加 @v2,让 amount 恒 0 的旧缓存自然失效。
- K 线 DataFrame 因此多一列 settle(0.7.0 未发布,列清单同步更新)。

测试:tests/test_ex_bars_fields.py 16 项(解析层双布局、日内 settle 归零、
API 透传、旧 pickle 兼容);全套 435 项通过。

* docs: 0.7.0 全站口径同步(DataFrame 出口 + EX 成交额限制 + 构造参数表)

- 首页:特性表补「DataFrame 直出」与 K 线批量性能,导航新增 DataFrame 卡片,
  统一方法数按实测修正为 35(+33 别名 = 68 个公开方法)
- 安装页:补 extras(polars/pandas/dataframe)与 df 用法;构造参数表补
  df / std_pool_cap / ex_pool_cap,并把 pool_max 默认值从 12 修正为 24(实际值)
- 场景示例:`转成 pandas` 一节改为先推 df= 开关,手动转换降级为局部场景
- 数据模型页:不再说「刻意不提供 DataFrame」,改为 dataclass 与 DataFrame
  是同一份数据、两种容器
- AI 接入页:返回值约定补 `df=` 开关的说法
- README:K 线列注释补 settle;已知限制补 EX 通道成交额/结算价的协议口径
- CHANGELOG:新增「升级」小节(升级命令、旧缓存无需清理)

---------

Co-authored-by: openbot-coder <openbot-coder@users.noreply.github.com>
2026-10-28 18:22:13 +08:00

326 lines
11 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""DataFrame 转换层测试(0.7.0)。
覆盖两条独立路径:
1. **转换口径**:直接调 :func:`to_frame`,喂合成模型对象,断言列名/列序/取值/dtype。
不碰网络,双后端各跑一遍。
2. **接线**:``TdxData(df=...)`` 的取值校验、装饰器的「内层不二次转换」语义。
"""
from __future__ import annotations
import math
import sys
import pytest
from pytdxdata import TdxData
from pytdxdata import frames as _frames
from pytdxdata.frames import FrameBackendError, framed, get_backend, to_frame
from pytdxdata.models import (
IndexBar,
IndexTradeStats,
IndicatorSet,
Market,
MinuteBar,
MinuteTradeStat,
SecurityBar,
SecurityQuote,
TdxBlock,
XdxrRecord,
)
BACKENDS = ("polars", "pandas")
# ---------------------------------------------------------------- 工具
def bar(**kw) -> SecurityBar:
base = {
"market": int(Market.SH), "code": "600000", "open": 10.0, "high": 11.0,
"low": 9.5, "close": 10.5, "vol": 1000.0, "amount": 10500.0,
"year": 2026, "month": 10, "day": 27, "hour": 0, "minute": 0,
"symbol": "sh600000",
}
base.update(kw)
return SecurityBar(**base)
def rows(df, backend: str) -> int:
return len(df) if backend == "pandas" else df.height
def cols(df, backend: str) -> list[str]:
return list(df.columns)
def cell(df, backend: str, name: str, index: int = 0):
if backend == "pandas":
return df[name].iloc[index]
# polars 2.x:Series[0] 会返回 Series 而不是标量,统一走 to_list()
return df.get_column(name).to_list()[index]
def missing(value) -> bool:
"""pandas 的数值列会把 None 变成 NaN,两者都算「缺」。"""
return value is None or (isinstance(value, float) and math.isnan(value))
# ---------------------------------------------------------------- 后端
def test_backend_instances_are_cached():
assert get_backend("polars") is get_backend("polars")
assert get_backend("pandas") is get_backend("pandas")
def test_unknown_backend_raises_value_error():
with pytest.raises(ValueError, match="未知的 DataFrame 后端"):
get_backend("numpy")
def test_missing_backend_raises_with_install_hint(monkeypatch):
monkeypatch.setitem(sys.modules, "polars", None) # 让 import polars 抛 ImportError
monkeypatch.delitem(_frames._BACKEND_CACHE, "polars", raising=False)
with pytest.raises(FrameBackendError) as excinfo:
get_backend("polars")
assert isinstance(excinfo.value, ImportError)
assert excinfo.value.name == "polars"
assert "pip install pytdxdata[polars]" in str(excinfo.value)
monkeypatch.delitem(_frames._BACKEND_CACHE, "polars", raising=False)
# ---------------------------------------------------------------- 转换口径
@pytest.mark.parametrize("backend", BACKENDS)
def test_bar_replaces_split_time_with_datetime(backend):
df = to_frame([bar(), bar(minute=30)], backend=backend)
assert cols(df, backend) == [
"market", "code", "open", "high", "low", "close", "vol", "amount",
"datetime", "trading_date", "symbol", "settle",
]
assert rows(df, backend) == 2
assert cell(df, backend, "market") == 1
assert cell(df, backend, "trading_date") == "2026-10-27"
stamp = cell(df, backend, "datetime", 1)
assert stamp.year == 2026 and stamp.hour == 0 and stamp.minute == 30
@pytest.mark.parametrize("backend", BACKENDS)
def test_int_enum_column_is_plain_int(backend):
df = to_frame([bar(market=Market.SZ)], backend=backend)
value = cell(df, backend, "market")
assert value == 0
assert not isinstance(value, Market)
@pytest.mark.parametrize("backend", BACKENDS)
def test_empty_list_falls_back_to_annotation_hint(backend):
df = to_frame([], backend=backend, model="SecurityBar")
assert rows(df, backend) == 0
assert cols(df, backend) == [
"market", "code", "open", "high", "low", "close", "vol", "amount",
"datetime", "trading_date", "symbol", "settle",
]
@pytest.mark.parametrize("backend", BACKENDS)
def test_quote_expands_bid_ask_levels(backend):
quote = SecurityQuote(
market=1, code="600000", price=10.5, pre_close=10.0, open=10.1, high=10.6,
low=9.9, vol=100.0, amount=1000.0, s_vol=40.0, b_vol=60.0,
bid=[(10.49, 100.0), (10.48, 200.0)], ask=[(10.51, 300.0)],
symbol="sh600000", fields={"0x7": 1.5},
)
df = to_frame([quote], backend=backend)
out = cols(df, backend)
assert "bid1_price" in out and "ask5_vol" in out
assert "bid" not in out and "ask" not in out
assert cell(df, backend, "bid2_vol") == 200.0
assert missing(cell(df, backend, "bid3_price")) # 只有两档,缺档给 None
assert cell(df, backend, "fields") == '{"0x7":1.5}'
@pytest.mark.parametrize("backend", BACKENDS)
def test_empty_dict_field_becomes_none(backend):
df = to_frame([SecurityQuote(
market=1, code="600000", price=1.0, pre_close=1.0, open=1.0, high=1.0,
low=1.0, vol=1.0, amount=1.0, s_vol=0.0, b_vol=0.0,
)], backend=backend)
assert missing(cell(df, backend, "fields"))
@pytest.mark.parametrize("backend", BACKENDS)
def test_minute_uses_real_clock(backend):
late = MinuteBar(price=1.0, vol=10.0, avg_price=1.0, hour=27, minute=59,
symbol="usAAPL", market=74, code="AAPL", date=20261026)
df = to_frame([late], backend=backend)
assert cols(df, backend) == [
"price", "vol", "avg_price", "datetime", "trading_date",
"symbol", "market", "code", "fields",
]
stamp = cell(df, backend, "datetime")
assert (stamp.year, stamp.month, stamp.day, stamp.hour, stamp.minute) == (
2026, 10, 27, 3, 59,
)
assert cell(df, backend, "trading_date") == "2026-10-26"
@pytest.mark.parametrize("backend", BACKENDS)
def test_xdxr_date_column(backend):
row = XdxrRecord(market=1, code="600000", year=2026, month=6, day=12,
category=1, name="分红", fenhong=0.5, symbol="sh600000")
df = to_frame([row], backend=backend)
assert cols(df, backend)[:3] == ["market", "code", "date"]
stamp = cell(df, backend, "date")
assert (stamp.year, stamp.month, stamp.day) == (2026, 6, 12)
@pytest.mark.parametrize("backend", BACKENDS)
def test_tdx_block_keeps_codes_list(backend):
df = to_frame([TdxBlock(name="银行", category=0, count=2, codes=["600000", "600036"])],
backend=backend)
assert cols(df, backend) == ["name", "category", "count", "codes"]
assert cell(df, backend, "codes") == ["600000", "600036"]
@pytest.mark.parametrize("backend", BACKENDS)
def test_indicator_set_merges_bars_and_indicators(backend):
item = IndicatorSet(
symbol="sh600000",
bars=[bar(), bar(minute=1), bar(minute=2)],
indicators={"ma5": [1.0, 2.0, 3.0], "boll": [None, None, 9.0]},
)
df = to_frame([item], backend=backend)
assert rows(df, backend) == 3
assert cell(df, backend, "symbol") == "sh600000"
assert cell(df, backend, "ma5", 2) == 3.0
assert missing(cell(df, backend, "boll"))
@pytest.mark.parametrize("backend", BACKENDS)
def test_indicator_series_shorter_than_bars_is_padded(backend):
item = IndicatorSet(symbol="sh600000", bars=[bar(), bar(minute=1)],
indicators={"ma5": [7.0]})
df = to_frame([item], backend=backend)
assert rows(df, backend) == 2
assert cell(df, backend, "ma5") == 7.0
assert missing(cell(df, backend, "ma5", 1))
@pytest.mark.parametrize("backend", BACKENDS)
def test_index_trade_stats_single_and_multi(backend):
stat = IndexTradeStats(
index="sh000300", date=20261027, member_count=300, active_members=299,
trade_count_members=299,
minutes=[MinuteTradeStat(time="09:31", volume=10.0, trade_count=5, amount=1000.0)],
)
single = to_frame(stat, backend=backend)
assert cols(single, backend) == ["index", "date", "time", "volume", "trade_count", "amount"]
assert cell(single, backend, "index") == "sh000300"
multi = to_frame([stat, stat], backend=backend)
assert rows(multi, backend) == 2
@pytest.mark.parametrize("backend", BACKENDS)
def test_batch_dict_is_concatenated(backend):
data = {
"600000": [bar(code="600000"), bar(code="600000", minute=1)],
"600036": [bar(code="600036", symbol="sh600036")],
}
df = to_frame(data, backend=backend)
assert rows(df, backend) == 3
assert cols(df, backend)[:2] == ["market", "code"]
@pytest.mark.parametrize("backend", BACKENDS)
def test_batch_dict_all_empty_returns_typed_empty(backend):
df = to_frame({"600000": [], "600036": []}, backend=backend, model="SecurityBar")
assert rows(df, backend) == 0
assert "datetime" in cols(df, backend)
@pytest.mark.parametrize("backend", BACKENDS)
def test_list_of_dicts_uses_key_union(backend):
df = to_frame([{"name": "银行", "rise": 1.2}, {"name": "券商", "amount": 5.0}],
backend=backend)
assert cols(df, backend) == ["name", "rise", "amount"]
assert missing(cell(df, backend, "amount", 0))
def test_unsupported_payload_raises_type_error():
with pytest.raises(TypeError, match="无法转换为 DataFrame 的返回值类型"):
to_frame(42, backend="polars")
with pytest.raises(TypeError, match="无法转换为 DataFrame 的列表元素"):
to_frame([1.0, 2.0], backend="polars")
def test_subclass_reuses_parent_plan():
df = to_frame([IndexBar(market=1, code="000001", open=1.0, high=1.0, low=1.0,
close=1.0, vol=1.0, amount=1.0, year=2026, month=1, day=5)],
backend="polars")
assert "datetime" in df.columns
assert "year" not in df.columns
# ---------------------------------------------------------------- 接线
def test_tdxdata_default_keeps_legacy_behaviour():
td = TdxData()
assert td._df_backend is None
def test_tdxdata_rejects_unknown_backend():
with pytest.raises(ValueError, match="未知的 DataFrame 后端"):
TdxData(df="numpy")
def test_tdxdata_accepts_supported_backend():
td = TdxData(df="polars")
assert td._df_backend == "polars"
class _Source:
"""最小替身:只验装饰器语义,不碰网络。"""
def __init__(self, backend: str | None) -> None:
self._df_backend = backend
@framed
async def single(self) -> list[SecurityBar]:
return [bar()]
@framed
async def nested(self) -> dict[str, list[SecurityBar]]:
got = await self.single() # 内层必须是原始 list
assert isinstance(got, list), f"内层被提前转换成了 {type(got)!r}"
return {"600000": got}
async def test_decorator_passthrough_when_no_backend():
src = _Source(None)
assert isinstance(await src.single(), list)
async def test_decorator_converts_outermost_only():
src = _Source("polars")
out = await src.nested()
assert cols(out, "polars")[0] == "market"
assert rows(out, "polars") == 1
def test_model_name_resolution():
# 注解是 dict[str, list[SecurityBar]] 时正则也能取到 SecurityBar;
# 但 "dict" 这类内建名在 models 里找不到,必须返回 None 而不是内建类型
assert _frames._model_by_name("dict") is None
assert _frames._model_by_name("SecurityBar") is SecurityBar
def test_datetime_dtype_is_datetime_like():
pl = pytest.importorskip("polars")
df = to_frame([bar()], backend="polars")
assert isinstance(df.schema["datetime"], pl.Datetime)