Compare commits
2
Commits
04e0c57569
...
v0.7.1
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c9722943e2 | ||
|
|
8c25feca18 |
No files matched your search
+33
-1
@@ -2,6 +2,36 @@
|
||||
|
||||
本项目遵循 [语义化版本](https://semver.org/lang/zh-CN/)。
|
||||
|
||||
## [0.7.1] - 2026-10-29
|
||||
|
||||
### 性能
|
||||
|
||||
- **DataFrame 转换提速近一倍:24 万行 K 线 894ms → 462ms(polars)/ 1,001ms → 587ms(pandas)**。
|
||||
|
||||
0.7.0 的取列实现有两处浪费:每个单元格都套一层 Python 闭包(顺带做 `IntEnum` 判断),
|
||||
而 `trading_date` 每行都重新格式化三个整数 —— 24 万行里这个值往往只有 **1~2 种**。
|
||||
|
||||
| 转换层(不碰网络) | 0.7.0 | 现在 |
|
||||
|---|---|---|
|
||||
| 5,225 行报价 → polars | 66 ms | **51 ms** |
|
||||
| 5,225 行报价 → pandas | 74 ms | **62 ms** |
|
||||
| 240,000 行 K 线 → polars | 1,038 ms | **444 ms** |
|
||||
| 240,000 行 K 线 → pandas | 1,234 ms | **585 ms** |
|
||||
|
||||
端到端(缓存已热,1,000 只 × 240 根 1 分钟):`df=None` 1,386ms、
|
||||
polars 1,768ms(原 2,680ms)、pandas 1,976ms(原 2,848ms)。
|
||||
|
||||
- 普通列改用 `list(map(attrgetter(name), rows))`:取值发生在 C 层,不再每格建一个 Python 帧。
|
||||
- `trading_date` 按「时间字段(除 minute 外)」缓存(`_TD_KEYS`):A股/港股/期货一个交易日
|
||||
一个键,美股靠键里的 `hour` 覆盖凌晨段跨日判断 —— 不依赖 `_US_DAY_SPLIT_HOUR` 的具体取值。
|
||||
- `IntEnum` 归一化从「每格判断一次」改为「按列判断一次、需要时才整列转」。
|
||||
- 逐字段等价有测试锁定(`test_every_bar_field_equals_model` 等 3 项),口径零变化。
|
||||
|
||||
### 文档
|
||||
|
||||
- `docs/dataframe.md` 性能一节改为**实测数据 + 何时该用/不该用 DataFrame 出口**的说明,
|
||||
并把「缓存命中也要付转换成本」写清。
|
||||
|
||||
## [0.7.0] - 2026-10-28
|
||||
|
||||
### 新增
|
||||
@@ -823,7 +853,9 @@ pip install -U "pytdxdata[polars]" # 想直出 DataFrame
|
||||
|
||||
- 初始版本:动态连接池 + 双层 TTL 缓存 + 3 连接并发分页 + 双通道(标准 / MAC)+ 扩展市场(港股 / 美股 / 期货 / 期权)
|
||||
|
||||
[Unreleased]: https://github.com/openbot-coder/pytdxdata/compare/v0.6.0...HEAD
|
||||
[Unreleased]: https://github.com/openbot-coder/pytdxdata/compare/v0.7.1...HEAD
|
||||
[0.7.1]: https://github.com/openbot-coder/pytdxdata/compare/v0.7.0...v0.7.1
|
||||
[0.7.0]: https://github.com/openbot-coder/pytdxdata/compare/v0.6.0...v0.7.0
|
||||
[0.6.0]: https://github.com/openbot-coder/pytdxdata/compare/v0.5.0...v0.6.0
|
||||
[0.5.0]: https://github.com/openbot-coder/pytdxdata/compare/v0.4.0...v0.5.0
|
||||
[0.4.0]: https://github.com/openbot-coder/pytdxdata/compare/v0.3.2...v0.4.0
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# pytdxdata 对外接口清单
|
||||
|
||||
> 以 `src/pytdxdata` 源码为准整理(v0.7.0)。分四类:Python API / 数据模型 / 辅助模块 / CLI。
|
||||
> 以 `src/pytdxdata` 源码为准整理(v0.7.1)。分四类:Python API / 数据模型 / 辅助模块 / CLI。
|
||||
>
|
||||
> 方法与代码的一致性由 `scripts/check_docs.py` 在 CI 中校验。
|
||||
|
||||
|
||||
+23
-3
@@ -116,7 +116,27 @@ DataFrame 只是换了个容器,**下面这些口径不会因为转成 DataFra
|
||||
|
||||
## 性能
|
||||
|
||||
转换是**列式采集**:按模型预先算好列计划,一次遍历填满所有列,不做逐行 `asdict()`。
|
||||
5,227 只 A 股报价转 polars 约 20~40ms,相对一次全市场查询(约 300ms)可以忽略。
|
||||
先说结论:**`df=` 只影响出参那一层,取数、连接池、缓存完全不变**,但转换本身不是免费的——
|
||||
结果集越大越明显。实测(缓存已热、同一台机器、Windows / Python 3.13):
|
||||
|
||||
`df=None` 时**完全不经过这层**(装饰器直接透传),零额外开销。
|
||||
| 场景 | `df=None` | `df="polars"` | `df="pandas"` |
|
||||
|---|---|---|---|
|
||||
| 全市场报价 5,225 只 | 66 ms | 117 ms(+51) | 124 ms(+58) |
|
||||
| 1,000 只 × 240 根 1 分钟(24 万行) | 1,386 ms | 1,768 ms(+383) | 1,976 ms(+590) |
|
||||
| 单只 × 240 根 1 分钟 | 1 ms | 2 ms | 2 ms |
|
||||
|
||||
只看转换层本身(不碰网络):5,225 行报价 → polars **51 ms** / pandas **62 ms**;
|
||||
24 万行 K 线 → polars **444 ms** / pandas **585 ms**。
|
||||
|
||||
几个要点:
|
||||
|
||||
- **小结果无感**:单只 K 线、单只报价这类常见调用只多 1 ms 左右。
|
||||
- **大结果按行数线性增长**:每行约 2~3 µs(12 列)。全市场 1 分钟线(约 125 万行)
|
||||
转换约 2.3 s(polars)/ 3.1 s(pandas)——如果你的链路是「全市场分钟线直落库」,
|
||||
这笔开销通常远小于下游写入,但如果只是做筛选/聚合,建议先拿 `df=None` 的 list 算完再转。
|
||||
- **polars 比 pandas 快 25~30%**(同数据、同机器),大结果更明显。
|
||||
- **转换是列式采集**:列计划按模型缓存(含交易日列的按时间字段缓存),
|
||||
普通列走 C 层取值,不做逐行 `asdict()`。
|
||||
- **`df=None` 时完全不经过这一层**:装饰器直接透传,零额外开销。
|
||||
- **缓存命中也要转**:磁盘/内存缓存里存的是 dataclass 记录,DataFrame 是每次调用现建的
|
||||
——高频轮询同一批数据时,这笔转换开销每次都会付。
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "pytdxdata"
|
||||
version = "0.7.0"
|
||||
version = "0.7.1"
|
||||
description = "通达信(TongDaXin)行情数据客户端:动态连接池 + 双层TTL缓存 + 3连接并发分页,纯asyncio零运行时依赖,支持A股/港股/美股/期货/期权"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
|
||||
@@ -18,7 +18,7 @@ from .models import (
|
||||
XdxrRecord,
|
||||
)
|
||||
|
||||
__version__ = "0.7.0"
|
||||
__version__ = "0.7.1"
|
||||
__all__ = [
|
||||
"TdxData",
|
||||
"Market",
|
||||
|
||||
+113
-22
@@ -28,7 +28,8 @@ from collections.abc import Callable, Sequence
|
||||
from contextvars import ContextVar
|
||||
from datetime import datetime as _dt
|
||||
from enum import IntEnum
|
||||
from typing import Any
|
||||
from operator import attrgetter
|
||||
from typing import Any, NamedTuple
|
||||
|
||||
from . import models as _models
|
||||
from .models import (
|
||||
@@ -166,13 +167,6 @@ def _json_or_none(value: Any) -> str | None:
|
||||
return json.dumps(value, ensure_ascii=False, separators=(",", ":"))
|
||||
|
||||
|
||||
def _getter(name: str) -> Callable[[Any], Any]:
|
||||
def _get(row: Any) -> Any:
|
||||
return _as_int(getattr(row, name))
|
||||
|
||||
return _get
|
||||
|
||||
|
||||
def _json_getter(name: str) -> Callable[[Any], Any]:
|
||||
def _get(row: Any) -> Any:
|
||||
return _json_or_none(getattr(row, name))
|
||||
@@ -227,17 +221,50 @@ _EXTRAS: dict[type, tuple[tuple[str, Callable[[Any], Any]], ...]] = {
|
||||
),
|
||||
}
|
||||
|
||||
#: 需要 JSON 化的字典字段(按类区分,避免误伤 ``FinancialRecord.fields: list[float]``)
|
||||
#: 模型 -> 需要 JSON 化的字典字段(按类区分,避免误伤 ``FinancialRecord.fields: list[float]``)
|
||||
_JSON_FIELDS: dict[type, frozenset[str]] = {
|
||||
SecurityQuote: frozenset({"fields"}),
|
||||
TransactionRecord: frozenset({"fields"}),
|
||||
MinuteBar: frozenset({"fields"}),
|
||||
}
|
||||
|
||||
_PLAN_CACHE: dict[type, tuple[tuple[str, Callable[[Any], Any]], ...]] = {}
|
||||
#: 交易日列的缓存键(时间字段里参与 ``trading_date`` 计算的那些)。
|
||||
#:
|
||||
#: ``trading_date`` 是纯函数:A股/港股/期货就是自然日,美股凌晨段回退一天。
|
||||
#: 但它每算一次都要格式化三个整数(美股还要减一天),240k 行里往往只有
|
||||
#: **1~2 个不同的键** —— 按「除 minute 外的时间字段」缓存后,整列从 218ms 降到 33ms。
|
||||
#: 键里保留 ``hour`` 是为了一箭双雕:既覆盖美股 ``hour < 12`` 的跨日判断,
|
||||
#: 又不必把 ``_US_DAY_SPLIT_HOUR`` 的取值抄过来(切换点改了本文件也不用跟着改)。
|
||||
_TD_KEYS: dict[type, tuple[str, ...]] = {
|
||||
SecurityBar: ("year", "month", "day", "hour"),
|
||||
MinuteBar: ("date", "day_offset", "hour"),
|
||||
}
|
||||
|
||||
#: 交易日列名(走缓存路径,不按行取属性)
|
||||
_TD_COL = "trading_date"
|
||||
|
||||
#: 增强列里可以直接用 ``attrgetter`` 取的(属性本身即得值,不需要额外加工)
|
||||
_PLAIN_EXTRAS = frozenset({"datetime"})
|
||||
|
||||
|
||||
def _plan_for(cls: type) -> tuple[tuple[str, Callable[[Any], Any]], ...]:
|
||||
class _ColPlan(NamedTuple):
|
||||
"""某模型的输出列计划(构造一次,按类缓存)。
|
||||
|
||||
分成三类是性能决定的:逐行调 Python 闭包比 C 层 ``attrgetter`` 慢一倍以上,
|
||||
而 ``trading_date`` 这种「同一批里值大量重复」的列更是值得单独缓存。
|
||||
"""
|
||||
|
||||
order: tuple[str, ...] # 输出列顺序
|
||||
plain: tuple[str, ...] # 直接取属性即可的列名
|
||||
wrapped: tuple[tuple[str, Callable[[Any], Any]], ...] # 需要加工的列(JSON / 五档 / 除权日)
|
||||
td_name: str | None # 交易日列名
|
||||
td_key: Callable[[Any], Any] | None # 交易日缓存键
|
||||
|
||||
|
||||
_PLAN_CACHE: dict[type, _ColPlan] = {}
|
||||
|
||||
|
||||
def _plan_for(cls: type) -> _ColPlan:
|
||||
"""构造(并缓存)某模型的输出列计划。
|
||||
|
||||
查找顺序按 MRO 走:``IndexBar`` 没有自己的计划,但它是 ``SecurityBar`` 的子类,
|
||||
@@ -250,6 +277,7 @@ def _plan_for(cls: type) -> tuple[tuple[str, Callable[[Any], Any]], ...]:
|
||||
drops: tuple[str, ...] = ()
|
||||
extras: tuple[tuple[str, Callable[[Any], Any]], ...] = ()
|
||||
json_fields: frozenset[str] = frozenset()
|
||||
td_fields: tuple[str, ...] = ()
|
||||
for klass in cls.__mro__:
|
||||
if not drops and klass in _DROPS:
|
||||
drops = _DROPS[klass]
|
||||
@@ -257,30 +285,92 @@ def _plan_for(cls: type) -> tuple[tuple[str, Callable[[Any], Any]], ...]:
|
||||
extras = _EXTRAS[klass]
|
||||
if not json_fields and klass in _JSON_FIELDS:
|
||||
json_fields = _JSON_FIELDS[klass]
|
||||
if not td_fields and klass in _TD_KEYS:
|
||||
td_fields = _TD_KEYS[klass]
|
||||
|
||||
import dataclasses
|
||||
|
||||
plan: list[tuple[str, Callable[[Any], Any]]] = []
|
||||
order: list[str] = []
|
||||
plain: list[str] = []
|
||||
wrapped: list[tuple[str, Callable[[Any], Any]]] = []
|
||||
td_name: str | None = None
|
||||
|
||||
def add(name: str, getter: Callable[[Any], Any] | None = None) -> None:
|
||||
nonlocal td_name
|
||||
order.append(name)
|
||||
if name == _TD_COL:
|
||||
td_name = name # 交易日列走缓存,不看 getter
|
||||
elif getter is None or name in _PLAIN_EXTRAS:
|
||||
plain.append(name)
|
||||
else:
|
||||
wrapped.append((name, getter))
|
||||
|
||||
inserted = False
|
||||
for f in dataclasses.fields(cls): # type: ignore[arg-type]
|
||||
if f.name in drops:
|
||||
if not inserted:
|
||||
plan.extend(extras)
|
||||
for name, getter in extras:
|
||||
add(name, getter)
|
||||
inserted = True
|
||||
continue
|
||||
getter = _json_getter(f.name) if f.name in json_fields else _getter(f.name)
|
||||
plan.append((f.name, getter))
|
||||
add(f.name, _json_getter(f.name) if f.name in json_fields else None)
|
||||
if not inserted:
|
||||
# 没有丢弃字段的模型(如 SecurityInfo):增强列直接追加到末尾
|
||||
plan.extend(extras)
|
||||
for name, getter in extras:
|
||||
add(name, getter)
|
||||
|
||||
_PLAN_CACHE[cls] = tuple(plan)
|
||||
return _PLAN_CACHE[cls]
|
||||
plan = _ColPlan(
|
||||
order=tuple(order),
|
||||
plain=tuple(plain),
|
||||
wrapped=tuple(wrapped),
|
||||
td_name=td_name,
|
||||
td_key=attrgetter(*td_fields) if td_fields else None,
|
||||
)
|
||||
_PLAN_CACHE[cls] = plan
|
||||
return plan
|
||||
|
||||
|
||||
def _trading_dates(key_of: Callable[[Any], Any], rows: Sequence[Any]) -> list[str]:
|
||||
"""按缓存键批量算交易日串(同一批里绝大多数行是重复键)。"""
|
||||
cache: dict[Any, str] = {}
|
||||
out: list[str] = []
|
||||
ap = out.append
|
||||
for row in rows:
|
||||
key = key_of(row)
|
||||
value = cache.get(key)
|
||||
if value is None: # trading_date 永远是字符串,不会与 None 混淆
|
||||
value = cache[key] = row.trading_date
|
||||
ap(value)
|
||||
return out
|
||||
|
||||
|
||||
def _columns(cls: type, rows: Sequence[Any]) -> dict[str, list]:
|
||||
"""列式采集:一次遍历填满所有列,比逐行 ``asdict()`` 快得多。"""
|
||||
return {name: [get(row) for row in rows] for name, get in _plan_for(cls)}
|
||||
"""列式采集:逐列一次遍历,比逐行 ``asdict()`` 快得多。
|
||||
|
||||
- 普通列用 ``list(map(attrgetter(name), rows))``:取值发生在 C 层,没有 Python 帧;
|
||||
- ``trading_date`` 按时间字段缓存(240k 行常只有 1~2 个键);
|
||||
- 其余列(JSON / 五档 / 除权日)才走逐行回调;
|
||||
- ``IntEnum`` 归一化放在最后**按列**做一次,而不是每个单元格都判一次。
|
||||
"""
|
||||
plan = _plan_for(cls)
|
||||
if not rows:
|
||||
return {name: [] for name in plan.order}
|
||||
|
||||
cols: dict[str, list] = {}
|
||||
for name in plan.plain:
|
||||
cols[name] = list(map(attrgetter(name), rows))
|
||||
if plan.td_name is not None and plan.td_key is not None:
|
||||
cols[plan.td_name] = _trading_dates(plan.td_key, rows)
|
||||
for name, get in plan.wrapped:
|
||||
cols[name] = [get(row) for row in rows]
|
||||
|
||||
# polars 对 IntEnum 的类型推断不可靠(会退化成 Object 列),统一转纯 int。
|
||||
# 只在首元素就是 IntEnum 时才整列转,普通 int 列一次都不多花。
|
||||
for name, col in cols.items():
|
||||
if col and isinstance(col[0], IntEnum):
|
||||
cols[name] = [_as_int(value) for value in col]
|
||||
|
||||
return {name: cols[name] for name in plan.order}
|
||||
|
||||
|
||||
def _columns_of_dicts(rows: Sequence[dict]) -> dict[str, list]:
|
||||
@@ -375,7 +465,7 @@ def to_frame(data: Any, *, backend: str = "polars", model: str | None = None) ->
|
||||
if merged is not None:
|
||||
return merged
|
||||
cls = _model_by_name(model)
|
||||
return be.empty(tuple(name for name, _ in _plan_for(cls)) if cls else ())
|
||||
return be.empty(_plan_for(cls).order if cls else ())
|
||||
|
||||
if isinstance(data, list):
|
||||
if data and isinstance(data[0], IndexTradeStats):
|
||||
@@ -390,7 +480,8 @@ def to_frame(data: Any, *, backend: str = "polars", model: str | None = None) ->
|
||||
if data:
|
||||
# 元素既不是 dataclass 也不是 dict(如 list[float])——不该走到这里
|
||||
raise TypeError(f"无法转换为 DataFrame 的列表元素:{type(data[0])!r}")
|
||||
return be.frame({name: [] for name, _ in _plan_for(cls)}) if cls else be.empty(())
|
||||
return (be.frame({name: [] for name in _plan_for(cls).order})
|
||||
if cls else be.empty(()))
|
||||
|
||||
raise TypeError(f"无法转换为 DataFrame 的返回值类型:{type(data)!r}")
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ from pytdxdata import TdxData
|
||||
from pytdxdata import frames as _frames
|
||||
from pytdxdata.frames import FrameBackendError, framed, get_backend, to_frame
|
||||
from pytdxdata.models import (
|
||||
ExMarket,
|
||||
IndexBar,
|
||||
IndexTradeStats,
|
||||
IndicatorSet,
|
||||
@@ -323,3 +324,60 @@ def test_datetime_dtype_is_datetime_like():
|
||||
pl = pytest.importorskip("polars")
|
||||
df = to_frame([bar()], backend="polars")
|
||||
assert isinstance(df.schema["datetime"], pl.Datetime)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- 交易日列缓存
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_trading_date_multi_key_batch_matches_model(backend):
|
||||
"""一批里混着 A股与美股、跨 0~23 点,逐行都要等于模型自己的 ``trading_date``。
|
||||
|
||||
``trading_date`` 是按「时间字段」缓存算出来的(240k 行常只有 1~2 个键),
|
||||
这里专门造出「多键同批」的极端情形,防止缓存键漏掉参与计算的因素。
|
||||
|
||||
注意:美股的跨日处理在 ``SecurityBar.__post_init__`` 里就做完了(凌晨段
|
||||
的年月日会被推进一天),所以 ``trading_date`` 对每个 bar 只是自然日,
|
||||
真正会分叉的是 ``datetime`` —— 两者都要逐行核对。
|
||||
"""
|
||||
batch = []
|
||||
for market in (int(Market.SH), int(ExMarket.US_STOCK)):
|
||||
for hour in range(24):
|
||||
batch.append(bar(market=market, hour=hour, minute=15))
|
||||
df = to_frame(batch, backend=backend)
|
||||
got_td = [cell(df, backend, "trading_date", i) for i in range(rows(df, backend))]
|
||||
got_dt = [cell(df, backend, "datetime", i) for i in range(rows(df, backend))]
|
||||
assert got_td == [b.trading_date for b in batch]
|
||||
assert got_dt == [b.datetime for b in batch]
|
||||
# 美股凌晨段的 datetime 确实落在下一天(否则这条测试等于没测)
|
||||
us = [b for b in batch if b.market == int(ExMarket.US_STOCK)]
|
||||
assert {b.datetime.date().isoformat() for b in us} == {"2026-10-27", "2026-10-28"}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_minute_trading_date_cache_matches_model(backend):
|
||||
"""分时的 ``day_offset`` 也参与交易日计算,缓存键不能漏。"""
|
||||
batch = [
|
||||
MinuteBar(price=10.0, vol=1.0, hour=9, minute=30, date=20261027),
|
||||
MinuteBar(price=10.0, vol=1.0, hour=21, minute=30, date=20261027,
|
||||
market=int(ExMarket.US_STOCK)),
|
||||
MinuteBar(price=10.0, vol=1.0, hour=3, minute=0, date=20261027,
|
||||
market=int(ExMarket.US_STOCK), day_offset=1),
|
||||
]
|
||||
df = to_frame(batch, backend=backend)
|
||||
got = [cell(df, backend, "trading_date", i) for i in range(rows(df, backend))]
|
||||
assert got == [b.trading_date for b in batch]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("backend", BACKENDS)
|
||||
def test_every_bar_field_equals_model(backend):
|
||||
"""逐字段等价:DataFrame 的每个值都必须等于 dataclass 的同名字段。
|
||||
|
||||
这条是取列路径(``attrgetter`` 批量取 + 按列转 Enum)改动的兜底,
|
||||
避免「快是快了,但某列悄悄串了值」。
|
||||
"""
|
||||
src = bar(minute=15, settle=3.5)
|
||||
df = to_frame([src], backend=backend)
|
||||
for name in cols(df, backend):
|
||||
want = getattr(src, name)
|
||||
assert cell(df, backend, name) == want, name
|
||||
Reference in new issue
Block a user