Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/evidence_package_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,16 @@ If any of the following are missing, keep the profile out of live settings:
- strategy repo: produces the evidence package
- platform repo: verifies runtime compatibility and gate status
- operator review: makes the final live decision

## Canonical promotion package (v2)

Promotion reruns must produce a new `strategy_evidence_package.v2` that validates against the packaged `strategy-evidence-package.v2.schema.json` and the dependency-free Python validator. Do not relabel or implicitly migrate a v1/alias package.

The closed v2 object binds strategy and input provenance, the exact `BacktestOrchestrator` `purged_walk_forward.v1` output, at least three ordered folds, positive purge/embargo, an independent locked OOS window of at least 12 calendar months, timing/cost/risk/metric identities, verified repo-relative artifact bytes and SHA-256 digests, and a human acceptance bound to the evidence-core digest.

Required lifecycle claims are fail-closed:

- learning: `learning_only=true`, `promotion_eligible=false`, `live_ready=false`, `size_zero_required=true`, `no_order=true`;
- accepted promotion evidence may set `promotion_eligible=true`, but must still keep `live_ready=false`, `size_zero_required=true`, and `no_order=true`.

Structural validation does not invent performance thresholds. Metric quality remains a bound human promotion decision. A requested stage, CI/PR/review/health result, or notification never grants paper, shadow, live, order, or capital authority; live/runtime requests remain `HOLD`.
13 changes: 13 additions & 0 deletions docs/evidence_package_template.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,16 @@ requested_stage: live_candidate
- 策略仓库:产出证据包
- 平台仓库:验证 runtime 兼容性和门槛状态
- 操作审批:做最终 live 决策

## Canonical 晋级包(v2)

promotion rerun 必须重新生成 `strategy_evidence_package.v2`,并同时通过 packaged schema 与 dependency-free Python validator。不得把 v1/alias 静默补默认值或改标签后冒充 v2。

封闭的 v2 object 必须绑定 strategy/input provenance、`BacktestOrchestrator` 的 `purged_walk_forward.v1` 原始输出、至少 3 个有序 folds、正数 purge/embargo、至少 12 个日历月的锁定独立 OOS、calendar/timezone/signal/execution timing、cost/risk/全部指标,以及 repo-relative artifact 实际 bytes 与 SHA-256。human acceptance 必须用 evidence-core SHA-256 绑定当前证据。

生命周期真值必须 fail closed:

- learning:`learning_only=true`、`promotion_eligible=false`、`live_ready=false`、`size_zero_required=true`、`no_order=true`;
- 完整且经绑定的人类接受的证据可以 `promotion_eligible=true`,但仍必须 `live_ready=false`、`size_zero_required=true`、`no_order=true`。

结构验证不臆造性能阈值;指标质量仍由绑定的人类 promotion acceptance 判断。requested stage、CI、PR、review、health 或 notification 都不能产生 paper/shadow/live、order 或 capital 权限;live/runtime 请求一律 `HOLD`。
16 changes: 16 additions & 0 deletions docs/strategy_promotion_risk_standard.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,3 +124,19 @@ AI 自动优化必须遵守以下规则:
- [ ] `position_control_allowed=true` 已绑定 `evidence_package_id`
- [ ] 证据包有效期明确
- [ ] `bounded budget` 已输出且可审计

## `strategy_evidence_package.v2` 晋级证据门

晋级重跑必须由 producer 生成新的 `strategy_evidence_package.v2`;v1/alias 只保留研究与监控兼容,不自动迁移成 v2。v2 必须同时绑定:

- strategy/source revision、input provenance/license/range/timestamp/manifest digest;
- `BacktestOrchestrator` 的 `purged_walk_forward.v1` 输出、至少 3 个有序 folds、正数 purge/embargo,以及锁定且独立的至少 12 个日历月 OOS;
- calendar/timezone/signal/execution timing、config/data-manifest/backtest/risk/IC/cost artifacts 及其实际 bytes/SHA-256;
- 上述全部风险指标及 `information_coefficient`。所有 metric/cost 必须存在、非 bool 且有限,cost/risk 状态必须为 `PASS`;
- human acceptance 的 decision/id/actor/time/authority-receipt SHA-256,并以 evidence-core SHA-256 绑定当前证据。

机器只判断结构、身份、有限性、日期、digest 与 PASS 状态;本文未冻结 Sharpe、return、MDD 或 IC 数值阈值,指标质量仍由绑定的人类 promotion acceptance 判断。

本 v2 门只产生研究晋级资格,不产生 paper/shadow/live 权限:`live_ready=false`、`size_zero_required=true`、`no_order=true` 始终成立。`requested_stage`、CI、PR、review、health 或 notification 不能改变这些真值;legacy/v2 live 或 runtime 请求都必须 `HOLD`。

本门完成也不改变 P3 的 `TERMINALLY_PARKED_NO_MEMBER` 状态。
279 changes: 10 additions & 269 deletions scripts/validate_strategy_evidence_package.py
Original file line number Diff line number Diff line change
@@ -1,214 +1,22 @@
#!/usr/bin/env python3
"""Validate a strategy evidence package JSON file."""
"""Compatibility CLI for the canonical strategy evidence package validator."""

from __future__ import annotations

import argparse
import hashlib
import json
import re
import sys
from datetime import datetime
from pathlib import Path
from typing import Any

ALLOWED_REQUESTED_STAGES = {
"research_backtest_only",
"ai_monitored_candidate",
"shadow_candidate",
"live_candidate",
"runtime_enabled",
}
ALLOWED_KELLY_LEVELS = {"K0", "K1", "K2", "K3", "K4"}
REQUIRED_ARTIFACTS = (
"returns",
"trades",
"positions",
"config",
"data_manifest",
"candidate_registry",
"benchmark_registry",
"cost_model",
"risk_report",
"kelly_readiness_report",
from quant_platform_kit.strategy_lifecycle.evidence_package_v2 import (

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restore src path setup for direct CLI runs

When this script is invoked directly from a source checkout without an editable install, this import runs before src is on sys.path, so python scripts/validate_strategy_evidence_package.py ... now exits with ModuleNotFoundError instead of validating anything. The sibling source-checkout wrapper adds ROOT / "src" before importing package code, and this script needs the same bootstrap or another standalone path to preserve the documented/tested direct CLI workflow.

Useful? React with 👍 / 👎.

STRATEGY_EVIDENCE_PACKAGE_SCHEMA_VERSION,
canonical_evidence_package_v2_bytes,
read_evidence_package_v2_json,
validate_evidence_package_v2,
validate_strategy_evidence_file,
validate_strategy_evidence_payload,
)
REQUIRED_RISK_METRICS = (
"sharpe_ratio",
"sortino_ratio",
"max_drawdown",
"annualized_return",
"annualized_volatility",
"calmar_ratio",
"information_ratio",
"var_95",
"cvar_95",
"turnover",
"trade_count",
"win_rate",
"profit_factor",
)
REQUIRED_RISK_BENCHMARK = ("name", "alpha", "beta")
REQUIRED_RISK_COST_STRESS = ("slippage_bps", "commission_bps", "passed")
REQUIRED_RISK_OOS = ("window_start", "window_end", "locked")
SHA256_RE = re.compile(r"^[A-Fa-f0-9]{64}$")


def validate_payload(payload: Any, *, base_dir: Path | None = None) -> list[str]:
issues: list[str] = []

if not isinstance(payload, dict):
return ["top-level JSON must be an object"]

for field in (
"schema_version",
"profile",
"market",
"requested_stage",
"generated_at",
"evidence_package_id",
"artifacts",
"validation",
"risk",
"kelly_readiness",
"ai_optimization",
):
if field not in payload:
issues.append(f"missing required field: {field}")

_check_non_empty_string(payload, "schema_version", issues)
_check_non_empty_string(payload, "profile", issues)
_check_non_empty_string(payload, "market", issues)
_check_non_empty_string(payload, "evidence_package_id", issues)

requested_stage = payload.get("requested_stage")
if not isinstance(requested_stage, str) or not requested_stage.strip():
issues.append("requested_stage must be a non-empty string")
elif requested_stage not in ALLOWED_REQUESTED_STAGES:
issues.append(f"unsupported requested_stage: {requested_stage!r}")

generated_at = payload.get("generated_at")
if not isinstance(generated_at, str) or not generated_at.strip():
issues.append("generated_at must be a non-empty string")
elif not _is_datetime_string(generated_at):
issues.append(f"generated_at is not a valid date-time: {generated_at!r}")

artifacts = payload.get("artifacts")
if not isinstance(artifacts, dict):
issues.append("artifacts must be an object")
else:
for name in REQUIRED_ARTIFACTS:
artifact = artifacts.get(name)
if not isinstance(artifact, dict):
issues.append(f"artifacts.{name} must be an object")
continue
_check_non_empty_string(artifact, "path", issues, prefix=f"artifacts.{name}")
sha256 = artifact.get("sha256")
if not isinstance(sha256, str) or not SHA256_RE.fullmatch(sha256):
issues.append(f"artifacts.{name}.sha256 must be a 64-character hex string")
continue
if base_dir is not None:
_validate_artifact_file(
name=name,
artifact=artifact,
expected_sha256=sha256,
base_dir=base_dir,
issues=issues,
)

validation = payload.get("validation")
if not isinstance(validation, dict):
issues.append("validation must be an object")
else:
if not isinstance(validation.get("oos_passed"), bool):
issues.append("validation.oos_passed must be a boolean")
if not isinstance(validation.get("overfit_report_present"), bool):
issues.append("validation.overfit_report_present must be a boolean")
if requested_stage in {"live_candidate", "runtime_enabled"}:
if validation.get("oos_passed") is not True:
issues.append(f"{requested_stage} requires validation.oos_passed=true")
if validation.get("overfit_report_present") is not True:
issues.append(f"{requested_stage} requires validation.overfit_report_present=true")

risk = payload.get("risk")
if not isinstance(risk, dict):
issues.append("risk must be an object")
else:
metrics = risk.get("metrics")
if not isinstance(metrics, dict):
issues.append("risk.metrics must be an object")
else:
for field in REQUIRED_RISK_METRICS:
value = metrics.get(field)
if field == "trade_count":
if not _is_int(value):
issues.append("risk.metrics.trade_count must be an integer")
elif value < 0:
issues.append("risk.metrics.trade_count must be >= 0")
elif not _is_number(value):
issues.append(f"risk.metrics.{field} must be a number")
if _is_number(metrics.get("win_rate")):
win_rate = metrics["win_rate"]
if win_rate < 0 or win_rate > 1:
issues.append("risk.metrics.win_rate must be between 0 and 1")

benchmark = risk.get("benchmark")
if not isinstance(benchmark, dict):
issues.append("risk.benchmark must be an object")
else:
_check_non_empty_string(benchmark, "name", issues, prefix="risk.benchmark")
if not _is_number(benchmark.get("alpha")):
issues.append("risk.benchmark.alpha must be a number")
if not _is_number(benchmark.get("beta")):
issues.append("risk.benchmark.beta must be a number")

cost_stress = risk.get("cost_stress")
if not isinstance(cost_stress, dict):
issues.append("risk.cost_stress must be an object")
else:
if not _is_number(cost_stress.get("slippage_bps")):
issues.append("risk.cost_stress.slippage_bps must be a number")
if not _is_number(cost_stress.get("commission_bps")):
issues.append("risk.cost_stress.commission_bps must be a number")
if not isinstance(cost_stress.get("passed"), bool):
issues.append("risk.cost_stress.passed must be a boolean")

oos = risk.get("oos")
if not isinstance(oos, dict):
issues.append("risk.oos must be an object")
else:
_check_non_empty_string(oos, "window_start", issues, prefix="risk.oos")
_check_non_empty_string(oos, "window_end", issues, prefix="risk.oos")
if not isinstance(oos.get("locked"), bool):
issues.append("risk.oos.locked must be a boolean")

kelly_readiness = payload.get("kelly_readiness")
if not isinstance(kelly_readiness, dict):
issues.append("kelly_readiness must be an object")
else:
level = kelly_readiness.get("level")
if not isinstance(level, str) or level not in ALLOWED_KELLY_LEVELS:
issues.append("kelly_readiness.level must be one of K0, K1, K2, K3, K4")
if kelly_readiness.get("full_kelly_allowed") is not False:
issues.append("kelly_readiness.full_kelly_allowed must be false")

ai_optimization = payload.get("ai_optimization")
if not isinstance(ai_optimization, dict):
issues.append("ai_optimization must be an object")

return issues


def validate_file(path: str | Path) -> list[str]:
evidence_path = Path(path)
try:
payload = json.loads(evidence_path.read_text(encoding="utf-8"))
except FileNotFoundError:
return [f"file not found: {evidence_path}"]
except json.JSONDecodeError as exc:
return [f"invalid JSON: {exc.msg} (line {exc.lineno}, column {exc.colno})"]
except OSError as exc:
return [f"failed to read file: {exc}"]
return validate_payload(payload, base_dir=evidence_path.parent)
validate_payload = validate_strategy_evidence_payload
validate_file = validate_strategy_evidence_file


def main(argv: list[str] | None = None) -> int:
Expand All @@ -224,72 +32,5 @@ def main(argv: list[str] | None = None) -> int:
return 0


def _check_non_empty_string(
payload: dict[str, Any],
field: str,
issues: list[str],
*,
prefix: str | None = None,
) -> None:
value = payload.get(field)
label = f"{prefix}.{field}" if prefix else field
if not isinstance(value, str) or not value.strip():
issues.append(f"{label} must be a non-empty string")


def _is_number(value: Any) -> bool:
return isinstance(value, (int, float)) and not isinstance(value, bool)


def _is_int(value: Any) -> bool:
return isinstance(value, int) and not isinstance(value, bool)


def _is_datetime_string(value: str) -> bool:
candidate = value.strip()
if candidate.endswith("Z"):
candidate = f"{candidate[:-1]}+00:00"
try:
datetime.fromisoformat(candidate)
except ValueError:
return False
return True


def _validate_artifact_file(
*,
name: str,
artifact: dict[str, Any],
expected_sha256: str,
base_dir: Path,
issues: list[str],
) -> None:
raw_path = str(artifact.get("path") or "").strip()
label = f"artifacts.{name}"
if not raw_path:
return
path = Path(raw_path)
if path.is_absolute():
issues.append(f"{label}.path must be repo-relative, got absolute path")
return

resolved = (base_dir / path).resolve()
try:
resolved.relative_to(base_dir.resolve())
except ValueError:
issues.append(f"{label}.path must stay within the evidence package directory")
return

if not resolved.is_file():
issues.append(f"{label}.path does not exist: {raw_path}")
return

actual_sha256 = hashlib.sha256(resolved.read_bytes()).hexdigest()
if actual_sha256.lower() != expected_sha256.lower():
issues.append(
f"{label}.sha256 mismatch: expected {expected_sha256.lower()}, got {actual_sha256.lower()}"
)


if __name__ == "__main__":
raise SystemExit(main())
Loading