Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions .github/workflows/phase1-receipt.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
name: Seal Phase 1 measurement receipt

# Each reviewed approval produces an independent immutable record; later requests must not cancel pending issuances.
on: # zizmor: ignore[concurrency-limits]
workflow_dispatch:
inputs:
kind:
description: Seal source measurement or its later publication reference
required: true
default: measurement
type: choice
options: [measurement, publication]
approval-path:
description: Reviewed default-branch qualification/phase1/*.json expectation
required: true
type: string

permissions:
contents: read
actions: read # Read source workflow attempts and download the exact artifacts being sealed.

jobs:
seal:
name: Seal reviewed measurement or publication
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.workflow_sha }}
persist-credentials: false
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
- name: Resolve reviewed expectation
env:
APPROVAL_PATH: ${{ inputs.approval-path }}
shell: python
run: |
import os
from pathlib import Path
path = Path(os.environ['APPROVAL_PATH']).resolve(strict=True)
root = Path('qualification/phase1').resolve()
if not path.is_relative_to(root) or path.suffix != '.json' or any(c in str(path) for c in '\r\n'):
raise ValueError('approval must be a reviewed qualification/phase1 JSON file')
with Path(os.environ['GITHUB_ENV']).open('a') as stream:
stream.write(f'REVIEWED_APPROVAL={path}\n')
- name: Seal independently approved complete source sweep
if: inputs.kind == 'measurement'
env:
GH_TOKEN: ${{ github.token }}
TRUSTED_WORKFLOW_SHA: ${{ github.workflow_sha }}
shell: python
run: |
import os, subprocess
subprocess.run(['uv', 'run', '--locked', 'python', '-m', 'infx.workflows.phase1_publication',
'--approval', os.environ['REVIEWED_APPROVAL'], '--archives', 'verified-archives',
'--output', 'receipt.json'], check=True)
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: inputs.kind == 'measurement'
with:
name: measurement-receipt
path: receipt.json
if-no-files-found: error
- name: Bind accepted source to reviewed publication
if: inputs.kind == 'publication'
env:
GH_TOKEN: ${{ github.token }}
TRUSTED_ISSUER_SHAS: ${{ vars.INFX_RECEIPT_ISSUER_SHAS }}
DEPLOYED_READER_SHA: ${{ vars.INFX_PHASE1_READER_REVISION }}
shell: python
run: |
import os, subprocess
subprocess.run(['uv', 'run', '--locked', 'python', '-m', 'infx.workflows.phase1_record',
'--approval', os.environ['REVIEWED_APPROVAL'], '--output', 'publication.json'], check=True)
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: inputs.kind == 'publication'
with:
name: publication-record
path: publication.json
if-no-files-found: error
22 changes: 22 additions & 0 deletions .github/workflows/recover-reused-ingest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,34 @@ concurrency:
jobs:
trigger-agentic-ingest:
name: trigger-agentic-ingest
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions:
contents: read
actions: read # Verify issuer workflow attempts and download accepted receipt artifacts.
steps:
- name: Checkout trusted receipt resolver
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.workflow_sha }}
persist-credentials: false
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
- name: Resolve accepted immutable publication
id: receipt
env:
GH_TOKEN: ${{ github.token }}
SOURCE_RUN_ID: ${{ inputs.source-run-id }}
MERGE_RUN_ID: ${{ inputs.merge-run-id }}
INFX_RECEIPT_ISSUER_SHAS: ${{ vars.INFX_RECEIPT_ISSUER_SHAS }}
INFX_RECEIPT_ISSUER_WORKFLOW: ${{ vars.INFX_RECEIPT_ISSUER_WORKFLOW }}
run: uv run --locked python -m infx.workflows.receipt_transport --publication-required
- name: Trigger agentic database ingest
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
SOURCE_RUN_ID: ${{ inputs.source-run-id }}
MERGE_RUN_ID: ${{ inputs.merge-run-id }}
RECEIPT_TRANSPORT: ${{ steps.receipt.outputs.payload }}
with:
# Cross-repository dispatch credential for maintainer-triggered ingest recovery.
github-token: ${{ secrets.FRONTEND_PAT }} # zizmor: ignore[secrets-outside-env]
Expand All @@ -37,6 +58,7 @@ jobs:
repo: "InferenceX-app",
event_type: "ingest-agentic-results",
client_payload: {
...JSON.parse(process.env.RECEIPT_TRANSPORT),
"source-run-id": process.env.SOURCE_RUN_ID,
"merge-run-id": process.env.MERGE_RUN_ID,
"database-target": "production"
Expand Down
44 changes: 44 additions & 0 deletions .github/workflows/run-sweep.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1000,12 +1000,33 @@ jobs:
needs.setup.outputs.reuse-enabled == 'true'
)
runs-on: ubuntu-latest
permissions:
contents: read
actions: read # Verify issuer workflow attempts and download accepted receipt artifacts.
steps:
- name: Checkout trusted receipt resolver
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ needs.setup.outputs.tooling-ref }}
persist-credentials: false
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
- name: Resolve accepted immutable publication
id: receipt
env:
GH_TOKEN: ${{ github.token }}
SOURCE_RUN_ID: ${{ needs.setup.outputs.reuse-enabled == 'true' && needs.setup.outputs.reuse-source-run-id || github.run_id }}
MERGE_RUN_ID: ${{ github.run_id }}
INFX_RECEIPT_ISSUER_SHAS: ${{ vars.INFX_RECEIPT_ISSUER_SHAS }}
INFX_RECEIPT_ISSUER_WORKFLOW: ${{ vars.INFX_RECEIPT_ISSUER_WORKFLOW }}
run: uv run --locked python -m infx.workflows.receipt_transport --publication-required --defer-unsealed
- name: Trigger database ingest
if: steps.receipt.outputs.ready == 'true'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
SOURCE_RUN_ID: ${{ needs.setup.outputs.reuse-enabled == 'true' && needs.setup.outputs.reuse-source-run-id || github.run_id }}
MERGE_RUN_ID: ${{ github.run_id }}
RECEIPT_TRANSPORT: ${{ steps.receipt.outputs.payload }}
with:
# Repository integration credential for the scoped sweep/ingest job.
github-token: ${{ secrets.FRONTEND_PAT }} # zizmor: ignore[secrets-outside-env]
Expand All @@ -1015,6 +1036,7 @@ jobs:
repo: "InferenceX-app",
event_type: "ingest-results",
client_payload: {
...JSON.parse(process.env.RECEIPT_TRANSPORT),
"source-run-id": process.env.SOURCE_RUN_ID,
"merge-run-id": process.env.MERGE_RUN_ID
}
Expand Down Expand Up @@ -1065,12 +1087,33 @@ jobs:
)
)
runs-on: ubuntu-latest
permissions:
contents: read
actions: read # Verify issuer workflow attempts and download accepted receipt artifacts.
steps:
- name: Checkout trusted receipt resolver
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ needs.setup.outputs.tooling-ref }}
persist-credentials: false
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
- name: Resolve accepted immutable publication
id: receipt
env:
GH_TOKEN: ${{ github.token }}
SOURCE_RUN_ID: ${{ needs.setup.outputs.reuse-enabled == 'true' && needs.setup.outputs.reuse-source-run-id || github.run_id }}
MERGE_RUN_ID: ${{ github.run_id }}
INFX_RECEIPT_ISSUER_SHAS: ${{ vars.INFX_RECEIPT_ISSUER_SHAS }}
INFX_RECEIPT_ISSUER_WORKFLOW: ${{ vars.INFX_RECEIPT_ISSUER_WORKFLOW }}
run: uv run --locked python -m infx.workflows.receipt_transport --publication-required --defer-unsealed
- name: Trigger agentic database ingest
if: steps.receipt.outputs.ready == 'true'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
SOURCE_RUN_ID: ${{ needs.setup.outputs.reuse-enabled == 'true' && needs.setup.outputs.reuse-source-run-id || github.run_id }}
MERGE_RUN_ID: ${{ github.run_id }}
RECEIPT_TRANSPORT: ${{ steps.receipt.outputs.payload }}
with:
# Repository integration credential for the scoped sweep/ingest job.
github-token: ${{ secrets.FRONTEND_PAT }} # zizmor: ignore[secrets-outside-env]
Expand All @@ -1080,6 +1123,7 @@ jobs:
repo: "InferenceX-app",
event_type: "ingest-agentic-results",
client_payload: {
...JSON.parse(process.env.RECEIPT_TRANSPORT),
"source-run-id": process.env.SOURCE_RUN_ID,
"merge-run-id": process.env.MERGE_RUN_ID,
"database-target": "production"
Expand Down
12 changes: 12 additions & 0 deletions .github/workflows/stage-results.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,16 @@ jobs:
GH_TOKEN: ${{ github.token }}
run: uv run --locked python -m infx.workflows.stage_results

- name: Resolve accepted immutable source receipt
id: receipt
env:
GH_TOKEN: ${{ github.token }}
SOURCE_RUN_ID: ${{ steps.request.outputs.run-id }}
MERGE_RUN_ID: ${{ steps.request.outputs.run-id }}
INFX_RECEIPT_ISSUER_SHAS: ${{ vars.INFX_RECEIPT_ISSUER_SHAS }}
INFX_RECEIPT_ISSUER_WORKFLOW: ${{ vars.INFX_RECEIPT_ISSUER_WORKFLOW }}
run: uv run --locked python -m infx.workflows.receipt_transport

- name: Acknowledge staging request
id: acknowledge
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
Expand Down Expand Up @@ -67,6 +77,7 @@ jobs:
RUN_DATE: ${{ steps.request.outputs.run-date }}
REQUESTED_BY: ${{ steps.request.outputs.requested-by }}
COMMENT_ID: ${{ steps.acknowledge.outputs.comment-id }}
RECEIPT_TRANSPORT: ${{ steps.receipt.outputs.payload }}
with:
# Cross-repository dispatch credential; trusted control code validates maintainer authorization.
github-token: ${{ secrets.FRONTEND_PAT }} # zizmor: ignore[secrets-outside-env]
Expand All @@ -76,6 +87,7 @@ jobs:
repo: 'InferenceX-app',
event_type: 'stage-results',
client_payload: {
...JSON.parse(process.env.RECEIPT_TRANSPORT),
'source-repository': `${context.repo.owner}/${context.repo.repo}`,
'pr-number': String(context.issue.number),
'run-id': process.env.RUN_ID,
Expand Down
2 changes: 1 addition & 1 deletion docs/eval-agentx-procedures.md
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ Treat fast results as bring-up evidence, never as a replacement for the canonica

## 8. Preserve trace and run provenance

AgentX defaults to recorded assistant-response replay. Live server outputs are measured but discarded when constructing later turns. Set `AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES=1` only for an explicitly different live-assistant experiment. The selected trace corpus is model-family dependent unless `WEKA_LOADER_OVERRIDE` pins it. The resolver logs both loader and Hugging Face dataset ([trace resolution](../benchmarks/benchmark_lib.sh#L2023-L2102), [replay semantics](../benchmarks/benchmark_lib.sh#L2104-L2270)).
At the pilot client revision `754356e9a39acc6cc6afb242d123bb57c3fb6f75`, AgentX always constructs later turns from recorded assistant-response deltas. Live server outputs are measured; `AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES` does not change this loader's behavior. A different replay methodology requires its own qualification. The legacy resolver chooses a model-family-dependent corpus unless `WEKA_LOADER_OVERRIDE` pins it. The prepared H100 client explicitly uses `semianalysis_cc_traces_weka_062126`, 393 entries, and no replay context filter ([Python client](../infx/benchmarks/agentx.py)).

Capture orchestration provenance immediately:

Expand Down
2 changes: 1 addition & 1 deletion docs/eval-agentx-procedures_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ Fast 结果只能作为 bring-up 证据,绝不能替代 canonical candidate。

## 8. 保留 trace 与运行 provenance

AgentX 默认 replay 已记录的 assistant response。实时服务输出会被测量,但构造后续 turn 时会丢弃。只有在明确要进行不同的 live-assistant 实验时,才设置 `AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES=1`。除非用 `WEKA_LOADER_OVERRIDE` 固定,否则所选 trace corpus 依赖模型 family;resolver 会同时记录 loader 与 Hugging Face dataset([trace 解析](../benchmarks/benchmark_lib.sh#L2023-L2102)、[replay 语义](../benchmarks/benchmark_lib.sh#L2104-L2270))。
在试点固定的客户端版本 `754356e9a39acc6cc6afb242d123bb57c3fb6f75` 中,AgentX 始终使用已记录的 assistant response 增量构造后续轮次。实时服务输出会被测量;`AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES` 不会改变此 loader 的行为。不同的回放方法需要单独验证。旧版 resolver 会根据模型系列选择语料,除非用 `WEKA_LOADER_OVERRIDE` 固定。预先准备的 H100 客户端明确使用 `semianalysis_cc_traces_weka_062126`、393 条记录,并且不按上下文长度过滤回放数据([Python 客户端](../infx/benchmarks/agentx.py))。

立即记录 orchestration provenance:

Expand Down
Loading
Loading