Skip to content

[wip ai slop] Add prepared H100 single-job execution / 新增 H100 预备式单任务执行 - #21

Draft
functionstackx wants to merge 3 commits into
codex/inferencex-phase1-base-984180e5from
codex/inferencex-phase1
Draft

functionstackx wants to merge 3 commits into
codex/inferencex-phase1-base-984180e5from
codex/inferencex-phase1

Conversation

@functionstackx

@functionstackx functionstackx commented Sep 19, 2026 •

Copy link
Copy Markdown

Prepared H100 jobs freeze one recipe/profile/runtime snapshot before allocation, then use a durable intent journal for native submission, reconciliation, observation and ownership-scoped cancellation. Structured custom clients use explicitly prepared argv, cwd, environments and mounted runtimes. Required-server failure fails the job; shutdown has bounded waits.

The direct single-node vLLM path also requires the listening socket to belong to the recorded worker process tree. The guard checks PID start times and PID/network namespaces before client traffic, during execution and before accepting exit 0. Port collisions, foreign/replaced listeners and unreadable ownership evidence fail closed. These are observed ownership checks, not a guarantee against every transition between polls.

Candidate: 62beb5ec4f8c33abc26851ba0adaded29957ca5b, based on the consumer’s NVIDIA dependency 984180e5b8755aef85e9995048b5a16cb5336bce. The target is a frozen copy of that baseline, separate from the fork’s newer main and AMD stack. Dependency-owned Bash templates and setup scripts remain supported. This draft remains unmerged under the user’s instruction.

On the actual H100 cluster, Slurm 25.05.7 has Step Manager enabled. Startup qualification ran worker 18324.5, but squeue --steps exposed only batch/extern. The updated native code uses a job-specific scontrol --oneliner show steps query, validates exact ownership and unambiguous running records, and uses the registry's job ID during cleanup. It preserves step-directed SIGTERM. Cluster evidence confirms the configuration; live compatibility of this new pin still requires a rerun.

Previous native 50c3dacc37def01606ee9e4e0ed873646d4f7cc5 passed InferenceX Linux CI 35481065144: 2,471 passed, two skipped, six GPU integration cases deselected, plus the installed nine-point preparation contract. Provisioning 35481075834 passed all 38 stages. The old sweep on that generation was cancelled during initial asset hashing before Slurm submission; it cannot qualify this new pin. Consumer PR #3299 now pins this candidate, with corrected generation 35484558552 passing all 38 provisioning stages. All 168 installed native source files match this pin.

Current-pin local validation: 2,497 passed, five failed, seven skipped, six integration cases deselected. All five failures reproduce on the unchanged macOS baseline (two Linux-affinity assumptions and three sourced-Bash path cases). Source Ruff, formatting and schema checks passed; advisory type-check output is unchanged from baseline. Focused step discovery/cleanup tests pass. Updated-pin Linux CI 35484873112 passed: 2,507 native tests, two skipped and six actual-GPU integration cases deselected. The separately installed 2.2.1.post3+g62beb5ec4 passed all nine prepare/render boundary cases. Source, lock and NVIDIA lineage checks passed. Actual startup cancellation 35486680437 passed: live step 18326.5 and its worker identity were observed, explicit cancellation closed only owned allocation 18326, and native completion confirms cleanup and restoration. Client cancellation 35486685532 also passed after corrected c28 server readiness: active client 18327.8, explicit owned cancellation, SIGTERM writer closure, terminal allocation CANCELLED and native cleanup/restoration complete. No prior teardown was found. Post-cancellation force-kill warnings remain; graceful model shutdown is not claimed. Corrected full nine-point sweep 35487222760 is now running; benchmark acceptance remains pending. The first startup probe reached owned terminal CANCELLED but missed its trigger; the first client probe ended FAILED during vLLM initialization before its writer started. Neither is qualified. The consumer separately fixes probe serving settings to match real c28 eval.

The requested [wip ai slop] title prefix remains; GitHub's conventional-title check rejects that prefix. This PR remains a draft and unmerged.

AI model disclosure

Codex is described by this runtime as based on GPT-6; the exact model/version identifier was not exposed and could not be verified. The primary agent integrated and reviewed the work. Delegated agents using the inherited runtime implemented/reviewed native execution and listener ownership, cancellation, prepared clients, provisioning and publication boundaries.

中文

H100 prepared job 在申请资源前冻结 recipe、profile 和 runtime 快照,随后通过持久化 intent journal 完成原生提交、恢复、观察和按归属取消。自定义客户端显式使用准备好的 argv、cwd、环境及运行时挂载。必要服务失败会使任务失败,关闭等待有明确时限。

单节点直连 vLLM 路径还要求监听 socket 属于已记录的 worker 进程树。Guard 在发送客户端流量前、执行期间及接受退出码 0 前校验 PID 启动时间和 PID/network namespace。端口冲突、外部或替换监听器以及无法读取归属证据都会失败。这些是实际观察到的归属检查,不代表能保证捕获轮询间的每次状态变化。

候选提交为 62beb5ec4f8c33abc26851ba0adaded29957ca5b,以消费方当前 NVIDIA 依赖 984180e5b8755aef85e9995048b5a16cb5336bce 为基线。目标分支是该基线的固定副本,与 fork 较新的 main 和 AMD stack 分开。依赖拥有的 Bash 模板和设置脚本仍受支持。按用户要求,本草稿保持未合并。

实际 H100 集群使用 Slurm 25.05.7 并启用 Step Manager。启动期验收中 worker 18324.5 已运行,但 squeue --steps 只返回 batch/extern。更新后的原生代码按指定任务查询 scontrol --oneliner show steps,验证准确归属和无歧义的运行记录,清理时使用 registry 自身的 job ID,保留逐 step SIGTERM 行为。集群证据 已确认配置;新 pin 已在启动期取消检查 35486680437 中验证实际活跃步骤发现与清理。

此前的原生 50c3dacc37def01606ee9e4e0ed873646d4f7cc5 已通过 InferenceX Linux CI 35481065144:2,471 项通过、两项跳过、六项 GPU 集成测试未选入,并通过已安装运行时的九点 preparation 契约。环境准备 35481075834 的 38 个阶段全部通过。使用该 generation 的旧 sweep 已在初始资源哈希阶段、Slurm 提交之前取消,不能验收新的 pin。消费方 PR #3299 现已固定本候选版本,独立的 generation 35484558552 已通过全部 38 个准备阶段,安装的 168 个原生源文件均与本 pin 一致。

当前 pin 的本地完整验证为:2,497 项通过、五项失败、七项跳过、六项集成测试未选入。五项失败均在原始 macOS 基线上复现(两项 Linux affinity 假设及三项 sourced-Bash 路径行为)。源代码 Ruff、格式及 schema 检查通过,建议性类型检查输出与基线相同。针对性的 step 发现/清理测试通过。更新 pin 的 Linux CI 35484873112 已通过:2,507 项原生测试通过、两项跳过、六项真实 GPU 集成测试未选入。独立安装的 2.2.1.post3+g62beb5ec4 通过全部九点 prepare/render 边界检查,源码、lock 和 NVIDIA 版本沿袭检查通过。实际 启动期取消 35486680437 已通过:观察到运行 step 18326.5 及 worker 身份,明确取消并关闭了唯一所属 allocation 18326,原生记录确认清理与恢复完成。客户端取消 35486685532 也已通过:修正后的 c28 服务就绪、客户端 18327.8 运行,明确取消所属资源后 writer 以 SIGTERM 关闭,allocation 达到终态 CANCELLED,原生清理与恢复完成;未发现提前 teardown。取消后的强制终止警告仍保留,不宣称模型优雅退出。修正后的 完整九点 sweep 35487222760 正在运行,基准验收仍待完成。首次启动探针虽确认所属资源以 CANCELLED 终止,但未满足触发条件;首次客户端探针在 writer 启动前因 vLLM 初始化失败而以 FAILED 结束。两者均未通过验收。消费方另行修正探针服务参数,使其与真实 c28 eval 一致。

保留用户要求的 [wip ai slop] 标题前缀;GitHub 的 conventional-title 检查不接受该前缀。本 PR 保持草稿且未合并。

AI 模型披露:运行时将 Codex 描述为基于 GPT-6,但未暴露可核实的精确模型/版本。主代理负责集成和审阅;继承相同运行时的子代理实现或审阅原生执行与监听归属、取消、客户端准备、环境准备和发布边界。

中文:为 H100 试点新增预备式单任务执行,冻结运行输入与依赖身份,持久化 Slurm 提交归属及恢复记录,并提供参数直传、保守终态观察、有界清理和对应行为测试。保留上游 Bash 依赖,硬件验收另行完成。

Signed-off-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
Bind prepared single-node direct-vLLM readiness and client execution to the recorded worker PID, process start time, namespaces, and owned listening sockets. Reject existing or replacement listeners and descendant PID reuse, and use existing client cleanup on ownership loss.

将预备的单节点 direct-vLLM 就绪检查和客户端执行绑定到实际 worker 进程及其监听套接字;拒绝外部监听器、替换服务和子进程 PID 复用,检测到所有权丢失时停止并回收客户端。

Validation: 77 focused tests passed locally before the additional Linux quick-exit case; final suite 2461 passed, 7 skipped, 6 deselected, with the same 5 macOS platform failures (Bash 3 and Linux CPU affinity). Five real Linux ownership tests await Linux CI; actual Pyxis /proc visibility still requires cluster qualification. No dependency lock changes.
Signed-off-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
@functionstackx functionstackx changed the title feat: add prepared H100 single-job execution / 新增 H100 预备式单任务执行 [wip ai slop] Add prepared H100 single-job execution / 新增 H100 预备式单任务执行 Sep 20, 2026
按明确的作业 ID 查询 scontrol,拒绝外部、缺失或歧义步骤记录;清理使用 registry 自身的作业归属,保留逐 step SIGTERM 行为。

Signed-off-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
@adibarra adibarra mentioned this pull request Sep 21, 2026
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant