Skip to content

Support multi-node multi-Draft training #21

Open

Description

@zhouaihui

Extend shared Target training to static multi-node clusters so independent Draft experiments can
reuse compatible features while keeping their own training state and progress.

Suggested scope

Preserve each Draft's weights, optimizer, schedule and checkpoints without a shared optimizer-step
barrier. Allow packing to be configured independently, including mixed packed and unpacked Drafts.

Support independent local filesystems and keep feature consumption on the producer's node.
Elastic membership and broader node-failure recovery remain separate work. The existing shared
Target and static multi-node paths provide the starting point.

Getting started

Implementation is in progress in PR #9, which adds
multi-node sharing, per-Draft packing and a training/recovery recipe. Use this issue to track the
feature and its remaining review and validation. Additional combinations can follow separately.

Expected outcome

  • A documented, runnable multi-node multi-Draft recipe works without a shared filesystem.
  • Drafts progress independently and can resume from different checkpoints, with faithful
    export and strict reloading of trained assets.
  • Compatible feature sharing and committed-work identity are preserved across packing and recovery.
  • Slow Drafts apply bounded backpressure; failed readers are safely fenced so healthy siblings
    can continue where the declared failure scope allows. Tested topology and limitations are recorded.

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions