[Fix] Invalidate KV cache before weight updates

[wildoranges]

Motivation

Fixes #4917.

Hot model-weight updates could leave prefix-cache KV blocks computed with the previous weights. Subsequent requests then mixed stale prefix KV with the new model weights, producing incorrect output without an error.

Modification

• Require the PyTorch engine to offload KV cache and have no active sessions or pending KV work before a weight update.
• Serialize synchronous and distributed weight updates with sleep and wakeup operations.
• Clear scheduler-owned prefix-cache entries when the engine enters sleep.
• Add the missing multi-process update_params path and guard serialized tensor payload handling.
• Keep weight-update errors in a lightweight module so management endpoint imports do not load the PyTorch/disaggregated engine.
• Document the required sleep/wakeup sequence in English and Chinese update-weight guides.

BC-breaking

Weight updates that are attempted while KV cache or active sessions remain available are now rejected with a conflict/error response. Clients must call /sleep first and then /wakeup?tags=weights before updating weights, as documented.

Use cases

This protects RLHF and other workflows that hot-update model weights while prefix caching is enabled.

Tests

• 128Core: 19 passed for tests/pytorch/engine/test_engine_sleep.py, tests/pytorch/paging/test_block_trie/test_kv_lifecycle.py, and tests/test_lmdeploy/serve/openai/test_management.py.
• Chinese and English Sphinx HTML builds: passed.
• Repository pre-commit hooks: passed.

Checklist

• Pre-commit and lint checks passed.
• Unit tests cover the modified behavior.
• Documentation updated for the new update-weight lifecycle.
• No downstream dependency changes.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[lvhan028]

In our RL workflow, the controller explicitly invokes the /sleep endpoint prior to /update_weights. This ensures that all weights are offloaded from GPU to CPU and the KV cache is fully destroyed. Consequently, when /update_weights is called, the KV cache is guaranteed to be in an uninitialized state, which aligns perfectly with the intended behavior.

We truly appreciate your contribution to the project. However, we prefer to avoid adding defensive logic here, as it would introduce unnecessary complexity to a workflow that is already well-orchestrated and functioning as expected.