Skip to content

[ascend] update attn op_backend - #4900

Merged
lvhan028 merged 1 commit into
InternLM:mainfrom
wanfengcxz:wq/update_ascend_attn
Aug 25, 2026
Merged

[ascend] update attn op_backend#4900
lvhan028 merged 1 commit into
InternLM:mainfrom
wanfengcxz:wq/update_ascend_attn

Conversation

@wanfengcxz

Copy link
Copy Markdown
Collaborator

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

DLINFER has updated the Ascend attention implementation from the legacy ATB
path to FIA/FIA v2 and unified graph replay with torch.npu.NPUGraph.update().

This PR adds the corresponding LMDeploy-side adaptations.
Dlinfer pr: DeepLink-org/dlinfer#350

Changes

  • Cache and reuse a fixed FIA causal mask per NPU device to avoid allocating
    the attention mask on every execution.
  • Use the fixed split-fuse causal mask for Dense, GQA, MLA and paged-prefill
    attention with sparse_mode=3.
  • Detect MLA models through k_head_dim != v_head_dim and explicitly pass
    is_mla to the DLINFER Ascend graph runner.
  • Validate the torch/torch-npu runtime when initializing the Ascend backend.

The companion DLINFER update includes:

  • Migrating prefill and paged attention to FIA/FIA v2.
  • Adding MLA paged-prefill support.
  • Removing unnecessary .tolist() calls from attention hot paths.
  • Removing the legacy ATB graph-task attention path.
  • Unifying graph replay through NPUGraph.update().

Dependency

This PR depends on the corresponding DLINFER attention update. The DLINFER PR
should be merged first, or both PRs should reference each other.

Required runtime:

  • torch >= 2.8.0
  • torch-npu >= 2.8.0.post1

Test

  • DeepSeek-V2-Lite MLA graph capture/replay
  • Qwen3.5 Dense graph capture/replay
  • Dense and MLA prefill/decode attention tests
  • MLA paged-prefill path test

@lvhan028
lvhan028 requested review from grimoire and a lite review from Copilot August 25, 2026 03:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adapts LMDeploy’s DLInfer Ascend backend to match DLInfer’s updated Ascend attention implementation (moving to FIA/FIA v2 and unified graph replay), including runtime validation and MLA detection.

Changes:

  • Cache and reuse a per-device FIA causal mask instead of allocating masks repeatedly.
  • Pass an explicit is_mla flag to the DLInfer Ascend graph runner based on k_head_dim != v_head_dim.
  • Validate the Ascend torch/torch-npu runtime during backend initialization via ensure_ascend_runtime().

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread lmdeploy/pytorch/backends/dlinfer/ascend/op_backend.py
Comment on lines +193 to +199
def get_fia_causal_mask():
device = step_context.block_offsets.device
mask = cls.fia_causal_masks.get(device)
if mask is None:
mask = torch.triu(torch.ones(2048, 2048, dtype=torch.int8, device=device), diagonal=1)
cls.fia_causal_masks[device] = mask
return mask

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kernel不是按绝对位置直接访问,而是计算当前 Q/KV tile 相对于 causal 对角线的位置。

@lvhan028
lvhan028 merged commit 7c857b2 into InternLM:main Aug 25, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants