SC'25 UltraAttn: Efficiently Parallelizing Attention through Hierarchical Context-Tiling
-
Updated
Aug 14, 2025 - Python
#
context-parallelism
Here are 4 public repositories matching this topic...
SILKern: sparse-index localization kernels for context-parallel decode — deterministic, allocation-free, CUDA-graph-safe
cuda pytorch triton reproducibility gpu-kernels determinism inference-optimization kv-cache sparse-attention vllm llm-inference paged-attention context-parallelism cuda-graphs
-
Updated
Aug 29, 2026 - Python
Hardware barrier-free distributed computing plane (FNG V3) fusing Burgers' viscous dissipation and moment correction directly into XLA registers to pre-rectify tensor skewness & minimize latency spikes inside distributed LLM attention rails.
hpc fault-tolerance distributed-computing gpu-acceleration numerical-analysis co-design mathematical-physics burgers-equation jax xla tpu-acceleration transformer-attention context-parallelism llm-infrastructure
-
Updated
Aug 11, 2026 - Python
Distributed training framework for DeepSeek-V3 (Multi-Head Latent Attention, DeepSeekMoE, auxiliary-loss-free load balancing) with composable DP/FSDP/HSDP/TP/PP/CP/EP parallelism, torchtitan-inspired.
pytorch moe data-parallelism distributed-training pipeline-parallelism kv-cache tensor-parallelism fsdp deepseek context-parallelism multi-head-latent-attention expert-parallelism
-
Updated
Sep 1, 2026 - Python
Add this topic to your repo
To associate your repository with the context-parallelism topic, visit your repo's landing page and select "manage topics."