expert-parallelism

Profile-guided layout optimization for llama.cpp GGUF MoE models — BOLT/PGO for LLM weights. Trace expert routing, rewrite the file so co-activated experts sit together, prefetch as merged reads.

Low-cost AMD/ROCm stationary-expert fabric for configurable trillion-parameter sparse MoE inference, with packed-INT4 HIP kernels, activation-only RoCE transport, hardware/BOM design, release gates, and reproducible roofline tooling.

Hand-built MoE expert parallelism: routing-skew capture, fused grouped-GEMM Triton kernels, 2-all2all dispatch/combine, overlap ablations — proving by counter-example why DeepEP exists.

Completed negative-result study of request-level route-aware scheduling for native diffusion MoE inference

Distributed training framework for DeepSeek-V3 (Multi-Head Latent Attention, DeepSeekMoE, auxiliary-loss-free load balancing) with composable DP/FSDP/HSDP/TP/PP/CP/EP parallelism, torchtitan-inspired.

AI-assisted build compatibility fix for hybrid EP (upstream PR #699); not accepted upstream.

Simulates Mixture-of-Experts routing strategies (TopK, NoisyTopK, ExpertChoice, Hash) across 432 configurations. Key findings: ExpertChoice achieves best practical balance (imbalance=1.40 vs TopK=5.52); NoisyTopK reduces drops by 40% on skewed distributions; capacity factor is the strongest control knob.

Correctness-first PyTorch/CUDA lab for FlashAttention, DeepSeek MLA/MoE, expert parallelism, reproducible benchmarks, and HPC-for-AI notes.