A 777M-parameter Mixture-of-Experts language model trained from scratch for $55. Multi-head Latent Attention, DeepSeekMoE, own tokenizer.
nlp machine-learning ai deep-learning tokenizer transformers pytorch moe language-model from-scratch fine-tuning multi-head-attention mixture-of-experts large-language-models llm deepseek moonfrost
-
Updated
Oct 2, 2026 - Python