TL;DR Scaling pretraining for a ~1-trillion-parameter Mixture-of-Experts (MoE) model on shared, public infrastructure was previously considered unfeasible due to unique architectural and interconnect bottlenecks. However, by leveraging the fine-grained flexibility… The post Training a 1T-Parameter MoE on EU public infrastructure with PyTorch and TorchTitan appeared first on PyTorch .
Training a 1T-Parameter MoE on EU public infrastructure with PyTorch and TorchTitan
calendar_today
November 12, 2026
person
The EUROPA consortium (Domyn, Fraunhofer IAIS and IIS)
domain
pytorch