E2063 - Loop the Loopies!
Published: July 21, 2026
Duration: 19:35
🤗 Upvotes: 56 | cs.CL, cs.AI
Authors:
Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, Bryan Dai
Title:
Loop the Loopies!
Arxiv:
http://arxiv.org/abs/2607.16051v2
Abstract:
We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N tim...