💡 The paper introduces Explorative Modeling, a new paradigm that unlocks a third pretraining axis for existing generative models, beyond parameters and data. The key problem addressed is that generative models are not trained end-to-end, due to the fact that they handle distributions with many modes, and existing scalable approaches factor the generation procedure, preventing end-to-end generation.

The proposed method, Explorative Modeling, factors the training loop instead, exploring K candidate matches between model generations and data, and training on the best predictions, which commit to modes rather than blurring them. This approach is found to be useful in two settings.

First, increasing exploration adds a third pretraining axis for existing generative models, where scaling exploration monotonically improves performance across both continuous and discrete domains, such as images, video, and language. Notably, gains from exploration increase with scale, climbing from 7 percent to 36 percent as data scales and from 13 percent to 23 percent as models grow, with efficiency gains more than doubling at three times the compute.

Second, Explorative Models enable end-to-end reconstructive generative modeling, matching diffusion on control tasks with 16-256 times fewer inference steps. The results establish Explorative Models as both a new pretraining axis for existing generative models and a standalone end-to-end generative modeling paradigm.

Overall, the paper's contributions include improving FLOP efficiency by 4.1 times, sample efficiency by 6.2 times, and parameter efficiency by 47 percent, and lifting the strongest image-generation recipes to near state-of-the-art 1.43 FID on ImageNet without guidance, enabling scaling and generalization.