💡 The paper introduces a new benchmark suite called RNG-Bench to evaluate the ability of multimodal large language models to reconstruct past observations and use them for decision-making in multi-step interactions. The problem addressed is that existing benchmarks do not adequately test a model's ability to recall and act on past observations, which is a crucial skill for deploying these models in real-world applications. The RNG-Bench suite consists of two games, Matching Pairs and 3D Maze, which are designed to test a model's ability to reconstruct past observations and use them to make decisions. The games have controlled difficulty parameters, including grid size, visual pattern, and observation modality, which allow for a thorough evaluation of a model's skills. The benchmark also introduces a head-to-head duel protocol to control for instance-level variance and a Memory Gap metric to distinguish between forgetting and poor decision-making. The results show that most residual errors in the models' performance are due to forgetting earlier observations rather than suboptimal decision-making. The paper also demonstrates that fine-tuning a model on optimal-policy rollouts and filtered model demonstrations can improve its performance on RNG-Bench and transfer to existing benchmarks without degrading its general multimodal capability. Overall, the paper provides a new benchmark suite and evaluation methodology for multimodal large language models, and demonstrates the importance of testing these models' ability to recall and act on past observations.