💡 The paper introduces M3Eval, a comprehensive evaluation framework and benchmark for assessing the memory capabilities of multi-modal models in video understanding systems. The problem addressed is that current multi-modal models have significant limitations in their memory capabilities, particularly in maintaining disentangled representations and demonstrating human-like interference patterns. To address this gap, the authors designed M3Eval, which is grounded in cognitive psychology and features carefully constructed tasks that isolate key aspects of memory.

The method involves conducting extensive experiments across representative multi-modal models using the M3Eval framework, which evaluates different memory dimensions such as what models retain, how faithfully information is preserved, and how robust memory remains under interference. The framework includes tasks that test the models' ability to maintain disentangled representations, exhibit human-like interference patterns, and demonstrate symbolic memory.

The results of the experiments reveal consistent weaknesses and distinctive behaviors in the models. The models struggle to maintain disentangled representations when processing parallel video streams, exhibit interference patterns that differ substantially from those observed in human memory, and ground memory sources more reliably in the spatial domain than the temporal domain. Additionally, the models demonstrate limited symbolic memory.

The paper's contributions include providing a valuable resource for future research in the form of the M3Eval benchmark and highlighting memory as a fundamental yet underexplored capability in multi-modal models. The findings offer insights for designing more effective memory mechanisms in multi-modal models, which can advance the field of video understanding systems. The code and dataset are made available to facilitate future research.