💡 The paper explores the differences between supervised fine-tuning and reinforcement learning in enhancing multi-task reasoning for large language models. The authors found that supervised fine-tuning suffers from severe task conflicts under multi-stage training, whereas reinforcement learning enables stable coexistence across diverse tasks. They attribute this to the parameter level, where reinforcement learning induces sparse and approximately orthogonal updates across tasks.

The authors provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. They found that interference in supervised fine-tuning is norm-limited, scaling with the absolute gradient magnitude, whereas interference in reinforcement learning is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks.

Leveraging this insight, the authors propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility. The results reveal a distinction between supervised fine-tuning and reinforcement learning, highlighting the benefits of reinforcement learning in multi-task learning for large language models. The proposed Parallel-RL approach has the potential to improve the efficiency and flexibility of multi-task training, making it a promising direction for future research.