💡 The paper addresses the issue of document set selection and ranking, which is crucial for large language models and AI agents that rely on search results. Existing evaluation systems score documents independently and aggregate them using metrics like DCG, ignoring interactions between documents such as redundancy, conflict, and complementarity. This limitation makes it difficult to determine what makes one document set better than another.

To address this issue, the authors propose a comprehensive evaluate-diagnose-optimize framework. They design Setwise Eval Kit, a three-level, nine-dimension document set evaluation benchmark that covers both short-form and long-form scenarios, comprising approximately 28,000 high-quality evaluation rubrics.

The authors systematically evaluate 12 rerankers and find that even the best method achieves no more than 45 percent coverage, and cross-document coordination dimensions are universally weak. No single method maintains top performance across both settings.

Building on this, the authors propose Rubric4Setwise, a training-free method that converts rubric-based evaluation criteria into document set selection signals. This method achieves the best downstream generation performance with fewer documents and search rounds. It is the only method that maintains state-of-the-art results across both scenarios, validating the effectiveness of closing the loop from evaluation to optimization.

The paper's contributions include a comprehensive evaluation framework, a new benchmark for document set evaluation, and a novel method for document set selection and ranking that outperforms existing methods. The results demonstrate the importance of considering cross-document interactions and using rubric-based evaluation criteria to improve document set selection and ranking.