💡 The paper addresses the problem of developing, evaluating, and deploying large language models LLMs in a cost-effective manner. Since no single LLM is optimal across all queries and budget constraints, model routing is essential for efficient deployment. However, existing routers adopt diverse formulations and implementations, making fair comparison and extension difficult.

To address this issue, the authors propose a unified formulation of LLM routing as a sequential decision process characterized by five components: context encoders, model encoders, scoring functions, decision rules, and learning signals. This formulation covers single-turn, multi-turn, and personalized routing.

Based on this formulation, the authors develop an automated pipeline for constructing routing supervision and evaluating routers jointly on response quality and inference cost. The resulting benchmark, xRouteBench, spans generic LLM, memory-augmented, vision, time-series, and personalized routing tasks.

The authors also introduce LLMRouter, an open-source modular infrastructure with more than 16 representative routers. Their empirical study shows that learned routers outperform the strongest fixed-model baseline by 14.6% relatively, lightweight routers become more competitive under tight cost constraints, and user-conditioned routing consistently improves personalization.

Overall, the paper provides a unified framework for LLM routing, a comprehensive benchmark, and a modular infrastructure, which can facilitate the development, evaluation, and deployment of LLMs in a cost-effective manner. The results demonstrate the effectiveness of the proposed approach and provide insights into the importance of routing in LLM deployment.