💡 The paper introduces TRACER, a system that uses production traces to train machine learning surrogates for large language model classification. The problem addressed is that large language models can be costly to deploy and may not always be necessary for every input. The authors propose a method where a lightweight surrogate model is trained on the production logs of the large language model, which can absorb a significant portion of future traffic at near-zero marginal inference cost.

The TRACER system trains the surrogate model on the production traces and governs its deployment through a parity gate, which activates the surrogate only when its agreement with the large language model exceeds a user-specified threshold. This approach provides interpretability into the handling boundaries of the surrogate model, describing which input regions it handles, where it plateaus, and why it defers.

The results show that TRACER achieves significant surrogate coverage on a 77-class intent benchmark, with coverage ranging from 83 to 100 percent depending on the quality target. On a 150-class benchmark, the surrogate fully replaces the teacher model. Additionally, the parity gate correctly refuses deployment on a natural language inference task when the embedding representation cannot support reliable separation. The TRACER system is available as open-source software, providing a cost-efficient and interpretable solution for large language model classification.