💡 The paper introduces BlockPilot, a method for improving the efficiency of speculative decoding in natural language processing tasks. Speculative decoding is a technique that uses a lightweight model to generate candidate tokens in parallel, which are then verified by a target model. Existing methods use a fixed block size for decoding, which can be suboptimal as the optimal block size varies across different input samples. The authors show that the optimal block size is critical to speculative decoding performance and that it exhibits a local structure, meaning that it tends to concentrate around the training block size.

To address this issue, the authors propose a sample-adaptive policy that predicts the optimal block size from the prefilling representation. This is done by formulating block size selection as a lightweight policy learning problem, where the optimal block size is predicted based on the representation of the prefilling stage. The prediction is performed only once after prefilling, allowing for seamless integration with existing models.

The authors evaluate their method on several benchmarks and demonstrate that it is plug-and-play, introduces minimal overhead, and consistently improves efficiency. The results show that BlockPilot achieves an acceptance length of 5.92 and a 4.20 times speedup on a specific model, indicating that it can significantly accelerate inference while maintaining accuracy. Overall, the paper contributes to the development of more efficient and adaptive speculative decoding methods, which can be useful for a wide range of natural language processing applications.