✨TimeBill: Time-Budgeted Inference for Large Language Models

📝 Summary:
TimeBill is a framework for LLMs in time-critical systems. It predicts execution time and adaptively adjusts KV cache eviction to balance inference efficiency and response performance within given time budgets, improving task completion rates.

🔹 Publication Date: Published on Dec 26

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.21859
• PDF: https://arxiv.org/pdf/2512.21859

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LLM #AI #RealTimeAI #InferenceOptimization #DeepLearning

✨POP: Prefill-Only Pruning for Efficient Large Model Inference

📝 Summary:
POP is a new stage-aware pruning method for large models. It omits deep layers during the computationally intensive prefill stage while using the full model for decoding. This achieves up to 1.37 times prefill speedup with minimal accuracy loss, overcoming limitations of prior pruning methods.

🔹 Publication Date: Published on Feb 3

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.03295
• PDF: https://arxiv.org/pdf/2602.03295

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#AI #MachineLearning #LLM #ModelPruning #InferenceOptimization

✨Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey

📝 Summary:
This survey analyzes dynamic routing systems that adaptively select among multiple independent LLMs based on query characteristics to optimize inference performance and cost. It covers diverse routing paradigms and presents a framework for understanding these systems, highlighting their ability t...

🔹 Publication Date: Published on Feb 23

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.04445
• PDF: https://arxiv.org/pdf/2603.04445

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LLM #AI #ModelRouting #InferenceOptimization #DeepLearning