✨Distribution-Aligned Sequence Distillation for Superior Long-CoT Reasoning

📝 Summary:
DASD-4B-Thinking is a new lightweight model achieving state-of-the-art reasoning by enhancing sequence-level distillation. It addresses limitations in current teacher-student knowledge transfer by better capturing the teachers full output distribution, using significantly fewer training samples.

🔹 Publication Date: Published on Jan 14

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2601.09088
• PDF: https://arxiv.org/pdf/2601.09088
• Project Page: https://github.com/D2I-ai/dasd-thinking
• Github: https://github.com/D2I-ai/dasd-thinking

🔹 Models citing this paper:
• https://huggingface.co/Alibaba-Apsara/DASD-4B-Thinking
• https://huggingface.co/Alibaba-Apsara/DASD-30B-A3B-Thinking-Preview

✨ Datasets citing this paper:
• https://huggingface.co/datasets/Alibaba-Apsara/Superior-Reasoning-SFT-gpt-oss-120b
• https://huggingface.co/datasets/Alibaba-Apsara/Superior-Reasoning-SFT-gpt-oss-120b-Logprob

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#AI #MachineLearning #LLM #KnowledgeDistillation #ChainOfThought

✨Shallow-π: Knowledge Distillation for Flow-based VLAs

📝 Summary:
Shallow-pi is a knowledge distillation framework that reduces transformer depth in vision-language-action models. It achieves over two times faster inference with less than one percent performance drop, enabling efficient real-world robotic deployment.

🔹 Publication Date: Published on Jan 28

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2601.20262
• PDF: https://arxiv.org/pdf/2601.20262
• Project Page: https://icsl-jeon.github.io/shallow-pi/
• Github: https://icsl-jeon.github.io/shallow-pi/

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#KnowledgeDistillation #Robotics #VLAModels #EfficientAI #DeepLearning

✨Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs

📝 Summary:
This paper introduces Knowledge Purification, consolidating multi-teacher LLM rationales to reduce conflicts and improve distillation efficiency. Methods improve model performance and reduce conflicts; router-based methods generalize robustly.

🔹 Publication Date: Published on Feb 1

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.01064
• PDF: https://arxiv.org/pdf/2602.01064

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LLM #KnowledgeDistillation #KnowledgePurification #AI #DeepLearning

✨Reinforcement-aware Knowledge Distillation for LLM Reasoning

📝 Summary:
RL-aware distillation RLAD improves knowledge transfer from RL-trained LLMs to smaller students. It addresses distribution mismatch and objective interference by using Trust Region Ratio Distillation TRRD. TRRD replaces the KL regularizer with a likelihood-ratio objective, balancing exploration, ...

🔹 Publication Date: Published on Feb 26

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.22495
• PDF: https://arxiv.org/pdf/2602.22495

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LLMs #KnowledgeDistillation #ReinforcementLearning #NLP #AI

✨PACED: Distillation at the Frontier of Student Competence

📝 Summary:
PACED optimizes distillation by focusing training on a student competence frontier using a Beta kernel weighting. Derived from gradient analysis, this avoids wasted compute at extremes, boosting distillation and self-distillation performance.

🔹 Publication Date: Published on Mar 11

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.11178
• PDF: https://arxiv.org/pdf/2603.11178

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#KnowledgeDistillation #DeepLearning #ModelOptimization #AIResearch #ComputeEfficiency

✨Efficient Universal Perception Encoder

📝 Summary:
EUPE enhances edge device performance through a novel two-stage knowledge distillation approach. It scales up to a large proxy teacher then down to an efficient encoder. This method provides superior, versatile representations for diverse tasks, outperforming prior techniques.

🔹 Publication Date: Published on Mar 23

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.22387
• PDF: https://arxiv.org/pdf/2603.22387
• Github: https://github.com/facebookresearch/eupe

🔹 Models citing this paper:
• https://huggingface.co/facebook/EUPE-ConvNeXt-S
• https://huggingface.co/facebook/EUPE-ViT-S
• https://huggingface.co/facebook/EUPE-ViT-B

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#KnowledgeDistillation #EdgeAI #ComputerVision #DeepLearning #RepresentationLearning