✨Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models

📝 Summary:
Masked Diffusion Language Models MDLMs show locality bias and poor context comprehension due to appended mask tokens acting as distractors. A mask-agnostic loss function was introduced. This function improves MDLM robustness by mitigating the masks distracting effect.

🔹 Publication Date: Published on Nov 26

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2511.21338
• PDF: https://arxiv.org/pdf/2511.21338

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LanguageModels #DiffusionModels #NLP #ContextComprehension #AIResearch

✨Scaling Behavior of Discrete Diffusion Language Models

📝 Summary:
Research on discrete diffusion language models DLMs shows their scaling behavior depends on noise type. Uniform diffusion is more parameter and data efficient than masked diffusion, making it promising for data-bound settings. A 10B parameter model confirmed this.

🔹 Publication Date: Published on Dec 11

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.10858
• PDF: https://arxiv.org/pdf/2512.10858
• Github: https://github.com/dvruette/gidd-easydel

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#DiffusionModels #LanguageModels #NLP #AIResearch #DeepLearning

✨Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

📝 Summary:
Canon layers are lightweight architectural components that enhance language model reasoning depth and breadth by promoting horizontal information flow. They improve performance across various architectures, validated in synthetic tasks and real-world pretraining.

🔹 Publication Date: Published on Dec 19

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.17351
• PDF: https://arxiv.org/pdf/2512.17351
• Project Page: https://physics.allen-zhu.com/part-4-architecture-design/part-4-1
• Github: https://github.com/facebookresearch/PhysicsLM4

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LanguageModels #LLM #AIArchitecture #DeepLearning #NLP

✨Bolmo: Byteifying the Next Generation of Language Models

📝 Summary:
Bolmo introduces competitive byte-level language models by efficiently converting existing subword models. This byteification overcomes subword limitations, matching performance with minimal training. Bolmo makes byte-level LMs practical.

🔹 Publication Date: Published on Dec 17

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2512.15586
• PDF: https://arxiv.org/pdf/2512.15586

🔹 Models citing this paper:
• https://huggingface.co/allenai/Bolmo-7B
• https://huggingface.co/allenai/Bolmo-1B

✨ Datasets citing this paper:
• https://huggingface.co/datasets/allenai/bolmo_mix

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LanguageModels #ByteLevelLMs #NLP #DeepLearning #AIResearch

✨End-to-End Test-Time Training for Long Context

📝 Summary:
This paper proposes End-to-End Test-Time Training TTT-E2E for long-context language modeling, treating it as continual learning. It uses a standard Transformer, learning at test time and improving initialization via meta-learning. TTT-E2E scales well and offers constant inference latency, being m...

🔹 Publication Date: Published on Dec 29, 2025

🔹 Paper Links:
• arXiv Page: https://arxivlens.com/PaperView/Details/end-to-end-test-time-training-for-long-context-6176-bf8fd7e6
• PDF: https://arxiv.org/pdf/2512.23675
• Github: https://github.com/test-time-training/e2e

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#TestTimeTraining #LongContext #LanguageModels #Transformers #ContinualLearning

✨ASTRA: Automated Synthesis of agentic Trajectories and Reinforcement Arenas

📝 Summary:
ASTRA automates training tool-augmented language models for multi-step decision-making. It uses synthetic data and verifiable reinforcement learning, integrating SFT and online RL. This achieves state-of-the-art performance.

🔹 Publication Date: Published on Jan 29

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2601.21558
• PDF: https://arxiv.org/pdf/2601.21558
• Github: https://lianjiatech.github.io/astra.blog/

🔹 Models citing this paper:
• https://huggingface.co/Emperorizzis/ASTRA-14B-Thinking-v1
• https://huggingface.co/Emperorizzis/ASTRA-32B-Thinking-v1

✨ Datasets citing this paper:
• https://huggingface.co/datasets/Emperorizzis/ASTRA-SFT-1k
• https://huggingface.co/datasets/Emperorizzis/ASTRA-RL-1k

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#AI #ReinforcementLearning #LanguageModels #MultiStepDecisionMaking #MachineLearning

✨Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Models

📝 Summary:
This work introduces Self-Rewarding Sequential Monte Carlo SMC to improve masked diffusion language model sampling. SMC uses multiple parallel diffusion processes and trajectory-level confidence as a self-rewarding signal, guiding generation to high-quality samples and boosting performance withou...

🔹 Publication Date: Published on Feb 2

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.01849
• PDF: https://arxiv.org/pdf/2602.01849
• Project Page: https://algolzw.github.io/sr-smc/
• Github: https://github.com/Algolzw/self-rewarding-smc

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#DiffusionModels #SequentialMonteCarlo #LanguageModels #GenerativeAI #MachineLearning

✨CL-bench: A Benchmark for Context Learning

📝 Summary:
Current LMs struggle with context learning, requiring new knowledge and reasoning beyond pre-training. The CL-bench, a new real-world benchmark, reveals models solve only 17.2 percent of tasks, showing a critical bottleneck for complex real-world applications.

🔹 Publication Date: Published on Feb 3

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.03587
• PDF: https://arxiv.org/pdf/2602.03587
• Project Page: https://www.clbench.com
• Github: https://github.com/Tencent-Hunyuan/CL-bench

✨ Datasets citing this paper:
• https://huggingface.co/datasets/tencent/CL-bench

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ContextLearning #LanguageModels #AIBenchmark #NLP #AIResearch

✨Proxy Compression for Language Modeling

📝 Summary:
Proxy compression trains language models on both raw bytes and compressed views. This enables efficient training on compressed inputs while offering a robust, end-to-end raw-byte inference. It improves training efficiency and eventually matches tokenizer performance.

🔹 Publication Date: Published on Feb 4

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.04289
• PDF: https://arxiv.org/pdf/2602.04289
• Github: https://github.com/LZhengisme/proxy-compression

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LanguageModels #Compression #MachineLearning #AI #Efficiency

✨Exposing the Systematic Vulnerability of Open-Weight Models to Prefill Attacks

📝 Summary:
A study reveals prefill attacks as a critical, underexplored vulnerability in open-weight language models. These attacks, which predefine initial response tokens, consistently compromise major models, necessitating urgent defense development.

🔹 Publication Date: Published on Feb 16

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.14689
• PDF: https://arxiv.org/pdf/2602.14689

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#PrefillAttacks #LLMSecurity #AIvulnerability #OpenWeightModels #LanguageModels

✨Can I Have Your Order? Monte-Carlo Tree Search for Slot Filling Ordering in Diffusion Language Models

📝 Summary:
McDiffuSE uses Monte Carlo Tree Search to optimize slot infilling order in Masked Diffusion Models, enhancing reasoning performance. It achieved significant gains, revealing non-sequential generation and larger exploration are key to overcoming model biases.

🔹 Publication Date: Published on Feb 13

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.12586
• PDF: https://arxiv.org/pdf/2602.12586

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#MonteCarloTreeSearch #DiffusionModels #NLP #LanguageModels #AI

✨HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam

📝 Summary:
HLE-Verified systematically validates and revises the HLE benchmark, resolving noisy items through expert review and model-based checks. This improves language model evaluation accuracy by 7-10 percentage points, especially on erroneous items, enabling more reliable measurement of model capabilit...

🔹 Publication Date: Published on Feb 15

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.13964
• PDF: https://arxiv.org/pdf/2602.13964

✨ Datasets citing this paper:
• https://huggingface.co/datasets/skylenage/HLE-Verified

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LLMEvaluation #Benchmarking #LanguageModels #AIResearch #NLP

✨Sink-Aware Pruning for Diffusion Language Models

📝 Summary:
Diffusion Language Models have high inference costs. This paper finds that their attention sinks are often unstable, unlike in autoregressive models. Sink-Aware Pruning identifies and removes these unstable sinks, improving efficiency and quality without retraining.

🔹 Publication Date: Published on Feb 19

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.17664
• PDF: https://arxiv.org/pdf/2602.17664
• Github: https://github.com/VILA-Lab/Sink-Aware-Pruning

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#DiffusionModels #LanguageModels #ModelPruning #NLP #AIResearch

✨One-step Language Modeling via Continuous Denoising

📝 Summary:
This paper introduces flow-based language models that use continuous denoising over one-hot token encodings. They surpass discrete diffusion models in quality and speed, particularly for few-step generation, challenging discrete diffusion's necessity for discrete data.

🔹 Publication Date: Published on Feb 18

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2602.16813
• PDF: https://arxiv.org/pdf/2602.16813
• Project Page: https://one-step-lm.github.io/
• Github: https://github.com/david3684/flm

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LanguageModels #GenerativeAI #DeepLearning #NLP #AI

✨Reward Prediction with Factorized World States

📝 Summary:
StateFactory transforms observations into hierarchical object-attribute structures using language models. This enables superior zero-shot reward prediction across domains by measuring semantic similarity, significantly improving agent planning performance.

🔹 Publication Date: Published on Mar 10

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.09400
• PDF: https://arxiv.org/pdf/2603.09400
• Project Page: https://statefactory.github.io/
• Github: https://github.com/yijunshens/StateFactory

✨ Datasets citing this paper:
• https://huggingface.co/datasets/YijunShen/RewardPrediction

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#RewardPrediction #AI #LanguageModels #MachineLearning #AgentPlanning

✨Training Language Models via Neural Cellular Automata

📝 Summary:
This paper introduces using Neural Cellular Automata NCA to generate synthetic data for pre-pre-training language models, addressing natural language limitations. This approach improves performance, accelerates convergence, and transfers to reasoning tasks, often outperforming extensive natural l...

🔹 Publication Date: Published on Mar 9

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.10055
• PDF: https://arxiv.org/pdf/2603.10055

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#AI #LanguageModels #NeuralCellularAutomata #SyntheticData #NLP

✨LoopRPT: Reinforcement Pre-Training for Looped Language Models

📝 Summary:
LoopRPT is a reinforcement pre-training framework for looped language models. It directly shapes intermediate representations by assigning reinforcement signals to latent steps, improving latent reasoning. This leads to better accuracy-computation trade-offs and enhanced early-stage reasoning.

🔹 Publication Date: Published on Mar 20

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.19714
• PDF: https://arxiv.org/pdf/2603.19714

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#ReinforcementLearning #LanguageModels #AI #NLP #DeepLearning

✨Diffutron: A Masked Diffusion Language Model for Turkish Language

📝 Summary:
Diffutron introduces a compact masked diffusion language model for Turkish. It uses resource-efficient LoRA-based pre-training and progressive instruction tuning. The model achieves competitive performance for non-autoregressive Turkish text generation despite its small size.

🔹 Publication Date: Published on Mar 20

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2603.20466
• PDF: https://arxiv.org/pdf/2603.20466

🔹 Models citing this paper:
• https://huggingface.co/diffutron/DiffutronLM-0.3B-Instruct
• https://huggingface.co/diffutron/DiffutronLM-0.3B-Base
• https://huggingface.co/diffutron/DiffutronLM-0.3B-1st-Stage

✨ Datasets citing this paper:
• https://huggingface.co/datasets/diffutron/DiffutronLM-Pretraining-Corpus

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#LanguageModels #TurkishNLP #DiffusionModels #NLP #AI

✨T5Gemma-TTS Technical Report

📝 Summary:
T5Gemma-TTS is an encoder-decoder codec language model that improves voice cloning and duration control for multilingual speech synthesis. It uses cross-attention for persistent text conditioning and Progress-Monitoring Rotary Position Embedding PM-RoPE for better target speech length tracking. I...

🔹 Publication Date: Published on Apr 2

🔹 Paper Links:
• arXiv Page: https://arxiv.org/abs/2604.01760
• PDF: https://arxiv.org/pdf/2604.01760
• Github: https://github.com/Aratako/T5Gemma-TTS

🔹 Models citing this paper:
• https://huggingface.co/Aratako/T5Gemma-TTS-2b-2b

✨ Spaces citing this paper:
• https://huggingface.co/spaces/Aratako/T5Gemma-TTS-Demo
• https://huggingface.co/spaces/litagin/T5Gemma-TTS-Demo

==================================

For more data science resources:
✓ https://xn--r1a.website/DataScienceT

#SpeechSynthesis #TTS #VoiceCloning #Multilingual #LanguageModels

🔥 Tmax: A simple recipe for terminal agents

💡 The paper presents a novel approach to training terminal agents using reinforcement learning, called Tmax. Terminal agents are a popular application of language models, but their training has been hindered by the lack of simple and effective methods, limited data, and challenging benchmarks. The authors address these issues by introducing a simplified recipe for training terminal agents, which achieves superior performance with fewer parameters than previous methods.

The method involves generating a large dataset of terminal environments using a novel taxonomy that combines difficulty control, personas, and verifier diversification. This allows for the cheap generation of large amounts of data, which is then used to train open-weight models using reinforcement learning with a simple outcome-only recipe.

The results show that Tmax achieves 27 percent on Terminal-Bench 2.0 with only 9 billion parameters, outperforming much larger models from prior work. The authors also release their terminal dataset, which is over 2.5 times larger than previously released terminal-agent datasets, as well as their models and code as a strong baseline for future academic work on terminal agents.

The contributions of the paper are threefold. First, it presents a simple and effective recipe for training terminal agents, which can be used as a baseline for future work. Second, it introduces a novel taxonomy for generating terminal environments, which allows for the cheap generation of large amounts of data. Third, it releases a large dataset of terminal environments, models, and code, which can be used by other researchers to advance the field of terminal agents. Overall, the paper provides a significant contribution to the field of terminal agents and reinforcement learning, and has the potential to advance the state of the art in this area.

📅 Published on Jun 22

🔗 Links:
• GitHub: https://github.com/huggingface
• arXiv: https://arxiv.org/abs/2606.23321
• PDF: https://arxiv.org/pdf/2606.23321
• Project Page: https://wai-org.com/blog/tmax/

🤖 Models citing this paper:
• https://huggingface.co/allenai/tmax-27b
• https://huggingface.co/allenai/tmax-9b
• https://huggingface.co/allenai/qwen35-9b-openthoughts

📊 Datasets citing this paper:
• https://huggingface.co/datasets/allenai/TMax-15K
• https://huggingface.co/datasets/allenai/tmax-15k-open-instruct
• https://huggingface.co/datasets/allenai/tmax-sft

━━━━━━━━━━━━━━━━━━━━━━━━
📢 By: https://xn--r1a.website/PaperNexus

#TerminalAgents #ReinforcementLearning #LanguageModels #TmaxAlgorithm #AgentTrainingMethods