💡 The paper addresses the challenge of fine-tuning large language models on single GPUs, which is limited by the models' memory-intensive nature. To overcome this, the authors propose SlideFormer, a system designed for single-GPU environments. The key innovations of SlideFormer include a lightweight asynchronous engine that overlaps GPU computation with CPU updates and multi-tier I/O, a heterogeneous memory management scheme that reduces peak memory usage, and optimized kernels that solve key bottlenecks and integrate advanced I/O.

The asynchronous engine treats the GPU as a sliding window, allowing for efficient processing. The heterogeneous memory management scheme significantly reduces memory usage, making it possible to fine-tune larger models. The optimized kernels improve performance by solving key bottlenecks and integrating advanced I/O.

The results show that SlideFormer achieves higher throughput and reduced memory usage compared to baselines. Specifically, it supports up to 8 times larger batch sizes and 6 times larger models, and achieves 1.40 to 6.27 times higher throughput while roughly halving CPU and GPU memory usage. The system sustains over 95 percent peak performance on both NVIDIA and AMD GPUs, demonstrating its effectiveness and efficiency. Overall, SlideFormer enables the fine-tuning of large language models on single GPUs, making it a significant contribution to the field of natural language processing.