Learn how new techniques make AI models faster, smarter, and more efficient by reducing memory use and speeding up training.

Learn how incorporating input-dependent parameters transforms state space models from time-invariant to time-varying.

Discover how SSMs improve sequence modeling by leveraging context compression, selectivity, and hardware-aware algorithms for efficient AI performance.

Explore state space models (SSMs), their structured architecture, and innovations like H3, Hyena, and RWKV that revolutionize AI sequence modeling efficiency.

Mamba, a new linear-time model, matches Transformer performance with 5× higher efficiency, excelling in language, audio, and genomics tasks.

Explore the simplified state space model (SSM) architecture that combines linear attention and MLP components into a unified block.

Mamba matches GPT-3 in language modeling, outperforming leading models without using traditional attention mechanisms, setting a new standard for efficiency.

Mamba’s memory system excels in solving sequence tasks, outperforming other models with its ability to remember key data and handle longer sequences.

New research shows real-valued SSMs outperform complex ones in discrete tasks. We explore initialization strategies and their impact on SSM efficiency.

Selection mechanisms are a powerful tool in sequence modeling, improving models like RNNs, CNNs, and SSMs.

Mamba excels in DNA modeling, outperforming HyenaDNA and Transformer++ in scaling laws for model size and context length.

Mamba introduces transformative concepts in language modeling with its selective SSMs, but the scaling of these models faces engineering challenges.

Mamba’s architecture is enhanced by the selective SSM layer, yielding significant performance improvements.

Mamba outperforms FlashAttention-2 and Transformers in both speed and memory efficiency, offering up to 20-40x faster inference throughput.

Mamba sets a new benchmark for audio waveform modeling and autoregressive speech generation, surpassing SaShiMi and other models.

Explore detailed experimental results from cutting-edge AI architectures, including Selective Copying, Induction Heads, and innovative models like Mamba and Hye

Selective SSMs achieve up to 7× faster processing than attention and optimal memory efficiency with kernel fusion and recomputation on modern GPUs.

Explore the mechanics of Selective SSMs, focusing on discretization, learnable biases, and zero-order hold formulas for efficient AI modeling.

Discover how new AI models handle longer tasks, improve learning, and bring smarter, faster performance to the next generation of technology.