Mamba uses selective state spaces to make parts of its state update depend on the input. Its original paper describes an architecture without attention or MLP blocks and reports linear scaling with sequence length (Gu and Dao, Mamba). That is a claim about this architecture, not every system described with the broad term state-space model.
For example, a builder considering a model for long streams of events could compare a Mamba-based implementation with an attention-based one on the same sequence lengths. Measure memory, throughput and task quality, including questions about events early in the stream. A favorable scaling rule alone does not decide whether the model retrieves the information your application needs.
Linear attention is a separate approach that reformulates attention using kernel feature maps (Katharopoulos et al., Transformers are RNNs). A convolutional hybrid is another distinct design. Older recurrent neural networks, including LSTMs, also carry a state from step to step, but update it with learned nonlinear transitions. In the related episode about Liquid AI’s architecture, Maxime Labonne says, “the architecture that we ended up with is a hybrid that has some attention layers.” (episode 43, 2:38).
That quote describes Labonne’s hybrid, not Mamba. For builders, inspect a model’s actual components before treating an architecture label as a deployment guarantee. Model architecture covers the wider set of choices; transformer defines the attention-based baseline. Are small language models better for production? connects architecture and size to a product’s requirements.
Sources
- Gu and Dao, Mamba — Primary source for the explanation and distinctions in this entry.
- Katharopoulos et al., Transformers are RNNs — Primary source for the explanation and distinctions in this entry.
