Moving beyond the attention mechanism central to the 2017 'Attention Is All You Need' paper that launched the AI era, 'Grassmann flow' models are gaining attention as a new alternative by leveraging geometric structures.
Imagine this: when we have a conversation with a friend, we don’t listen to every single word they say with the same level of weight. We grasp the context by “focusing” (paying attention) only on the important words. In 2017, the Google Brain team published a groundbreaking paper titled Attention Is All You Need, which applied this principle to artificial intelligence. The structure introduced in this paper, called the “Transformer,” has become the brain of the AI we use every day, such as ChatGPT, Claude, and Gemini. [Source 5, Source 9, Source 18]
However, an interesting movement has recently been detected in the AI academic community. New models are emerging that pose a fundamental question: “Is ‘Attention’ truly the only correct answer for AI?”
Why Is This Important?
While the Transformer has clearly brought about a quantum leap in AI, it also possesses fundamental limitations that are difficult to solve. The most representative issue is that the computational load increases exponentially as sentences get longer. If a more efficient structure can be found that processes information in a way different from the Transformer, AI will be able to think more intelligently with far fewer resources than it does now. This means lower AI usage costs and the ability to run more powerful AI on everyday devices like the smartphones we use daily.
Understanding It Simply: A Tangled Thread or a Flow?
Existing Transformers use an “attention mechanism” to understand the relationships between words. Using a simple analogy, it’s a method of connecting countless threads between important destinations (words) on the map that is a sentence. When a sentence gets long, this web of threads becomes too complex to handle. It is similar to the high energy required to decide whose words to listen to first when many people are talking at the same time. [Source 13]
On the other hand, the recently proposed “Grassmann flows” method has a completely different approach. [Source 3] If attention is the “art of connection” that examines the relationships between words one by one, Grassmann flows view information as a point on a geometric space (manifold) and track how this information moves across that space. [Source 4]
Simply put, instead of complexly connecting threads, think of it as a method of finding a “path” where points of information flow while drawing a smooth curve. This allows for a much more efficient grasp of the overall flow without having to compare words one by one. [Source 3, Source 4]
Current Status: The Transformer’s Monopoly and a New Challenge
The Transformer is still the standard for AI. This is because its ability to compute data all at once—parallel processing—demonstrates much faster and more powerful performance than RNNs (Recurrent Neural Networks), which were the sequential methods of the past. [Source 10, Source 13] Researchers are not stopping here, but are constantly breaking through the limitations of Transformers by adding various optimization techniques like “FlashAttention” and “Mixture of Experts.” [Source 11]
Nevertheless, the fact that a new alternative called an “attention-free model” has appeared to challenge the Transformer’s dominance is highly significant. Although models like Grassmann flows are only in their infancy, they are showing new possibilities that break the existing mold in the field of sequence modeling. [Source 3]
What Will Happen in the Future?
AI models will develop beyond simply “how much larger data can we learn” toward “how can we find a more efficient structure.” If these geometric approaches currently being researched are perfected to a commercial level, AI will be able to understand data faster and more accurately with much less power. The technology we currently call “Attention” may one day become a classic technique that marks a page in the history of AI technology.
Technological progress never stays in one place. The efforts of scientists to cross the massive wall called “Attention” and find another efficient path are leading us toward a more astonishing future.
References
- Abstract page for arXiv paper 1706.03762:AttentionIsAllYouNeed
-
[AttentionIsNotWhatYouNeed alphaXiv](https://www.alphaxiv.org/overview/2512.19428v1) -
[AttentionisNotWhatYouNeed Aelias](https://aelias.dev/reading-log/attention-is-not-what-you-need/) - Attentionisn’t allyouneed: The wanderer’s algorithm
- AttentionIsNOTAll That WENeed
- AttentionIsAllYouNeed— на русском — IRI Lab
- AttentionIsAllYouNeed：關於Transformer你所需要知道的關鍵論文
- AttentionisAllYouNeed: статья, изменившая AI — AI-Uchi
- 3 Lessons from the paper “AttentionIsAllYouNeed” as a Beginner
- AttentionisAllYouNeed
- RNN
- Grassmann flows
- Transformer
- Attention-based model
- Attention-free sequence model
- An evolution of existing RNNs
- Parallel processing allows for shorter computation time
- Uses less memory
- Does not require an internet connection