We introduce the concept of 'arithmetic intensity,' which determines how efficiently AI's brain, 'attention,' processes data, along with the latest technologies to increase it.
Imagine you are a chef, but for every ingredient you need, you have to walk 100 meters between the kitchen and the refrigerator. You would likely spend much more time traveling back and forth to fetch ingredients than actually cooking. No matter how fast your knife skills are, your overall cooking speed would be frustratingly slow.
The exact same thing is happening in the world of AI we use today. The core brain of the latest AI models, ‘Attention’ (an AI structure that identifies relationships between words in a sentence) Source 12, must constantly travel between memory (where data is stored) and hardware, just like our chef traveling to the refrigerator, when processing information. Today, I want to explain very easily why AI cannot run faster and about a secret indicator called ‘arithmetic intensity’ that engineers are focusing on to solve this problem.
Why It Matters
If the response speed of the AI chatbot we use is slow, it is not just a matter of frustration. This is because the cost of an AI service is directly linked to processing efficiency. In simple terms, if an AI can perform more calculations each time it fetches data from memory exactly once, we can build much faster and cheaper AI services with the same machine.
In other words, ‘engineering optimization’—how efficiently we squeeze the capabilities of AI onto hardware without waste—is just as important as increasing the AI’s intelligence, and it is the key that changes our daily AI experience.
The Explainer
AI engineers use an indicator called ‘Arithmetic Intensity’ to measure this efficiency Source 10.
To use an analogy, it is the ratio representing “how many calculations (FLOPs, floating-point operations) hardware performs when it fetches 1 byte of data from memory” Source 7, 11.
- Low Arithmetic Intensity: A situation where you walk back and forth to the refrigerator several times to barely chop one onion. (There is a lot of data movement, but little calculation is actually done)
- High Arithmetic Intensity: A situation where you bring all the ingredients from the refrigerator at once to boil a large pot of kimchi stew. (A lot of calculations are done with data brought once)
In the Transformer-based AI models we use today, the part with the highest computational cost is the attention layer Source 1. However, because attention creates too many intermediate data structures, it is stuck in a bottleneck where the speed of reading and writing data to and from memory is slower than its actual computational capability—a ‘memory-bound’ state Source 2, 13.
For example, based on the past A100 GPU, the required arithmetic intensity for efficient computation was 156 FLOPs/byte, but the actual intensity of a typical attention mechanism was only about 65 FLOPs/byte Source 2. This is similar to driving a high-end sports car but creeping along at 30 km/h due to a jammed road.
Where We Stand
To overcome this problem, engineers are redesigning the attention structure itself. Representative technologies include ‘Multi-Query Attention (MQA)’ and ‘Grouped-Query Attention (GQA)’ Source 6, 9.
These technologies dramatically reduce the amount of information that must be read from memory when calculating attention. Since the same result can be produced by reading less data, ‘arithmetic intensity’ naturally increases, and the overall processing speed improves Source 6, 9. In recent studies, there are also very active attempts to optimize the projection matrix of attention to nearly double the arithmetic intensity Source 9.
What’s Next
Future AI will develop in a direction that breaks through the hardware performance limits as much as possible rather than just increasing the size of the model Source 4. We will encounter AI that understands longer context with less power, and this will create an environment where we can run more powerful AI even on personal devices like smartphones Source 14.
MindTickleBytes’ AI Reporter View
The advancement of AI is not just about building a smarter brain. ‘Engineering efficiency’—how smartly we put that brain to work—is what accelerates the popularization of technology. This silent war to increase arithmetic intensity is the practical engine that will make AI deeply embedded in our daily lives.
References
- Transformer Inference Estimations: Arithmetic Intensity, Throughput
- 2.1: Standard Attention — The IO Problem
-
[Attention at Inference: Arithmetic Intensity… Aleksandr Timashov](https://timashov.ai/blog/2025/mha-during-inference/) - Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference
- Native Sparse Attention: Hardware-Aligned and Natively
- Multi-Query Attention is All You Need
-
[Attention & KV Cache Bottlenecks in Inference Medium](https://medium.com/@alice_gjw/deep-dive-2-attention-kv-cache-bottlenecks-in-inference-35ea2d52a34d) -
[[Tech] Why MLA and MTP Fight Each Other: Attention Through Arithmetic Intensity Changyi Yang’s Site](https://changyi.fun/posts/attention-arithmetic-intensity/) - Hardware-Efficient Attention for Fast Decoding
- FlashAttention: Making Attention I/O-Aware
-
[What is arithmetic intensity? GPU Glossary](https://modal.com/gpu-glossary/perf/arithmetic-intensity) -
[What is an attention mechanism? IBM](https://www.ibm.com/think/topics/attention-mechanism) - ELI5: Flash Attention
- Arithmetic Intensity In Decoding: A Hardware-Efficient Perspective…
- Total processing time vs. computation amount
- The ratio of data moved per operation
- The number of operations (FLOPs) performed per 1 byte of data moved from memory
- Because the amount of data movement is much larger than the amount of computation
- Because the hardware's computation speed is too slow
- Because data is not stored in memory
- By increasing the model's parameters
- By reducing the number of memory data reads required during attention calculation
- By increasing the computer's voltage