How Does AI Think? The Mathematical Key to Peering Inside Massive Neural Networks

Imagine you are a very clever dog trainer. The dog performs your commands perfectly, but you have no idea what the dog is thinking to perform those actions. Is it simply the result of training, or does the dog have its own logic?

AI models like ChatGPT, which we use every day, are similar. They produce amazing results after learning from vast amounts of data, but what happens inside that massive neural network remains veiled, much like a ‘black box.’ Today, we look at a landmark 2021 study by Anthropic, ‘A Mathematical Framework for Transformer Circuits,’ which sought to open this black box and peer into the inside of AI mathematically. Source: A Mathematical Framework for Transformer Circuits

Why Is This Important?

As AI spreads throughout society, questions like ‘Why did the AI give this answer?’ and ‘Can we really trust it?’ have become critical topics. If an AI provides biased information or makes a wrong judgment, we must be able to find the cause internally and fix it.

This research goes beyond mere curiosity; it is an effort to draw a ‘mathematical map’ to allow us to perfectly control and understand the massive technology that is AI. Source: A Mathematical Framework for Transformer Circuits \ Anthropic This study is evaluated as the pioneer of the field of ‘Mechanistic Interpretability’ (the logical and mathematical analysis of how AI processes data internally), attempting to translate internal AI operations into precise mathematical language. Source: [Review] A Mathematical Framework for Transformer Circuits

Easy Understanding: Dissecting AI’s ‘Brain Circuit’

The core of this research begins with a very simple question: “Can we explain the small-scale algorithms executed by AI in precise mathematical terms, and can we read exactly what they are doing just by looking at their weights (the numerical values adjusted by AI during learning)?” Source: Circuits 01 — A Mathematical Framework for Transformer Circuits

To this end, the researchers broke down the Transformer (the core AI structure that understands relationships between words in a sentence) model into a very simple form with two or fewer layers for analysis. Source: A Mathematical Framework for Transformer Circuits

To use an analogy: Imagine a very complex 100-story skyscraper in front of you. The blueprints are so complicated that they are difficult to understand at a glance. Instead of digging into the entire structure of the building, the researchers separated only the first and second floors and began observing how the wires inside were connected with a microscope. Source: A Mathematical Framework for Transformer Circuits

The researchers viewed the ‘residual stream’ (a type of communication channel where AI stores and continuously updates information while processing sentences), which is the pathway through which AI exchanges information, as a communication channel that transmits information via addition. Source: mathematicalframeworkfortransformercircuits Simply put, it is similar to the process where several people write in a single notebook simultaneously, accumulating information. By applying an Attention mechanism (a function that determines which words in a sentence are important) to this, they decomposed and analyzed it into a mathematical framework consisting of matrices (QK) that determine whether to focus on specific information and matrices (OV) that determine how to reflect that information. Source: mathematicalframeworkfortransformercircuits

Current Situation: How Far Have We Come?

Currently, this research has become an important foundation for providing a ‘mental model’ for AI researchers to infer the inside of AI models. Source: Review: A Mathematical Framework for Transformer Circuits However, the latest models we use are like giant monsters with trillions of parameters (numerical values finely adjusted by AI as it learns). They are far more complex than the two-layer model covered in this study. Source: A Mathematical Framework for Transformer Circuits Therefore, applying the methodology of this study perfectly to actual massive models remains a challenging task.

What Will Happen in the Future?

The ‘mathematical language’ presented by this study continues to evolve. Researchers are striving to gradually apply the simple algorithmic patterns discovered here to larger and more complex models. Source: A Mathematical Framework for Transformer Circuits Perhaps one day, when we ask an AI, “Why did you give that answer?”, the day may come when the AI can explain its internal circuits based on mathematical evidence.

MindTickleBytes AI Reporter’s View

In the tide of the massive technology known as AI, the attempt to dissect its interior is a noble effort to secure the ‘transparency’ and ‘trust’ of the technology. Only when we understand AI not as a magic box, but as a machine with clear rules called mathematics, can we confidently welcome a future where we coexist with AI.

References

1. A Mathematical Framework for Transformer Circuits
2. A Walkthrough of A Mathematical Framework for Transformer Circuits — Neel Nanda
3. A Mathematical Framework for Transformer Circuits \ Anthropic
4. A Mathematical Framework for Transformer Circuits
5. Arxiv Dives - A Mathematical Framework for Transformer Circuits - Part 1
6. A Walkthrough of A Mathematical Framework for Transformer Circuits - YouTube
7. A Mathematical Framework for Transformer Circuits
8. Circuits 01 — A Mathematical Framework for Transformer Circuits
9. Review: A Mathematical Framework for Transformer Circuits
10. Review: A Mathematical Framework for Transformer Circuits

11. [A Mathematical Framework for Transformer Circuits…

    HackerNews](https://news.ycombinator.com/item?id=49672365)

12. A Mathematical Framework for Transformer Circuits \ Anthropic
13. A Mathematical Framework for Transformer Circuits - nikkie-memos
14. mathematicalframeworkfortransformercircuits
15. A Mathematical Framework for Transformer Circuits: How LLMs…

16. [TransformerCircuits1: Summary of Results

    3rd layer](https://3rdlayer.uk/posts/framework-01-summary/)