AI Reads and 'Summarizes' Papers? Does It Really Understand? Implementing GPT-2 Using Only CMake

Imagine. What if we could take apart the ‘brain’ of the AI assistant we use on our smartphones today to see exactly how it crafts sentences? To most people, AI feels like ‘magic.’ It’s like a black box that spits out answers when you press a button. However, developers want to open that box.

Recently, going beyond just peeking inside, there’s a trend of experimental projects that rebuild the structure of this massive AI from the ground up using only the most basic tools. There has even been an attempt to implement an AI model called GPT-2 using only CMake (a tool for automating program builds). [Source 8, Source 11, Source 12]

Why Is This Important?

Why are people taking on this ‘hardship’ in their busy lives? It’s like building a castle by carving wood and molding mud yourself, rather than using pre-assembled Lego blocks. Today, most AI development happens on top of huge, convenient tools like PyTorch (a complex library for AI development). However, these tools are so convenient that they often obscure the core process of what mathematical calculations AI actually performs within the data.

These ‘from scratch’ experiments lower the barrier to entry for AI development and help average developers fundamentally understand how AI works. [Source 10, Source 13] If we build a model ourselves, we can grasp the logical path behind why AI gives specific answers much more deeply.

Understanding It Simply: Building the ‘Brain’ of AI

Simply put, current AI models are a massive collection of numerous ‘weights’ (numerical values multiplied when processing data). These weights are complexly connected to complete sentences. To use a metaphor to understand this, AI is like a complex plumbing system with tens of thousands of faucets connected. Depending on which faucet you open and how much (how you adjust the weights), the amount and direction of the water (result) changes.

Andrej Karpathy (an AI scientist formerly at OpenAI) showed an amazing experiment through the ‘llm.c’ project, packing this huge AI into about 1,000 lines of code using only pure C language. [Source 2, Source 3, Source 17, Source 18] It was a ‘diet’ that left only the essential code and showed the core structure of a task that would normally require the help of external libraries spanning hundreds of thousands of lines.

The CMake implementation that appeared here is a case that took this experiment a step further. [Source 8, Source 11] By utilizing CMake, a management tool normally used to make programs into executable files, they wove in the calculation logic of AI. It is being accepted among developers as a kind of ‘technical amusement’ and a ‘challenge to limits,’ similar to making ‘bricks’ yourself with a ‘blueprint’ for building a house. [Source 9]

Current Situation: How Far Has It Come?

Of course, these experimental implementations cannot replace ChatGPT right now. In the case of the model implemented in CMake, the speed at which the program runs is inevitably very slow. This is because CMake originally works like an interpreter (a way of interpreting code line by line) and undergoes inefficient processes, such as converting numbers into strings every time during calculation. [Source 12]

Nevertheless, these attempts are very valuable. OpenAI’s GPT-2 model also has aspects that are not fully understood, such as its robustness or behavior in worst-case scenarios. [Source 4] Therefore, these ‘clean room’ implementations (building from scratch without external libraries) serve as the most perfect textbook for learning while taking apart the internal structure of AI piece by piece. [Source 10, Source 13]

What Will Happen in the Future?

AI technology will become increasingly popularized in the future. While only a very small number of engineers can implement AI right now, as projects like ‘llm.c’ or ‘microgpt’ that explain principles in around 265 lines of code increase, AI technology will become more transparent. [Source 16, Source 17]

We might soon live in an era where we can easily check how AI works, from mathematical principles down to the unit of code. The next time AI summarizes meeting materials for you, instead of just being amazed, why not imagine for a moment, ‘Ah, the core of that massive model started from this single line of code’?

MindTickleBytes’ AI Reporter Perspective

After stripping away the shells of complex technology, all that remains is simple mathematics and logic. As technology advances, these attempts to explore its ‘essence’ will cultivate the real literacy we need in the AI era.

References

1. [Vue HN 2.0

   Implementation of GPT-2 in pure CMake](https://vue-hackernews-ssr-5cavbdjcta-ew.a.run.app/item/49412909)

2. Andrej Karpathy Trains GPT-2 in Pure C Without PyTorch
3. Why Implement GPT-2 in Pure C Language? Karpathy Responds to Online Criticism - Boardor
4. GitHub - openai/gpt-2: Code for the paper “Language Models are…”
5. Need help with implementing gpt-2 from scratch - Deep Learning…
6. project — CMake 4.4.2 Documentation
7. Free GPT Image 2 AI Image Generator & Editor (No Signup, Unlimited)
8. Implementation of GPT-2 in pure CMake - GitHub
9. The Ultimate Tech Flex: Implementing GPT-2 in Pure CMake
10. GitHub - shaktsin/gpt2.c: GPT2 Inference Implementation in …
11. Implementation of GPT-2 in pure CMake - thenote.app

12. [Implementation of GPT-2 in pure CMake

    Hacker News](https://news.ycombinator.com/item?id=49412909)

13. Deconstruction Series #1: Rebuilding GPT-2 in Pure C
14. NanoEuler Tutorial: Run GPT-2 in Pure C/CUDA — AI Tutorial
15. GitHub - angry-kratos/GPT-2-in-C: GPT 2 implementation in pure C
16. GitHub - NJX-njx/microgpt: The most atomic GPT-2 …
17. Andrej Karpathy’s “llm.c” is Revolutionizing GPT-2 with a …
18. Andrej Karpathy Trains GPT-2 in Pure C Without PyTorch