20,000 tokens per second on a $250 FPGA? The truth behind this amazing experiment

Imagine if you could use an artificial intelligence on a single small device at home that reads and writes text hundreds of times faster than the conversational AIs we commonly use. When we think of ‘Artificial Intelligence (AI),’ we usually first think of NVIDIA’s high-performance GPUs (Graphics Processing Units) that cost hundreds of thousands of dollars. However, recently, interesting experimental results that shatter this conventional wisdom are pouring out from developers.

Recently, a developer used a Field Programmable Gate Array (FPGA) board—a semiconductor chip whose logic circuits can be programmed in the field—costing only $250 to run a language model, recording a speed of over 21,000 tokens (text fragments) per second. Source 1, Source 8 This is a surprising figure that makes one doubt their eyes even when compared to existing high-priced equipment. How on earth is this possible?

Why is this important?

Until now, AI technology has developed in the direction of demanding ‘larger, more computations.’ Because of this, massive power and expensive hardware have been essential to run Large Language Models (LLMs). However, this experiment poses the fundamental question: “Must AI always run on expensive equipment?”

If sufficiently fast AI inference is possible on ultra-low-power, low-cost hardware, the story changes completely. This is because we will be able to use AI assistants inside the home appliances, cars, and various wearable devices we use without having to send personal information to external servers, entirely in an ‘offline’ state. This will significantly increase the accessibility of AI technology and become a new breakthrough that resolves data security issues. Source 5, Source 11

In simple terms (an analogy)

Why are special semiconductors like FPGAs faster and more efficient than existing GPUs? Let’s use a library as an analogy.

Running a giant model on a GPU is like keeping books (model data) in a distant warehouse (external memory) in the library and having a librarian (data path) fetch the books whenever needed. This ‘memory bottleneck,’ where fetching the books takes longer than reading them, is the main culprit holding back modern AI performance. Source 5

On the other hand, the FPGA-based model used in this experiment adopted a method of spreading all the books out on the desk in advance (storing model weights directly inside the chip). Source 5, Source 11 Since data doesn’t need to be moved, the speed is incredibly fast, and there is almost no power wasted moving data. In fact, the ‘TerEffic’ architecture proposed by the research team reportedly shows 19 times higher power efficiency than existing equipment. Source 10, Source 13

How far have we come?

Surprising records are already appearing one after another in the field.

• High-Speed FPGA Experiments: Speeds of 21,000 tokens per second were measured in a $250 FPGA environment, a figure stable enough to handle 2,000 simultaneous users without performance degradation. Source 1, Source 15
• Ultra-Low-Cost Microcontrollers: It has even been confirmed that small language models run at a speed of about 10 tokens per second on a $10 microcontroller. Source 2, Source 7
• Extreme Efficiency: Cases of models working completely offline on an $8 ESP32-S3 chip (512KB RAM) have been reported. Source 4

Of course, there are clear limitations. These small models lack the high intelligence to answer complex questions or write advanced code, and are optimized for short sentence generation or simple classification tasks. Source 7

What can we expect?

We are now facing an era of AI that lives and breathes inside a tiny chip in our pockets, not AI located in giant server rooms. Researchers are working to implement smarter AI on smaller devices by introducing more efficient computation methods (such as ternary operations). Source 11, Source 13 In the near future, smart home appliances that perfectly understand our voices and react immediately without an internet connection will become a part of everyday life.

AI Opinion

In an AI market dominated by giant models, a paradigm shift toward ‘small and efficient’ hardware optimization is underway. This is a significant technical milestone that will accelerate the democratization of AI. If attempts to optimize algorithms according to the characteristics of hardware continue—moving away from the method of pouring power into performance blindly—AI will permeate our lives more quickly and lightly.

References

1. [Taalas-Style On-Chip Weights on a $250 FPGA: a Language Model at 60k tok/s

   Michael Ayles](https://www.mikeayles.com/blog/on-chip-llm-kv260/)

2. Dev proves LLMs will run on anything – even a $10 microcontroller

3. [Token Generation Speed Visualizer

   LLM Performance Demo](https://shir-man.com/tokens-per-second/)

4. How This Tiny $8 Chip Runs an LLM With Almost No RAM - YouTube
5. r/AIToolsPerformance on Reddit: Karpathy’s MicroGPT hits 50,000 tok/s on FPGA
6. LLM Token Generation Speed Simulator & Benchmark

7. [The next age of LLMs? Dev gets a small LLM running at 10 tokens a second locally on a $10 microcontroller

   TechRadar](https://www.techradar.com/pro/the-next-age-of-llms-dev-gets-a-small-llm-running-at-10-tokens-a-second-locally-on-a-usd10-microcontroller)

8. [Nuxt HN

   Show HN: A tiny LLM running at 21,000 tok/s](https://hn.nuxt.dev/item/49242475)

9. An LLM Writes Shakespeare on an FPGA — and We … - LinkedIn
10. Researchers Deliver Dramatic Performance, Efficiency Gains for LLMs with the FPGA-Driven TerEffic
11. Can an FPGA Actually Run a Tiny LLM? (Part 1: Memory Wall)
12. NLnet; LLM2FPGA
13. TerEffic: Highly Efficient Ternary LLM Inference on FPGA
14. FPGA-Accelerated Large Language Models Used for ChatGPT
15. ShowHN: A tiny LLM running at 21,000 tok/s on a $250 FPGA