We explore the principles of 'distillation' technology, which compresses the complex process of diffusion models generating data into just a few steps, and the background of why this technology is necessary.
Imagine you have to assemble 1,000 complex puzzle pieces. If you had to place each piece very carefully, it would take days to complete. But what if there were an ‘expert assistant’ beside you who knew the patterns of this puzzle very well? Even if you only placed a few key pieces, the expert assistant would predict the overall picture and finish the puzzle in an instant.
The process by which ‘diffusion models’ (AI models that gradually create images from random noise), which have been a hot topic in the generative AI field recently, draw images is similar to this. Behind the wonderful images we see lies the hidden effort of the AI performing tens or hundreds of iterations to remove noise and refine the image. However, this process is often too slow, which is inconvenient. The technology that appeared to solve this is ‘diffusion distillation’.
Why is this important?
AI image generation technology is increasingly aiming for higher resolution and higher quality. However, the amount of computation is increasing exponentially as a result. Previous diffusion models had to break down difficult and long tasks into countless small steps to generate complex data [Source: The paradox of diffusion distillation].
While this method produces excellent quality results, it has the fatal drawback that users have to wait too long to receive them. If you want to use AI in real-time changing videos or apps that need to react quickly, this speed issue is a task that must be solved. Distillation technology dramatically increases this speed, helping to mount AI on our daily lives much faster and lighter [Source: Latent Adversarial Diffusion Distillation].
Understanding it easily
When you hear ‘distillation,’ you might usually think of whiskey or distilled water. Distillation in AI has a similar meaning. Just as you boil the raw liquid (vast learning knowledge) contained in a large vat to extract only the core ingredients, AI distillation means “compressing complex iterative learning processes into a few shortened executions.”
To use an analogy, let’s say you are teaching a student who is learning to cook for the first time a complex recipe that spans 100 steps. At first, they have to follow every step, but once the student builds their cooking skills, they can grasp the essentials and create a wonderful dish in just 5 steps, right? Like this, the core of diffusion distillation is training based on the weights of existing models so that they can produce similar results in fewer steps [Source: GitHub - Hramchenko/diffusion_distiller].
At this time, researchers use a strategy of minimizing ‘Integral KL divergence’ (a mathematical way to calculate the difference between two probability distributions and measure how accurate the model is). Through this, they significantly reduce the number of steps in the process of generating images while maintaining the capabilities of the original model as much as possible [Source: The paradox of diffusion distillation].
How far has it come?
Diffusion distillation technology is currently being actively researched. Beyond simply reducing steps, it is evolving to a level where high-quality images can be generated with just a single-step execution [Source: [Paper Review] One-step Diffusion with Distribution Matching Distillation (DMD)]. This is a bold attempt to completely transcend the speed limitations of the existing iterative generation method.
However, like all technologies, there are limits to distillation. If you try to create something in fewer steps, there is a risk of missing very subtle details or textures that the original model had. Finding the optimal contact point between ‘speed’ and ‘quality’ is the biggest concern that engineers are currently wrestling with [Source: The paradox of diffusion distillation].
What will happen in the future?
In the future, high-quality image or video generation, which was only possible on supercomputers for professionals, will become possible on personal computers or mobile devices. If you distill a heavy model to be light and put it into a smartphone, it will become an everyday experience for AI to change the art style of the photos you take in real-time or transform them like a movie.
In short, the more ‘distillation’ technology develops, the faster AI will become, and we will be able to use the results that AI quickly draws as easily as using photo filter apps. We look forward to a new era of creation that the innovation of speed will bring.
References
- Dieleman, S. (2024). The paradox of diffusion distillation. https://sander.ai/2024/02/28/paradox.html
- Hacker News. (2024). The paradox of diffusion distillation (2024). https://news.ycombinator.com/item?id=49553830
- Sauer, A., et al. (2024). Designing Parameter and Compute Efficient Diffusion Transformers. https://arxiv.org/html/2502.14226
- Kim, D., et al. (2025). Autoregressive Distillation of Diffusion Transformers. https://openaccess.thecvf.com/content/CVPR2025/papers/Kim_Autoregressive_Distillation_of_Diffusion_Transformers_CVPR_2025_paper.pdf
- Hramchenko, A. (n.d.). diffusion_distiller: PyTorch Implementation. https://github.com/Hramchenko/diffusion_distiller
- Emergent Mind. (2024). Latent Adversarial Diffusion Distillation. https://www.emergentmind.com/papers/2403.12015
- Tamir, M. (2024). The paradox of diffusion distillation. https://www.linkedin.com/posts/miketamir_the-paradox-of-diffusion-distillation-activity-7201659030103052290-0GXd
- arXiv. (2025). A Survey on Pre-Trained Diffusion Model Distillations. https://arxiv.org/html/2502.08364
- Kim, S. (2024). The paradox of diffusion distillation by Sander Dieleman. https://www.threads.com/@sung.kim.mw/post/C36Y-ykJfmr
- Kim, J. (2023). [Paper Review] On Distillation of Guided Diffusion Models. https://kimjy99.github.io/논문리뷰/on-distillation/
- Kim, J. (2024). [Paper Review] One-step Diffusion with Distribution Matching Distillation (DMD). https://kimjy99.github.io/논문리뷰/dmd/
- Su, D., et al. (2024). D4M: Dataset Distillation via Disentangled Diffusion Model. https://openaccess.thecvf.com/content/CVPR2024/papers/Su_D4_Dataset_Distillation_via_Disentangled_Diffusion_Model_CVPR_2024_paper.pdf
- YouTube. (n.d.). LADD: Fast High-Resolution Image Synthesis with Latent… https://www.youtube.com/watch?v=9T352z1woNc
- Practical Diffusion. (2025). Schedule - 6.S183: A Practical Introduction to Diffusion Models. https://www.practical-diffusion.org/2025/schedule/
- Paper Notes. (2025). [Paper Note] Adversarial Distribution Matching for Diffusion Distillation. https://en.papernotes.org/ICCV2025/video_generation/adversarial_distribution_matching_for_diffusion_distillation_towards_efficient_i/
- Chan, A. (n.d.). Diffusion Models. https://andrewkchan.dev/posts/diffusion.html
- Generate a perfect image at once
- Break down difficult tasks into several simple denoising tasks
- Randomly synthesize existing images
- Increase AI memory
- Increase image generation speed
- Make AI larger
- Random data deletion
- Minimizing Integral KL divergence (IKL)
- Infinite hardware performance expansion