What if AI wrote text like it paints? The challenge of 'continuous diffusion' language models

Imagine this: You wake up in the morning and tell your AI assistant, “Summarize today’s meeting materials and email them to me.” While previous AI would string words together one by one according to fixed probabilities, a new type of AI begins with a blurry idea and gradually refines the sentences, much like an artist completing an increasingly sharp painting on a blank canvas. This is the future envisioned by ‘Continuous Diffusion’ language models, a hot topic in recent AI research.

Why is this technology important?

Most large language models (LLMs) we currently use, which learn from massive amounts of text data to write like humans, utilize an ‘autoregressive’ approach, generating words one by one in a fixed sequence. This is like running while looking only a step ahead, posing limitations in envisioning the big picture of an entire text at once.

Conversely, ‘diffusion models,’ which have conquered the fields of image and video generation, produce exceptional results by gradually refining data. Reference 4, Reference 9 If this approach can be successfully applied to text, it could enable writing with far more creative and logical structures than currently possible. Reference 16

Simply put: Why is text different from images?

A diffusion model is a process of gradually clearing away noise—a random state without data—from a space to find a clear image. ‘Pixel values,’ which are photo brightness or color information, consist of continuous numbers, making this process very natural. Reference 11

However, text is a completely different world. To use an analogy, the world of images is like a smooth hill, while the world of text is like a disconnected staircase. There is no intermediate value between the word ‘apple’ and ‘pear.’ Text is composed of ‘discrete tokens,’ making it very tricky to create text by smoothly clearing away noise like in images. Reference 11

To solve this, researchers use ‘embeddings’—a technique that arranges the meanings of words in a mathematical vector space—to represent text as coordinates existing in a continuous space. Reference 12 Recent research, such as ‘Riemannian Diffusion Language Models (RDLM),’ maps out how words are distributed using mathematical maps called ‘statistical manifolds’ (complex geometric spaces where data resides). By treating words like points rolling on a giant hypersphere, researchers are opening a path to handle text in a continuous manner. Reference 3, Reference 14

How far have we come?

In fact, research on text diffusion models began as early as 2022 with attempts like ‘Diffusion-LM.’ Reference 1 Unfortunately, continuous diffusion methods to date have been evaluated as having somewhat lower performance compared to models that build text word-by-word. Reference 2, Reference 15 While new models utilizing mathematical geometry are appearing one after another, bridging the gap between the ‘discreteness of language’ and the ‘continuous diffusion process’ remains a challenging hurdle at the forefront of AI research. Reference 6

What can we expect?

Going forward, it is highly likely that AI will utilize diffusion models as a ‘latent reasoner,’ where it goes beyond simply writing well to infer complex thoughts step-by-step. Reference 16, Reference 17 In the multimodal era where text and images are processed simultaneously, continuous diffusion will become a core technology for breaking down the boundaries between text, video, and images. The AI assistant you see next will have the ability to think more deeply and unfurl its thoughts more smoothly than today.

MindTickleBytes AI Reporter’s Perspective

If diffusion models can align the meaning of text just as they align the pixels of an image, we will move beyond simple sentence generation to see the AI’s thought process as a ‘process of convergence.’ This will be an important inflection point where communication between AI and humans becomes even more sophisticated.

References

1. Continuous diffusion language models – Sander Dieleman

2. [LangFlow: Continuous Diffusion Rivals Discrete Models in…

   LinkedIn](https://www.linkedin.com/posts/hangke-sui_langflow-continuous-diffusion-rivals-discrete-activity-7450571557388828674-Lv6p)

3. Continuous Diffusion Model for Language Modeling [Quick Review]
4. Advances in Continuous Diffusion Language Models - YouTube
5. Continuous Diffusion for Discrete Text
6. Continuous Diffusion Model for Language Modeling - AI for…
7. Diffusion Language Models: How a New AI Paradigm Is Challenging…
8. Simple Diffusion Language Models - YouTube
9. ELF: 임베딩 공간에 머무는 연속 확산 언어 모델(Continuous Diffusion…
10. Think In Diffusion: Continuous Latent Diffusion Language Model
11. Block Diffusion Language Models: Combining autoregression and…
12. [Revue de papier] Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models
13. Models — Google DeepMind
14. [Paper Note] Continuous Diffusion Model for Language Modeling

15. [Continuous Diffusion Model for Language Modeling

    OpenReview](https://openreview.net/forum?id=VGv5y60sXC)

16. Diffusion Reasoners: Iterative Inference Models
17. Coevolutionary Continuous Discrete Diffusion… - Microsoft Research