Research results show that even when using 'distillation'—a technique for transferring knowledge from large models to smaller ones—the political censorship characteristics of the original model are not necessarily transferred.
Imagine this: you are studying under a teacher who is incredibly intelligent but refuses to speak on certain topics or only provides biased opinions. Will a student learning under this teacher inevitably adopt the same biased way of thinking? The AI industry has been grappling with a similar question. This is the very issue surrounding the censorship controversy of the recently prominent Chinese AI model, ‘DeepSeek.’
DeepSeek has been evaluated as refusing to answer politically sensitive questions or modifying content in a way favorable to specific countriesSource: Semafor. Many developers were concerned that during the ‘distillation’ process—where DeepSeek’s vast knowledge is extracted to create small, efficient models—these censorship habits would be inherited as well. However, interesting research results that partially alleviate these concerns have recently emerged, becoming a hot topic.
Why is this important?
In the AI model development process, developers favor ‘model distillation’ technology, where a massive, high-performing model (the teacher) is built first, and then a lighter, faster small model (the student) is trained using the teacher’s answers as study materialSource: Forbes.
If the teacher model’s ‘censorship habits’ were to be transferred to the student model, developers would have to bear the enormous cost of training massive data from scratch every time to create useful AI. However, this study offers technical hope to developers looking to efficiently lightweight AI, suggesting that “censorship is not necessarily replicated.”
In simple terms: AI model distillation
Comparing AI model distillation to a school class makes it easier to understand. The large ‘teacher’ model is like an encyclopedia that has studied vast amounts of data. In contrast, the ‘student’ model is much lighter and operates efficiently.
- Distillation: This is the process of having the teacher model solve difficult problems and then training the student model on the sophisticated way the teacher answers those problemsSource: Semafor.
- Censorship Transfer: There was a concern that if the teacher avoided certain answers for political reasons, the student would do the sameSource: Semafor.
However, recent studies suggest that censorship characteristics do not necessarily carry over in this processSource: ModernOrange. In other words, even if the teacher tries to avoid providing specific information, the student model has the potential to provide more free and flexible answers in the process of learning the core of the knowledge.
Current Status: What kind of model is DeepSeek?
DeepSeek is currently classified as an ‘open weight’ modelSource: Reddit. This means that the model’s architecture and learned weights are public, allowing anyone to research or modify the model based on them.
Various derivative models utilizing DeepSeek (e.g., DeepSeek-R1-Distill-Llama, etc.) have already been created and are actively in useSource: GroqDocs. Many developers are running these models on their local computers and modifying them to suit their own purposesSource: Reddit.
What will happen in the future?
Moving forward, more developers will build efficient small models based on the knowledge of giant models. Since the possibility that distillation technology can break free from the shackles of censorship has been confirmed, it is expected that specialized AI—freer and more professional, without being confined by the bias of a specific model—will emerge more rapidlySource: ModernOrange, Source: YouTube.
MindTickleBytes’ AI Reporter Perspective
The issues of AI censorship and model distillation are truly a hot potato for developers. This research demonstrates the technical possibility that when lightweighting AI, unwanted characteristics may not necessarily be replicated. This suggests that AI is evolving beyond a tool that simply transmits knowledge, and can evolve more freely and diversely according to the developer’s intent.
References
- Exclusive: Censorship in Chinese AI models can be undone, new research shows
- Since DeepSeek is open source, can’t we just make a version without the censorship? : r/DeepSeek
- ShowHN: Distilling DeepSeek into GPT-OSS doesn’t transfer censorship. Try it
-
[Fine Tune DeepSeek R1 Build a Medical Chatbot - YouTube](https://www.youtube.com/watch?v=qcNmOItRw4U) - DeepSeek-R1-Distill-Llama-70B - GroqDocs
- Did DeepSeek Copy Off Of OpenAI? And What Is Distillation?
- A technology that teaches art to AI
- A technology that trains a small model (student) using data created by a large model (teacher)
- A technology that completely deletes AI models
- Yes, it was transferred perfectly
- No, censorship is not necessarily transferred
- It is impossible to confirm whether it was transferred
- Fully open source
- Open weight model
- Private commercial model