AI Develops 'Bias' Toward Strangers? The Secret of Discrimination Without Training

Imagine this: You have become a hiring manager at a new company. As you evaluate candidates, you notice that at some point, you develop a habit of assigning only easy tasks to people from a certain group and only difficult tasks to others. However, the surprising fact is that you have never heard any bad things about those candidate groups, nor have you ever been trained to discriminate against them. You were simply following a method that happened to succeed a few times while working, and before you knew it, you had become a person with biases without even realizing it.

Recently, similar chilling research findings were announced in the field of Artificial Intelligence (AI). It turns out that Large Language Models (LLMs, AI structures that grasp relationships between words in sentences) have begun to autonomously create ‘biases’ against new, virtual groups about which they have no information [Source 8].

Why is this important?

AI is no longer just a simple chatbot. It is establishing itself as a practical decision-maker that directly impacts human life, including hiring, loan approvals, and legal judgments [Source 2, Source 3].

What if, even if we clean up only the existing data to eliminate AI bias, the AI autonomously cooks up new biases while it works? This research warns that the methods we currently use to ‘remove’ bias from AI are insufficient [Source 8, Source 11]. In particular, these biases tend to become more severe as technology advances and the scale of AI models increases [Source 8].

Easy to understand: Misinterpreting AI’s ‘Success Formula’

Let me explain using an analogy. AI is like a very competent and diligent new employee. This employee wants to learn quickly and has a habit of recording successful experiences as a formula.

Let’s assume that one day, when the AI happened to pick a candidate from ‘Group A’, it luckily achieved good results. The AI stores this as a formula: ‘Group A is competent.’ Conversely, if an error occurred while picking a candidate from ‘Group B’, it learns that ‘Group B is incompetent.’ This happens even though there was actually no difference in ability between Group A and Group B.

The researchers had the AI repeatedly make decisions using methods taken from psychological literature [Source 8]. The results were shocking. Even though the AI had received no prior training, it autonomously learned from coincidental outcomes (spurious outcomes) and created results that discriminated against specific groups. The frequency of this bias formation was even higher than that of humans [Source 10]. It is as if the AI is picking up the habit of ‘bad bias’ faster than humans as it learns about the world.

Current situation: The limitations of data cleansing

Currently, many companies and research institutes are focusing on removing existing racial and gender biases mixed into AI training data. However, this study warns that “making data clean is not enough to solve the problem” [Source 2].

This phenomenon has already been confirmed in several latest AI models [Source 8]. AI is not just imitating the given data; it is ‘adaptively’ expanding its knowledge by interacting with the environment. In that process, it is creating biases unintentionally [Source 7].

What will happen in the future?

As AI’s decision-making capabilities become more advanced, we may have to fight against ‘moving biases’ rather than ‘fixed biases.’ Future research is expected to focus on how to manage AI’s decision-making ‘algorithms’ themselves more fairly, going beyond simply correcting data to prevent biases that arise during the learning process [Source 6]. The era has arrived where the smarter we make AI, the more carefully we must also observe its habits.

MindTickleBytes AI Reporter’s Perspective

Bias may be an ‘inevitable byproduct’ that occurs when AI learns something. As long as AI learns about the world in real-time, the battle against bias will be a never-ending task. Now that technology has gone beyond being a tool and has become a subject of judgment, we need a system that transparently monitors the ‘process’ of reaching a conclusion as much as the results of the AI.

References

1. arXiv:2511.06148v4 - Large Language Models Develop Novel Social Biases Through Adaptive Exploration
2. ICML Virtual - Large Language Models Develop Novel Social Biases Through Adaptive Exploration
3. Papers with Code - Large Language Models Develop Novel Social Biases Through Adaptive Exploration
4. Hugging Face Space - Reproduction of LLM Social Bias Research
5. J-GLOBAL - Research Detail
6. HR Executive - AI hiring tools can invent their own bias, research finds
7. Princeton Computational Cognitive Science Lab - Publications
8. arXiv - Large Language Models Develop Novel Social Biases Through Adaptive Exploration (Abstract/Details)
9. OpenReview - Discussion for ICML Oral Paper
10. SAI Science - Paper and Code Review