Why Do AI Chatbots Only Draw Pelicans Riding Bicycles? Uncovering the ‘Diversity Collapse’ Phenomenon

Imagine this: You have asked a high-performing AI chatbot to create an image. Not a generic picture of a cat or a stunning landscape, but something a bit whimsical—a “pelican riding a bicycle.” You would surely expect the AI to showcase a variety of colorful and imaginative images that would surprise us. Instead, the results are identical: the same-looking pelican and the exact same bicycle. It is a situation where it feels like every AI is churning out drawings from the same factory.

Recently, something called the “Pelican Test” has been gaining attention in the field of Artificial Intelligence (AI). It is a type of benchmark used to evaluate the performance of Large Language Models (LLMs), specifically how well they follow instructions and generate creative results. However, a surprising fact is emerging from this “pelican riding a bicycle” test: AIs are increasingly generating only similar results. This is the reason experts are calling it “LLM diversity collapse.” Source: The Emperor Pelican Has No Clothes - arcfu.com Source: Why the Pelican Test Worked and Why It Is Done - LLM …

Why Is This Important?

As AI technology embeds itself deeper into our lives, how diverse and creative AI outputs are has become a crucial question. What happens if AI always thinks in the same way and produces identical results? This implies that the creativity of AI is limited and that its ability to find solutions to unexpected problems or suggest new ideas is diminished.

The true value of AI lies not just in processing information quickly and accurately, but in expanding human thinking and suggesting new perspectives. To use an analogy, it is as if every chef in the world used only one single recipe for every dish. The diversity and creativity of culinary art would vanish, and our sensory experience would become extremely monotonous. Imagine if an AI suggested only one “optimal” path in medical diagnostics, legal advice, or complex scientific research without exploring other possibilities; we might miss out on chances for diagnosing rare diseases or making revolutionary scientific discoveries.

In fact, after researchers analyzed over 13 million labels from 18 LLMs, they discovered that ‘it is possible to make almost any hypothesis statistically significant.’ This suggests that changing LLM settings even slightly can lead to vastly different results; conversely, it also means that highly uniform results can be easily derived depending on specific settings. Source: LLMHacking: 6 Strategies to Avoid False Findings in Your Research. The fact that LLMs produce uniform results raises suspicions that AI is stopping at the repetition of learned patterns rather than achieving true ‘understanding.’

What Is the Problem with ‘Pelicans Riding Bicycles’?

The “pelican riding a bicycle” prompt was designed to comprehensively test an LLM’s image generation capabilities and how well it follows complex instructions. Source: Why the Pelican Test Worked and Why It Is Done - LLM … Source: I tested the top local LLMs against ChatGPT and Claude … - MUO To pass this test, several criteria must be met. First, the generated image must be recognizable as a “pelican.” Also, the bicycle must be properly equipped with wheels and a frame, and most importantly, the pelican must actually be in a position of “riding” the bicycle. Source: I tested the top local LLMs against ChatGPT and Claude … - MUO

In the past, each LLM generated unique and diverse pelican drawings in response to this prompt. Some pelicans wore helmets, some sported mustaches, and the types of bicycles varied as well. This was evidence that LLMs had excellent capabilities in interpreting prompts and implementing them creatively. Source: Simon Willison on pelican-riding-a-bicycle

However, in recent months, the “pelican riding a bicycle” drawings produced by LLMs have become strikingly similar. Source: The last six months in LLMs, illustrated by pelicans on bicycles Source: The Emperor Pelican Has No Clothes - arcfu.com As mentioned earlier, this is the “diversity collapse” phenomenon of LLMs.

The Current Situation: Standardized Pelicans

This standardization raises fundamental questions about how LLMs are developed and evaluated. There is analysis suggesting that while AI models might show better performance in certain settings, such as “high reasoning,” this can conversely become a factor that harms output diversity. Source: Simon Willison on llm-reasoning This is because when a model runs in a “high reasoning” mode, its tendency to derive the most “safe and verified” result for a specific prompt can become stronger. Consequently, it generates only the most common and predictable images, excluding original or unexpected interpretations. Source: 2025 in LLMs so far - ai.engineer

This raises suspicions that AI is not truly “understanding” or “thinking,” but merely mimicking patterns that appear most frequently in training data. If an LLM repeats only the same appearance, it suggests there are limits to AI’s role as a tool to stimulate our imagination and open up new possibilities.

What Will Happen in the Future?

The “diversity collapse” phenomenon of LLMs clearly demonstrates the dual nature of AI technology advancement. On one hand, AI is advancing at a staggering speed, nearing human intelligence; on the other hand, as results become standardized, it poses questions about the uniquely human domain of creativity.

Moving forward, AI developers will need to focus not just on improving performance, but also on research into how diverse and creative AI outputs can be. Also, as shown by the “pelican riding a bicycle” test, we must remember that the very method of evaluating AI can affect the diversity of its results. As mentioned earlier, research published by Bauman et al. warns of the risk that current evaluation methods are focused more on finding settings advantageous for deriving specific results than on measuring AI’s true capabilities. Source: LLMHacking: 6 Strategies to Avoid False Findings in Your Research.

For AI to truly become a tool that complements and expands human intelligence, it must overcome the trap of uniform results and evolve in a direction that embraces a wider range of creativity and unpredictability. The standardization shown by pelicans riding bicycles demands important reflection on the path AI should take.

References

1. Why the Pelican Test Worked and Why It Is Done - LLM … - https://braindetox.kr/en/posts/llm_benchmark_pelican_test_2026.html
2. The last six months in LLMs, illustrated by pelicans on bicycles - https://simonwillison.net/2025/Jun/6/six-months-in-llms/
3. 2025 in LLMs so far - ai.engineer - https://ai.engineer/talks/YpY83-kA7Bo-llm-pelican-benchmark
4. I tested the top local LLMs against ChatGPT and Claude … - MUO - https://www.makeuseof.com/tested-the-best-local-llms-against-claude-and-chatgpt-with-one-a-pelican/
5. The Emperor Pelican Has No Clothes - arcfu.com - https://arcfu.com/post/the-emperor-pelican-has-no-clothes/
6. Simon Willison on pelican-riding-a-bicycle - https://simonwillison.net/tags/pelican-riding-a-bicycle/
7. LLM- Detailed dive into LLM w Andrej Karpathy… - Obsidian Publish - https://publish.obsidian.md/followtheidea/Content/AI/LLM+-+Detailed+dive+into+LLM+w+Andrej+Karpathy
8. LLMHacking: 6 Strategies to Avoid False Findings in Your Research. - https://www.linkedin.com/pulse/llm-hacking-6-strategies-avoid-false-findings-your-mikel-k-ngueajio-nogqf
9. Simon Willison on llm-reasoning - https://simonwillison.net/tags/llm-reasoning/