Learn how to maximize AI learning efficiency and performance through the 'NGU' learning technique, which forces AI models to repeatedly attempt hard problems during training until an answer is found.
Imagine you are doing math homework, but the problem is so difficult that you can’t get close to the answer even after trying 100 times. The teacher doesn’t show you the answer and only says, “Keep thinking about it.” If this situation continues, you would probably want to give up on the homework.
Surprisingly, AI models often find themselves in the same situation. When using ‘Reinforcement Learning’ (a method of training models through rewards), which is one of the ways AI learns new knowledge, if a problem is too difficult, the AI never finds the correct answer even once. Since it has never seen the correct answer, it has no way to learn what it did right. Recently, a ‘Never Give Up’ learning method has emerged to solve this problem.
Why Is It Important?
To make the AI chatbots we use better at logical reasoning or complex coding tasks, AI also needs the experience of solving ‘difficult problems’ on its own, just like humans. However, with current reinforcement learning methods, AI often became frustrated (zero probability of correct answers) as soon as the problem level increased slightly [Source: POPE: Learning to Reason on Hard Problems].
This research is designed to encourage AI to persistently challenge itself until it finds the correct answer. This is an important advancement that goes beyond simply increasing AI’s intelligence, allowing us to entrust AI with more complex and important tasks in our daily lives.
Understanding It Simply
To understand this learning method, let’s explain two core approaches using metaphors.
The first is the ‘NGU (Never Give Up)’ learning method. Simply put, it is a system that makes the AI persistently attempt to solve a difficult problem multiple times until it finds the correct answer [Source: Learning to Solve Hard Problems in RL for LLMs by Never Giving Up]. For example, while easy problems can be solved in one or two tries, difficult problems might need dozens of attempts to even get close to the correct answer. NGU helps the AI quickly pass over easy problems while concentrating more computational resources on difficult problems until it gets the right answer [Source: Learning to Solve Hard Problems in RL for LLMs by Never Giving Up].
The second is the ‘ReGFT (Reference-Guided Fine-Tuning)’ method. This is like a math teacher solving 80% of a problem and guiding the student (AI) to think and solve the rest on their own, instead of showing them the whole answer [Source: Learn Hard Problems During RL with Reference Guided Fine-tuning]. Based on the hints provided, the AI uses its own logic to reach the final answer, and through this process, it develops ‘thinking muscles’ to solve difficult problems on its own [Source: How to Explore to Scale RL Training of LLMs on Hard Problems?].
Current Situation
In the AI industry today, reinforcement learning is actively used in fields where there is a ‘clear correct answer,’ such as math problems or programming code [Source: How I Learned RL for LLMs]. However, with the emergence of techniques like NGU and ReGFT, an environment is being created where AI can learn on its own even for creative writing or complex decision-making problems where there isn’t a single clear answer [Source: Learning to Solve Hard Problems in RL for LLMs by Never Giving Up].
However, the fact that training costs may increase as AI concentrates computational resources to solve difficult problems is a challenge that must be addressed in the future [Source: Learning to Solve Hard Problems in RL for LLMs by Never Giving Up].
What Will Happen in the Future?
In the future, the era of ‘thinking AI’—where AI goes beyond simply memorizing data to strategizing on its own and learning through repeated failure—is expected to accelerate. In particular, the ability to solve high-difficulty problems while minimizing human assistance (hints) will be greatly strengthened [Source: Learning to Solve Hard Problems in RL for LLMs by Never Giving Up]. The AI you meet in the future may be a little more persistent and a little more logical than it was yesterday.
AI’s Perspective
MindTickleBytes AI Reporter’s Perspective: “It suggests that it is much more valuable to ‘practice the process of finding the answer’ than to ‘give the AI the answer.’ Human education and AI learning are ultimately heading in the same direction.”
References
- Learning to Solve Hard Problems in RL for LLMs by Never Giving Up
- Learning to Solve Hard Problems in RL for LLMs by Never Giving Up (HTML version)
- How to Explore to Scale RL Training of LLMs on Hard Problems?
- Learn Hard Problems During RL with Reference Guided Fine-tuning
- How I Learned RL for LLMs: A Researcher’s Detour in Five Parts
- Learning to Solve Hard Problems in RL for LLMs by Never Giving Up (HuggingFace)
- Learning to Solve Hard Problems in RL for LLMs by Never Giving Up (AlphaXiv)
- POPE: Learning to Reason on Hard Problems
- Repeated sampling until the correct answer is found
- Human inputting all correct answers
- Doubling the model size
- Training costs are too low
- There is too much ground truth data
- There is no learning signal because it never sees a correct result
- Showing the entire correct answer as is
- Providing only a partial answer (hint) so the AI solves the rest
- Making the AI memorize the correct answer