Did AI Do My Homework? Tracking the 'Code Provenance' in Programming Education

Imagine a student, Student A, who has to submit a computer programming assignment tonight. In the past, they might have stayed up all night rewriting code dozens of times and wrestling with errors, but now, they can simply tell an AI, “Write Python code to solve this problem,” and get a perfect result in just a few seconds.

Is this code really the product of Student A’s knowledge, or is it the AI’s? With AI generating functional code in an instant, programming education is facing a fundamental question: “How do we evaluate a student’s actual learning level and effort?” Source: Analyzing Code Provenance in LLMs for Programming Education.

Why is this important?

The essence of programming education lies more in learning the “thinking process” required to write code than in simply “creating” it. However, as AI allows students to skip this process, educators are finding themselves in a situation where they cannot tell whether submitted code truly came from a student’s mind or if it was spat out by an AI.

This directly affects the reliability of assessments. If every student uses AI to submit assignments, it becomes impossible to know who truly possesses the skills. To solve this, researchers have recently been developing “Code Provenance” tracking technology to find the origins of code Source: Analyzing Code Provenance in LLMs for Programming Education.

Understanding the basics: How AI code tracking works

So, how can we tell if AI wrote the code? Researchers use a method called the “Min-K% probability approach” Source: Analyzing Code Provenance in LLMs for Programming Education.

Simply put, it measures probabilistically how deeply embedded a piece of code is within an AI’s “memory” (training data). Think of it like this: when we transcribe a famous poem, we write it down without hesitation, but when we write something difficult we’ve never seen before, we pause. By intensively analyzing the parts where an AI model is “less surprised” when generating code, the system can determine whether the code was generated by simply memorizing the dataset or by developing logic on its own Source: Analyzing Code Provenance in LLMs for Programming Education.

The researchers added a technique called “synthetic non-member code generation” to increase accuracy. This allows for more precise tracking of a model’s code memorization by virtually creating and comparing new code that the AI is unlikely to have learned Source: Analyzing Code Provenance in LLMs for Programming Education, Source: GitHub - my625/student-code-provenance-LLMs-MinK.

Current status: “Memory” differs by model

This study analyzed various language models, ranging from free models like GPT-2 and DialoGPT to the latest commercial models like GPT-5, Gemini 2.5, and LLaMA 3.3 Source: Analyzing Code Provenance in LLMs for Programming Education.

Interestingly, free models reacted more sensitively to detecting code memorization information than commercial models Source: Analyzing Code Provenance in LLMs for Programming Education. This is because the way code is processed differs depending on the model’s scale and training method. In the process of creating massive models with millions of dollars in costs and vast GPU resources Source: Model Provenance Testing for Large Language Models - arXiv.org, the subtle balance between creativity and memory in code varies from model to model.

What’s next?

In the future, education will become more focused on proving one’s own logic while collaborating with AI, rather than simply having the ability to write code. The researchers plan to develop educational assessment methods in a direction that secures “process visibility,” showing how students interacted with AI tools to write their code, rather than just determining the origin Source: Analyzing Code Provenance in LLMs for Programming Education.

We can no longer exclude AI. Instead, the technical and educational consideration of how to respect each student’s unique learning path while keeping AI as a powerful colleague has only just begun.

MindTickleBytes AI Reporter’s View

As technology advances, the ability to prove “how you thought” will become a more important skill than “what you created.” Code provenance tracking technology may become the minimum seatbelt needed to protect the integrity of learning in the AI era.

References

1. Analyzing Code Provenance in LLMs for Programming Education (https://arxiv.org/abs/2611.09177)
2. GitHub - my625/student-code-provenance-LLMs-MinK (https://github.com/my625/student-code-provenance-LLMs-MinK)
3. Model Provenance Testing for Large Language Models - arXiv.org (https://arxiv.org/pdf/2502.00706)