IN BRIEF
More than 6,000 middle-school students were randomly assigned to combinations of mathematics practice, AI tutoring and mastery requirements.
AI-supported students moved through fewer problems, but after mistakes they were more likely to improve on subsequent attempts.
The most promising delayed-learning signal appeared when AI was combined with mastery structure, although that evidence remained limited.
WHY IT MATTERS
AI's educational value may depend less on generating immediate answers than on how it is integrated into instructional design.
The experiment suggests that making students slow down, address errors and reach mastery before advancing may matter more than simply giving them access to an AI assistant.
WHAT MADE IT POSSIBLE
The tutor delivered structured assistance after incorrect answers, supporting students before their next attempt.
Researchers separately tested AI support and a mastery rule, then evaluated what happened when the two were combined.
THE STORY
During practice, AI-supported students completed fewer questions but showed better performance on attempts following an error.
On an assessment roughly one week later, the AI-plus-mastery condition produced the most encouraging result. The gain was marginally significant and concentrated on material students had actually practiced.
WHAT WE KNOW
Random assignment provides more informative causal evidence than observational studies of students who choose to use AI tools. [1][2]
The clearest effect occurred in the learning process immediately after mistakes: students received structured help and were more likely to correct the next attempt.
WHAT WE DON'T KNOW
The working paper has not yet been peer reviewed.
Exposure was brief, so the experiment cannot establish whether effects persist through sustained classroom use.
There is not yet strong evidence for broad transfer to mathematics content students did not practice.
WHAT HAPPENS NEXT
Longer trials can test whether the pattern persists when AI tutoring is embedded in normal curricula for weeks or months.
Future work can disentangle how much of the benefit comes from the AI itself and how much comes from the pedagogical structure governing its use.
SOURCE TRACE
The researchers ran a randomized field experiment with more than 6,000 middle-school students in Hamilton County Schools, Tennessee.
Students receiving AI support progressed through fewer questions but were more accurate conditional on reaching an attempt.
After a mistake, AI support improved the probability of answering the next attempt correctly and reduced the number of attempts needed to return to a correct answer.
The strongest delayed-learning evidence appeared when AI support was embedded in the mastery workflow, with marginally significant gains concentrated on practiced material.
Specialist reporting on the experiment describes a brief intervention and a delayed assessment about one week later, underscoring that the study does not establish long-term classroom effects.
SOURCES
- Making AI Tutoring Productive: Evidence from a Mastery-Based Math Practice ExperimentOpen source
- Slow math: Kids may learn more when AI makes them review mistakesOpen source
CURRENT EVIDENCE
EARLY EVIDENCE
Randomized field experiment with more than 6,000 middle-school students; working paper, not yet peer reviewed; delayed assessment approximately one week after practice.