IN BRIEF
The study evaluated a course-specific AI assistant across seven university course sections in Chile.
Simply encouraging students to use AI increased reported adoption but did not improve midterm performance.
Guidance that steered use toward tutor-style engagement improved final-exam performance by 0.21 standard deviations.
WHY IT MATTERS
The result separates two questions that are often conflated: giving students access to AI and designing how they use it do not necessarily produce the same educational effect.
If replicated, it would suggest that value depends more on interaction design than on tool availability alone.
WHAT MADE IT POSSIBLE
The assistant was course-specific and supported learning interactions within the academic setting.
The effective intervention did more than provide access: it guided students toward tutor-like engagement.
THE STORY
The researcher ran two randomized experiments across seven course sections with real academic outcomes.
In the first intervention, higher awareness and reported AI use did not improve midterm performance.
In the second, explicit guidance toward tutor-style interaction increased perceived usefulness by 0.38 standard deviations and final-exam performance by 0.21 standard deviations.
The evidence is promising but preliminary: it is a working paper from one selective university context.
WHAT WE KNOW
Both evaluations were randomized and embedded in real university courses. [1]
Simple encouragement to adopt AI did not improve the measured midterm outcome.
Tutor-style guidance was associated with a 0.21-standard-deviation improvement in final-exam performance.
WHAT WE DON'T KNOW
The paper has not yet been peer reviewed.
It is unclear how well the result generalizes to other universities, disciplines or student populations.
The accessible abstract does not provide enough detail to reconstruct every statistical specification or heterogeneous treatment effect.
WHAT HAPPENS NEXT
Peer review and independent replication.
Testing in different institutions and disciplines.
Experiments that isolate which guidance patterns produce durable learning rather than short-term performance.
SOURCE TRACE
The study deployed a course-specific AI assistant and evaluated it with two randomized controlled trials across seven course sections at a selective university in Chile.
An encouragement intervention increased awareness and reported AI use but did not improve midterm performance.
A guidance intervention shifted interactions toward tutor-style engagement, increased perceived usefulness by 0.38 standard deviations and improved final-exam performance by 0.21 standard deviations.
SOURCES
- Guidance Over Adoption: Experimental Evidence on AI-Assisted Learning in Higher EducationOpen source
CURRENT EVIDENCE
EARLY EVIDENCE
randomized field experiments with real course outcomes; working paper, not peer reviewed
WHAT WOULD CHANGE THIS?
- If preregistered replications at other universities find no exam advantage or show that the effect disappears over longer follow-up, the positive interpretation should be downgraded.