NEW YORK — What happened: A consortium of educational researchers released a two-year longitudinal study tracking four thousand high school students using an interactive, chain-of-thought mathematical reasoning tutor powered by a distilled open-weight language model.
Why it matters: Educational technology is flooded with unvetted claims about personalized learning. This independent study provides rigorous, controlled data on where LLM-based tutors actually help versus where they introduce pedagogical errors.
Deep dive: The study compared classrooms using the AI tutor alongside human teachers against traditional lecture-based instruction. Results showed a statistically significant 14 percent improvement in standardized algebra test scores for students in the bottom third of the class.
Report check (claims vs what is verified vs still rumor): Software vendors claimed the AI could independently foster critical thinking across all STEM disciplines. Independent auditors verified gains in procedural algebra, but noted persistent hallucinations when students asked open-ended questions about geometric proofs.
Open questions: Does reliance on step-by-step AI hints diminish a student's ability to engage in productive struggle and independent problem-solving over the long term?