A teacher uses Shaia, an AI-powered lesson planning tool, during a mathematics lesson in Bogotá, Colombia.

IPA’s Partnership for Tech in Education (P4T-Ed) initiative supports the use of data and evidence to drive learning and improvement in the edtech sector. As part of the P4T-Ed initiative, IPA and the Jacobs Foundation are supporting three randomized controlled trials (RCTs) to help generate rigorous evidence on edtech interventions’ potential impact on learning outcomes. 

This is the second blog post in a series highlighting key findings, insights, and lessons learned from the RCTs. The series showcases how evidence is helping bridge the gap between innovation, implementation, and impact in edtech. 

Funded by the Jacobs Foundation and the Bogotá Education Secretariat (SED) and conducted in collaboration with Mentu Labs and the Shaia Foundation, IPA is evaluating whether Shaia, an AI-powered lesson planning tool, helps teachers deliver more effective math instruction and improves student learning. 

Juan Muñoz-Morales, professor at IÉSEG School of Management, and Felipe Barrera-Osorio, professor at Vanderbilt University, are leading the research. 

Every morning in Bogotá, thousands of students walk into math classrooms carrying very different expectations. For some, mathematics represents opportunity and future stability. For others, it is a subject that has become progressively harder and more discouraging over time.

The challenge is visible in national exam results: 38 percent of students fail to reach high performance levels in mathematics. For education policymakers and teachers alike, the question is pressing: what can be done differently to support learning?

Artificial Intelligence (AI) is one active area of research. Not as a replacement for teachers, but as a tool to strengthen an often invisible part of teaching: lesson planning.

Funded by the Jacobs Foundation, and in collaboration with Mentu Labs, in partnership with Shaia Foundation and Bogotá’s Education Secretariat (SED), IPA is evaluating whether Shaia, an AI-based lesson planning tool, can help teachers deliver stronger math instruction and whether this translates into better student learning.

Where Students and Teachers Started

To understand whether AI can make a difference, it is essential to know where students and teachers began. 

The study follows 72 public schools, covering grades 6 through 9, and was designed as a randomized evaluation. Some schools received access to the AI-powered Shaia tool, while others continued with their standard approach.

At baseline, there were no meaningful differences between the two groups. Students in schools assigned to receive the tool scored an average of 31.8 percent correct answers on a standardized math assessment, compared to 32.1 percent in control schools. Teachers also shared similar profiles: on average, 20 years of teaching experience, most holding postgraduate degrees or specializations in mathematics.

This similarity matters. It means that any patterns observed later are less likely to reflect pre-existing differences and more likely to reflect what happens during implementation.

The baseline also revealed persistent inequalities. Migrant students consistently underperformed, and among girls, math performance declined steadily as grade levels increased. Yet students’ attitudes were broadly positive: most believed math was important for their future and associated success with effort rather than innate ability.

The problem, it appeared, was not motivation, but access to support.

Experienced Teachers, Limited Resources

One of the study’s more striking findings challenges common narratives about underqualified teachers. Bogotá’s public-school math teachers are overwhelmingly experienced and well trained.

What they often lack are high-quality digital resources and institutional support for integrating technology. Many teachers reported only a basic understanding of AI, even as over 84 percent of their students reported using AI tools independently, primarily for studying and research.

Baseline data also identified why lesson planning matters specifically. Student performance was positively correlated with both teacher experience and time spent planning lessons. If AI could meaningfully support teachers in this process—saving time or improving structure—it might help amplify existing strengths rather than attempt to compensate for weaknesses.

What the Midline Data Show

Midline data, collected halfway through implementation, offer early signals worth noting, though not yet definitive conclusions.

Students in schools where teachers had access to the Shaia tool scored an average of 34.8 percent, compared to 31.6 percent in non-exposed schools. Schools that received the tool improved by nearly 3 percentage points over the baseline period, while schools that did not receive the tool saw a slight decline.

Further analyses of the midline data have also reported positive results, showing that students in treatment schools improved between 0.22 and 0.46 standard deviations more than students in comparison schools, depending on the analytical approach used. These effect sizes are considered meaningful relative to many rigorously evaluated education interventions. However, because these findings are based on midline data with a smaller and selected sample, they should be interpreted with appropriate caution. While they are consistent with the encouraging descriptive patterns presented here, the endline analysis will provide the most comprehensive assessment of whether these differences are sustained over time and can be robustly attributed to the intervention.

The most notable pattern involves gender. In schools that did not receive the tool, boys outperformed girls by an average of 2.09 percentage points. In schools where teachers used the Shaia AI tool, that gap narrowed to 1.26 percentage points.

The evidence suggests that something different may be happening in classrooms where teachers have access to AI-supported planning—even if the mechanism is not yet fully visible.

Practice Stability, Perception Shifts

Interestingly, teachers’ reported classroom practices did not change dramatically at midline. Time allocated to lesson planning, classroom organization, and instructional structure remained largely stable across groups. What did change was how teachers perceived AI.

Teachers exposed to the Shaia tool were more likely to describe AI as useful, particularly for lesson preparation and saving time. Most reported actively using the tool, though some cited barriers such as limited time or insufficient training.

This distinction matters. Technology can be introduced quickly, but changes in teaching practice develop more gradually. Early adoption appears to shift attitudes first, with practice changes emerging more gradually. Continued follow-up will help determine whether increased familiarity with AI tools translates into longer-term changes in instructional approaches.

Access Is Not Enough

One of the clearest midline insights is that access alone is not enough. Schools where teachers used the tool more intensively and participated more actively in training showed stronger student performance patterns. The implication is direct: AI-supported tools appear most effective when integrated into daily practice, rather than used occasionally.

In this sense, AI strengthens what is already there, especially when experienced teachers engage with it consistently.

Students in Bogotá are already digitally connected. The challenge is not introducing technology into schools, but using it in ways that meaningfully support teaching and learning. Early evidence suggests that AI-supported lesson planning may help, particularly for reducing gender-based performance gaps.

Sustained impact will depend on institutional commitment, consistent teacher engagement, and ongoing support structures.

The final evaluation, scheduled for 2026, will determine whether these early trends translate into lasting learning gains. For now, the message is clear: artificial intelligence does not replace teachers. But when used thoughtfully, it may become a powerful ally in helping students overcome persistent learning gaps, one lesson plan at a time.