Type(s) of RFE services: A/B testing, Monitoring technical assistance

Organizations face a challenge in fostering a culture and building the institutional capacity for continuous learning that drives program refinement. EdTech organizations are well positioned to address this by leveraging their digital infrastructure to generate and use actionable data for decision-making that leads to improved learning outcomes. Mentu developed Shaia, a web-based platform that helps teachers be more effective and efficient in their daily work, providing AI-enabled resources that help improve classroom practices and learning outcomes. The challenge was developing a structured approach and the technological infrastructure needed to conduct rigorous A/B testing cycles to enable continuous refinement of Shaia.

Through the Partnership for Tech in Education (P4T-Ed) initiative, IPA partnered with Mentu to build the internal capacity and data systems needed to conduct rapid A/B testing cycles. The collaboration followed a phased approach, starting with the design of a Learning Roadmap to identify key learning needs and guide experimentation, followed by establishing systems and processes required to implement it.

The initial phase involved creating Shaia’s theory of change, prioritizing A/B testing ideas using a cost-effectiveness framework, and identifying gaps in Mentu's existing data infrastructure to facilitate experimentation. During the implementation of the Learning Roadmap, IPA assisted Mentu in selecting and integrating an A/B testing tool, provided guidance on data analysis and visualization and interpreting experiment results to inform evidence-based decisions.

The collaboration generated a theory of change that mapped how Shaia’s features could lead to better teaching practices and, ultimately, improve student outcomes. To track if key outcomes were occurring as expected, IPA defined a set of monitoring indicators and identified data sources within Mentu’s system that later enabled experimentation. Building on this foundation, we built a roadmap of four experiments that focused on expanding user reach, maximizing platform impact, and enhancing cost efficiency.

The integration of an A/B testing platform and the development of a monitoring dashboard equipped Mentu to rigorously design, execute, monitor, and analyze A/B tests. Through guided implementation of their first A/B test, focused on increasing teacher use of a feature in the platform, Mentu gained practical experience across the entire testing cycle and established protocols for future tests. The insights from this A/B test directly informed tangible improvements to Shaia.

Mentu now has institutionalized processes and systems to continuously test innovations, measure early outcomes related to their impact, and make evidence-based decisions. This capability enables them to maximize Shaia's cost-effectiveness by investing in enhancements that demonstrably improve outcomes for more teachers and students throughout Latin America.

Implementing Partner