Evidence Generation
Asia
Evidence Generation
Asia
User engagement data is useful to EdTech organizations because it can enable meaningful indicators related to learning outcomes. For example, early evidence suggests that time-on-task, session frequency, and completion rates can signal whether users are using the product as it is pedagogically intended, which allows organizations to improve product design. However, EdTech organizations also face challenges in managing and using large amounts of user engagement data to understand whether and how programs lead to meaningful improvements in learning. NABU, a leading EdTech reading platform, is investing in a structured and consistent approach to using their app and web data to understand how access and interaction with their educational content translates into better literacy outcomes. To support these efforts, NABU needed a clearer user engagement and impact framework, structured product priorities, and stronger data systems to connect platform engagement data to early signs of program impact on reading skills.
Through the Partnership for Tech in Education (P4T-Ed), IPA partnered with NABU to refine its theory of change, which set the foundations to develop an engagement, retention, and impact framework adapted from industry standards. IPA analyzed NABU's reading app data to validate the engagement framework, understand current user engagement and retention patterns, and identify pain points in the user journey.
IPA then conducted a data systems assessment to identify gaps, and based on the findings, designed a data system1 that takes raw platform data, cleans and structures it, and transforms it into analysis-ready indicators prioritized by Nabu for program monitoring. This data structure supports the engagement and impact framework, and organizes and expands access to platform data across NABU’s team. IPA also designed a dashboard prototype based on prioritized indicators and decision-use cases, and provided direct technical assistance to support implementation of the data architecture and dashboard.
Following the engagement, NABU has a clearer and more actionable theory of change and structured engagement and impact framework that helps the organization prioritize what to measure and when, providing actionable data about user engagement to guide product strategy and improvement.
The medallion data architecture and monitoring dashboard equip NABU's product and program teams to access and query data independently, reducing reliance on ad hoc requests to the technology team and organizing large volumes of data into structured and actionable datasets. NABU can now use the dashboard to monitor meaningful user engagement and track progress towards outcomes on an ongoing basis, connecting engagement data to program decisions in a more systematic way.
Together, these tools and systems strengthen NABU's capacity to use large volumes of engagement data to make evidence-based product decisions and generate insights on the drivers of literacy outcomes for the children it serves.
1. The design of the data infrastructure is based on industry standards of a medallion architecture that organizes data in three progressive layers: bronze, silver, and gold. Each layer cleans, organizes and aggregates the data a step further toward analysis-ready form.
