Splash: Finding the Right Timing for Impact Evaluation

Lack of safe drinking water, inadequate sanitation, and poor hygiene contribute to poor health outcomes for children in many countries across the world. Based in Seattle, WA, the NGO Splash provides safe water and hygiene education to approximately 290,000 children in schools and orphanages in urban areas of Asia and Africa. Splash provides urban institutions with water purification systems, teaches children about hygiene, and also works to make safe water provision sustainable by working with local partners to scale up its programs. The program started in 2007 in Cambodia, China, and Nepal and has since expanded to Ethiopia, India, and Thailand.

Splash is eager to learn about the impact of its program on the health and development of schoolchildren through a rigorous impact evaluation. Before an organization can measure its impact, however, it is important it has a strong monitoring and evaluation system in place, and in 2015, when this analysis was completed, Splash’s M&E system was still under development. In these early stages, we recommend that organizations understand whether their program is operating as planned. This means developing a strong framework to track implementation—which enables organizations to show accountability to donors and quickly identify and resolve issues. Given the technical and organizational requirements for carrying out a high-quality randomized evaluation, we recommend that Splash further develop its monitoring system before conducting an impact evaluation.

If the organization decides to pursue an impact evaluation, the next step would be to use its monitoring data to identify the most important impact or operational questions to research.

Lessons for Others

1. Monitoring systems should focus on validating key elements of the theory of change.

In particular, monitoring data on take up and engagement can help inform ongoing program design.

2. Develop a credible, actionable monitoring system before you evaluate.

In the early phase of building a monitoring system, an organization is often not well placed to conduct a rigorous impact evaluation. An evaluation is likely to distract from the development and refinement of critical monitoring tasks.

3. Credible data can position an organization for a well-designed impact evaluation.

The first goal of an M&E system is to gather credible data on program implementation. When and if the time is right for impact evaluation, this data can inform the evaluation design and help pinpoint the most relevant research questions

4. Review existing evidence first.

Before embarking on an impact evaluation, organizations should first look at literature that is relevant to their program. Evidence may already exist on similar programs that may enable them to estimate their impact.