A policymaker with limited resources is trying to decide which evidence-based program to implement to improve student learning outcomes. How important is the cost? What if the available options, all with different price points, aren’t equally effective, or some are not effective at all? How can we make this decision simpler?

Policymakers have limited access to cost-effectiveness data: in a review by McEwan (2015) of 77 randomized education studies, he found that most include minimal cost data, putting decision-makers in the difficult position of having to choose between interventions without having sufficient information about what they cost. One way to support policymakers in using evidence in their decision-making is to provide them with a cost-effectiveness analysis (CEA). A CEA is a ratio of costs to impacts and can provide powerful insight into which programs might offer the greatest value for money, assuming the programs aim to achieve the same goal.

Collecting detailed cost data is crucial to conducting accurate CEAs and giving policymakers clear information to make informed decisions. However, collecting cost data for a program—especially after a research study has been completed—is highly challenging. It requires collecting information from multiple sources: program budgets, academic papers, or program reports for a description of the program structure; researchers and implementers for additional program information and costs; and public sources for components such as local wages and transportation costs. To address this challenge, IPA’s policy team has created an automated detailed cost collection tool to input costs for the duration of a project on a monthly basis. Simplifying the cost collection process makes it easier to share CEA results with policymakers, and helps policymakers compare results from different programs—even when performed in different countries, by different organizations, and in different years.
 

Cost Collection Challenges in Practice

Working with researchers in Zambia, IPA calculated a CEA for the Home-Based Growth Charts project several years after the project had been completed because the government was interested in scaling the program and wanted an estimate of its cost. Since we were calculating the CEA retrospectively, we compared project budgets to expense reports, estimated unit costs, and double-checked figures with the researchers and country office staff. We also documented all the assumptions and decisions we made to arrive at the final CEA calculation. Without an accurate CEA, Zambian policymakers may have under or overestimated the cost of the potential scale-up and possibly misallocated limited budget resources.

Retroactive cost collection is difficult, but even real-time cost collection can be challenging: it requires collaborating with partners to get cost data, which can be sensitive information. For example, partners may be hesitant to share inputs such as staff time (salaries), benefits, and other program costs.

For the Strengthening Teacher Accountability to Reach All Students (STARS) project, a government-implemented differentiated instruction project in Ghana, we wanted to collect opportunity cost data for the time spent by teachers, head teachers, and circuit supervisors on the project (that is, the time these staff would have otherwise spent on their normal activities, if not for their work on the project). To collect this data, we conducted a phone survey with questions related to monthly salary and percentage of time spent on the project per term; the survey was administered to a sample of the teachers, head teachers, and circuit supervisors who participated in the project. From the responses of the surveyed individuals, we calculated their average salaries and time spent on the project and then multiplied these averages by the total number of teachers, head teachers, and circuit supervisors involved in the project to calculate the total opportunity cost for each. Collecting this data helps us share information not only about a policymaker’s budgetary costs but also about the costs that a particular program will impose on staff working on existing programs. Opportunity costs are often a key factor that influences decision-making.

Cost collection requires gathering cost data from different partner expenditure reports, distributing costs between treatment arms, and making cost and program assumptions where data are not available. For example, different partners may be involved in delivering different components of a project, or partners may disburse funds to one another, complicating the data-gathering process.

For example, during the STARS project, which took place over the course of the 2018-2019 school year, we collected paper copies of expenditure reports submitted by the Ghana Ministry of Education during each school term of the project (three terms in total). We then manually entered each line item into our cost collection tool by month and categorized each line item into its associated cost ingredient—a time-consuming process. We had to double, if not triple, check to ensure that no errors were made in data entry.

From such experiences, we have learned the importance of working with the implementing partner from the very beginning of the project to ensure we can collect data digitally and avoid the time-consuming manual entry of cost data and the potential inaccuracies manual entry can introduce. Careful cost collection allowed us to present the Ghanaian government with an accurate picture of each component of the program’s costs so policymakers could make informed budgeting decisions when planning for the scale-up of the project. The government is now planning to roll out the STARS project to 10,000 schools across the country.

A CEA by itself may not provide enough information for a policymaker to select which program to implement. But knowing the cost of a program is a key factor in helping policymakers decide which program to choose. Better policies start with better decision-making, and better decision-making starts with better information about cost.

Visit our new CEA webpage with links to our automated cost collection tool as well as the library of CEAs IPA has conducted. You can also read this case study about how we applied these strategies to conduct a CEA on the Strengthening Accountability to Reach All Students (STARS) program with government partners in Ghana.