Abstract

Previous research suggests having community members monitor health service providers can improve the delivery of health services, and greatly improve child health as a result. In Uganda, researchers conducted a large-scale randomized evaluation of a program called Accountability Can Transform (ACT) Health that followed this model. It provided community members and health care workers information about the quality of their local health services and brought them together to create action plans for how to improve local health service accountability, delivery, and quality. The evaluation found that the program marginally improved the quality of treatment patients received and increased patient satisfaction twenty months after the program began. However, the program did not affect how often people sought health care or improve health outcomes. There is also no evidence that the program increased monitoring or bottom-up pressure by community members.

Policy Issue

According to the World Health Organization, the the under-five mortality rate in low-income countries in 2015 was 76 deaths per 1000 live births, which was about 11 times the average rate in high-income countries.¹ In sub-Saharan Africa, more than half of these deaths were caused by diseases that could have been prevented or treated if the children had had access to a small number of proven, inexpensive services.² Improving local access to these services is therefore a priority for the region.

To address these issues across the region, the development sector has embraced a potentially promising approach: the bottom-up monitoring of service providers by community members. The idea is that providing citizens with information about service delivery shortfalls—along with information allowing them to compare local outcomes with national standards and with outcomes in other communities—will put them in a position to monitor and apply pressure on underperforming service providers. A 2009 study of a program that followed this model, called Power to the People (P2P), validated this approach. It generated striking results: infant weight increased, under-5 mortality declined by 33 percent, immunization rates rose, waiting times at clinics fell, staff absenteeism dropped, utilization increased, and communities became more engaged and monitored clinics more extensively.³ Yet, in other research, the effectiveness of information provision and citizen monitoring has been mixed.

Context of the Evaluation

The study took place in 16 districts across four regions of Uganda. While Uganda has seen a decline in child, infant, and neonatal mortality rates in recent years—under-five mortality rates were cut in half between 2006 and 2016— the country still falls in the bottom quartile of the world distribution on most basic health measures. Since 2001, public health services in Uganda have been free of charge. However, Ugandan health staff are underpaid and facilities are poorly funded. Resource management tends to be weak, resulting in missing medicines and high absenteeism rates among health workers.

Details of the Intervention

Researchers and IPA conducted a randomized evaluation to measure the impact of the full ACT Health program and individual components of the program on utilization rates, treatment quality, patient satisfaction, and health outcomes, including child mortality.

The ACT Health program was implemented by a consortium of civil society organizations coordinated by GOAL Uganda. The program consisted of three components:

1. Information: Citizen Report Cards (CRCs) were provided to health centers and included information about health services in the catchment area, including people’s knowledge of their rights and responsibilities; utilization of the various services offered at the health center; perceptions of the quality of these services; and satisfaction with the health care they received. For most outcomes, the health center data was presented alongside district averages as a comparison.

2. Mobilization: Community meetings were held where the CRC results were presented and discussed. At the meetings, participants developed an action plan to improve health service delivery. Facilitators also held separate meetings with health center staff to discuss the CRC results and formulate a health center action plan which listed steps that the staff could take to improve health outcomes.

3. Interface: Facilitators brought health staff together with representatives of the community to discuss how to work together to improve the quality of health care in the community. Citizens and health care workers produced a social contract laying out specific steps they could each take to contribute to improvements in health outcomes.

Before the program began, the research team collected data about health delivery in facilities in each of the communities and used this information to write the CRCs. Then, researchers randomly assigned the 376 health centers and their catchment areas to one of four groups:

1. Full ACT Health program: information, mobilization, and interface (92 health centers)

2. Partial ACT Health program: information and mobilization (92 health centers)

3. Partial ACT Health program: interface only (97 health centers)

4. Comparison: No program at the time of study (95 health centers)

The research team conducted three rounds of data collection using a household survey, a health clinic survey, and administrative data from the health centers (verified with physical checks of drug stocks and storage conditions). Surveys were conducted before the program was rolled out, and one and two years later.

Results and Policy Lessons

Overall, communities that received the full program reported receiving marginally better care and were slightly more satisfied with their care than communities that did not receive any program. However, no version of the program increased utilization rates or improved health outcomes.

Quality of care: Households that received the full ACT Health program received better care than those in the comparison group. Quality of care improved by 0.059 standard deviations, a very small effect. People in ACT Health program communities were more likely to report having had privacy during their most recent exam and having had their diagnosis clearly explained to them. Health centers in program communities were also less likely to have had stockouts of key drugs during the previous three months.

Patient satisfaction: Patient satisfaction slightly improved for those in the full ACT Health group. Households in program communities were more likely to report that the services offered at the health center were of “very” or “somewhat” high quality; that they were “satisfied” or “very satisfied” with the quality of the care they received during their most recent visit to the clinic; that the person conducting their examination behaved politely, showed respect, appeared interested in their health condition, and listened to what they had to say; and that, compared to the year before, the availability of medical staff had improved. Patient satisfaction overall improved by 0.079 standard deviations, also a very small effect.

Health utilization: The program did not affect utilization rates (how much people sought care).

Health outcomes: No version of the program had any impact on health outcomes (including child mortality) on average or in specific groups in either the short term or longer term (at midline or endline).

Contrary to the theory of change that motivated the intervention, there is no evidence that the improvement in treatment quality was caused by an increase in monitoring or bottom-up pressure by community members. Indeed, the study provides suggestive evidence that providing information to top-down, as well as bottom-up, principals may be a stronger lever for changing the behavior of frontline service providers than mobilizing pressure from the bottom-up alone.

The findings suggest a combination of information provision and increased oversight can marginally change the behavior of frontline service providers, but cast doubt on the power of information to foster community monitoring or to generate improvements in health outcomes, including child mortality, at least in the short term.

Sources

^{1 "}Under-five mortality," World Health Organization. 2015. http://www.who.int/gho/child_health/mortality/mortality_under_five_text/en/

² Fact Sheet, Children: Reducing Mortality. World Health Organization. https://www.who.int/news-room/fact-sheets/detail/children-reducing-mortality

³ Svensson, Jakob, and Martina Björkman. "Power to the People: Evidence from a Randomized Field Experiment of a Community-Based Monitoring Project in Uganda." Quarterly Journal of Economics 124, no. 2 (2009): 735-769.