Abstract

Policy Issue

Over eight million people die annually in low- and middle-income countries from largely treatable conditions. One prominent reason for this is the low quality of care, which goes hand-in-hand with low patient confidence in health providers as well as under-utilization of health clinics. Under-utilization can be particularly damaging in the context of health crises such as epidemics, when complying with response efforts through early testing and treatment can prove critical to containing their spread.

As a consequence, improving the performance of health workers and boosting patient utilization of health systems is a priority for many decision-makers. One potential way to bring about these improvements is through the use of performance-based financial incentives—but the evidence is mixed on these programs in the health sector, and they can be too expensive for resource-constrained governments.

One promising alternative involves introducing competition among health providers, and giving status awards to high performers as a form of motivation.¹ Another utilizes community monitoring, in which health providers and community members are brought together to jointly address obstacles to adequate healthcare provision.² This study investigated if these two accountability interventions improved utilization and health outcomes in the context of Sierra Leone’s health sector, before and then during the 2014-2015 Ebola crisis in West Africa. 

Context of the Evaluation

Starting in 2010, the government of Sierra Leone embarked on a concerted effort to improve health care throughout the country, including through the launch of an ambitious free health care initiative. The initiative removed user fees and increased health worker salaries, but the government remained worried that that the lack of accountability in the system would lead workers to continue charging fees illegally while exerting little effort. The government expressed interest in understanding if these issues could be addressed through low-cost accountability interventions, which the researchers implemented in conjunction with the government’s Ministry of Health and Decentralization Secretariat.

In 2014, roughly one year after the researchers’ endline survey, Sierra Leone was struck with the worst Ebola outbreak ever recorded. Like other infectious diseases, early isolation and treatment is key to preventing the spread of Ebola, but fears of sub-standard care and stigma deterred patients with symptoms from visiting health clinics.³

The timing of the evaluation relative to the epidemic offered a unique opportunity to examine whether accountability interventions could improve health clinic utilization and health outcomes under both normal and crisis conditions, and to examine whether the interventions contributed to the health system’s capacity to respond to crises.

Details of the Intervention

Researchers working with IPA partnered with the Government of Sierra Leone, the World Bank, and three international NGOs to evaluate the impact of two accountability programs—community monitoring and non-financial awards—on health care utilization and health outcomes. They also examined the cost-effectiveness of the programs, both individually and in relation to each other. 

They randomly assigned 254 government-run clinics in four districts to either one of two intervention groups or a control group, with a third of the study clinics allocated to each group. The two accountability interventions included the following components:

• The community monitoring intervention introduced health scorecards that provided information regarding the state of health care in each clinic, and facilitated three sets of meetings over nine months that involved: just community members; just health facility staff; as well as interface meetings with both sides. Through the interface meetings, community members and clinic staff articulated complaints, but also made mutual commitments to improve services by addressing areas such as staff absenteeism, maternal mortality, and vaccination rates. The goal was to establish mutual accountability through the development of these commitments.
• The non-financial awards intervention facilitated competition among clinics, and gave (non-financial) status awards to the highest performing and most improved clinics in each district. Staff at winning clinics received letters of commendation from politicians and an award at a public ceremony.

Clinics were ranked on key measures such as worker absenteeism, staff attitude and charging of illegal fees, though the criteria for winning was not publicly revealed to avoid distorting staff effort.

Researchers conducted a follow-up survey with the study’s full sample after one year to measure effects over the medium-run. They surveyed the 254 clinics; and surveyed 10 households in two communities near each clinic, collecting data from over 5000 households.

After the Ebola outbreak occurred in 2014-2015, the research team was uniquely positioned to examine the effects of the accountability interventions on the epidemic. They matched data on reported Ebola cases—both confirmed cases and patients who test negative for the virus—from the Viral Hemorrhagic Fever (VHF) database (maintained by the government and external partners, including the US Centers for Disease Control and Prevention) to the areas around the 254 clinics in their sample. This enabled them to analyze impacts on Ebola-related outcomes. While unplanned, this represented a rare opportunity to examine outcomes in the context of a public health crisis.

Results and Policy Lessons

Before the Ebola crisis (after one year)

Community monitoring: Community monitoring significantly increased the perceived quality of health care delivered by the clinics. For example, satisfaction with health workers and health care was significantly higher among those living near intervention clinics, versus clinics in the comparison group.

Individuals living near community monitoring clinics also had higher levels of clinic utilization: they were 7 percent more likely to have sought care in a government-run clinic; and pregnant women were also 11 percent more likely to have given birth at this type of facility.

Health outcomes in community monitoring clinics also improved, driven mainly by significant improvements in child health. The likelihood of households experiencing the death of a child under age five declined by 38 percent, while child weight-for-length and vaccinations also showed some improvement.  

Non-financial awards: Non-financial awards also increased the perceived quality of care. Patients at these clinics similarly reported higher levels of satisfaction, possibly due to more positive personal interaction with staff; and general utilization of clinics rose.

However, the non-financial awards did not increase maternal utilization of clinics, or significantly improve the health outcomes of children.

Results in both groups also revealed a significant increase in trust in Western medicine relative to traditional and spiritual healers.

During the Ebola crisis (after two years)

The research team also analyzed results on longer-run outcomes related to the Ebola crisis. Overall, the interventions increased reporting of Ebola cases – both confirmed cases and cases in which patients tested negative for the virus – and reduced mortality among patients.

Pooled together, total reporting of Ebola cases in the areas where the interventions clinics were increased: reported cases were about 60 percent higher than in areas with control clinics. The intervention built confidence and trust in the health system; the researchers argue that individuals were therefore more likely to report to clinics if they suspected they had Ebola symptoms and get tested.

The researchers also ruled out two alternative explanations for why reported cases were higher.

• First, underlying rates of Ebola transmission were not higher in clinics that received the programs: All types of cases—patients who tested positive and negative—were higher in program clinics; while the proportion who tested positive was the same in program and control clinics.
• Second, clinics receiving the programs did not have more specialized facilities for Ebola testing or treatment.

Higher rates of testing should have bolstered efforts to contain the disease.  The authors estimate the higher rates of reporting slowed the reproduction rate of the disease by 19 percent more in the areas around program clinics.

In addition, death rates out of Ebola cases were lower in the areas around program clinics (about one in seven) than in areas around comparison clinics (one in four) -- with this reduction driven by community monitoring clinics.

The larger effects of community monitoring on medium-run health outcomes and longer-run Ebola patient deaths suggest that getting the community directly involved may be especially effective at improving patient outcomes.

Overall, these results highlight the potential for accountability interventions to improve health systems by increasing the perceived quality of care and building confidence in health providers. Not only can accountability programs produce gains over the medium-run under “normal” conditions, but they may also produce substantial gains during health crises, when trust in the health system becomes critical, as individuals face decisions about whether to cooperate with response efforts, such as complying with quarantines or volunteering for medical testing.

Acknowledgements

In addition to our partners and IPA team members, we thank the survey respondents and clinic staff in Sierra Leone who donated their time. We are also grateful to Carolina Bernal, Afke de Jager, Kevin Grieco, Niccolo Meriggi, Nick Otis, Moritz Poll, and Mirella Schrijvers for excellent research assistance. Finally, we acknowledge several institutions that provided financial or other support: the Njala University Ebola Museum and Archive, USAID-DIV, the IGC, NWO, ESRC, Royal Netherlands Embassy in Ghana, and UCLA’s CCPR.

Sources