June 30, 2009
Day in the Life
Request to share experiences in explaining evaluation
When we return to the US after months or years of working in the field, we often experience "reverse culture shock" - aspects of US culture seem surprising and unusual to us. I felt this when explaining "What is IPA?" to family members and friends, and realized I needed an entirely new approach.
In Peru, it takes several examples to illustrate why we use randomized control trials. I talk about the need for a control group, and why individuals or village banks or schools that will receive a particular treatment need to be selected randomly. In other words, we use a methodology that is new to our partners, and it is important that they understand it before we launch a study.
But... I have never needed to explain that some development interventions work and some do not. Our partners look for ways to reach poorer families, or get people to maintain savings accounts, or get students (and teachers!) to actually show up in schools, and they recognize that research helps them figure out what is the best way to do that. It is obvious to everyone in Peru that aid organizations, NGOs, and government agencies often spend money on projects that do not actually help.
And then, at a dinner party a few weeks ago in New York City, it dawned on me that it had not occurred to my lawyer and investment banker buddies that evaluation was even necessary. Not to assess whether there is corruption in aid delivery (though preventing corruption is of course important as well) but to know whether what is being delivered is at all worthwhile.
So my question for you US-based folks - is this something you need to explain to friends and family? Especially to those who do not work in development? And if so, what analogies do you use?
Get Out the Vote
When talking to folks in the US, I like the “Get Out the Vote” case study that JPAL has used in its training course. It’s not about poverty or development, but I find it a good way to walk people through the pitfalls of choosing a comparison group and the consequences of getting biased results. (The case is available through MIT Open Courseware at http://ocw.mit.edu/ans7870/resources/pal/cs1_getoutvote.pdf.)
Researchers had volunteers call 60,000 potential voters in the lead up to the 2002 US congressional elections then used public voting records after the election to see who voted and who didn’t.
In the JPAL case based on the study the authors present five different comparisons. The first is simply a comparison within the 60,000 between those that were reached and those that were not. They then control for differences between the two groups using multiple regression and matching. Finally, they reveal the results of the randomized experiment. The case follows a pattern of presenting a comparison/analysis that seems reasonable and then a discussion of why the results could be biased.
The first comparison suggests a large positive impact of the program on voter turnout (10.8%), but with each subsequent comparison, the effect diminishes until it completely disappears in the randomized controlled experiment.
In my experience, the case is a great illustration of how important it is to get the comparison group right. The ramifications of concluding that a program has a 10% impact in the desired direction when in fact it is near zero are obvious.
Another important feature of
Another important feature of evaluation that I like to bring up is heterogeneous treatment effects (i usually express it as "different stuff effects people differently"). Basically, I say that when thinking about aid programs its not just important whether a given intervention has a positive impact but who the intervention impacts. Is it the very poor, the urban poor, the rural poor or more working class groups? Does it effect men or women more? Young or old? The answers to these questions are not always clear.
Without an understanding of these heterogeneous treatment effects its hard to know how widespread the benefits of anti-poverty programs are. Furthermore, without an understanding of the heterogeneous treatment effects of different programs its hard to target interventions to certain regions. For example, if a given region has lots of problems with out of work young men, it wouldn't make sense to implement a program that mainly helps older women.
One of the big reasons we evaluate then is not just so we can see what works, but so we can see how programs work and who they work for.
debunking the black box fallacy
My original post had to do with the idea that it doesn't even occur to some folks, especially those who have less developing country experience, that evaluations are even necessary. But Alex points out something important - when explaining RCTs (in the field or in the US or anywhere) it's necessary to explain that we don't just look at the outcomes and say yes, there was X effect on the treatment group that was not present in the control group. I think NGOs become much more interested in our work when I point out that we can look at intermediate steps (take up, knowledge, attitudes about a particular product) as well as whether an intervention is more effective for older or younger clients, richer or poorer, female or male. If you can't account for the process, then the results of an evaluation are not as useful to practitioners and policy makers.
A couple concrete examples:
-Business education (http://financialaccess.org/research/projects/0020) helped clients improve their sales, even for clients who said in the baseline that they weren't interested in being trained. This indicates that it might be necessary to make the training mandatory.
-Provision of textbooks in Kenya raised test scores only for those students who were already better off, but not for those that did not perform as well to begin with (http://www.povertyactionlab.org/projects/project.php?pid=33). This makes sense to everyone in Peru, because like in Kenya, the books are not written in the students' native languages - rural speakers of Quechua are not going to benefit from textbooks in Spanish unless they are already better off to begin with.
my own blogged explanations
I've made my own attempts on my blog (http://blogs.cgdev.org/open_book/tag/rcts). In illustrating the power of RCTs, I used the example of hormone replacement therapy and breast cancer in the U.S. I also have a post about the limits of RCTs. Comments welcome.
The same thing happens to me
The same thing happens to me all the time. Many people do not consider that some initiatives to help the poor don't work. It might be quite a shock to them to learn that their charitable contributions don't always translate into positive impacts.
Explaining randomized evaluations
When I explain the design of the microfinance impact study in Mexico that I was working on, I often hear, "Wait, so you're basically just saying 'tough luck' to half the city and then depriving these poor people of money!?" That kind of response from strangers requires a quick explanation so that I do not come off as completely heartless. (And of course, in a more selfless vein, so that support for randomized evaluations continues to grow.)
My favorite strategy is to compare randomized evaluations to medical trials, where we actually aren't sure whether and how an intervention is going to work. People seem to accept that we don't want to give people "development medicine" that doesn't work either.
The medical analogy doesn't work, though, when people assume the treatment is already proven to be effective. I have found that microfinance in particular, enjoys (or, for more cynical readers, suffers from) such a strong and popular narrative that assumptions about how, why, and even if it works are especially hard to shake. I then try to explain a few salient controversies in the field. The trick (which I have NOT mastered) is how to do this in a way that leaves the other person truly convinced, rather than launching into something overly technical that leaves them thinking "Oh, it's just some complicated economic research stuff."
it depends
I find it very easy to explain to conservatives or libertarians or anyone else who generally dislikes big government. They also tend to be skeptical of "do-gooders" or anyone spending money that is not their own, without using it to try to make more money.
For others, I ask them to consider what they would do if they had a million dollars to give away, and wanted it to have the biggest impact. How would they chose from among the millions of non-profit organizations out there? How will they know their money is being used effectively? A lot of people will actually say they want to give it to organizations with "low overhead," which is a good starting point for talking about efficiency and effectiveness.
But yeah, a lot of people don't realize that evaluation is necessary. A lot of philanthropy is just about the act of giving. Everyone dresses up nicely, drinks top shelf liquor and fine wine, and congratulates each other for giving to such a "good cause."