April 13, 2009
Commentary
Uncovering the Secret to Successful Aid?
This seems to be our week to blog about Bill Easterly's blog. Although in this one, we aren't going to agree as much. I was really excited to see his report about Women's Trust until I got to the final paragraph:
"This kind of aid project is based on a lot of personal, face to face interaction, developing trust and a shared vision, so it is small scale, it has to let things proceed at their own pace, it can’t meet rigid pre-set output targets, it could never be judged by a rigorous “randomized controlled trial” methodology. In short, it involves the kind of tacit knowledge and individual adaptation that could never be converted into a routinized project implemented by the official aid bureaucracies. It breaks all the rules, and it works."
We would be happy to take on an evaluation of just this type of project. The key here is to ask the right question. Easterly is assuming that the "question" in an RCT has to be rigid and non-reactive. But that is not the case. We can evaluate a "process" just as well as we can evaluate a fixed intervention with no flexibility. In fact, we are doing just that. In Ghana (Returns to consulting) and Mexico (mentoring MSMEs) we are engaged in projects that study the impact of a fairly fluid process of "business advice."
But even if it is possible to evaluate a process, that doesn't mean that Women's Trust is ready for an RCT. If, in fact, the project is only working with 50 women or so, then it might simply be too small. In that case, we would agree that this particular project should not be evaluated for impact (though it would still be good to have some monitoring to track the expenses per person reached, e.g.). The reality is that not all projects can or should get evaluated rigorously. But we should still use rigorous methods to learn whether the “idea” is good, and to learn where and when it should be implemented. For that, this exact idea is perfectly well suited to be evaluated with an RCT, ideally using both qualitative and quantitative measurement tools.
Ironically, it seems that the feature Easterly is most concerned about is the small-scale highly-adaptive nature of this project. But there's another obvious issue with a program that is so small and specialized that it can't be ramped up to a larger scale. Aside from a nice story, what impact can we possibly expect from it? Can we really expect thousands of NGO's to pop up and do just this, on a tiny scale?
Or is that the exact key to success, to somehow stimulate thousands (no, millions) of nano-level NGOs, doing proactive, highly flexible, tailored work to help individuals one-by-one? In other words, how excited should we be by this story? Personally, I'm torn until I see the data, to see which approach can make a big impact. Thoughts?
But does it scale?
I don't see why Easterly can claim that RCTs are inherently impotent in evaluating the impacts of small, flexible, fluid programs on measurable outcomes. Indeed, an RCT on such a program could be of great value to an organization that wanted to know how much good it was doing. The leap of faith, it seems to me, is in external validity. Generally speaking, the more subtle and personal the intervention, the more possible mechanisms exist to explain success (or failure).
One example comes to mind: credit officers from an MFI distributed flyers (each containing a randomly-selected interest rate) inviting people to apply for loans. Our goal was to see how people reacted to the different rates--would more be enticed by the lower rates, as expected? Well, yes; but that wasn't the whole story. It turned out that the identity of the credit officer was a stronger predictor of take-up than the interest rate itself! That suggests that some credit officers were simply better salesmen (and saleswomen), and that their marketing skills were an important driver of take-up.
So where does that leave the policymaker? Interest rates are easy to identify and (relatively) easy to set, but marketing skills may not be. Analogously, in the case of consulting for MSMEs, say, training modules may be easy to export, but it's hard to control for the quality and character of delivery by consultants.
Is there a nifty way to get around this roadblock? One initial thought (in the case of the interest rate experiment) would be to stratify by credit officer--treat each as his/her own experiment. Of course, this drives sample size up geometrically... Any thoughts would be greatly appreciated!
Small scale doesn't necessarily mean better services
It doesn't seem that Easterly is misguided in his overall appreciation of WomensTrust's work - they do seem to be providing useful financial & social services to their clients. However, it was my experience in working at a similarly-sized MFI that small-scale organizations aren't necessarily any better at providing individualized service than conscientious larger ones might be. For instance, despite my MFI's low client-loan officer ratios and deep familiarity with the economic situation of the villages in which they lent, they didn't leverage any of these advantages to, say, provide individualized business advice or financial literacy training. Nor were their repayment rates any higher than those of larger MFIs; on the contrary, they were lower than those of groups like Grameen. Institutional learning and new program design were also hampered by their small staff, as no one could regularly be spared to follow new developments at other organization, or conduct thorough monitoring of the social performance of their loans. Ultimately, small-scale service provision seemed less cost-effective and no more personalized than the same service provision through a large-scale MFI would have been.
Of course, there could be some selection bias here; nano-MFIs like mine and WomensTrust might be working in areas where larger MFIs couldn't operate profitably at all, thus skewing the perceived correlation between size and effectiveness. And one anecdote certainly does not a theory make. I think it's just wise to keep in mind that program size is only one of many channels through which effective service provision might operate.