RCTs: The Backlash
The media attention surrounding the release of More Than Good Intentions and Poor Economics has led to a few criticisms of RCTs cropping up around the interwebs:
- Arvind Subramanian worries about the macroeconomic effects of aid, something not addressed by RCTs.
- If Christopher Blattman was picking research like stocks, he would be shorting evaluations.
- Martin Ravallion asks - Are we really assessing development impact? (and David McKenzie and Berk Özler respond).
- Marc Bellemare thinks that the most valuable criticism of RCTs is in the questions that they cannot answer.
- Edward Carr, Prof of Geography, argues for the need for greater integration with qualitative work. Part One and Part Two
- David Week asks for the evidence that RCTs have been effective in driving development.
- Jennifer Lentfer talks about the need to listen to what the poor want. (which Dean Karlan and Jacob Appel discuss in Foreign Policy)
- And finally, there is a spoof post on the ever-entertaining "Stuff Expat Aid Workers Like."
Whats that, you want a handy 5-bullet-point definitive rebuttal list? Sure thing;
- RCTs can't answer every question. Nobody ever said that.
- Having said that, there are probably more questions that they can answer than you originally thought.
- Of the set of questions that they can answer, they are probably the best way of answering them. This shouldn't be too controversial a claim.
- Billions of dollars, and probably the vast majority of aid spending, still goes on projects that have not been rigorously evaluated. So we aren't quite yet in the position to be worried about over-kill.
- Yes we should be careful about extrapolation and generalizing. But this also applies to other sources of evidence, and is not an argument against relying on evidence to the fullest extent possible, or generating as much new evidence as is cost-effective.
Update:
Tom Murphy just sent me remarks on the "evaluation fad" by Jonathan C. Lewis at the Microfinance USA Conference.
Dennis Whittle says we need to "make the beneficiary king".
Sasha Dichter laments extrapolation from too few studies.
Ian Thorpe suggests that RCTs are going through a familiar "hype cycle."
Do add any other links we may have missed here.