In round two of Lisbeth Schorr versus evidence-based policy (see round 1 here), she writes in the Chronicle of Philanthropy that,

If government agencies and private grant makers, afraid of being considered not rigorous, unscientific, or wasteful, choose to support only those efforts that meet the randomized-trial test, we will be robbed of:

* Good programs that do not lend themselves to random-assignment evaluations.

* Reforms that are deeper and wider than individual programs.

* Innovations of all kinds.

(Full article here)

Not surprisingly, we disagree. Although Ms. Schorr's article is entitled "To Judge What Will Best Help Society's Neediest, Let's Use a Broad Array of Evaluation Techniques" -- a sentiment that we can't disagree with -- we're worried she wants to throw out the baby with the bathwater. Here's our response:

To the Editor:

Lisbeth Schorr raises common concerns about how "evidence-based" requirements will be interpreted by funders and donors.

At Innovations for Poverty Action, we share Ms. Schorr's desire to avoid "squandering money … on efforts that do no good" but strongly disagree with her assessment that a newfound emphasis on rigorous evaluation methods will inhibit innovation and effectiveness in social policy.

The reason that we and so many others strongly support randomized trials is that they offer one of the best ways to understand what works and help successful programs scale up.

Ms. Schorr's speculation both overestimates what is expected of randomized trials and underestimates what they can accomplish.

Contrary to Ms. Schorr's assertion, few would argue that randomized trails should be applied in every evaluation of every program. Rather, advocates support the strategic use of the trials when appropriate to provide evidence on whether and how to scale up programs with potential to improve people's lives.

The idea that the trials are the "gold standard" necessarily implies that there are other standards and approaches to inform program evaluations. Ms. Schorr even suggests that some funders would prefer to leave needy people in the lurch rather than invest money in a field with few trials. If this were true, many programs today would not be funded.

No one suggests that we immediately cut off programs that lack evidence. But it's critical that we invest in understanding what truly works to make sustained progress on solving social problems. Policy makers and funders should move forward with the best evidence available today, while at the same time working to improve the evidence base itself.

Ms. Schorr's second assertion, that trials are limited to measuring "single, isolated remedies," is also incorrect. In fact, researchers working in the U.S. and abroad have been able to test complex packages of interventions and dynamic processes using creative study design.

Finally, far from inhibiting innovation, randomized trials can provide a way for nonprofit groups and policy makers to take a chance on new ideas and gather evidence through pilots before making massive spending decisions.

Randomized trials allow innovators to prove that their ideas get results without having to rely on fads or rhetoric. There is no better argument for a revolutionary idea than the ability to present strong, clear evidence that it works.

We, like Ms. Schorr, and everyone else in this sector are driven to help those in need. We believe that these people deserve not just good intentions but our best efforts to ensure that the help we are offering will actually do them good. That's why the "evidence-based" movement is so important.

Delia Welsh

Managing Director
Innovations for Poverty Action
New Haven, Conn.