This is the second in a series of blog posts summarizing the discussions from a May 2017 researcher gathering on measuring women’s empowerment in impact evaluations. Read the first post on household decision-making power here.

Painting a full picture of women’s empowerment in impact evaluations using surveys alone can be challenging. Qualitative methods can help researchers better understand a program’s impact on women’s lived experience and identify reasons why a program worked or didn’t work, but economists rarely have incentives to incorporate robust qualitative methods into their research.

In this post, we feature an interview with Dr. Sarah Baird, an economist and Associate Professor of Global Health and Economics at George Washington University. Sarah uses field experimental methods to understand what works to improve the lives of young people, particularly adolescent girls in Sub-Saharan Africa. Here, she shares her thoughts on how qualitative methods can complement quantitative methods based on her experience as part of an interdisciplinary research team conducting a multi-country longitudinal study on gender and adolescence. She also talks about the barriers to conducting more mixed methods research in the social sciences.

Next week, we’ll publish an interview with the director of the study featured here, Dr. Nicola Jones, to understand the qualitative researcher’s perspective on this mixed methods story.

Nellie:   One argument for putting more focus on mixed methods work is to untangle the mechanisms behind multi-faceted programs for women and girls. This is an important component of the DfID-funded Gender and Adolescence Global Evidence (GAGE) mixed methods study that you're working on with Nicola Jones, who I’m going to speak with next week. Could you tell me about the study and your role?

Sarah:    GAGE is a nine-year mixed methods longitudinal research and evaluation program examining what works to transform the lives of poor adolescents in general and poor adolescent girls specifically. Now we're finalizing the program and study designs, as well as the survey instruments, and we hope to launch data collection in early September.

Because we're working with adolescent girls and thinking about their transition to adulthood, the interventions are very multifaceted. This set-up makes it clear that we're going to need both quantitative and qualitative work to get at the bottom of what we're ultimately impacting and why. Life skills training in safe spaces, for example, is trying to teach a whole host of skills that affect many different dimensions—economic, psychological, health—of girls’ lives. Quantitatively it's difficult to define the primary outcomes and the mechanism by which those outcomes are impacted. Qualitative research can unearth the details in these multifaceted programs, especially when you’re thinking about adolescent girls. Is it the effect of spending time with peers or the curriculum developed for the safe space group that led to lower pregnancy rates? The mechanism can lead to very different scale-up strategies and cost.

Nellie:   In the work you've done already, how have the qualitative researchers contributed to the quantitative aspects of the research?

Sarah:    It’s been very useful to get the other team’s perspective on what can be better captured qualitatively. For example, in many surveys, we ask open-ended questions, such as “What do you like about school?” and then write a descriptive answer. These questions are meant to unpack the mechanisms behind our findings. If you find that an intervention kept girls in school, you really want to understand why. But looking at mechanisms qualitatively instead is a good division of labor. We may not pull all these questions out of the quantitative questionnaire, because with a much larger sample we can talk more generally about the population. But our surveys can stick to what we think are going to be the key mechanisms and leave the secondary and tertiary possibilities to the qualitative work.

Being able to remove these questions from the survey and instead explore them in qualitative work is important, because when you want to capture outcomes across six dimensions, you very quickly get to a point where you need have a three- to four-hour questionnaire to capture everything that you wanted to. No one, particularly young adolescents, has that kind of attention span.

Nellie:   Despite these potential benefits, we don’t see as many partnerships between researchers across disciplines as you might expect. Tell me about a study you worked on that you think could have benefited from better integrated qualitative components. What was missing?

Sarah:    My best example is a cash transfer study in Malawi that I worked on right out of grad school. We used qualitative in-depth interviews to elucidate what people understood about the program and how they understood their treatment arm in comparison to somebody else's. Those interviews became footnotes in the paper to address referees’ critiques. Economists typically view qualitative research as a way to fill-in those little gaps in the quantitative analysis.

We also applied for funding to analyze the qualitative data more rigorously, but the funder came back with funding just for the data collection. I don’t think they fully appreciated that qualitative analysis is a totally different skill set than what our team of economists could offer. Now we have hundreds of in-depth interviews sitting on my computer that no one has analyzed yet because of a lack of funding. That's a shame.

If I could go back, I would bring in an expert on qualitative methods to design an integrated study using their state of the art methods along with ours. Even if you have the funding, the challenge is in the publication at the end. There's still a pretty limited audience for that type of work, and if you're just out of your doctorate and aiming for tenure, you want to publish in the best journal in your discipline. It’s hard to spend significant resources on qualitative work that’s going to be left out of the paper, because it just won't be able to get into the outlets that you’d like.

Those interviews became footnotes in the paper to address referees’ critiques. Economists typically view qualitative research as a way to fill in those little gaps in the quantitative analysis.

Nellie:   So, what could journals do to encourage more inter-disciplinary research?

Sarah:    Right now the most common way around this issue is to publish two versions of the paper—one on the qualitative findings and another on the quantitative. That's a good first step, and from a policy perspective, if a colleague and I then present to the government together, we can also convey the nuance of the results.

But most of the academics reading one paper are not reading the other. From that perspective, we might learn a lot more by integrating the findings. I’d like to see a journal that development economists, as well as demographers, sociologists, and others doing qualitative work, all respect to develop a reputation for publishing excellent mixed methods work. There could still be papers on the quantitative and qualitative findings but also a third paper in this journal that looks at the two together. For this to happen, senior researchers who have less pressure on them for tenure would need to be excited (a) to do work that goes to this journal and (b) to read this journal.

For now though, public health is a good space to move this mixed methods agenda forward. Our global health department at GW is cross-disciplinary by design. Journals in economics, public health, and sociology are all equally valued. As an individual within the economics world, I have my hierarchy of journals, and my ego wants to speak to that world, but as a member of this department, they're going to be equally happy if I publish in the Quarterly Journal of Economics or the Lancet. This is probably true for public policy departments as well.

As an organization committed to making data and information meaningful and useful for policymakers, IPA also has the potential to move this agenda forward by fostering a conversation about the need for and encouraging more mixed methods work.

Read the full interview with Professor Baird on GAGE's website here.