Open science discussion at Innovations for Poverty Action
Template G Content Blocks
Sub Editor
Innovations for Poverty Action was pleased to recently host a talk by Courtney Soderberg from the Center for Open Science (COS). COS aims to help researchers increase the transparency and reproducibility of their work. Founded in 2013 by psychologists Brian Nosek and Jeff Spies, it has grown quickly and has received a lot of interest (its work has been featured in the Washington Post, Nature and the Economist).
The talk began with a discussion of issues which compromise the reliability of research. Publication bias – the tendency to publish positive results at a greater rate than null results – is a big problem in research, as is selective reporting of results. Researchers such as John Ioannidis have discussed these problems at length (see for example Ioannidis’ highly-cited article, “Why Most Published Research Findings are False.”)
It’s increasingly clear that we need ways of tackling these systematic problems. Research transparency – including publicly sharing data and code, and pre-registering studies – has potential to improve research. Sharing materials beyond just publications can allow other researchers to dig into the analysis with re-analyses and robustness checks. And pre-registering studies provides a public record of the study regardless of future publications, which helps to combat publication bias.
But as Courtney Soderberg highlighted, there are incentive problems. Research transparency is not yet a widely-adopted norm in the social sciences. Researchers are rewarded for publications in the current system and there is often little professional reward for sharing data. There are time constraints as well – it can be time-consuming to prepare materials to share, particularly if that was not the goal from the beginning of the project.
This is where COS comes in. Courtney described COS’s three-pronged approach: infrastructure, community and meta-research.
- The Open Science Framework is COS’s platform for making workflow easier: researchers can upload their materials as they work, from pre-registering studies, to version-
controlled code, to publicly sharing data. The platform integrates with other sites such as Github, Dataverse and Figshare. - On the community front, COS is developing badges for rewarding researchers for good practices, including preregistration and open data. Journals are invited to adopt and give out the badges, with icons that can be placed directly in articles.
- On meta-research: some of COS’ most-discussed projects include the Reproducibility Projects (including a collaboration with Science Exchange on cancer biology) and the Many Labs project in psychology, in which different labs conduct the same studies independently to see if they can reproduce the same results. Another very interesting new meta-research study is a crowdsourced data analysis project, in which many teams tackle the same question using the same data.
IPA and other groups join COS in the wider movement towards research transparency. Some examples:
- IPA has adopted data publication guidelines and launched a repository for data from randomized controlled trials in the social sciences, hosted at Dataverse. IPA, along with J-PAL, has also adopted a pre-registration requirement (with optional pre-analysis plans), and many of its studies are registered on the AEA registry.
- Berkeley Institute for Transparency in the Social Sciences hosts workshops and conferences on topics related to reproducible research, and hosts a blog on research transparency.
- There is a new center at Stanford called the Meta-Research Innovation Center.
At IPA, we’re excited to be a part of the movement towards improving research through greater transparency, and will post further updates in the coming year. If you have questions or comments on the IPA Research Transparency Initiative, please email researchsupport@poverty-action.org.
Stephanie Wykstra is a Research Manager at IPA and directs the research transparency initiative. She is a member of IPA's Research and Knowledge Management team.
April 15, 2015