Over the last decade, discussions around the importance of reproducibility in research have become increasingly common. While many peer-reviewed journals had previously taken a "good-faith" approach, accepting research methodologies in submitted research without much questioning, there has been a gradual shift toward increased accountability in academic publishing. Authors are increasingly being asked to meet strict criteria for documentation and transparency around data, with many journals now including reproducibility checks as part of their review process.
This shift has been propelled by the so-called reproducibility crisis in research — specifically, the inability of researchers to replicate or independently verify published findings in peer-reviewed publications. While the extent of this crisis is not entirely known, reproducibility remains a crucial aspect of any rigorous study.
This is particularly important for SPIA. Our country and causal impact studies span diverse geographies, and the ex-post findings aim to contribute to CGIAR-wide decision-making. For that to work, the findings first need to be reproducible, meaning that others should be able to re-run our analysis and arrive at the same findings. Without this basic step, nuanced questions about whether results hold in other contexts and/or can be scaled are premature. This blog shares what we've learned from our efforts to make SPIA research reproducible.
Understanding the terms
Reproducibility is often conflated with related terms, such as "replicability" and "robustness". Within SPIA, we rely on the following definitions, which have been broadly adapted from the Turing Way Handbook and the National Academies of Sciences, Engineering and Medicine:
- Reproducibility is the ability to generate the same results with the same data and the same analysis scripts as used by the authors;
- Replicability is the ability to produce recurring findings when new data is run through the same analysis;
- Robustness means the finding holds up under different reasonable analysis choices on the same data; and
- Generalizability means the findings hold up in new populations, contexts, and settings.
Reproducibility is therefore the base upon which replicability, robustness and, eventually, generalizability build on.
Three types of reproducibility
Reproducibility can be further broken down into different types, each of which may apply to different strands of research.
Empirical reproducibility focuses on how evidence is generated. It places the onus on researchers to maintain careful records of the data collection instruments used, sampling methods, protocols, and field implementation guides, so that the study can be repeated.
Statistical reproducibility, on the other hand, focuses on whether the inferences of a study hold up, and whether statistically compatible conclusions can be derived from equivalent models and methods.
Finally, computational reproducibility — the subject of this blog — focuses on whether the same numerical outputs and figures can be produced, provided that the same data and code are used. This has been the focus of much of SPIA's recent reproducibility work.
Frequent bottlenecks for computational reproducibility
Curating research scripts and creating comprehensive reproducibility packages that allow external researchers to explore a study's data and methodology often falls by the wayside during the research process. This is exacerbated by the fact that researchers are incentivized to publish results as early as possible rather than on ensuring that the underlying work is reproducible. However, in the evolving research landscape, getting a handle on one's data documentation is likely to pay off. This requires embracing reproducible workflows at the outset and integrating them into the earlier phases of research.
What we've learned about best practices for computational reproducibility
Reproducibility may have been a nice-to-have in the past, but stricter journal and funder requirements, and the broader movement to make research more transparent and credible, are rapidly shifting the incentives for researchers. While the requirements are often geared toward making materials publicly available for others to verify, availability by itself cannot guarantee reproducibility.
To be fully reproducible, packages should be designed with their users in mind. This includes clear labelling of files, detailed set-up descriptions, a list of installed libraries, working file paths, and usage directions for different operating systems. The standard for computational reproducibility should be a package that can run from scratch in a clean environment with minimal intervention. Accompanying it with a README file helps ensure the package can be used easily and without consulting the authors.
Even so, thorough documentation alone does not guarantee that a package will run. Computational reproducibility depends not only on the data and code, but also on the environment in which the code is executed: the software versions, packages, operating system, and file structure on the author's machine. Because these conditions are rarely made explicit, a package that runs smoothly for its authors can fail elsewhere. Such failures do not bear weight on the quality of research, but they can make otherwise sound results impossible to verify. Since such dependencies are invisible to the authors until someone else tries to run the code, they are best identified and rectified early, by testing the package outside the environment in which it was built.
The biggest beneficiaries of a reproducible package may be the authors themselves. After all, when you're six months out from publication and faced with a referee inquiry or a follow-up study, you probably won't remember every detail of your data cleaning choices or methodology. But you'll certainly be grateful that "past you" took the time to document the data and steps in painstaking detail.
Want to work with SPIA's data? Check our Data Resources page for stocktake datasets and replication files.
