As we finalize the last cohort of the CGIAR Research Program 2020 Reviews, CAS reflects on CRP data quality and its role in assessing the effectiveness of CGIAR’s agricultural research.
Effectiveness–recognized as an important evaluation criterion in the CGIAR evaluation policy and the Quality of Research for Development Framework–can shed light on whether a development project has achieved its goal. It is assessed by considering program design, completion of activities, delivery of outputs, and progress towards outcomes.
As part of the CRP 2020 Reviews, the CGIAR Advisory Services (CAS)
employed external reviewers to assess the effectiveness of the 12 CGIAR research programs. To do so, the reviewers looked at the data generated by the CRPs, from the inception of the reviews in 2017 to 2019. The quantitative analysis relied on data from the CGIAR Results Dashboard–a platform providing an overview of results achieved by the CGIAR research programs. Forms of qualitative data collection and analysis are not covered by this blog.
To support the external evaluators, CAS performed a pre-analysis on the Results Dashboard data. Issues with interoperability and resulting data quality between the management information systems were discovered. Data cleaning, with the help of the MARLO and MEL teams, involved more than 900 Milestones, 300 OICRs, 1400 Innovations, and 400 policies.
The pre-analysis looked at the entire data set (Fig. 1) and focused especially on milestones since they are a common unit across all CRPs.
CAS pre-analysis revealed the following limitations of CGIAR’s reporting and repository systems:
- Missing information;
- Broken linkages between plans and results;
- Non-standardized use or application of definitions for key indicators, such as innovations and milestones;
- Technical issues affecting CRP data quality made it difficult to compare different years and identify trends.
Likely causes of this could be changing reporting requirements over the years and interoperability challenges among the different platforms and systems in play.
The CAS, CRP management units, MARLO, and MEL teams worked together to clarify these issues and enhance the data sets and ultimately strengthen data quality. In this perspective, the 2020 CRP Reviews are proving to be an important opportunity to improve the data quality at the CRP level and to understand the potential of such data to be comparable across CRPs for One CGIAR. CAS sees this as an opportunity for system-wide learning, rather than an evaluable factor of a specific CRP.
While CRP Reviews’ purpose is driven by accountability to funders, the 2020 review process has been a true learning experience. CGIAR can take advantage of lessons learned and apply them to improve data monitoring and quality. Perhaps CGIAR can take advantage of this experience to sow the seed of a constructive debate on data quality?
The large amount of CRP data generated over the years positions One CGIAR to ask the “Why” questions, by going beyond data trends, to eventually demonstrate the quality and effectiveness of CGIAR’s international agricultural research. Data quality and big data are a priority for agricultural research to face the challenges of the coming years in light of a constant increase in world population, climate change, loss of biodiversity, and the new challenges presented by the COVID crisis.
For more info and details on the newly released RTB, MAIZE, and FISH CRP reports, please visit the CRP 2020 Reviews page. Follow us on LinkedIn for future discussions and announcements!
Acknowledgements
I would like to thank Angela Gimenez Barrera and Ram Ravi, who provided me with valuable support in developing this analysis.
