As the CGIAR Research Program 2020 Reviews reach their halfway point, Paolo Sarfatti (CAS Senior Technical Advisor) and Max Rünzel (CAS Evaluation Consultant) explain why bibliometrics matter when it comes to assessing the Quality of Science (QoS) in Research for Development in the second CRP 2020 Review blog series.
We have now completed the first six of the twelve 2020 CRP reviews. Throughout the process, we have made use of the CGIAR Independent Science for Development Council (ISDC)’s Quality of Research for Development (QoR4D) framework that established relevance, scientific credibility, legitimacy, and effectiveness as the key elements of R4D quality [ISDC 2020].
The two elements that QoS focused on are:
- Scientific credibility - which considers the quality of sources of knowledge, data, and research methods used, presentation of findings, and interpretation. It recognizes the importance of good scientific practice, such as peer review.
- Legitimacy - which focuses on fairness and ethical considerations within the research process, transparency, management of potential conflicts of interest, recognition of the responsibilities that go with public funding, partners’ involvement and recognition, and the quality of partnerships.
In terms of these two elements, the CAS CRP 2020 Reviews adopted bibliometrics and Altmetrics as part of a mixed-methods approach, using them as proxies to measure and evaluate QoS during the 2020 Reviews. Bibliometrics use statistical methods to analyze documents based on citations (usually books, publications, or articles), while Altmetrics are non-traditional bibliometrics calculating scholarly impact based on the social media uptake and a range of online news outlets.
A key concept in bibliometrics is to analyze the occurrence and co-occurrence of words, authors, or citations, which tries to answer two questions:
- How many times does this word, author, or citation come up in this selection (occurrences)?
- How many times does the same word appear in combination with another word (co-occurrence)?
Combined, this information can then be shown in a graph.
Below, we look under the hood of bibliometric analysis, highlighting two of many potential ways to support QoS assessment.
- Identifying a Country Collaboration Network
First, as a parameter for legitimacy, we identified the country collaboration network (see image no. 1) as a proxy to assess the geographical scope of the CRP’s research partnerships. The network is a visual representation of the occurrence and co-occurrence of joint authorships. That is, the size of the circle indicates the number of publications from a given country as per the affiliation of the authors, and the thickness of the connecting lines indicates the number of publications between two countries. From the graph, it becomes apparent that most authors are affiliated with a US-based organization and that China, United Kingdom, Netherlands, Ethiopia, Germany, Italy, Australia, and France are among the key collaborating countries. This graphical representation shows that the level of south-south collaboration is apparently less-developed than the north-south axis.
PIM Country Collaboration Network
- Analyzing keywords for research significance
As an innovative parameter for assessing the significance of research outputs to next stage users, the image below shows an author keyword analysis of 414 ISI publications from the MAIZE CRP (report forthcoming Dec 2020). The keyword analysis considers the 30 most-used author keywords among all ISI publications the CRP has produced between 2017 and 2019. The three largest circles in the graph, and thus, the three most-used author keywords are maize, conservation agriculture, and climate change. Further analyzing the clusters of co-occurrences, we see that the cluster of the breeding-related keywords (e.g., genomic selection, genomic prediction, etc.) is far, and almost isolated, from the clusters related to cropping systems, crop utilization, and sustainable management of natural resources (e.g., conservation agriculture, sustainable intensification, soil fertility, food security, biofortification, etc.). Finally, it has to be considered that author keywords are chosen subjectively by the author and may thus target the journal-specific audience.
CGIAR Research Program on Maize – Author keyword occurrences and co-occurrences
Analyzing scientific credibility
The CRP Reviews assessed the quality and quantity of different outputs, scientific publications, knowledge products, analytical models and data sets, and their communication and access to implementation partners and next users. A sub-set of articles that scored highest on the bibliometric and Altmetric Score were reviewed concerning their (potential) contribution to the CRPs objectives. For scientific credibility, the reviews looked at the top 15 scientific journals containing the highest number of ISI publications per CRP. Then, these journals’ impact factors (IFs), Journal Citation Report (JCR) category, and the rank within the category (including its quartile) were used as a parameter for assessing the scientific credibility of the CRP’s publications. For example, 48% of GLDC publications are in good to excellent journals, showing high Impact Factors. You can find more information about this in Annex 5b of the CRP Review 2020 on Grain Legumes and Dryland Cereals.
Sourcing and cleaning the data
The data used to perform the bibliometric analysis were obtained from the CGIAR Results Dashboard - the CGIAR tool to showcase CRPs’ outputs. We carried out a cleaning process to obtain the DOIs – a scientific publication’s unique identifier - which we looked up in Web of Science to download the corresponding metadata related to the publication in a standardized manner. Arriving at these graphs and preparing the data for the subject matter experts highlighted the importance of coherent, accessible, searchable, and high-quality data and the value it can bring to the organization as a whole.
In conclusion
Throughout preparing data and subsequently assessing Quality of Science, we have learned that the provided quantitative instruments (i.e., parameters/proxies rather than criteria) to measure and evaluate the QoS within the realm of bibliometrics and Altmetrics laid a solid foundation for the mixed-method approach that was deployed by the reviewers. Further development is needed to solidify the Quality of Science analysis methods using both quantitative and qualitative methods. Furthermore, we have learned that standardized, harmonized, and consistent data on research outputs is paramount to provide high-quality quantitative instruments. This should be a priority throughout the transition towards One CGIAR.
Paolo Sarfatti (CAS Evaluation Senior Technical Advisor)
Max Rünzel (CAS Evaluation Consultant)
Please visit the CRP Reviews 2020 webpage to find out more.
