Skip to main content

Main menu

  • Home
  • About
  • Who we are
  • News
  • Events
  • Publications
  • Search

Secondary Menu

  • Independent Science for Development CouncilISDC
    • Who we are
    • News
    • Events
    • Publications
    • Featured Projects
      • Inclusive Innovation
        • Agricultural Systems Special Issue
      • Proposal Reviews
        • 2025-30 Portfolio
        • Reform Advice
      • Foresight & Trade-Offs
        • Megatrends
      • QoR4D
      • Comparative Advantage
  • Standing Panel on Impact AssessmentSPIA
    • About
      • Who We Are
      • Our Mandate
      • Impact Assessment Focal Points
      • SPIA Affiliates Network
    • Our Work
      • Country Studies
        • Community of Practice
        • Bangladesh Study
        • Ethiopia Study
        • Uganda Study
        • Vietnam Study
      • Causal Impact Assessment
        • Call for Expressions of Interest: Accountability and Learning Impact Studies
      • Use of Evidence
      • Cross-Cutting Areas
        • Capacity Strengthening
        • Methods and Measurement
        • Guidance to IDTs
    • Resources
      • Publications
      • Blog Series on Qualitative Methods for Impact Assessment
      • SPIA-emLab Agricultural Interventions Database
    • Activities
      • News
      • Events
      • Webinars
  • Evaluation
    • Who we are
    • News
    • Events
    • Publications
    • Evaluations
      • Science Group Evaluations
      • Platform Evaluations
        • CGIAR Genebank Platform Evaluation
        • CGIAR GENDER Platform Evaluation
        • CGIAR Excellence in Breeding Platform
        • CGIAR Platform for Big Data in Agriculture
    • Framework and Policy
      • Evaluation Method Notes Resource Hub
      • Management Engagement and Response Resource Hub
      • Evaluating Quality of Science for Sustainable Development
      • Evaluability Assessments – Enhancing Pathway to Impact
      • Evaluation Guidelines
  • Independent Science for Development CouncilISDC
  • Standing Panel on Impact AssessmentSPIA
  • Evaluation
  • Home
  • About
  • Who we are
  • News
  • Events
  • Publications
  • Search
F. De La Cruz/Bioversity
Blog

Why Bibliometrics Matter for Assessing Quality of Science

You are here

  • Home
  • News
  • Why Bibliometrics Matter for Assessing Quality of Science

As the CGIAR Research Program 2020 Reviews reach their halfway point, Paolo Sarfatti (CAS Senior Technical Advisor) and Max Rünzel (CAS Evaluation Consultant) explain why bibliometrics matter when it comes to assessing the Quality of Science (QoS) in Research for Development in the second CRP 2020 Review blog series.

We have now completed the first six of the twelve 2020 CRP reviews. Throughout the process, we have made use of the CGIAR Independent Science for Development Council (ISDC)’s Quality of Research for Development (QoR4D) framework that established relevance, scientific credibility, legitimacy, and effectiveness as the key elements of R4D quality [ISDC 2020].

The two elements that QoS focused on are:

  • Scientific credibility - which considers the quality of sources of knowledge, data, and research methods used, presentation of findings, and interpretation. It recognizes the importance of good scientific practice, such as peer review.
  • Legitimacy - which focuses on fairness and ethical considerations within the research process, transparency, management of potential conflicts of interest, recognition of the responsibilities that go with public funding, partners’ involvement and recognition, and the quality of partnerships.

In terms of these two elements, the CAS CRP 2020 Reviews adopted bibliometrics and Altmetrics as part of a mixed-methods approach, using them as proxies to measure and evaluate QoS during the 2020 Reviews. Bibliometrics use statistical methods to analyze documents based on citations (usually books, publications, or articles), while Altmetrics are non-traditional bibliometrics calculating scholarly impact based on the social media uptake and a range of online news outlets.

A key concept in bibliometrics is to analyze the occurrence and co-occurrence of words, authors, or citations, which tries to answer two questions:

  1. How many times does this word, author, or citation come up in this selection (occurrences)?
  2. How many times does the same word appear in combination with another word (co-occurrence)?

Combined, this information can then be shown in a graph.

Below, we look under the hood of bibliometric analysis, highlighting two of many potential ways to support QoS assessment.

  1. Identifying a Country Collaboration Network

First, as a parameter for legitimacy, we identified the country collaboration network (see image no. 1) as a proxy to assess the geographical scope of the CRP’s research partnerships. The network is a visual representation of the occurrence and co-occurrence of joint authorships. That is, the size of the circle indicates the number of publications from a given country as per the affiliation of the authors, and the thickness of the connecting lines indicates the number of publications between two countries. From the graph, it becomes apparent that most authors are affiliated with a US-based organization and that China, United Kingdom, Netherlands, Ethiopia, Germany, Italy, Australia, and France are among the key collaborating countries. This graphical representation shows that the level of south-south collaboration is apparently less-developed than the north-south axis.

PIM Country Collaboration Network

PIM Country Collaboration Network

  1. Analyzing keywords for research significance

As an innovative parameter for assessing the significance of research outputs to next stage users, the image below shows an author keyword analysis of 414 ISI publications from the MAIZE CRP (report forthcoming Dec 2020). The keyword analysis considers the 30 most-used author keywords among all ISI publications the CRP has produced between 2017 and 2019. The three largest circles in the graph, and thus, the three most-used author keywords are maize, conservation agriculture, and climate change. Further analyzing the clusters of co-occurrences, we see that the cluster of the breeding-related keywords (e.g., genomic selection, genomic prediction, etc.) is far, and almost isolated, from the clusters related to cropping systems, crop utilization, and sustainable management of natural resources (e.g., conservation agriculture, sustainable intensification, soil fertility, food security, biofortification, etc.). Finally, it has to be considered that author keywords are chosen subjectively by the author and may thus target the journal-specific audience.

CGIAR Research Program on Maize – Author keyword occurrences and co-occurrences

MAIZE Author Keyword

Analyzing scientific credibility

The CRP Reviews assessed the quality and quantity of different outputs, scientific publications, knowledge products, analytical models and data sets, and their communication and access to implementation partners and next users. A sub-set of articles that scored highest on the bibliometric and Altmetric Score were reviewed concerning their (potential) contribution to the CRPs objectives. For scientific credibility, the reviews looked at the top 15 scientific journals containing the highest number of ISI publications per CRP. Then, these journals’ impact factors (IFs), Journal Citation Report (JCR) category, and the rank within the category (including its quartile) were used as a parameter for assessing the scientific credibility of the CRP’s publications. For example, 48% of GLDC publications are in good to excellent journals, showing high Impact Factors. You can find more information about this in Annex 5b of the CRP Review 2020 on Grain Legumes and Dryland Cereals.

Sourcing and cleaning the data

The data used to perform the bibliometric analysis were obtained from the CGIAR Results Dashboard - the CGIAR tool to showcase  CRPs’ outputs. We carried out a cleaning process to obtain the DOIs – a scientific publication’s unique identifier - which we looked up in Web of Science to download the corresponding metadata related to the publication in a standardized manner. Arriving at these graphs and preparing the data for the subject matter experts highlighted the importance of coherent, accessible, searchable, and high-quality data and the value it can bring to the organization as a whole.

In conclusion

Throughout preparing data and subsequently assessing Quality of Science, we have learned that the provided quantitative instruments (i.e., parameters/proxies rather than criteria) to measure and evaluate the QoS within the realm of bibliometrics and Altmetrics laid a solid foundation for the mixed-method approach that was deployed by the reviewers. Further development is needed to solidify the Quality of Science analysis methods using both quantitative and qualitative methods. Furthermore, we have learned that standardized, harmonized, and consistent data on research outputs is paramount to provide high-quality quantitative instruments. This should be a priority throughout the transition towards One CGIAR.

Paolo Sarfatti (CAS Evaluation Senior Technical Advisor)

Max Rünzel (CAS Evaluation Consultant)


 

Please visit the CRP Reviews 2020 webpage to find out more.

Share on

Evaluation
Nov 27, 2020

Related News

Posted on
28 Apr 2025
by
  • Allison Poulos
  • Solomon Adebayo

Driving Change Through Evaluation: Key Insights from PCU’s Solomon Adebayo and Allison Poulos

Posted on
10 Apr 2025
by
  • Patrick Caron
  • Ibtissem Jouini

Speaking Truth to Power: The Role of Independent Evaluations and Integrated Partnership Board in Driving Meaningful Change in CGIAR

Posted on
24 Mar 2025
by
  • Cristiano Rossignoli
  • Moogdho Mahzab

Strengthening MELIA: Insights from Cristiano Rossignoli (WorldFish) & Moogdho Mahzab (IFPRI) on CGIAR Evaluations

More News

Related Publications

MELIA cover page
Evaluation Reports & Reviews
Evaluation
Issued on 2025

Summary of Learning on Monitoring, Evaluation, Learning and Impact Assessments (MELIA): Knowledge Product

Evaluation Reports & Reviews
Evaluation
Issued on 2025

Review of CGIAR Management Response System to Independent Evaluations

Technical Notes
Evaluation
Issued on 2025

Social Network Analysis For The Evaluation Of Development Interventions: A Methods Note

More publications

CGIAR Independent Advisory and Evaluation Service (IAES)

Alliance of Bioversity International and CIAT
Via di San Domenico,1
00153 Rome, Italy
  • IAES@cgiar.org
  • (39-06) 61181

Follow Us

  • LinkedIn
  • Twitter
  • YouTube
JOIN OUR MAILING LIST
  • Terms and conditions
  • © CGIAR 2025

IAES provides operational support as the secretariat for the Independent Science for Development Council and the Standing Panel on Impact Assessment, and implements CGIAR’s multi-year, independent evaluation plan as approved by the CGIAR’s System Council.