*Since publication of this blog, the review of 19 Initiative proposals and a further 12 Initiative proposals was completed and online for SC14 and SC15.
As Initiative Design Teams (IDTs) prepare their 32 CGIAR Initiative proposals, the Independent Science for Development Council (ISDC) polishes its final steps in preparation for its external review. Nineteen Initiative proposals are expected on September 30th* and ISDC stands ready with a roster of 100+ diverse subject matter experts (SMEs) to provide constructive criticism that will drive the CGIAR 2030 Research and Innovation Strategy into fruition.
Good Science Is Not Sufficient for Transformational Change
The Initiatives are not competitive grant proposals. Ultimately all Initiatives are expected to form cohesive components of a synergistic, interconnected research program. The external review of the Initiatives is an essential part of good governance and quality assurance; it will deliver benefits for the researchers, leadership, and System Council (i.e., funders). The main benefit will be an assurance that the best possible science is conducted to deliver the intended development outcomes. This will provide confidence to funders that their investments in One CGIAR research are appropriately targeted with high chances for success. The Quality of Research for Development in the CGIAR Context (Qo4RD [2020]) is a framework that facilitates CGIAR System-wide agreement on the nature and assessment of the quality of science.
To operationalize the QoR4D framework for the Initiative assessments, ISDC embarked on a codesign process with One CGIAR scientists that resulted in 17 criteria using the four elements of QoR4D: relevance, scientific credibility, legitimacy, and effectiveness. The QoR4D criteria also align with the Eschborn Principles—a set of co-designed principles developed by system funders and other stakeholders and endorsed by System Council in April 2020. The criteria were framed to ensure proposals present context understanding, anticipate needs and opportunities of end-users, and build partnerships and activities. The ISDC Chair presented the proposal assessment process during the 13th meeting of the CGIAR System Council (SC13) in June 2021.
How Will the Review Process Work?
Each Initiative will be reviewed by an independent and anonymous review team comprised of three external SMEs, led by an ISDC member and supported by the CGIAR Advisory Services Shared Secretariat (CAS Secretariat). The CAS Secretariat identified SMEs through a competitive roster enrollment that currently contains more than 100 social and biophysical scientists representing more than 25 countries. Although the reviews are anonymous, an aggregated list of reviewers for each Initiative proposal wave will be presented along with select demographic data.
SMEs will be matched based on their expertise to each Initiative review team, with one serving as a coordinator who will aggregate and coordinate the review, working closely with the ISDC member lead. Because the Initiatives are multidisciplinary and cut across all five CGIAR Impact Areas, each SME team will have a social scientist to ensure perspective diversity. Reviewers will attend a workshop in September where the CAS Secretariat will provide detailed guidance on the expectations and process.
How Will Reviews be Reported?
Criteria were designed as a means of delivering feedback for improving individual proposals and their implementation, and to provide advice to System Council. Two templates were developed: 1) for the review teams and 2) for final reporting. Proposals will be assessed by a combination of qualitative data and Likert scores per criterion. Scores will not be weighted.
The external reviewers will score each of the 17 QoR4D criteria on a 4-point Likert scale (criteria mapped against the current proposal template). The external reviewers also will have an opportunity to provide short narratives for each of the criteria, an overall assessment of the proposal, and two proposal sections of greatest strengths and weaknesses. The three external templates developed by the SMEs then will be aggregated to generate the final report (summary) for System Council and the authors.
The final report for each Initiative currently includes the following sections:
- Review summary and actionable recommendation(s)
- Areas of divergence among review team and ISDC resolution (if applicable)
- Low-scoring criteria, i.e., criteria that scored an average of 1 or 0
Next Steps
The CAS Secretariat and ISDC currently are matching SMEs to proposals and building review teams. Each SME will be contracted and provided an honorarium. The review period will commence on September 30 and end with the final reporting by 25 November in advance of the December 2021 System Council meeting. All final reporting will be posted online for transparency. ISDC expects the additional 13 proposals to be reviewed in 2022.
