In 2020, CGIAR embarked on an ambitious reformulation of its partnerships, knowledge, assets, and global presence to support the UN's Sustainable Development Goals (SDGs). This institutional convergence, embraced by all CGIAR Centers under the umbrella of One CGIAR, involves the harmonization of 'CGIAR's policies and internal business services across various domains, including Human Resources, Information Technology, Finance, Procurement, Communications, Resource Mobilization, and Research Performance and Results Management. Achieving this goal requires implementing integrated business applications with standardized and consistent data models/definitions to improve interoperability.
The CGIAR System Council approved the Performance Results & Monitoring Framework 2022-2030 (PRMF) and operationalized by adopting the CGIAR Integration Framework Agreement (IFA) in December 2022. The revised Performance and Results Management System (PRMS), designed for reporting against the PRMF, aims to integrate management information from diverse systems into a single, streamlined annual report. With support from Digital Services, the 'CGIAR's Portfolio Performance Unit (PPU) and the Program Coordination Unit (PCU) were tasked with delivering reporting arrangements through PRMS by 2022.
As a backdrop to this reform, in 2019, the Multilateral Organisation Performance Assessment Network (MOPAN) assessed CGIAR's performance, emphasizing organizational effectiveness and result achievement. MOPAN rated CGIAR's Performance Management as unsatisfactory in terms of strong and transparent results function and highly inadequate in evidence-based planning and programming. The system-wide implementation of CGIAR's multi-year evaluation plan, as approved by the CGIAR System Council following the IAES Terms of Reference, relies on the alignment of quality assurance in project management, performance monitoring, evaluation, and learning (MEL) across CGIAR.
The Risk Management Framework of the CGIAR System depicts internal audits and independent evaluations within the three-line model as key elements of independent assurance (see Figure 3 in the CGIAR Evaluation Policy, 2022). The effective functioning of the two assurance mechanisms aligns with the charter(s) of internal assurance functions(s), encompassing internal audit and CGIAR's Evaluation Framework and Policy.
Implementing the Evaluation Policy also assumes and relies on the ability of a PRMS to produce evidence meeting criteria such as completeness, accuracy, and information integrity—ensuring the availability of quality data, often in real-time (see related blog). The inclusion of the PRMS evaluative study in the 'System's multi-year evaluation plan builds on lessons encountered during, and recommendations coming out of, the IAES-commissioned independent evaluations across 2020-2021: 2020 independent reviews of 12 CRPs (see related blog), the evaluation of the CGIAR Platform for Big data in agriculture (2021) and the Technical Note on use of bibliometrics towards evaluating Quality of Science (2021).
PRMS Study: Scope and the Process
In 2022, the CGIAR Internal Audit Function (IAF) and IAES jointly formed the 'PRMS Study Team' to provide upfront advice and recommendations on 'CGIAR's design and implementation of PRMS and related products (SC16 Background Resource). Following industry standards, the PRMS study team relied on the PRMS Project Risk Assessment for guidance and reference. The overarching risk is that the PRMS does not meet PRMF requirements. The study scope included governance/oversight, risk and project management, and PRMS process redesign. The proposed focus of engagement in terms of risks and status of PRMS was linked to the potential auditable/evaluable components of the PRMS ecosystem as of Q2 of 2022. The anticipated value to One CGIAR from a collaborative IA/IAES engagement is envisioned as follows:
Increased agility and responsiveness as Internal Audit/Evaluation flags risks and proposes potential solutions as the PRMS evolves.
Application of innovative approaches and solutions to the design and implementation of the PRMS through the advice of Internal Audit/Evaluation, bringing in best practices.
In a two-pronged approach, the study combined project risk analysis and formative evaluation in line with the "evaluability" and "measurability" standards of the CGIAR-wide Evaluation Framework (2022) and the Policy's pillars[1]. Mixed methods included stakeholder mapping and analysis, key informant interviews (KII), case studies, and desk reviews. Given the evolving nature of the PRMSs, providing real-time and formative advice required agility and real-time learning.
PRMS Study: Lessons Learnt on Value-Add
The joint study team and relevant stakeholders found the engagement approach largely beneficial and aligned with the Terms of Reference. An After-Action Review conducted after the joint engagement assessed the value-added outcomes and processes, yielding insights into the lessons learned:
1. Increased agility and responsiveness as Internal Audit/Evaluation identified risks and proposed potential solutions:
The adaptive nature of the study was crucial for the transitional moment in One CGIAR and the agile nature of the PRMS project. Learning in a rapidly changing environment required revising the TOR and scope of engagement in agreement with governance committees. This adaptive nature allowed everyone to narrow the scope and enhance the study's efficiency and advice.
The PRMS study team considered evolving contexts and real-time learning and continually reassessing assumptions based on the following:
- The collected evidence assesses the risks, draws conclusions, and provides towards assessing risks, draw conclusions, and provide evidence-based advice and recommendations in real-time evidence-based guidance.
- Adapting to the challenge of obtaining core documentation in a standard ex-post performance evaluation was exacerbated by the real-time nature of the process-focused evaluative exercise.
- A mixed-methods and multi-disciplinary approach was used to understand the operating environment, track contextual and programmatic assumptions, and answer questions by leveraging primary and secondary sources.
2. Applying innovative approaches and solutions to the PRMS design and implementation by leveraging best practices across industries:
- Real-time assurance and advisory approaches, now widely used in both industries, have supported improved project delivery and performance during project design, development, and implementation rather than at project completion.
- CGIAR benefits from the two assurance functions' mutually reinforcing and complementary mandates, approaches, and entry points. This joint lens provides a holistic approach
3. Enhancing Quality Assurance and Accountability Culture in CGIAR.
- It implies a need for a more collaborative and constructive approach. IAES/Evaluation should consider sharing PRMS findings more effectively with Senior Directors, Managers, and the Monitoring, Evaluation, and Learning (MEL) Community. This joint approach could engender long-term adoption of the report.
- IAES/Evaluation looked at the effectiveness of the data quality control measures, partially attributing constraints to the resource availability.
- The initial risk assessment was based on the best available knowledge, and the PRSM Core team provided a risk register in the interim.
- The engagement of the PRMS study teams hinged on the participation of its key stakeholders. The extensive exploration and learning in the original TOR involved a deep collaborative engagement with the selected MEL community of practice members. While it enriched the content of the advisory commitment, it also provided an opportunity for the CGIAR IAF and IAES functions to foster relationships.
Opportunities for Joint Assurance Providers
The collaborative process produced valuable insights into the optimal utilization of the respective responsibilities of assurance functions, leveraging internal audit and evaluation professional practices and standards. Through the joint execution of assurance activities involving two different assurance functions, several benefits and opportunities, each contributing significant value, were identified, including:
- Early buy-in: consultation and real-time engagement facilitate buy-in and focus when the study's results are communicated to them or the PRMS core team.
- Increased Confidence: The collaborative engagement of IAF and IAES enhances the credibility and reliability of the assurance process. Their joint involvement has led to more robust and comprehensive evaluation and advisory work, resulting in increased confidence in the results and recommendations of the assurance work. For CGIAR and its centers, this collaborative engagement demonstrates a commitment to transparency, accountability, and good governance, positively influencing the internal and external stakeholders' confidence.
- Enhanced Objectivity: The collaboration between IAF and IAES enhances objectivity to the assessment. Each assurance function brings its methodologies, perspectives, and areas of expertise, which can help mitigate cognitive biases and ensure a more balanced evaluation. This heightened objectivity contributes to the overall integrity and quality of the assurance/advisory work.
- Comprehensive Coverage: Their joint engagement allows a wider scope of coverage of CGIAR activities, risks, and controls. Each function concentrates on different aspects of the same business but applies different cognitive lenses, resulting in a more comprehensive assessment. This inclusive approach aids in identifying potential gaps, weaknesses, or opportunities for improvement that a single assurance provider might overlook.
- Cross-validation: There is an opportunity to cross-validate the discovered findings. The IAF and IAES conducted thorough reviews and validated each other's work, identifying discrepancies and inconsistencies. This collaborative joint effort resulted in a more accurate and reliable assessment. This validation process strengthens overall engagement outcomes, serving as a tool to improve accountability in future evaluation processes.
- Diverse Disciplines: Members of the IAF and IAES bring diverse backgrounds, offering specialized knowledge, skills, and industry experience. Their broad expertise has helped bring on a broader range of perspectives and insights, resulting in a more thorough understanding of risks, better identification of emerging trends, and valuable recommendations for improvement.
Conclusion
Overall, the PRMS Study provides valuable insights into the application and execution of PRMF at CGIAR. It highlights the importance of being adaptable, learning in real-time, and utilizing innovative techniques in the planning and execution of PRMS. In addition, this collaborative study highlights the advantages of joint engagement between the IAF and IAES, contributing to increased confidence, impartiality, comprehensive coverage, and achieving PRMS's effectiveness and efficiency.
[1] Related Pillars 1. Evaluation Embedded in CGIAR Assurance and Management; 2. A Holistic and Consistent CGIAR-wide Approach to MEL; 3. 3. Adaptive Implementation and Management
