Introducing AI in the evaluator’s toolkit
It is hard to believe that Artificial Intelligence has been a buzzword since the 1980s with the term first appearing in 1984. What followed was an “AI Winter”, a period characterized by reduced funding and interest in AI research. Today, AI is reshaping how we live, work, shop, and even make decisions. For the evaluation community, this shift is arriving at a particularly crucial moment when evaluators are being asked to do more for less; expanded mandates, complex portfolios, geopolitical conflicts, pandemic disruptions, shrinking budgets and tighter deadlines. Through these new nuanced realities, something has to give and that “something” is the repetitive, time-consuming work that AI tools seem well suited to handle.
At CGIAR's Independent Advisory and Evaluation Service (IAES), we have been quietly but deliberately building our AI capabilities and practical experience in our day-to-day practice. In June 2025, we released an AI technical note that provides grounded, forward-looking guidance for evaluators seeking to explore the role of Artificial Intelligence throughout the evaluation lifecycle. In addition, one of the recommendations from a recent high-level review of CGIAR’s Advisory Bodies by the System Council’s Strategic Impact, Monitoring and Evaluation Committee (SIMEC) was to explore ways in which some of the tasks can be supported by AI tools. This blog is part of a broader IAES effort to document that exploration in practice. In one of the pieces by the IAES director, What’s A Human To Do? A New Blog Series on AI Use Cases in CGIAR Advisory Bodies sets out the organizational mandate: to identify the AI use cases that make us lighter and leaner without weakening our ethics, legitimacy, or rigor. This blog documents the Evaluation Function's contribution to that collective roadmap. Our approach has been consistent; we have been deliberate not to replace evaluative thinking which remains irreducibly human but to reclaim the time and cognitive bandwidth that gets consumed by tasks that are repetitive and/or well suited for AI tools. This blog shares an account of how we are doing that, where it is working, and where we are still learning.
Being honest about risks
Responsible AI adoption requires transparency about limitations, not just possibilities. We are acutely aware that AI introduces real risks, hallucinations, gender bias, data confidentiality concerns, the temptation to outsource judgment, as well as the potential for a false sense of rigor among others. We have encountered cases where AI-generated outputs appear comprehensive and authoritative yet contain subtle errors that only a trained evaluator would catch. We have also found AI genuinely useful as part of our quality assurance process: stress-testing assumptions, surfacing inconsistencies, and flagging risks we might have overlooked.
The honest framing is that AI introduces both risks and opportunities and navigating that reality thoughtfully is itself part of responsible adoption. In every use case described below, the evaluator remains firmly in the driver's seat.
Data Analysis: AI-Assisted Coding with MAXQDA
Qualitative data coding has long been among the most time-intensive tasks in evaluation. Reading, interpreting, and coding large volumes of text-including interview transcripts, focus group notes, and document reviews- demand sustained concentration and is difficult to accelerate without sacrificing rigor. MAXQDA has significantly changed that equation for our team. The software offers AI-assisted coding, automatic document summarization in multiple languages and formats, text paraphrasing, and the ability to suggest new codes with explanatory annotations. These features have meaningfully reduced the time required to work through large qualitative datasets while improving consistency across the coding process.
That said, AI-assisted coding is not without its limitations. The software tends to over-code and generate duplicate codes, requiring careful human review and cleaning after each AI-coded segment. Accurate results depend on clear instructions, well-defined code descriptions, and detailed coding memos from the outset. Large documents must also be divided into smaller sections before processing, and sensitive information must be manually removed prior to upload, as the software cannot automatically detect or redact confidential content. These are manageable constraints, but they demonstrate that AI assistance does not eliminate the need for skilled human oversight it redirects it.
Stress-testing our own thinking
For Evaluation designs to be robust and comprehensive, adversarial thinking is important; the kind that asks, what are we missing? where is the logic fragile? Are there opportunities for stakeholders to push back on? In the past, this has traditionally been the purview of peer reviewers, team discussions, and of course the invaluable experience of senior evaluators. These still remain paramount and indispensable. However, AI has become a useful “sparring” partner in between these iterations. In this end, we have fed an AI model with evaluation questions and asked it to assess whether they are too broad to be answerable and though it does not always produce novel insights, it reliably surfaces obvious problems that familiarity can cause us to overlook. Similarly, in cases where review questions or evaluation matrices evolves across multiple iterations and contributions they tend to accumulate inconsistencies including overlapping scope, varying levels of specificity and even mixed use of the DAC criteria. We have found AI models useful at flagging these inconsistencies, proposing alternative phrasing and even checking if sub-questions nest logically under their parent criteria.
Checking bibliometric indicators
Bibliometric analysis has gained increasing traction within CGIAR’s work and indicators are often highly technical and require careful handling. We have used AI in two specific ways here. First, as a clarification tool, asking AI models to distinguish between closely related indicators such as field-weighted citation impact versus normalized citation impact producing clear explanations and definitions that help align understanding across the evaluation team and with external consultants. Second, as a redundancy checker where we used AI to map indicator lists against one another and asking them to flag potential overlaps, prompting us to justify each indicator's inclusion. This does not replace methodological rigor, but it does accelerate the review cycle and reduces the risk of indicator duplication.
Beyond these two uses, AI can also help evaluation managers assess the feasibility of engaging in new technical domains particularly when tasked with negotiating scope or methods with external firms. A quick AI-assisted scan of an unfamiliar domain can give managers enough grounding to ask the right questions, identify gaps in a firm's proposed approach, and engage more confidently in technical discussions without relying solely on the firm's own framing.
Emerging frontiers
As noted earlier, CGIAR Governance has requested the assurance bodies to explore the kind of work where AI tools can add genuine value. This was not a prescriptive request but rather an exploratory one. For instance identify for AI tools capable of identifying cross-cutting themes, convergent findings, and divergent conclusions across a set of evaluation reports can meaningfully accelerate our work. We are exploring how to do this responsibly, with appropriate quality checks and clear attribution.
Our colleagues at the Standing Panel on Impact Assessments (SPIA) are pushing this frontier further. They tested three AI tools on a task of producing donor-tailored summaries of CGIAR evaluation and impact assessment reports and documented their findings in the blog Can AI Actually Understand Independent Evidence? We Put Three Platforms to Test. The findings are instructive for the Evaluation Function as well. All three platforms produced usable draft material, but none performed consistently without meaningful human review. Accuracy, specificity, and donor tailoring varied significantly across platforms and document types and the review effort required to verify AI-generated outputs was itself a non-trivial cost. The conclusion that AI serves best as a drafting tool rather than a publishing one applies equally to our own workflows.
We are also watching developments in peer organizations closely. UNFPA recently launched an Evaluation Assistant; an AI-powered platform that allows users to query UNFPA's evaluation evidence base drawing from 14 years of the organization’s evaluative data and insights in natural language. This is a significant development, and it raises a legitimate question; could a comparable tool serve IAES staff, consultants, and research analysts in engaging with CGIAR's growing body of evaluation evidence? This can help to conduct multi-level synthesis, conduct document comparisons and even perform advanced visualizations from a chat box interphase. It is worth noting that our colleagues from the Standing Panel on Impact Assessment (SPIA) have already started building such a tool.
The evaluator stays in charge
None of what we have described above changes the fundamental accountability structure of evaluation. Evaluative judgement, weighing evidence, reaching defensible conclusions, and communicating findings with integrity remains a human responsibility. AI can inform that process; it cannot perform it. The takeaway is clear: AI can assist, but evaluators must stay in the driver's seat. For CGIAR's Evaluation Function, that means applying AI tools within a framework that is transparent about what was used and how, attentive to data confidentiality and participant protection, and honest about limitations. It means building the internal capacity to use these tools critically rather than deferring to them. And it means continuing to document what works so that what we are learning can be shared with the broader evaluation community we are part of. Finally, we are constantly guided by the Standards and Principles of CGIAR-wide Evaluation Framework and integrating AI tools requires a governance approach that balances innovation with ethical responsibility, ensuring transparency, fairness, accountability, and inclusivity.
