One of the primary ways that CGIAR research contributes to development outcomes is through influencing how polices are designed and implemented. According to available data, research oriented to ‘improving policies’ has been a steadily growing part of the CGIAR portfolio, accounting for 17% of CGIAR research investment by 2006 (the most recent year for which we have comparable data at system-level) and policies and institutions feature prominently in the CGIAR Strategy and Results Framework. As the development agenda broadens to include development impacts beyond agricultural production and food security, the importance of the ‘policy pathway’ will continue to grow. Identifying an appropriate approach for assessing the impact of research on policy will be essential to showing the CGIAR’s contribution to development outcomes.
SPIA asked Mitch Renkow to reflect on the opportunities and challenges of assessing the impact of policy-oriented research (POR) in CGIAR, and his findings are reported in this paper and related Technical Note . The starting point is a data base of plausible claims of significant policy influence made by CGIAR centers between 2006 and 2014. The data base includes 94 claims covering 22 countries as well as global and multi-lateral institutions. While we often think of national laws and regulations as being the primary target for CGIAR policy research, the majority of policy claims relate to other types of policy change such as creation of institutions, changes in investment priorities and budget allocations, innovations in operations and management of agencies and programs, or influence on international treaties, declarations or agreements. While most CGIAR policy claims relate to agricultural policies and programs, almost a third were about natural resource management.
While in many cases the research results underlying an influence claim came from policy or institutional analysis, there are also examples where other types of results, including findings from biophysical science, were important for influencing policy. This raises the question of what kind of research is ‘policy-oriented’ and whether this category should be broader than social science. As the CGIAR increasingly becomes known not just as a developer of technologies and practices but as a provider of knowledge and information to governments and development organizations to inform policies and programs, the line between policy and technology research blurs. Is there research in CGIAR that is not at least potentially policy-oriented? Are we capturing this in how we understand and measure impact?
To help us think about how to assess the impact of policy oriented research, Renkow presents a theory of change that shows that there are two clear phases in the policy process: one in which research contributes to a policy outcome and another in which that policy change leads to welfare and environmental outcomes. Each phase has its own set of conceptual and empirical impact assessment challenges, related, for example, to isolating a contribution of research or defining an appropriate counterfactual. While all types of policy research are potentially valuable, Renkow suggests that some types of policy outcomes and/or stages of the policy process could be more amenable to impact assessment.
A handful of the claims in the database have been rigorously assessed at each phase, but the vast majority stop with policy influence. In the past SPIA has worked hard to promote more studies quantifying the ex post impact of policy outcomes on development outcomes—to be able to compare results to investment in other areas of research. Renkow reflects on how successful that effort has been, and the simple answer is not very. The explanation lies as much with demand as with supply. While rigorous, quantitative ex post impact assessments may be feasible, many research managers, donors and others find “evidence-based narratives” (p. 16) more useful.
Renkow concludes that pushing costly, quantitative studies at the expense of high-quality qualitative analyses would be a mistake. Rather, improving the rigor and quality of qualitative studies would support evidence-based decision making and it may indirectly lead to more quantitative studies since what is being “ learned from these qualitative studies about policy processes and impact pathways … should help increase the number of cases in which it is feasible to pursue full-blown quantitative impact evaluations” (p. 16). How to enhance the quality and rigor of policy influence studies and how SPIA can best support this are important questions that the paper raises but does not try to answer.
Another important question that the paper raises is how we can learn from past studies and from experience to more effectively identify entry points for research in policy processes and bring research to bear on key policy issues. If policymakers tend to use research to confirm rather than change existing beliefs, what does that say about how and with whom CGIAR needs to work? Recognizing that policy outcomes are highly context specific, how can even well-done studies be synthesized to identify generalizable lessons and support system-level learning as well as accountability? Approaches from fields such as political science or realist evaluation have the potential to complement the economics tool kit and enrich our answers to this question.
