Over the last decade, there has been a growing interest in the use of remote sensing methods for impact evaluation purposes. Increased satellite data availability, together with its increased spatial and temporal resolution, has brought about new possibilities for impact assessment studies. However, these new opportunities come with new challenges. On October 30th 2020, SPIA, together with emLab at UC Santa Barbara, organized a workshop to discuss these challenges and their possible solutions.
One of SPIA's objectives is to expand and deepen evidence of the impact of CGIAR research investments on CGIAR Strategy and Results Framework (SRF) outcomes and associated Sustainable Development Goals. One of the ways we do this is by supporting and funding four impact assessment studies addressing environmental externalities of CGIAR innovations (following a call for proposals issued in 2019). The funded projects all feature the use of remote sensing methods. The “Remote Sensing for Impact Evaluation” workshop aimed to facilitate cross-project learning among projects and start a dialogue between specialists in impact evaluation and remote sensing.
All panelists were clear on the advantages of remote sensing data for impact evaluation purposes. In contrast to ground-based data, remote sensing data can provide large sample sizes at a relatively low cost. Sampling bias is easily avoidable, and data are available for places that would otherwise be costly to reach. However, the use of remote sensing methods comes with significant challenges, which can lead to biased casual estimates if not treated carefully. For example, there are fundamental characteristics of satellite data that can cause problems such as non-classical measurement error, spatial correlation, and the modifiable areal unit problem. Throughout the workshop, panelists and participants extensively discussed the following four key challenges:
- For satellite-generated measures, which involve a process of machine learning classification, the models need to be trained on ground-based data, and these data are often noisy. Therefore, it is essential to understand the outcome being used and the underlying assumptions that make it a suitable proxy for public policy-relevant measures.
- As panelist Jennifer Alix-Garcia (OSU) showed, remote sensing data is likely to involve non-classical measurement error. In contrast to classical measurement error that can cause attenuation bias, non-classical measurement error does not have a predictable effect. The latter could cause either positive or negative bias, which is a problem when using remote sensing data for impact evaluation.
- Remote sensing data measures are prone to contain spatial correlation. It is, therefore, particularly important to define an appropriate level and method that accounts for it when clustering the standard errors, which helps address this problem (but does not necessarily solve it).
- Given that remote sensing data is usually combined with other data sources, it is essential to be aware that there is a trade-off between efficiency loss when aggregating data and the risk of missing the decision of interest if choosing too granular data for the analysis. In order to combine remote sensing data with socioeconomic surveys, aggregation of granular satellite images may be needed to match the relevant levels from which surveys are collected, reducing the number of observations. On the other hand, even when it is possible to perform the analysis on a more granular level, the choice of too high-resolution units of analysis can be misleading. As pointed out by Kathy Baylis (UCSB): "Smaller is not always better." The problem of working with high-resolution units of analysis is that there is a risk of omitting the relevant underlying decision-making process, which means a focus on the study's main objective is lost.
The workshop included further discussions in breakout sessions, where the four research teams shared their intended methods to overcome the challenges of working with remote sensing data and received input from workshop participants and panelists. A recurring topic, in both presentations and breakout sessions, was that of training remote sensing models. Workshop participants discussed the trade-off between quantity and accuracy of training data. Opinions differed regarding the capacity of models to perform better than the data they are trained on.
If you are interested in learning more about the possibilities and challenges of remote sensing for impact evaluation, a recording of the workshop is available here.
SPIA extends a warm thank you to panelists and participants for making this an engaging, enjoyable, and fruitful workshop!
