There is growing interest and demand to include qualitative methods for impact assessment to improve both accountability and learning in the CGIAR System. There is great potential for qualitative methods to complement quantitative methods at different stages of the impact assessment process. Additionally, there are impact questions which are best addressed relying entirely on rigorous qualitative methods. SPIA will provide guidance on the use of qualitative methods in impact assessment through a series of blogs. This introductory blog provides a synopsis of the topics to be featured in upcoming blogs. Note that this overview is only indicative and may be subject to changes in terms of sequencing and detailed content of each blog.
This blog will offer guidance on how quantitative and qualitative approaches can be fruitfully combined. We will draw upon recent work that conceptualizes how economics can mix with more qualitatively oriented disciplines like anthropology and qualitative sociology, how disagreement between quantitative and qualitative findings need not result in tension, but in fact, a far richer and deeper understanding of the issues, and how machine learning and natural language processing are pushing the boundaries of combining qualitative and quantitative methods in innovative ways, see here and here. The blog will also draw upon a few short pieces (see here) on common misperceptions, the types of questions that qualitative research can help answer, the stages in an evaluation that qualitative work can be used, the value of qualitative data in advancing equity in policy and finally, examples of the use of qualitative research in agriculture, see here and here.
In their book “Qualitative Literacy”, Mario L. Small and Jessica M. Calarco identify five elements that are essential indicators of a well-done qualitative study, regardless of the methods used (interviews, focus groups, ethnographic engagement etc.):
- Cognitive empathy
- Heterogeneity
- Palpability
- Follow-up
- Self-awareness
The blog will provide guidance on what these indicators mean when collecting and analyzing qualitative data in the context of development projects, particularly for impact assessments.
- Training qualitative researchers
Hiring skilled qualitative researchers to study development issues or evaluate research for development programs is a challenge. There is little common understanding of what constitutes rigor in qualitative research, there is a misconception that qualitative studies are cheap or that firms that specialize in collecting quantitative data can also easily collect qualitative data. Moreover, graduate programs in the social sciences pay inadequate attention to teaching qualitative methods.
The blog will provide guidance on training qualitative researchers for development projects, drawing upon the experiences of the authors.
- Coding and analysis in qualitative research
There are several software packages that are widely used for qualitative analysis, but this blog will provide guidance on basic questions related to coding and analysis:
- What is a coding tree? Why is it important?
- Is the analytical approach used by qualitative researchers inductive or deductive?
- What is iterative analysis?
- What is the role of interpretation in qualitative research?
- How do qualitative researchers think about…..?
Researchers working with qualitative and quantitative data oftentimes do not speak the same language on several key concepts. This blog(s) will examine how qualitative researchers think about:
- Sampling
- Generalizability
- Causality
- Objectivity
- Assessing institutional and policy outcomes
This blog will build on a session that was organized during the SPIA Matchmaking Event in Nairobi in July 2022 on ways to help design impact assessments that are not amenable to experimental design, see here for the recording.
Additional resources:
- A general overview of the state of the debate around the logic of “process tracing”
- An older article discussing process tracing as a methodology
- How-to pieces written largely for MEL audiences, available here and here
- A video webinar on process tracing
- Ethical issues to consider when collecting and analyzing qualitative data.
Fieldwork throws up several ethical challenges for the qualitative researcher – how should the researcher respond when a respondent shares a traumatic personal event during an interview? How can the researcher ensure privacy for the respondent when the physical space does not allow for private communication? How should the researcher respond when an interviewee expresses concern over audio recording of interviews? What steps can the researcher take to ensure their own safety in the field?
Based on the authors’ experiences, the blog will provide guidance on how researchers can prepare themselves prior to going to the field.
- Translation and transcription issues
“Context matters” is a commonly used phrase by researchers. Language is key to context, but not enough attention is paid to how words are used and interpreted. This blog will provide guidance on how to reduce the distance between the language of the “field” and the language of the researcher and assess the latest technologies (e.g., API methods) to help in translation issues.
