In a previous blog, we highlighted that the assumptions on objectivity shared by interpretive qualitative researchers are not the same as how researchers in the quantitative tradition approach objectivity. To summarize briefly, while quantitative approaches seek to isolate, control and manipulate for predictive capacity and replicability, qualitative methods are designed to understand a dynamic social reality unfolding in its natural setting. Quantitative researchers seek value neutrality and aim to cut through the mess and the noise of a context to answer causal questions. Qualitative researchers, on the other hand, understand their research setting to be a complex social universe with multiple actors and partial viewpoints, of which they themselves are one partial observer and participant.
A critical feature of qualitative research is an awareness of positionality, which refers to the ways in which a researcher’s social identity (for example, their racial/ethnic/national identity, class or gender position as well as their broad cultural orientation as a scientist hailing from a particular social location) may predispose them to particular interpretations of their research subjects. While qualitative researchers begin from a position of partial understanding stemming from their background, experiences and knowledge, the goal of the inquiry is to move beyond this partial view to include how their research subjects make sense of their situation. Hence, qualitative researchers do not consider it a loss of objectivity when questions are not standardized, if knowledge is co-produced with the research subjects, or if the research relies on observations of natural settings rather than on experimental design.
Thus, while the keywords for achieving objectivity in quantitative inquiry are removal of bias, replicability and statistical generalization, the keywords in interpretive qualitative inquiry are awareness of positionality, use of reflexivity, improving trustworthiness and transparency, all in an effort to “maximize objectivity”. In this blog, we consider what this means in practice – what strategies do qualitative researchers use to maximize objectivity?
Maximizing objectivity during study design
Broadly speaking, qualitative researchers aim to arrive at an understanding of the perspectives, beliefs, and motivations of the subjects of the research. A starting point to ensure that we maximize objectivity is to ask: who are the different actors that reasonably constitute the ‘subjects’ of the research? For example, let’s say we want to investigate a research puzzle using qualitative methods – why are rice farmers not adopting pest-resistant varieties even though it is in their interest to do so? One approach is to interview farmers and ask them about how, and why, they select which variety to plant. The researcher can then arrive at an understanding of the underlying beliefs and motivations that inform the farmers’ decisions. This is a sound approach, but it does not maximize objectivity.
An alternative study design would be to include a 360-degree view of the research problem, taking into account the perspectives of all the actors in the decision-to-action chain: the scientists who develop the rice varieties, the policymakers who decide to release the varieties, the agriculture extension officers who support farmers’ decision-making, private sector actors operating in the marketplace, and last but not least, the farmers who ultimately grow the rice.
Why is this important? By only including farmers’ perspectives, we assume that the levers of change need to be turned at the lowest level of the decision-making chain. In the development sector, we are often told to think about the theory of change. By including all actors, we can begin to ask more expansive questions – what are the scientists’ theory of change? Is it the same as that of the farmers? For example, do the pest-resistant varieties target the same pests that farmers are most worried about? Do the pest-resistant varieties have other traits that make them less attractive to farmers? If these gaps exist and are well understood, have the policymakers and implementers adjusted their recommendations? If so, how? And if not, why not?
Qualitative researchers are comfortable with multiple perspectives and different interpretations. Positionality matters such that every level in the decision-making chain is just as important and valid; in this sense, the approach is democratic. It is not just the researcher or the scientist or the policymaker - those at the ‘top’ of the chain - who have conceptions and theories about how the world works and have ‘theories of change’. The agriculture extension workers have a ‘theory of change’ as do the farmers.
The qualitative researcher is an interlocutor of multiple worlds and a participant with other stakeholders in the research puzzle. So, the first order of business in answering the research puzzle - why do farmers not use pest-resistant rice varieties - is to understand and delineate the different worldviews of all actors with equal levels of curiosity and interest and that is why it is important to include all stakeholders to arrive at the points of shared understanding and the nodes of disagreement, through which we can begin to frame policies that bring these worlds together.
Maximizing objectivity during data collection
Having designed a study that includes all stakeholders, we now have to embark on data collection. How do we maximize objectivity at this stage?
In designing questions, the goal is to understand the respondent’s worldview and do so with cognitive empathy (for more on cognitive empathy, see here). So, for example, when interviewing scientists about the development of pest-resistant varieties, or interviewing farmers about their pesticides usage, we don’t need to feel stymied because we are not knowledgeable about the science or because we have no first-hand experience of farming. Our goal is to enter the worldview of the respondent and understand the logic that drives their thinking. What is the best way of achieving this?
First, we shift our understanding of data, not as something to be “found” but as something to be “co-produced”. Second, we expect that data are not “static” - respondents may not have well-formed, coherent viewpoints, or their answers may shift depending on the question, the interviewer or their motivation, or even because the interviewer approached them at an inconvenient time. Quantitative researchers often read exhortations to “co-produce” or treat data as “dynamic” and feel nervous because it seems too chaotic, subjective and uncontrolled. Qualitative researchers, on the other hand, believe that the world is messy and we must take this messiness into account. The strategies that qualitative researchers use to “account for messiness” is to think through questions such as: how best can questions be asked so that the respondent and researcher comfortably co-produce data? What types of questions or ways of asking inhibit co-production? How best can the researcher build rapport with the respondent? How can they gain the trust of the respondent? If data are dynamic, how do we know we are on the right track? In a practical sense, co-production and dynamic forms of data-gathering mean that we:
- Start with a set of open-ended questions but be alert to modifying or changing the questions depending on the responses.
- Be aware of positionality and engage in self-reflexivity.
For example, the qualitative researcher enters a village that they have never before visited with the goal of interviewing farmers for a study on pesticides. The researcher wants to ensure that they talk to different types of farmers who cut across social divisions of class, gender, race, caste etc. Some of these divisions are more easily observed, e.g., a male or female farmer. But what about, say, social class? Wealthier farmers are likely to own more land, but the fields are far away and there is no way of immediately knowing who owns which plot of land. However, the researcher observes a car parked and learns that it belongs to the person who they are about to interview. The car is a visual clue, but how to attach meaning to the car, i.e., convert the observation into ‘data’? The particular make and model of the car would be considered affordable for a lower-middle-class person in the city where the researcher lives. However, to interpret what the ownership of this car means for this farmer’s social class in the village requires being aware of positionality and that one’s judgments are colored by one’s background (in this case, an urban-rural divide) and that the same object - the car - may not ‘mean’ the same thing in the village as it does in the city.
Reflexivity means being rigorously aware of positionality. By being self-reflexive, the qualitative researcher constantly questions their assumptions and recognizes the possibility of misjudgment. To understand what ownership of that particular car means in this particular context, the researcher can employ different strategies. Asking the car owner direct questions runs the risk of creating awkwardness and impeding the possibility of building trust and rapport. If the researcher is with a local guide or translator, one could ask them what ownership of the car suggests about the respondent’s social status. The researcher can also look around and see what other cars are in the village and after the completion of the visit to the village, discuss their interpretation with the local team.
The key points here are that data collection does not start and end just with the ‘interview’. Reflexivity is an ongoing process and results from value awareness rather than value neutrality. Thus, positionality and reflexivity are critical elements in maximizing objectivity and ensuring that the data is of the highest quality.
- Pay attention to how power asymmetries might shape the perceptions of actors and therefore, have a healthy distrust of self-serving statements. E.g., if a scientist tells the interviewer about how farmers think, the interviewer should at least wonder about the basis for the scientist’s conclusions - is it direct field experience or observation? Or prejudices? Likewise, if a farmer tells the interviewer that they use pest-resistant varieties, the researcher should be attuned to the possibility of social desirability bias - that the farmer is telling the interviewer what they think the interviewer wants to hear. The key is to recognize the complexity of social speech and how it is shaped by power and the dynamics of status, respect etc. A trained interviewer is very attuned and aware of reading between the lines, interpreting emotions, and asking follow-up questions to ascertain the source without being rude or losing rapport.
- Triangulate through different sources of evidence - e.g., matching observational data with interview data, matching observational data across team members, matching interviews with the same person conducted at different points in time etc.
Qualitative studies in the development sector usually involve more than one researcher - oftentimes there is a local interpreter, guide or collaborator. The ‘locals’ can be a source of information or bias, but again, it is possible to account for this - by acknowledging and recognizing the positionality of the local researcher and by considering them as one more, rather than the sole, perspective.
- Be alert to factual discrepancies.
Maximizing objectivity during data analysis
Finally, in the data analysis stage, we can maximize objectivity in a few different ways. At its heart is the recognition and acknowledgement that the process of knowledge production is itself a social process that involves actors, institutions, histories, asymmetries of power, perspectives and values. Qualitative researchers look for safeguards so that there is both rigor and fairness in the representation of complex realities. In practice this can be achieved by:
- Have the data be read, discussed and coded by multiple coders. The resolution of disagreements ensures that the interpretation is nuanced and includes multiple viewpoints. Increasingly, Large Language Models (LLMs) are being used for coding. In a future blog, we will share some thoughts and considerations on the use of LLMs.
- Pay attention to the factual statements on actions taken and underlying beliefs, but also to the emotions expressed in interviews. The quote below from a farmer in Vietnam highlights why this is important.
“I have been conflicted about applying pesticides and ignoring others’ advice. But it could affect my harvest and revenue afterwards. For example, if there was a brown planthopper outbreak and other farmers sprayed but I did not, then I would have nothing left to harvest while the others’ harvest wouldn’t be damaged at all. I would feel ashamed because my family could blame me for not working hard and working perfunctorily.”
If we pay attention to the action, we understand that the farmer is not following recommended guidelines. If we go deeper, we begin to see that the farmer’s decision is dependent on what fellow farmers are doing on their plots. Thus, pesticide application is a “social” act, not just an individual one. Finally, it is critical to understand the emotions of fear and shame, and the loss of both income and face if things were to go wrong in the field. Attending to all the elements in this short excerpt is a way to enter the farmer’s world and maximize objectivity.
How do these findings translate into policy? One clear implication is that policy must be thought of, and implemented, at the level of the collective. The usual practice is to train some farmers with the assumption that untrained farmers will learn upon observing the trained farmers. However, policymakers could consider a more concerted attempt to train farmers together and discuss solutions at the collective level. Finally, it is important to acknowledge the emotions being voiced. Scientific language can oftentimes be abstract and distancing and for change to occur, it is worth considering how to bring the language of science and policy closer to the voices of the farmers.
- Brainstorm about the findings with all the stakeholders. This raises trustworthiness in the data.
- Provide an account of the strengths and limitations of the study including how the positionality of the researcher (or multiple researchers) may affect participant responses. Being reflexive in this manner is not simply about writing a perfunctory statement to satisfy some “methods checklist”. It is a critical element of the research steps, contributing to transparency of data collection and analysis, and enhancing accountability to the respondents, other stakeholders and the larger public.
This blog drew from the following sources:
Burawoy, M. (1998). The extended case method. Sociological theory, 16(1), 4-33.
Harding, S. (1995). Can feminist thought make economics more objective? Feminist economics, 1(1), 7-32.
Mahoney, J., & Goertz, G. (2006). A tale of two cultures: Contrasting quantitative and qualitative research. Political analysis, 14(3), 227-249.
Murphy, A. K., Jerolmack, C., & Smith, D. (2021). Ethnography, data transparency, and the information age. Annual Review of Sociology, 47(1), 41-61.
Rao, V. (2023). Can Economics Become More Reflexive? Exploring the Potential of Mixed Methods. In: Deshpande, A. (eds) Handbook on Economics of Discrimination and Affirmative Action. Springer, Singapore. https://doi.org/10.1007/978-981-19-4166-5_19
Small, M. L., & Calarco, J. M. (2022). Qualitative literacy: A guide to evaluating ethnographic and interview research. Univ of California Press.
Zolvinski, S. (2008). Listening to farmers: qualitative impact assessments in unfavorable rice environments.
