Sonal Zaveri is an independent evaluation expert based in India, who has worked in over twenty countries and incorporated a diverse range of methodologies and approaches throughout her 30-year career. Particularly passionate about young people and gender issues, she is a founder and board member for the Community of Evaluators – South Asia, a founder and regional coordinator for the Gender and Equity Network South Asia (GENSA), and the co-chair of EvalGender+, and a Board Member for the International Development Evaluation Society (IDEAS) and the International Evaluation Academy (IEAc) among several other roles and titles.
Zaveri is also a member of the CGIAR’s Independent Advisory and Evaluation Service (IAES) Evaluation Reference Group, which played a role in the development of the service’s new set of evaluation guidelines for assessing the quality of research for development (QoR4D) – both within CGIAR, and in other like-minded organisations. We spoke to her to learn from her expertise as an implementer of real-time evaluation (RTE) and hear her take on its potential integration into CGIAR research contexts.
Q: Can you tell us more about your experience to date as an RTE implementer?
A: I had the opportunity to do a real-time evaluation for UNICEF South Asia (ROSA) during the COVID-19 pandemic, addressing the effectiveness of their gender response in South Asiafor one year, from September 2020 to September 2021. In an RTE, you provide evaluative feedback while the project is being implemented, to help improve it, provide insights, change course, and adapt on the basis of evidence gathered in real time. We were right in the middle of the pandemic, and had no idea how things were going to unfold or how long it was going to last – but we did know that women and girls were being disproportionately affected.
Historically, real-time evaluation has been used for humanitarian interventions, where the situation is very dynamic, uncertain and you need instant feedback and rapid turnaround. But this RTE with UNICEF was long-term, and gathered evidence and learning to a) integrate gender in a humanitarian response, in a fast-moving pandemic as well as b) ensure gender mainstreaming at the nexus of development programming.
Q: What are the situations where you have felt the approach to be particularly powerful?
A: RTE is powerful because it is very flexible – in terms of evaluating at multiple times, and particularly when you're not very sure about how the external environment will unfold, how it will impact implementation or how fast the situation will change. This was exactly the situation when I used the RTE to evaluate UNICEF’s gender-related interventions during the COVID-19 pandemic. I believe that RTE has the potential to be used with a wide array of projects, such as when you are innovating and you're not sure whether or how the intervention is going to work or if the project design/objectives may need to be tweaked or changed. I think RTE has a very wide scope for use, beyond humanitarian settings, and that it can be extremely powerful when done well. I also think that the potential for it to be used in a more long-term way is something that we have not yet explored enough.
Q: What have the commissioning organizations and evaluators derived from using RTE that added value?
A: For evaluators, I think RTE offers a unique opportunity to understand a project and organization better, including the ecosystem in which it is functioning, the key challenges, and so on. Otherwise, we evaluators get very little time to truly understand the challenges and contexts in which projects are being implemented. Using this approach, evaluators learn a lot – we learn in real time – and we also become more realistic about what can or cannot happen within the time period for a particular project. I think this understanding, this wisdom that we get when we are in a longer-term relationship with the evaluand, is absolutely priceless.
For the commissioning organizations, there is an added element of confidence in the timely use of evaluative evidence. A pre-condition is to choose the “right”, competent evaluator – one who provides robust and timely evidence for appropriate adaptation during the implementation; one who is willing to share learnings, dialogue about findings, reflect about the way forward,how to tweak the implementation, use the evidence to change course, and so on. The RTE process is important, because at the end of the project you want to make sure you achieve the broad objectives for which you started the project.
In many dynamic contexts or innovations, you don't have a baseline, so with such an absence you do require some other way to monitor what's going on. Often, monitoring is very sparse, or may not be robust enough for the dynamic, fast-changing situations. I think RTE is also very helpful during these times, because it keeps one’s finger on the pulse of what is happening and offers the opportunity to engage in timely dialogue as well as working closely with the implementation team.
So, I think for both commissioners and for evaluators, RTE presents a win-win situation. The commissioners will likely be more confident that evidence gathered will be used to ensure that the project’s broader project objectives are fulfilled. And the evaluator gets a far better understanding of the project, and the opportunity to engage in open dialogue, that enables participation and inclusion in the evaluative process – rather than ‘parachuting in’, evaluating something that's been going on for a long time and trying to make the best sense of it, and then ‘parachuting out’.
Q: What particular challenges and/or benefits can you envisage to using RTE in the CGIAR context?
A: For CGIAR, I think it is definitely one of the evaluation approaches that should be considered. It brings all the advantages I mentioned earlier, including better ownership of the findings. There are also many tools and methods used in RTE that are seldom employed in conventional evaluation, because in RTE you have the luxury of time, continuity, and are able to build a relationship of trust. I think the relationship building is something very unique; especially when we talk about participation, inclusion, and ownership of the process. RTE really checks all of those boxes.
One challenge that people must be aware of when they use RTE is that you need continuity of the evaluator. In my case, we started off with a team of two people: I was leading the project and I had another colleague with me, but she left for better opportunities, about a third of the way into the project. I continued, and that was really critical. You have to make sure that the evaluator is available to you for the entire span of the evaluation, especially if it is a long-term proposition.
Q: Any further advice?
A: I would also say that you need to choose your evaluator well. The facilitation that is required by an evaluator is critical and because the evaluator is going to be with the project team for a long time, he/she needs to build a trusting relationship with the evaluand. Equally, the project team and the evaluation commissioners should have confidence in the evaluator’s technical expertise and that what the evaluator is advising makes sense. They also need to be in close contact with the evaluator, because whatever is being suggested is likely to involve design and implementation changes, and will have implications for budgets, staffing, resources, and so on. So, there is an added responsibility for the commissioners as well. If you get all these pieces in place, it works well.
Another piece of advice is that because you’re working in real time, it's important for the evaluator to be agile and quickly understand what is changing politically and strategically. Evaluators have to go beyond the conventional interviews and focus group discussions; they may need to observe and perhaps engage at the grassroots directly, or through intermediaries. They also need to be present at relevant meetings and events – perhaps not participate, but listen in to understand the organization better and what is happening on the ground. This method is known as the “Fly on the Wall”. Here, the evaluator can be a silent or a participant observer, of course with the permission of the evaluand!
The last piece of advice I would give is that if the evaluator is able to build trust and confidence, and provide meaningful and relevant evaluative guidance to the project team, the commissioner is likely to demand more from the evaluator. This has implications for budgets and resources. So, one may need to negotiate the time available for the evaluator and the evaluation – or the contract has to be a bit flexible depending on the needs and requirements from the field.
