In our last post on measuring downstream impacts, we offered a language for talking about impacts. In this one, we discuss the value of qualitative methods for assessing them. Where to begin? First of course, you need to be prepared to wait for the impacts to happen. How long depends upon the kinds of impacts you’re trying to achieve. Referring back to the table in our last post, depending on the size of the school and the intensity of the intervention, it may be possible to observe changes in school climate in just a few months, though whether they will be sustained over time is another question (and one that is often neglected). Many substance abuse, employment and criminal justice metrics look at changes six months post program. Impacts such as community change on the other hand may take years to observe. The evaluation team needs to be prepared to follow-up several months, or possibly several years, after the program has concluded and this waiting has to be built into the project timeline and budget.
But the problem of attributing impacts to interventions remains. With so many plausible explanations for observed changes—or lack of changes— how do you draw a line from a program to its impacts? We believe qualitative methods represent the best approach. Why? Because there is no better way of learning whether Program A had Impact B than by asking. Large sample randomized control trails (RCTs) can also be effective but they are very hard for nonprofits to implement both from a financial and practical perspective.
Of course asking is not enough. In our work, we typically encourage participants to trace a narrative describing what things were like for them before the program, to what they are like now. As we listen to their stories, we keep our eyes and ears open for evidence that ties the program to their present circumstances. When we hear something that suggests a possible impact, we ask how that impact came about and whether and how it relates to the program in question. We recently worked with a South African NGO (ConnectNetwork) that provides capacity building training to grassroots organizations serving women and children in the informal settlements surrounding Cape Town. One goal of the program, one of the impacts its designers hoped to bring about, included an expansion of the services offered by the participating organizations. In describing her work, one interviewee talked about the project planning component of the training and how it enabled her organization to better keep program managers on track. These improvements in management practices in turn supported the organization’s ability to expand its programs.
In some instances it may be important to set up pre-program interviews in order to establish a baseline. Such an approach, while considerably more costly, insures against any problems with interviewee recall. Another option is to employ a pre-program survey. Such a survey could even be used to draw sample for the follow-up interviews, for example, if participants were grouped by some quality the program seeks to address. Thinking about the ConnectNetwork project, we might have administered a survey to organizations at the start of the program designed to assess their overall level of capacity. We then could have conducted interviews with an equal number of high capacity and low capacity groups in order to learn how the program’s impacts were distributed.
We believe that qualitative approaches enable us to see, very specifically, the multitude of factors that often combine to bring about long term impacts. Unlike randomized control studies where impacts are attributed to a program based on a process of elimination (the control and experimental groups are assumed to be sufficiently similar so that any differences in outcomes can only have been brought about by the program), qualitative studies develop evidence about impacts directly. In our next post, we’ll discuss how an evaluation we did for the Coro New York Leadership Center used this kind of approach. If you’d like to be notified when this post has been published give us your contact information and we’ll keep you posted.