We came across an opinion piece in in the Chronicle of Philanthropy by Kelly Campbell and Matt Forti of the Bridgespan Group in which they make a number of arguments for conducting rigorous outcome evaluations of nonprofit programs. While the piece concludes with a statement about the value of ongoing assessment in the service of continuous program improvement, our sense was the it mainly emphasizes the external value of evaluation. Often, the goal of such outwardly focused work is the production of evidence about program outcomes; either to satisfy existing funders or to interest new ones.
We believe that it’s time for nonprofit organizations to push back against such a conception of outcome evaluation. First, of course, evaluation is difficult and expensive to do and few funders are willing to pay for it. Good evaluation– the kind that really documents the efficacy of a program model– is very expensive. Second, without a broad-based, large N study, preferably involving randomized control groups, outcomes are often inconclusive. Lastly, most programs, at best, have only very modest effects. The people nonprofit organizations serve– at risk youth, substance using adults, the homeless, etc.– live in devastating situations that may have been a lifetime in the making. The challenges they face are deeply woven into the fabric of society as a whole, that is to say they are structural, and a single program, even a very well-run and comprehensive program, is not likely to make a substantial and measurable difference for all of its participants.
What this means is that single case (or single organization) outcome evaluations, done with limited time and dollar resources, are not likely to show much change. Is this a reason to abandon evaluation? Of course not. But it does provide a rationale for shifting questions away from those that ask whether an organization has achieved a set of specific outcomes, and towards those that ask, in a more open-ended fashion, what it has achieved, for whom, and under what circumstances. Those are the important things to look at. People, communities and indeed whole societies do change, but incrementally and sometimes, only imperceptibly. Effective organizations use evaluation not just to justify their work– does a dance program for inner city kids really need to be justified– but to deepen their understanding of what they do and how it affects those they serve. Doing so shows that they’re in it for the long haul and that in and of itself, ought to interest funders enough to invest.