Hiring consultants of any kind is always a challenge for not-for-profit organizations. Unlike a physical product you don’t know what you’re really getting until you’ve signed the contract. Moreover, consulting services are typically very expensive, highly customized and the results of the work are sometimes critical to your organization’s success. In a study we did for the United Way of New York City (UWNYC) some years ago, we found that while most of the consulting engagements UWNYC funded for member organizations were successful, several wound up doing more harm than good and in fact actually set the client back. We learned a lot about what works and what doesn’t work in consulting and what we found can be reduced to four key points:
- The consultant was difficult to work with (he or she was often inflexible or unavailable or did not get along with staff or only thought in terms of billable hours)
- The consultant had a canned, one size fits all ‘process’ and was unwilling to collaborate or adequately understand the nonprofit’s unique situation– the consultant in other words was a know-it-all
- The consultant was unable to understand the nonprofit’s ‘big picture’ or the environment in which it operated
- The project failed to build the nonprofit’s internal capacity or help it grow in a strategic way
The study focused on consulting in general and was not specific to program evaluation though we did look at a number of evaluation projects. Since we are sometimes approached by executives in the nonprofit sector with stories of evaluations gone bad, we thought it might be useful to share some thoughts about how to hire an evaluation consultant.
The place to start is with the goals of the project. What do you hope to accomplish? Here are five different kinds of evaluation projects we have been involved in along with some thoughts about the kinds of skills an evaluation firm needs to have in order to manage each one:
| Project Goals | Comments |
|---|---|
| Emergency evaluation | You’re in the middle of a program (or worse, at the end of it) and the funder reminds you that you were supposed to evaluate it. You need a consultant who is flexible enough to deal with a difficult situation, can work under considerable time pressure and understands that not every project can be completed with the highest standards of academic rigor. |
| Evaluation of new initiative | A new program has been funded and you need an evaluation plan that will help you monitor progress and assess outcomes throughout the life of the program. You need a firm that can think creatively and anticipate the many ways in which evaluation data can be used and one that has the ability to stick with your project over the long haul. Perhaps put best, you need a firm that can partner with you and help make your program a success. |
| Evaluation system | You’ve decided that rather than thinking about evaluation on a program by program basis, you would like to incorporate it strategically into your organization’s work. You need a firm that understands that such a system is likely to be something you will need to know how to operate, in other words, one that can coach and train you on how to perform routine evaluation work. At the same time, your consultant needs to appreciates that incorporating evaluation does not mean making evaluation your organization’s primary goal. Most important of course, you need a firm that has the experience to anticipate the many ways evaluation data can be used to support your organization’s goals. |
| Formative or process evaluation | You need to know how a program is actually working out on the ground. For example, you have questions about whether the program design is being correctly implemented across various sites or you may want to gather data to use in making mid-course corrections. For this kind of project, you need an evaluator who can understand the key elements of your program design and has the ability to work collaboratively with the program’s leadership team and with the people actually implementing the initiative. |
| Summative or outcomes evaluation | You need to know whether your program was able to achieve its goals. The firm you select will need to be sensitive to both expected outcomes and inadvertent ones. It will also need to know what kinds of group comparisons may be relevant. For example, you will most likely want to know which subgroups in your population benefited the most and which benefited the least. |
Of course a competent firm ought to be able to handle any of these kinds of projects—there are no firms that we know of that specialize in one or the other. Our purpose in laying them out is to encourage you to think about your goals and about the elements of success that are unique to them.
In a future post, we’ll touch on five things to look for as you begin the process of locating and engaging an evaluation firm for your organization.
