- Technical research expertise
- Project management skills
- Excellent thinking skills
- Flexibility
- An orientation towards collaboration
- Excellent writing and communication skills
- A strong client services orientation
The first three attributes have to do with getting your project done. The next four relate to your consultant’s ability to work with you so that the evaluation has maximum impact and comes off smoothly.
Technical research skills are of course paramount. If you are going to use evaluation to improve your program or simply communicate what you have achieved to your stakeholders, you need to be certain that the findings you report are accurate and complete. You need confidence, in other words, that data collection and analysis were handled correctly; that your surveys asked valid questions; that any interviews were conducted in a neutral and evenhanded way; that the sample that was drawn was of the correct size; that the statistical procedures that were used were the correct ones and that they were performed correctly; and that any qualitative data that was collected was analyzed in a complete and unbiased manner.
We’ve been concerned for a long time that evaluation can sometimes appears easier than it really is. After all, we’ve all been interviewed for something in our lives and most of us have completed a questionnaire. Given this situation, you may ask yourself whether you really need a professional evaluator or whether it is simply a matter of outsourcing a set of evaluation related tasks to someone outside your organization. But in order to really answer your organization’s unique questions, you need to be certain that the data upon which those answers are based, represents what is really going on in your program. Only a firm with well-developed research skills can assure you that it is.
How can you assess a firm’s expertise? The best measure of research expertise is specific training in research methodology. And it’s not enough to see a list of advanced degrees among the firm’s principals. The consultant managing your project is the one who has to have the training. Look for degrees, the higher the better, in sociology, evaluation research, psychology and anthropology. Look for experience teaching research in a university setting as well.
One additional note: While most professional programs require a research course or two, few can match the level of training in methodology students receive in the fields mentioned above. That said, there are several excellent post-Master’s certificate programs (TEI, Claremont) designed to give consultants deeper exposure to research. If you can find an individual who has completed one of these you are most likely in very good hands.
Project management skills are important too. Your project actually has to get done– on time and on budget. To succeed here you need a consultant who understand what it takes to get a survey out and follow-up with non-respondents. You need a consultant who can find a venue for your focus group or track down missing participants from a longitudinal study. Without these skills, the best laid plans will never be realized.
How can you assess a firm’s project management skills? We think the best way is to look at their experience. How long have they been in business? How many similar surveys, interviews, focus groups, have they done in the past? Firms that have lots of experience under their belts usually know how to make a study happen. Those are the firms you will want to look at the most closely.
Excellent thinking skills are critical as well. By excellent thinking skills we mean the ability to interpret data in meaningful ways– ways that can offer insights and provoke the kinds of questions that help programs and organizations improve what they do. But it goes beyond interpretive ability. A consultant with excellent thinking skills will understand the contexts and settings within which your organization operates and will be able to work with you, as a thinking partner, in ways that go beyond the findings of a single study. Ultimately, you should be able to rely on such a consultant as a trusted strategic advisor.
How can you assess a consultant’s thinking skills? This one is actually a bit more straightforward than you would think. Education of course plays a role but experience is most important, but it really comes down to their level thoughtfulness. Do they ask meaningful questions? Can they think on their feet? Do you sense that your prospective consultant can see the forest for the trees, the big picture in other words? Try asking each firm you interview who their favorite evaluation scholar is. If they can’t answer that one, if they’re not thoughtful about the field of evaluation, how thoughtful are they likely to be about your organization?
Flexibility is about recognizing that program evaluation takes place in the real world. Evaluation is often underfunded, done under time pressure, and may involve working with stakeholders who are not fully committed to it. These realities require an firm that realizes that evaluation is not your organization’s main business. Flexibility means understanding that evaluation work can have tremendous value even if it is not done perfectly. It means, accepting that as in all things, the perfect should not become the enemy of the good. The key is having the expertise to know where and how to make methodological compromises in order to get the project done. We like Bamberger, Rugh and Mabry’s Real World Evaluation: Working Under Budget, Time, Data and Political Constraints. It has a lot of practical advice to offer on this subject and you can find it here.
How can you assess a consultant’s flexibility? The answer here is pretty straight forward. First of course you ought to be able to get an idea in your initial interview. Does the consultant talk research techniques with you or does he or she ask about your program? The former can sometimes indicate an over emphasis on academic rigor and a concomitant lack of consideration for the kinds of compromises that have to be made to get work done in the real world. One quality often associated with flexibility is comfort with uncertainty. Try simply asking the firms you bring in to interview to describe an evaluation gone awry and ask how they handled the situation. Reference checks are also a great way to assess flexibility.
Collaborative evaluators undertand that no matter how good a job they do collecting data, you know your program better than they do—what makes it tick, what underlying issues it faces. Your knowledge about what works and what doesn’t, your questions and concerns, and your thoughts about how to best manage the evaluation have an important role to play as the evaluation plan develops and the project moves forward. When it comes time to develop an interpretation of the study’s findings, the consultant you work with should have a willingness to include you in the analytical process as well. Such an approach results in richer and deeper answers to the evaluation’s key questions.
While a collaborative approach leads to a better evaluation product, importantly, it also increases the chances that the study’s results will be used. In a collaborative approach, the ownership of findings is shared. Much more than in a ‘go it alone’ model, both you and your evaluator have a shared stake in the project. You understand the report you get at the end of the project because you had a voice in shaping it. Because of such an understanding, you and your team will have more confidence in implementing the study’s recommendations. Collaboration in other words, makes the report actionable.
How do you assess how collaborative a consultant’s is? The best way of course is to ask– both the consultant and his or her references. Two great questions are: “What kind of role do you anticipate we will play in the evaluation project,” and “how do you see that changing as the project unfolds?” If the consultant just tells you that you’ll be helping to arrange interviews or sending out survey reminders, his or her orientation may not be well geared to collaboration.
Excellent communication skills are important because the utilization of evaluation findings depends upon the ability to get them out to program stakeholders in a timely manner and in a format and language they understand. Reports, summaries, etc. need to present data and other evidence that is accessible and easy to navigate. For many nonprofits the 30 page evaluation report may not be the best way to go. Your evaluation consultant needs to be able to understand what kind of communication strategy will work best for your organization. He or she may also need to be open to creating multiple reports for your different stakeholders. For example, program managers may be best served with a report that emphasizes how a program was deployed whereas your development group, in order to fulfill funder requirement, may need one focused on outcomes.
Just as important is the timeliness of the information. Evaluation knowledge makes the greatest contribution to organizational improvement when stakeholders receive it in real time. This lessens the surprises sometimes found in a final report, breaks the content of such a report up into manageable chunks, and generates excitement among stakeholders. A consultant who expects to submit only a final report may not understand this value.
How do you assess a consultant’s communication skills? That’s an easy one, just ask for some writing samples. When you get them, evaluate them for overall tone and readability. Can you see your stakeholders understanding what the consultant has written? Can you see them viewing it as reasonable and accurate? It’s very much a matter of fit here. Some organizations may need fairly formal reports written in a very authoritative tone. Others may do better with something more down-to-earth. If you’d like to learn about best practices for reporting and communications, take a look at Evaluation Strategies for Communicating and Reporting. We use it when we teach evaluation and often refer back to it when we start a new project.
Client oriented means all of the things we have listed so far and more. It has to do, essentially, with how easy it is to work with your consultant. We’ve covered the big things like flexibility and collaborativity, but the little things matter as well, perhaps even more so. A client oriented consultant is one who returns phone calls and emails promptly; one who is trustworthy and accountable; one who understands confidentiality and who has your best interests at heart. He or she is willing to go above and beyond the minimum requirements of the work plan or contract and to get the job done for you on time and within budget.
How do you know if a consultant is client oriented? This is really the hardest thing of all to judge. Of course flexibility and collaborativity play some role, but at the end of the day it’s very much a judgment call. You may get some inkling in the back and forth that typically occurs during the selection process but besides following up with references, you will most likely need to check in with your right brain on this one. Trust your instincts and if in doubt, move on if the relationship doesn’t feel right.
In our final post in this series, we’ll cover the process of locating and hiring an appropriate consultant for your project. Please stay tuned for more.
