John Gargani has put together a humorous and very thoughtful guide to assessment of evaluation quality. Stakeholders he says, view an evaluation as high quality of it supports their program. They find numerous flaws in the evaluation if, on the other hand, it comes up with negative findings. You’ll find it here. We’d like to add a few of our own:
Stakeholder’s Second Law of Evaluation Quality
There exists an positive relationship between the amount of stakeholder involvement in an evaluation project and the amount of time it takes to complete the project.
The Funder’s Outcome Paradox
The greater the funder’s insistence on quantitative outcomes, the higher the statistical significance of the outcomes AND the lover practical significance or effect size they are likely to have. For example, an evaluator found that students exhibited a statistically significant increase in knowledge of the funder’s hobby horse curriculum. The actual differences however were meaningless. Students went from a ‘grade’ of 45% to a grade of 58%. In others words nobody passed the course even with the intervention.
Corollary to the Funder’s Outcome Paradox
Funders will remember only those aspects of college statistics that show their programs in a positive light. Statistical significance trumps all else when it is observed. (Funders were in school before statistics classes covered the topic of effect size.) Speaking of effect size, check out this site. It’s the best we’ve found on this very important topic in evaluation. For somewhat more detail, try this. And be sure to look for more discussions of effect size in the future.