The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume IX, Number 4, Winter 2003/2004
Issue Topic: Reflecting on the Past and Future of Evaluation
Ask the Expert
Michael Scriven on the Differences Between Evaluation and Social Science Research
Michael Scriven is a professor of evaluation at the University of Auckland in New Zealand and a professor of psychology at Claremont Graduate University in California. One of the world’s most renowned evaluators, Dr. Scriven has authored more than 330 publications in 11 fields, including Evaluation Thesaurus, a staple of evaluation literature. Dr. Scriven is a former president of the American Evaluation Association (AEA), and received the AEA’s esteemed Lazarsfeld Award for his contributions to evaluation theory.
How are evaluation and social science research different?
Evaluation determines the merit, worth, or value of things. The evaluation process identifies relevant values or standards that apply to what is being evaluated, performs empirical investigation using techniques from the social sciences, and then integrates conclusions with the standards into an overall evaluation or set of evaluations (Scriven, 1991).
Social science research, by contrast, does not aim for or achieve evaluative conclusions. It is restricted to empirical (rather than evaluative) research, and bases its conclusions only on factual results—that is, observed, measured, or calculated data. Social science research does not establish standards or values and then integrate them with factual results to reach evaluative conclusions. In fact, the dominant social science doctrine for many decades prided itself on being value free. So for the moment, social science research excludes evaluation.¹
However, in deference to social science research, it must be stressed again that without using social science methods, little evaluation can be done. One cannot say, however, that evaluation is the application of social science methods to solve social problems. It is much more than that.
What unique skills needs do evaluators need?
Evaluators need a few special empirical research skills along with a range of evaluative skills. The repertoire of empirical skills mainly includes those used for social science research, with its emphasis on hypothesis testing. But for an evaluator, empirical skills must include more than those required for traditional social science research.
For example, evaluators often need to know how to search for a program or policy’s side effects—a skill that is tremendously important for evaluation, but not for hypothesis testing. For an evaluator, discovering side effects may be what swings the overall evaluative conclusions from bad to good or vice versa.
Evaluative skills also include abilities like determining relevant technical, legal, and scientific values that bear on what is being evaluated, and dealing with controversial values and issues.
Evaluators also need synthesis skills in order to integrate relevant evaluative and factual conclusions. In fact, the ability to synthesize is probably the key cognitive skill needed for evaluation. Synthesis includes everything from making sure that judgments are balanced to reconciling multiple evaluations (which may be contradictory) of the same program, policy, or product (Scriven, 1991).
Why aren’t the differences between evaluation and social science research widely understood or accepted?
One has to understand the difference between a profession and a discipline. Program evaluation began to take shape as a profession during the 1960s and has become increasingly “professional” in the decades since. This progress has mostly involved the development of evaluation tools, the improved application of these tools, the growth of a professional support network, and a clearer understanding of the evaluator’s status and role. This is very different from what it takes to develop into a discipline.
A discipline recognizes the boundaries of a field and its relation to other fields. It has a concept of itself, as well as an appropriate philosophy of operation that defines the logic of that particular field. The recognition that allows a profession to be thought of as a discipline comes well after that profession has developed. For evaluation that recognition has come only recently.
Evaluation’s move toward becoming a discipline was delayed by the prominence of the value-free doctrine in the standard social sciences centering on the assertion that evaluation could not be objective or scientific and therefore had no place as a scientific discipline. It was not until the late twentieth century that this thinking was confronted seriously and its inaccuracies discovered.
While evaluation has been practiced for many years, it is only now developing into a discipline. In this way evaluation resembles technology, which existed for thousands of years before there was any substantive discussion of its nature, its logic, its fundamental differences from science, and the details of its distinctive methods and thought.
In recent years we have begun to see more discussions within the field about evaluation-specific methodology. We are moving toward the general acceptance of evaluation as a discipline, but there is still a long way to go.
Reference
Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage.
¹ Note, however, that this is changing as social science is being asked to be more involved with serious social problems, interventions, or issues. In order to do so, social science will have to incorporate evaluation or evaluative elements.
Julia Coffman, Consultant, HFRP
Email: julia_coffman@msn.com