The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume IX, Number 4, Winter 2003/2004
Issue Topic: Reflecting on the Past and Future of Evaluation
Promising Practices
Using Narrative Methods to Link Program Evaluation and Organization Development
Charles McClintock, Dean of the Fielding Graduate Institute’s School of Human and Organization Development, illustrates how narrative methods can link a program’s evaluation and its organization development.
The field of program evaluation has evolved over the past half century, moving from focusing primarily on research methods to embracing concepts such as utilization, values, context, change, learning, strategy, politics, and organizational dynamics. Along with this shift has come a broader epistemological perspective and wider array of empirical methods—qualitative and mixed methods, responsive case studies, participatory and empowerment action research, and interpretive and constructivist versions of knowledge.
Still, evaluation has remained an essentially empirical endeavor that emphasizes data collection and reporting and the underlying skills of research design, measurement, and analysis. Related fields, such as organization development (OD), differ from evaluation in their emphasis on skills like establishing trusting and respectful relationships, communicating effectively, diagnosis, negotiation, motivation, and change dynamics. The future of program evaluation should include graduate education and professional training programs that deliberately blend these two skill sets to produce a new kind of professional—a scholar-practitioner who integrates objective reflection based on systematic inquiry with interventions designed to improve policies and programs (McClintock, 2004).
Narrative methods represent a form of inquiry that has promise for integrating evaluation and organization development. Narrative methods rely on various forms of storytelling that, with regard to linking inquiry and change goals, have many important attributes:
- Storytelling lends itself to participatory change processes because it relies on people to make sense of their own experiences and environments.
- Stories can be used to focus on particular interventions while also reflecting on the array of contextual factors that influence outcomes.
- Stories can be systematically gathered and claims verified from independent sources or methods.
- Narrative data can be analyzed using existing conceptual frameworks or assessed for emergent themes.
- Narrative methods can be integrated into ongoing organizational processes to aid in program planning, decision making, and strategic management.
The following sketches describe narrative methods that have somewhat different purposes and procedures. They share a focus on formative evaluation, or improving the program during its evaluation, though in several instances they can contribute to summative assessment of outcomes. For purposes of comparison, the methods are organized into three groups: those that are relatively structured around success, those whose themes are emergent, and those that are linked to a theory of change.
Narratives Structured Around Success
Dart and Davies (2003) propose a method they call the most significant change (MSC) technique and describe how it was applied to the evaluation of a large-scale agricultural extension program in Australia. This method is highly structured and designed to engage all levels of the system from program clients and front-line staff to statewide decision makers and funders, as well as university and industry partners. The MSC process involves the following steps:
- Identify domains of inquiry for storytelling (e.g., changes in decision-making skills or farm profitability).
- Develop a format for data collection (e.g., story title, what happened, when, and why the change was considered significant).
- Select stories by voting at multiple levels (e.g., front-line staff, statewide decision makers and funders) on those accounts that best represent a program’s values and desired outcomes.
- Conduct a content analysis of all stories (including those not selected in the voting) in relation to a program logic model.¹
As described by Dart and Davies (2003), one of the most important results of MSC was that the story selection process surfaced differing values and desired outcomes for the program. In other words, the evaluation storytelling process was at least as important as the evaluation data in the stories. In addition, a follow-up case study of MSC revealed that it had increased involvement and interest in evaluation, caused participants at all levels to understand better the program outcomes and the dynamics that influence them, and facilitated strategic planning and resource allocation toward the most highly valued directions. This is a good illustration of narrative method linking inquiry and OD needs.
A related narrative method, structured to gather stories about both positive and negative outcomes, is called the success case method (Brinkerhoff, 2003 [A review of this book is available in this issue.]). The method has been most frequently used to evaluate staff training and related human resource programs, although conceptually it could be applied to other programs as well.
This method has two phases. A very short email or mail survey is sent to all program participants to identify those for whom the training made a difference and those for whom it did not. Second, extreme cases are selected from those two ends of the success continuum and respondents are asked to tell stories about both the features of the training that were or were not helpful as well as other organizational factors that facilitated or impeded success (e.g., support from supervisors and performance incentives). Based on the logic of journalism and legal inquiry, independent evidence is sought during these storytelling interviews that would corroborate the success claims.
The purpose of the success case method is not just to evaluate the training, but to identify those aspects of training that were critical—alone or in interaction with other organizational factors. In this way, the stories serve both to document outcomes, but also to guide management about needed organizational changes that will accomplish broader organizational performance goals. Kibel (1999) describes a related success story method that involves more complex data gathering and scoring procedures and that is designed for a broader range of human service programs.
Narratives With Emerging Themes
A different approach to narrative methods is found within qualitative case studies (Costantino & Greene, 2003). Here, stories are used to understand context, culture, and participants’ experiences in relation to program activities and outcomes. As with most case studies, this method can require site visits, review of documents, participant observation, and personal and telephone interviews. The authors changed their original practice of summarizing stories to include verbatim transcripts, some of which contained interwoven mini stories. In this way they were able to portray a much richer picture of the program (itself an intergenerational storytelling program) and of relationships among participants and staff, and they were able to use stories as a significant part of the reported data.
Nelson (1998) describes a similar approach that uses both individual and group storytelling in evaluating youth development and risk prevention programs. The individual stories elicit participant experiences through a series of prompts, while the group stories are created by having each group member add to a narrative about a fictitious individual who participates in the program and then has a set of future life outcomes. Group storytelling is a means of getting at experiences an individual is reluctant to claim or at material that might not be accessible to conscious thought.
Both of these approaches can result in wide differences in the quality and detail of the stories. Especially with group storytelling, the narrative can become highly exaggerated. The point of narrative in these instances is not so much to portray factual material as it is to convey the psychological experience of being in the program. Analysis can take many forms, depending on the conceptual framework or evaluation contract, and can include thematic coding, verbatim quotes, and narrative stories as the substance of the analysis.
Narratives Linked to a Theory of Change
The previous uses of narrative emphasize inquiry more than OD perspectives. Appreciative inquiry (AI) represents the opposite emphasis, although it relies heavily on data collection and analysis (Barrett & Fry, 2002). The AI method evolved over time within the OD field as a form of inquiry designed to identify potential for innovation and motivation in organizational groups. AI is an attempt to move away from deficit and problem-solving orientations common to most evaluation and OD work and move toward “peak positive experiences” that occur within organizations. AI uses explicitly collaborative interviewing and narrative methods in its effort to draw on the power of social constructionism to shape the future. AI is based on social constructionism’s concept that what you look for is what you will find, and where you think you are going is where you will end up.
The AI approach involves several structured phases of systematic inquiry into peak experiences and their causes, along with creative ideas about how to sustain current valued innovations in the organizational process. Stories are shared among stakeholders as part of the analysis and the process to plan change. AI can include attention to problems and can blend with evaluation that emphasizes accountability, but it is decidedly effective as a means of socially creating provocative innovations that will sustain progress.
This brief overview of narrative methods shows promise for drawing more explicit connections between the fields of program evaluation and OD. In addition, training in the use of narrative methods is one means of integrating the skill sets and goals of each profession to sustain and improve programs.
¹ A logic model illustrates how the program’s activities connect to the outcomes it is trying to achieve.
References
Barrett, F., & Fry, R. (2002). Appreciative inquiry in action: The unfolding of a provocative invitation. In R. Fry, F. Barrett, J. Seiling, & D. Whitney (Eds.), Appreciative inquiry and organizational transformation: Reports from the field. Westport, CT: Quorum Books.
Brinkerhoff, R. O. (2003). The success case method: Find out quickly what’s working and what’s not. San Francisco: Berrett-Koehler Publishers.
Costantino, R. D., & Greene, J. C. (2003). Reflections on the use of narrative in evaluation. American Journal of Evaluation, 24(1), 35–49.
Dart, J., & Davies, R. (2003). A dialogical, story-based evaluation tool: The most significant change technique. American Journal of Evaluation, 24(2), 137–155.
Kibel, B. M. (1999). Success stories as hard data: An introduction to results mapping. New York: Kluwer/Plenum.
McClintock, C. (2004). The scholar-practitioner model. In A. DiStefano, K. E. Rudestam, & R. J. Silverman (Eds.), Encyclopedia of distributed learning (pp. 393–396). Thousand Oaks, CA: Sage.
Nelson, A. (1998). Storytelling for prevention. Evergreen, CO: The WHEEL Council.
Charles McClintock, Ph.D.
Dean
School of Human and Organizational Development
Fielding Graduate Institute
2112 Santa Barbara Street
Santa Barbara, CA 93105
Tel: 805-687-1099
Email: cmcclintock@fielding.edu