Volume lll, Number 1, 1997

Issue Topic: Methodology

Questions & Answers

Interview With Robert K. Yin

Robert K. Yin is president of COSMOS Corporation, an evaluation research firm based in Bethesda, Maryland. Dr. Yin's extensive publications on case study methodology include Case Study Research and Applications of Case Study Research. COSMOS is currently using case study methodology in three of four of its evaluations of community-based collaborative initiatives. We asked Dr. Yin to share with us some of his insights about the use of case study methodology in evaluating such initiatives.

How can case study methodology be used to study the effectiveness of comprehensive and collaborative community-based initiatives?

As evaluators of program effectiveness, we are often faced with the challenge of identifying why and how interventions lead to observed results or outcomes. Case study methodology, by investigating phenomena in their real-life context, can be a very important tool in opening the “black box” of how interventions and program effectiveness are linked. This is an advantage over traditional experimental and quasi-experimental designs which may measure outcomes and some process variables but fall short in dealing with the dynamic that is inherent in community-based collaborative initiatives.

Our case study work uses a tool called the “logic model.” The logic model outlines the cause and effect steps that link interventions with expected outcomes. It thus lays out the mystery of the “black box” as a set of linkages and hypotheses about how a collaborative really works. These hypotheses can then be “tested” using both quantitative and qualitative data.

The logic model concept is not new, as it was first used in evaluability assessment. However, we are using this approach to address one of the perennial challenges evaluators face—determining causality. We use the logic model also to help us identify conflicting models, or rival hypotheses, to use the traditional evaluation term. In traditional evaluation designs, comparison and control groups are used to establish causality. In community-based initiatives, however, it is virtually impossible to identify comparisons, much less control groups. We use the logic model to help us identify alternative hypotheses about both processes and outcomes and collect data on these as well. We recognize that this approach does not lead to results having the same level of certainty as the perfect experiment, but it can help us to begin to pick things apart and understand what is going on in these very complex initiatives.

What role can the case study evaluator play in providing formative input to community-based initiatives?

The logic model, as we apply it, facilitates a participatory approach. We believe everyone's practice is based on a theory about how things work. The logic model enables practitioners to get these theories, as well as rivals, out in the open. Such an approach may reveal that a theory is flawed, that there are gaps in interventions and what they are expected to yield. This exercise thus helps to build a better theoretical model. We use evaluation workshops and include the evaluators, project director, and staff in the development of the model. We find as well that this process serves an important program development function. This process enables us to refine the theories as we learn more about them and can also make evaluation a better experience.

While there might be concerns about the objectivity of results that come from a more participatory evaluation, I think that there is an increasing yielding of the idea of the distant third-party evaluator. However, the participatory approach should not be seen as corrupting the evaluator's role. Although the evaluator interacts more with those involved in a program, he/she is committed to collect all relevant data and make conclusions on that basis. The participation of others in the process early on helps them to understand that in this case, the evaluator plays both a program development and an assessment role. In many cases, we find that practitioners are surprised that those who are typically viewed as third-party evaluators can play an important role in providing ongoing feedback.

What are the methodological issues one needs to consider when using case study methodology to examine community-based initiatives in multiple sites?

The important aspect of these evaluations is the role of the rival hypotheses. If you don't think about the alternative hypotheses that might explain what you are seeing, it won't work. The articulation of rival hypotheses, as discussed earlier, helps to improve the validity of the evaluation.

One also needs multiple evidence. There are many sources of evidence that can be used for evaluation, including surveys, archival data, and observations. One needs to use these in a converging manner and be able to understand and address their various strengths and weaknesses.

In evaluating multiple sites, the issue that frequently comes up is whether one or several case studies need to be conducted. I believe that one can test theories with a single case. One well-known use of a single case to test three theories is Graham Allison's analysis of the Cuban Missile crisis (Essence of Decision).

Multiple cases strengthen single-case results. One has to understand that in case study design we are talking about small numbers; two cases rather than one strengthen the results greatly, while six cases rather than two strengthen the results manifold. One way to use multiple case design is to find sites that operate under different hypotheses in reaching the same goal. For example, if there are three theories—A,B,C—about different interventions to reduce violence in the community, we might choose Site 1 because it is addressing the goal primarily using Theory A; we might choose Site 2 because it is primarily using an approach based on Theory B; etc. By collecting data on all three interventions at all three sites, we should begin to understand the contributions of the different interventions. Such a design, while difficult to implement, does help us to begin to get at the challenge of causality.

Karen Horsch, Research Specialist, HFRP

Cami Anderson, Research Assistant, HFRP