Search Publications |  The Evaluation Exchange |  FINE Network |  OST Database

HFRP > Evaluation > The Evaluation Exchange > Issue Archive > Community-Based Initiatives > Implementing a Theories of Change Evaluation in the Cleveland Community-Building Initiative

Volume lll, Number 3&4, 1997

Issue Topic: Community-Based Initiatives

Promising Practices

Implementing a Theories of Change Evaluation in the Cleveland Community-Building Initiative

Sharon Milligan, Claudia Coulton, and Peter York of the Center on Urban Poverty and Social Change and Ronald Register, Executive Director of the Cleveland Community-Building Initiative, explain how a theories of change approach can be used to address the constraints of traditional evaluation techniques.¹

Comprehensive community initiatives have proven difficult to evaluate because they do not lend themselves to traditional experimental methods. Many audiences, however, are interested in whether these efforts are actually creating change and improving the lives of residents. A theories of change approach offers promise as a means to address the constraints of traditional evaluation techniques.

A theories of change approach makes explicit the short- and long-term outcomes of an initiative, the strategies pursued to achieve them, and the linkages between these. By doing so, it offers several advantages over traditional experimental methods. First, it can be applied to whole community interventions in which untreated control groups are not possible. Second, it makes explicit many of the assumptions about the ingredients of community and system change and how these are expected to improve conditions for residents and their local institutions. Finally, by tracking progress on the steps in the change process, it can provide corrective feedback that can help distinguish theory failure from implementation failure.

The Cleveland Community Building Initiative (CCBI), in collaboration with the Center on Urban Poverty and Social Change, has begun the process of implementing a theories of change approach to evaluation. CCBI seeks to strengthen communities and reverse persistent poverty in four areas of Cleveland by supporting individualized approaches shaped by community members. Each of the four areas, or villages, has a council which is representative of various stakeholders and is charged with developing action plans to address poverty in the local neighborhood. Village coordinators work with the village councils to develop and maintain the councils and to provide technical assistance. The CCBI Board of Trustees is responsible for supporting villages and providing linkages to necessary financial, intellectual, and technical expertise to implement the village-based agendas. The CCBI executive director has responsibility for managing CCBI's overall operations.

The components of the CCBI evaluation include: articulation of the theories of change; identification of key benchmarks for strategies and outcomes; collection of data; analyses to determine whether the theories need to be refined; modification of theories as needed; and provision of regular feedback to stakeholders on progress and results. Little is currently known about the procedures that should be used in identifying theories of change and in selecting elements of the theories to track in the evaluation. Since CCBI now has experience with these two steps in the evaluation process, it is possible to describe the methods used, their results, and lessons learned.

The CCBI evaluators used a four-step process to define CCBI's theories of change:

1. Determining key stakeholders whose theories need to be elicited: CCBI evaluators identified a number of key stakeholders who could help articulate the initiative's theories of change—the CCBI board and staff members, village councils and council coordinators, and the CCBI executive director.

2. Eliciting theories of change from these designated stakeholders: CCBI evaluators used various methods to help stakeholder groups to articulate the initiative's theory of change. Theory development began by soliciting the opinions of CCBI staff members through small group interviews; a separate interview with the executive director yielded his theory of change. The interviews consisted of a series of questions designed to elicit theories of change—beginning with conversation about short-term strategies and outcomes and then moving to intermediate and long-term outcomes. Evaluators diagrammed these theories, solicited feedback, and made modifications.

   The evaluators interviewed each board member separately to solicit his/her theory of change, including outcomes and accomplishments each viewed as important to CCBI, the specific activities each felt served as the means to achieve these, and the linkages or relationships among them. These individual theories were consolidated into one initial draft that was shared with the board and subsequently modified.

   Focus groups were used to solicit the theories of change from each of the four village councils and where possible, evaluators tried to link the theory development process with the village's strategic planning process. Again, the theories were shared with the groups for refinement.

3. Examining the contributing stakeholders' theories for common and unique elements: Once the stakeholders' theories were developed, the CCBI evaluators had to compare and reconcile them. The evaluators incorporated the executive director's theory with those of the staff members—where common elements existed among the theories, no changes were made to the staff members' theories; strategies and outcomes suggested by the director but not included in the staff members' theories were added to the latter. The refined model was then shared with the staff members and the director, and further refinements made. A similar approach was used to reconcile the board members' theories, which again was shared with the members and further refined.

4. Agreeing on the theory or theories which will guide the evaluation: The next task was to compare the theories among stakeholder groups to develop one theory for the initiative. Evaluators found that the staff members' and board members' theories contained more similarities than differences. Furthermore, village councils' theories seemed to provide more detail to the theories of the other two stakeholder groups.

The CCBI evaluation is now moving toward evaluating the early parts of the theory since these now have sufficient detail and consensus. The evaluators recognized that there was a need to simplify the theory in order to identify benchmarks by which to assess progress. The simplification process consisted of grouping elements into major strategies and outcomes, each of which could then be further specified for evaluation. The evaluators identified numerous benchmarks for each strategy that would need to be reviewed by stakeholders so they would be aware of and agree upon these visible signs of progress. CCBI staff and evaluators are also beginning to consider the possible indicators that will be used to examine long-term outcomes of the initiative as well as identifying possible data sources.

In trying to operationalize the theories of change concept in a community setting, the CCBI evaluation offers these preliminary lessons:

• Evaluators must take on new roles: A theories of change approach thrusts evaluators into new, less traditional roles. In this evaluation, evaluators as well as CCBI staff members, board members, and village councils were all collaborators and co-discoverers. The evaluator is a trainer and a facilitator as well as a technical advisor.

• Stakeholders must be committed: Theory development is an iterative process and requires substantial time and energy if it is to work. The stakeholders in the CCBI evaluation made a substantial time commitment to work with the evaluators throughout the iterative process to define the assumptions underlying the initiative

• Group processes work best in theory development: The use of a group process not only helps to build consensus and to yield a common theory, but it also allows group members to see each other's perspectives more clearly. The CCBI experience suggests that groups be smaller than eight persons.

• Being explicit about the detailed steps along the pathway of change is difficult: Many missing links exist between the early strategies and long-term outcomes. While stakeholders have a fairly clear vision of what they want to see in their communities, the steps that would be necessary to achieve these outcomes are not yet clear. The CCBI evaluators anticipate that once members and village residents experience success with initial action projects, they will be able to revisit the long-term outcomes and begin to make choices about the preconditions for change.

• The distinction between process and outcomes is difficult to apply: The early work of CCBI, as most community initiatives, involves putting processes and structures into place that stakeholders believe will build the community. Within the theory, these are identified as strategies, the accomplishment of which is an early outcome. This might be, for example, putting into place a recruitment process for village council members which contributes to village council formation and operation. CCBI evaluators have tried to phrase many of the processes in terms of short-term outcomes that could be observed at a point in time as signs that they are accomplished well.

• Establishing standards and thresholds is difficult: In the field of CBIs, values and goals are not stated as clear thresholds or levels. The experience of practitioners may be a useful source of a threshold for what works, but experience is not always codified or readily available to evaluators. A knowledge base in this area needs to be developed.

• To be compelling, models will need to be fully specified: Frequent feedback of findings and opportunities to modify the theory will prove useful for program improvement. This information should help provide staff and residents the evidence they need to distinguish between strategies that are incompletely implemented and strategies that were done according to the standards but did not produce the desired changes in the community. It is less clear how to build a compelling case that the theory itself is powerful and valid—as it now stands, the theory is not adequate to account for competing explanations and the influence of factors is not included in the model. In the early stages of the evaluation, where most of the predicted effects are internal to the initiative itself, this is not as problematic. The later community and individual-level changes that are predicted, however, are much less internally controlled. Therefore, there is a need to specify the model fully so that the influence of external factors can be ruled out or explicitly brought into the change process.

The CCBI evaluation is a work in progress. This early work has helped identify both the struggles and the opportunities that can derive from using a theories of change approach. While considerable work remains to be done to address important methodological concerns, the early experience of CCBI demonstrates that the collaborative, iterative process of developing, implementing, and refining theories of change is a good fit with the way that CBIs work.

This article is based on a longer piece of the same title which will appear in New approaches to evaluating community initiatives, volume II: Theory, measurement and analysis, (1997). Karen Fulbright-Anderson, Anne C. Kubisch, and James P. Connell (Eds.), Washington, DC: Aspen Institute. For information, please contact: The Aspen Institute, Publications Office, P.O. Box 222, Houghton Lab Lane, Queenstown, MD 21658. Fax: 410-827-9174.

¹ While the term “theory of change” is typically used to identify the approach described herein, we use the term “theories of change” to acknowledge that multiple theories about the process of social change often operate.

Sharon Milligan, Co-Director

Claudia Coulton, Co-Director

Peter York, Research Assistant

Center on Urban Poverty and Social Change
Case Western Reserve University

Ronald Register
Executive Director
Cleveland Community-Building Initiative