Volume XI, Number 3, Fall 2005

Issue Topic: Democratic Evaluation

Evaluations to Watch

Untangling Logic Models and Indicators: Reflections on Engaging Stakeholders

Seema Shah, a researcher at the Institute for Education and Social Policy, shares her experience of engaging community organizing groups to develop a logic model on how community organizing leads to better student outcomes.

Three years ago, the Institute for Education and Social Policy at New York University undertook the massive project of studying community organizing efforts on school reform in eight cities over 6 years.¹ Using a mix of qualitative and quantitative methodologies, we have based our work on the principle that quality research that should be collaborative (engaging the communities under study), contextual (designed to adapt to local conditions), and utilization-focused (capable of being used by communities).

Developing and Refining the Logic Model
Our initial fieldwork, conducted during the first 2 years of our study, included conversations with organizing staff and parent leaders at each of the study sites about their organizations' theories of change. These theory-of-change conversations focused on each group's methodology, in particular the critical organizational activities that lead to the achievement of the organization's social change goals. As an outgrowth of this fieldwork, we went on to create an overarching logic model that illustrated how community organizing activities lead to particular improvements in school capacity for effective instruction and how these improvements, in turn, increase the likelihood of improved student achievement. We published this proposed logic model in our report Constituents of Change.² Though the report was the result of iterative conversations with the sites, we sought to use the publication not as an “end product” but as a catalyst for further dialogue.

In the fall of 2004, at the start of the study's 3rd year, we sought to foster continued dialogue through a research symposium at which community organizers and leaders from the eight study sites revisited the model and highlighted the aspects that both fit and clashed with their experiences. To facilitate conversation among symposium participants, we distributed blank, poster-sized templates of the logic model. Using the proposed logic model as a reference, groups filled in the organizing activities and outcomes most relevant to their individual organizations on the templates. This dialogue helped us refine the model and enabled us to launch the next phase of the research—delineating a set of relevant indicators to assess the outcomes identified in the model. The symposium built on our previous attempts to get feedback by providing a valuable opportunity for face-to-face interaction among stakeholders.

Grappling With Outcome Indicators
Our original model had two key sets of outcomes—those that occur for the community organizing group and those that occur within schools and school districts. Initially, we focused on the changes that increase the school's capacity to deliver an effective instructional program. Based on the logic model and informed by the literature, our research team identified and documented each organizing group's campaigns and corresponding indicators of change. Toward the end of the study's 3rd year, we again engaged community organizing groups as we worked to develop and refine our outcome indicators. Because organizers and leaders had already engaged in developing and fine-tuning the initial logic model, grappling with the indicators represented a natural next step.

During this phase of the project, our conversations with representatives from the sites added nuance and specificity to the indicators framework. For example, one of our proposed indicators was defined as “a decrease in disciplinary actions” within the school. Through our conversations with organizers, however, we learned that in some cases schools were too lax in enforcing discipline, while others were too strict. Consequently, we revised this indicator to read as “more appropriate implementation of discipline policies” in order to reflect the organizing group's work more accurately. While most of our conversations occurred with organizing staff, many of the organizers to whom we spoke said that they intended to share the spreadsheets documenting the indicators with their parent and youth leaders for additional feedback. This feedback, when relayed back to us, will allow the indicators framework to reflect multiple layers of input

Developing a set of indicators rooted in the original logic model has proven to be a time-consuming and detail-oriented task. Yet it is clear that as our research team learns the language of organizing, the organizing groups in our study are also internalizing the vocabulary of research. Now, 4 years into our 6-year project, we believe that this mutual learning process is due, in large part, to an approach that strives to be conscious of collaboration, context, and utilization. As a result, we have been able to generate a logic model and an indicators framework grounded in both research and practice—one that will authentically inform the next steps of our research.

¹ This research is generously funded by the Charles Stewart Mott Foundation.
² Mediratta, K. (2004). Constituents of change: Community organizations and public education reform. New York: New York University, Institute for Education and Social Policy. steinhardt.nyu.edu/iesp/publications/pubs/cip/ConstituentsofChange.pdf

Seema Shah, Ph.D.
Project Director
New York University Institute for Education and Social Policy
82 Washington Square East, 7th Floor
New York, NY 10003
Tel: 212-998-5887
Email: seema.shah@nyu.edu