The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume IX, Number 2, Summer 2003
Issue Topic: Evaluating Education Reform
Evaluations to Watch
Flexibility and Feedback in a Formative Evaluation
Marjorie Wechsler of SRI and Jane David of the Bay Area Research Group describe the importance of flexibility and feedback in conducting formative evaluation.
How do you evaluate a reform agenda that is constantly adapting to changing circumstances and feedback? That is the challenge we face as formative evaluators for the Bay Area School Reform Collaborative (BASRC). Created in 1995 in response to the Hewlett-Annenberg Challenge,¹ BASRC is a San Francisco-based reform organization dedicated to improving student achievement and closing the achievement gap. Schools and districts receiving BASRC funds participate in regional networking opportunities and inquiry-focused activities around teacher practices, equity, assessment, and leadership. The cornerstone of BASRC is inquiry- and data-driven decision making—not just for its grantees, but for itself. Accordingly, BASRC contracted with the Bay Area Research Group and SRI International to conduct a formative evaluation to help increase its effectiveness.
Given BASRC’s learning stance and the developmental nature of its work, the goal of our formative evaluation is to increase the efficiency and effectiveness of its organizational learning. Ironically, it is BASRC’s learning stance that complicates our task, because as BASRC learns, it adopts new strategies for supporting reform. With the launch of its second five-year funding period in 2001, BASRC changed from focusing on individual schools to providing support for collaboratives of schools and their districts. In 2002 BASRC again shifted its emphasis, providing comprehensive support to five districts and lighter support for school collaboratives in other districts.
Asking the Right Questions
Our research questions not only get at the heart of our formative task, but also remain pertinent in a changing environment. The following overarching questions guide our work: To what extent are BASRC’s strategies to promote and support reform effective? How might they become more effective, through either modifying or better implementing the current strategies? For each of BASRC’s primary strategies, we gather data to answer three questions:
- Is the intent of the strategy clear to BASRC staff and to the field?
- Is the implementation of the strategy consistent with its intent?
- Does the strategy contribute constructively to the progress of reform in schools and districts?
Each question includes an implied “why or why not,” from which we draw inferences about how strategies might be better designed, targeted, or strengthened in practice. These questions are applicable to any new strategies implemented.
Being Flexible and Responsive
Because BASRC’s reform effort is complex and is coupled with a serious attempt to learn along the way and adjust accordingly, plans never roll out exactly as expected. As BASRC adjusts to feedback from the field directly and from us, its plans and strategies change. Our evaluation tasks must change accordingly. For example, when BASRC’s focus on creating collaboratives of schools shifted to concentrating attention on both schools and central offices in fewer districts, our emphasis shifted as well.
Timely Feedback
BASRC develops its annual strategic plan for the following academic year while it is still implementing the plan for the current year. Our feedback must correspond with BASRC’s planning and decision-making cycles. Therefore, an end-of-year report is not sufficient. BASRC also needs just-in-time feedback so it does not waste time or resources on failing strategies and builds on success.
Continual Feedback
We provide feedback in multiple ways throughout the year, including brief memos and occasional reports summarizing data collected from grantees, “real-time” feedback during events, and frequent email and conversations with staff in response to observations or questions. We meet regularly with planning groups and with management teams to present findings, hear reactions to our interpretations, discuss strategies, and solicit their questions to guide subsequent inquiries. Such an approach encourages staff to think about what their next steps will be. The key to making feedback useful is to ensure that there is a mechanism for translating the results into actions.
Our guiding questions and flexible stance have enabled us to provide constructive feedback to BASRC. Some of our influence is tangible. For example, our recommendations contributed to BASRC’s decision to differentiate training for experienced and novice reform coaches who help schools implement inquiry-based practices. Often our influence is subtler. For example, we may steer thinking in a different direction by the questions we raise, or provide validation to support a decision. Either way, our task as formative evaluators is to focus on BASRC’s strategies and actions that are amenable to feedback and improvement, and to direct our energies and BASRC’s attention to the areas where changes will have the biggest impact on its grantees.
For more information visit SRI’s website at: www.sri.com/policy/cep/edreform/BASRC.htm.
¹ In 1995, the Hewlett-Annenberg Challenge was established for public school renewal in the San Francisco Bay area’s six counties. The five-year $100 million grant has been supported by William R. Hewlett, the William and Flora Hewlett Foundation, the Annenberg Foundation, and matching local funds.
Marjorie E. Wechsler
Education Policy Analyst
SRI International
333 Ravenswood Ave.
Menlo Park, CA 94025
Tel: 650-859-4822
Email: marjorie.wechsler@sri.com
Jane L. David
Director
Bay Area Research Group
3144 David Ave.
Palo Alto, CA 94303
Tel: 650-493-4425
Email: jld@bayarearesearch.org