The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume VIII, Number 1, Spring 2002
Issue Topic: Family Support
Evaluations to Watch
Strengthening Programs and Summative Evaluations Through Formative Evaluations
Two evaluators from SRI describe the benefits realized by the Parent Institute for Quality Education when they prefaced their summative evaluation with a formative evaluation.
In 1999, SRI International (SRI) received support from the Stuart Foundation to conduct an in-depth formative and summative evaluation of the Parent Institute for Quality Education (PIQE). PIQE is a not-for-profit community organization in California that aims to increase parents’ knowledge and skills to support their children’s academic achievement and ultimate enrollment in postsecondary education. In its 15-year history, PIQE has graduated more than 200,000 parents from its eight-week Parent Institute, many of whom were new immigrants to the United States.
PIQE, as a learning organization, values the opportunity to partcipate in both formative evaluations (which enable them to judge the worth of the program during the program activities) and summative evaluations (which enable them to judge the program at the end of the activities). The formative evaluation discussed below has allowed PIQE to assess and strengthen its current practices. Originally scheduled to last only one year, the formative evaluation was expanded to two and a half years to allow time for PIQE to implement program refinements based on the evaluation findings and to allow for a second round of observation and feedback on the new methods and materials.
This article describes how SRI and PIQE worked together on the formative evaluation and how it supported refinement of the program, identification of promising practices, and articulation of child, parent, and teacher outcomes, which will be measured in the summative evaluation.
Developing a “Theory of Change”
As an initial step in the evaluation, SRI assisted PIQE in articulating a “theory of change” to describe how PIQE expects the program components (e.g., the eight-week class, the teacher workshop) to produce their desired short-term and long-term outcomes (e.g., improved parent and teacher attitudes and behaviors, child educational achievement). All observation and interview protocols used in the evaluation have been based on this theory of change and developed collaboratively with PIQE staff.
The first year of formative evaluation (the observations and interviews discussed below) raised several questions about the goals and strategies outlined in the first conceptualization of PIQE’s theory of change. As a result, PIQE revised their theory of change and articulated new measurable outcomes for parents, teachers, and students. If we had not had the opportunity to refine the theory of change through a formative evaluation, the summative evaluation would not have included measurement of some activities and outcomes now considered essential for program success. In addition to supporting a stronger summative evaluation design, collaboration around developing and refining the theory of change helped PIQE to refine its program objectives and description and to identify implementation areas that are crucial for program success.
Comprehensive Observations and Interviews
SRI conducted observations of six complete eight-week sessions of the Parent Institute, as well as interviews of host school principals and teachers, parent participants, and program facilitators. SRI and PIQE staff met several times to discuss the findings and the ways SRI could provide further feedback or support.
SRI also developed a draft set of benchmarks (milestones that would indicate that the desired changes had occurred) to help support PIQE staff in tracking their progress. PIQE staff reviewed the draft benchmarks and helped refine them so they could be used to manage their change process. Program staff have used the formative evaluation findings and data collection tools in several ways. For example, they have clarified lesson objectives, developed sample lesson plans, revised program curriculum, created new facilitator training materials, and revised the program operations manual.
Real-Time Feedback
To examine the extent to which the program refinements were successfully being implemented, PIQE staff asked SRI to conduct additional “real-time” observations (provide feedback on each class observed within a day or two of the observation so that they could use the information to help other instructors who were about to teach the same lesson). SRI provided real-time feedback on 10 class sessions, each taught by a different instructor, during the 2000-2001 school year. After each observation, SRI researchers distributed detailed observation notes and held telephone debriefing meetings with PIQE staff to discuss the notes and give immediate verbal feedback. SRI produced a summary of the real-time feedback with descriptions of what did and did not work in program implementation for PIQE to use as an instructor training tool.
Real-time feedback allowed SRI to look at parts of the program that PIQE wished to have examined in more detail and areas in which PIQE was specifically trying to apply changes based on earlier feedback. The real-time feedback was extremely useful, but it would not have been possible or as useful without the earlier comprehensive observations and interviews. The combination of these two approaches worked extremely well.
The Value of the Formative Evaluation Method
Formative evaluation is often overlooked when organizations believe that their programs are already running according to plan or when resources are limited. Our experience with the PIQE program, however, suggests that formative evaluation is valuable for mature as well as new programs. In addition, we found that our formative work has built a stronger foundation for the summative evaluation scheduled to begin in May 2002. We hope that our experience encourages others involved in program evaluations to include formative research methods in their evaluations whenever possible.
Shari Golan, Ph.D.
Dana Petersen, M.A., M.P.H.
SRI International
333 Ravenswood Avenue
Menlo Park, CA 94025
650-859-2000
shari.golan@sri.com
dana.petersen@sri.com
www.sri.com/policy/cehs