You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Jennifer Maltby from Boston After School & Beyond describes the evaluation design and goals of the Partners for Student Success initiative.

In fall 2006, Boston After School & Beyond (Boston Beyond) and its partners launched Partners for Student Success (PSS),1 an alignment initiative between the city's out-of-school time (OST) service providers and the Boston Public Schools. PSS aims to improve academic outcomes and promote healthy development for struggling students by increasing the availability of, participation in, and coordination among OST programs. PSS is designed to create change within individual programs and schools, as well as to create systemic change, by promoting greater coordination between schools and OST providers.

PSS's strategies include establishing frameworks and systems to identify strengths and needs of struggling students, matching these students with appropriate OST programs, promoting effective communication and coordination between schools and OST providers, and improving the professional capacity of educators and OST providers to meet students' needs. PSS is also addressing system-wide barriers to participation in OST programs, such as insufficient transportation, and creating an integrated data system that will allow for better communication between schools and OST staff. A fundamental strategy is the creation of a new position within each participating school—the Manager for Extended Learning Services, who coordinates and oversees the implementation of PSS's strategies.

PSS will operate in 5 public elementary schools in 2006–2007 and expand to 15 schools over the next 3 years. Prior to the fall 2006 implementation, Boston Beyond staff and partners engaged in a 9-month design and planning process. Elizabeth Reisner from Policy Studies Associates (PSA) advised the design team on evaluation issues and the creation of an evaluation plan. The plan evolved through a panel discussion with three evaluators experienced in OST research,2 conversations with key stakeholders and funders, and PSS team meetings. PSA began implementation of the evaluation in September.

The PSS team decided to conduct an implementation analysis during the early stages of the initiative. Due to a variety of factors—including the anticipated start-up time for the initiative and the time and dosage that research suggests are required for OST programs to demonstrate developmental and academic outcomes—the team concluded that an outcome evaluation was not appropriate for the initial stage. Instead, an outcome evaluation will be conducted after the implementation analysis. Central to both phases of the evaluation is the theory of change, which proposes that a) barriers to learning can be mitigated by high-quality OST programs and other wrap-around services and b) services that are matched to students' unique characteristics can lead to improved developmental outcomes and ultimately to improved academic outcomes.

The implementation phase of the evaluation calls for a mixed-methods approach, using both quantitative and qualitative methods, across 2 years. The first stage will consist of detailed case studies of the five sites implemented in 2006. These case studies will describe the implementation at each site and identify which sites have implemented the initiative with sufficient adherence to the theory of change. The case study approach will accommodate the unique characteristics and anticipated variation across sites and allow us to learn from the sites' diversity. Data collection methods will include parent and student surveys and focus groups, interviews with school and OST program staff, attendance data, and school records. A second stage of the implementation evaluation will be conducted with the five new sites in 2007. Through these implementation studies, PSS hopes to identify the processes and actions that lead to high-quality implementation of each of PSS's components (for example, improving program quality and increasing communication between school and OST staff).

In fall 2008, the evaluator will begin the outcome evaluation with the sites identified as having effectively implemented the initiative. Based on the theory of change, PSS predicts that sites that do not implement the initiative well cannot expect to achieve long-term outcomes. The outcome study will utilize a quasi-experimental design, which will compare outcomes for students in PSS schools with youth in similar schools that are not implementing PSS. The outcomes measured will be informed by the implementation analysis and may include school and OST program engagement, positive interactions with peers and adults, and academic measures (e.g., grade promotion, test scores). PSS anticipates having preliminary results in the summer of 2007, with final results in the fall of 2009.

1 Partners for Student Success is an initiative of Boston After School & Beyond in collaboration with the City of Boston, Boston Public Schools, The Boston Foundation, Full-Service Schools Roundtable, Harvard University's Program in Education, Afterschool and Resiliency (PEAR), Ludcke Foundation, Massachusetts 2020 Foundation, Nellie Mae Education Foundation, and the Wallace Foundation.
2 Boston Beyond thanks Priscilla Little of Harvard Family Research Project, Beth Miller of Miller-Midzik Research Associates, and Marty Williams of the Center for Research Management for their consultation.

Jennifer Maltby
Knowledge Projects Manager
Boston After School & Beyond
89 South Street, Suite 402
Boston, MA 02111
Tel: 617-345-5322 ext. 22


‹ Previous Article | Table of Contents | Next Article ›

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project