The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume XII, Number 1 & 2, Fall 2006
Issue Topic: Building and Evaluating Out-of-School Time Connections
Evaluations to Watch
An Impact Evaluation of Academic Instruction for After School Programs
Alison Black and Fred Doolittle from MDRC describe the evaluation of an enhanced academic instruction approach for after school programs.
As pressure for students to meet challenging academic standards grows, parents, principals, and policymakers are increasingly turning their attention to the out-of-school hours as an opportunity to provide additional academic support. Indeed, the federal government has made a substantial investment toward this goal through its 21st Century Community Learning Centers (21st CCLC) funding, and the Department of Education (ED) has launched a program of research seeking to strengthen the ability of 21st CCLC-funded after school programs to support academic growth.
An ED early study found that after school programs primarily provided homework help for students and that these services had little or no effect on young people's academic performance, as measured by grades and test scores.1 There are a growing number of academic approaches in after school besides homework help but, to our knowledge, none of these approaches have scientifically sound evidence indicating that they improve academic outcomes. We are currently conducting an evaluation to determine whether there is such evidence for one academically focused approach—adapting and extending an in-school math and reading program to after school settings.
Study Goals
Because the goals of the No Child Left Behind act and ED-funded programs are to improve academic performance, the National Center for Education Evaluation at the U.S. Department of Education's Institute of Education Sciences contracted with MDRC2 to develop a study of new models of academic instruction in after school programs. More specifically, this study examines instructional approaches that are a) adapted from the regular school day and b) diagnostically driven, to provide differentiated instruction on specific topics with which students need the most help. The study examines whether such approaches produce better academic outcomes than typical after school academic supports.
Harcourt School Publishers was selected to adapt and extend their existing in-school math program for use in after school programs, while the Success for All Foundation was chosen to develop reading program materials. These materials are being implemented in a randomized study begun in the 2005–2006 school year and continuing through the 2006–2007 school year, in which half of the students are receiving the enhanced academic instruction and half are participating in the services regularly provided by their programs.
The key research questions of this study are:
- What is the impact of the enhanced after school program on student academic performance?
- What is the impact of the enhanced after school program once the program has been in operation for a year, when after school teachers have more experience?
- What is the impact on student academic performance after 2 years of program implementation?
Research Methodology
Fifty after school centers—most are 21st CCLC grantees—were selected to test either the reading or math program among second- through fifth-grade students. We selected centers that could provide a “fair test” of the new math/reading program.3 Key selection criteria included: a) the program's ability to implement the models with reasonable fidelity; b) the expectation of a clear service contrast between the program and control groups; and c) the program's track record of serving low-performing students, having reasonably stable program funding and operation, and having appropriately qualified staff.
Within these programs, students were recruited who had already enrolled in the program and whom staff identified as performing below grade level and in need of academic support. Of this group, students whose parents agreed to allow them to participate were randomly assigned to either the enhanced or regular after school program for the first 45 minutes of the program. All study sites, therefore, had students in both the enhanced and control conditions.
Prior to random assignment, all students completed the reading or math Abbreviated Battery of the Stanford Achievement Test (SAT10), and parents completed application and informed consent forms. After random assignment, the following data were collected for all students, in order to examine implementation and estimate differences across groups:
- Attendance data, to examine the intensity of participation
- Field research on program implementation, to understand the methods and context of implementation, the service contrast between the enhanced and control groups, and implementation lessons
- Student surveys, to understand reactions to the after school program and any other academic supports students may have received
- Program staff surveys, to understand the background and experience of staff providing academic support
- School-day teacher surveys, to understand the students' academic performance and any special support they received during the school day
- Abbreviated battery of the SAT10 (for all students) and the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) test (for second and third graders in reading sites), to measure impacts on achievement scores
A report addressing the key questions for the first year of the study will be released in fall 2007.
1 Dynarski, M., Pistorino, C., Moore, M., Silva, T., Mullens, J., Deke, J., et al. (2002). When schools stay open late: The national evaluation of the 21st-Century Community Learning Centers Program. Washington, DC: U.S. Department of Education, Office of the Under Secretary; James-Burdumy, S., Dynarski, M., Moore, M., Deke, J., Mansfield, W., & Pistorino, C. (2005). When schools stay open late: The national evaluation of the 21st Century Community Learning Centers Program: Final report. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.
2 The evaluation is being conducted by MDRC in collaboration with Bloom Associates, Public/Private Ventures, and Survey Research Management.
3 This analysis will not attempt to generalize statistically beyond the observed sample of centers because sites were not picked to be nationally representative.
Alison Black
Research Associate
Tel: 212-340-8827
Email: alison.black@mdrc.org
Fred Doolittle
Vice President
Tel: 212-532-3200
Email: fred.doolittle2@mdrc.org
MDRC
16 East 34th Street, 19th floor
New York, NY 10016