You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Research Description

Overview and Components This meta-analysis synthesized 35 out-of-school time (OST) program studies that employed control or comparison groups to estimate program effects for students at risk of failure in reading or math.
Start Date May 2003
Scope national
Type after school, summer/vacation
Location urban, suburban, rural
Setting varied, including but not limited to public schools, college campuses, and community-based organizations
Participants kindergarten through high school students
Number of Sites/Grantees not applicable
Number Served not applicable
Study Details In order to be included in the analysis, studies had to: (a) concern an OST program for K–12 students; (b) be published or reported in or after 1985 and implemented in the United States; (c) include some type of direct assessment of students’ academic achievement in reading, math, or both (e.g., standardized tests, classroom assessments, or grades); (d) examine the effectiveness for students at risk for school failure (low performance on achievement assessments or characteristics typically associated with lower achievement and school dropout); (e) include a control or comparison group (i.e., a group of youth who did not participate in the OST program under investigation, whose achievement results were compared with those for youth who did participate); (f) disaggregate results for specific OST programs; (g) not be designed for and delivered only to special populations such as special education students, English language learners, or migrant students; and (h) have sufficient quantitative information for calculation of effect sizes (ES), defined as a measure of the magnitude of the relationship between participation in the program and outcomes. Studies could be published or unpublished and could include evaluation reports, conference presentations, and dissertations.
Funding Level not available
Funding Sources United States Department of Education
Researchers Patricia A. Lauer, Rocky Mountain Center for Health Promotion and Education

Motoko Akiba, University of Missouri, Columbia

Stephanie B. Wilkerson, Magnolia Consulting, LLC

Helen S. Apthorp and Mya L. Martin-Glenn, Mid-continent Research for Education and Learning

David Snow, Billings Public Schools
Research Profiled Out-of-School Time Programs: A Meta-Analysis of Effects for At-Risk Students
Research Planned none
Report Availability Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L. (2006). Out-of-school time programs: A meta-analysis of effects for at-risk students. Review of Educational Research, 76, 275–313.


Contacts

Research Linda Brannan
Information Resources Director
Mid-continent Research for Education and Learning
4601 DTC Boulevard, Suite 5000
Denver, CO 80237
Tel: 303-632-5506
Email: lbrannan@mcrel.org

Helen S. Apthorp, Ph.D.
Principal Researcher
Mid-continent Research for Education and Learning
4601 DTC Boulevard, Suite 5000
Denver, CO 80237
Tel: 303-632-5622
Email: hapthorp@mcrel.org
Profile Updated September 29, 2006

Research Study: Out-of-School Time Programs: A Meta-Analysis of Effects for At-Risk Students


Research Description

Research Purpose To address the following research questions: (a) Based on rigorous research and evaluation studies, what is the effectiveness of OST programs in assisting at-risk students in reading and math, and (b) how does the effectiveness of OST differ by program and study characteristics?
Research Design Experimental and Quasi-Experimental: Each of the 35 studies was coded for information about the OST program, the youth sample, the research design, statistical results, and research quality. Program information included the nature of the program (e.g., homework help, one-on-one tutoring), content (e.g., reading, math), focuses (academic, social), timeframe (e.g., after school, summer school), descriptions of specific strategies related to reading or math, qualifications of those implementing the programs, and program duration (number of program hours made available to attendees). Youth information included gender, ethnicity, grade level, and how the study identified youth as at risk. Research design was coded as either experimental (studies with random assignment to treatment and control groups) or quasi-experimental (studies without random assignment to control groups but that often used procedures to equate or match participants to a comparison group). Designs were also coded for whether youth were pretested on achievement prior to program implementation and posttested afterward or only posttested. For research quality, studies were coded for four types of validity: construct (treatment properly defined and fidelity of the intervention), internal (groups were comparable and no identified processes or events that could be alternative explanations for estimated effects), external (participant characteristics, settings, and outcomes are represented in the sample; effectiveness was tested for important participant subgroups), and statistical (i.e., ES can be calculated for outcome measures). More weight was assigned to construct and internal validity, and points were assigned in each category. Based on these criteria, studies were divided into low-, medium-, and high-quality studies.
Data Collection Methods Secondary Source/Data Review: Researchers searched the ERIC database using the following parameters: 1985–2003, not college, and English-language only. Separate searches were conducted using specific keywords: supplementary, summer school, after school, and vacation. They next searched for the terms literacy, reading, math, and algebra anywhere in the citation. Similar searches were conducted in the PsycInfo and Dissertation Abstracts databases. Researchers next read the abstracts of the resulting citations, along with descriptions of additional studies from five research reports: Fashola (1998), Cooper et al. (2000), Redd et al. (2002), Scott-Little et al. (2002), and Miller (2003), as well as from the following organizations’ websites: Afterschool Alliance, After School Corporation, Harvard Family Research Project, and National Institute on Out-of-School Time.

References:
Fashola, O. S. (1998). Review of extended-day and after-school programs and their effectiveness (Report No. 24). Baltimore, MD: Center for research on the Education of Students Placed at Risk.

Cooper, H., Charlton, K., Valentine, J. C., & Muhlenbruck, L. (2000). Making the most of summer school: A meta-analytic and narrative review. Monographs of the Society for Research in Child Development (Serial No. 260), 65(1), 1–118.

Redd, Z., Cochran, S., Hair, E., & Moore, K. (2002). Academic achievement programs and youth development: A synthesis. Washington, DC: Child Trends.

Miller, B. M. (2003). Critical hours: Afterschool programs and educational success. Quincy, MA: Nellie Mae Education Foundation.
Data Collection Timeframe Data were collected between May 2003 and July 2003.


Findings:
Summative/Outcome Findings

Academic Studies showed that programs had significant and positive effects on reading achievement (p < .05), with estimated average ES ranging from .05 to .13. There were significant differences in estimated ES by: (a) grade level (p < .01), with larger ES reported in studies of lower elementary (K–2) and high school students; (b) program duration (p < .01), with programs of moderate duration (44 to 84 hours) showing the most positive and significant average ES; (c) group size (p < .01), with one-on-one tutoring programs showing the largest estimated average effects; (d) study quality (p < .05), with medium and high quality studies showing the largest ES; and (e) publication type (p < .05), with publications in peer-reviewed journals showing the largest ES. There was no significant variation by whether the study used pretest scores or whether the program had an academic focus or a focus on academics plus social development.

Studies showed that programs had significant and positive effects on math achievement (p < .05), with estimated average ES ranging from .09 to .17. ES significantly differed by: (a) grade level (p < .01), with larger and more significant ES reported in studies of middle and high school students; (b) program duration (p < .05), with programs of moderate duration showing the most positive and significant average ES; (c) focus (p < .05), with programs combining an academic and social focus having larger ES than strictly academic programs; (d) group size (p < .01), with programs that used mixed grouping showing the largest ES; (e) study quality (p < .01), with medium and high quality studies showing the largest ES. Estimated ES did not significantly differ by whether the program was after school or in the summer, by publication type, or by whether the study used pretest scores.

 

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project