You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Research Description

Overview and Components The Study of Promising After-School Programs aims to (a) identify after school programs that feature promising practices for enhancing youth’s academic and social development and their emotional and physical well-being and (b) test the hypothesis that disadvantaged youth who participate in promising after school programs achieve greater learning and developmental gains over a 2-year period than disadvantaged nonparticipants. Promising after school programs are those that offer high-quality environments for youth, are sustainable, and promote positive youth outcomes.
Start Date The study began in the fall of 2002. (completed in spring 2005).
Scope national
Type after school
Location urban, suburban, rural
Setting public schools, community-based organizations
Participants elementary and middle school students (ages 8–14)
Number of Sites/Grantees 37 in 2002–2003; 35 in 2003–2004; 34 in 2004–2005
Number Served average enrollment of 328 youth for elementary school programs and 504 youth for middle school programs in 2002–2003
Study Details Year 1 of the study identified promising programs. Year 2 examined intermediate impacts of promising programs and other after school experiences on youth outcomes. Year 3 examined longer terms impacts of programs and other after school experiences on youth outcomes.

Selection criteria for promising programs stipulated that programs be school linked or school based; serve elementary or middle school youth from low-income families; meet at least 3 days a week; charge families no fee or only a small fee; expect to be sustained for the next 3 years; have been in operation for at least 3 years; offer opportunities for sustained youth involvement in substantive activities that support skill building, mastery, and engagement; have access to resources and materials to support substantive activities; use staffing patterns that contribute to positive and supportive youth relationships with staff and peers (low child–adult ratios and staff turnover, staff with training and expertise); serve at least 30 youth in target grades (grades 3–4 and 6–7); and show evidence of positive youth impacts in a previous evaluation.
Funding Level Not available
Funding Sources The C. S. Mott Foundation funded the study. Government agencies, private/corporate donors, school districts, and private foundations funded the programs.
Researchers Wisconsin Center for Education Research and Policy Studies Associates
Research Profiled Descriptive Report of the Promising Programs.

Examination of Intermediate Outcomes in Year 2

Examination of Longer Term Outcomes After Two Years of Program Experience
Research Planned None
Report Availability Vandell, D. L., Reisner, E. R., Brown, B. B., Pierce, K., Dadisman, K., & Pechman, E. M. (2004). The study of promising after-school programs: Descriptive report of the promising programs. Madison: Wisconsin Center for Education Research.

Vandell, D. L., Reisner, E. R., Brown, B. B., Dadisman, K., Pierce, K. M., Lee, D., & Pechman, E. M. (2005). The study of promising after-school programs: Examination of intermediate outcomes in year 2. Madison: Wisconsin Center for Education Research.

Vandell, D. L., Reisner, E. R., Pierce, K. M., Brown, B. B., Lee, D., Bolt, D., & Pechman, E. M. (2006). The study of promising after-school programs: Examination of longer term outcomes after two years of program experiences. Madison: Wisconsin Center for Education Research.

Available at www.wcer.wisc.edu/childcare/statements.html


Contacts

Research Deborah Lowe Vandell
Chair, Department of Education
University of California, Irvine
2001 Berkeley Place
Irvine, CA 92697-5500
Tel: 949-824-7840
Fax: 949-824-2965
Email: dvandell@uci.edu
Profile Updated November 17, 2006

Research Study 2: Examination of Intermediate Outcomes in Year 2


Research Description

Research Purpose To examine program impacts on youth outcomes in year 2.
Research Design Quasi-Experimental: Several year 1 programs were withdrawn in year 2 due to loss of funding, administrative staff changes, or school district concerns. Replacement programs were selected for year 2, resulting in a sample of 19 elementary and 16 middle school programs in 13 cities and 9 states across the country. Programs maintained their ratings on nearly all program quality indicators from year 1 to 2. The only significant change was a decrease in mastery orientation (p < .05). All elementary students in grades 3–4 and middle school students in grades 6–7 were recruited from programs’ host schools for study participation. In schools with over 500 youth in targeted grades, 5 classrooms were randomly selected for recruitment.

The sample consisted of 1,820 elementary youth (45% of those approached) and 2,926 middle school youth (38% of those approached) who provided signed parental consent. Compared to host schools, the sample contained significantly smaller proportions of boys (p < .01) and youth receiving free/reduced-price lunch (p < .001); the middle school sample had significantly higher proportions of White and Black youth and a smaller proportion of Hispanic youth (p < .001). About half of all youth had minimal program dosage (0–4 days; 44% elementary, 52% middle school), with smaller percentages classified as having low (5–29 days; 12%, 18% respectively), moderate (30–59 days; 9%, 11%), substantial (60–59 days; 7% each), and high (90+ days; 28%, 12%) dosage.

Youth who attended programs more than 5 days in at least one semester were placed in the program group (n = 1,017 elementary and 540 middle school youth); all others were placed in the comparison group (n = 803 elementary and 579 middle school youth). Program youth participated in other types of structured, supervised activities (except coached sports at the elementary level) to a significantly greater extent at baseline than comparison youth (p < .001). Also in contrast to comparison youth, the elementary program group was significantly more likely to be Black or White and reside in single-parent homes with mothers who worked full time; and less likely to be Hispanic and qualify for free/reduced-price lunch (p < .001). Elementary program youth also had poorer baseline teacher ratings of social skills than comparison youth (p < .05). The middle school program group was significantly more likely to be male (p < .001), live in single-parent homes (p < .001), and have mothers who were employed full time (p < .05); and less likely to be White (p < .05) than the middle school comparison group. To address the fact that initial differences between participants and nonparticipants are likely to contribute to differential decisions regarding whether to attend after school programs, child and family characteristics were controlled for in the outcome analysis, including (a) youth’s behavioral adjustment, gender, and ethnicity; (b) parents’ employment and education; and (c) family structure (e.g., two parent household) and income.

Classroom teacher, youth, and program staff surveys were collected in fall 2003 (baseline) and spring 2004 (follow-up). Teacher surveys were collected from the regular classroom elementary school teachers and language arts middle school teachers on study sample youth. Also at baseline, program activities were observed. Of the elementary sample, 94% of youth completed surveys at baseline and 89% at follow up; teacher surveys were completed for 81% of youth at baseline and 84% at follow up; and program staff surveys were completed for 78% of program youth at baseline and 80% at follow up. Of the middle school sample, 92% of youth completed surveys at baseline and 87% at follow up; teacher surveys were received for 88% of youth at baseline and 85% at follow up; and program staff surveys were received for 83% of program youth at baseline and 79% at follow up.

Cluster analysis (a statistical technique to group individuals based on scores on a set of variables) was used to create youth clusters based on 8 variables: program status (program vs. comparison group); program dosage (amount of program participation); and the amount of time spent in 6 other after school contexts—school-based extracurricular activities, coached sports, lessons, home alone, home with siblings but no adults present, and hanging out with peers unsupervised by adults. Four clusters were identified: (a) high program/high activity—high involvement levels in both programs and other enrichment activities (16% of elementary and 18% of middle school youth); (b) high program/low activity—high program participation rates and low scores on most of the other variables (33% of elementary and 28% of the middle school youth); (c) low supervision—low program participation levels, moderate to high amounts of time in other activities, and high amounts of unsupervised time, especially hanging out with peers (16% of elementary and 15% of middle school youth); and (d) supervised at home—relatively low scores on all variables (34% of elementary and 38% of middle school youth). The low supervision group was contrasted with the other clusters to examine whether the programs and other enrichment activities were protective for youth at risk socially and academically.

Clusters differed significantly on a number of demographic variables. In the elementary sample, the high program/high activity group had a higher proportion of girls than the other clusters, whereas males predominated in the low supervision cluster (p < .001). The high program/low activity group had higher family incomes than the other clusters (p < .05). The low supervision group had the greatest proportion of mothers who did not graduate from high school (p < .01). The supervised at home group contained relatively more youth from two-parent households and youth whose mothers did not work full time (p < .001). The high program/high activity and supervised at home groups had higher proportions of Hispanic youth (p < .001). In the middle school sample, the high program/ high activity cluster had a comparatively high proportion of girls (p < .001) and, along with the low supervision group, a greater percentage of mothers who worked full time and higher incomes (p < .05). The high program/low activity cluster was predominantly male and Hispanic, with a lower proportion of White youth than the other clusters (p < .001). The supervised at home group had the highest proportion of two-parent households (p < .01), the lowest percentage of mothers working full time (p < .05), and lower maternal education levels (p < .05). These differences were controlled for in the analyses.

Clusters also differed significantly on a number of baseline outcome variables. At the elementary level, the low supervision group had higher ratings of youth-reported misconduct (p < .001) and consistently poorer ratings from teachers (p < .01 for academic performance, and p < .001 for all other ratings). Program staff perceived that youth in the high program/high activity cluster had better work habits and social skills than youth in the high program/low activity and low supervision clusters (p < .01). Significant differences at baseline also were evident for middle school youth. The low supervision cluster had the highest youth-reported rates of misconduct and substance use (p < .001) and was rated by classroom teachers as more aggressive than the high program/low activity cluster (p < .05). Program staff rated youth in the high program/low activity cluster as exhibiting greater task persistence than youth in the low supervision cluster (p < .05). These differences were controlled for in the analyses.
Data Collection Methods Observation: Programs were observed to confirm that they met the study’s quality criteria.

Secondary Source/Data Review: Program attendance data were collected and used to identify dosage levels based on the number of days youth attended across year 2.

Surveys/Questionnaires: Surveys included measures of youth’s work habits (all surveys); task persistence/self-efficacy (teacher, staff, and middle school youth surveys)’ social skills and aggressive/prosocial behavior with peers (teacher and staff surveys); substance use (middle school youth surveys); academic performance (teacher surveys); and misconduct and time spent in various after school activities (youth surveys).

Tests/Assessments: A modified version of the Promising Practices Rating System (PPRS, Vandell et al., 2004), developed for this study, was used during observations. It included 8 program processes related to quality: supportive relations with adults; supportive relations with peers; youth engagement in program activities; appropriate program structure; opportunities for cognitive growth; mastery orientation (e.g., “activities involve a graded progression of skills”); chaos in the program; and over-control of youth by staff.

Surveys incorporated items from assessment scales (often modified for this study) to examine youth outcomes, including (a) the Mock Report Card’s (Pierce et al., 1999) Work Habits Scale, a measure of classroom work habits (all surveys), and Academic Performance Scale, a measure of reading and oral/written language performance (all teacher surveys; elementary teachers also rated math, science, and social studies); (b) the Self-Reported Behavior Index’s Misconduct Scale (Brown, Clasen, & Eicher, 1986), a measure of behavior (youth surveys), and items that ask about frequency of tobacco, alcohol, marijuana, and other drug use (middle school youth surveys); (c) Walker & Arbreton’s Self-Efficacy Scale (2001), a measure of task persistence/self efficacy; (middle school youth, teacher, and staff surveys); (d) the Teacher Checklist of Peer Relations’ Prosocial Behavior Scale (Coie & Dodge, 1988), a measure of social skills with peers (teacher and staff surveys); and (e) the Child Behavior Scale (Ladd & Profilet, 1996), a measure of aggressive, withdrawn, and prosocial behaviors, with items forming 2 scales, Aggressive with Peers and Prosocial with Peers (teacher and staff surveys).

References:
Brown, B. B., Clasen, D. R., & Eicher, S. A. (1986). Perceptions of peer pressure, peer conformity, dispositions, and self-reported behavior among adolescents. Developmental Psychology, 22, 521–530.

Coie, J. D., & Dodge, K. A. (1988). Multiple sources of data on social behavior and social status in the school: A cross-age comparison. Child Development, 59, 815–829.

Ladd, G. W., & Profilet, S. M. (1996). The Child Behavior Scale: A teacher-report measure of young children’s aggressive, withdrawn, and prosocial behaviors. Developmental Psychology, 32, 1008–1024.

Pierce, K. M., Hamm, J. V., & Vandell, D. L. (1999). Experiences in after-school programs and children’s adjustment in first-grade class classrooms. Child Development, 70, 756–767.

Vandell, D. L., Reisner, E. R., Brown, B. B., Pierce, K., Dadisman, K., & Pechman, E. M. (2004). The study of promising after-school programs: Descriptive report of the promising programs. Madison: Wisconsin Center for Education Research.

Walker, K. E., & Arbreton, A. J. A. (2001). Working together to build Beacon Centers in San Francisco. Philadelphia: Public/Private Ventures.
Data Collection Timeframe Data were collected in fall 2003 and spring 2004.


Findings:
Summative/Outcome Findings

Academic No significant differences were found between program and comparison groups or by dosage on any academic outcomes. However, significant associations were found between academic outcomes and nonprogram after school contexts: in the elementary sample, more time hanging out with peers was associated with poorer youth- and staff-reported work habits (p < .001) and teacher- and staff-reported task persistence. More time at home alone or with siblings was associated with poorer teacher- and staff-reported work habits (p < .05). More time in organized activities was associated with better youth-reported work habits (p < .001). In the middle school sample, more time hanging out with peers was associated with poorer youth-reported work habits (p < .01).

Compared to the low supervision group, significant gains were seen at the elementary level in the other groups’ youth-reported work habits (p < .001 for the high program/high activity group, p < .05 for the other two); (b) the high program/low activity and supervised at home groups’ teacher-reported work habits (p < .05), task persistence (p < .01, p < . 05, respectively) and academic performance (p < .01, p < . 05, respectively); and (c) the two program groups’ staff-reported work habits and task persistence (p < .05). At the middle school level, the high program/high activity group’s teacher-reported work habits and the supervised at home group’s teacher-reported academic performance (p <. 05) showed significant gains relative to the low supervision group.
Prevention Program and comparison groups did not significantly differ on any prevention outcomes, although youth had significantly increased staff-reported aggressive behavior when they attended the programs more often (p < .05 for elementary youth, p < .01 for middle school youth). In addition, significant associations were found between prevention outcomes and other after school contexts: In the elementary sample, more time hanging out with peers (p < .001), at home alone or with siblings (p < .05), and in organized activities (p < .05) were associated with greater youth-reported misconduct than amongst youth spending less time in these contexts. More time spent hanging out with peers was also associated with more staff-reported aggressive behavior with peers (p < .05) compared to youth who spent less time hanging out with peers. In the middle school sample, more time hanging out with peers was associated with more misconduct and substance use (p < .001).

Compared to the low supervision group, significant improvements were seen at the elementary level in the other three clusters’ youth-reported misconduct (p < .001) and in the high program/low activity cluster’s teacher-reported aggressive behavior (p < .05). At the middle school level, compared to the low supervision group, improvements were seen in the other three clusters’ youth-reported misconduct (p < .01) and substance use (p < .01 for the high program/low activity group and p < .001 for the other two).
Youth Development No significant differences were found between program and comparison groups or by dosage in youth development outcomes. However, significant associations were found between youth development outcomes and other after school contexts. For elementary youth, more time hanging out with peers was associated with less staff-reported prosocial behavior (p < .01) and social skills (p < .05), while more time at home alone or with siblings was associated with poorer teacher-reported social skills (p < .01) than among youth spending less time in these contexts. For middle school youth, more time in organized activities was associated with poorer teacher reports of social skills (p < .05).

Compared to the low supervision cluster, at the elementary level, significant gains (p < .05) were seen in the high program/low activity cluster’s teacher-reported prosocial behavior and social skills and the supervised at home cluster’s teacher-reported social skills. No significant differences were found at the middle school level.

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project