The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
HFRP
> Out-of-School Time
> OST Database & Bibliography
> Database
> Promising After-School Programs
A Profile of the Research Study of Promising After-School Programs
Profile Information
- Full Text (HTML)
- Request reprint permission
- How to cite
- About the OST Database
Research Description
| Overview and Components | The Study of Promising After-School Programs aims to (a) identify after school programs that feature promising practices for enhancing youth’s academic and social development and their emotional and physical well-being and (b) test the hypothesis that disadvantaged youth who participate in promising after school programs achieve greater learning and developmental gains over a 2-year period than disadvantaged nonparticipants. Promising after school programs are those that offer high-quality environments for youth, are sustainable, and promote positive youth outcomes. |
| Start Date | The study began in the fall of 2002. (completed in spring 2005). |
| Scope | national |
| Type | after school |
| Location | urban, suburban, rural |
| Setting | public schools, community-based organizations |
| Participants | elementary and middle school students (ages 8–14) |
| Number of Sites/Grantees | 37 in 2002–2003; 35 in 2003–2004; 34 in 2004–2005 |
| Number Served | average enrollment of 328 youth for elementary school programs and 504 youth for middle school programs in 2002–2003 |
| Study Details | Year 1 of the study identified promising programs. Year 2 examined intermediate impacts of promising programs and other after school experiences on youth outcomes. Year 3 examined longer terms impacts of programs and other after school experiences on youth outcomes. Selection criteria for promising programs stipulated that programs be school linked or school based; serve elementary or middle school youth from low-income families; meet at least 3 days a week; charge families no fee or only a small fee; expect to be sustained for the next 3 years; have been in operation for at least 3 years; offer opportunities for sustained youth involvement in substantive activities that support skill building, mastery, and engagement; have access to resources and materials to support substantive activities; use staffing patterns that contribute to positive and supportive youth relationships with staff and peers (low child–adult ratios and staff turnover, staff with training and expertise); serve at least 30 youth in target grades (grades 3–4 and 6–7); and show evidence of positive youth impacts in a previous evaluation. |
| Funding Level | Not available |
| Funding Sources | The C. S. Mott Foundation funded the study. Government agencies, private/corporate donors, school districts, and private foundations funded the programs. |
| Researchers | Wisconsin Center for Education Research and Policy Studies Associates |
| Research Profiled | Descriptive Report of the Promising Programs. Examination of Intermediate Outcomes in Year 2 Examination of Longer Term Outcomes After Two Years of Program Experience |
| Research Planned | None |
| Report Availability | Vandell, D. L., Reisner, E. R., Brown, B. B., Pierce, K., Dadisman, K., & Pechman, E. M. (2004). The study of promising after-school programs: Descriptive report of the promising programs. Madison: Wisconsin Center for Education Research. Vandell, D. L., Reisner, E. R., Brown, B. B., Dadisman, K., Pierce, K. M., Lee, D., & Pechman, E. M. (2005). The study of promising after-school programs: Examination of intermediate outcomes in year 2. Madison: Wisconsin Center for Education Research. Vandell, D. L., Reisner, E. R., Pierce, K. M., Brown, B. B., Lee, D., Bolt, D., & Pechman, E. M. (2006). The study of promising after-school programs: Examination of longer term outcomes after two years of program experiences. Madison: Wisconsin Center for Education Research. Available at www.wcer.wisc.edu/childcare/statements.html |
Contacts
| Research | Deborah Lowe Vandell Chair, Department of Education University of California, Irvine 2001 Berkeley Place Irvine, CA 92697-5500 Tel: 949-824-7840 Fax: 949-824-2965 Email: dvandell@uci.edu |
|
| Profile Updated | November 17, 2006 | |
Research Study 1: Descriptive Report of the Promising Programs
Research Description
| Research Purpose | To examine the quality and implementation of potentially promising after school programs. |
| Research Design | Non-Experimental: Based on a review of published materials, recommendations from after school experts (academics, practitioners, youth-serving organizations, consultants, and federal and state officials), and interviews with program directors, 35 elementary and 26 middle school programs met the study criteria and thus were identified as potential study sites. Considerations of geographic diversity and accessibility to the researchers led to the selection of 29 elementary and 28 middle school sites for fall 2002 site visits. During these visits, activities were observed over 2 days and interviews were conducted with program directors, activity leaders (one at each site), and principals at all host schools. In spring 2003, researchers returned to 19 elementary and 18 middle school programs that seemed especially promising based on fall visits, during which researchers observed activities again over 2 days and conducted program director and activity leader surveys. Of the sites visited in the spring, 13 were in California, 11 in the Northeast, 9 in the Northwest, and 4 in the Midwest; about 40% (slightly more at the middle school than elementary level) were in major metropolitan centers, another 40% in other urban areas, and the remaining 15–20% in smaller towns or rural areas. On average, elementary programs had been in operation for over 5 years and middle school programs for 8 years. |
| Data Collection Methods | Interviews/Focus Groups: Director and activity leader interviews asked about program activities/schedules, youth opportunities for autonomy and choice, and staff training. Directors were also asked about program goals and funding, relationships with community agencies, youth enrollment/attendance, and staff education. Activity leaders were asked about planning opportunities and relationships with partner schools. Principal interviews asked about school enrollment/demographics and perceptions of the program. Observation: Observations collected data about program quality and implementation. Surveys/Questionnaires: All surveys asked about staff training/background and partner school–parent relations. Directors were asked about staff meetings, program enrollment, attendance, space/material resources, funding, and community agency relations. Activity leaders were asked about their experience, planning opportunities, and job satisfaction/support. Test/Assessments: Two observations tools were used: (a) the Promising Practices Rating System (PPRS, Vandell et al., 2004), developed for this study, was used during fall site visits to quantify 7 program processes related to quality, including supportive relations with adults; supportive relations with peers; youth engagement; appropriate program structure; cognitive growth opportunities; mastery orientation (e.g., “activities involve a graded progression of skills”); and autonomy opportunities; and (b) the After-School Activity Observation Instrument (AOI, Pechman & Marzke, 2003), an observation/interview measure modified for spring site visits to collect data during 15-minute observations of content and skill areas, use of space/materials, youth interactions with peers/staff, youth engagement, and activity structure. The AOI identifies 21 types of activities (e.g., arts and crafts, snack, etc.), of which all but two were observed in programs. To get a sense of how well each type of activity reflected the PPRS processes, means scores on scales related to each of these features was calculated (all except autonomy opportunities), and activities types in which the mean score for a given scale were especially high were identified. References: Pechman, E., & Marzke, C. (2003). After-School Activity Observation Instrument (AOI) observation manual. Adapted for the WCER/PSA Study of Promising After-School Programs. Washington, DC: Policy Studies Associates. Vandell, D. L., Reisner, E. R., Brown, B. B., Pierce, K., Dadisman, K., & Pechman, E. M. (2004). The study of promising after-school programs: Descriptive report of the promising programs. Madison: Wisconsin Center for Education Research. |
| Data Collection Timeframe | Data were collected in fall 2002 and spring 2003. |
Findings:
Formative/Process Findings
| Activity Implementation | The majority of programs (71% of elementary and 78% of middle school programs) operated at least 5 days a week. On average, programs were open just under 4 hours a day. Observations revealed that all programs offered academic assistance (e.g., homework help, tutoring, study skills, test prep). Board games/puzzles, arts and crafts, and performing arts activities were also common. Computer-oriented activities were more common in middle school programs, while elementary programs were more likely to have reading/language arts enrichment. Program staff reported a variety of other activities that were not observed, including photography, youth-run businesses, video production, outdoor skills, drama, textile production, cooking, gardening, oral history, drill team, and community service. Program schedules reflected community needs and the constraints imposed by location, resources, staffing, and funding. Marked schedule differences were sometimes found from one day or season to the next. Most programs had multiple classes going simultaneously at various points in the year. Activities tended to change regularly; the majority of directors noted that they offered activities in distinct sessions. Most elementary sessions ran 7–12 weeks, while nearly half of middle school sessions ran 1–6 weeks. The number of days an activity was offered per week was inconsistent, but for the majority of elementary programs, daily activity sessions lasted an hour or less, while middle school programs tended to allow 1–2 hours. Average program quality ratings ranged from 3.0 (somewhat characteristic of the program) to just below 4.0 (highly characteristic of the program). Average ratings were highest for supportive relationships with peers and adults and for youth engagement. Middle school programs outscored elementary programs on autonomy opportunities and emphasis on mastery. Cognitive growth opportunities were comparatively limited, especially in middle school programs. Nearly three quarters of elementary and almost all middle school programs received average quality scores of at least 3.5 out of 4 on more than half of the 7 processes measured, and a fifth of elementary and half of middle school programs had such ratings on at least 6 processes. Over half of programs received 4s for youth engagement and supportive peer relationships. Less than 20% of programs received top scores for enhancing cognitive growth. Top ratings were more prevalent among middle school programs for all processes except appropriate structure. Most directors and activity leaders reported youth leadership opportunities, which most commonly arose without formal organization. In some programs, staff systematically encouraged older youth or youth with more advanced skills to take responsibility for other youth. Some programs had high school youth on staff to help with activities. High ratings by activity type covered all program process areas except appropriate structure and each process area was facilitated by at least two activity types. No activities type received high ratings in all 5 of the remaining areas, but a number of activities received high rating in 4 of the 5 areas: at the elementary level in performing arts and snacks and at the middle school level in reading/language arts enrichment, tutoring, computer skill building, higher education/ career prep, and study skills/test prep. |
| Costs/Revenues | Over two thirds of programs drew funds from local/federal government agencies; about half from private/corporate donors, over 40% from local school districts, and a third from private foundations. Many also had national foundation grants (33% of elementary and 43% of middle school programs). In terms of dollar amounts, the federal government accounted for over a third of elementary and 45% of middle school program budgets, while local governments accounted for a fourth of elementary and nearly a fifth of middle school program budgets. Much of the rest of elementary budgets came from private/corporate donors. The last third of middle school budgets came more equally from several sources. |
| Parent/Community Involvement | Community organizations (schools, businesses, and social service agencies) provided youth activities, volunteers, and supplies at almost all programs. Many sites also counted on these partners to provide mentors, classes for parents, funding, youth referrals, and activity space. Programs relied on multiple organizations for resources; many reported associations with 5 or more organizations for any given resource. In general, elementary programs reported more community linkages than did middle school programs. Directors sent information home and held events for parents at least once a semester. For elementary directors, the majority spoke to (90%) and met with (80%) parents several times a month. In addition, many programs offered classes for parents, dealing with childrearing, employment skills, health care, and English language learning. |
| Program–School Linkages | Over two thirds of elementary program directors spoke to school staff “almost weekly” about homework and individual youth’s needs. The majority of directors spoke to school staff about curriculum concepts at least a couple of times per semester. Over half of directors met with the principal at least 2–3 times a month; over a third did so weekly. Such meetings were more frequent in elementary programs. Over a third of activity leaders spoke to school staff about individual youth almost weekly. Nearly half of elementary leaders spoke to school staff about homework almost weekly. Communication about curriculum concepts was less frequent, especially among elementary leaders. About 40% of leaders spoke to school staff about classroom space at least occasionally. |
| Program Context/Infrastructure | Almost all programs had classroom, storage, staff planning, and meeting space; at least three quarters had access to outdoor space, a cafeteria, and specialty areas (gym, library, or art room); and a majority also had auditorium, computer lab, and kitchen access. The only area that was less common was science labs for elementary programs. Storage and library space, however, were judged to be inadequate by over a third of directors, and many elementary directors found art, music, and office space to be limited. Over 80% of directors had access to all types of materials listed on the survey, and very few reported limited materials of any particular type. Almost all directors, however, felt that games and puzzles were in short supply, almost 1 in 3 elementary directors felt that computer access was inadequate, and 1 in 4 middle school directors noted a need for more physical education equipment and expanded transportation services for field trips. Across the schools associated with the programs, 83% of elementary youth and 75% of middle school youth qualified for free/reduced-price lunch. |
| Recruitment/Participation | Directors reported that 65% of elementary and 50% of middle school youth attended programs 3 or more days a week. Three quarters of elementary programs indicated that at least half of their participants attended 3 or more days a week, and 1 out of 5 claimed this benchmark was achieved by more than 90% of their participants. By contrast, only 57% of middle school programs reported that a majority of their youth attended 3 or more days a week, and none indicated that 90% or more of their youth achieved this benchmark. Latinos made up the highest percentage of participants (56% elementary, 43% middle school), followed by White (23% elementary, 33% middle school), Black (18% elementary, 16% middle school), and Asian and other (1–5%) youth. Participants closely matched the ethnic compositions of the associated schools. |
| Staffing/Training | Activity leaders’ expressed satisfaction overall with working conditions and staff relations. Elementary programs reported 6–28 staff (average = 15) and middle school programs reported 5–44 (average = 16), including paid and volunteer staff in a variety of roles, from administrators to activity leaders to custodians. An average youth–staff ratio, including teen volunteers, was observed to be 10:1 in elementary and 8:1 in middle school programs. On average, directors had been at programs for 3.6 years and worked just under 5 days a week. They worked for 29 hours a week at elementary sites and 38 hours at middle school sites. All were paid by the program; many also worked for the school (38% at elementary and 47% at middle school sites). The majority held a bachelor’s degree or higher (84% of middle school and 69% of elementary directors). Almost 40% of middle school and 7% of elementary directors were certified teachers. Most directors (80%) had previously worked in youth recreation/childcare positions; about 3 of 4 had classroom instruction experience; and over a third had administrative experience in youth and/or social service work. On average, activity leaders worked 10–12 hours across a little over 3 days a week. They had been at the program for an average of 2.2 years at elementary sites and 1.6 years at middle school sites. About 80% were paid by the program; about half of middle school and a fourth of elementary leaders also worked for the school. About half held a bachelor’s degree or higher. Nearly a third of middle school and a fifth of elementary leaders were certified teachers. The majority had prior experience in youth and/or social services or classroom instruction. Few directors reported hiring teen staff. At elementary sites, teens worked an average of 15 hours over 4 days a week and had 1.5 years of experience; all were paid by the program. At middle school sites, teens worked an average of 3 hours over 2 days a week and had an average of a few months of experience; none were paid by the program. Other staff worked, on average, 3–4 days for 10–12 hours per week and had been at the program for 2 years. The majority were paid by the program (83% at elementary and 69% at middle school sites), and some also worked for the school (15% and 40% respectively). Staff training was provided in a variety of areas, but most commonly in classroom management and academic enrichment. For middle school programs, staff training was also commonly provided in conflict resolution, diversity training, and youth development. In elementary programs, staff were less likely to receive arts or diversity training than in middle school programs, and training was more likely to be offered to paid staff than to volunteers. On average, directors reported receiving 51 hours of training in the last year; activity leaders reported 21 hours. Most activity leaders reported that they held major responsibilities for planning activities, which occurred at least weekly in nearly 80% of cases. About a quarter of elementary and over a third of middle school program activity leaders said that they engaged in activity planning on a daily basis. Nearly all leaders met with other staff to discuss program issues; a third of elementary and close to half of middle school leaders reported that such meetings occurred weekly. |
Research Study 2: Examination of Intermediate Outcomes in Year 2
Research Description
| Research Purpose | To examine program impacts on youth outcomes in year 2. |
| Research Design | Quasi-Experimental: Several year 1 programs were withdrawn in year 2 due to loss of funding, administrative staff changes, or school district concerns. Replacement programs were selected for year 2, resulting in a sample of 19 elementary and 16 middle school programs in 13 cities and 9 states across the country. Programs maintained their ratings on nearly all program quality indicators from year 1 to 2. The only significant change was a decrease in mastery orientation (p < .05). All elementary students in grades 3–4 and middle school students in grades 6–7 were recruited from programs’ host schools for study participation. In schools with over 500 youth in targeted grades, 5 classrooms were randomly selected for recruitment. The sample consisted of 1,820 elementary youth (45% of those approached) and 2,926 middle school youth (38% of those approached) who provided signed parental consent. Compared to host schools, the sample contained significantly smaller proportions of boys (p < .01) and youth receiving free/reduced-price lunch (p < .001); the middle school sample had significantly higher proportions of White and Black youth and a smaller proportion of Hispanic youth (p < .001). About half of all youth had minimal program dosage (0–4 days; 44% elementary, 52% middle school), with smaller percentages classified as having low (5–29 days; 12%, 18% respectively), moderate (30–59 days; 9%, 11%), substantial (60–59 days; 7% each), and high (90+ days; 28%, 12%) dosage. Youth who attended programs more than 5 days in at least one semester were placed in the program group (n = 1,017 elementary and 540 middle school youth); all others were placed in the comparison group (n = 803 elementary and 579 middle school youth). Program youth participated in other types of structured, supervised activities (except coached sports at the elementary level) to a significantly greater extent at baseline than comparison youth (p < .001). Also in contrast to comparison youth, the elementary program group was significantly more likely to be Black or White and reside in single-parent homes with mothers who worked full time; and less likely to be Hispanic and qualify for free/reduced-price lunch (p < .001). Elementary program youth also had poorer baseline teacher ratings of social skills than comparison youth (p < .05). The middle school program group was significantly more likely to be male (p < .001), live in single-parent homes (p < .001), and have mothers who were employed full time (p < .05); and less likely to be White (p < .05) than the middle school comparison group. To address the fact that initial differences between participants and nonparticipants are likely to contribute to differential decisions regarding whether to attend after school programs, child and family characteristics were controlled for in the outcome analysis, including (a) youth’s behavioral adjustment, gender, and ethnicity; (b) parents’ employment and education; and (c) family structure (e.g., two parent household) and income. Classroom teacher, youth, and program staff surveys were collected in fall 2003 (baseline) and spring 2004 (follow-up). Teacher surveys were collected from the regular classroom elementary school teachers and language arts middle school teachers on study sample youth. Also at baseline, program activities were observed. Of the elementary sample, 94% of youth completed surveys at baseline and 89% at follow up; teacher surveys were completed for 81% of youth at baseline and 84% at follow up; and program staff surveys were completed for 78% of program youth at baseline and 80% at follow up. Of the middle school sample, 92% of youth completed surveys at baseline and 87% at follow up; teacher surveys were received for 88% of youth at baseline and 85% at follow up; and program staff surveys were received for 83% of program youth at baseline and 79% at follow up. Cluster analysis (a statistical technique to group individuals based on scores on a set of variables) was used to create youth clusters based on 8 variables: program status (program vs. comparison group); program dosage (amount of program participation); and the amount of time spent in 6 other after school contexts—school-based extracurricular activities, coached sports, lessons, home alone, home with siblings but no adults present, and hanging out with peers unsupervised by adults. Four clusters were identified: (a) high program/high activity—high involvement levels in both programs and other enrichment activities (16% of elementary and 18% of middle school youth); (b) high program/low activity—high program participation rates and low scores on most of the other variables (33% of elementary and 28% of the middle school youth); (c) low supervision—low program participation levels, moderate to high amounts of time in other activities, and high amounts of unsupervised time, especially hanging out with peers (16% of elementary and 15% of middle school youth); and (d) supervised at home—relatively low scores on all variables (34% of elementary and 38% of middle school youth). The low supervision group was contrasted with the other clusters to examine whether the programs and other enrichment activities were protective for youth at risk socially and academically. Clusters differed significantly on a number of demographic variables. In the elementary sample, the high program/high activity group had a higher proportion of girls than the other clusters, whereas males predominated in the low supervision cluster (p < .001). The high program/low activity group had higher family incomes than the other clusters (p < .05). The low supervision group had the greatest proportion of mothers who did not graduate from high school (p < .01). The supervised at home group contained relatively more youth from two-parent households and youth whose mothers did not work full time (p < .001). The high program/high activity and supervised at home groups had higher proportions of Hispanic youth (p < .001). In the middle school sample, the high program/ high activity cluster had a comparatively high proportion of girls (p < .001) and, along with the low supervision group, a greater percentage of mothers who worked full time and higher incomes (p < .05). The high program/low activity cluster was predominantly male and Hispanic, with a lower proportion of White youth than the other clusters (p < .001). The supervised at home group had the highest proportion of two-parent households (p < .01), the lowest percentage of mothers working full time (p < .05), and lower maternal education levels (p < .05). These differences were controlled for in the analyses. Clusters also differed significantly on a number of baseline outcome variables. At the elementary level, the low supervision group had higher ratings of youth-reported misconduct (p < .001) and consistently poorer ratings from teachers (p < .01 for academic performance, and p < .001 for all other ratings). Program staff perceived that youth in the high program/high activity cluster had better work habits and social skills than youth in the high program/low activity and low supervision clusters (p < .01). Significant differences at baseline also were evident for middle school youth. The low supervision cluster had the highest youth-reported rates of misconduct and substance use (p < .001) and was rated by classroom teachers as more aggressive than the high program/low activity cluster (p < .05). Program staff rated youth in the high program/low activity cluster as exhibiting greater task persistence than youth in the low supervision cluster (p < .05). These differences were controlled for in the analyses. |
| Data Collection Methods | Observation: Programs were observed to confirm that they met the study’s quality criteria. Secondary Source/Data Review: Program attendance data were collected and used to identify dosage levels based on the number of days youth attended across year 2. Surveys/Questionnaires: Surveys included measures of youth’s work habits (all surveys); task persistence/self-efficacy (teacher, staff, and middle school youth surveys)’ social skills and aggressive/prosocial behavior with peers (teacher and staff surveys); substance use (middle school youth surveys); academic performance (teacher surveys); and misconduct and time spent in various after school activities (youth surveys). Tests/Assessments: A modified version of the Promising Practices Rating System (PPRS, Vandell et al., 2004), developed for this study, was used during observations. It included 8 program processes related to quality: supportive relations with adults; supportive relations with peers; youth engagement in program activities; appropriate program structure; opportunities for cognitive growth; mastery orientation (e.g., “activities involve a graded progression of skills”); chaos in the program; and over-control of youth by staff. Surveys incorporated items from assessment scales (often modified for this study) to examine youth outcomes, including (a) the Mock Report Card’s (Pierce et al., 1999) Work Habits Scale, a measure of classroom work habits (all surveys), and Academic Performance Scale, a measure of reading and oral/written language performance (all teacher surveys; elementary teachers also rated math, science, and social studies); (b) the Self-Reported Behavior Index’s Misconduct Scale (Brown, Clasen, & Eicher, 1986), a measure of behavior (youth surveys), and items that ask about frequency of tobacco, alcohol, marijuana, and other drug use (middle school youth surveys); (c) Walker & Arbreton’s Self-Efficacy Scale (2001), a measure of task persistence/self efficacy; (middle school youth, teacher, and staff surveys); (d) the Teacher Checklist of Peer Relations’ Prosocial Behavior Scale (Coie & Dodge, 1988), a measure of social skills with peers (teacher and staff surveys); and (e) the Child Behavior Scale (Ladd & Profilet, 1996), a measure of aggressive, withdrawn, and prosocial behaviors, with items forming 2 scales, Aggressive with Peers and Prosocial with Peers (teacher and staff surveys). References: Brown, B. B., Clasen, D. R., & Eicher, S. A. (1986). Perceptions of peer pressure, peer conformity, dispositions, and self-reported behavior among adolescents. Developmental Psychology, 22, 521–530. Coie, J. D., & Dodge, K. A. (1988). Multiple sources of data on social behavior and social status in the school: A cross-age comparison. Child Development, 59, 815–829. Ladd, G. W., & Profilet, S. M. (1996). The Child Behavior Scale: A teacher-report measure of young children’s aggressive, withdrawn, and prosocial behaviors. Developmental Psychology, 32, 1008–1024. Pierce, K. M., Hamm, J. V., & Vandell, D. L. (1999). Experiences in after-school programs and children’s adjustment in first-grade class classrooms. Child Development, 70, 756–767. Vandell, D. L., Reisner, E. R., Brown, B. B., Pierce, K., Dadisman, K., & Pechman, E. M. (2004). The study of promising after-school programs: Descriptive report of the promising programs. Madison: Wisconsin Center for Education Research. Walker, K. E., & Arbreton, A. J. A. (2001). Working together to build Beacon Centers in San Francisco. Philadelphia: Public/Private Ventures. |
| Data Collection Timeframe | Data were collected in fall 2003 and spring 2004. |
Findings:
Summative/Outcome Findings
| Academic | No significant differences were found between program and comparison groups or by dosage on any academic outcomes. However, significant associations were found between academic outcomes and nonprogram after school contexts: in the elementary sample, more time hanging out with peers was associated with poorer youth- and staff-reported work habits (p < .001) and teacher- and staff-reported task persistence. More time at home alone or with siblings was associated with poorer teacher- and staff-reported work habits (p < .05). More time in organized activities was associated with better youth-reported work habits (p < .001). In the middle school sample, more time hanging out with peers was associated with poorer youth-reported work habits (p < .01). Compared to the low supervision group, significant gains were seen at the elementary level in the other groups’ youth-reported work habits (p < .001 for the high program/high activity group, p < .05 for the other two); (b) the high program/low activity and supervised at home groups’ teacher-reported work habits (p < .05), task persistence (p < .01, p < . 05, respectively) and academic performance (p < .01, p < . 05, respectively); and (c) the two program groups’ staff-reported work habits and task persistence (p < .05). At the middle school level, the high program/high activity group’s teacher-reported work habits and the supervised at home group’s teacher-reported academic performance (p <. 05) showed significant gains relative to the low supervision group. |
| Prevention | Program and comparison groups did not significantly differ on any prevention outcomes, although youth had significantly increased staff-reported aggressive behavior when they attended the programs more often (p < .05 for elementary youth, p < .01 for middle school youth). In addition, significant associations were found between prevention outcomes and other after school contexts: In the elementary sample, more time hanging out with peers (p < .001), at home alone or with siblings (p < .05), and in organized activities (p < .05) were associated with greater youth-reported misconduct than amongst youth spending less time in these contexts. More time spent hanging out with peers was also associated with more staff-reported aggressive behavior with peers (p < .05) compared to youth who spent less time hanging out with peers. In the middle school sample, more time hanging out with peers was associated with more misconduct and substance use (p < .001). Compared to the low supervision group, significant improvements were seen at the elementary level in the other three clusters’ youth-reported misconduct (p < .001) and in the high program/low activity cluster’s teacher-reported aggressive behavior (p < .05). At the middle school level, compared to the low supervision group, improvements were seen in the other three clusters’ youth-reported misconduct (p < .01) and substance use (p < .01 for the high program/low activity group and p < .001 for the other two). |
| Youth Development | No significant differences were found between program and comparison groups or by dosage in youth development outcomes. However, significant associations were found between youth development outcomes and other after school contexts. For elementary youth, more time hanging out with peers was associated with less staff-reported prosocial behavior (p < .01) and social skills (p < .05), while more time at home alone or with siblings was associated with poorer teacher-reported social skills (p < .01) than among youth spending less time in these contexts. For middle school youth, more time in organized activities was associated with poorer teacher reports of social skills (p < .05). Compared to the low supervision cluster, at the elementary level, significant gains (p < .05) were seen in the high program/low activity cluster’s teacher-reported prosocial behavior and social skills and the supervised at home cluster’s teacher-reported social skills. No significant differences were found at the middle school level. |
Research Study 3: Examination of Longer Term Outcomes after Two Years of Program Experience
Research Description
| Research Purpose | To examine program impacts of sets of experiences in different after school contexts across years 2 and 3. |
| Research Design | Quasi-Experimental: Two middle school programs in year 2 closed; of these, 1 was transferred to a new school, and thus included in the study for a total of 19 elementary and 15 middle school programs in year 3 (2003–2004). Programs maintained their ratings on all program quality indicators from year 2 to 3. In year 3, 80% of the year 2 elementary sample (n = 1,434) and 76% of the middle school sample (n = 855) remained at the participating schools and were available for data collection. Youth who dropped from the study were significantly less likely to attend the programs during year 2 (p < .001), and more likely to be in the supervised at home cluster in the analysis (p < .01). The elementary youth who left the study were also significantly more likely to be White and less likely to be Hispanic than those who stayed in the study (p < .05). The program sample was defined as those who attended a program for at least 5 days in at least one semester. In year 3, 57% of the remaining elementary sample and 42% of the remaining middle school sample attended a program at least 1 day, with average attendance rates of 86 and 62 days, respectively. For years 2 and 3 combined, 64% of the elementary sample and 59% of the middle school sample attended a program at least 1 day, with average 2 year attendance rates of 141 and 79 days, respectively. Over the 2 years, the largest proportion of the sample had minimal program dosage (0–9 days; 39% elementary, 49% middle school), with smaller percentages classified as low (10–59 days; 17%, 23% respectively), moderate (60–119 days; 12%, 14%), substantial (120–179 days; 11%, 7%), and high (180 or more days; 21%, 7%) dosage. Over the 2 years, youth participated in other organized after school activities an average of about once a week or less, with youth engaging in lessons most often, and caring for siblings least often. Correlations of the frequency of these activities over 3 time periods (fall 2003, spring 2004, and spring 2005) were small to moderate, suggesting that participation in these activities was somewhat fluid across a single year and across years. Classroom teacher, youth, and parent surveys were collected and programs were observed over 2 days in spring 2005. Teacher surveys were collected from the regular classroom elementary school teachers and language arts middle school teachers on study sample youth. Of the elementary sample, 70% of youth completed surveys, teacher surveys were completed for 68% of youth, and parent surveys were completed for 56% of youth. Of the middle school sample, 68% of youth completed surveys; teacher surveys were received for 63% of youth; and parent surveys were received for 43% of youth. Cluster analysis (a statistical technique to group individuals based on scores on a set of variables) was used to create youth clusters based on 8 variables across the 2 years: program status (program vs. comparison group); program dosage (amount of program participation); and the amount of time spent in 6 other after school contexts (school-based extracurricular activities, coached sports, lessons, home alone, home with siblings but no adults present, and hanging out with peers unsupervised by adults). Four clusters were identified: (a) program + activities—high involvement levels in both programs and other enrichment activities (18% of elementary and 17% of middle school youth); (b) program only—high program participation rates and low scores on most of the other variables (37% of elementary and 32% of the middle school youth); (c) self care + activities—low program participation levels, moderate amounts of time in other activities (mostly coached sports), and high amounts of unsupervised time, especially hanging out with peers (15% of both elementary and middle school youth); and (d) supervised at home–relatively low scores on all variables (30% of elementary and 36% of middle school youth). The self care + activities cluster was contrasted with the other clusters to examine whether the programs and other enrichment activities were protective for youth at risk socially and academically. Clusters differed significantly on a number of demographic variables. In the elementary sample, the program + activities group had the largest proportion of girls, whereas boys predominated in the self-care + activities cluster (p < .001. The program + activities and supervised at home groups had the highest proportions of Hispanic youth (p < .001). The supervised at home group contained relatively more youth from two-parent households and youth whose mothers did not work full-time (p < .001). The self-care + activities group had the highest proportion of mothers who did not graduate from high school (p < .05). Family incomes did not differ between elementary clusters. In the middle school sample, the supervised at home group had the largest proportion of girls (p < .05) and youth from two-parent households (p < .01) as well as the smallest proportion of mothers who worked full-time (p < .01). The self-care + activities cluster had the highest percentage of mothers who worked full time (p < .01), and the supervised at home and program only groups had lower incomes than the other groups (p < .05). The program + activities and program only clusters were predominantly Hispanic and had fewer proportions of White students (p < .05). Maternal education did not vary significantly between the middle school clusters. These differences were controlled for in the analyses. |
| Data Collection Methods | Observation: Programs were observed to confirm that they met the study’s quality criteria. Secondary Source/Data Review: Program attendance data were collected and used to identify 5 dosage levels based on the number of days youth attended across years 2 and 3. Surveys/Questionnaires: Surveys included measures of youth’s work habits (all surveys); task persistence/self-efficacy (teachers and middle school youth surveys); social skills, aggressive/prosocial behavior with peers, and academic performance (teacher surveys); substance use (middle school youth surveys); misconduct and time spent in other after school activities (youth surveys); and peer reliance and compliance (parent surveys). Test/Assessments: A modified version of the Promising Practices Rating System (PPRS, Vandell et al., 2004), developed for this study, was used during observations. It included 8 program processes related to quality: supportive relations with adults; supportive relations with peers; youth engagement in program activities; appropriate program structure; opportunities for cognitive growth; mastery orientation (e.g., “activities involve a graded progression of skills”); chaos in the program; and over-control of youth by staff. Surveys incorporated items from assessment scales (often modified for this study) to examine youth outcomes, including: (a) the Mock Report Card’s (Pierce et al., 1999) Work Habits Scale, a measure of classroom work habits (youth and teacher surveys), and Academic Performance scale, a measure reading and oral/written language performance (all teacher surveys; elementary teachers also rated math, science, and social studies); (b) the Self-Reported Behavior Index’s Misconduct Scale (Brown, Clasen, & Eicher, 1986), a measure of behavior (youth surveys), and items that ask about frequency of tobacco, alcohol, marijuana, and other drug use (middle school youth surveys); (c) Walker & Arbreton’s Self-Efficacy Scale (2001) a measure of task persistence/self efficacy; (middle school youth and teachers); (d) the Teacher Checklist of Peer Relations’ Prosocial Behavior Scale (Coie & Dodge, 1988), a measure of social skills with peers (teacher surveys); (e) the Child Behavior Scale (Ladd & Profilet, 1996), a measure of aggressive, withdrawn, and prosocial behaviors, with items forming 2 scales, Aggressive with Peers and Prosocial with Peers (teacher surveys); and (f) the Child Adjustment Scale (Santrock & Warshak, 1979), a measure of socioemotional adjustment, which includes scales of work habits, peer relations, and compliance (parent surveys). References: Brown, B. B., Clasen, D. R., & Eicher, S. A. (1986). Perceptions of peer pressure, peer conformity, dispositions, and self-reported behavior among adolescents. Developmental Psychology, 22, 521–530. Coie, J. D., & Dodge, K. A. (1988). Multiple sources of data on social behavior and social status in the school: A cross-age comparison. Child Development, 59, 815–829. Ladd, G. W., & Profilet, S. M. (1996). The Child Behavior Scale: A teacher-report measure of young children’s aggressive, withdrawn, and prosocial behaviors. Developmental Psychology, 32(6), 1008–1024. Pierce, K. M., Hamm, J. V., & Vandell, D. L. (1999). Experiences in after-school programs and children’s adjustment in first-grade class classrooms. Child Development, 70, 756–767. Santrock, J. W., & Warshak, R. A. (1979). Father custody and social development in boys and girls. Journal of Social Issues, 35, 112–125. Vandell, D. L., Reisner, E. R., Brown, B. B., Pierce, K., Dadisman, K., & Pechman, E. M. (2004). The study of promising after-school programs: Descriptive report of the promising programs. Madison: Wisconsin Center for Education Research. Walker, K. E., & Arbreton, A. J. A. (2001). Working together to build Beacon Centers in San Francisco. Philadelphia: Public/Private Ventures. |
| Data Collection Timeframe | Data were collected in spring 2005 (year 3) and compared to data collected during the 2003–2004 school year (year 2). |
Findings:
Summative/Outcome Findings
| Academic | Compared to the self-care + activities cluster, significantly greater gains were seen over the 2 years at the elementary level in the other three groups’ youth- and teacher-reported work habits (p < .01 for the program only group’s youth reports and the supervised at home group’s teacher reports, p < .001 for the rest), teacher-reported task persistence (p < .001 for the programs + activities cluster, p < .01 for the other two); the program + activities (p < .01) and the supervised at home (p < .001) groups’ teacher-reported academic performance, and the supervised at home group’s parent-reported work habits (p < .05). At the middle school level, significantly greater gains were seen in the two program groups’ youth-reported work habits (p < .05) compared to the self-care + activities cluster. |
| Prevention | Compared to the self-care + activities cluster, significantly greater decreases were seen over the 2 years for all youth-reported misconduct (p < .001); elementary-teacher-reported aggression with peers (p < .05 for the programs + activities group, p < .01 for the program only group, and p < .001 for the supervised at home group). All three middle school groups’ youth-reported substance abuse declined more than the self-care + activities group (p < .01 for the supervised at home group and p < .001 for the other two). |
| Youth Development | Compared to the self-care + activities cluster, significantly greater gains were seen over the 2 years at the elementary and middle school level for the program only group’s (p < .001) and supervised at home group’s (p < .01) parent-reported compliance; At the elementary level, all three other groups’ teacher-reported social skills (p < .05 for the program only group and p < .01 for the other two) and prosocial behavior (p < .01 for the supervised at home group and p < .05 for the other two) improved relative to the self-care + activities group. |