Search Publications | The Evaluation Exchange | FINE Network | OST Database

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

A Profile of the Evaluation of 21st Century Community Learning Centers—national

Program Description

Overview	The 21st Century Community Learning Centers (21st CCLC) program, authorized under the federal Elementary and Secondary School Act, provides expanded learning opportunities for participating children in a safe, drug-free, and supervised environment. Administered by the U.S. Department of Education, states receive funding to allocate to local educational agencies.
Start Date	1997
Scope	national

Type	after school, summer/vacation, before school, weekend, comprehensive services
Location	urban, rural
Setting	public schools
Participants	elementary and middle school students
Number of Sites/Grantees	7,500 rural and inner-city public schools in more than 1,400 communities (2002)
Number Served	1.2 million children and youth and 400,000 adults (2001)
Components	Programs receiving 21st CCLC funds must provide a variety of activities that offer high-quality expanded learning opportunities for youth in the community and that contribute to reduced drug use and violence. Grantees must also carry out at least four of the following activities: literacy education programs; senior citizen programs; children's day care services; integrated education, health, social service, recreational, or cultural programs; summer and weekend school programs in conjunction with recreation programs; nutrition and health programs; expanded library service hours to serve community needs; telecommunications and technology education programs for individuals of all ages; parenting skills education programs; support and training for child day care providers; employment counseling, training, and placement services for individuals who leave school before graduating from secondary school; and services for individuals with disabilities. In order to receive grant money, the legislation states that applicants must collaborate with other community-based organizations, related public agencies, businesses, or other appropriate organizations.
Funding Level	$846 million (FY 2001); $1 billion (FY 2002)
Funding Sources	US Department of Education
Other	Through a unique public-private partnership, the Charles Stewart Mott Foundation is complementing the work of the US Department of Education by providing technical assistance to help 21st CCLC grantees develop, implement, and sustain high-quality after school programs. The Mott foundation also supports the program by building public will for after school programs through a media campaign and supporting evaluation and research work in the field.

Evaluation

Overview	The evaluation consists of an impact and an implementation evaluation of the 21st CCLC program. These evaluation components are funded by the US Department of Education. In addition, the Charles Stewart Mott Foundation is providing support to supplement the Department of Education-funded studies and to conduct a series of special studies that will explore issues related to after school program access, services, and best practices. The foundation for this evaluation work is a logic model, based on available research and practice knowledge that lays out the hypothesized relationship between program context, program implementation, intermediate effects, and long-term effects.
Evaluator	Mathematica Policy Research, Inc. and Decision Information Resources, Inc.
Evaluations Profiled	When Schools Stay Open Late: The National Evaluation of the 21st-Century Community Learning Centers Program, First Year Findings Study of After School Time Use: 2000–2002
Evaluations Planned	The evaluators currently are collecting another year of follow-up data and have expanded the study to include more programs serving elementary school students. The additional data from the second follow-up year and from the newly included programs will be the basis for two future reports. The first will update the findings for middle school students using another year of follow-up data and will present first-year findings for elementary school students using a larger number of elementary school programs. The second will update the findings for elementary school students using another year of follow-up data. Also, Deborah Vandell of the University of Wisconsin is conducting a separate study entitled, Study of After School Time Use: 2000–2002.
Report Availability	U.S. Department of Education, Office of the Under Secretary. (2003). When schools stay open late: The national evaluation of the 21st-Century Learning Centers program, first year findings. Washington, DC, Author. Available at www.ed.gov/pubs/21cent/firstyear. Moore, M., Dynarksi, M., Mullens, J., James-Burdumy, S., & Rosenberg, L. (2000). Enhancing the 21st-Century Community Learning Centers evaluation: A concept paper. Princeton, NJ: Mathematica Policy Research, Inc.

Contacts

Evaluation	Mark Dynarski, Ph.D. Mary Moore, Ph.D. Principal Investigators Mathematica Policy Research, Inc. P.O. Box 2393 Princeton, NJ 08543-2393 Tel: 609-799-3535 Email: mdynarski@mathematica-mpr.com
	Betsy Warner Project Officer Planning and Evaluation Service US Department of Education 400 Maryland Ave., SW Washington, DC 20202 Tel: 202-401-3600 Email: elizabeth_warner@ed.gov
Program	Robert Stonehill Office of Elementary and Secondary Education US Department of Education 400 Maryland Ave., SW Washington, DC 20208-5524 Tel: 202-260-9737 Email: rstonehi@inet.ed.gov
Profile Updated	April 25, 2003

Evaluation 1: When Schools Stay Open Late: The National Evaluation of the 21st-Century Community Learning Centers Program, First Year Findings

Evaluation Description

Evaluation Purpose	To examine how 21st CCLC programs impact students' academic skills and test scores, sense of safety and self-care, and behaviors in the classroom and out of school, and whether various types of students are affected differently by programs and what program factors, if any, are related to improved outcomes.
Evaluation Design	Experimental and Quasi-Experimental: The evaluation includes a quasi-experimental middle school study and a random assignment experimental elementary school study. Quasi-Experimental The middle school study is based on a nationally representative sample of after school programs and participants and a matched comparison group of students similar to the program participant group. Similar students were identified in host schools or in other schools in the participating districts, using propensity score matching techniques. Thirty-four school districts and 62 centers in these districts are included in the study. Participant and comparison group students were similar on many baseline characteristics, including the percentage of families receiving food stamps and TANF, the percentage of families who did not speak English as their primary language at home, and measures of parental discipline, student empathy, good behavior, and the extent to which students feel that they control their own futures. At the same time, a larger proportion of middle school participants lived in households with low annual incomes than did comparison students, and participants were more likely than comparison students to be from a single-parent household and to have a mother or father with less than a four-year college degree. These participants also demonstrated more behavioral problems and less confidence in their reading abilities relative to the matched controls. The regression models used to estimate impacts adjust for observed participant-comparison differences in these characteristics. Experimental The elementary school study uses random assignment of students to treatment and control groups. The first year of the study involves seven school districts. The elementary school programs that were part of the study appear to be typical of elementary school 21st CCLC programs along most dimensions (although they tended to be more urban and served a larger percentage of minority students than the average elementary program). The evaluators warn that caution should be exercised in applying the findings to all elementary school programs. Programs in the study had more applicants for their slots than they could serve, which facilitated the use of an experimental design, but the programs were not statistically sampled. Because the first-year findings may change when the full set of elementary school grantees is included in the analysis, the evaluators also caution that elementary school findings in this report should be viewed as preliminary. Evaluators collected baseline and follow-up data for 4,400 middle school students and 1,000 elementary school students, and also conducted site visits. Since the program operates through grants administered to local education agencies each year, there are different cohorts of grantees operating 21st CCLC programs. Programs in this evaluation come from either the first, second, or third cohort of grantees. At the start of data collection activities, first-cohort grantees were beginning their third year of funding and second and third cohort grantees were beginning their second full year of funding. Second-cohort grants were awarded in November 1998 and some grantees may not have begun serving students until the fall of 1999. Evaluation activities in 6 of the 34 middle school programs are funded by the Mott Foundation (hereafter referred to as Mott sites). The Mott sites were selected based on the extent to which they were aligned with the vision of 21st CCLC programs and represented geographic and demographic diversity. The Mott sites were selected after all candidate sites were screened through phone interviews and site visits.
Data Collection Methods	Document Review: Annual performance reports submitted by grantees to the US Department of Education were analyzed to gather information on general characteristics and context of 21st CCLC programs. Observation: Evaluators made site visits, lasting between two and four days, to all grantees at least once. The six Mott sites were visited twice. Site visit reports were coded using qualitative analysis software, and site visitors completed several assessment forms that allowed researchers to categorize center programs (for example, to distinguish the degree of emphasis placed on academics and developmental activities). Secondary Sources/Data Review: School records of elementary and middle school students provided information about grades, promotion, and school attendance. Data on sample students' daily attendance in the 21st CCLC program were also collected. Surveys/Questionnaires: Questionnaires administered to parents, teachers, and students provided background information about school characteristics, out-of-school time, program content, and program functioning and staffing. Other items measured included: academic/cognitive outcomes, social/emotional outcomes, and behavior outcomes, as well as mediating factors for program outcomes, such as individual and family characteristics, parent-child relationships, and community and parent involvement in the program. Teacher surveys also included information about programs' collaboration with the school and with other community organizations. Teachers who received questionnaires were English teachers of the middle school students and regular classroom teachers of the elementary school students. Tests/Assessments: SAT-9 reading and math test scores were collected for elementary students in the impact sample. When districts did not test students on a schedule consistent with the evaluation, Mathematica administered the reading test to students.
Data Collection Timeframe	Data were collected during the 2000–2001 school year.

Findings:
Formative/Process Findings

Activity Implementation	Programs were typically offered four to five days per week. Nearly nine of ten middle school centers (89%) provided homework help. Slightly more than half of middle-school centers (54%) provided homework help and other academic support, such as tutoring, state test preparation, and help sessions in reading, writing, and math skills. Nonacademic activities at the middle school centers included recreation, such as using the gym, playing board games, or using computers, and cultural activities, such as crafts, drama, music, mentoring, role modeling and conflict resolution, and issue forums. Site visitors observed that homework sessions at middle school centers usually were organized with students in large groups proctored by teachers or other staff members, with students talking to each other and staff members not checking the homework for quality or completeness. The elementary centers tended to be open most of the after school hours (3pm–6pm, Monday through Friday). A typical after school schedule in an elementary school center included time for a snack and homework, followed by one or two sessions of academic activities or enrichment and recreation. Typically, centers were open 10 or more hours per week, after school, and one-third were open 20 hours or more per week. Some were open on Saturdays, and many offered summer programs. In general, students at elementary school centers had 45 minutes to an hour to work on homework, an hour of another academic activity, and one to two hours for other activities. Elementary school centers varied in how much opportunity students were given to select after school activities. Four of the grantees that focused on academic support required students at their centers to attend both the homework period and the cognitive activity. Two grantees allowed students at their centers more choice. Elementary site visitors witnessed several chaotic homework sessions in which few students focused on their work and few staff members were engaged with students. Although some centers maintained better control, evaluators report that centers did not give students much help with their homework.
Costs/Revenues	Center budgets averaged about $196,000 a center, or about $1,000 per enrolled student, with the 21st CCLC grant accounting for about 70% of budgets. Programs typically were free both for students and parents. The average grant award for the first three cohorts of grantees was slightly under $400,000. Fifty-five percent of middle school coordinators were paid by the hour, with an average hourly wage of about $17. Most other staff members also were paid by the hour, with an average hourly wage of about $16.
Program Context/Infrastructure	Most grantees that were part of the study had operated some type of after school program before receiving a 21st CCLC grant and were using their grant funds to expand or modify their services and activities. About 65% of middle school grantees and about 57% of elementary school grantees in the study had operated after school programs in one or more schools that were part of the 21st CCLC grant. Nationwide, the average grantee ran three or four centers. Most centers (95%) were located in elementary or middle schools or located in schools that included some combination of K–8. Sixty-six percent of host schools were considered high-poverty (at least half their students were eligible for free or reduced-price lunches). Nationally, 17% of schools are high-poverty. Middle school program students felt no safer after school than did similar middle school students. Programs did not show effects on whether elementary students felt safe or unsafe after school either.
Program-School Linkages	Centers at three elementary school sites had strong links with the regular school program. The links arose from shared curricula or lessons that focused on the same standardized tests, and natural links created when centers hired staff members from the regular school to run or supervise activities. A common element among grantees that did not have strong links was that few teachers from the regular school worked in the after school centers.
Recruitment/Participation	Many centers' policies allowed students to participate on a drop-in basis, choosing each day whether or not to participate. Program attendance averaged about two days per week for elementary students and about one day per week for middle school students. Nationwide, the average grantee reported enrollment of almost 700 students over the course of the school year. Of all participants, 57% were minority students, compared with 37% of students nationwide. Attendance varied by day, with some students attending regularly and others more occasionally, and with students enrolling and exiting from the program throughout the year. Program staff attributed the low attendance to the lack of interesting or appealing activities and to competition from other organized activities, especially sports. Middle school students in the study attended centers for 32 days—about one day a week—during the 2000–2001 school year. More than half attended for fewer than 25 days, a quarter attended for more than 50 days, and almost 10% attended for more than 75 days. The analysis suggests that frequent middle school participants were more likely to be from disadvantaged households and more likely to want to improve in school than non-frequent participants. For example, middle school participants who attended frequently were more often black (37%, compared to 20%) and living in single-parent households (33%, compared to 27%) than were non-frequent participants. They also had lower average household incomes and higher rates of public assistance receipt. However, mothers of frequent participants were less likely to have dropped out of high school. Elementary school students attended for 58 days, on average, during the school year, and more than one-third of students attended for more than 75 days. These attendance levels may not be typical of attendance levels of elementary school programs in general because the evaluation looked only at oversubscribed programs.
Staffing/Training	The student-staff ratio across centers was about 11 to 1. Academic activities had much lower ratios than recreational activities. For coordinators, the workweek was four to five days a week, five hours a day. For other staff members, it was three days a week, three hours a day, often in cycles and not continuously throughout the school year. About a third of coordinators and three-fifths of other staff members were teachers. Survey data showed that middle school teachers believed that, as a result of working with students at the centers, they improved their teaching skills and had better relationships with some students.
Systemic Infrastructure	Programs did not collaborate much with other community organizations. In general, centers contracted with community agencies to provide specific after school sessions rather than as partners with shared governance or combined operations. Programs were slow to begin planning to sustain themselves after the end of their 21st CCLC grant. Even among those grantees within months of their grant's end, sustainability planning was almost nonexistent.

Summative/Outcome Findings

Academic	Fewer than two in five middle school students (38%) said that the centers were a good place to get homework done. Three-quarters of parents of middle school participants said they believed participation would help their child do better in school. Teachers reported a statistically significant increase in classroom effort by middle school participants compared to comparison students (p<.01). Teachers were no more likely, however, to report that program participants performed better than comparison students academically. Teachers reported that about one-third of each group achieved at an “above-average” or “very high” level, and 52% of teachers in each group reported that students “get good grades on tests.” Middle school participants exhibited a statistically significant improvement in school attendance compared to comparison group members (9 days absent during the 2000–2001 school year vs. 10.1 days absent for comparison students, p<.01). Middle school participants were also late for class significantly fewer times than comparison students (5 days vs. 6.2 days, p<.01). For middle school students, grades in most subjects were not different than for similar students not attending the 21st CCLC program. Grades for math were higher for 21st CCLC participants, but the overall difference was small and only marginally statistically significant (80.3 points for participants vs. 79.5 points for comparison students, p<.10). A subgroup analysis found improved math grades for black (1.7 points, p<.05) and Hispanic (1.5 points, p<.10) middle school students and no math grade improvement for whites. School records also showed less absenteeism and tardiness for black and Hispanic participants compared with black and Hispanic comparison students. Records indicated that black participants were absent from class 1.6 fewer days (p<.05) and late for class 1.8 fewer days (p<.01) than black nonparticipants. Hispanic participants were absent from class 1.3 fewer days per week (p<.05) and were late for class 1.6 fewer days (p<.05) than Hispanic nonparticipants. Middle school participants were more likely than comparison group students (82.9% vs. 79.7%, p<.05) to say they expected to graduate from college. Teachers for middle school students were more likely to say assignments were completed to their satisfaction (58% of participants vs. 53.3% of comparisons, p<.01), although program participants were no more likely than comparison students to do or complete the homework assigned. For middle school students, programs that emphasized academic activities over recreation and other activities were no more likely to increase test scores or grades. A subgroup analysis found that students who attended programs more frequently, both at the middle school and elementary school levels, did not have higher academic outcomes compared with students that attended less frequently. For elementary students, the programs showed no significant effects on reading or math grades or reading test scores. For example, in spring 2001, program students had an average percentile reading score of 34.3, compared with a score of 34.1 for similar students. Social studies grades were higher by a statistically significant margin (83, compared with 80, p<.05), but grades in other subjects were not. Although grades in these other subjects generally appeared higher for treatment students, the differences were not statistically significant. According to elementary students' teachers, program students were more likely than similar students to try hard in reading or English class, although this difference was only marginally statistically significant (57% of participants vs. 49.5% of control group students, p<.10). According to parents, however, program students were marginally less likely than similar students to work hard in school (80.7% of participants vs. 87.4% of control group students, p=.06). For elementary students, there were no significant differences in participants' vs. nonparticipants' reports of homework completion or time spent reading for fun.
Family	At the middle school level, programs were associated with increased parent involvement at their child's school. Parents of program participants were more likely to volunteer at their child's school (17.8% of participants' parents vs. 14.5% of comparison group members' parents, p<.05) and attend open houses (27.4% of participants' parents vs. 19.1% of comparison group members' parents, p<.01) or parent-teacher organization meetings (33.8% of participants' parents vs. 27.6% of comparison group members' parents, p<.01) three or more times per year. For middle school students, increases in involvement for parents in two-parent families were larger than for parents in single-parent families. For example, participation led to a 14 percentage point increase in parents from two-parent households attending open houses, but only a 6 percentage point increase for single parents (p<.01). High-attendance grantees had a larger impact (14.6 percentage points) than low- and medium-attendance grantees (0 to 4 percentage points) on attendance at parent-teacher organization meetings (p<.05). Centers serving elementary students increased the percentage of parents helping their child with homework at least three times in the last week, with 68% of parents of treatment students and 58% of parents of control students doing so, an effect size of 21% (p<.05). Centers also increased the percentage of parents asking their child about class work: 73% of parents of treatment students and 65% of parents of control students asked about class work at least seven times in the past month (effect size 16%, p<.10). Centers also increased parent attendance at after school events: 52% of parents of treatment students and 42% of parents of control students attended at least three after school events in the past year (effect size 19%, p<.05). Elementary centers did not affect several other indicators of parent involvement, such as whether parents checked that their child had completed homework, attended school events such as open houses and parent-teacher organization meetings, or volunteered in the school.
Prevention	Although rates were not high, middle school participants were more likely to report that they sold drugs (3.3% of participants vs. 1.8% of comparison group students, p<.01), smoked marijuana (3.7% of participants vs. 2.7% of comparison group students, p<.10), and had their personal property damaged (16.9% of participants vs. 14.1% of comparison group students, p<.05). Other measures of behavior—such as suspensions, absences, and teacher reports of discipline problems—were the same in both groups. A subgroup analysis found that girls were especially more likely than boys to have had their personal property damaged (p<.01) or to have been “picked on” (p<.05). Programs did not show effects on elementary school students' behavior in school. Suspensions, absences, and teacher reports of discipline problems were the same for both the treatment and control groups.
Youth Development	Middle school program students had the same values as comparison students on a social engagement composite variable, which includes the extent to which students report that they get along with others their age and feel included. Middle school program students also had similar values on a peer interaction/empathy composite variable, which includes the extent to which students report being good at working with others in a team and believing the best about other people. These participants were less likely than their nonparticipant peers to rate themselves as good or excellent at working out conflicts with others (60.7% of participants vs. 65% of comparison group students, p<.05). Elementary program students were no more likely than control students to report getting along with others their age, feeling included, being good at working with others in a team, or setting a goal and working to achieve it. Elementary school participants were no more or less likely to have spent more time watching television than control students. Centers did not seem to affect whether students were in self-care (defined as students being by themselves, with friends, or with younger siblings after school, and not being cared for by an adult). In the middle school sample, about 17% of both participants and comparison group students were in self-care at least three days in a typical week at the time of the follow-up survey. In the elementary school sample, roughly 2% of both treatment and control group students were in self-care at least three times in a typical week. Frequent middle school participants were significantly less likely than moderate middle school participants to be in self-care at least three times per week (13.2% vs. 18.3%, p=.01). Centers increased the amount of time children spent in non-parent care after school, and decreased the amount of time spent in parent or sibling care. For example, 53.2% of middle school participants were in their parents' care at least three days after school in a typical week, as compared to 59.2% of comparison students (p<.01). Over 4% (4.6%) of middle school participants were in sibling care three days per week, as compared to 7.2% of comparison students (p<.01). In contrast, 20.2% of middle school participants were in non-parent care at least three times per week as compared to 11.7% of comparison students. For middle school participants, this shift in care meant significantly more time spent in tutoring (0.6 days per week vs. 0.3 days per week, p<.01), non-homework reading, writing, or science activities (1.4 days per week vs. 1.2 days per week, p<.05), school activities such as band or drama (1.0 days per week vs. 0.6 days per week, p<.01), organized sports (1.4 days per week vs. 1.2 days per week, p<.01), clubs (0.4 days per week vs. 0.2 days per week, p<.01), surfing the Internet (2.0 days per week vs. 1.8 days per week, p<.01), and “hanging out” with friends (2.7 days per week vs. 2.4 days per week, p<.01) than comparison students. In the elementary centers, the treatment group was moderately significantly less likely than control students to be in their parents care three days per week (62.8% vs. 70.2%, p<.10) or in their siblings' care three days per week (2.0% vs. 4.6%, p<.10). Elementary treatment students were significantly more likely than control students to be in non-parent adult care (31.2% vs. 20.7%, p<.01). There were virtually no significant differences, however, in the types of activities elementary treatment students participated in after school. Control group students were significantly more likely than treatment group students to have participated in clubs such as the Boy and Girl Scouts or the Boys and Girls Clubs than were treatment group students (27.8% of controls vs. 18.2% of treatment students, p<.05). Treatment group students, on the other hand, were significantly more likely than controls to report taking care of a brother or sister after school (44.8% of treatment students vs. 28.1% of controls, p<.01). Frequent elementary participants were significantly more likely than moderate elementary participants to be engaged in tutoring, school activities such as band or drama, and various types of lessons such as art lessons, music lessons, or dance lessons (p<.05). These frequent participants were also significantly less likely to be taking care of a brother or sister after school (p=.01) and moderately significantly less likely to be doing chores around the house after school (p=.10) than were control students.

A Profile of the Evaluation of 21st Century Community Learning Centers—national

Profile Information