Non-Experimental: The study took place over five years, three of which were spent visiting sites, with each year focused on different content areas: reading and math in the first year, science and arts in the second year, and technology and homework help in the third year. High-functioning afterschool sites were identified through a comprehensive screening of 21st CCLC-funded programs throughout the United States. Programs (which contained multiple sites) selected for inclusion in the study:

• Had large numbers of youth attending
• Had been in operation for more than 2–3 years
• Delivered the requisite content and dosage of that content within their sites
• Met or exceeded their stated goals
• Had high levels of quality as reported by teacher surveys
• Demonstrated increased academic performance of program attendees

Based on these criteria, 53 programs were recruited for the study. Two sites from each program were selected for site visits, during which researchers interviewed a total of 338 program staff (project directors, site coordinators, and site instructors), and administered surveys to 645 program staff, 2,600 parents of participants, and 424 teachers at schools that program participants attended.

To assess quality, researchers identified 13 key indicators of success in delivering quality content in an afterschool setting, based on extensive literature review and consultation with experts in the field. Specifically, successful programs had:

1. Clear goals
2. Research-based activities aligned to achieve goals
3. Afterschool materials aligned with state standards
4. Links between afterschool activities and school-day activities
5. Research-based curriculum and teaching strategies
6. A positive environment
7. Motivational strategies to engage youth participants in learning
8. Promotion of youth engagement
9. Effective program management/support/resources
10. Opportunities for youth to practice skills
11. Periodic evaluation to check program effectiveness
12. Periodic assessment to review youth participants’ progress
13. Goals reset according to assessment results

These 13 components were used to develop the instruments with which to validate promising practices during site visits.

Interviews/Focus Groups: Staff interviews gathered information on the general nature and structure of the program to better identify the qualities of an exemplary “best practices” program. Different interview protocols were developed for project directors, site coordinators, and site instructors to specifically address questions that were most relevant to each role. All interview protocols covered general program background information, content area instructional strategies, program organization and structure, external communication and support, youth–adult interactions, program outcomes, and impact on youth participants.

Observation: A structured observation protocol was used during site visits. The protocol included questions focusing on the content and quality of afterschool instructional practice.

Secondary Source/Data Review: APR/PPICs data provided information on programs including their objectives, grade levels served, number of youth served, youth participant demographics and academic achievement data, time spent on specific content curriculum, number of staff, and percentage of credentialed staff members.

Surveys/Questionnaires: The staff survey included questions about content-specific curriculum, general instructional practices and activities, and organizational and structural characteristics.

The parent survey focused on perceived program satisfaction, opportunities for parental involvement, and perceptions of the programs’ impacts on youth.

The teacher survey looked at teachers’ knowledge of the afterschool curriculum, as well as their perceptions of changes in student attitudes, knowledge, and performance due to participation in the program.

The majority of the programs reported offering three or more activities each day. Most included homework help or tutoring activities. Other activities focused on academic subjects (e.g., math, reading, writing, science), enrichment (e.g., arts and crafts, cooking, gardening, health and nutrition, cultural activities, computers), and recreation (e.g., sports, dance, drill team, outdoor games).

The content-specific instruction varied from 45 minutes to a little over an hour per session. All programs offered their content-specific instruction at least twice a week.

The majority of programs employed specific strategies to engage youth participants in the afterschool program, placing a particular emphasis on making learning fun in order to maintain interest after a long day at school. These strategies included cross-content integration, diversity of activities, real world examples, dialogic and cooperative learning, culturally significant programming, special consideration for participants’ activity preferences, and the incorporation of enrichment and recreational activities. Intentional or “disguised” learning was also popular with sports, games, discussions, and journal writing; these activities were used to engage learners while pushing learning into the background.

Diversity in activities was a key way of maintaining interest among youth participants. The daily schedule generally comprised a combination of homework help and tutoring, academics, and some enrichment and/or recreational activity. Most programs offered short periods of 30 minutes to an hour per activity, and a few used learning or activity centers that allowed children to move from one activity to the next at their own pace.

Programs tried to engage youth participants by offering real-world examples that connected the curriculum to their lives outside of school. This was particularly salient among science and technology programs. As one site coordinator stated, “We use things like BMX bikes or other things that interest them.”

Programs attempted to make content more relevant by tying it to current events. A few sites used newspapers and magazines in class; others mentioned studying popular culture like hip-hop and movies; some sites studied popular gadgets; and several sites used television as the starting point for an educational activity.

All 53 programs appeared to structure their curricular design and program practices to facilitate meeting their program goals. For example, three of the math programs and four of the reading programs specifically targeted students who were struggling academically.

The common goal of arts programs was to provide art experiences to youth who otherwise would have little to no exposure to the arts. Similarly, all science staff responded to questions about curricular goals by suggesting a primary interest in simply offering positive science experiences to youth.

Technology programs tended to focus on developing youth participants’ applied knowledge. Nearly all of the technology staff interviewed discussed the goal of teaching youth the mechanics of a broad range of technology skills in the interests of encouraging the youth to use those skills to enhance learning in other content areas, and teaching the youth a technical skill that could be relevant to their real lives both now and in the future.

Survey results revealed that many staff included content in their curriculum tied to state academic standards. However, within each content area, some standards were used more often than others. In the content area of reading, more staff reported using standards developed for lower grade levels such as “Vocabulary development” (96%) than standards developed for upper grade levels like “Applying self-correcting strategies to decode text” (55%). This was also true in math, where lower grade-level standards such as “Addition and subtraction of whole numbers” (100%) were more often used than higher grade-level standards such as “Problem solving using equations” (50%).

Most programs indicated that they designed their own curriculum, rather than relying on existing curriculum.

Some of the common research-based practices across the programs included encouraging youth’s learning by infusing the learning process with cultural and social experiences relevant to their lives; using “hands on” approaches in science, arts, and technology; giving youth participants the freedom to test and expand their own knowledge; and shifting between guided work and self-paced work.

Staff across all 53 programs consistently offered a broad range of means by which they shared program and youth progress information with parents. Some of the most frequently cited means of communication were monthly/bimonthly newsletters; pre-academic year orientation; parents’ nights; community outreach activities such as potlucks and youth performances; back-to-school nights; and contact with parents as needed regarding youth progress, attendance, or behavior via letters, notes, phone calls, and scheduled meetings.

The great majority of program staff interviewed acknowledged that although they encouraged parent participation in their programs, parent volunteerism was extremely low. The key reason appeared to be that programs in the study predominantly served families where both parents worked and thus were unavailable to afterschool activities. Staff and parents both also reported low levels of program staff’s meeting with parents on a regular basis.

On a 10-point scale of 1 (never) to 10 (always), parents had an average rating level of approximately “9” on items tapping how often they felt that they were welcome to visit the program at any time, that there were staff available to speak to them in their home language, that they were comfortable talking about program materials, and that they were notified in a timely manner if their children were not paying attention.

Many of the 53 programs mentioned ways in which they connected with the community. The activities most frequently mentioned were service learning (e.g., taking trips to nursing homes to sing to the senior citizens, cultivating community gardens).

Most staff reported benefiting from various types of support from and involvement in the local community. Benefits from these partnerships were especially valuable for arts, science, and technology programs where supplies could be costly. For instance, some arts programs benefited from donated materials and supplies, funding, arts-related outings, and artists-in-residence programming. Many programs also reported getting volunteers through their affiliations with local universities, high schools, or community organizations.

Staff expressed high levels of agreement with the following statements (mean level of agreement above 4.0 on a 5-point scale): youth understand the standards for behavior, programs address youth’s behavioral issues promptly, and program has adequate materials and resources.

Across programs, program staff commonly mentioned a need for additional space. Many programs relied on access to common space, such as an auditorium or a classroom shared with school-day teachers, which often caused logistical problems and sometimes prevented planned activities from taking place. Furthermore, some programs expressed difficulty with not having consistent access to classrooms.

Most programs reported limited access to technology, particularly computers. Although all of the technology programs indicated they had sufficient computers and tools to work with (many received support from technology companies such as Dell or Apple), other programs were frustrated by old equipment and a lack of current software.

When asked about program goals, about half of the staff members mentioned social development. Many strategies were used to enhance social development; the most commonly reported strategies included the use of group activities, multi-age groupings, rewarding prosocial behaviors, and encouraging peer collaboration. Social development was often emphasized through the enrichment activities (e.g., field trips, sports, character-building workshops), which helped to strengthen youth’s self-concept, character, creativity, and community immersion.

Overall, youth interactions with one another at the 53 programs appeared to be positive, nurturing, and respectful.

Interview and survey responses indicated that even though quasi-experimental or experimental studies were rarely carried out, all of the programs were conducting some evaluation activities that were performed either internally by their own staff, externally by outside evaluators/researchers, or both. The methods used in external evaluations typically included pre- and post-testing of various program outcomes, comparison groups of similar youth not involved in the program, surveys, focus groups, and structured observational assessments. The methods used in internal evaluations varied: They included informal conversations between afterschool staff, school staff, and parents; formal surveys of youth participants, parents, and staff; and tracking of test scores, grades, and attendance records.

Only the technology programs reported having content-specific evaluations (i.e., evaluation activities specific to their technology focus). These evaluations were undertaken by a variety of public and private organizations. The evaluations tracked the overall effectiveness of the technology and larger curriculum, often using computer software to monitor youth’s academic progress in school and using that data as a barometer to adjust afterschool programming.

Interviewees most frequently mentioned using the results of internal evaluation to serve as a baseline for instruction, to monitor youth progress, and to document program impact.

The community-based programs all reported conducting internal evaluations on a regular basis, some more intensive and extensive than others. They appeared to have a broader focus than other programs, and utilized a mixture of surveys of parents, staff, youth participants, and classroom teachers to seek feedback. These data served to inform programs from multiple perspectives, provided a continuous feedback loop for self-improvement, and measured program satisfaction rates.

Slightly less than half of the afterschool staff indicated that they communicated with the school staff frequently or regularly, while an equal number of afterschool staff indicated that they never or rarely communicated with school staff.

Most afterschool staff reported in interviews a casual form of communication with the school-day teachers, typically on an “as needed” basis in regard to individual youth progress and/or behavioral issues, and mostly through brief, informal contacts such as email or casual drop-in conversations. An exception was the science programs, where the majority of site coordinators (90%) and site instructors (93%), reported that specific staff members were assigned to or responsible for communication with the school staff.

District-affiliated programs seemed to have easier access to student academic databases, and about half of these programs made use of this access to establish baselines and determine areas of academic need for their participants. The student academic database and classroom grades were also used as outcome measures in determining academic growth and progress, especially for programs with the explicit goal of raising students’ achievement scores.

Most program staff reported in interviews and on surveys that they were familiar with the state academic standards, although levels of familiarity varied. Some program staff stated that they had a strong knowledge of the standards, often because they were also school-day teachers, or because their program built on the school’s curricular structure. In about half of the programs, especially those that were community based, one or more of the interviewees responded that they were not very familiar with the state standards.

On a 10-point scale of 1 (never) to 10 (always), parents had average ratings of approximately “9” on items tapping how often they felt that staff cared about and respected their children and dealt with their children’s behavioral problems promptly.

The majority of program staff indicated in survey responses that site instructors had an active voice in decisions about program curriculum and instruction (mean level of agreement = 4.4 out of 5), and took active roles in program leadership and decision-making (mean level of agreement = 4.1 out of 5).

The majority of staff stayed at their programs over 3 years. Math and reading program staff members had an average of 3.5 years of experience at the current site. Approximately 75% of the science, technology, and arts program staff had between 1–7 years of services at the current site, with 46% of science, 42% of the arts, and 38% of the technology staff working over 4 years at their current site. Homework programs had 60% of the staff members at the current site for 1–7 years and 30% from 4–7 years.

The largest share of program staff reported having a Bachelor’s degree (47%), and another 24% had a Master’s degree in addition to their Bachelor’s degree.

Although 57% of the staff reported that their programs offered professional development 2–4 times a year, only 26% participated in professional development offered through their programs at this frequency, while 60% reported that they never participated, and 14% reported that they participated once.

According to interviews, the most common types of professional development offered across programs were general techniques in working with youth, such as discipline issues and general teaching strategies. Instructors particularly appreciated professional development that was directly applicable in the classroom, such as teaching strategies, innovative approaches, and classroom management.

Positive relationships and interactions between the staff and the youth participants were observed in all of the 53 programs, particularly in terms of expectations for youth performance, disciplinary issues, and youth having a voice in the program.

About 75% of the parents perceived that their children’s skills and interest (i.e., subject-specific skills, interest in the subject, and interest in schoolwork in general) improved since participating in the program, whereas a minimal number of parents (1%) saw a decrease. Approximately 12% of the parents did not perceive any changes in their children’s performance or attitudes, while about 7% were not sure about any changes.

About half of school-day teachers agreed that their students in these afterschool programs had improved in school attendance, were paying more attention in class, and had fewer classroom discipline problems. In addition, 58% of teachers agreed that their students who participated in the afterschool program increased their frequency of class participation, and most teachers (61%) agreed that their students in the afterschool program put more effort into school work.