Program participants were recruited from youth enrolled in afterschool programs at the selected centers who were behind grade level, but not by more than 2 years; were identified by afterschool program staff as in need of supplemental academic support to meet local academic standards; and had a baseline test score on either the math or the reading portion of the Stanford Achievement Test Series, Tenth Edition (SAT 10).

Two curriculum developers adapted their school-day materials to be used as a structured enhanced model for math (Mathletics) and reading (Adventure Island). These models provided 45 minutes of daily instruction after school, 4 days per week. Mathletics instruction is built around five mathematical themes: numbers and operations, measurement, geometry, algebra and functions, and data analysis and probability. Adventure Island includes elements identified by the National Reading Panel (2000): phonemic awareness, phonics, fluency, vocabulary, comprehension, and strategic reading.

References:
Harcourt School Publishers. (n.d.). Harcourt Mathletics. Boston, MA: Houghton Mifflin Harcourt.

National Reading Panel. (2000). Teaching children to read: An evidence-based assessment of the scientific literature on reading and its implications for reading instruction. Bethesda, MD: National Institute of Child Health and Human Development.

Findings After the First Year of Implementation

Final Report

Black, A. R., Doolittle, F., Zhu, P., Unterman, R., & Grossman, J. B. (2008). The evaluation of Enhanced Academic Instruction in After-School Programs: Findings after the first year of implementation (NCEE 2008-4021). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Available at: http://ies.ed.gov/pubsearch/pubsinfo.asp?pubid=NCEE20084021

Black, A. R., Somers, M., Doolittle, F., & Unterman, R. (2009). The evaluation of Enhanced Academic Instruction in After-School Programs: Final report (NCEE 2009-4077). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Available at: http://ies.ed.gov/ncee/pubs/20094077/index.asp

To address the following questions:

• Does enhanced academic instruction in afterschool programs improve math or reading proficiency over what students would achieve in regular afterschool programs?
• What are the impacts of enhanced afterschool instruction on subgroups of students?
• Does enhanced afterschool instruction affect other in-school academic behavior outcomes?

The evaluation also examined how the instruction was implemented.

Experimental and Non-Experimental: Youth in grades 2 through 5 were randomly assigned by grade into either the enhanced or regular program group. For the math centers, 1,081 youth were assigned to the Mathletics enhanced program and 880 were assigned to the regular program. For the reading centers, 1,048 youth were assigned to the Adventure Island enhanced program and 780 were assigned to the regular program. Youth assigned to the enhanced instruction were offered special instruction during a 45-minute time block, while youth assigned to the regular program received the existing academic support services (e.g., homework assistance). Both enhanced and regular program groups received similar services for the remainder of the afternoon schedule.

A standardized assessment was administered in Fall 2005 (before random assignment) and in Spring 2006 (posttest) to assess math and reading skills. A literacy assessment was fielded in Spring 2006 for students in grades 2 and 3 in both the enhanced and the regular reading programs. State and local achievement test score data were also obtained from school records (with response rates generally between 80% and 90% depending on the sample).

Instructor surveys were conducted with all instructors in both years, including data for 209 instructors serving the enhanced program group and 160 instructors serving the regular program group; these data were collected from February to April 2006. In addition, surveys were conducted with regular school-day teachers in the spring of 2006 (surveys were available for over 98% of youth in the sample).

Evaluators observed the 45-minute instruction modules for half of the instructors serving the enhanced program group (n = 51 for the math group and n = 50 for the reading group, randomly sampled). In addition, local district coordinators observed all Mathletics and Adventure Island instructors three times per year across the school year. Following the instruction module observation of a session, evaluators interviewed half of the instructors serving the enhanced program group (randomly sampled). Interviews were conducted from February to April 2006.

Interviews/Focus Groups: Interviews with enhanced program instructors focused on their perspectives of the strengths and weaknesses of the enhanced program, how program implementation evolved over time, challenges in implementing the enhanced programs, how these challenges were addressed, and suggestions for improvement.

Observation: Observations of the 45-minute instruction modules involved ratings of the use of instructional practices, which included organization, clarity of presentation, modeling of concepts, monitoring of student progress, cooperative peer learning, and classroom management practices. Each of these elements was rated on a scale of 1 (needs improvement) to 4 (outstanding). District coordinator observations focused on the fidelity of instruction to the Mathletics and Adventure Island models.

Secondary Source/Data Review: Data were collected from schools housing the afterschool centers about school context, student body demographics, and student-to-teacher ratios.

Daily program attendance data were collected from program centers for all days when the enhanced instruction was offered.

Data on Adequate Yearly Progress (AYP) status of the schools housing afterschool centers were collected from state departments of education websites for the 2005–06 school year.

Surveys/Questionnaires: Instructor surveys focused on staff characteristics, the nature of activities that staff led or participated in, their experience with the materials they used, and the support they received to implement the services they provided.

Student surveys covered such issues as receipt of academic support outside regular school hours from sources other than the afterschool program, sources of help with homework, sense of adult support, and expectations from afterschool program staff.

Regular-school-day teachers covered such topics as whether students received individual academic help during the regular school day, whether they completed their homework, and their behavior in class.

Test/Assessments: The Stanford Achievement Test, Tenth Edition (SAT 10), abbreviated battery for math or reading (depending on the intervention implemented) was used to measure the gains in achievement. The Dynamic Indicators of Basic Early Literacy Skills (DIBELS) measured reading fluency for the second- and third-grade students.

State-administered tests from regular-school-day student records, which included test scores on standardized tests, were collected to provide another measure of academic achievement.

References:
Center on Education Teaching and Learning. (2007). Dynamic Indicators of Basic Early Literacy Skills Data System. Eugene, OR: University of Oregon College of Education. Available at: http://dibels.uoregon.edu/index.php

Harcourt Assessment, Inc. (n.d.). Stanford Achievement Test Series, Tenth Edition—Abbreviated Battery, Areas of Assessment. San Antonio, TX: Author. Available at: http://harcourtassessment.com/HAIWEB/Cultures/en-us/Productdetail.htm?Pid=SAT10A&Mode=summary&Leaf=SAT10A_2

Students in the enhanced math sites received an average of 170 minutes of instruction per week (intended amount of instruction was 180 minutes). All of the math centers offered Mathletics for a minimum of 70 days during the school year, with 4 centers offering 70–79 days of instruction, 4 centers offering 80–89 days, 4 centers offering 90–99 days, 10 centers offering 100–109 days, and 3 centers offering 110 days or more.

Students in the enhanced reading programs received an average of 176 minutes of instruction per week (intended amount of instruction was 180). All of the reading centers offered Adventure Island for a minimum of 70 days during the school year, with 6 centers offering 70–79 days of instruction, 4 centers offering 80–89 days, and 15 centers offering more than 90 days of instruction.

The majority of enhanced math instructors surveyed reported that the Mathletics materials addressed the topics on which students needed help (74%) and were at “about the right level of difficulty” (88%).

The majority of enhanced reading teachers surveyed reported that the materials of Adventure Island were appropriate for their students (93%) and that the materials were at “about the right level of difficulty” (94%).

All but one of the enhanced math teachers interviewed indicated that they experienced some challenges related to keeping up with the intended pacing of the enhanced program model. Some found the pacing a consistent problem (16%), sometimes a problem (30%), or initially a problem that then improved (8%). The remaining 46% found pacing a rare problem.

All of the enhanced reading instructors interviewed indicated experiencing some challenges related to keeping up with the intended pacing of the enhanced program model. These instructors found the pacing a consistent problem (42%), sometimes a problem (32%), or rarely a problem (26%).

Of the 25 schools that housed the afterschool centers offering the enhanced math instruction, 10 were located in a large or midsize city, 8 were within the fringe of the large or midsize city, 4 were in a large or small town, and 3 were in a rural area. The racial and ethnic composition of the student body in these schools was 44% Black, 32% White, and 19% Hispanic. Approximately 75% of students received free or reduced-price lunch and the average student-to-teacher ratio in these schools was 15:1. Five of the 25 schools did not meet AYP for the 2005–06 school year.

Of the 25 schools that housed the afterschool centers offering the enhanced reading instruction, 19 were located in large or midsize cities, 4 were located on the fringe of a large or midsize city, and 2 were located in a large or small town. The racial and ethnic composition of the student body in these schools was 60% Black, 13% White, and 24% Hispanic. Approximately 81% of students received free or reduced-price lunch and the average student-to-teacher ratio in these schools was 15:1. Eleven of the 25 schools did not meet AYP for the school year 2005–06.

Students in the enhanced math program attended significantly more out-of-school math instruction over the school year than those in the regular program, in both the number of days and hours attended: They attended 12 more days and averaged 49 more hours of math instruction than the regular program group (p < .05 for each).

Students in the enhanced reading group attended significantly more out-of-school reading instruction than the regular program group, in both the number of days and hours attended: They attended 7 more days and averaged 48 more hours of reading instruction than the regular program group (p < .05 for each).

Students in the enhanced math program participated in significantly (p < .05) more classes or activities in math outside of school (29%), compared with students assigned to the regular program group (21%).

Students in the enhanced reading program participated in significantly (p < .05) more classes or activities in reading outside of school (39%), compared with students assigned to the regular program group (31%).

Students in the enhanced math group and the regular program received similar amounts of individualized instruction per week (an average of 50 minutes vs. 49 minutes).

Students in the enhanced reading group and the regular program group received similar amounts of individualized instruction per week (an average of 84 minutes vs. 87 minutes).

In the math programs, enhanced program instructors were significantly (p < .05 for each) more likely to be certified teachers (97% vs. 62%) and to have had more than four years of elementary teaching experience (78% vs. 65%), and less likely to have no elementary teaching experience (0% vs. 11%). Additionally, the enhanced program averaged a significantly smaller student-to-instructor ratio than the regular program (9:1 vs. 11:1, p < .05).

In the reading programs, enhanced program instructors were significantly (p < .05 for each) more likely to be certified teachers (99% vs. 60%) and to have had more than four years of elementary teaching experience (75% vs. 55%), and significantly less likely to have no elementary teaching experience (0% vs. 13%). Additionally, the enhanced program averaged a significantly smaller student-to-instructor ratio than the regular program (9:1 vs.14:1, p < .05).

Instructors providing math support in the regular programs were significantly less likely than instructors in the enhanced programs (p < .001 for each) to report having received high quality training to carry out their work (55% vs. 94%) or 30 minutes or more of paid daily preparation time (12% vs. 91%).

Instructors providing reading support in the regular programs were significantly less likely than instructors in the enhanced programs (p <.001 for each) to report having received high quality training to carry out their work (58% vs. 97%) and 30 minutes or more of paid daily preparation time (35% vs. 84%).

Participants in the enhanced math program showed significantly greater average pretest to posttest improvements than the regular program group (p < .05 for each) in their scores on SAT 10, abbreviated battery math test overall (average gain of 35.8 vs. 33.0 points), and for the two subtests of problem-solving (2.5 scaled score points higher for the enhanced group) and procedures (4.3 scaled score points higher for the enhanced group).

For the younger math subgroup (grades 2 and 3), the difference between the enhanced and regular program groups in total math scores was not significant. However, for the older math subgroup (grades 4 and 5), gains on math test scores were significantly higher for students in the enhanced math program (p = 0.01).

Participants in the enhanced math program did not show any significant differences in homework completion, attentiveness in class, or disruptiveness in class when compared to students in the regular program group.

Participants in the enhanced reading program did not show any significant differences as compared to the regular program group on any measure of reading scores or academic behavior, either for the full analysis sample or for the various subgroups.