Program participants were recruited from youth enrolled in afterschool programs at the selected centers who were behind grade level, but not by more than 2 years; were identified by afterschool program staff as in need of supplemental academic support to meet local academic standards; and had a baseline test score on either the math or the reading portion of the Stanford Achievement Test Series, Tenth Edition (SAT 10).

Two curriculum developers adapted their school-day materials to be used as a structured enhanced model for math (Mathletics) and reading (Adventure Island). These models provided 45 minutes of daily instruction after school, 4 days per week. Mathletics instruction is built around five mathematical themes: numbers and operations, measurement, geometry, algebra and functions, and data analysis and probability. Adventure Island includes elements identified by the National Reading Panel (2000): phonemic awareness, phonics, fluency, vocabulary, comprehension, and strategic reading.

References:
Harcourt School Publishers. (n.d.). Harcourt Mathletics. Boston, MA: Houghton Mifflin Harcourt.

National Reading Panel. (2000). Teaching children to read: An evidence-based assessment of the scientific literature on reading and its implications for reading instruction. Bethesda, MD: National Institute of Child Health and Human Development.

Findings After the First Year of Implementation

Final Report

Black, A. R., Doolittle, F., Zhu, P., Unterman, R., & Grossman, J. B. (2008). The evaluation of Enhanced Academic Instruction in After-School Programs: Findings after the first year of implementation (NCEE 2008-4021). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Available at: http://ies.ed.gov/pubsearch/pubsinfo.asp?pubid=NCEE20084021

Black, A. R., Somers, M., Doolittle, F., & Unterman, R. (2009). The evaluation of Enhanced Academic Instruction in After-School Programs: Final report (NCEE 2009-4077). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Available at: http://ies.ed.gov/ncee/pubs/20094077/index.asp

To address the following questions:

• Does enhanced academic instruction in afterschool programs improve math or reading proficiency over what students would achieve in regular afterschool programs?
• What are the impacts of enhanced afterschool instruction on subgroups of students?
• Does enhanced afterschool instruction affect other in-school academic behavior outcomes?

The evaluation also examined how the instruction was implemented.

Experimental and Non-Experimental: Youth in grades 2 through 5 were randomly assigned by grade into either the enhanced or regular program group. For the math centers, 1,081 youth were assigned to the Mathletics enhanced program and 880 were assigned to the regular program. For the reading centers, 1,048 youth were assigned to the Adventure Island enhanced program and 780 were assigned to the regular program. Youth assigned to the enhanced instruction were offered special instruction during a 45-minute time block, while youth assigned to the regular program received the existing academic support services (e.g., homework assistance). Both enhanced and regular program groups received similar services for the remainder of the afternoon schedule.

A standardized assessment was administered in Fall 2005 (before random assignment) and in Spring 2006 (posttest) to assess math and reading skills. A literacy assessment was fielded in Spring 2006 for students in grades 2 and 3 in both the enhanced and the regular reading programs. State and local achievement test score data were also obtained from school records (with response rates generally between 80% and 90% depending on the sample).

Instructor surveys were conducted with all instructors in both years, including data for 209 instructors serving the enhanced program group and 160 instructors serving the regular program group; these data were collected from February to April 2006. In addition, surveys were conducted with regular school-day teachers in the spring of 2006 (surveys were available for over 98% of youth in the sample).

Evaluators observed the 45-minute instruction modules for half of the instructors serving the enhanced program group (n = 51 for the math group and n = 50 for the reading group, randomly sampled). In addition, local district coordinators observed all Mathletics and Adventure Island instructors three times per year across the school year. Following the instruction module observation of a session, evaluators interviewed half of the instructors serving the enhanced program group (randomly sampled). Interviews were conducted from February to April 2006.

Interviews/Focus Groups: Interviews with enhanced program instructors focused on their perspectives of the strengths and weaknesses of the enhanced program, how program implementation evolved over time, challenges in implementing the enhanced programs, how these challenges were addressed, and suggestions for improvement.

Observation: Observations of the 45-minute instruction modules involved ratings of the use of instructional practices, which included organization, clarity of presentation, modeling of concepts, monitoring of student progress, cooperative peer learning, and classroom management practices. Each of these elements was rated on a scale of 1 (needs improvement) to 4 (outstanding). District coordinator observations focused on the fidelity of instruction to the Mathletics and Adventure Island models.

Secondary Source/Data Review: Data were collected from schools housing the afterschool centers about school context, student body demographics, and student-to-teacher ratios.

Daily program attendance data were collected from program centers for all days when the enhanced instruction was offered.

Data on Adequate Yearly Progress (AYP) status of the schools housing afterschool centers were collected from state departments of education websites for the 2005–06 school year.

Surveys/Questionnaires: Instructor surveys focused on staff characteristics, the nature of activities that staff led or participated in, their experience with the materials they used, and the support they received to implement the services they provided.

Student surveys covered such issues as receipt of academic support outside regular school hours from sources other than the afterschool program, sources of help with homework, sense of adult support, and expectations from afterschool program staff.

Regular-school-day teachers covered such topics as whether students received individual academic help during the regular school day, whether they completed their homework, and their behavior in class.

Test/Assessments: The Stanford Achievement Test, Tenth Edition (SAT 10), abbreviated battery for math or reading (depending on the intervention implemented) was used to measure the gains in achievement. The Dynamic Indicators of Basic Early Literacy Skills (DIBELS) measured reading fluency for the second- and third-grade students.

State-administered tests from regular-school-day student records, which included test scores on standardized tests, were collected to provide another measure of academic achievement.

References:
Center on Education Teaching and Learning. (2007). Dynamic Indicators of Basic Early Literacy Skills Data System. Eugene, OR: University of Oregon College of Education. Available at: http://dibels.uoregon.edu/index.php

Harcourt Assessment, Inc. (n.d.). Stanford Achievement Test Series, Tenth Edition—Abbreviated Battery, Areas of Assessment. San Antonio, TX: Author. Available at: http://harcourtassessment.com/HAIWEB/Cultures/en-us/Productdetail.htm?Pid=SAT10A&Mode=summary&Leaf=SAT10A_2

Students in the enhanced math sites received an average of 170 minutes of instruction per week (intended amount of instruction was 180 minutes). All of the math centers offered Mathletics for a minimum of 70 days during the school year, with 4 centers offering 70–79 days of instruction, 4 centers offering 80–89 days, 4 centers offering 90–99 days, 10 centers offering 100–109 days, and 3 centers offering 110 days or more.

Students in the enhanced reading programs received an average of 176 minutes of instruction per week (intended amount of instruction was 180). All of the reading centers offered Adventure Island for a minimum of 70 days during the school year, with 6 centers offering 70–79 days of instruction, 4 centers offering 80–89 days, and 15 centers offering more than 90 days of instruction.

The majority of enhanced math instructors surveyed reported that the Mathletics materials addressed the topics on which students needed help (74%) and were at “about the right level of difficulty” (88%).

The majority of enhanced reading teachers surveyed reported that the materials of Adventure Island were appropriate for their students (93%) and that the materials were at “about the right level of difficulty” (94%).

All but one of the enhanced math teachers interviewed indicated that they experienced some challenges related to keeping up with the intended pacing of the enhanced program model. Some found the pacing a consistent problem (16%), sometimes a problem (30%), or initially a problem that then improved (8%). The remaining 46% found pacing a rare problem.

All of the enhanced reading instructors interviewed indicated experiencing some challenges related to keeping up with the intended pacing of the enhanced program model. These instructors found the pacing a consistent problem (42%), sometimes a problem (32%), or rarely a problem (26%).

Of the 25 schools that housed the afterschool centers offering the enhanced math instruction, 10 were located in a large or midsize city, 8 were within the fringe of the large or midsize city, 4 were in a large or small town, and 3 were in a rural area. The racial and ethnic composition of the student body in these schools was 44% Black, 32% White, and 19% Hispanic. Approximately 75% of students received free or reduced-price lunch and the average student-to-teacher ratio in these schools was 15:1. Five of the 25 schools did not meet AYP for the 2005–06 school year.

Of the 25 schools that housed the afterschool centers offering the enhanced reading instruction, 19 were located in large or midsize cities, 4 were located on the fringe of a large or midsize city, and 2 were located in a large or small town. The racial and ethnic composition of the student body in these schools was 60% Black, 13% White, and 24% Hispanic. Approximately 81% of students received free or reduced-price lunch and the average student-to-teacher ratio in these schools was 15:1. Eleven of the 25 schools did not meet AYP for the school year 2005–06.

Students in the enhanced math program attended significantly more out-of-school math instruction over the school year than those in the regular program, in both the number of days and hours attended: They attended 12 more days and averaged 49 more hours of math instruction than the regular program group (p < .05 for each).

Students in the enhanced reading group attended significantly more out-of-school reading instruction than the regular program group, in both the number of days and hours attended: They attended 7 more days and averaged 48 more hours of reading instruction than the regular program group (p < .05 for each).

Students in the enhanced math program participated in significantly (p < .05) more classes or activities in math outside of school (29%), compared with students assigned to the regular program group (21%).

Students in the enhanced reading program participated in significantly (p < .05) more classes or activities in reading outside of school (39%), compared with students assigned to the regular program group (31%).

Students in the enhanced math group and the regular program received similar amounts of individualized instruction per week (an average of 50 minutes vs. 49 minutes).

Students in the enhanced reading group and the regular program group received similar amounts of individualized instruction per week (an average of 84 minutes vs. 87 minutes).

In the math programs, enhanced program instructors were significantly (p < .05 for each) more likely to be certified teachers (97% vs. 62%) and to have had more than four years of elementary teaching experience (78% vs. 65%), and less likely to have no elementary teaching experience (0% vs. 11%). Additionally, the enhanced program averaged a significantly smaller student-to-instructor ratio than the regular program (9:1 vs. 11:1, p < .05).

In the reading programs, enhanced program instructors were significantly (p < .05 for each) more likely to be certified teachers (99% vs. 60%) and to have had more than four years of elementary teaching experience (75% vs. 55%), and significantly less likely to have no elementary teaching experience (0% vs. 13%). Additionally, the enhanced program averaged a significantly smaller student-to-instructor ratio than the regular program (9:1 vs.14:1, p < .05).

Instructors providing math support in the regular programs were significantly less likely than instructors in the enhanced programs (p < .001 for each) to report having received high quality training to carry out their work (55% vs. 94%) or 30 minutes or more of paid daily preparation time (12% vs. 91%).

Instructors providing reading support in the regular programs were significantly less likely than instructors in the enhanced programs (p <.001 for each) to report having received high quality training to carry out their work (58% vs. 97%) and 30 minutes or more of paid daily preparation time (35% vs. 84%).

Participants in the enhanced math program showed significantly greater average pretest to posttest improvements than the regular program group (p < .05 for each) in their scores on SAT 10, abbreviated battery math test overall (average gain of 35.8 vs. 33.0 points), and for the two subtests of problem-solving (2.5 scaled score points higher for the enhanced group) and procedures (4.3 scaled score points higher for the enhanced group).

For the younger math subgroup (grades 2 and 3), the difference between the enhanced and regular program groups in total math scores was not significant. However, for the older math subgroup (grades 4 and 5), gains on math test scores were significantly higher for students in the enhanced math program (p = 0.01).

Participants in the enhanced math program did not show any significant differences in homework completion, attentiveness in class, or disruptiveness in class when compared to students in the regular program group.

Participants in the enhanced reading program did not show any significant differences as compared to the regular program group on any measure of reading scores or academic behavior, either for the full analysis sample or for the various subgroups.

To address the following questions: What is the impact on student achievement of offering students the opportunity to participate in the enhanced afterschool program for 1 school year? Is this impact different in the second year of implementation than in the first year? What is the impact of offering students the opportunity to participate in the enhanced afterschool programs for 2 consecutive years? The evaluation also examined how the instruction was implemented.

Experimental and Non-Experimental: The second-year study included 27 afterschool centers (15 for math, 12 for reading) that voluntarily agreed to participate in the study for a second year. The centers provided the same type of enhanced afterschool program (math or reading) as they had provided in the first year of the study.

This study is based on a two-stage random assignment design of students, in which students were randomly assigned by grade within each afterschool center on two separate occasions—once at the beginning of the first year of the study (first stage in Fall 2005) and then again at the beginning of the second study year (second stage in Fall 2006).

In each of the math and reading sites there were students who had 0, 1, and 2 years of being offered the program. For the analyses involving 2-year impacts the sample size was 270 students (169 enhanced program) in the reading sites, and 367 students (227 enhanced program) in the math sites.

There were some analyses of a single year of program offerings included in the second-year study. These included students in school year 2005–06 offered 1 year of the program (Cohort 1) and students in school year 2006–07 offered 1 year of the program (Cohort 2).  In the study sample, the number of students offered 1 year of treatment across both school years includes 905 in Cohort 1 (504 enhanced program) and 626 (352 enhanced program) in reading, and 1,144 in Cohort 2 (634 enhanced program) and 792 (461 enhanced program) in math.

A standardized assessment was administered in the fall and spring of each school year, and student surveys were administered in the spring of the school year (with response rates above 80% for virtually all samples and measures). A literacy assessment was fielded in Spring 2006 for students in grades 2 and 3 in both the enhanced and regular reading programs. The literacy assessment was fielded again in the spring of 2007 for all students in both programs. State and local achievement test score data were also obtained from school records (with response rates generally between 80% and 90% depending on the sample).

Instructor surveys were conducted with all instructors in both years, including data for approximately 230 instructors serving the enhanced program group and 180 instructors serving the regular program group; these data were collected from February to April 2006 and February to April 2007. In addition, surveys were conducted with regular-school-day teachers in the spring of 2006 and 2007 (and were available for over 99% of youth in the sample).

Research staff interviewed half of the instructors serving the enhanced program group (randomly sampled). Interviews were conducted from February to April 2006. Evaluators also interviewed two randomly sampled instructors serving the regular afterschool program group at each center from March to April 2007. Research staff also interviewed all district coordinators from March to April 2007.

Evaluators observed the 45-minute instruction modules for half of the instructors serving the enhanced program group (n = 51 for the math group and n = 50 for the reading group, randomly sampled). In addition, local district coordinators observed all Mathletics and Adventure Island instructors three times per year across each school year. Following the instruction module observation of a session, evaluators interviewed half of the instructors serving the enhanced program group (randomly sampled). Interviews were conducted from February to April in each year.

Interviews/Focus Groups: Interviews with enhanced instructors focused on their perspectives of the strengths and weaknesses of the enhanced program, how their implementation evolved over time, challenges in implementing the enhanced programs, how these challenges were addressed, and suggestions for improvement.

Interviews with regular program instructors focused on the academic focus of the afterschool activities, the content covered each day, the use of assessments, and where materials were drawn from.

Interviews with district coordinators focused on the strengths and weaknesses of the enhanced program, how it evolved over time, how challenges were addressed, and recommendations for the future.

Observation: Observations of the 45-minute instruction modules involved ratings of the use of instructional practices, which include organization, clarity of presentation, modeling of concepts, monitoring of student progress, cooperative peer learning, and classroom management practices. Each of these elements was rated on a scale of 1 (needs improvement) to 4 (outstanding). District coordinator observations focused on the fidelity of instruction to the Mathletics and Adventure Island models.

Secondary Source/Data Review: Data were collected from schools housing the afterschool centers about school context, student body demographics, and student-to-instructor ratios.

Daily program attendance data were collected from program centers for all days when the enhanced instruction was offered.

Data on Adequate Yearly Progress (AYP) status of the schools housing afterschool centers were collected from state departments of education websites for the 2005–06 school year.

Surveys/Questionnaires: Instructor surveys focused on staff characteristics, the nature of activities that staff led or participated in, their experience with the materials they used, and the support they received to implement the services they provided.

Student surveys covered such issues as receipt of academic support outside regular school hours from sources other than the afterschool program, sources of help with homework, sense of adult support, and expectations from afterschool program staff.

Regular-school-day teacher surveys covered such topics as whether students received individual academic help during the regular school day, whether they completed their homework, and their behavior in class.

Test/Assessments: The Stanford Achievement Test, Tenth Edition (SAT 10), abbreviated battery for math or reading (depending on the intervention implemented) was used to measure the gains in achievement. The Dynamic Indicators of Basic Early Literacy Skills (DIBELS) measured reading fluency for the second- and third-grade students.

State-administered tests from regular-school-day student records, which included test scores on standardized tests, were collected to provide another measure of academic achievement.

References:
Center on Education Teaching and Learning. (2007). Dynamic Indicators of Basic Early Literacy Skills Data System. Eugene, OR: University of Oregon College of Education. Available at: http://dibels.uoregon.edu/index.php

Harcourt Assessment, Inc. (n.d.). Stanford Achievement Test Series, Tenth Edition—Abbreviated Battery, Areas of Assessment. San Antonio, TX: Author. Available at: http://harcourtassessment.com/HAIWEB/Cultures/en-us/Productdetail.htm?Pid=SAT10A&Mode=summary&Leaf=SAT10A_2

On average, students in the enhanced program group in Cohort 1 received 48 more hours of academic instruction in math during the school year than students in the regular program group. This difference—which was significant (p < .001)—represented an estimated 30% increase in total math instruction over and above what was received by these students during the regular school day. In Cohort 2, enhanced program students received 42 more hours—also a significantly greater amount of time (p < .001) than received by those in the regular program group—and an estimated 26% increase in total math instruction. However, the added hours of math instruction were significantly less in the second year of implementation than in the first year of implementation (42 hours vs. 48 hours, p < .001).

In the classes with students at the first- and second-grade reading levels (in Adventure Island, students are grouped by their initial reading level, not by grade), average fidelity scores (a measure of how well instructors implemented the enhanced program as intended) did not significantly differ across the first and second years of implementation; in the classes with students reading above the second-grade level, average fidelity scores were significantly lower in the second year than in the first year (p < .001).

Observations revealed that, in any given year, implementation of the program lacked consistency, as indicated by variation in the number of program components implemented by instructors. In particular, in the second year, returning instructors had significantly higher implementation fidelity scores than instructors who were new to the program (p < .001).

On average, students in the enhanced program reading group in Cohort 1 received 54 more hours of academic instruction in reading during the school year than students in the regular program group. This difference—which was significant (p < .001)—represents a 22% increase in total reading instruction over and above what was received by these students during the regular school day. In Cohort 2, enhanced program students received 56 more hours—also a significantly greater amount of time (p < .001) than received by those in the regular program group—and an estimated 23% increase in total reading instruction. However, the net difference in added hours of instructional reading between implementation years was not significant.

The enhanced math program students in Cohort 2 did not significantly differ from regular program students on gains in SAT 10 total math scores, in contrast to the differences found in Cohort 1, where enhanced program students significantly outperformed their control group peers (3.5 scaled score difference in Cohort 1, p < .05). The difference in impacts between Cohort 1 and 2 samples was not significant.

One year of enrollment in the enhanced math program was associated with increases in students’ performance on locally administered standardized math tests for Cohort 2 (p = .01), and the difference across cohorts was not significant.

One year of enrollment was not associated with impacts on regular-school-day teacher reports of academic behavior (homework completion, attentiveness in class, and disruptiveness in class) in either year.

Participating in the enhanced afterschool program for 2 consecutive years was not associated with any significant impacts for any math outcome above and beyond participating for 1 year.

One year of enrollment in the enhanced reading program did not have a significant impact on any measure of reading achievement (nor on school-day teacher reports of academic behavior (homework completion, attentiveness in class, and disruptiveness in class).

Participating in the enhanced afterschool program for 2 consecutive years was associated with declines in SAT 10 total reading scores compared to students in the control group (p < .05).