Search Publications | The Evaluation Exchange | FINE Network | OST Database

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

A Profile of the Evaluation of Extended-Day Tutoring Program

Program Description

Overview	During the 1995–1996 school year, the Extended-Day Tutoring Program was implemented with first through fourth graders in Title I Memphis City Schools. The program used a curriculum based on Success For All (SFA) with the purpose of improving participating students' literacy skills.
Start Date	1995
Scope	local
Type	after school
Location	urban
Setting	public schools
Participants	elementary school students
Number of Sites/Grantees	13 Title I sites
Number Served	656 (1995–1996)

Components	The Extended-Day Tutoring program was developed by the Memphis City Schools' Success For All (SFA) coordinator in collaboration with faculty experts from the University of Memphis and Johns Hopkins University. The program ran for two school years, 1995–1996 and 1996–1997. It used a Success for All (SFA) curriculum as its basis. SFA is a school-wide research-based reform model developed by Robert Slavin and colleagues at Johns Hopkins University based on the notion that all students can and must succeed in the early grades. Founded on the idea that reading is fundamental to other skill areas, the program targets students in lower elementary school grades. The SFA approach includes intensive instruction in language arts for students, professional development to help teachers effectively work with all students, and an active family support component (www.successforall.com/index.htm). The tutoring program was offered three days a week for one hour after school to selected students in grades one through four. Students were nominated for participation by their teachers due to the teachers' perceptions that the students were likely to benefit from the program, but program participation was voluntary. For the first 15 minutes of the program, students were led in SFA Story Telling and Retelling (STaR) and/or Listening Comprehension, depending on their grade level. Next, the children spent 30 minutes reading and doing related follow-up activities using Scott Foreman Book Festival kits. These kits provide multiple copies of each book so that students can engage in partner reading. The final 15 minutes of the program were spent focusing on any of the following activities which are alternated depending on the day: writing, Book Club, computer skills, and test-taking strategies for the Tennessee Comprehensive Assessment Program (TCAP), the state-mandated standardized achievement test. During the first year of implementation, the 1995–1996 school year, the tutoring program began in December due to the time required for program design, obtaining materials, and recruiting teachers.
Funding Level	N/A
Funding Sources	Title I federal funding

Evaluation

Overview	Ross et al. conducted an evaluation to identify the program's strengths, weaknesses, and preliminary process and achievement outcomes after the first year of implementation.
Evaluators	Steven M. Ross, Tracey Lewis, Lana Smith, and Allan Sterbin, University of Memphis
Evaluations Profiled	Evaluation of the Extended-Day Tutoring Program in Memphis City Schools: Final Report to CRESPAR
Evaluations Planned	not available
Report Availability	Ross, S. M., Lewis, T., Smith, L., & Sterbin, A. (1996). Evaluation of the Extended-Day Tutoring Program in Memphis city schools: Final report to CRESPAR. Memphis, TN: University of Memphis.

Contacts

Evaluation	Steven M. Ross Associate Director and Senior Researcher Center for Research in Educational Policy 325 Browning Hall The University of Memphis Memphis, TN 38152 Tel: 901-678-3413 Email: smross@memphis.edu
Program	Not available since the program was discontinued in 1997.
Profile Updated	June 6, 2002

Evaluation: Evaluation of the Extended-Day Tutoring Program in Memphis City Schools: Final Report to CRESPAR

Evaluation Description

Evaluation Purpose	To identify the strengths and weaknesses of the program as well as to examine preliminary academic outcomes after the first year of implementation.
Evaluation Design	Quasi-Experimental: The evaluation employed a matched-pair design in which one student in each pair participated in the tutoring program while the other student did not. Students in grades two through four were matched by the principal using the 1995 Total Reading Normal Curve Equivalent (NCE) scores on the Tennessee Comprehensive Assessment Program (TCAP) as the primary criterion for matching and attitude and behavior as secondary matching criteria. There were 328 matched pairs included in the analysis. While first graders also participated in the tutoring program, they were not included in part of the evaluation since they do not undergo TCAP testing. The intention of the researchers was that one member of each pair would be randomly assigned to the program; in fact, it is unclear whether principals executed random assignment or whether they selected students for participation in the program and then found a matching student who would not receive tutoring. Nonetheless, comparisons of treatment and comparison group students reveal no significant differences in pre-program TCAP NCE scores.
Data Collection Methods	Observations: Observations of the program were made by four Memphis city teachers working as consultants for Title I. All four teachers had expertise with the SFA reading program. They made a total of 153 separate observations of program activities, rating the quality of each observed activity as: “not observed,” "poor,” “satisfactory,” or “good.” The observers also provided accompanying comments to justify their ratings. In addition to these formal observations, two evaluators made a total of six informal observations in order to gain a general impression of the program. Observations were made throughout the 1995–1996 school year. Surveys/Questionnaires: Teachers who taught in the after school tutoring program, of whom there were 62, were surveyed regarding their perceptions of the strengths and weaknesses of the program. The survey also elicited suggestions for program improvement. The surveys had a response rate of 77%. Surveys were conducted in the spring of 1996. Tests/Assessments: In addition to using 1995 TCAP NCE test scores to place students in matched pairs, 1996 TCAP NCE scores of students in both the treatment and comparison groups were studied to determine if there was an impact of the program on children's reading abilities. The TCAP is given in the spring.

Findings:
Formative/Process Findings

Activity Implementation	Generally, observers felt that the program was well implemented by teachers in all of its components, citing satisfactory quality and fidelity to the planned activities. Some teachers were observed to be more skilled than others and more conscientious about adhering to the program plan. Three problems that appeared to come up again and again in observations were: (1) time management, (2) less emphasis on writing and TCAP skill-building than had been intended, and (3) an overemphasis on teacher presentations at the expense of active student learning. First Grade Observations For the first grade, many observations were made of the Star/Listening Comprehension and Trade Book activities, but few observations were made of the skills instruction (TCAP or computer), the writing, book club, or peer sharing activities. For both Star/Listening Comprehension and Trade Books, 73% and 65% of ratings, respectively, were “good.” Observers noted that first graders appeared to be very involved and attentive during the Star/Listening Comprehension activities. The activity tended to include a story being read to the students, followed by group discussion and comprehension questions from the teacher. In many rooms, partner reading was observed and new vocabulary was introduced to the students and discussed. In most cases in the observed first-grade classes, the Trade Book activity entailed the class reading the book together as a group in addition to some partner reading. Sometimes, the teacher asked predictive questions about the story. Also, a variant was that students listened to a story on tape. The activity generally included time for students to discuss the stories and work on vocabulary as well as engage in various follow-up activities including drawing illustrations, discussing rhyming words, story mapping, discussing opposites, comparing different objects, and classifying things. Observers noted that most students seemed interested and engaged in the activity. Observations of the skills/TCAP/computer activity were rare in the first grade, but one class was seen practicing skills in letter/sound associations, another was observed practicing skills related to directions, and a third completed an activity for the Book Festival. Observations of first grade writing activities were infrequent, but observed activities included writing stories or sentences with a partner or silent reading and writing vocabulary words. Second Grade Observations For second grade, most of the observations were of Star/Listening Comprehension and Trade Book activities. For both of these areas, almost all ratings were “satisfactory” or “good,” with 55% of Star/Listening Comprehension activities rated as “good” and 56% of Trade Book activities rated as “good.” A total of 10 observations were made of skills/TCAP/computer activities and six were made of writing activities; all of these were rated “satisfactory” or “good.” Only two Book Club activities were observed and no peer sharing activities were observed. Second-grade students appeared to be interested and engaged in the Star/Listening Comprehension activity. Typically, teachers read the story to students and led a group discussion about it. Sometimes, the children listened to the story on tape. The teachers asked predictive and comprehension questions about the story and discussed new vocabulary with the students. Some observers noted a problem that teachers spent too much time reading and used time that was allotted for the Trade Book activity. The Trade Book activity for the second grade usually took the form of partner reading, although some observed classes had the teacher reading the book to the students, students reading silently, alternating student silent reading with the teacher reading aloud, listening to the book on tape, and the students reading the book chorally as a class. The students reviewed new vocabulary, made predictions, and answered the teachers' questions. Writing and sometimes drawing activities were observed during most second grade Trade Book activities. Most students appeared engaged, but in some of the classes students did not appear to be focused on the lesson. During the skills/TCAP/computer activity, second graders were observed practicing sounds, compound words, sequencing, rhyming words, and contractions. Some students were observed working on reading skills in the computer lab. Generally, students appeared to be engaged in this activity. Of the few observations of second grade classes engaged in writing activities, observers noted that some students wrote about their vocabulary words while others typed letters on the computer. Second graders engaged in Book Club were observed on several occasions making oral reports. These Book Club sessions appeared to be engaging for students. Third Grade Observations Observers made relatively fewer visits to third-grade classes. The few ratings made of the Star/Listening Comprehension and Trade Book activities were almost always “good.” Third graders in the Star/Listening Comprehension activity were observed to be actively engaged. Third graders in the Trade Book activities were observed reviewing vocabulary words, being read to, discussing stories, and asking questions. Sometimes, children listened to the story on tape while other times the teacher and students took turns reading a page at a time, and still other times the students read silently. Many varied activities were observed among third graders in this activity including: partner reading and writing answers or summaries after reading a chapter in the book. Some students were observed reading their answers to the class. Relatively few observations were made of the skills/TCAP/computer sessions for third graders. Observed activities included: computer lab work on reading skills, computer lab work on language arts skills, TCAP skills training incorporated into a vocabulary lesson. Fourth Grade Observations Observations of fourth grade activities yielded relatively fewer “good” ratings and more “poor” ratings than for lower grades, but most of the ratings were still positive. Fourth grade teachers were generally observed to be following Star/Listening Comprehension procedures. Students discussed vocabulary, read and discussed the story, made predictions, and appeared to be interested and engaged. There were many positive comments from observers about the establishment of clear lesson plans. In some classes listening comprehension activities were not observed to be occurring during the scheduled time. Instead, fourth graders were engaged in story mapping and round robin reading aloud, among other activities. Most fourth graders were observed to be doing partner reading during the Trade Books activity, although some students were observed to be silently reading. In some classes, they worked on vocabulary, made predictions, reviewed characters in the story, and wrote about elements of the story. According to observers, some teachers did not appear to be following any type of model and there were no questions, interactions, or predictions. During the skills/TCAP/computer, fourth graders were predominantly observed completing skill sheets from one of their books. Some engaged in TCAP practice such as synonyms, while others did webbing for a character in the story, read a story from a computer monitor, or organized ideas for writing a paragraph. For the fourth grade, one observation was made of a writing activity in which students were observed writing meaningful sentences from words in the story. Fourth graders were observed once engaged in the Book Club activity during which time three students were presenting oral book reports to the class. Another observation of a Book Club activity showed students engaged in a skills activity instead.
Recruitment/Participation	Ninety-seven percent of the tutored students were promoted to the next grade at the end of the 1995–1996 school year. The mean attendance rate for the program was 75%; 71% of the children attended more than 80% of the sessions.
Satisfaction	According to teacher surveys, the most frequently cited strength of the program was the books, frequently described as high-interest material that met student needs, with 43% of teachers reporting this strength. The second most common strength noted was the program materials (25% of teachers), closely followed by cooperative learning (17%), the training program (15%), the partner reading/discussion (11%), and the small groups/classes (11%). The most frequent survey response from teachers when asked about program weaknesses was to omit answering the question (26%). Of the answers actually given, 15% of teachers noted that the program should have begun earlier in the year, 15% mentioned behavior problems, 9% felt that there was not enough teacher input in the program, 9% felt that students had too few skills, 7% felt that the program was not congruent with SFA, 7% felt that groups were too large, 7% felt that kids were tired and bored, 4% felt that erratic attendance was a problem, and 2% felt that math skills need reinforcement too. The most frequent of the suggestions for improvement made by teachers in surveys were to begin the program earlier in the year and to provide more money for material and supplies. Also, there were recommendations from teachers that they increase their flexibility to adapt the program to individual children's needs, involve parents more, and have smaller groups.
Staffing/Training	Teachers reported that the training they received was effective and that they felt ready to implement the program when they began.

Summative/Outcome Findings

Academic

The evaluators used ANCOVA, with the covariate specified as 1995 TCAP sores, and found that the pretest (1995) scores were significantly (p<.001) related to posttest (1996) scores. They were then able to control for this in subsequent analyses to determine the program effect.

The evaluators analyzed the TCAP test results by doing two major analyses: the first analysis compared a treatment group of students assigned to tutoring who attended at least 50% of the time against a comparison group of students originally assigned to the treatment group who attended less than 50% of the time and students who had originally been assigned to the comparison group. The second analysis compared only those students in a matched pair where both students had taken the pretest and posttest (1995 and 1996 TCAP) to each other. Within these two main groups, evaluators looked separately at students who attended more than 80% of the time and students who attended more than 50% of the time and alternately included and left out two schools which showed discrepancies between the pretest scores of the matched pairs. At one of the two “outlier” schools, there was only a .47 correlation between the 1995 TCAP scores of the matched pairs and at the other there was a correlation of -.10; all the remaining schools had correlations of at least .94 between the two pretest scores of the matched pairs.

First Analysis: Participants Attending More Than 50% of the Time Versus Low- or Non-Attenders (Less Than 50% of the Time) Plus the Original Comparison Group

Looking only at students attending more than 80% of the time (excluding students who attended between 50% and 80% of the time) and excluding the two outlier schools, there was a significant difference only between tutored students in the third grade and low-attenders and non-tutored students in the third grade (p<.009). The mean TCAP NCE score was 8.5 points higher for third graders who participated more than 80% of the time in the tutoring program. For second and fourth graders there were non-significant differences between the tutored students and low- or non-attenders, 6 points higher for the tutored students in grade two and 6 points lower for the tutored students in grade four.
Looking only at students attending more than 80% of the time (excluding students who attended between 50% and 80% of the time) and including the two outlier schools, there were comparable results with only grade three showing a significant (p<.009) program effect of a 9-point mean score difference, which favored tutored students. Non-significant differences for grades two and four were as follows: tutored second graders had a mean score 5 points higher than low- or non-attenders and tutored fourth graders had a mean score 6 points lower than low- or non-attenders.
Looking at those students who attended more than 50% of the time and excluding the two outlier schools, again only grade three showed significant (p<.034) program effects. This difference favored tutored third graders by close to 6 TCAP NCE points. For grades two and four, there were non-significant differences between the tutored and low- or non-attending students of 2 points and close to 0 points, respectively.
Looking at those students who attended more than 50% of the time and including the two outlier schools, there were still only significant (p<.035) program effects for grade three of 5.5 mean NCE points higher for the tutored group as compared to the low-attending and non-tutored group. Non-significant differences of 2.6 points (favoring the tutored children) and 0 points were found for grades two and four, respectively.

Second Analysis: Tutored Students Versus Matched Pair Comparison Group Students

Looking only at students attending more than 80% (excluding students who attended between 50% and 80% of the time) of the time and excluding the two outlier schools, only the third grade results showed significant (p<.002) program effects. The tutored third grade students showed an 11-point mean NCE score advantage over their non-tutored matched pairs. Second and fourth grades showed differences of 1.2 points and -2.1 points, i.e., the matched pair non-tutored fourth grade students scored 2.1 points lower than the tutored students, respectively.
Looking at those students who attended more than 50% of the time and excluding the two outlier schools, the results still hold: only grade three showed significant (p<.022) program effects of 6.8 points favoring the tutored students over their matched pairs. For grades two and four, the non-significant results were .2 points and -2.1 points, respectively.
For both 80% attendance and 50% attendance using the full samples of matched pairs (including the two outlier schools), the results were nearly identical to the above results. Both of these analyses showed significant program effects for grade three, non-significant score advantages for tutored second graders over their matched pairs, and non-significant score disadvantages for tutored fourth graders over their matched pairs.
Throughout the analyses, only grade three showed significant program effects. These program effects were strongest, approximating 8.5–11 points, when the tutoring sample only included those students who attended 80% or more of the sessions and when the two outlier schools were removed. In contrast, the program effect was weakest, about 5.5 points, when the tutoring sample included students who attended 50% of the time and all schools were included.
In all analyses, TCAP NCE non-significant score differences directionally favored tutored students in grade two and were close to zero or slightly higher for the comparison group in grade four.

A Profile of the Evaluation of Extended-Day Tutoring Program

Profile Information