You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Program Description

Overview The Summer Bridge (SB) program in Chicago, Illinois, is a central component of Chicago Public Schools’ (CPS) efforts to end social promotion. The goal of the program is to give low-achieving students the extra help they need to remediate poor skills and meet test-score cutoffs for promotion to the next grade.
Start Date summer 1997
Scope local
Type summer/vacation
Location urban
Setting public schools
Participants elementary and middle school students (grades 3, 6, and 8)
Number of Sites/Grantees about 400 elementary schools
Number Served over 21,000 per year in 1997–2000
Components All CPS students scoring below a given test score on the Iowa Test of Basic Skills (ITBS) are required by CPS to attend SB. The initial promotional test score cutoffs were 1 year below grade level for third graders, 1.5 years below grade level for sixth graders, and 1.8 years below grade level for eighth graders. Special education students and students in bilingual education for 4 years or less (3 years or less prior to 1999) are exempt from the policy. Third and sixth graders attend SB 3 hours per day, 5 days per week, for 6 weeks. Eighth graders attend 4 hours per day, 5 days per week, for 7 weeks. Students are taught by regular CPS teachers in classes of approximately 16–18 students.

The SB curriculum is closely aligned with the ITBS. Teachers are not given any flexibility in deviating from the prescribed curriculum but can tailor it to their students’ needs. For example, teachers are allowed to emphasize reading if most of their students only need to pass the reading cutoff score. The reading curriculum focuses on developing reading comprehension concepts (e.g., main idea, inference) and skills by using a variety of reading materials repeatedly over the course of the summer. The math curriculum focuses on problem solving, data interpretation, and computation and provides instruction in both the application of basic math computation skills and in analysis and estimation.
Funding Level $34 million per summer in 1997
Funding Sources Chicago Public Schools


Evaluation

Overview The 1997–2000 evaluation aimed to determine the program’s short- and long-term impacts, how these impacts varied for student subpopulations, and the nature of students’ and teachers’ experiences in SB as well as the overall learning environment.
Evaluators Melissa Roderick, Mimi Engel, Jenny Nagaoka, Susan Stone, Consortium on Chicago School Research
Evaluations Profiled Ending Social Promotion: Results From Summer Bridge

Getting It Right the Second Time Around: Student Classroom Experience in Chicago’s Summer Bridge Program
Evaluations Planned Future evaluations will examine outcomes for students retained under CPS’ policies, many of whom attended SB.
Report Availability Roderick, M., Engel, M., & Nagaoka, J. (2003). Ending social promotion: Results from Summer Bridge. Chicago: Consortium on Chicago School Research. Available at www.consortium-chicago.org/publications/p59.html.

Stone, S. I., Engel, M., Nagaoka, J., & Roderick, M. (2005). Getting it right the second time around: Student classroom experience in Chicago’s Summer Bridge Program. Teachers College Record, 107, 935–957.


Contacts

Evaluation Melissa Roderick, Ph.D.
Associate Professor, University of Chicago
Consortium on Chicago School Research
1313 East 60th Street
Chicago, IL 60637
Tel: 773-702-1171
Fax: 773-702-2010
Email: m-roderick@uchicago.edu
Program Chicago Public Schools
Department of School and Regions
125 S. Clark
10th Floor
Chicago, IL 60603
Tel: 773-553-2150
Fax: 773-553-2151
Profile Updated May 19, 2006

Evaluation 1: Ending Social Promotion: Results From Summer Bridge


Evaluation Description

Evaluation Purpose To answer the following questions: (a) Is SB effective in increasing students’ test scores and allowing more students to be promoted, and do these effects vary by students’ preprogram skill levels? (b) Does SB produce uniform effects across students and schools? (c) Does SB provide a positive learning environment? (d) How do staffing characteristics, teachers’ implementation of the curriculum, and classroom learning environments shape SB’s impacts? and (d) Does SB provide help for low-achieving students that is sustained over time?
Evaluation Design Quasi-Experimental and Non-Experimental: Data were gathered to look at students’ and teachers’ experiences in SB and the quality of the learning environments offered to students. Evaluators also examined how the quality of the learning environments varied according to characteristics of the schools and classrooms offering SB, adjusting statistically for teacher and student characteristics (and classroom-level characteristics when looking at quality by school-level characteristics).

To determine SB’s short-term impacts, test-score increases of SB students were compared to those reported in a meta-analysis of other summer school programs nationwide,¹ as well as to gains made by SB students during the regular school year. More specifically, evaluators used students’ test-score data from their entire academic testing history to predict their pre-SB test scores and subtracted this from their actual post-SB test scores to obtain estimates of the gains related to participation in SB. Test-score data were obtained for all SB participants who completed the program and were tested at the end of the year; 80% of students who were required to attend SB were tested at the end of the program (although it is not known whether the remaining 20% actually attended or not, nor if all students who were tested attended SB). In total, about 14,000 students in grades 3, 6, and 8 were included in the analytic sample: 6,300 in third grade, 4,400 in sixth grade, and 3,300 in eighth grade.

To determine longer term impacts, evaluators compared the 2-year learning trajectories of SB students who were slightly below the test-score cutoffs (assessed on the ITBS as up to approximately 3 Grade Equivalent months below the cutoff) before the program (“high-scoring SB students”) to those of students who did not attend the program because they were slightly above the cutoffs (assessed on the ITBS as up to approximately 3 Grade Equivalent months above the cutoff; “the comparison group”).

The evaluators then compared differences in trajectories found between high scoring SB students and the comparison group to differences found between two analogous groups of scorers prior to SB’s implementation (two groups of students with the same ranges of ITBS scores but who obtained these scores without then being eligible or ineligible for SB). This latter comparison is meant to correct for the fact that SB students may have simply experienced bad testing days and therefore demonstrated “regression to the mean” or alternatively, that SB students may “mature” and catch up to their higher scoring peers. The analysis is restricted to first-time SB enrollees (to distinguish between the effects of SB and the effects of retention). This analysis was also restricted to the SB youth who did not meet the reading score cutoff. (Although some SB youth also failed to meet the math score cutoff, almost all failed to meet the reading score cutoff.) Sample sizes in the longer term impacts analysis were approximately 5,000 in the SB third grade group and 4,000 in the third grade comparison group, and 3,400 in the SB sixth grade group and 3,400 in the sixth grade comparison group.

The evaluators caution that the summative findings for third graders may be less reliable than those for children in higher grades because third grade test score data demonstrate that these young children have a higher probability of scoring below “chance,” or the expected test scores of students who answer every question at random. This may indicate that third graders do not yet have the rudimentary test-taking skills necessary to complete the assessments used in the SB evaluation.

¹ Cooper, H., Charlton, K., Valentine, J., & Muhlenbruck, L. (2000). Making the most of summer school: A meta-analysis and narrative review. Monographs of the Society for Research in Child Development, 65, 260. Malden, MA: Blackwell.
Data Collection Methods Interviews/Focus Groups: Evaluators interviewed a subsample of youth from the Consortium’s larger evaluation of Chicago’s initiative to end social promotion. This original longitudinal sample included 102 low-achieving African American and Latino sixth and eighth graders. Evaluators interviewed 48 out of the 63 students who failed to meet test score cutoffs and were therefore enrolled in SB. These interviews focused on students’ experiences in SB and during the regular school year. In addition, 37 interviews were conducted with SB teachers, asking about their students, the curriculum, and the support they received in SB. Student interviews were conducted in the summer of 1999 and teacher interviews were conducted in the summer of 2000.

Observation: Classroom observations took place in 12 SB schools. Researchers observed reading and math classes four times in one third, one sixth, and one eighth grade classroom in each of the 12 schools for a total of approximately 140 observations in each subject. Evaluators selected schools by choosing pairs of one high-achieving and one low-achieving school that were from the same area and had similar demographics. The observations examined content of lessons, teachers’ instructional styles, how much time was spent on different activities, classroom configuration, classroom management, and student behavior. Observations of instruction were coded either “tailored” (i.e., instruction that was deemed above and beyond the requirements, adjusting to individual classroom and student needs), “sufficient,” “minimal,” or “insufficient” based on the pace, clarity, and accuracy of presentation, engagement of students, and monitoring of students’ understanding.

Secondary Source/Data Review: CPS provided access to all CPS students’ administrative records, which included demographic data (age, gender, race, ethnicity, grade level, schools attended, and special education/bilingual status).

Surveys/Questionnaires: SB teachers were administered surveys during the last week of the program in 1999. These surveys asked about the quality of the SB curriculum and materials, their instruction, and their students’ needs and performance. Surveys were sent to all SB schools and collected from 1,335 teachers who came from 87% of programs containing third graders, 84% of programs containing sixth graders, and 79% of programs containing eighth graders. The evaluators could not calculate exact response rates within schools because of an inability to determine the number of teachers in each school.

Students were surveyed in the summer of 1999 regarding perceptions of their experiences in SB and comparisons of these experiences with their experiences during the school year. In total, 4,829 sixth graders (66% completion rate) and 4,225 eighth graders (69% completion rate) were included in this sample. Of these students, 2,519 sixth graders and 2,303 eighth graders also responded to a survey in the spring of 1999 as part of the Consortium on Chicago School Research’s biannual survey, which questioned students about experiences in their schools. Thus, evaluators were able to match 55% of sixth graders and 52% of eighth graders who completed summer surveys.

Tests/Assessments: ITBS scores were obtained for students’ entire school careers, for all SB and comparison group students.
Data Collection Timeframe Data were collected between 1997 and 2000.


Findings:
Formative/Process Findings

Activity Implementation Over half of teachers surveyed reported that they could not cover all of the materials and topics required in the time allotted. However, observations revealed that throughout the summer, over 90% of teachers remained on pace or taught the daily lessons at a pace faster than the prescribed pace.

Teachers tended to spend more time on reading than on math, since this was the subject in which the majority of students still needed to meet the promotional cutoff. Some teachers spent more time on reading for the entire class, others used groupings of students to adjust time spent on reading and math for individual students, and still others provided certain students with extra work and attention outside of class time.

Over two thirds of teachers surveyed reported assigning extra work for individual students (in addition to regular homework) at least once a week. About half also reported that they worked with students before or after class on a regular basis (about once a week or more).

Seventy-three percent of reading class observations were found to be either sufficient or tailored, while 18% were found to be insufficient. Sixty-eight percent of math class instructional observations were found to be either sufficient or tailored, while 27% were found to be insufficient. In only about 20% of observations did evaluators find teachers going beyond providing clear and accurate instruction to engage students actively in the learning process, challenge students, or ensure that students were mastering concepts.

Site observations revealed that SB classrooms were characterized by high levels of basic skills practice and instruction focused on prescribed topics. On average, only 8% of SB reading lessons and 5% of SB math lessons were devoted to understanding (more extensive problem solving or other in-depth student work).

Observations revealed that little time was spent on noninstructional activities and that over three quarters of class time was devoted to either providing instruction, practicing skills, or setting up and reviewing assignments and activities.

Despite being encouraged to use small group instruction frequently, observations showed that teachers used small groups, on average, only 5% of the time.

Approximately 57% of teachers agreed that they supplemented the curriculum (as a whole) with basic activities or materials, as compared to only 26% who supplemented it with more advanced activities or materials.
Costs/Revenues SB teachers were paid an average of about $42 per hour; the average third or sixth grade SB teacher was paid about $5,544 for the summer, and the average eighth grade teacher earned approximately $7,719.

The average cost per student for SB teachers and materials was around $528. The cost of an SB teacher was approximately $347 per student in third and sixth grades and $445 in the eighth grade. The cost of all SB materials was approximately $136 per student for third and sixth graders and $171 for eighth graders. Adding costs for administration and operations increased the cost of the program by approximately $100 to $628 per student.

In its first 2 years, transportation for SB cost approximately $1 million per summer. After CPS required most students to attend their neighborhood schools for SB, transportation costs dropped to about $200,000.
Parent/Community Involvement About half of SB teachers reported talking to students’ parents at least once a week.
Program Context/Infrastructure Teachers who knew more of their students, had tutors assisting in their classrooms, and had smaller class sizes were more likely to report adapting instruction to meet their students’ needs and to report an overall positive learning environment in the classroom.

The degree to which students scored below the promotional cutoffs was not related to teachers’ assessments of the curriculum, the overall learning environment, or the degree to which they worked individually with students.

Teachers in high achieving schools were more likely to report providing individualized student support and to feel that the atmosphere of SB was conducive to student learning.

Eighth grade teachers provided more individual attention and better assessments of the curriculum than did teachers in other grades, in part because eighth grade teachers knew a larger proportion of their students prior to SB.
Program-School Linkages Over 80% of SB students in 1999 and 2000 attended the same school during SB as they did during the regular school year. This was a slight drop from analogous rates in 1997 (over 90%), most likely due to the fact that SB changed its policy regarding student transportation. In 1997, SB provided transportation for students whose home schools were not their neighborhood schools, but by 1999 SB required participants to attend their neighborhood schools for SB regardless of whether they attended those schools during the regular school year.

Teacher surveys revealed that 87% of third and sixth grade teachers and 83% of eighth grade teachers reported teaching SB in the same schools in which they taught during the school year. Over 80% of SB teachers were regular classroom reading or math teachers during the school year.

Interviews revealed that SB students perceived the SB learning environment as different from and more positive than during the school year. In particular, they noted the slower pace and more individual attention, as well as an emphasis placed on careful explanation so that students could better understand material. A small minority of interviewees, however, found the pace too slow and repetitive.

In comparing SB survey responses to similar questions on the prior spring’s regular school day survey, students perceived higher levels of academic press (e.g., encouragement, high expectations) and “personalism” (e.g., listening, willingness to provide extra help) in SB. (Survey responses were generally 1–2 points higher for SB on a scale ranging of 1–10.)

Over 85% of surveyed student reported that teachers had more time to help them in SB than during the regular school year.

Eighth graders in predominantly African American schools (85% or more African American students) had larger increases on the measure of personalism from the school year to SB and more positive perceptions of the program (p <. 05 for both) than did eighth graders in schools with different racial/ethnic compositions. Similar patterns were not as prominent for sixth graders in African American schools. There were no differences by racial composition for academic press.
Recruitment/Participation Nearly all (80–100%) high-risk students (those over 1.5 years below grade level averaged across the students’ entire academic trajectory) were required to attend SB. Less than half of sixth and eighth grade and 64% of third grade moderate-risk students (those between 0.5 and 1.5 years below grade level across students’ academic trajectories) were required to attend SB. Less than 10% of low-risk students (those who were less than 0.5 years below grade level across students’ academic trajectories) were required to attend.

Approximately 97% of SB students were African American or Latino, while 85–87% of students who were subject to the promotional policy across the district were African American or Latino.

In 1997, the average SB third, sixth, and eight grader was 0.8, 0.9, and 1.1 years below grade level, respectively, and the learning rate for all three grades over the previous year was only 0.3 years. In 1999, the prior year learning rate was 0.5 months for third graders, 0.4 months for sixth graders, and 0.7 months for eighth graders, meaning that, in 1999, SB students entered SB closer to the promotional cutoff than SB students did in 1997.

Evaluators estimated that 80%–90% of students who were required to attend SB actually attended the program. Eighty percent of students were actually tested at the end of the program, indicating that at least that many participated.

Attendance rates appeared to be similar across grades, gender, race/ethnicity, and students’ prior achievement level.
Satisfaction Over three quarters of sixth graders and 88% of eighth graders reported being “mad” that they had to go to summer school.

Over 70% of student survey respondents reported liking summer school better than school during the regular academic year.

About one third of student survey respondents felt that the work in SB was too difficult.

Eighty percent of teachers surveyed reported being somewhat to very satisfied with the content of the lesson plan guides, and 86% were somewhat to very satisfied with the quality of the materials.

Approximately three quarters of teachers felt positively about the reading and math curricula, agreeing or strongly agreeing that the materials were useful and appropriate and that the curricula were well organized and of high quality. However, 30% of teachers surveyed agreed that the reading curriculum was too difficult, and 48% agreed that it was too standardized (26% and 42% for the math curriculum).

Only half of teachers (55%) were satisfied with the pacing of the lesson plans, primarily because they found the pace too fast.
Staffing/Training Surveys conducted in 1999 revealed that eighth grade SB teachers were more likely than third or sixth grade teachers to have taught the same grade during the previous school year (54% vs. 36% and 54% vs. 31% for third and sixth grade teachers, respectively). Over twice as many eighth grade teachers than third grade teachers (41% vs. 18%) reported knowing almost all (80–100%) of their SB students before the program.

The SB student to teacher ratio of 16:1 represented a drop from class sizes that SB students experienced during the regular school year, which in some cases were as high as 30:1. In interviews, teachers emphasized that SB’s small class sizes were critical in allowing them to pay more attention to individual students.

About half of SB teachers reported having tutors available to assist with instruction.

Teachers rated as tailoring instruction were found to differ from other teachers in the quality and variety of interactions with students. These teachers provided more feedback to students, both as a group and individually, and made a greater effort to present the material so that it was accessible to both their strongest and their weakest students.


Summative/Outcome Findings

Academic Approximately half of sixth and eighth graders who attended SB met the promotional cutoffs in both subjects by the end of the summer, compared with 40% of third graders. These rates were consistent across years, except among eighth graders, where the percent meeting the cutoffs dropped from about 55% to approximately 38%.

The average adjusted short-term gains in reading scores for SB participants from 1997–2000 were 2 months for third graders, 3.9 months for sixth graders, and 5.9 months for eighth graders. Gains for eighth graders dropped in 2000 compared to previous years.

The average adjusted short-term gains in math scores for SB participants from 1997–2000 were 2.9 months for third graders, 3.6 months for sixth graders, and 4.8 months for eighth graders. Gains for all grades dropped in the summer of 2000 compared to previous years.

SB students’ reading test scores typically increased almost twice as fast per week in the summer as in the school year for all three grade levels. For eighth graders, reading test score gains per week were over three times higher for the summer than for the school year in 1997–1999.

SB students’ math test scores increased over twice as fast during the summer as in the school year for third graders in 1997, increased at similar or slower rates than in the school year for the summers of 1998 and 2000, and increased about 1.4 times faster than during the school year for the summer of 1999. For sixth graders, test score increases per week were over 2 times faster in the summers of 1997 and 1998, 1.7 times faster in 1999, and 1.2 times faster in 2000. For eighth graders, test score increases per week were approximately 2.5 times faster than during the school year in the summers of 1997 and 1999, about 3 times faster in 1998, and about 1.75 times faster in the summer of 2000.

Across program years, approximately 80% of low-risk, 40% of moderate-risk, and 11–12% of high-risk third graders met the promotional cutoffs after attending SB. For sixth graders, a little over 80% of low-risk, 57% of moderate-risk, and 27–28% of high-risk students met the promotional cutoffs after attending SB. For eight graders, approximately 75% of low-risk, 55% of moderate-risk, and 29% of high-risk SB students met the promotional cutoffs after attending SB.

Higher risk third grade students experienced the largest gains in reading relative to low- and moderate-risk students. Low-risk third graders, on the other hand, experienced the largest gains in math relative to the other groups.

Lower risk sixth grade students experienced the largest gains in both reading and math relative to moderate- and high-risk students.

Lower risk eighth grade students experienced the largest gains in reading relative to moderate- and high-risk students. Moderate-risk eighth graders, on the other hand, experienced the largest gains in math relative to the other groups.

Learning gains did not differ substantially by students’ race/ethnicity, although sixth and eighth graders in predominantly African American schools experienced larger gains in reading than students in predominantly Latino schools (and to a lesser extent students in other schools). Eighth grade students in predominantly African American schools also experienced larger gains in math than students in predominantly Latino or other schools.

In the third and sixth grades, boys had significantly lower adjusted test score gains in math than girls. In third grade, boys also experienced smaller test score gains in reading (p < .01 for each) than girls.

Math score gains for all grades and reading score gains for third graders were significantly higher (p < .05) for students in high-achieving schools (those with over 40% of students at or above national norms in those subjects) than in moderate- or low-achieving schools.

Adjusted learning gains were significantly greater in schools where teachers reported knowing a larger proportion of their students before SB (p < .10 in third grade, p < .05 in sixth grade, and p < .01 in eighth grade).

Combining site observations with analysis of test score gains, the evaluation found that students whose teachers provided tailored instruction had substantially larger reading score gains than the average SB student. Students whose teachers provided sufficient instruction had slightly higher test score gains than the average SB student.

When evaluation results were compared with results from a meta-analysis of other summer school programs, the adjusted short-term reading gains experienced by third grade SB students were similar to estimates from other summer programs for 1997–1999 and were larger for third grade SB students in 2000. For sixth graders, short-term SB reading gains were larger for 1997–1999 and were even larger than other programs for the summer of 2000. For eighth graders, short-term SB reading gains were substantially larger than other summer programs for 1997–1999 and also for the summer of 2000.

The adjusted short-term math gains experienced by third grade SB students were greater than the estimate for other summer programs for 1997–1999 but were smaller than for other programs for the summer of 2000. For sixth graders, short-term SB reading gains were larger for 1997–1999 and were approximately equal to other programs for the summer of 2000. For eighth graders, short-term SB reading gains were larger than other summer programs for 1997–1999 and for the summer of 2000.

Third graders who attended SB had slightly larger ITBS reading test score gains in the years following SB than did students with similar (but above the cutoff) test scores who did not participate. These larger gains mainly seemed to close the gap between the two groups following SB rather than allowing SB students to surpass those students who were initially just above them in their ITBS scores. Similarly, sixth grade SB students had larger test score gains between fifth and eighth grade than students in the comparison group, although they too continued to lag slightly behind their peers with similar scores who did not have the benefit of extra instruction in SB. The third grade gains are significant at p < .01 and the sixth grade gains are at p < .05.

Comparing SB high scorers and their comparison group to analogous groups prior to the policy’s implementation, both third and sixth graders’ reading score gains diverged slightly from the 2 years prior to the summer between third and fourth grades to the 2 years after. After the policy’s implementation, the SB eligible group’s learning gains contributed to a narrowing of the initial gap between SB high scorers and the comparison group, although the magnitude of these differences were very small. For the prepolicy groups, the third grade differences were not significant and the sixth grade difference was significant (p < .01). The gap between the high scorers and the comparison group was also still significant.

SB third and sixth graders had slightly larger overall reading learning gains across 4 years (the 2 prior to SB and the 2 following SB) than did comparison group students, although analysis of students’ rates of change revealed that SB did not appear to accelerate students’ learning, but rather provided a one time boost that increased their overall 4 year average gains, helping them get a little closer to students right above the promotional cutoffs.

About three quarters of students surveyed reported learning more in SB than during the school year.

Over 80% of student survey respondents reported that they worked harder in summer school than the regular school year.

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project