Search Publications | The Evaluation Exchange | FINE Network | OST Database

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

A Profile of the Evaluation of Supplemental Educational Services

Program Description

Overview	The U.S. Department of Education’s (DOE) No Child Left Behind (NCLB) Act of 2001 requires low-performing schools to offer Supplemental Educational Services (SES) to students from low-income families. These services are defined to include academic instruction (e.g., tutoring, remediation) that is provided outside of the regular school day. The program’s goal is to improve youth’s academic achievement. SES providers may include nonprofit, for-profit, and faith-based organizations, charter schools, private schools, and public and private colleges and universities. Local school districts can provide services, but only if they have not been identified as low performing.
Start Date	Fall 2002
Scope	national
Type	after school, summer/vacation, before school, weekend
Location	urban, suburban, rural

Setting	public school, private school, community-based organization, religious institution, private facility, recreation center
Participants	elementary through high school students
Number of Sites/Grantees	1,890 providers nationwide (spring 2004)
Number Served	430,000 youth in 2004–2005
Components	All youth from low-income families (e.g., those who qualify for the free or reduced price meal [FRPM] program) who attend Title I schools that are not meeting their state’s adequate yearly progress goals are eligible to receive SES or be transferred to another school. Each state is required to develop provider selection criteria, approve or deny provider applications, and provide school districts with a list of available providers in their area. Districts must inform parents of SES availability and provide them with information to select providers for their children, collect parent applications or signatures, coordinate provider activities with schools, estimate per-pupil expenditures, contract with providers to provide SES, manage provider activity, and develop specific educational goals for each student in consultation with parents and providers. Parents may select any approved provider in the area served by the district or within a reasonable distance of the district. Providers must measure and report on youth’s progress regularly to teachers and parents. Schools are not required to play a role in SES implementation. Services are required to be consistent with state academic content standards and district instruction.
Funding Level	NCLB requires districts to set aside 20% of their Title I, Part A allocation for school choice-related transportation and SES. Districts must spend at least 5% of this Title I, Part A allocation on SES unless demand is less than estimated.
Funding Sources	U.S. Department of Education

Evaluation

Overview	The DOE commissioned a multiyear study aimed at gaining insights from the early efforts of particular states and districts that could assist others in improving SES implementation. In addition, the Civil Rights Project conducted a study to examine the ability of districts to implement the requirement that schools offer SES to students attending poorly performing schools.
Evaluators	Policy Studies Associates, Inc. and the Civil Rights Project at Harvard University (now at the University of California–Los Angeles)
Evaluations Profiled	Early Implementation of Supplemental Educational Services Under the No Child Left Behind Act: Year One Report Case Studies of Supplemental Services under the No Child Left Behind Act: Findings From 2003–04 Increasing Bureaucracy or Increasing Opportunities? School District Experience With Supplemental Educational Services
Evaluations Planned	Many states have commissioned evaluations.
Report Availability	U.S. Department of Education, Office of the Under Secretary, Policy and Program Studies Service. (2004). Early implementation of supplemental educational services under the No Child Left Behind Act: Year one report. Washington, DC: Author. Available at: www.policystudies.com/studies/school/nclb.html U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service. (2005). Case studies of supplemental services under the No Child Left Behind Act: Findings from 2003–04. Washington, DC: Author. Available at: www.ed.gov/rschstat/eval/disadv/supplementalyear2/index.html Sunderman, G. L., & Kim, J. (2004). Increasing bureaucracy or increasing opportunities? School district experience with supplemental educational services. Cambridge, MA: The Civil Rights Project at Harvard University. Available at: www.civilrightsproject.ucla.edu/research/esea/increasing_bureaucracy.pdf

Contacts

Evaluation	Katrina G. Laguarda Senior Research Associate Policy Studies Associates, Inc. Suite 400 1718 Connecticut Avenue, NW Washington, DC 20009 Tel: 202-939-5321 Fax: 202-939-5732 Email: laguarda@policystudies.com Gail L. Sunderman Senior Research Associate The Civil Rights Project 8370 Math Sciences Los Angeles, CA 90034 Tel: 310-267-5562 Fax: 310-206-6293 Email: glsunderman@yahoo.com
Program	Jackie Jackson Director Office of Elementary and Secondary Education 400 Maryland Ave., SW Washington, DC 20202 Tel: 202-260-0826 Fax: 202-205-0310
Profile Updated	May 31, 2007

Evaluation 2: Case Studies of Supplemental Services Under the No Child Left Behind Act: Findings From 2003–04

Evaluation Description

Evaluation Purpose	To answer the following questions about SES in its 2nd year: How are states and districts implementing SES? How do states select providers? How do districts reach out to and involving parents regarding SES? What SES are provided? What types of providers offer and provide SES? What are the implementation challenges and successes?
Evaluation Design	Non-Experimental: A sample of 6 states and 9 school districts were selected that seemed to be relatively far along in implementing SES and that varied in location, size, and student population, including the 5 districts in the year 1 study (see Evaluation 1) that were still required to offer SES in year 2. Interviews were conducted with state administrators responsible for administering SES, district staff involved in planning or implementing SES, and up to 3 principals and SES providers in each district. In up to 3 schools in each district, teachers participated in focus groups, and parents of SES-eligible children participated in interviews or focus groups. The following criteria were used in selecting the sample: schools with the highest number of youth receiving SES in a given district; middle and high schools; providers serving the largest number of youth, and districts that were themselves state-approved providers.
Data Collection Methods	Interviews/Focus Groups: Interviews and focus groups asked about SES implementation. Since 2 states were new to the sample in year 2, interviews included specific questions about year 1.
Data Collection Timeframe	Data were collected between January and April 2004, during the second school year in which SES were offered.

Findings:
Formative/Process Findings

Activity Implementation	Providers typically offered 1–2 hour sessions immediately after school, 2–3 times a week. SES content and structure varied widely by provider and sometimes by tutor. Providers used a range of approaches to teach reading and math: many used diagnostic test results to define SES objectives; some included guided literature reading, discussions, and extension activities to improve oral fluency, vocabulary, and comprehension; others focused on phonics and decoding skills; still other used self-paced worksheets to strengthen basic skills. Some did not use a prescribed curriculum, instead focusing on class work or homework, organizational strategies, note taking, critical thinking skills, and test prep. In several districts, parents returned interest forms in September or early October, but SES did not begin until November or December (or even March, in one district). Several parents felt that it was easy to lose track of what they signed up for. By contrast, parents in another district were pleased that SES programs were up and running much earlier in year 2 than in year 1.
Costs/Revenues	The percent of Title I (or other) funds that districts set aside for SES ranged from 2% to 21%. In the 6 districts that set aside less than 20%, 2 districts explained that they set aside enough for all eligible students, while 4 districts explained that their set aside was based on year 1 demand. Average SES per-pupil expenditure for the 9 districts was $1,408 in year 2. For the 5 districts in both years of the study, the average increased from $967 in year 1 to $1,280 in year 2. Several districts expressed concern over the administrative costs of implementing SES. Although districts could use Title I funds for implementation, these costs were substantial. District administrators explained that there were costs associated with mailings to schools and parents, working with providers, and legal costs associated with writing provider contracts. Provider costs and total contact hours varied for many reasons. Staff costs depended on youth–tutor ratios, staff qualifications, and local salary scales. Some districts let providers use school facilities for free, while others charged rent, and off-site providers also paid rent. Providers who offered transportation incurred additional costs. District policies for reimbursing providers based on attendance also affected the income generated by each slot, and thus the number of tutoring hours. Further, since districts paid providers a maximum amount equal to their per-pupil Title I allocation, low-cost providers could offer more hours than high-cost ones in the same district.
Parent/Community Involvement	While some providers said that they sent reports to parents every 4–6 weeks and others reported informal communication with parents, most parents reported receiving nothing from providers about youth’s work or progress. Several parents reported that they relied on teachers to keep them informed. Others said they had no idea what their child did after school. Five states required that providers communicate with parents in their native language.
Program Context/Infrastructure	Most district administrators reported beginning SES administration by contacting all state-approved providers to determine their availability in the district, and asking them to submit information about their services offered. Districts were adept at entering into provider contracts; many used boilerplate contracts that they created and vetted in year 1. However, the time needed to finalize contracts was substantial. For example, in 2 districts, school board involvement in the vetting process caused delays. Most districts’ provider contracts contained similar content, including specifications for the following: number of tutoring hours per youth; session duration and frequency; amount paid per youth; instruction type; provision of SES on- or off-site; academic test results; program attendance, progress reports, and evaluations; agreements on youth goals and progress; service termination; liability insurance; district responsibilities; and payment/compensation terms. Most contracts were accompanied by examples of forms that providers were to complete periodically. Of the 24 providers in the sample, SES was provided in: schools (N = 18); off-site in their own facilities, churches, or a Boys & Girls Club (N = 4); and in homes (N = 2). Within districts, some providers served several different school sites. Other districts assigned one provider per school. Just under half of providers offered transportation, including all providers in 2 districts. Some providers organized their programs so that they could be attended in combination with other activities. For example, after school programs of various kinds might operate for 3–4 hours after school, with SES tutoring taking place for 1–2 of those hours. Schools faced challenges in communicating to parents the distinction between SES and other after school programs; many school staff said they did not understand the differences themselves.
Program–School Linkages	Districts often relied on schools to recruit youth. Many districts gave schools sample letters to send to families of eligible youth. In some cases, school staff called parents to encourage them to enroll. One district administrator reported that teachers were instrumental in explaining services to parents. Several teachers in this district said that many parents were persuaded to sign up because their child’s teacher was providing the SES. At least 2 districts with relatively high participation rates used school-based parent coordinators or district parent liaisons to contact parents. Schools in several districts held parent meetings on SES availability and services, and some schools hosted provider fairs for parents to hear from providers directly. Districts increasingly relied on school staff to help coordinate SES. Some principals noted that they were required to help with SES logistics, including ensuring that providers had space and that parents turned in permission forms. Further, some districts allowed schools to define eligibility beyond low-income status. Principals relied on teachers to determine which youth most needed SES. School staff in some districts said they wanted to play a bigger role in defining eligibility. Providers worked within schools whenever possible; youth typically attended programs in their own school. Most providers hired teachers from those same schools or from the district at large. Some providers reported formal mechanisms to communicate with teachers. One provider sent teachers checklists to identify youth’s strengths and weaknesses. In one school, providers attended school faculty meetings. School staff who worked as SES tutors informally kept teachers informed of youth’s progress. Many teachers, however, did not know which of their students received SES or from whom, and few knew much about what their students did in SES; none received written progress reports from providers. Teachers saw this lack of communication as a missed opportunity and expressed a desire to work with tutors to address particular youth’s needs; some teachers also felt that they could help ensure that youth attended SES sessions. Although providers maintained that their curriculum aligned with state standards, some were unable to give any evidence that they did so. Others reported that they conducted alignment studies and purchased instructional materials to ensure that state standards were covered. Still others explained that they used results of state assessments to develop individual tutoring plans. Providers in at least 3 sites used diagnostic tests based on state assessments to determine tutoring content. District administrators reported that they assumed that states assessed the alignment of provider curricula with state standards as part of their provider selection criteria. Most providers did not attempt to coordinate their instruction with district approaches, and district administrators did little to help with this. In several districts, teachers and providers agreed that explicit coordination was less important than whether SES provided academic skills. In at least 2 sites, the district’s approach to teaching reading differed considerably from that of providers. Teachers in several districts noted that some providers’ reliance on worksheets and traditional drills ran counter to their beliefs and practices. Administrators in at least 3 districts reported relying on principals to monitor provider performance.
Recruitment/Participation	Most districts mailed parents letters about SES availability. These letters were relatively straightforward and referenced the availability of math and reading “tutorial services” or “extra help.” Two districts lapsed into legislative jargon; parents in one of these districts complained that the language was inaccessible to them. Letters were usually accompanied by materials that included descriptions of provider services, a form for parents to rank their provider choices, and provider brochures. Some districts included answers to frequently asked questions, application material checklists, and due dates. In addition to sending materials, districts made other efforts to reach parents and reported being eager to do so. One district held provider fairs at schools, sent fliers and reminders home, and had school staff conduct home visits to encourage parents to enroll. Other strategies included newspaper announcements, TV and radio ads for provider fairs, and labeling eligibility “scholarships” to give SES a more positive spin. Although districts generally reported successfully informing parents about SES, some school staff and providers disagreed, claiming that materials were confusing, complicated, too heavily reliant on written information, or lacked adequate information on providers. Districts also cited problems communicating to parents about SES eligibility; as one district administrator explained, it was difficult for parents to understand why their child did not qualify for SES. Other parents wondered why they received a letter when their child was doing well in school. Three of the districts in the sample in year 1 and 2 changed their targeting strategies as a result of low year 1 participation. Districts’ year 2 experiences suggested that targeting SES was unnecessary, since so few parents enrolled their children in SES. Several districts set attendance policies, which typically involved dropping youth from the program for specific numbers of absences and/or not reimbursing providers if youth failed to attend. Some providers complained that youth absences were out of their control. Most districts required providers to include attendance and progress reports for each participant with their monthly invoices. District administrators said that monitoring attendance was difficult. In some districts, providers had trouble managing attendance policies in a way that ensured that they got paid for all youth and sessions. For example, one district wanted copies of participant sign-in sheets and parent signatures to confirm their child attended. Providers found that it was difficult to get parent signatures; youth often failed to give the form to their parents to sign. In year 2, every district knew how to estimate the number of youth they could serve with SES, compared to year 1, when few had a strong grasp of their capacity to serve eligible youth. Eight districts provided SES to 7–28% of all eligible youth, and the 9th served 86%. In 6 districts, more youth were eligible than could be served with the maximum required funding; the other 3 districts could have served all eligible youth. Only one district had participation rates that reached the maximum they could support; this district went beyond its funding capacity to serve another 126 youth. Two districts provided SES to 86% or more of youth they could serve, while the other 6 districts had participation rates between 13% and 62% of youth that they could serve. Among districts with under 50% participation, districts reported that some youth did not participate because they had sufficient reading and math achievement, family responsibilities that precluded them from staying after school, or parents who could not pick them up after school. Parents did not enroll their children in SES for a number of reasons. Many found other after school options more accessible, convenient, or appropriate, or thought that their children got enough help from other after school programs or informal teacher tutoring; several had safety concerns about their children being at school after hours or traveling home late in the day; some simply did not know enough about SES or how to sign up; and some wanted to enroll but were told that their children were not eligible. Some district and school staff noted that parents were sometimes overwhelmed by too many provider options. The 5 districts in the sample for both years offered local, state, and federally funded after school programs that operated in tandem with SES. By year 2, most parents knew the names of the most active SES providers and most understood that their services were distinct from schools’ other after school programs. However, parents still tended to regard SES as one of many equally attractive after school activities. Some parents reported that they signed their child up for SES in addition to other programs. Parents spoke highly of arrangements where youth could participate in both SES and in recreational after school activities on the same day. Providers in several districts reported that as SES became established, attendance was higher and more predictable. However, most respondents observed that attendance was a problem. Teachers observed that attendance was often better when services were provided in the school. Attendance was especially a challenge at the middle and high school levels. In several middle schools, school staff estimated that fewer than half of SES participants attended on any given day. In elementary schools, attendance rates were higher, sometimes approaching 95%. To combat attendance problems, several providers offered youth incentives to attend regularly. For some parents, who studied provider brochures, choosing a provider was fairly straightforward. Other parents reported attending open houses where providers made presentations or speaking with providers at their child’s school. In several schools, providers had a regular and visible presence, attending school events, presenting in classrooms, and talking with parents. These face-to-face contacts were crucial sources of information for some parents. Most parents were aware of the school choice option, but few seriously considered it; some wanted their child to attend school in the community, while others were reluctant to send their child to a school with teachers they didn’t know. Parents typically heeded teacher and principal advice when deciding to sign up for SES and choosing providers. In many schools, teachers who worked for providers were a key point of contact for SES and helped recruit parents. In at least 2 districts, schools with staff who were actively involved in recruiting parents had much higher participation rates than schools with less active staff. Many parents reported that they chose a particular provider because a teacher they knew and trusted worked for that provider or had encouraged them to do so. In other schools, the principal or the Title I parent coordinator acted as a gatekeeper, directing parents to providers based on their knowledge of providers and of student needs. Parents tended to choose providers in their child’s school; few sent their child off-site when given an on-site choice. In a couple of districts, providers that offered SES in youth’s homes were also an attractive option for some parents. While many parents reported that transportation availability was critical in selecting a provider, in 2 districts, providers that offered transportation tended to have lower enrollments than those that did not. Other parents tended to choose programs that ran until 6 p.m., so that they could get pick up their child after work. In explaining why they signed their child up for SES, parents said that they sought to keep their child safe and occupied after school and that any “extra learning” would benefit their child. They hoped tutoring would give their child the structure and extra attention lacking during the school day and at home. Homework help was also seen as an important feature. However, some parents questioned whether a few extra hours of tutoring each week could make a difference and a few objected to providers’ instructional approaches. All 6 states required applicants provide evidence of their capacity to serve special populations of youth (e.g., disabled and ELL). School and district staff reported that providers offered SES that were designed to benefit all students, including high school, English language learner (ELL), and special education. High school students had more limited choices than other youth, since fewer providers served them. No district or school staff reported problems with ELL or special education students’ SES access. Spanish-speaking parents in 2 focus groups expressed satisfaction with SES. The total number of tutoring hours each youth received ranged widely, from 18 to 120. At a rate of 2–6 hours per week, youth tended to “cycle out” of SES after 10–20 weeks. As a result, youth often received SES for only one marking period or semester, so providers who began SES early tended to serve two cohorts of youth in one academic year. Providers required a minimum number of youth to enroll to cover their costs, which ranged from 10 youth per site to 80. Competition for youth was intense in some schools; in one district, providers raffled off prizes to youth who signed up. In almost every district, at least one provider was unable to offer services due to low interest.
Staffing/Training	Tutoring was most often provided in small groups, with tutor–youth ratios ranging from 1:3 to 1:15. More than half of providers put tutors with groups of 5 to 10 youth. All 6 states required providers to produce evidence that their staff were qualified and had undergone background checks. Of the 24 providers, 15 reported that a teaching certificate was required. A few providers had less stringent requirements. For example, one provider hired college students who were finishing teacher certification programs. About half of providers sought to hire tutors and site coordinators directly from schools where they provided SES. Providers, teachers, and parents pointed out that teachers could be on-site immediately after school, knew youth and their parents, and could strengthen SES–school links. Others had mixed feelings; in the words of one provider, using these teachers was complicated by the fact that “teachers can’t separate their biases and attitudes from the school day.” Many districts reported a need for additional staff for tasks such as monitoring provider services and attendance and the fact that SES sometimes took staff from their regular duties. Providers reported that tutors got 4–20 hours of training at the beginning of the year. Many tutors reported that some tutors skipped this training, and that they typically did not receive follow-up support during the year. One problem was that most tutors worked just a few hours a week, so some were unwilling to invest large amounts of unpaid (or even paid) time in training. Two established national providers reported that they checked on tutors regularly and conducted formal performance evaluations. Other providers had no formal systems to supervise tutors. Providers found it challenging to hire and retain appropriate numbers of tutors to meet fluctuating demand. This sometimes led to providers’ inability to serve youth requesting SES, letting go of staff when SES demand was low, or suspending SES if youth stopped showing up. At least 3 districts found ways to fund an SES coordinator in their district to help improve SES management and communication between providers and the district. In addition, to help distinguish among and coordinate between the various after school activities, schools in more than one district hired after school program coordinators.
Systemic Infrastructure	In year 2, states asked applicants for evidence that they met provider criteria, though some asked providers only for assurances. States also added criteria to legislative requirements. For example, one state required providers serving areas with sizable ELL populations to show experience in improving this group’s academic achievement. In year 2, all 6 states issued approved provider lists before the start of the school year. However, at least 3 districts reported receiving lists several months after states reported publishing them and 3 states continued to review and approve applications during the year. By year 2, all 6 states refined their provider application review process, developing evaluation criteria and rubrics to score applications and organizing committees of reviewers. Despite states’ reports of limited attempts to encourage potential providers to apply, the number of organizations applying and getting approved grew from year 1 to 2. However, state service administrators reported concerns over the limited supply of providers in small and rural districts. In one state, many of the rural areas were only served by online providers, which was problematic for schools that lacked computers and high-speed Internet connections. Private providers other than faith-based and online were the dominant provider types in 4 of the study states and nationally. Intermediate education agencies, school districts, and school-based providers made up 25% of providers nationwide, while in 4 of the study states, these types of providers were even more common, representing 31–46% of providers (in the remaining 2 states, they were somewhat less common). The proportion of online and faith-based providers was low in all study states and nationwide. Most states planned to monitor and evaluate provider performance, which included visiting providers and evaluating district data. Some states worked with universities or external evaluators, while others developed surveys and evaluations themselves. States generally did not attempt to monitor the number of contact hours per child or provider costs. While states knew that providers who failed to contribute to increased academic proficiency for 2 consecutive years had to be removed from state lists, states had yet to develop criteria to evaluate this requirement, and as of year 2, few providers had been removed (and none due to poor quality). One state, an exception, developed a target for provider performance in reading. Several respondents indicated that they welcomed federal assistance in this monitoring; they asked for sample data collection instruments/templates and greater interaction with the U.S. DOE on these issues. Although most districts required agreements for providers to follow about each SES participant’s goals and progress (developed by the district in consultation with parents and providers), few districts were able to enforce this requirement and most could not say whether these agreements were fashioned in the manner mandated under NCLB.

Summative/Outcome Findings

Academic

Across districts, parents gave numerous examples of ways their children benefited from SES, including improved grades, math or reading skills, and attitudes toward school. In one district, several parents appreciated that their high-school-age children earned school credit for participation that would allow them to graduate on time. Other parents reported that they observed little benefit from SES and saw no improvement in their children’s reading and math skills.

A Profile of the Evaluation of Supplemental Educational Services

Profile Information