You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Program Description

Overview Sacramento START (Students Today Achieving Results for Tomorrow) is an after school program that provides children from low-income neighborhoods in Sacramento, California, with homework assistance and enrichment activities. The program is a partnership between the city of Sacramento, the county of Sacramento, and six local school districts. START is built around the idea that high quality after school programs promote youth development and build on student assets by helping children form new friendships, develop new interests, and build their self-esteem. This positive youth development, in turn, promotes social and emotional growth and greater attachment to school. START aims to be particularly beneficial to students who are struggling academically.
Start Date 1995
Scope local
Type after school
Location urban
Setting public schools
Participants kindergarten and elementary school students (K–6)
Number of Sites/Grantees 18 elementary schools in 1995–1996; 29 elementary schools in 1998–1999; 40 elementary schools in 1999–2000; 32 elementary schools in 2000–2001; and 37 elementary schools in 2001–2002, and 2002–2003. START involves six school districts in the Sacramento region.
Number Served 4,209 in 1999–2000; 3,820 in 2000–2001; 5,136 in 2001–2002; and 6,939 in 2002–2003
Components Each START program includes three components: homework and tutoring assistance, literacy, and enrichment. The following activities are incorporated in all START programs at least twice a week: (1) reading—group and individual reading opportunities that emphasize accuracy and fluency; (2) literacy and math—activities presented at each grade level to support and reinforce the chosen curriculum at the school site; and (3) enrichment—activities such as art, cooking, crafts, science, health and physical education, cooperative games, sports, drama and dance, music and movement, cultural and holiday-themed activities, field trips, nutrition, gardening, and environmental education.

Each START site is also provided with an array of educational resources, such as games, language review resources, literacy center resources, and activity resources.

Program Leaders serve classes of approximately 20 students. Site Directors supervise Program Leaders and manage programs. Regional Directors supervise five to twelve Site Directors and coordinate activities with school districts and citywide functions (e.g., training, special events, and enrichment activities). A classroom teacher at each school serves as an Academic Alignment Coach to link the START program with the school curriculum and school staff. The Academic Alignment Coach is paid an annual stipend. Recreation Managers and Program Managers supervise the program for the city of Sacramento.
Funding Level $4.8 million (2001–2002)
Funding Sources the city of Sacramento, six local school districts, and numerous corporations, foundations, and individuals
Other The city of Sacramento Department of Neighborhood Services serves as the fiscal agent for the program, employs and trains staff, and prepares reports for funders. START also requires five core training classes for all program staff offering direct services, which cover an array of topics related to providing after school services.


Evaluation

Overview START is evaluated annually to look at program implementation and its impacts on participating youth.
Evaluator Minicucci Associates
Evaluations Profiled Achieving Results: Evaluation Report 1999/2000

Supporting Student Achievement: Evaluation Report 2000/2001

Evaluation Report 2001/2002
Evaluations Planned The 2002–2003 evaluation is in process. Annual evaluations are required by the California Department of Education, which provides major funding for the program.
Report Availability Lamare, J. (1998). Sacramento START: An evaluation report, September 1996 – May 1997. Sacramento, CA: Sacramento Neighborhood Planning and Development Services Department. Summary available at www.aypf.org/rmaa/pdfs/sacSTART.pdf (Acrobat file).

Minicucci Associates. (2001). Achieving results: Evaluation report 1999/2000. Sacramento, CA: Author. Available at www.sacstart.org/pdf/1999-2000_eval.pdf (Acrobat file).

Minicucci Associates. (2001). Supporting student achievement: Evaluation report 2000/2001. Sacramento, CA: Author. Available at www.sacstart.org/pdf/2000-2001_eval.pdf (Acrobat file).

Minicucci Associates. (2002). Evaluation report 2001/2002. Sacramento, CA: Author. Available at www.sacstart.org/pdf/2001-2002_eval.pdf (Acrobat file).


Contacts

Evaluation Cathy Adams Minicucci
Minicucci Associates
1540 River Park Dr., Ste. 212
Sacramento, CA 95815
Tel: 916-920-7800 x224
Fax: 916-649-3161
Email: minicucci@pacbell.net
Program Sacramento START
8795 Folsom Boulevard, Suite 101
Sacramento, CA 95826
Tel: 916-808-6089
Fax: 916-808-1214
Profile Updated May 11, 2004

Evaluation 1: Achieving Results: Evaluation Report 1999/2000


Evaluation Description

Evaluation Purpose To examine START participants' outcomes in the areas of academic achievement and school attendance.
Evaluation Design Quasi-Experimental: Outcome data were collected on START participants in grades three through six in four school districts for the current program year (1999–2000) and the previous year (1998–1999), although the evaluators were unable to determine whether or how many 1999–2000 START participants were also participating in START during 1998–1999. Evaluators attempted to identify 50 students at each grade level who attended START at least 30 days in February and March of 2000 at each school. The rationale for this selection criteria was that standardized testing usually occurs in April or May, so for START to have helped raise test results, the children would have to have been present in the program a sufficient number of days leading up to the SAT-9 testing dates.

Because START participation was not high enough to obtain 50 students per grade per school, the study team identified as many students as possible that met the criteria for each grade level. In addition, some students lacked a complete two years' worth of data due to not having been enrolled in the school for both years, having been absent during test times, or having been exempted from testing as an English Language Learner (ELL). No more than 40 students met the criteria at any grade level. In total 1,369 students were identified for inclusion in the evaluation, with complete two-year data available for 871 of these. This sample was more female (58%) than male (42%), and was racially and ethnically diverse (31% African American, 27% Asian, 23% Hispanic, 18% white, and 2% other). Approximately half spoke English as their primary language, while 22% spoke Hmong, 16% spoke Spanish, 5% spoke Cantonese, 2% spoke Mien, and 4% spoke some other language.
Data Collection Methods Secondary Source/Data Review: Program participants' school attendance data were obtained from school district databases. START registration information was gathered to examine students' date of birth, bilingual language information, and gender.

Test/Assessments: Participants' SAT-9 reading and math test scores were obtained from school districts. SAT-9 data were reported in percentile rank format.
Data Collection Timeframe Data were collected during the 1999–2000 school year, including academic data for the current and previous school year.


Findings:
Summative/Outcome Findings

Academic From 1998–1999 to 1999–2000, SAT-9 achievement scores showed overall growth in percentile rank scores for fifth and sixth graders in reading, with a decline for third graders and virtually no change for fourth graders (an average of one-percentile rank point decline across grades).

From 1998–1999 to 1999–2000, SAT-9 achievement scores showed overall growth in percentile rank scores for all grade levels in math (an average of seven percentile rank points growth across grades).

From 1998–1999 to 1999–2000, the proportion of students scoring below the twenty-fifth percentile dropped from 45% to 44% in reading and from 43% to 33% in math. Of students scoring below the twenty-fifth percentile in reading in 1998–1999, 23% scored above the twenty-fifth percentile in 1999–2000. The corresponding figure for math was 36%.

Low-performing ELL students showed more improvement in reading than English-speaking low performers (28% and 20% of low performers moving above the twenty-fifth percentile, respectively). This was also true for math scores (40% vs. 31%, respectively).

Among START students with complete school attendance data across both school years, the average number of days absent dropped from eight days in 1998–1999 to five days in 1999–2000.

For START students with school attendance problems (missing 10 or more days in 1998–1999), the average number of days absent dropped from 17.6 days in 1998–1999 to 8.9 days in 1999–2000.

Evaluation 2: Supporting Student Achievement: Evaluation Report for 2000/2001


Evaluation Description

Evaluation Purpose To examine (1) outcomes for START students in academic achievement and school attendance and (2) program implementation strengths, challenges, and emerging solutions.
Evaluation Design Quasi-Experimental and Non-Experimental: Outcome data were collected for the 1999–2000 and 2000–2001 school years on START participants. A total of 1,200 students from six districts were selected from grades three through six who participated in START for at least 30 days in February and March of 2001. Of these, 748 had complete standardized testing information for both years, while 705 students had complete attendance information for both years. A subsample of 227 START students in the 2000–2001 evaluation sample, who had also been included in the previous year's evaluation, allowed for a separate longitudinal analysis of students participating in START for two consecutive years.

The evaluation group sample was slightly more female than male (54% vs. 44%, 2% unknown). Thirty percent were third graders, 33% were in fourth grade, 24% were in fifth grade, and 13% were in sixth grade. The sample was ethnically and racially diverse, with 29% Hispanic students, 27% African-American students, 23% Asian students, 15% white students, and 1% other. Approximately one-third of the sample were English Language Learners (ELL). ELL students' languages included Spanish (45%), Hmong (31%), Hindi (7%), Mien (5%), and Chinese (5%).
Data Collection Methods Interviews/Focus Groups: End-of-the-year focus groups were held with 25 Site Directors and five Regional Directors in May 2001. Focus groups concentrated on perceptions of the benefits of START, perceptions of START students' needs, and START's milestones in the 2000–2001 school year.

Secondary Source/Data Review: School attendance data were obtained from school district databases. Program registration information was also gathered, including participants' date of birth, bilingual language information, and gender.

Surveys/Questionnaires: Pretest and posttest questionnaires were collected from 76 START students (about 10% of the achievement test sample for 2000–2001) that asked questions about how often in the past 30 days the students wanted to go to school, studied for a test, felt unsafe at school, and talked with a parent/guardian about school or homework. Pretest/posttest data were gathered for as many START students as possible.

Tests/Assessments: SAT-9 reading and math test scores were obtained from participating school districts. SAT-9 data were reported in Normal Curve Equivalent scores (NCE).
Data Collection Timeframe Data were collected during the 2000–2001 school year, with data collected from the current and previous school year. For the whole sample, academic data were available in school years 1999–2000 and 2000–2001. For the subsample who had participated in START for two years, academic data were also available for school year 1998–1999.


Findings:
Formative/Process Findings

Activity Implementation Staff reported a number of milestones during the 2000–2001 year, including involvement of high school students as tutors and mentors in a science program, participation in 4-H Cooperative Extension enrichment activities, a University of California-Davis Steps to College program at eight START sites, an art and garden project at four START schools, and a special nutrition program at 11 START sites.

Site Directors reported that students often come into the program tired and in need of some fun, a snack, and a break, which made it difficult to launch immediately into academic program content.
Program Context/Infrastructure A number of Site Directors reported that many START children come into the program needing various support services, such as basic needs (e.g., food and clothing), social services, or health services.
Program-School Linkages Program staff expressed frustration with the pressure they felt to stress reading and math with students who wanted more music and art, which was often lacking in the regular school day.
Staffing/Training Many Site Directors and Regional Directors noted that turnover in Program Leaders was one of Start's largest challenges, and could be time consuming in terms of needing to continually train new staff.

According to Site Directors and staff, successful strategies for training staff included shadowing experienced START staff, observing skilled classroom teachers during the school day, and having classroom teachers observe new hires and offer suggestions.

Site Directors felt that lower staff/child ratios and higher hourly pay would help to retain quality staff.


Summative/Outcome Findings

Academic Overall, SAT-9 scores showed a 1 NCE point decline in reading and a 3 NCE point gain in math between 1999–2000 and 2000–2001. Both changes were statistically significant (p < .05).

ELL students remained steady in reading from the first year to the second year, while English speakers lost 1 NCE point. In math, ELL students gained 4 NCE points, as opposed to 2 NCE points for English speakers.

In reading, 20% of students in the bottom quartile in 1999–2000 moved out of this quartile in 2000–2001. The corresponding figure for math scores was 42%.

Overall, there were 2% more students in the bottom quartile in reading in spring 2001 than in spring 2000. In math there was a seven-percentage point drop in the percentage of students in the bottom quartile (31% in spring of 2000 vs. 24% in spring of 2001).

ELL students showed a 1% drop in the proportion scoring in the lowest quartile in reading and a 10% drop in the lowest quartile in math. The corresponding drops for English speakers were 3% and 6%, respectively.

For the subsample with data for three consecutive years (two years in START), results showed a decline of 2 NCE points in reading and an increase of 4 NCE points in math between spring 1999 and spring 2001. For the lowest quartile scorers in this subsample, 22% moved out of this quartile in reading and 45% moved out of this quartile in math.

START students showed very little overall change in school attendance between 1999–2000 and 2000–2001, missing 5.5 days and 5.4 days, respectively. Among problem attenders (those missing 10 or more days in 1999–2000), the number of days absent dropped from 14.8 in 1999–2000 to 10.1 in 2000–2001, a reduction of 4.7 days absent.

Among the subsample with data for three consecutive years (two years in START), students missed an average 7.3 days in 1998–1999, 5 days in 1999–2000, and 5.1 days in 2000–2001, for a two-year drop of 2.2 days absent. For problem attenders in this group, the corresponding figures were 15.9 days absent in 1998–1999, 9.8 days absent in 1999–2000, and 9.7 days absent in 2000–2001, for a two-year drop of 6.2 days absent.

Questionnaire data revealed a drop from pretest to posttest in the percentage of students reporting that they liked going to school and studied hard for a test in the past 30 days.
Family A higher proportion of questionnaire respondents from pretest to posttest reported speaking daily with a parent or guardian about school in the past 30 days.

Staff focus groups revealed that many staff felt that their provision of reliable after school care enabled parents to work, particularly for families coming off of welfare.
Youth Development There were no changes from pretest to posttest questionnaire in the percentage of respondents reporting always feeling safe at school.

Evaluation 3: Evaluation Report 2001/2002


Evaluation Description

Evaluation Purpose To examine (1) program implementation strengths, challenges, and emerging solutions and (2) program participants' outcomes in START attendance, school attendance, academic achievement, and behavior and attitudes.
Evaluation Design Quasi-Experimental and Non-Experimental: Eleven sites were chosen for an in-depth examination of implementation issues. To assess the impacts of the program on participating youth, evaluators constructed an evaluation group and a comparison group. The evaluation group consisted of 1,440 third through sixth grade START students who attended START at least 60 days during 2001–2002 and had complete standardized test information for 2000–2001 and 2001–2002. Based on school district records, a total of 814 comparison students were successfully matched to the evaluation group on five characteristics: school, language fluency, gender, grade, and equivalent SAT-9 reading scores for 2000–2001 (within plus or minus 5 Normal Curve Equivalence [NCE] points). The evaluation sample was 33% African American, 26% Hispanic/Latino, 25% Asian/Pacific Islander, 14% white, and 2% other. The comparison group was 23% African American, 28% Hispanic/Latino, 25% Asian/Pacific Islander, 21% white, and 3% other.
Data Collection Methods Interviews/Focus Groups: Individual interviews were conducted during site visits in spring 2002 at 11 schools with 20 Program Leaders, 11 Site Directors, 11 Principals, and 8 Literacy Coaches. In addition, nine focus groups were conducted with groups of 4–8 classroom teachers, and another 18 focus groups were conducted with groups of 8–10 students. Interviews and focus groups concentrated on program implementation.

Observation: Observations were conducted in 25 START classrooms during site visits at 11 schools in spring 2002 to examine START implementation.

Secondary Source/Data Review: School attendance and test score data were collected from data files maintained by school districts and school sites. START attendance data were provided by individual program sites.

Surveys/Questionnaires: Pretest and posttest questionnaires were collected from 240 START students that asked questions about how often in the past 30 days the students wanted to go to school, studied for a test, and felt unsafe at school. Pretest/posttest data were collected from as many START students as possible at the beginning and end of the 2001–2002 year.

Tests/Assessments: Students' SAT-9 test scores were provided by the school districts for both the 2000–2001 and 2001–2002 school years.
Data Collection Timeframe Data were collected during the 2001–2002 school year, including academic records from the current and previous school year.


Findings:
Formative/Process Findings

Activity Implementation School sites varied in the extent of enrichment activities offered. Four of the 11 sites that were visited offered enrichment at least three days per week, while the remaining seven delivered enrichment one day per week. Most stakeholders concurred that more enrichment was necessary, such as art, science, and sports, which would otherwise not be available to many START students. Some sites responded by bringing in visitors and guest speakers to teach enrichment activities such as drama, dance, or art.
Parent/Community Involvement Strengthening ties with parents was found to be a promising practice, including follow up with parents when a child was absent and daily contact with parents when they picked up their children from the program.
Program-School Linkages Based on interviews, focus groups, and observations, the evaluators found that the linkages between START and the regular school day were enhanced when Program Leaders worked directly with classroom teachers, supported by the Literacy Coach and Site Director. Examples of promising practices included: Program Leaders observing classroom teaching, classroom teachers present during START time, and classroom teachers regularly reporting progress to the program. In addition, Literacy Coaches facilitated the schools' communication with both Program Leaders and Site Directors and provided SAT-9 test material to support START in connecting to school curriculum. Other practices contributing to strong program-school linkages were hiring school staff as Program Leaders and inviting START staff to school staff meetings and retreats.

One program challenge identified by the evaluators was the shared use of space with classroom teachers. In particular, some school staff reported in interviews that START left their classrooms untidy or with missing or broken items. START staff alleviated this problem by replacing missing or broken items and voicing teacher concerns to Program Leaders and Site Directors.
Recruitment/Participation START students were approximately evenly divided between males and females, although girls attended START at a slightly higher rate than boys.

The presence of male Program Leaders and the availability of structured competitive sports were found to help in attracting and retaining boys in the upper grades.

Most START students were in grades one through five; participation tapered off slightly at sixth grade. Similarly, attendance days tended to decline somewhat in the upper grades, particularly at sixth grade. Kindergarteners were also eligible to attend, but because many kindergarten students attended school for half a day, kindergarten students made up only 6% of all START students.

START students participated in the program for an average of 107 days during 2001–2002 (out of an average program operation length of 148 days).
Staffing/Training Based on interviews and observations, the evaluators found that the Literacy Coach position played a pivotal role in creating a high quality after school program. Sites varied in their access to a high quality Literacy Coach. Quality Literacy Coaches made daily contact with Site Directors and Program Leaders, provided training on school curriculum, provided ongoing consultation on teaching methodology and classroom management, and served as a liaison between classroom teachers and START staff.

The need to upgrade the quality of Start's instruction and classroom management was a challenge noted by the majority of teachers and principals interviewed and by evaluators during site observations. Practices that were found to help with this problem were strengthening the presence of the Literacy Coach in START classrooms and classroom teachers' mentoring of Program Leaders.

One promising practice (i.e., a practice found to promote a high quality after school program) revealed through interviews with Site Directors was hiring school staff and college students as Program Leaders for START. College students were found to contribute skills in delivering academic instruction and support.

Using the Literacy Coach as a “floating tutor” in START classrooms was found to be a promising practice, providing students with extra academic support from a classroom teacher and also letting Program Leaders observe effective academic instruction and classroom management.

Some sites surveyed students for their preferences for enrichments activities and took these preferences into consideration in their selection in hiring Program Leaders.


Summative/Outcome Findings

Academic Among students in the lowest quartile of reading SAT-9 scores during 2000–2001, the START evaluation group experienced greater growth in reading achievement than comparison group students (12.5 NCE points vs. 3.9 NCE points). Similar patterns prevailed for math SAT-9 scores (19.1 NCE points vs. 8.5 NCE points).

The results for higher performing START students were mixed for reading scores. For the two middle quartiles of achievers in 2000–2001, the second lowest quartile in both groups made slight gains, while the third quartile dropped approximately 10 NCE points. Among the top performing quartile in 2000–2001, the START group's scores dropped approximately 17.5 NCE points, while the comparison group dropped by 4.8 NCE points.

Results for higher performing START students in math were also mixed. Among the second lowest quartile, START students made larger gains than comparison group students (6.7 NCE points vs. 4 NCE points); among the third quartile, both groups dropped slightly (a 3-NCE-point decrease vs. a 0.9-NCE-point decrease); and among the highest quartile, the START group dropped 16.6 NCE points while the comparison group dropped 4.6 NCE points.

The gains in student achievement among low-performing students generally increased with the number of days they participated in START. Low-performing START students who participated between 60 and 90 days gained 10 NCE points in reading and 12 NCE points in math. For 91–120 days of participation, gains were 14 NCE points in reading and 23 NCE points in math. For 121–150 days, gains were 12.5 NCE points in reading and 23 NCE points in math. And for over 150 days, gains were 26 NCE points in reading and 24 NCE points in math.

START students were classified as having good school attendance and had no change in attendance from 2000–2001 to 2001–2002, with six days absent on average for each year.

For the over 200 students in the START evaluation group whose school attendance was classified as a problem (missing over nine days in 2000–2001), attendance improved substantially in 2001–2002; their days absent dropped from an average of 15 to 11 days.

Among the 183 START evaluation group students who participated in both 2000–2001 and 2001–2002, these two-year participants showed a statistically significant growth in math NCE scores (p < .05) and a nonsignificant decline in reading NCE scores.

The average days absent from school for two-year attendees increased slightly from 2000–2001 to 2001–2002 (from 3 days to 4.5 days), but overall these students were classified as good attendees. Among two-year attendees classified as problem attenders (missing more than nine days of school in 2000–2001), the average number of days absent dropped from 13.7 to 7.3.

Among the 60 START evaluation group students who participated in 1999–2000, 2000–2001, and 2001–2002, these three-year participants showed sustained growth in both reading and math NCE scores (47 NCE points in spring of 1999 to 52 NCE points in spring of 2002 in reading; 32 NCE points in spring of 1999 to 43 NCE points in spring of 2002 in math).

School attendance data for three-year attendees showed that this group had consistently good attendance, missing roughly four days of school per year.

The percentage of surveyed students responding that they wanted to go to school everyday in the past 30 days increased from 56% at pretest to 60% at posttest.

Fewer surveyed students reported studying for a test daily at the posttest than on the pretest, but more students reported studying for tests a few times per week at the time of the posttest than at the time of the pretest.
Youth Development The percentage of surveyed students reporting that they never felt unsafe at school increased from 45% at pretest to 71% at posttest.

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project