You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼


Program Description

Overview The Santa Ana Unified School District in Southern California received renewable 3-year funding from the State of California to operate the After School Education and Safety Program in four urban public middle schools. (The After School Education and Safety Program was known as the After School Learning and Safe Neighborhoods Partnerships Program [ASLSNPP] at the time of the evaluation in this profile and therefore is referred to by that name.) The purpose of the program is to support positive development for Santa Ana students, many of whom live in unsafe neighborhoods that offer few out-of-school learning opportunities. In particular, the program aims to provide a safe environment while targeting improvements in academic achievement, attendance, and behavior. The District and its partners designed the program integrate academics with recreational enrichment so that the multiple academic and social needs of participating students are met.
Start Date September 1999
Scope local, state
Type after school
Location urban
Setting public schools
Participants middle school students
Number of Sites/Grantees The Santa Ana Unified School District is the recipient of a 3-year renewable grant for ASLSNPP programs in four middle schools.
Number Served 617 in 1999–2000
Components In the Santa Ana Unified School District, after school programs operate at four middle schools from 3pm to 6pm Monday through Friday free of charge. These four middle schools serve predominantly Latino students with limited English proficiency and from high poverty backgrounds. Although each site’s schedule varied, a typical program schedule included a 1-hour homework period, a 1-hour arts or life skills component, and a 1-hour sports component. Students who had completed their homework participated in a literacy or math enrichment activity for the remainder of the homework hour.
Funding Level $50 million was given for the statewide ASLSNPP programs during 1999–2000 school year. An additional $35 million was given beginning in school year 2000–2001.
Funding Sources State of California, through the After School Learning and Safe Neighborhoods Partnerships Program; Santa Ana City Parks, Recreation, and Community Services Agency; the Department of Education at the University of California, Irvine; the Orange County Social Services Agency; the Orange County Department of Education; and the Santa Ana Police Department


Evaluation

Overview This evaluation examined first-year differences (1999–2000) in outcomes between participants and nonparticipant matches at the four middle school ASLSNPP sites in Santa Ana. Specifically, the evaluation measured outcomes in the areas of student achievement, attendance, and feelings of safety.
Evaluator Jenel Prenovost, Ed.D., University of California, Irvine and University of California, Los Angeles
Evaluations Profiled A First-Year Evaluation of After School Learning Programs in Four Urban Middle Schools in the Santa Ana Unified School District
Evaluations Planned none
Report Availability Prenovost, J. K. E. (2001). A first-year evaluation of after school learning programs in four urban middle schools in the Santa Ana Unified School District. Irvine, California: Author.


Contacts

Evaluation Jenel Prenovost
After School Collaboration for Educational Enrichment
3321 South Fairview Street
Santa Ana, CA 92704
Tel: 714-513-3397
Fax: 714-433-3487
Email: jprenovost@sausd.k12.ca.us
Program Peggy Adin
Santa Ana Unified School District
3321 South Fairview Street
Santa Ana, CA 92704
Tel: 714-433-3484
Profile Updated December 3, 2002

Evaluation: A First-Year Evaluation of After School Learning Programs in Four Urban Middle Schools in the Santa Ana Unified School District



Evaluation Description

Evaluation Purpose To determine the characteristics of participants in the Santa Ana ASLSNPP program, assess whether the program participants have greater gains on the SAT-9 Reading and Math Tests than matched pair comparison groups who do not attend the program by the end of the first year of program implementation (1999–2000), assess whether program participants have greater gains in school attendance than matched pair comparison groups who do not attend the program by the end of the first year of program implementation (1999–2000), and assess whether program participants feel safer at school than matched pair comparison groups who do not attend the program.
Evaluation Design Quasi-Experimental: The quasi-experimentally designed experiment had three groups, two of which were treatment groups and one of which was a matched nonparticipant comparison group, at each of the four sites. The nonparticipant comparison group was composed of students who did not participate in the program, but who had similar characteristics to program participants, including school site, grade level, gender, ethnicity, Limited English Proficiency (LEP) rating, free and reduced-price lunch (FRL) status, and spring 1999 grade point average. The first treatment group was the high-dosage group consisting of students who attended the program for more than 38 days during the 181-day school year, the median number of program attendance days for all participants. The second treatment group was the low-dosage group consisting of students who attended the program for fewer than 38 days. Participants joined the program as a result of fliers given to parents at the beginning of the school year and as a result of referrals from teachers or counselors due to a student’s low grades or scores on the SAT-9 test or due to a perception that the student could benefit from the program.
Data Collection Methods Document Review: School and district records were reviewed to obtain the following information about each student: school site, grade level, gender, ethnicity, LEP rating, FRL status, spring 1999 grade point average, identity of classroom teachers, and length of time in the after school program. In addition, student school attendance data were collected from the four schools.

Surveys/Questionnaires: Students completed a survey required by the State ASLSNPP program, which included questions about whether or not students wanted to go to school in the last 30 days, whether or not students have ever felt unsafe at school in the last 30 days, and whether or not students have studied hard for a test in the last 30 days. This survey was taken in June of 2000, toward the end of the first year of the programs.

Tests/Assessments: Students’ reading and math scores on the spring 2000 Stanford Achievement Test (SAT-9) were used to assess program impact on achievement.
Data Collection Timeframe Data were collected during 1999 and 2000.


Findings:
Formative/Process Findings

Activity Implementation The typical program schedule consisted of a 20-minute snack period, 1-hour of homework, 1-hour of arts or life skills, and 1-hour of sports.

Each site maintained a 20:1 student teacher ratio.
Parent/Community Involvement Staff engaged in developing relationships with partnering staff representing multiple agencies, experience, and training. They also engaged in developing relationships with administrators, parents, teachers, and students.
Program Context/Infrastructure The average 1999 National Percentile Ranking (NPR) in reading scores for sixth graders among Santa Ana students was 17.

Forty percent of the population at two of the schools live within the Empowerment Zone, a designated area of the city targeted to receive funding for improved employment, housing, and education opportunities.
Recruitment/Participation Program participants were generally similar the comparison group, as expected due to the matching procedure. Ninety-four percent of matched students did not participate in any other after school program.

Program participants (and the nonparticipating matched comparison group) were found to be significantly (p < .001) different than the general school population across several demographic variables. Program participants were significantly more likely than the general school population to be Hispanic (93% vs. 82%) to be designated as LEP (71% vs. 60%), to be in the sixth grade (44% vs. 32%), to be male (62% vs. 43%), to be eligible for free or reduced-price lunches (84% vs. 73%).

The program participants missed more days of school at baseline than the remainder of the school population (7.39 days vs. 6.72 days), although this difference was not statistically significant.

The program participants scored significantly (p < .05) lower on the SAT–9 Reading and Math tests than the rest of the school population at baseline.

Program participants in the low-dosage group, those who had attended the after school program less than the median number of days, were significantly (p < .001) more likely to have been suspended during their school career (24% of this group had been suspended) than higher dosage participants (16% of whom had been suspended), the comparison group who did not attend the program (16% of whom had been suspended), and the remaining school population (14% of whom had been suspended).

The mean number of days students attended the program during the 1999–2000 school year was 56.07 days and the median was 38 days.

Girls had a significantly (p < .035) higher program attendance than boys, with a mean number of days for girls of 63.45 while the mean number for boys was 52.24.
Staffing/Training Daily staff included one site coordinator, one credentialed teacher, one instructional assistant, one arts or life skills instructor, and one sports instructor from the City of Santa Ana Parks, Recreation and Community Services Agency. Volunteers from the local university also worked with the programs.

All of the site coordinators were Latino and bilingual and lived in Santa Ana. There were several teachers at each site who rotated into the daily credentialed teacher position. Each site selected a Lead Teacher responsible for providing academic enrichment lessons and working with the Program Director to create engaging and enriching lessons. The Lead Teacher also served as liaison between the Program Director, site coordinator, and the other teachers. This position became integral in coordinating and planning curricula, as well as scheduling teachers.


Summative/Outcome Findings

Academic No statistically significant differences were found in SAT-9 Reading improvement scores or NPR scores between treatment and comparison groups from 1999 to 2000. However, high-dosage participants improved somewhat more than matches; this was especially true for eighth graders, females, and students of limited English proficiency.

There were site differences in improvements in SAT-9 Reading scores, which were not statistically significant.

No statistically significant differences were found in SAT-9 Math improvement scores from 1999 to 2000 or NPR scores between treatment and comparison groups. However, high-dosage participants improved somewhat more than matches and all program participants improved more than the general school population. The subgroups of high-dosage sixth graders, high-dosage male participants, and high-dosage LEP participants also improved more than matches. NPR scores favored high-dosage LEP and high-dosage LEP participants when compared to the low-dosage participants.

At two of the schools, there were significant differences in SAT-9 Math test scores between particular subgroups of program participants and comparison group matches. At the first, high-dosage males (p < .038) and high-dosage sixth graders (p < .048) improved significantly more that the matches. These same two subgroups also had significantly higher NPR scores on the SAT-9 Math test than matches. High-dosage participants at the second school had significantly (p < .084) higher NPR scores than low-dosage participants.

There were significantly (p < .005) fewer days of school missed by high-dosage participants (5.56 days) as compared to low-dosage participants (7.46 days) and the matches (6.80 days). In addition, high-dosage LEP students missed significantly (p < .002) less school than low-dosage participants and the matches. Higher-dosage sixth and eighth graders, on the other hand, had higher means in days absent than the matches, although this also was not statistically significant.

The program was associated with a nearly significant (p < .082) difference in improvement in school attendance. There was a statistically significant (p < .031) finding that high-dosage students improved more in their school attendance (1.36 days) than low-dosage students (.32 days). Also, high-dosage LEP students improved their attendance significantly (p < .05) more than low-dosage participants (.29 days) and matches (-.18 days).

Site-level differences in the program impact on attendance were apparent, with different subgroups at the four schools evidencing significantly higher school attendance over the comparison groups.

In response to the survey question about how often students wanted to come to school in the last 30 days, the treatment and comparison groups answered almost identically. No statistically significant differences were found among treatment groups in answers to the survey question about how many times in the last 30 days the students had studied hard for a test. However, males in the program studied slightly harder than the matches and female matches studied slightly harder than either the high- or the low-dosage participants.

When parents were asked how the program has made a difference in their child, 73% of parents answered that their child does more homework and performs better in school.

Sixty-seven percent of program participants responded that program had made a difference in them. Fifty-six percent of these students commented that the program has had a positive effect on their ability to do homework or get better grades, even though students were not specifically asked about homework or grades in the survey question.
Prevention There were no significant differences in feelings of safety at school in the last 30 days, according to the self-reported survey responses, between the treatment and comparison groups.

 

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project