You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Program Description

Overview The California 21st Century High School After School Safety and Enrichment for Teens (ASSETs) Program funds school–community partnerships to establish out-of-school time (OST) programs that provide high school students in California with academic support, enrichment, and family activities. Funding priority is given to programs that serve students who attended schools with the poorest academic ranking in the state.
Start Date 2003 (reauthorized by the California Legislature in 2006)
Scope state
Type afterschool, before school, summer, weekends
Location urban, suburban, rural
Setting public school, private school, community-based organization, religious institution, private facility, recreation center
Participants high school students
Number of Sites/Grantees ASSETs funds have been awarded to 43 grantees serving 57 high schools in three cohorts. Across cohorts, grantees included: 25 local education agencies (LEAs), 4 county offices of education, 4 public entities (e.g., city/county government, higher education), and 10 private entities (e.g., community-based organizations, faith-based organizations, private education institutions).
Number Served 27,900 youth (2005–2006 school year)
Components ASSETs grantees are required to provide three components to support high school students: (a) academic assistance, including activities that support meeting state academic standards (e.g., tutoring, homework help, high school exit exam prep, and college prep); (b) enrichment activities (e.g., community service, service learning, opportunities to mentor and tutor younger pupils, career and technical education, job readiness, arts, computer and technology training, physical fitness, and recreation activities); and (c) family literacy services. These components must be embedded within a developmental framework supporting the acquisition of personal and social assets that promote adolescent well-being and a successful transition to adulthood.
Funding Level $2.5 million annually, as of 2003
Funding Sources California Department of Education (CDE), through federal 21st Century Community Learning Centers (21st CCLC) Initiative funding

Evaluation

Overview A multi-year, mixed-methods evaluation examined program participation, activity implementation, and program outcomes.
Evaluator WestEd
Evaluations Profiled Interim Report

ASSETs Final Evaluation Report
Evaluations Planned None.
Report Availability Hipps, J., Diaz, M., & Wingren, G. (2006). California 21st Century High School After School Safety and Enrichment for Teens (ASSETs) Program independent evaluation: Interim report. San Francisco: WestEd. Available at: www.wested.org/online_pubs/assets_interim_report.pdf

Hipps, J., & Diaz, M. (2007). ASSETs final evaluation report: California 21st Century High School After School Safety and Enrichment for Teens (ASSETs) Program. San Francisco: WestEd. Available at: www.wested.org/cs/we/view/rs/840


Contacts

Evaluation Jerome Hipps
Project Director
WestEd
300 Lakeside Drive, 25th Floor
Oakland, CA 94612-3534
Tel: 510-302-4227
Fax: 510-302-4242
Email: jhipps@wested.org
Program Patricia Terry
Consultant
California Department of Education
After School Partnerships Office
1430 “N” Street, 6th Floor
Sacramento, CA 95814
Tel: 916-319-0329
Email: pterry@cde.ca.gov
Profile Updated May 9, 2011

Evaluation 2: ASSETs Final Evaluation Report


Evaluation Description

Evaluation Purpose To examine the program’s impact on participating schools and benefits to youth participants, the extent to which ASSETs programs address and integrate a youth development approach into their design and implementation, what factors contribute to program effectiveness, and any unintended consequences of program implementation.
Evaluation Design

Quasi-Experimental and Non-Experimental: Data were collected on all 43 grantees. CDE received grantees’ data for both the 2004–2005 and 2005–2006 program years on program participants from all but five grantees serving eight schools. All but two grantees, which served three schools, submitted a completed Evaluation Guidebook for the 2005–2006 year.

For Cohort 1, 2005 and 2006 test scores on the California High School Exit Exam (CAHSEE) and on the English language arts (ELA) section of the California Standards Tests (CST) were analyzed for program participants (program group) and nonparticipants attending the same schools (comparison group). Data were available for 11 of the 15 Cohort 1 schools, although only 6 of these schools submitted the data needed to compare student outcomes on the CAHSEE. CST data included only students with CST results from both years to compare changes in the program group to the comparison group. Performance of these two groups was also compared to students statewide. For the ELA CST, the program group was examined as a whole and by the number of days they attended the program.

In the spring of 2006, 1–2 day site visits were conducted at eight of the programs: four in Cohort 1 (selected because they were found to have many characteristics of quality afterschool programs), two in Cohort 2 (selected from those in the same districts as the four Cohort 1 programs), and two in Cohort 3 (selected because data indicated they were making substantial progress). During these visits, interviews were done with 8 program directors, 7 program coordinators, 7 principals, 52 youth participants, and 5 collaborative partners.
Data Collection Methods Document Review: Programs completed the ASSETs Program Evaluation Guidebook which included program data on youth participant demographics and participation levels; assessed needs and goals; activities and their links to goals and standards; links to the school-day curriculum; administration and staffing characteristics; professional development; family literacy; institutional capacity; youth involvement; youth’s physical, intellectual, emotional, psychological, and social development; advisory group implementation; sustainability; collaborating organizations; and evaluation narrative (challenges, supporting data, and next steps).

Interviews/Focus Groups: Interviews and focus group questions related to progress during the 2005–2006 school year, program benefits to schools and youth, the role of ASSETs activities in helping prepare youth for the CAHSEE, and facilitators to program implementation (e.g., program–school linkages, youth development approaches present in the program, and coordination of the ASSETs program with other programs at the school).

Secondary Source/Data Review: CDE collected youth data on program attendance, demographics, academic test scores, school absences/suspensions, and high school enrollment. The 21st CCLC PPICS system was used to obtain data about program staffing.

Test/Assessments: The CAHSEE is an exam that, beginning with the class of 2006, all California students are now required to pass in order to graduate from high school. It has two parts: English-Language Arts (ELA) and math. Students take this exam in grade 10; those who do not pass a section on their first try may retake the exam in grades 11 and 12.

The CSTs, a component of California’s Standardized Testing and Reporting program, were developed to assess student achievement on state ELA and math curriculum standards. All students in grades 9–11 take a grade-level ELA test. Results from math CSTs were not examined, since there are eight different CSTs that a high school student may take, depending on the specific math course a student is enrolled in. Results are reported as five performance levels: advanced, proficient, basic, below basic, and far below basic. The state’s goal is for all students to score at the advanced or proficient levels.
Data Collection Timeframe Data were collected for the 2005–2006 school year, with some data collected for the 2004–2005 school year to provide a baseline (as noted above).


Findings:
Formative/Process Findings

Activity Implementation Programs’ educational enrichment activities related to many broad areas such as college and career readiness, leadership and mentoring, technology, visual and performing arts, physical education and recreation, and academics. Some sites used a project-based learning or thematic approach in their enrichment classes. In describing this component, several programs indicated how they sought to augment students’ school-day experiences with learning activities that contributed to their development in multiple areas.

Academic assistance at programs provided support to help youth succeed academically through such methods as tutoring, homework help, walk-in support, and study groups. Some sites adopted a case management approach, using data to pinpoint youth’s needs then assigning staff to work with specific groups of youth. A few sites offered peer-to-peer tutoring, sometimes with older youth working with younger ones.

Most programs provided activities to help youth pass the CAHSEE. Activities were held both after school and on Saturdays, depending on the location. Afterschool classes were 1–2 hours long and were held 2–4 times a week. Saturday classes lasted 2–3 hours. Prep activities were offered for 4–12 weeks with some programs offering classes 2–3 times a year in coordination with the schedule for CAHSEE administration.

Programs used different approaches to target students for CAHSEE assistance, depending on what program and school staff felt made most sense given a school’s needs and resources. Specific groups targeted included: seniors needing to pass at least one portion of the CAHSEE, students who had not passed CAHSEE by the end of grade 10, sophomores with a good chance of passing the CAHSEE on their first try, special needs students, and English learners.

Program curricula for CAHSEE prep came from multiple sources and often included test taking strategies in addition to test-related content. Some programs drew on the preparation curriculum used during the school day. Other programs, including those administered by community-based organizations (CBOs), worked with local high school teachers to develop a curriculum. A third group of programs purchased CAHSEE preparation materials from established publishers like Kaplan and Princeton Review.

Programs made resources available after school that would not otherwise be available, such as libraries, computer labs, and media centers.

Sites exposed youth to possible options after high school through formal and informal conversations with staff, guest speakers from the community, exposure to potential interests and hobbies, and job skills development events.
Parent/Community Involvement Multiple providers worked with youth in the academic assistance component, including local college students, Americorps members, and CBO staff. For CAHSEE assistance, some programs also teamed with county Regional Occupational Programs or local community adult schools.

Through partnerships with local universities, police departments, churches, businesses, and CBOs, programs tapped into resources to support youth development in many areas. For example, one program convened monthly meetings with youth, parents, and community members to discuss issues affecting the community. Some programs used community partners to leverage supports available to youth in the community.
Program Context/ Infrastructure Ethnic minorities comprised over 50% of students attending ASSETs schools. Hispanic youth were the largest ethnic group at 43 schools, consistent with state demographics. Further, Hispanic youth were the largest group of students attending schools in 8 of the 13 counties where ASSETs schools were located. African American youth were the largest group of students at 6 schools while Asian youth were the largest group at 5 schools, all in northern California.

English learner enrollment at ASSETs schools ranged from under 10% to over 50%, and was greater than the state average of 25% at 36 of the 56 ASSETs schools.

Programs addressed issues of safety for youth participants by having security staff on site and connecting adult supervisors through walkie-talkies. In rural areas, some programs worked with school districts to arrange for participants to take a late bus home.

Programs provided a structured setting while allowing youth flexibility to be involved at a level that facilitated a sense of independence in achieving their goals. Often, grantees reported providing youth with explicit rules and expectations for their behavior and performance.

Programs gave youth opportunities to take part in group activities that emphasized inclusion of all youth and maintained a social environment that recognized, appreciated, and encouraged differences in cultural values, gender, race/ethnicity, and sexual orientation.

According to stakeholders, by providing activities that supported the whole student, programs captured a side of youth that often was missed during the school day due to strict curricular guidelines, large class sizes, and limited class time.

In interviews, youth, principals, and program staff commented that programs added to the activities available to youth after the school day. At some schools, these activities were important because their communities did not offer many other resources for youth. Youth reported that these activities gave them reasons to come to school. Other youth felt that having activities like tutoring and homework assistance helped youth do better in school.

Every program worked to involve youth in decision-making, usually using multiple methods. Most grantees conducted surveys and/or focus groups to get youth feedback in such areas as activity satisfaction, program benefits, feelings of safety at school, and connections with adults. Student advisory boards were also widely used to get youth input. Youth were also sometimes members of a program’s formal advisory board. Further, several grantees noted that their service-learning activities promoted youth involvement by allowing youth to choose how to focus their energies. This involvement helped programs in many ways. For example, one program noted: “Youth tend to stay focused on the activities when they were co-facilitated by youth...”

Many grantees were surprised by programs’ abilities to create a positive afterschool culture (i.e., creating a sense of community, facilitating interactions that would not otherwise happen, and teaching in innovative ways that provided youth flexibility).
Program–School Linkages Grantees worked to keep their enrichment activities tied to state content standards. Several programs reported that they asked instructors to develop a syllabus for their activities and demonstrate how they incorporated content standards into their plans.

Most ASSETs programming occurred on high school campuses, which allowed an easy transition of the school day social norms to afterschool programming. The majority of programs maintained the rules for behavior from the school day and embedded them in a more relaxed environment with increased flexibility in time management.

Two schools used afterschool hours to “backfill” learning in areas where students were falling behind in their courses. For example, at one school, when assessments showed students did not understand concepts in their regular algebra course, they participated in a “shadow” class scheduled during the afterschool program where they received help to master their weak areas.

Many programs viewed the regular school and afterschool hours as a seamless continuum; eight sites spoke explicitly of their programs as either being an extension of the school day or offering a seamless school day. Several schools framed afterschool as additional school periods, adding two periods to the usual six period school day. At another site, staff viewed afterschool as an opportunity to engage students academically and build relationships with students in ways that were not possible in the regular school setting.

School-day teachers were involved in programs in a variety of ways and played key roles in linking school and afterschool activities. Guidebook data provided examples of teacher involvement in areas where schools and programs worked together beneficially: publicizing program activities, recruiting youth, developing curriculum, and leading program activities. Programs at smaller rural high schools reported that teachers involved in afterschool activities were familiar with students, which helped them tailor academic assistance to student needs.

All but four grantees (three of which were CBOs) employed teachers in their program to supervise activities that were a part of the academic assistance and educational enrichment components, although the number of teachers working with programs varied greatly from site to site, according to grantee reports. Teachers represented anywhere from 3 to 70% of staff working with youth.

During site visits, program coordinators and school principals reported that programs provided opportunities for students to work with teachers who were not their regular school-day instructors, which they felt was beneficial in some cases since some students might connect better with a teacher they did not work with during the school day. However, some students saw the opportunity to work afterschool with their school-day teachers as a plus.

Some schools had difficulties attracting teachers to the program because teachers were tired at the end of the day or were new to teaching and needed to prepare for the next day. As such, programs limited the time commitment required of teachers who wanted to lead afterschool activities. At most sites (perhaps with the exception of tutoring and homework help), afterschool activities met just 2–3 times a week. Programs also lessened time demands by scheduling activities that lasted for 6 weeks rather than an entire semester. One program increased the number of math teachers working afterschool from just five in the 2004–2005 school year to twelve the next year by responding to teachers’ feedback that they preferred to work with their own students.

Many programs grappled with staff pay when teachers served as staff. District contracts with teachers’ unions often specified an hourly pay rate that teachers were to receive when working outside the school day. The rate was sometimes too high for programs’ limited budgets. Grantees dealt with the problem by hiring the teachers through a CBO or, in the case of non–local-education-agency (LEA) grantees, employing teachers as employees of the grantee.

Guidebook data revealed multiple strategies that programs used to communicate with schools. Program staff attended faculty and department meetings, participated in professional development provided to teachers, and attended meetings of the school site council. Participating in meetings like these allowed programs to develop a visible presence on the school campus and facilitated communication with the school.

Programs shared information about students with schools in many ways. Some grantees and schools had formal referral processes that provided programs with information about students’ academic needs. Programs informed schools about students’ progress through formal feedback forms on students provided to teachers, regular reports to teachers and principals, email contacts with teachers, and face-to-face conversations.

CBOs established multiple strategies to work with schools, which they felt contributed to their programs’ success. Some used formal mechanisms to keep teachers and counselors informed about students’ progress after school. For instance, one CBO developed a “Confirmation of Tutoring Form” to communicate with teachers. Another CBO found that co-location with the school allowed for “greater access to students, student records, school staff, and resources.”

Guidebook responses indicated that programs often faced challenges in obtaining sufficient classroom space for afterschool activities.

Guidebook responses revealed discrepancies between grantees in the degree to which CBOs had access to schools’ student data to support afterschool activities. Some CBOs could not directly access student data systems and relied on school staff to provide any student data.

The coordinator position often had high turnover at programs, which disrupted relationships built with schools and youth. One grantee handled this challenge by hiring a schoolteacher to serve as the coordinator, which brought stability to the position.
Recruitment/ Participation Over 27,900 youth attended ASSETs program activities during the 2005–2006 year.

Participants were largely from minority ethnic groups: 56% Hispanic; 17% African American; 9% Asian, Pacific Islander, or Filipino; and 5% White. Participant ethnic composition was similar to the school at 37 of 53 sites with available data. At seven schools, participant ethnic composition differed from the overall school (the remaining programs lacked sufficient ethnicity data).

Overall, 26% of youth attending ASSETs programs were English learners. At 31 schools, the proportion of English learners attending programs was approximately equal to their school enrollment. At four sites, English learners were underrepresented among participants, while at six sites, English learners were overrepresented. Twelve programs had insufficient data to judge the rate of English learner involvement in ASSETs activities.

High school program attendance rates declined as age increased: 28% of participants were in grade 9, 28% were in grade 10, 23% were in grade 11, and 18% were in grade 12 (for 4% of participants, grade was unknown).

An average of 31% of students attending host schools participated in programs, ranging from 1–91% per school.

Youth participants attended the program an average of 26 days, ranging from 4 to 164 days on average per school.

Program attendance at the 40 reporting schools increased 84% from 2004–2005 to 2005–2006, from about 13,900 to over 25,600, and the percent of youth involved 30 days or more increased 37%. Some of this growth reflected the fact that 2005–2006 was the first full year of operation for some programs.

Data available from eleven Cohort 1 schools showed that 32% more youth attended a program in 2005–2006 than attended in 2004–2005. Further, attendance was up at ten of the eleven Cohort 1 schools. However, the number of youth attending 30 days or more declined both overall by 38% and at nine of the eleven schools. Interview data suggest that these decreases were partly related to changes in site coordinators and learning how to effectively staff that position.

All grantees targeted specific groups of students for academic support using criteria including quantitative data (e.g., grade point averages [GPA], report cards, achievement test scores) and referrals from school staff and parents. Some programs also targeted youth who were at risk of dropping out, in foster care, athletes needing to maintain their GPA, or youth who were new to the school. At 14 programs, all participants were required to be involved in academic support either every day they participated or just 1–2 times a week. Another 25 programs did not have specific participation requirements, though many encouraged certain youth to participate and program staff worked with school staff, parents, and youth to move targeted students into support activities. Lastly, 13 programs required specific groups of students to receive academic assistance.

Some grantees offered incentives to youth for participating in academic assistance activities. In their guidebooks, 23 sites reported tangible incentives, which were almost always academically oriented (e.g., earning extra credit, making up school absences).

Some grantees reported in their guidebook obstacles to attracting youth. Several indicated difficulty drawing youth because participating in afterschool activities was not a part of the school’s culture. Programs believed that as participants got excited about program activities and talked to their friends about their experiences, attendance would grow.
Staffing/Training Grantees offered many examples of ways their programs provided supportive relationships, and youth attested to the positive relationships they had with adults in the program. Grantees often trained staff in developing positive relationships with youth, modeled compassionate ways of interacting with one another, nurtured mutual respect for listening to others ideas and their personal property, and encouraged personalized attention to youth. Some programs prided themselves in their ability to develop caring relationships with youth.

Summative/Outcome Findings

Academic Separate analyses were conducted for students who had yet to pass the ELA portion of the CAHSEE and those who had yet to pass the math portion. Of these youth, 10th grade students who did not participate in ASSETs activities had significantly higher ELA and math scale scores than ASSETs participants (p = .00 for both tests). For grades 11 and 12, ELA and math scores did not significantly differ between ASSETS participants and their classmates.

On the ELA CAHSEE test, youth who did not attend ASSETs programs had significantly lower passing rates than youth who attended programs for at least 1 day (p < .01), 10 days (p = .01), and 30 days (p = .01). For students who attended programs for at least 1 day, passing rates in grade 12 (but not in grades 10 or 11) were significantly higher (p < .01) for ASSETs participants than comparison students. Students in grades 10–12 who attended 10 or more days had significantly higher passing rates than other students (p = .05 for grades 10 and 11 and p = .01 for grade 12). Students in both grades 10 and 12 who attended afterschool for 30 days or more passed at significantly higher rates than non-attending students (p = .05 for grade 10 and p = .01 for grade 12).

Compared to nonparticipating English-learners’ CAHSEE ELA passing rates, ASSETs participants who were English learners passed the ELA test at a significantly higher rates amongst those that attended at least 1 day (p = .01), 10 days (p = .01), and 30 days (p = .05).

Participants who attended at least 1 day, 10 days, or 30 days were significantly more like to pass the CAHSEE math test than were nonparticipants (p = . 01 for each). Participants were significantly more likely to pass the math test than other students at grades 11 and 12 (p = .01 for each). At grades 10–12, ASSETs participants who attended 10 or more days and those who attended 30 or more days were significantly more likely to pass the math test than non-participating students (p = .01 for 10 days in grades 10 and 11, and 30 days for grade 11; p = .05 for 30 days in grade 10 and 12, and 10 days in grade 12).

Compared to nonparticipating English-learners’ CAHSEE math passing rates, ASSETs participants who were English learners were significantly more likely (p = .01) to pass this test, with the percent passing increasing as days of attendance increased.

On the 2006 CAHSEE, ASSETs participants who passed the ELA test averaged a significantly greater number of days attending programs than participants who did not pass, both overall (p = . 01) and at grade 11 (p = .03). Program participants who passed the math portion attended program activities a significantly greater number of days both overall (p = .02) and at grade 10 (p = .04) than participants who did not pass.

A greater percentage of participants scored at or above proficient on the ELA CST than nonparticipants at the same grade level at their schools. Differences between the two groups were small at grade 10 (1%) but larger at grades 9 and 11 (5% and 7%, respectively). However, these differences were not statistically significant.

At grades 9 and 10, youth who attended more program days were more likely to score at or above proficient on the ELA CST. At grade 11, however, the percent of youth scoring at or above proficient declined as they attended over 50 days. Statewide, there was no change from 2005 to 2006 at grades 9 and 11. For grade 10, this percent increased from 2005 to 2006 by 1%. Among Cohort 1 schools at each grade level, the change between 2005 and 2006 was identical for both program participants and nonparticipants. At grades 9 and 11, a greater percent of youth scored at or above proficient in 2006 than in 2005. Further, for grade 11 participants, the 2005 to 2006 change increased positively as youth attended 30 or more, 40 or more, and 50 or more days. At grade 10, in contrast, the 2005 to 2006 change tended to be more negative as youth attended a greater number of program days.

When asked how they benefited from the program, the majority of youth pointed out its academic benefits. They connected the skills they learned to immediate outcomes such as finishing homework, learning from another teacher, passing a class, and learning English.

Program participation helped some youth develop their English language skills. Youth surveys administered during one school’s English-as-Second-Language class revealed that all students who participated in the program reported learning English more quickly than their classmates.

At several sites, youth were able to make up missing credits by working with a credentialed teacher afterschool in credit reclamation courses. This approach allowed 22 students at one school to graduate, boosting the school’s graduation rate.
Youth Development Program staff felt that programs offered youth avenues to explore talents they otherwise would not have been able to explore. Furthermore, several programs pointed to the varied ways in which they helped prepare youth for life after high school.

Both youth and adults reported during site visits that youth benefited from relationships developed in programs. Youth said that their participation allowed them to expand their circle of friends since they got to know people they would not otherwise have met. They also felt that they learned key interpersonal skills like leadership, teamwork, trust, and communication. The adults observed that programs provided youth with opportunities to connect to caring adults at the school, which they saw as “vitally important” in keeping youth connected to the school and in a safe environment after school.

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project