Lodestar Management/Research, Inc.

Harder+Company Community Research

EVALCORP Research & Consulting

Annual Evaluation Report for 2003–04: Findings for Elementary School Programs

Annual Evaluation Report for 2003–04: Findings for Middle School Programs

Assessment of Program Quality and Youth Outcomes

Kaiser, M., & Lyons, M. (2001). Woodcraft Rangers: State of California After School Learning and Safe Neighborhoods Partnerships Program with the Los Angeles Unified School District. Annual evaluation report, 1999–2000. Los Angeles, CA: Author.

Lodestar Management/Research. (2002). Woodcraft Rangers: State of California After School Learning and Safe Neighborhoods Partnerships Program with the Los Angeles Unified School District. Annual evaluation report, 2000–01. Los Angeles, CA: Author.

Lodestar Management/Research. (2003). Woodcraft Rangers: Los Angeles Unified School District After School Education and Safety Program annual evaluation report 2001–02. Los Angeles, CA: Author.

Lodestar Management/Research (2004). Woodcraft Rangers: Los Angeles Unified School District After School Education and Safety Program annual evaluation report for 2002–03. Los Angeles, CA: Author.

Lodestar Management/Research (2005). Woodcraft Rangers: Annual evaluation report for 2003–04. Los Angeles, CA: Author.

Lodestar Management/Research. (2006). Woodcraft Rangers After-School Program: Summary of program youth outcomes for middle school sites 2004–05. Los Angeles, CA: Author.

Lodestar Management/Research (2006). Process evaluation report: Key factors related to program recruitment, retention, and outcomes. Los Angeles, CA: Author.

Lodestar Management/Research (2007). Woodcraft Rangers: Annual evaluation report for 2005–06. Los Angeles: Author.

Harder+Company Community Research. (2008). Woodcrafts Rangers annual evaluation report 2006–2007: Middle school programs. Los Angeles, CA: Woodcraft Rangers.

EVALCORP Research & Consulting. (2011). Assessment of program quality and youth outcomes: A study of the Woodcraft Rangers’ Nvision After-School Program. Irvine, CA: Author.

To answer the following questions:

• Whom is WR serving?
• Does WR increase youth engagement in afterschool activities?
• Does WR help schools to keep students safely occupied during afterschool hours?
• Are school administrators, parents, and participating youth satisfied with WR’s quality?
• Do youth who participate in WR attend school more regularly, develop prosocial interests and behaviors and avoid at-risk behaviors, improve their attitudes toward school and learning, improve their learning skills and habits, or have higher levels of academic achievement?

Quasi-Experimental and Non-Experimental: The evaluation included 11 of the 15 middle school programs. To examine implementation, at the end of the program year, 77 parents from all 11 schools participated in focus groups, and 15 school staff (school administrators and learning support staff) from 10 schools were interviewed.

Relationships between participation and outcomes were explored using two data sets. The first contained data on youth who participated in WR for at least 3 months to test the effects of youth’s program exposure (measured by days of program attendance) on outcomes. A total of 654 youth participants were surveyed at the beginning of the year (baseline) and 544 at the end of the year. WR staff also completed an assessment on 157 youth from baseline to first follow-up (3–6 months after enrollment), 241 youth from baseline to second follow-up (6–9 months after enrollment), and 46 from baseline to third follow-up (9–12 months after enrollment).

The second data set included a subset of youth who participated in WR for at least 6 months and a random sample of youth from the same schools who did not participate in WR. Demographic and academic data were collected for both groups. The comparison group was similar to participants in gender, grade, free/reduced-price lunch participation, and English learner status, but differed in ethnicity, with WR serving a significantly higher percentage of African American youth (p < .05). Analyses accounted for this difference by statistically controlling for this variable. Analyses also examined differences in academic progress between the 2002–03 and 2003–04 school years.

Interviews/Focus Groups: Focus groups with participants’ parents solicited perceptions of program benefits, satisfaction with their child’s club experience, and sense of youth safety.

Interviews with key school staff sought feedback on WR’s value to the school.

Secondary Source/Data Review: Gender, ethnicity, current school, grade level, and WR attendance data were collected from program records for all program youth in 2003–2004.

Demographic and academic data for the program and comparison groups for the 2002–03 and 2003–04 school years were collected from the Los Angeles Unified School District, including free/reduced-price lunch participation, language proficiency, participation in special education programs, spring semester school suspensions and absences, spring semester English as a Second Language (ESL) level, spring English and math grades, spring grade point average (GPA), and standardized academic test scores.

Surveys/Questionnaires: Youth surveys included items on school attitude, academic skills, sense of efficacy, problem-solving skills, and risk-taking behaviors. Baseline surveys also incorporated questions about youth’s prior participation in afterschool activities and their decision to participate in WR. End-of-year surveys added program satisfaction items.

Tests/Assessments: Evaluators examined test score data from the California Achievement Test/6th Edition (CAT6) Normal Curve Equivalent (NCE) for Reading/English and Math.

WR staff assessments of participants included items on youth’s peer interactions, attitude toward school, academic and social skills, and attitude toward risk behaviors.

On surveys, 93% of participants reported feeling safe “a lot” or “some” of the time when at WR. Similarly, almost all (99%) of focus group parents indicated feeling that their children were “very” or “somewhat” safe at WR.

During focus group discussions, children’s safety was one of the most common program benefits that parents cited. They appreciated that the program provided a safe place for their children to spend the afternoon.

A total of 2,407 middle school youth were served during the year (approximately 8% of the schools’ total student population, an average of 219 youth per school) for at least 5 days or more.

Youth participated in WR for an average of 5 months during the 2003–2004 year. Approximately 75% stayed for at least 3 months; one third stayed for almost a full school year (7–9 months) or the whole year (10–12 months).

For the 10 schools that had offered WR for more than 1 year, 18% of participants had also attended the previous year.

Participants were primarily Latino (83%) or African American (14%) and received free/ reduced-price lunch (96%). Three quarters had a native language other than English, with almost half (48%) identified as limited English proficient. Participants were almost evenly split between male (51%) and female (49%).

Participants included students from grades 6–8, with one third from each grade.

The top reasons given by surveyed youth for joining WR included “sounded fun” (47%), “wanted sports” (45%), and “friend was in it” (34%).

The majority of participants (68%) had not participated in any afterschool activities prior to joining WR. Another 24% had only participated in one activity prior to WR.

According to school administrator interviews, two of WR’s main impacts were increasing student participation in afterschool activities (all 10 agreed) and decreasing the number of students without supervision after school (9 out of 10 agreed).

More than half of participants indicated the highest level of satisfaction with WR in all areas. For example, 65% of participants indicated the staff were friendly and cared “a lot” and 60% said “I like going to the program.” Parents in focus groups were also satisfied with the program; a large majority indicated they were “very satisfied” with all areas, including space (91%), overall program quality (82%), opportunities for parents to participate (80%), activities (78%), staff (77%), and help with homework (72%).

More than half of participants maintained or improved their English and math scores on the CAT/6 from spring 2003 to spring 2004.

The more often youth participated in WR, the lower their math grades at the end of the year; this difference was significant (p < .05). When comparing WR participants to nonparticipants, there were no significant differences in math grades.

For CAT6 English and math scores, and spring English grades, there were no significant relationships between WR participation and change in these areas, either for different participation levels or between participants and comparison youth.

Most participants (73%) maintained or improved their math grade from spring 2003 to spring 2004 and two thirds (64%) maintained or improved their English grade from spring 2003 to spring 2004.

More than half (57%) of participants maintained or improved their GPA from spring 2003 to spring 2004. Controlling for spring 2003 GPA and key demographics, youth who participated more often in WR had higher spring 2004 GPAs than those who participated less often (p < .05). In addition, those who participated in WR for 6 or more months improved their GPA significantly more than the comparison group (50% vs. 44%, p < .05).

Slightly less than half (48%) of WR participants maintained or improved their school attendance from spring 2003 to spring 2004. In addition, analyses controlling for spring 2003 absences and key demographics indicated that youth who participated more often in WR had significantly fewer school absences in 2003–2004 than those who participated less often (p ≤ .05). While the average number of school absences increased between 2002–2003 and 2003–2004 for participants, it also increased for the comparison group. The increase was less substantial for WR 6-month-or-more participants (8.1–9.6 days) than for the comparison group (8.0–10.0 days), a difference that was marginally significant (p < 0.1).

The vast majority of participants did not have any school suspensions: 89% had no spring semester suspensions from 2002–2003 to 2003–2004. The average number of suspensions changed very little over the 2 years, from 0.17 days to 0.18 days.

Approximately 30% of participants surveyed improved on the relationship to school scale from baseline to end of year. However, the average rating decreased during that period. Youth who participated for more than 1 year or who had higher levels of involvement were significantly more likely to improve their school attitudes than youth who participated less often (p < .05).

WR staff observed little change in participants’ attitudes toward school: The average rating of this item increased from 2.6 at baseline to 2.7 at 6-month follow-up, which was not statistically significant.

Half of the school administrators surveyed indicated that WR positively impacted students’ attitudes toward school.

According to parents in focus groups, one main benefit of WR was improvement in their children’s academic skills. Many parents also noted that youth were more interested in school.

WR staff assessments indicated that 55% of participants improved in the academic skills scale at the 6-month follow-up (p < 0.5).

Seven of the 10 school staff interviewed reported that WR had a positive impact on students’ academic skills. Other key school staff members also commented on this impact.

Approximately one quarter of participants said they improved in reading for enjoyment (27%), finishing homework on time (25%), and doing a good job on homework (25%).

Among participants whose primary language was not English, 59% improved their ESL level, and another third maintained their fluency level (35%). The average ESL level of participants was 2.2 in spring 2003 and increased to 3.0 at follow-up (on a 5-point scale where 5 is most English proficient, p < .01).

WR staff reported that 57% of participants improved in prosocial skills/risk avoidance behaviors after 6 months of participation. More frequent participation in WR was associated with significantly greater improvements in this area (p < .05). Staff’s average ratings of participant development improved for all nine items of the scale and significant changes were found for three items: “expresses negative attitudes toward risk behaviors,” “approaches new tasks or projects with confidence,” and “demonstrates that he/she values others’ feelings and needs.”

Seven of the 10 interviewed school administrators indicated that the program had a positive impact on youth’s at-risk behaviors.

According to youth surveys, 35% of participants improved in the avoidance of risk-taking behaviors by the end of 2003–2004.

WR staff noted improvement in activity engagement for 54% of participants assessed at 3–5 months and for 50% at 6–8 months. The scale changes were significant between baseline and both of these time points (p ≤ .05). Of the six items that constitute the activity engagement scale, significant change was found between baseline and 6–8 months for “able to provide assistance to other children” and “comfortable in taking a leadership role in activities and projects.” Youth who participated more often in WR had significantly more improvement in their activity engagement level than those who participated less often (p ≤ .05).

According to all 10 school administrators interviewed, one of the main program impacts was that program participation improved youth’s social skills and peer relations.

According to youth surveys, approximately one third of participants improved their sense of efficacy (36%) and problem-solving skills (30%) by the end of the year. Further investigation indicated that youth who attended more often had significantly higher levels of efficacy; this relationship was significant (p < .05).

Many parents commented in focus groups that they saw a positive change in their children’s attitudes. They reported that youth were happier and more motivated.