Lodestar Management/Research, Inc.

Harder+Company Community Research

EVALCORP Research & Consulting

Annual Evaluation Report for 2003–04: Findings for Elementary School Programs

Annual Evaluation Report for 2003–04: Findings for Middle School Programs

Assessment of Program Quality and Youth Outcomes

Kaiser, M., & Lyons, M. (2001). Woodcraft Rangers: State of California After School Learning and Safe Neighborhoods Partnerships Program with the Los Angeles Unified School District. Annual evaluation report, 1999–2000. Los Angeles, CA: Author.

Lodestar Management/Research. (2002). Woodcraft Rangers: State of California After School Learning and Safe Neighborhoods Partnerships Program with the Los Angeles Unified School District. Annual evaluation report, 2000–01. Los Angeles, CA: Author.

Lodestar Management/Research. (2003). Woodcraft Rangers: Los Angeles Unified School District After School Education and Safety Program annual evaluation report 2001–02. Los Angeles, CA: Author.

Lodestar Management/Research (2004). Woodcraft Rangers: Los Angeles Unified School District After School Education and Safety Program annual evaluation report for 2002–03. Los Angeles, CA: Author.

Lodestar Management/Research (2005). Woodcraft Rangers: Annual evaluation report for 2003–04. Los Angeles, CA: Author.

Lodestar Management/Research. (2006). Woodcraft Rangers After-School Program: Summary of program youth outcomes for middle school sites 2004–05. Los Angeles, CA: Author.

Lodestar Management/Research (2006). Process evaluation report: Key factors related to program recruitment, retention, and outcomes. Los Angeles, CA: Author.

Lodestar Management/Research (2007). Woodcraft Rangers: Annual evaluation report for 2005–06. Los Angeles: Author.

Harder+Company Community Research. (2008). Woodcrafts Rangers annual evaluation report 2006–2007: Middle school programs. Los Angeles, CA: Woodcraft Rangers.

EVALCORP Research & Consulting. (2011). Assessment of program quality and youth outcomes: A study of the Woodcraft Rangers’ Nvision After-School Program. Irvine, CA: Author.

• Whom does WR serve?
• Does WR increase youth engagement in afterschool activities?
• Does WR help schools to keep students safely occupied during afterschool hours?
• Are school administrators, parents, and participating youth satisfied with WR’s quality?
• Do youth who participate in WR attend school more regularly, develop prosocial interests and behaviors and avoid at-risk behaviors, improve their attitudes toward school and learning, improve their learning skills and habits, and have higher academic achievement levels?

Quasi-Experimental and Non-Experimental: The evaluation included all 14 elementary school programs. To examine implementation, at the end of the program year, 105 parents of participants from 13 of the 14 schools participated in focus groups, and 11 staff members (school administrators and learning support staff) from 10 of the 14 schools were interviewed.

Relationships between participation and outcomes were explored using two data sets. The first contained data on youth who participated in WR for at least 3 months to examine the effects of program exposure (measured by days of program attendance) on outcomes.

A total of 1,033 program youth were surveyed at the beginning of the year (baseline) and 876 of these also completed a survey at the end of the year. WR staff also completed an assessment on 151 WR participants from baseline to first follow-up (3–6 months after enrollment), 433 youth from baseline to second follow-up (6–9 months after enrollment), and 55 youth from baseline to third follow-up (9–12 months after enrollment).

The second data set included a subset of WR participants who participated for at least 6 months, and a comparison group composed of a random sample of youth from the same schools who did not participate in WR. Demographic and academic data were collected for both groups. The comparison group was ethnically similar to WR participants, but fewer WR participants received free/reduced-price meals and were identified as limited English proficient than in the comparison group (p < .05 for each). In addition, participants were somewhat younger than comparison youth (p < .05). Analyses accounted for these differences by controlling for demographic variables. Analyses also examined differences in academic progress between the 2002–03 and 2003–04 school years.

Interviews/Focus Groups: Focus groups with participants’ parents solicited perceptions of program benefits, satisfaction with their child’s club experience, and sense of youth safety.

Interviews with school administrators sought feedback on WR’s value to the school.

Secondary Source/Data Review: Gender, ethnicity, current school, grade level, and WR attendance data were collected from program records for all program youth in 2003–2004.

Demographic and academic data were collected for program and comparison group youth for the 2002–03 and 2003–04 school years from the Los Angeles Unified School District, including free/reduced-price lunch participation, English language proficiency, participation in special education programs, spring semester school suspensions and absences, spring semester English as a Second Language (ESL) level, spring English and math grades, and standardized academic test scores.

Surveys/Questionnaires: Youth surveys included items on school attitude, academic skills, sense of efficacy, problem-solving skills, and risk-taking behaviors. Baseline surveys also incorporated questions about youth’s prior participation in afterschool activities and their decision to participate in WR. End-of-year surveys added program satisfaction items.

Test/Assessments: Evaluators used test score data from the California Achievement Test/6th Edition (CAT6) Normal Curve Equivalent (NCE) for Reading/English and Math.

WR staff assessments of participants included items on youth’s peer interaction, attitude toward school, academic and social skills, and attitude toward risk behaviors.

WR elementary sites served 3,574 youth in 2003–2004 for at least 5 days or more.

WR served approximately one quarter of the elementary schools’ total student populations, with an average of 255 students per site.

Youth participated in WR for an average of 6 months (two thirds of the school year) during 2003–2004, a slightly higher average than the previous year (5 months). More than three quarters of participants stayed in WR for at least 3 months, while 42% stayed for almost a full school year (7–9 months) or the whole year (10–12 months).

For the seven schools that had offered WR for more than 1 year, 29% (588) of 2003–2004 participants had also attended the previous year.

WR participants were primarily Latino (91%), received free/reduced-price lunch (97%), and had a native language other than English (72%). A slight majority were female (52%). A higher percentage were African American than in previous years (7% vs. less than 1%). Participants included students from grades K–6, with most from grades 3 and 4.

Almost half of participants surveyed who joined WR for the first time in 2003–2004 said they joined because their parents signed them up (45%). Other commonly cited reasons included “sounded fun” (35%) and “homework help” (28%).

Most participants (73%) had not participated in afterschool activities prior to WR.

According to school staff interviews, WR’s two main impacts were on decreasing the number of students without supervision after school (all 11 staff agreed) and increasing student participation in afterschool activities (10 of the 11 staff agreed).

More than three quarters of 3-month participants maintained or improved their reading and math grades from spring 2003 to spring 2004.

Almost half of participants maintained or improved their CAT/6 score in reading and math from spring 2003 to spring 2004.

Youth who participated more often in WR outperformed those who participated less often on changes in math and reading grades and changes in CAT6 math and reading scores (p < .05 for math grades, reading grades, and CAT6 math scores; p < .10 for CAT6 reading scores). No significant results were found when similar analyses were conducted to compare the 6-month-or-more sample with the nonparticipant comparison group.

WR participants were absent from school an average of 2 days in both the 2002–03 and 2003–04 school years.

The majority of participants surveyed in 2003–2004 (82%) reported that they never skipped school. The average rating from baseline to follow-up showed an increase in students reporting this behavior, although it was not significant.

Participants’ average survey rating for how often they wanted to go to school was very high at baseline (4.68 on 5-point scale) and stayed almost the same at end-of-year follow-up (4.63). At the end of the year, a majority of youth reported that they wanted to go to school “every day” (78%).

The vast majority of participants did not have any school suspensions. Almost all (98%) had no spring semester suspensions in either 2002–03 or 2003–04.

Participants’ average survey rating of “I liked being at school” decreased from 4.5 at baseline to 4.4 at the end of the year (p < .05), on a 5-point scale (where 5 is the most positive). However, higher levels of WR attendance were significantly correlated with more positive change in this school attitude rating (p ≤ .05).

Average ratings on the survey’s “youth relationship to school” scale decreased significantly from baseline to follow-up (from 4.4 to 4.2 on a 5-point scale where 5 is the most positive; p < .05).

About 30% of participants surveyed improved in their ratings of “I finished all my homework on time” and “I did a good job on my homework” between baseline and follow-up.

Over one third of the participants (37%) indicated they read for enjoyment more often at the spring follow-up than before they started the program.

WR staff observed significant improvements for youth between baseline and 6-month follow-up in the areas of “talks about own future in school and career” and “verbalizes a positive attitude toward school and learning” (p < .001 for each). On 5-point scale where 5 is the most frequent, average ratings of this item increased from 1.9 to 2.4 for the first item and 2.4 to 2.7 for the second.

WR staff assessments of youth indicated that 56% of participants improved in the academic skills scale by the time of the 6-month follow-up.

In focus groups, parents most often commented that the major program benefit was their children’s homework completion.

Three quarters of school staff interviewed believed that WR positively impacted students’ attitudes toward school and academic skills (e.g., homework completion, language acquisition). Most school staff discussed the helpful role WR played in assisting youth with homework.

Among participants whose primary language was not English, more than half (56%) improved their ESL level, and the remaining 44% maintained their fluency level. The average ESL level of participants increased from 2.5 in 2002–2003 to 3.2 in 2003–2004 (on a 5-point scale where 5 is most English proficient).

During focus groups, several parents noted that a benefit of WR was that their children communicated more often with them about school (e.g., about homework and looking forward to school).

In focus groups, parents said that WR helped them become more involved in their children’s education, for example through volunteering for the program. Parents also said that they were more aware of school activities and ways to volunteer at the school.

Three quarters of school staff interviewed reported that WR helped increase youth’s sense of safety at school.

WR staff reported that 62% of participants improved their prosocial skills/risk avoidance behaviors after 6 months. Significant changes were found for all nine items of the scale, with most substantial improvement for the following items: “expresses negative attitudes toward risk behaviors,” “forms friendships with prosocial peers,” and “seeks understanding of peers’ family traditions, practices, etc.”

The majority of participants surveyed did not engage in risk-taking behaviors. For example, at follow-up, 92% said they “never” did things that they knew weren’t good for them and 66% said they “never” hung around with kids who get into trouble. However, risky behaviors did significantly increase from baseline to follow-up (p < .05).

WR staff noted improvement in activity engagement for 57% of participants assessed at 3–5 months and for 61% at 6–8 months. Significant changes (p < .05) were found between baseline and the 6–8 month assessment for all six items of the activity engagement scale. Specific items in which staff noted most change at 6–8 months included “comfortable in taking a leadership role in activities and projects,” “plans or joins with others to do new things,” and “approaches new tasks or projects with confidence.”

According to youth surveys, nearly half of participants improved their sense of efficacy (49%) and problem-solving skills (45%). Of the items that constitute these two scales, the most improvement was seen for “I tried new activities,” “I finished a project on my own,” and “I solved a problem I was having with another student without arguing/fighting.”

According to school staff interviews, one of the main impacts of WR was to improve students’ social skills and peer relations; three quarters cited this as a positive impact.

Parents in the focus groups commonly indicated that the program helped their children to overcome shyness.