Lodestar Management/Research, Inc.

Harder+Company Community Research

EVALCORP Research & Consulting

Annual Evaluation Report for 2003–04: Findings for Elementary School Programs

Annual Evaluation Report for 2003–04: Findings for Middle School Programs

Assessment of Program Quality and Youth Outcomes

Kaiser, M., & Lyons, M. (2001). Woodcraft Rangers: State of California After School Learning and Safe Neighborhoods Partnerships Program with the Los Angeles Unified School District. Annual evaluation report, 1999–2000. Los Angeles, CA: Author.

Lodestar Management/Research. (2002). Woodcraft Rangers: State of California After School Learning and Safe Neighborhoods Partnerships Program with the Los Angeles Unified School District. Annual evaluation report, 2000–01. Los Angeles, CA: Author.

Lodestar Management/Research. (2003). Woodcraft Rangers: Los Angeles Unified School District After School Education and Safety Program annual evaluation report 2001–02. Los Angeles, CA: Author.

Lodestar Management/Research (2004). Woodcraft Rangers: Los Angeles Unified School District After School Education and Safety Program annual evaluation report for 2002–03. Los Angeles, CA: Author.

Lodestar Management/Research (2005). Woodcraft Rangers: Annual evaluation report for 2003–04. Los Angeles, CA: Author.

Lodestar Management/Research. (2006). Woodcraft Rangers After-School Program: Summary of program youth outcomes for middle school sites 2004–05. Los Angeles, CA: Author.

Lodestar Management/Research (2006). Process evaluation report: Key factors related to program recruitment, retention, and outcomes. Los Angeles, CA: Author.

Lodestar Management/Research (2007). Woodcraft Rangers: Annual evaluation report for 2005–06. Los Angeles: Author.

Harder+Company Community Research. (2008). Woodcrafts Rangers annual evaluation report 2006–2007: Middle school programs. Los Angeles, CA: Woodcraft Rangers.

EVALCORP Research & Consulting. (2011). Assessment of program quality and youth outcomes: A study of the Woodcraft Rangers’ Nvision After-School Program. Irvine, CA: Author.

• Whom does WR serve?
• Does WR increase youth engagement in afterschool activities?
• Does WR help schools to keep students safely occupied during afterschool hours?
• Are school administrators, parents, and participating youth satisfied with WR’s quality?
• Do youth who participate in WR attend school more regularly, develop prosocial interests and behaviors and avoid at-risk behaviors, improve their attitudes toward school and learning, improve their learning skills and habits, and have higher academic achievement levels?

Quasi-Experimental and Non-Experimental: The evaluation included all 14 elementary school programs. To examine implementation, at the end of the program year, 105 parents of participants from 13 of the 14 schools participated in focus groups, and 11 staff members (school administrators and learning support staff) from 10 of the 14 schools were interviewed.

Relationships between participation and outcomes were explored using two data sets. The first contained data on youth who participated in WR for at least 3 months to examine the effects of program exposure (measured by days of program attendance) on outcomes.

A total of 1,033 program youth were surveyed at the beginning of the year (baseline) and 876 of these also completed a survey at the end of the year. WR staff also completed an assessment on 151 WR participants from baseline to first follow-up (3–6 months after enrollment), 433 youth from baseline to second follow-up (6–9 months after enrollment), and 55 youth from baseline to third follow-up (9–12 months after enrollment).

The second data set included a subset of WR participants who participated for at least 6 months, and a comparison group composed of a random sample of youth from the same schools who did not participate in WR. Demographic and academic data were collected for both groups. The comparison group was ethnically similar to WR participants, but fewer WR participants received free/reduced-price meals and were identified as limited English proficient than in the comparison group (p < .05 for each). In addition, participants were somewhat younger than comparison youth (p < .05). Analyses accounted for these differences by controlling for demographic variables. Analyses also examined differences in academic progress between the 2002–03 and 2003–04 school years.

Interviews/Focus Groups: Focus groups with participants’ parents solicited perceptions of program benefits, satisfaction with their child’s club experience, and sense of youth safety.

Interviews with school administrators sought feedback on WR’s value to the school.

Secondary Source/Data Review: Gender, ethnicity, current school, grade level, and WR attendance data were collected from program records for all program youth in 2003–2004.

Demographic and academic data were collected for program and comparison group youth for the 2002–03 and 2003–04 school years from the Los Angeles Unified School District, including free/reduced-price lunch participation, English language proficiency, participation in special education programs, spring semester school suspensions and absences, spring semester English as a Second Language (ESL) level, spring English and math grades, and standardized academic test scores.

Surveys/Questionnaires: Youth surveys included items on school attitude, academic skills, sense of efficacy, problem-solving skills, and risk-taking behaviors. Baseline surveys also incorporated questions about youth’s prior participation in afterschool activities and their decision to participate in WR. End-of-year surveys added program satisfaction items.

Test/Assessments: Evaluators used test score data from the California Achievement Test/6th Edition (CAT6) Normal Curve Equivalent (NCE) for Reading/English and Math.

WR staff assessments of participants included items on youth’s peer interaction, attitude toward school, academic and social skills, and attitude toward risk behaviors.

WR elementary sites served 3,574 youth in 2003–2004 for at least 5 days or more.

WR served approximately one quarter of the elementary schools’ total student populations, with an average of 255 students per site.

Youth participated in WR for an average of 6 months (two thirds of the school year) during 2003–2004, a slightly higher average than the previous year (5 months). More than three quarters of participants stayed in WR for at least 3 months, while 42% stayed for almost a full school year (7–9 months) or the whole year (10–12 months).

For the seven schools that had offered WR for more than 1 year, 29% (588) of 2003–2004 participants had also attended the previous year.

WR participants were primarily Latino (91%), received free/reduced-price lunch (97%), and had a native language other than English (72%). A slight majority were female (52%). A higher percentage were African American than in previous years (7% vs. less than 1%). Participants included students from grades K–6, with most from grades 3 and 4.

Almost half of participants surveyed who joined WR for the first time in 2003–2004 said they joined because their parents signed them up (45%). Other commonly cited reasons included “sounded fun” (35%) and “homework help” (28%).

Most participants (73%) had not participated in afterschool activities prior to WR.

According to school staff interviews, WR’s two main impacts were on decreasing the number of students without supervision after school (all 11 staff agreed) and increasing student participation in afterschool activities (10 of the 11 staff agreed).

More than three quarters of 3-month participants maintained or improved their reading and math grades from spring 2003 to spring 2004.

Almost half of participants maintained or improved their CAT/6 score in reading and math from spring 2003 to spring 2004.

Youth who participated more often in WR outperformed those who participated less often on changes in math and reading grades and changes in CAT6 math and reading scores (p < .05 for math grades, reading grades, and CAT6 math scores; p < .10 for CAT6 reading scores). No significant results were found when similar analyses were conducted to compare the 6-month-or-more sample with the nonparticipant comparison group.

WR participants were absent from school an average of 2 days in both the 2002–03 and 2003–04 school years.

The majority of participants surveyed in 2003–2004 (82%) reported that they never skipped school. The average rating from baseline to follow-up showed an increase in students reporting this behavior, although it was not significant.

Participants’ average survey rating for how often they wanted to go to school was very high at baseline (4.68 on 5-point scale) and stayed almost the same at end-of-year follow-up (4.63). At the end of the year, a majority of youth reported that they wanted to go to school “every day” (78%).

The vast majority of participants did not have any school suspensions. Almost all (98%) had no spring semester suspensions in either 2002–03 or 2003–04.

Participants’ average survey rating of “I liked being at school” decreased from 4.5 at baseline to 4.4 at the end of the year (p < .05), on a 5-point scale (where 5 is the most positive). However, higher levels of WR attendance were significantly correlated with more positive change in this school attitude rating (p ≤ .05).

Average ratings on the survey’s “youth relationship to school” scale decreased significantly from baseline to follow-up (from 4.4 to 4.2 on a 5-point scale where 5 is the most positive; p < .05).

About 30% of participants surveyed improved in their ratings of “I finished all my homework on time” and “I did a good job on my homework” between baseline and follow-up.

Over one third of the participants (37%) indicated they read for enjoyment more often at the spring follow-up than before they started the program.

WR staff observed significant improvements for youth between baseline and 6-month follow-up in the areas of “talks about own future in school and career” and “verbalizes a positive attitude toward school and learning” (p < .001 for each). On 5-point scale where 5 is the most frequent, average ratings of this item increased from 1.9 to 2.4 for the first item and 2.4 to 2.7 for the second.

WR staff assessments of youth indicated that 56% of participants improved in the academic skills scale by the time of the 6-month follow-up.

In focus groups, parents most often commented that the major program benefit was their children’s homework completion.

Three quarters of school staff interviewed believed that WR positively impacted students’ attitudes toward school and academic skills (e.g., homework completion, language acquisition). Most school staff discussed the helpful role WR played in assisting youth with homework.

Among participants whose primary language was not English, more than half (56%) improved their ESL level, and the remaining 44% maintained their fluency level. The average ESL level of participants increased from 2.5 in 2002–2003 to 3.2 in 2003–2004 (on a 5-point scale where 5 is most English proficient).

During focus groups, several parents noted that a benefit of WR was that their children communicated more often with them about school (e.g., about homework and looking forward to school).

In focus groups, parents said that WR helped them become more involved in their children’s education, for example through volunteering for the program. Parents also said that they were more aware of school activities and ways to volunteer at the school.

Three quarters of school staff interviewed reported that WR helped increase youth’s sense of safety at school.

WR staff reported that 62% of participants improved their prosocial skills/risk avoidance behaviors after 6 months. Significant changes were found for all nine items of the scale, with most substantial improvement for the following items: “expresses negative attitudes toward risk behaviors,” “forms friendships with prosocial peers,” and “seeks understanding of peers’ family traditions, practices, etc.”

The majority of participants surveyed did not engage in risk-taking behaviors. For example, at follow-up, 92% said they “never” did things that they knew weren’t good for them and 66% said they “never” hung around with kids who get into trouble. However, risky behaviors did significantly increase from baseline to follow-up (p < .05).

WR staff noted improvement in activity engagement for 57% of participants assessed at 3–5 months and for 61% at 6–8 months. Significant changes (p < .05) were found between baseline and the 6–8 month assessment for all six items of the activity engagement scale. Specific items in which staff noted most change at 6–8 months included “comfortable in taking a leadership role in activities and projects,” “plans or joins with others to do new things,” and “approaches new tasks or projects with confidence.”

According to youth surveys, nearly half of participants improved their sense of efficacy (49%) and problem-solving skills (45%). Of the items that constitute these two scales, the most improvement was seen for “I tried new activities,” “I finished a project on my own,” and “I solved a problem I was having with another student without arguing/fighting.”

According to school staff interviews, one of the main impacts of WR was to improve students’ social skills and peer relations; three quarters cited this as a positive impact.

Parents in the focus groups commonly indicated that the program helped their children to overcome shyness.

To answer the following questions:

• Whom is WR serving?
• Does WR increase youth engagement in afterschool activities?
• Does WR help schools to keep students safely occupied during afterschool hours?
• Are school administrators, parents, and participating youth satisfied with WR’s quality?
• Do youth who participate in WR attend school more regularly, develop prosocial interests and behaviors and avoid at-risk behaviors, improve their attitudes toward school and learning, improve their learning skills and habits, or have higher levels of academic achievement?

Quasi-Experimental and Non-Experimental: The evaluation included 11 of the 15 middle school programs. To examine implementation, at the end of the program year, 77 parents from all 11 schools participated in focus groups, and 15 school staff (school administrators and learning support staff) from 10 schools were interviewed.

Relationships between participation and outcomes were explored using two data sets. The first contained data on youth who participated in WR for at least 3 months to test the effects of youth’s program exposure (measured by days of program attendance) on outcomes. A total of 654 youth participants were surveyed at the beginning of the year (baseline) and 544 at the end of the year. WR staff also completed an assessment on 157 youth from baseline to first follow-up (3–6 months after enrollment), 241 youth from baseline to second follow-up (6–9 months after enrollment), and 46 from baseline to third follow-up (9–12 months after enrollment).

The second data set included a subset of youth who participated in WR for at least 6 months and a random sample of youth from the same schools who did not participate in WR. Demographic and academic data were collected for both groups. The comparison group was similar to participants in gender, grade, free/reduced-price lunch participation, and English learner status, but differed in ethnicity, with WR serving a significantly higher percentage of African American youth (p < .05). Analyses accounted for this difference by statistically controlling for this variable. Analyses also examined differences in academic progress between the 2002–03 and 2003–04 school years.

Interviews/Focus Groups: Focus groups with participants’ parents solicited perceptions of program benefits, satisfaction with their child’s club experience, and sense of youth safety.

Interviews with key school staff sought feedback on WR’s value to the school.

Secondary Source/Data Review: Gender, ethnicity, current school, grade level, and WR attendance data were collected from program records for all program youth in 2003–2004.

Demographic and academic data for the program and comparison groups for the 2002–03 and 2003–04 school years were collected from the Los Angeles Unified School District, including free/reduced-price lunch participation, language proficiency, participation in special education programs, spring semester school suspensions and absences, spring semester English as a Second Language (ESL) level, spring English and math grades, spring grade point average (GPA), and standardized academic test scores.

Surveys/Questionnaires: Youth surveys included items on school attitude, academic skills, sense of efficacy, problem-solving skills, and risk-taking behaviors. Baseline surveys also incorporated questions about youth’s prior participation in afterschool activities and their decision to participate in WR. End-of-year surveys added program satisfaction items.

Tests/Assessments: Evaluators examined test score data from the California Achievement Test/6th Edition (CAT6) Normal Curve Equivalent (NCE) for Reading/English and Math.

WR staff assessments of participants included items on youth’s peer interactions, attitude toward school, academic and social skills, and attitude toward risk behaviors.

On surveys, 93% of participants reported feeling safe “a lot” or “some” of the time when at WR. Similarly, almost all (99%) of focus group parents indicated feeling that their children were “very” or “somewhat” safe at WR.

During focus group discussions, children’s safety was one of the most common program benefits that parents cited. They appreciated that the program provided a safe place for their children to spend the afternoon.

A total of 2,407 middle school youth were served during the year (approximately 8% of the schools’ total student population, an average of 219 youth per school) for at least 5 days or more.

Youth participated in WR for an average of 5 months during the 2003–2004 year. Approximately 75% stayed for at least 3 months; one third stayed for almost a full school year (7–9 months) or the whole year (10–12 months).

For the 10 schools that had offered WR for more than 1 year, 18% of participants had also attended the previous year.

Participants were primarily Latino (83%) or African American (14%) and received free/ reduced-price lunch (96%). Three quarters had a native language other than English, with almost half (48%) identified as limited English proficient. Participants were almost evenly split between male (51%) and female (49%).

Participants included students from grades 6–8, with one third from each grade.

The top reasons given by surveyed youth for joining WR included “sounded fun” (47%), “wanted sports” (45%), and “friend was in it” (34%).

The majority of participants (68%) had not participated in any afterschool activities prior to joining WR. Another 24% had only participated in one activity prior to WR.

According to school administrator interviews, two of WR’s main impacts were increasing student participation in afterschool activities (all 10 agreed) and decreasing the number of students without supervision after school (9 out of 10 agreed).

More than half of participants indicated the highest level of satisfaction with WR in all areas. For example, 65% of participants indicated the staff were friendly and cared “a lot” and 60% said “I like going to the program.” Parents in focus groups were also satisfied with the program; a large majority indicated they were “very satisfied” with all areas, including space (91%), overall program quality (82%), opportunities for parents to participate (80%), activities (78%), staff (77%), and help with homework (72%).

More than half of participants maintained or improved their English and math scores on the CAT/6 from spring 2003 to spring 2004.

The more often youth participated in WR, the lower their math grades at the end of the year; this difference was significant (p < .05). When comparing WR participants to nonparticipants, there were no significant differences in math grades.

For CAT6 English and math scores, and spring English grades, there were no significant relationships between WR participation and change in these areas, either for different participation levels or between participants and comparison youth.

Most participants (73%) maintained or improved their math grade from spring 2003 to spring 2004 and two thirds (64%) maintained or improved their English grade from spring 2003 to spring 2004.

More than half (57%) of participants maintained or improved their GPA from spring 2003 to spring 2004. Controlling for spring 2003 GPA and key demographics, youth who participated more often in WR had higher spring 2004 GPAs than those who participated less often (p < .05). In addition, those who participated in WR for 6 or more months improved their GPA significantly more than the comparison group (50% vs. 44%, p < .05).

Slightly less than half (48%) of WR participants maintained or improved their school attendance from spring 2003 to spring 2004. In addition, analyses controlling for spring 2003 absences and key demographics indicated that youth who participated more often in WR had significantly fewer school absences in 2003–2004 than those who participated less often (p ≤ .05). While the average number of school absences increased between 2002–2003 and 2003–2004 for participants, it also increased for the comparison group. The increase was less substantial for WR 6-month-or-more participants (8.1–9.6 days) than for the comparison group (8.0–10.0 days), a difference that was marginally significant (p < 0.1).

The vast majority of participants did not have any school suspensions: 89% had no spring semester suspensions from 2002–2003 to 2003–2004. The average number of suspensions changed very little over the 2 years, from 0.17 days to 0.18 days.

Approximately 30% of participants surveyed improved on the relationship to school scale from baseline to end of year. However, the average rating decreased during that period. Youth who participated for more than 1 year or who had higher levels of involvement were significantly more likely to improve their school attitudes than youth who participated less often (p < .05).

WR staff observed little change in participants’ attitudes toward school: The average rating of this item increased from 2.6 at baseline to 2.7 at 6-month follow-up, which was not statistically significant.

Half of the school administrators surveyed indicated that WR positively impacted students’ attitudes toward school.

According to parents in focus groups, one main benefit of WR was improvement in their children’s academic skills. Many parents also noted that youth were more interested in school.

WR staff assessments indicated that 55% of participants improved in the academic skills scale at the 6-month follow-up (p < 0.5).

Seven of the 10 school staff interviewed reported that WR had a positive impact on students’ academic skills. Other key school staff members also commented on this impact.

Approximately one quarter of participants said they improved in reading for enjoyment (27%), finishing homework on time (25%), and doing a good job on homework (25%).

Among participants whose primary language was not English, 59% improved their ESL level, and another third maintained their fluency level (35%). The average ESL level of participants was 2.2 in spring 2003 and increased to 3.0 at follow-up (on a 5-point scale where 5 is most English proficient, p < .01).

WR staff reported that 57% of participants improved in prosocial skills/risk avoidance behaviors after 6 months of participation. More frequent participation in WR was associated with significantly greater improvements in this area (p < .05). Staff’s average ratings of participant development improved for all nine items of the scale and significant changes were found for three items: “expresses negative attitudes toward risk behaviors,” “approaches new tasks or projects with confidence,” and “demonstrates that he/she values others’ feelings and needs.”

Seven of the 10 interviewed school administrators indicated that the program had a positive impact on youth’s at-risk behaviors.

According to youth surveys, 35% of participants improved in the avoidance of risk-taking behaviors by the end of 2003–2004.

WR staff noted improvement in activity engagement for 54% of participants assessed at 3–5 months and for 50% at 6–8 months. The scale changes were significant between baseline and both of these time points (p ≤ .05). Of the six items that constitute the activity engagement scale, significant change was found between baseline and 6–8 months for “able to provide assistance to other children” and “comfortable in taking a leadership role in activities and projects.” Youth who participated more often in WR had significantly more improvement in their activity engagement level than those who participated less often (p ≤ .05).

According to all 10 school administrators interviewed, one of the main program impacts was that program participation improved youth’s social skills and peer relations.

According to youth surveys, approximately one third of participants improved their sense of efficacy (36%) and problem-solving skills (30%) by the end of the year. Further investigation indicated that youth who attended more often had significantly higher levels of efficacy; this relationship was significant (p < .05).

Many parents commented in focus groups that they saw a positive change in their children’s attitudes. They reported that youth were happier and more motivated.

Quasi-Experimental and Non-Experimental: Program data were collected from all WR sites.

WR site coordinators were surveyed to identify key components of program implementation. A total of 55 of the 57 elementary and middle school site coordinators completed the survey (42 of 43 elementary sites; 13 of 14 middle school sites). From these survey data, two different methods, a factor analysis approach and a benchmark approach, were used to identify key components. The factor analysis approach used statistical methods to identify quality indicators. The benchmark approach used ratings from site coordinator surveys to develop a benchmark or minimum criteria for a “high quality” site response. For each item, if the site’s actual response met or exceeded the management team’s benchmark, a point for that item was assigned. If a benchmark category contained five items, and the site met the high-quality criteria for four of them, the site's score would be 80% on that benchmark (i.e., 4 divided by 5).

The program quality factors identified in the factor analysis included:

1. Core elements—the extent that the program had a collaborative relationship with school administrators, how often it followed a site schedule, and the extent that youth were involved in decisions that impacted the program design
2. Cycle plans—the extent the program implemented cycle plans (i.e., descriptions of goals and objectives of program activities), how useful the cycle plans were during the implementation of club activities, and the extent that the cycle plans’ objectives were met
3. Value of ad hoc assistance—the value provided to staff from WR parent volunteers, youth volunteers, and WR traveling specialists (i.e., professionals who run clubs related to their field of expertise)
4. Connections—how many activities promoted youth involvement in the community, how involved parents were in the program, and how much access the program had to school facilities to implement program activities
5. Educational supports—the value provided to staff from WR teacher liaisons and school personnel and the extent that the program used academic concepts in activities.

Only the first three factors, however, were retained when relating quality factors to youth outcomes, as connections and educational supports did not have sufficient reliability to be considered stable factors using the current survey measurement tool.

The quality indicators gleaned from the benchmark analysis were:

1. Cycle plans—the extent to which cycle plans (i.e., descriptions of goals and objectives of program activities) were used and useful in activity implementation
2. Time distribution—students involved for expected number of minutes in required components of homework, fitness, nutrition/snack, interest-based club activities, and closing activities/all-together time
3. Club activities—activities offered in six areas: sports, performing arts, visual arts, recreation activities, computers or technology/multi-media, and leadership opportunities/youth development
4. Student engagement—extent of youth engagement in additional youth development activities
5. Club contributors—who is involved in club activities, including teacher liaisons, activity consultants, traveling specialists, and school personnel
6. Ad hoc staff value—the extent to which club contributors have appropriately important roles
7. Club selection—whether staff help to select club activities; “staff” includes regional managers, site coordinators, club leaders, youth councils, youth participants, and principals/school administrators
8. Student involvement—the extent of choice and leadership opportunities in club activities
9. Parent involvement—extent that parents are involved in club activities, including communication with staff and attending events and activities
10. School collaboration—the extent to which the clubs work with schools, including engaging with school staff in specific communications and activities
11. Facilities access—how much access the program had to school facilities to implement program activities
12. Site coordinator qualities—characteristics of site coordinators, including management style, organizational skills, and staff communication
13. Program staff qualities—characteristics of program staff, including leadership, organization, and student interactions.

To examine youth outcomes, WR participants were administered a survey when they first joined the program (baseline) and again at the end of the program year. In total, 2,304 elementary youth and 406 middle-school youth had matched baseline and year-end surveys. Academic test score data were also collected on participants for the 2007–08 and 2008–09 school years; only those youth with data from both years (N = approximately 3,300 elementary school youth and 4,900 middle school youth) were included in analyses. Lastly, school attendance data were collected for approximately 4,700 youth in both the elementary and middle school samples.

Quality indicators were examined to see whether and how they were related to youth outcomes. Two of the quality factors identified through factor analysis (connections and educational supports) were not included in the quality-outcome analysis because they did not have strong enough internal consistency to be considered stable enough to use in that analysis. In addition, two other factors—program attendance days and demographics of the local communities in which WR sites were located—were used to account for other contributions to youth outcomes beyond quality. Analyses were performed separately for elementary and middle schools.

Secondary Source/Data Review: School attendance data for the 2007–08 and 2008–09 school years were collected from the district.

Program attendance data for the 2008–09 program year were collected from the WR data system.

Demographic data from the U.S. Census 2000 for each school region was collected on the communities in which the WR schools were located. Indicators used include median family income as well as percentages of families in poverty, persons (aged 16 and older) in the labor force, persons with high school degree or greater; and persons who speak a language other than English in their home.

Surveys/Questionnaires: The site coordinator survey consisted of more than 150 quantitative and qualitative questions about how the site operated. Questions focused on how the site operated in terms of staffing, site structure and activities, daily club activities, youth engagement and leadership, school collaboration, parent involvement, and community involvement.

Youth surveys measured changes in attitudes, academic and social skills, sense of efficacy, and risk-related activities. The end-year surveys also included questions about youth’s experience and satisfaction with the program.

Tests/Assessments: Academic data included scores on the California Standards Test (CST) in English/Language Arts (ELA) and in Mathematics. Proficiency scores on the CST range from 1 (Far Below Basic) to 5 (Advanced).

For the factor analysis indicators, sites’ average scores were on the high end of the possible range for two of the five indicators: core elements and cycle plans (10.9 out of 12 and 10 out of 11, respectively).

There were no significant differences in the quality factor scores between programs at elementary schools and programs at middle schools.

According to the benchmark indicators—which represent the degree to which sites were able to reach the quality benchmark, with the goal to reach 100% of the benchmark—quality tended to be high (i.e., reaching a 75% or higher quality rating) across sites in 3 of the 13 benchmark areas for both elementary and middle school programs: site coordinator qualities, school collaboration, and program staff qualities. In addition, the majority of elementary sites reached at least a 75% quality rating in cycle plans and student engagement, while the majority of middle school sites reached at least a 75% quality rating in club activities and club contributors.

According to the benchmark indicators, the range of quality across sites was quite large for almost all the areas. For example, the average quality score for student involvement was high at 74% for elementary sites, yet this score ranged from as low as 20% to the highest possible score of 100%.

Only two benchmark indicators had quality scores that were consistently lower than the WR model’s expectations. For time distribution, the average percentage of the benchmark reached was 44% for elementary sites and 38% for middle school sites. This indicates that most sites did not spend the same amount of time on each of the daily required components as expected by the WR model. For ad hoc staff value, the average percentage of the benchmark reached was 46% in elementary sites and 60% in middle school sites.

Average overall benchmark quality scores ranged from 60–74% for elementary school sites, and 48–80% at middle school sites. For elementary sites, the overall quality average was 67%. For middle schools, the overall quality average was 68%.

Of the 16 quality indicators included in the outcome analysis, 5 had relationships that were favorable with a subset of the youth outcomes in elementary sites: value of ad hoc assistance, student engagement, club contributors, access to facilities, and site coordinator qualities. This indicates that when program quality was higher in each of these areas, the more positive were at least one or more of the desired youth outcome results. Of the remaining 11 indicators, 7 had a combination of negative and positive relationships with outcomes (time distribution, club activities, ad hoc value, club selection, student involvement, school collaboration, program staff qualities); 3 had only negative relationships (cycle plans [factor analysis], cycle plans [benchmark analysis], parent involvement); and 1 had no relationship (core elements). The overall quality indicator was generally unrelated to youth outcomes in elementary sites.

Of the 16 quality indicators included in the outcome analysis, 7 had favorable relationships with a subset of youth outcomes in middle school sites: cycle plans (factor analysis), cycle plans (benchmark analysis), student involvement, parent involvement, school collaboration, facilities access, and site coordinator qualities. This indicates that when program quality was higher in each of these areas, the more positive were at least one or more of the desired youth outcome results. Of the remaining 9 indicators, 6 had a combination of negative and positive relationships with outcomes (core elements, time distribution, club contributors, ad hoc value, club selection, and program staff qualities), 2 had only negative relationships (club activities and student engagement), and 1 had no relationship (ad hoc assistance).

While there was no one indicator favorably associated with all thirteen middle school outcome indicators, five of the indicators were associated with at least half of the outcomes, which was not the case in elementary sites. Also unlike elementary sites, overall middle school program quality was related to three outcomes: increased CST ELA scores, decreased problem behaviors, and increased positive attitude toward school.