Search Publications | The Evaluation Exchange | FINE Network | OST Database

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

A Profile of the Evaluation of CHAMPS

Program Description

Overview	CHAMPS is a program designed to add an academic enrichment component to existing after school programs in the School District of Palm Beach County (SDPBC) in Florida. CHAMPS’ objectives are to (a) increase after school program staff’s skills in facilitating academic enrichment curriculum and (b) increase CHAMPS participants’ academic achievement.
Start Date	fall 2002
Scope	local
Type	after school
Location	suburban
Setting	public schools, community-based organizations

Participants	kindergarten and elementary school students (grades K–6)
Number of Sites/Grantees	10 in 2002–2003; 40 in 2003–2004; 38 in 2004–2005
Number Served	892 in 2002–2003; 3,771 in 2003–2004; 3,775 in 2004–2005
Components	CHAMPS is designed to be implemented in the same way across all program sites. During the after school programs at the sites, youth rotate through a variety of activities—including snack time, free time to play outdoors, homework assistance, CHAMPS activities, and indoor recreational activities—tailored to meet the interests of youth and resources of each site. Youth participate in CHAMPS for 1 hour per day, 4 days per week. On average, youth complete one CHAMPS lesson per day. CHAMPS operates under the guidance of CHAMPS staff and a head teacher at each program site. Each school principal and after school director selects a certified school-day teacher to act as the head teacher for the CHAMPS program site for a total of 8 hours per week. This head teacher mentors the instructors and performs the following duties: (a) reviewing the instructors’ lessons, (b) overseeing implementation of the lessons in the classroom, (c) observing and coaching instructors with regard to their teaching skills, (d) preparing materials for the instructors, (e) assisting teachers who need help with facilitation techniques, (f) completing CHAMPS program paperwork, and (g) requesting CHAMPS supplies. CHAMPS instructors complete training in facilitative teaching strategies prior to implementing the program. CHAMPS activities consist of the implementation of academic curricula in existing after school programs. In the 1st year of implementation, after school program staff facilitated the curriculum Foundations Travels, in the 2nd year, the curriculum Foundations Celebrates, and in the 3rd year, the curriculum Blast Off. The content and activities of all sets of curricula were aligned to the state of Florida’s Language Arts Literacy and Mathematics standards by the curricula developers. Although the curricula were designed for students in third grade and above, instructors reduced the complexity of the curricula so that the content and activities were developmentally appropriate for students in lower grade levels as well. CHAMPS was implemented in the following three settings: 1. School Age Child Care Programs (SACCs): SACCs are financially self-supporting programs in which participating families pay a fee for each child who attends. To encourage youth from lower income families and underserved groups to participate in the program, CHAMPS provides scholarships that are distributed based on the judgment of the administration at each school. SACC staff must hold high school or general education diplomas, complete first aid and CPR training, and training to care for preschoolers. 2. Beacon Programs: Beacon schools partner with community-based organizations to offer after school childcare and additional programs that create positive changes in youth development, academic achievement, and healthy family functioning. Youth from the community, who do not necessarily attend the Beacon school during the day, are eligible to participate in after school programs offered at Beacon schools. Youth participate in CHAMPS at no cost. 3. Community Based Organizations (CBOs): CBOs are designed to serve the individual needs of the communities in which they are located. Each organization’s mission and operating procedures are unique, although all provide after school programs for school-aged children.
Funding Level	$382,600 in 2002–2003; $1.35 million in 2003–2004; $1.35 million in 2004–2005
Funding Sources	The Toppel Family Foundation, the Children’s Services Council of Palm Beach County, and the Picower Foundation

Evaluation

Overview	The evaluation sought to understand how the addition of an academic enrichment component to after school programs supports improved student achievement and increased professionalism of after school staff.
Evaluators	Kimberly A. Knutson, School District of Palm Beach County
Evaluations Profiled	Evaluation of the CHAMPS After-School Program: FY2003 CHAMPS SACC Program Evaluation Report CHAMPS After-School Program Outcome Evaluation Year Two: FY2004
Evaluations Planned	CHAMPS’ academic outcomes will be evaluated through the 2004–2005 school year. The final evaluation will address the question of how the effect on student achievement produced by CHAMPS compares to the effect on student achievement produced by (a) the standard academic enrichment activities offered through the district SACC program and (b) standard district after school care with homework help and no academic enrichment activity component.
Report Availability	Knutson, K. A. (2003). CHAMPS SACC program evaluation report. Coral Springs, FL: Author. Knutson, K. A. (2003). Evaluation of the CHAMPS after-school program: FY2003. Coral Springs, FL: Author. Knutson, K. A. (2004). CHAMPS after-school program outcome evaluation year two: FY2004. Coral Springs, FL: Author. Knutson, K. A. (2005). CHAMPS after-school program outcome evaluation year three: FY2005. Coral Springs, FL: Author.

Contacts

Evaluation	Kimberly A. Knutson, Ed.D. Specialist, Testing and Evaluation School District of Palm Beach County 3370 Forest Hill Blvd., Suite 228 West Palm Beach, FL 33406-5871 Tel: 561-434-8782 Fax: 561-963-3842 Email: Knutson@palmbeach.k12.fl.us
Program	Judy Nee Director, CHAMPS Program 8487 155th Place North Palm Beach Gardens, Florida 33418 Tel: 561-741-4641 Email: judynee@bellsouth.net
Profile Updated	January 5, 2006

Evaluation 3: CHAMPS After-School Program Outcome Evaluation Year Two: FY2004

Evaluation Description

Evaluation Purpose	To answer the following questions: (a) Were average spring achievement test scores of CHAMPS participants greater than those of comparable nonparticipants, (b) did CHAMPS participants achieve greater reading and math gains on achievement test scores than comparable nonparticipants, (c) was there a difference in the reading and math achievement by the types of sites in which CHAMPS was implemented, and (d) was the reading and math achievement of CHAMPS participants at the SACC sites different in the 2nd year of implementation than in the 1st year?
Evaluation Design	Quasi-Experimental: Of the 3,771 youth from pre-kindergarten through sixth grade enrolled in CHAMPS, 2,824 attended at least 80% of the total number of program days that CHAMPS was conducted during the 2003–2004 school year. Of these 2,824 participants, only second through fifth grade participants enrolled in school before October 1, 2003, and who attended the same school for the entire school year were included in the evaluation, for a total of 1,830 youth. Participants were compared to a sample of 1,761 youth not attending CHAMPS who matched the demographic characteristics of CHAMPS participants (grade, race/ethnicity, FRL status, and LEP status). In addition to being comparable demographically, comparison group youth attended the same schools during the day as CHAMPS participants. All youth in the comparison group had the opportunity to sign up for after school care but did not. Treatment and comparison group youth had similar 2002–2003 achievement test scores, with the only significant difference being among fourth graders at Beacon sites, where CHAMPS participants were less likely to have low reading achievement scores and less likely to have low or high math achievement scores than comparison group youth (p < .05). Subgroup analyses were done by grade and by site and within grades and sites for race/ethnicity, FRL status, and LEP status.
Data Collection Methods	Secondary Source/Data Review: Achievement test scores and demographic data were obtained from SDPBC. Test/Assessments: The achievement test scores obtained from SDPBC consisted of the Florida Comprehensive Assessment Test of Sunshine State Standards (FCAT) and the Scholastic Reading Inventory (SRI). SRI scores are first available beginning in second grade, while FCAT scores are first available beginning in third grade. The FCAT is a criterion-referenced test that measures the degree to which students have mastered the Sunshine State Standards for reading, writing, and math. SRI scores measure reading comprehension. Using these test results, three measures were examined: (a) the average performance on the spring FCAT Developmental Scale (FCAT-DS) and SRI scores, and (b) the amount of spring to spring gain on the FCAT-DS and the amount of fall to spring gain on the SRI, and (c) the effect size of statistically significant results of each analysis. (For test score gains, evaluators also calculated “effect size differences,” which are the differences in calculated pretest to posttest effect sizes for the treatment and comparison groups). Effect size differences were considered meaningful if they reached the following thresholds: .20 = small difference, .50 = moderate difference, and .80 = large difference.
Data Collection Timeframe	Data were collected for fiscal year 2004.

Findings:
Summative/Outcome Findings

Academic

Among second graders at Beacon sites, the average SRI scores for all, African American, FRL, and LEP CHAMPS youth were significantly higher than their counterparts in the comparison group (p < .05). The effect sizes were small for all youth and the FRL subgroup, and moderate for African American and LEP youth.

Among second graders at SACC sites, there were no significant differences on SRI scores overall, although the average SRI scores for CHAMPS African American, Hispanic, and FRL youth were significantly higher than youth in respective comparison groups (p < .05). The effect sizes were small for African American and FRL youth and moderate for Hispanic youth.

Among second graders at CBO sites, the average SRI scores for CHAMPS youth did not differ from comparison group youth, either overall or by subgroup.

Among third graders at SACC sites, there were no significant differences on the SRI or FCAT overall, although the average SRI scores for CHAMPS African American, LEP, and FRL youth were significantly higher than youth in their respective comparison groups (p < .05). The effect sizes were small for FRL youth and moderate for African American and LEP youth. In addition, the average FCAT math score for African American CHAMPS youth was significantly higher than for comparison group youth (p < .05), although the effect size was small.

Among third and fourth graders at CBO and Beacon sites as well as fifth graders at SACC sites, the average FCAT and SRI reading and math scores for CHAMPS youth did not differ from their comparison group counterparts, overall or by subgroup.

Among fourth graders at SACC sites, there were no significant differences on the SRI or FCAT overall, although Hispanic CHAMPS youth’s average SRI-I score was significantly higher than the Hispanic youth in the comparison group (p < .05).
Among fifth graders at Beacon sites, there were no significant differences on the SRI or FCAT overall, although the average FCAT math and reading scores of African American CHAMPS youth were significantly higher than their comparison group counterparts (p < .05). The effect size was moderate for both reading and math.

Among third graders at Beacon sites, all, African American, and FRL CHAMPS youth demonstrated less gain on the FCAT reading than their counterparts in the comparison group (p < .05). The effect sizes were small.

Among third graders at CBO sites, CHAMPS youth demonstrated less gain on the FCAT reading and math than comparison group youth. The effect size was moderate for both. In math, CHAMPS African American and FRL youth demonstrated significantly less gain than youth in their respective comparison groups (p < .05). The effect sizes were moderate for both groups. On the SRI-I, the gains demonstrated by all youth, African American youth, LEP youth, and FRL youth participating in CHAMPS was significantly greater than the amount of gain demonstrated by youth in the comparison groups (p < .05). The amount of gain was small for all youth, African American youth, and FRL youth and moderate for LEP youth.

Among third graders at SACC sites on the SRI-I, there were no significant differences on the SRI or FCAT overall, although White CHAMPS youth demonstrated less gain than White youth in the comparison group (p < .05). The effect size was small. LEP CHAMPS youth demonstrated greater gains than LEP comparison group youth (p < .05). The effect size was moderate. On FCAT reading, African American and FRL youth demonstrated greater gains than their comparison group counterparts (p < .05). The effect sizes were moderate. On FCAT math, African American and FRL youth demonstrated greater gains than their comparison group counterparts (p < .05). The effect size was moderate for African American and small for FRL youth.

Among fourth graders at Beacon sites, there were no significant differences on the SRI or FCAT overall, although Hispanic and LEP CHAMPS youth demonstrated less gain on FCAT reading than their comparison group counterparts (p < .05). The effect size was large. In math, all, Hispanic, and LEP CHAMPS youth demonstrated greater gains than their comparison group counterparts (p < .05). The effect size was small for all youth, large for Hispanic youth, and moderate for LEP youth.

Among fourth graders at CBO sites, there were no significant differences on the SRI or FCAT overall, although LEP CHAMPS youth demonstrated greater gains on the SRI-I than comparison group youth (p < .05). The effect size was large. On the FCAT math, all, LEP, and FRL CHAMPS youth demonstrated greater gains than their counterparts in the comparison group (p < .05). The effect sizes were small.

Among fourth graders at SACC sites, there were no significant differences on the FCAT or SRI overall, although all and FRL CHAMPS youth demonstrated greater SRI-I gains than their comparison group counterparts (p < .05). The effect size was small for all youth and moderate for FRL youth. On FCAT reading, all, African American, Hispanic, and FRL CHAMPS youth demonstrated less gain than their comparison group counterparts (p < .05). The effect size was small for all groups. On the FCAT math, all, African American, and Hispanic CHAMPS youth demonstrated greater gains than their comparison group counterparts (p < .05). LEP CHAMPS youth demonstrated less gain than their comparison group counterparts (p < .05). The effect size for all groups was small.

Among fifth graders at Beacon sites, there were no significant differences on the FCAT or SRI overall, although all CHAMPS youth and African American CHAMPS youth demonstrated greater SRI-I gains than comparison group youth (p < .05). The effect sizes were small. On the FCAT math, all, African American, Hispanic, LEP, and FRL CHAMPS participants demonstrated greater gains than comparison group youth. The effect sizes were small. White CHAMPS youth demonstrated less gain on the FCAT math than their comparison group counterparts. The effect size was large.

Among fifth graders at CBO sites, there were no significant differences on the SRI or FCAT overall, although LEP youth demonstrated greater SRI-I gains than their comparison group counterparts (p < .05). The effect size was small. On the FCAT math, all, African American, and FRL CHAMPS youth demonstrated greater gains than their comparison group counterparts (p < .05). The effect size was small. LEP CHAMPS youth demonstrated less gain on the FCAT math than their comparison group counterparts (p < .05). The effect size was moderate.

Among fifth graders at SACC sites, there were no significant differences on the SRI or FCAT reading, although all, African American, White, LEP, and FRL CHAMPS participants demonstrated greater gains on the FCAT math than comparison group youth (p < .05). The effect size for all groups was small.

When comparing estimated effects in reading across program site contexts, the following findings emerged: (a) CHAMPS third graders at CBO program sites performed better relative to their comparison group counterparts on SRI-I than did CHAMPS third graders at SACC and Beacon sites, (b) CHAMPS third graders at SACC sites performed better on FCAT reading relative to their comparison group counterparts than did CHAMPS third graders at Beacon and CBO sites, (c) CHAMPS fourth graders at SACC sites performed better on SRI-I relative to their comparison group counterparts than did CHAMPS fourth graders at Beacon and CBO sites, (d) LEP CHAMPS fourth graders at CBO sites performed better on SRI-I relative to their comparison group counterparts than did LEP CHAMPS fourth graders at Beacon and SACC sites, and (e) Hispanic CHAMPS fourth graders at Beacon sites performed better on FCAT reading relative to their comparison group counterparts than did Hispanic CHAMPS fourth graders at SACC or CBO sites.

When comparing estimated effects on FCAT math scores across program site contexts, the following findings emerged: (a) CHAMPS third graders at SACC sites performed better relative to their comparison group counterparts than did CHAMPS third graders at Beacon or CBO sites, (b) African American CHAMPS fourth graders at SACC sites performed better relative to their comparison group counterparts than did African American CHAMPS fourth graders at Beacon and CBO sites, (c) Hispanic and LEP CHAMPS fourth graders at Beacon sites performed better relative to their comparison group counterparts than did Hispanic and LEP CHAMPS fourth graders at SACC and CBO sites, (d) FRL CHAMPS fourth graders at CBO sites performed better relative to their comparison group counterparts than did FRL CHAMPS fourth graders at SACC and Beacon sites, and (e) CHAMPS fifth graders at Beacon sites performed better relative to their comparison group counterparts than did CHAMPS fifth graders at SACC and Beacon sites.

When comparing effect size estimates between year 1 and year 2 of the evaluation for reading, the following findings emerged: (a) The estimated impact of CHAMPS on the reading performance of LEP third graders was greater in year 2 than in year 1; (b) on SRI-I, the estimated impact of CHAMPS on the reading performance of fourth graders was greater in year 2 than in year 1 for all and FRL youth; (c) on FCAT, the estimated impact of CHAMPS on the reading performance of fourth graders was less in year 2 than in year 1; and (d) on SRI-I, the estimated impact of CHAMPS on the reading performance of fifth graders was less in year 2 than in year 1.

When comparing effect size estimates between year 1 and year 2 of the evaluation for math, the following findings emerged: (a) The estimated impact of CHAMPS on the math performance of all and Hispanic fourth graders was greater in year 2 than in year 1; (b) the impact of CHAMPS on the math performance of FRL and LEP fourth graders was less in year 2 than in year 1; (c) the estimated impact of CHAMPS on the math performance of all, White, and LEP fifth graders was greater in year 2 than in year 1; and (d) the estimated impact of CHAMPS on the math performance of African American and FRL fifth graders was less in year 2 than in year 1.

A Profile of the Evaluation of CHAMPS

Profile Information