The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume VII, Number 2, Spring 2001
Issue Topic: Out-of-School Time Issue #2
Evaluations to Watch
National Evaluation of the 21st Century Community Learning Centers
Mark Dynarski and Mary Moore of Mathematica Policy Research, reveal the challenges of evaluating a national program implemented in multiple locations with inherently different key elements.
In any field, large, nationally implemented programs are faced with the challenge of demonstrating the effectiveness of a program model implemented in multiple locations with inherently different key elements. The U.S. Department of Education (ED) and the C.S. Mott Foundation (Mott) face this challenge with the implementation of the 21st Century Community Learning Centers (21st CCLC) Program. Through an evaluation design that is as complex and multifaceted as the program it aims to evaluate, the ED and Mott hope to gather evidence on whether these 21st CCLC programs improved student performance and behavior in and out of school.
The Scope of the Program
The 21st CCLC Program, authorized under the Federal Elementary and Secondary School Act, provides expanded learning opportunities for participating children in a supervised, safe environment through grants to local education agencies. Administered by the ED, in collaboration with other public and nonprofit agencies, the program involves about 3,600 rural and inner-city public schools in 903 communities. Federal support for the program has grown from $40 million in fiscal year 1998 to an appropriation of $846 million in fiscal year 2001.
Mott complements the work of the ED by providing technical assistance support to help grantees develop, implement, and sustain high-quality after school programs, building public will for after school programs, and supporting evaluation and research work in the field.
The Scope of the Evaluation
Mathematica Policy Research, Inc. (Mathematica) contracted with the ED to conduct an evaluation of both the impact and implementation of the 21st CCLC Program. In addition, Mathematica received a grant from Mott to supplement both of the ED-funded studies—impact and implementation—as well as to conduct a series of special studies that will further explore issues related to after school program access, services, and best practices. Together, Mott and ED are committed to obtaining credible evidence that these programs result in improved student performance and behaviors in and out of school.
Both the ED core studies and the Mott enhancements will rely on qualitative and quantitative methods for data collection. The impact study will include experimental and quasi-experimental design components, randomly sampling programs that have been implemented in the original 21st CCLC grant. The implementation study will focus on understanding the ways in which 21st CCLC programs were, in actuality, implemented. Underlying the overall evaluation plan is a logic model that lays out the five main topical areas that are central to the evaluation: (A) the context in which each after school program operates, (B) implementation of the after school program, (C) family, individual, and community conditioning factors that affect (D and E) students’ intermediate and long-term outcomes.
Evaluation Design
Impact Study
The impact study will investigate the conditioning factors, intermediate effects, and long-term effects of 21st CCLC programs that were implemented as intended.
The ED-funded component of the study will include two distinct designs, one targeting elementary school children and another, middle school children. The Mott-funded enhancement to the impact study will add up to 10 additional sites representing programs considered to be of “high quality” in order to learn more about how quality features can be developed.
Data will be collected through surveys, student records, and reading achievement tests.
Mott also is considering a longitudinal component to the impact study that will provide important information about the long-term effects of after school program participation on outcomes such as course-taking and behavior in high school, high school completion and expectations, and plans for post-secondary education.
Implementation Study
The ED-funded implementation study will examine the components of 21st CCLC programs as well as student experiences and patterns of participation. This study is meant to understand what programs look like, as implemented, documenting adjustments that have been made to the 21st CCLC model and contextual factors that promote or detract from successful implementation. The study will include 30 middle school sites and up to 20 elementary school sites and focus on gaining an in-depth understanding of a range of key implementation components.
Data will be collected through interviews, focus groups, observations, and document review during two visits to each site.
The Mott-funded enhancement of the implementation study consists of in-depth analysis of up to 10 additional sites, documenting how programs foster developmentally appropriate youth outcomes, implement strong community partnerships, create and sustain parent involvement, and what contributes to overall program quality.
Study of After School Time Use
Deborah Vandell and Associates at the University of Wisconsin-Madison are conducting this Mott-funded special study. Designed to supplement the national impact and implementation studies, this study will examine student experiences in after school programs and student engagement in program activities.
Using a technique called experience sampling, students in the study will be given “wrist watches” that are programmed to beep 15 times over a five-day period. When beeped, students will record information about where they are, who they are with, what activities they are involved in, their level of involvement or engagement, and their affect or emotion.
Surveys of Program Participants and Nonparticipants
Surveys geared toward documenting programs’ effects on participants will be administered to students in all of the Mott middle school sites in the national study as well as 1,000 nonparticipating students at Mott middle school sites.
Surveys of Principals and Program Staff
To complement the implementation study, surveys of principals and staff in the Mott middle school and elementary school sites will provide information about how these key actors in after school programs conduct their roles, tap resources, and coordinate regular and after school programs.
Study of Other Mature After School Programs (non-21st CCLC)
This study, conducted by Policy Studies Associates, will examine the activities and experiences of after school programs not funded by the 21st CCLC program in order to provide an opportunity to learn about and promote practices that may benefit and help sustain 21st CCLC programs.
The national 21st CCLC evaluation represents a unique public-private partnership and commitment to ensuring that as much as possible is learned from the program and that these learnings will contribute to more effective, accessible, and sustainable after school programs nationwide.
For more information about the evaluation, please contact: Mark Dynarski at 609-275-2397 or mdynarski@mathematica-mpr.com.
Mark Dynarski, Ph.D.
Mary Moore, Ph.D.
Principal Investigators
Mathematica Policy Research, Inc.
P.O. Box 2393
Princeton, NJ 08543-2393