Volume X, Number 1, Spring 2004

Issue Topic: Evaluating Out-of-School Time Program Quality

Evaluations to Watch

Investigating Quality: The Study of Promising After-School Programs

Deborah Lowe Vandell and Elizabeth R. Reisner describe their three-year study into whether and how participation in high quality after school programs matters for youth outcomes.

More and more, stakeholders in out-of-school time (OST) are asking, What is quality and does it matter? To address the question, the Study of Promising After-School Programs¹ (the Study) is investigating the short-term and long-term impacts of high quality after school programs on the development of youth from high-poverty communities. By first identifying high quality programs and then examining their associated youth outcomes, the Study goes beyond the question of whether any participation matters for youth outcomes, to address the question of whether, and how, participation in high quality programs matters.

The Study operates from a research-driven theory of change² that links family background characteristics, program features, program participation, and youth outcomes. The Study’s theory of change acknowledges that these components are interdependent. For example, child and family characteristics may be associated with access to and outcomes of high quality programs.

Testing the relationships among these components requires a comprehensive, multifaceted study that employs a nationwide sample of programs. The Study includes 18 elementary schools and 18 middle schools that serve predominantly low-income children at risk for school failure. At the outset, OST experts identified potential school-based or school-linked programs whose quality they deemed high. This sample was then narrowed to those programs meeting criteria both for high quality practices as defined by previous research and for research feasibility (i.e., enrollment of at least 30 students in each of the targeted grades and availability of similar students not served by the program).

Programs included in the study met the following criteria for quality:

• They have been in operation at least three years and are likely to remain in operation for the near future.
• They offer opportunities for sustained involvement in substantive activities intended to promote cognitive and social growth.
• They use staffing methods conducive to supportive relationships with youth (e.g., low child-adult ratios, low staff turnover, staff training).

Children who participate regularly in high quality programs are compared with classmates who are similar in terms of family background but who do not participate. Children with varying levels of participation are also compared. Assessment of participation is based not only on a yes-or-no scale, but also on the frequency of participation.

Program quality is assessed in terms of two related but distinct sets of features: structural and institutional features, such as staff education, child-staff ratio, and financial resources; and process and content features, such as positive relationships, variety in program offerings, availability of activities that promote sustained cognitive engagement, and opportunities for autonomy and choice.

The Study is also examining four categories of developmental outcomes: academic, social, psychological, and behavioral. Each category is being assessed by multiple sources, including youth, parents, teachers, and staff. Developmental outcomes are being assessed both in the intermediate term (in spring 2004) and in the longer term (in spring 2005).

Preliminary analyses of baseline data collected from programs in spring 2002 and fall 2003 reveal some of the programs’ characteristics. In terms of structural/institutional features, analysis has so far shown that programs have strong ties with community organizations, offer a wide variety of activities ranging from tutoring to sports, and demonstrate the likelihood of sustainability over several years. The majority of staff members are college educated, a large proportion with at least three years of OST experience, and most reporting high levels of job satisfaction. In terms of process and content features, preliminary analysis shows that most programs get high ratings for having supportive relationships, opportunities for cognitive growth, and student engagement. These results are consistent across both elementary and middle school programs in the sample.

In 2004 and 2005, further analysis will be conducted to address questions of participation and outcomes. The specific questions will include the following:

1. How do child and family characteristics relate to participation?
2. Is frequency of participation related to youth outcomes?
3. Does participation have intermediate effects on school attendance and work habits, on social skills and peer relationships, on self-efficacy and persistence, and on misconduct?
4. Does participation have long-term effects on grades and test scores, on positive attitudes and behaviors at home, and on delinquent and risk-taking behaviors?

The final results of this research, in combination with the preliminary results discussed above, will help to document the components that constitute high quality OST programs and their effects on participating youth.

For more information about the Study of Promising After-School Programs, visit www.wcer.wisc.edu/childcare/des3.html.

¹ This nationwide study, which spans the years 2002–2005, is being conducted by the Wisconsin Center for Education Research, in collaboration with Policy Studies Associates, and with support from the Charles Stewart Mott Foundation.
² A program’s theory of change explicitly articulates how the activities provided lead to the desired results by linking service receipt to intermediate changes (such as changes in attitudes or in-program behaviors) to short- and longer-term outcomes in participants.

Deborah Lowe Vandell
Professor of Human Development
Wisconsin Center for Education Research
School of Education
University of Wisconsin-Madison
1025 West Johnson Street, Ste. 785
Madison, WI 53706
Tel: 608-263-1902
Email: dvandell@facstaff.wisc.edu

Elizabeth R. Reisner
Principal
Policy Studies Associates, Inc.
1718 Connecticut Avenue, NW, Ste. 400
Washington, DC 20009
Tel: 202-939-5323
Email: ereisner@policystudies.com