Volume XII, Number 1 & 2, Fall 2006

Issue Topic: Building and Evaluating Out-of-School Time Connections

Evaluations to Watch

After School Programs for High School Students: Launching the Evaluation of After School Matters

Barton Hirsch and Larry Hedges present their innovative design for evaluating After School Matters, a Chicago initiative that draws on connections with community members, businesses, and schools.

In the eyes of policymakers, after school programs are an increasingly important vehicle for achieving a host of objectives, from promoting positive youth development and workforce preparation to reducing rates of school dropout and community violence. We have recently launched an evaluation of the country's largest provider of after school programs for high school students: After School Matters (ASM).1

Located primarily in 35 Chicago public high schools, ASM provides paid apprenticeships to adolescents in partnership with the public schools, libraries, park district, children and youth services, and community-based organizations. The apprenticeships last for 10 weeks (90 hours) each fall and spring and emphasize developing skills by working on challenging tasks that lead to a public final product or performance. Instructors have expertise—and often earn their livelihood—in the content areas of the apprenticeships, which focus on technology (e.g., computer repair), the arts (e.g., improvisational theater), a combination of the two (e.g., producing social documentaries), or sports (e.g., lifeguarding).

Although ASM hopes to improve academic performance and reduce problem behaviors, its primary objectives are to promote positive youth development and teach marketable job skills. Our evaluation examines outcomes in all four of these domains through a mixed-methods approach that incorporates both quantitative and qualitative methods. The evaluation uses an experimental design, in which youth who sign up for a specific apprenticeship are randomly assigned to the intervention or control condition. Pending full funding, this first study will include 560 intervention and control group youth, across 14 apprenticeships, based in approximately nine high schools.

A Series of Evaluation Studies
The after school field has become sensitized to the political context of evaluation work since the federal government sought to slash funding for after school programs in response to the preliminary, controversial analysis of the 21st Century Community Learning Centers.2 Because ASM evaluation findings are likely to have important policy implications, it is vital that a research strategy be put in place that does not prematurely lead to a summative “final” judgment about ASM's effectiveness. For that reason, we believe that it is important to conduct a sequenced series of ASM evaluation studies, rather than an all-in-one single investigation.

The first randomized controlled trial of ASM will focus on whether the intervention works when it is implemented well. We will study apprenticeships taught by experienced instructors who do a good job of implementing design features—an important and reasonable first step that will provide a good test of whether the underlying ASM model leads to significant youth gains. This strategy will give the intervention its “best shot” at demonstrating impact, while still utilizing rigorous scientific methodology.

If findings from this initial evaluation are positive, ASM will then evaluate whether the scaled-up model is effective under the range of implementation conditions that exist in a broader set of sites. This sequencing will enable us to determine whether the underlying theoretical model is sound before addressing issues of implementation during scale-up and their effects on program impact—critical knowledge that would not be available if we evaluated the scaled-up version first. Additionally, the scaled-up evaluation will be expensive, and positive findings from the initial experimental trial will help to justify the cost.

Evaluating Marketable Job Skills
In designing the ASM evaluation, we purposefully tied our methods to the program's goals and components. We were excited by the opportunity to develop an assessment of marketable job skills suitable for high school students. It became clear from our reading of relevant literature and consultations with senior human resource (HR) professionals that employers rely heavily on job interviews when hiring for entry-level positions. As part of our evaluation strategy, therefore, we designed a mock job interview conducted by HR professionals. At the end of the interview, each youth is rated on specific skills, attitudes, and behaviors. These include “soft skills,” such as problem solving, teamwork, and communication, which are increasingly important in the global economy. Interviewers also make an overall recommendation as to whether they would hire the young person for an actual entry-level position.

Ratings made by HR professionals on youth employability in private sector jobs provide the kind of hard-outcome data favored by policymakers. Given that many of the job skills assessed are also indices of adolescent development, this study may yield the kind of research evidence that can persuade policymakers of the benefit of a positive youth development approach to after school programs.

1 Initial funding for the evaluation is provided in part by the Searle Fund.
2 For more information on this debate, and for experts' reactions, click here to see an article from the spring 2003 Evaluation Exchange.

Barton J. Hirsch
Professor of Human Development and Social Policy
Northwestern University
2120 Campus Drive
Evanston, IL 60208
Tel: 847-491-4418
Fax: 847-491-8999
Email: bhirsch@northwestern.edu

Larry V. Hedges
Board of Trustees Professor in Statistics, Human Development and Social Policy
Institute for Policy Research
Northwestern University
2040 Sheridan Road
Evanston, IL 60208
Tel: 847-491-8899
Email: l-hedges@northwestern.edu