The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume IX, Number 1, Spring 2003
Issue Topic: Evaluating Out-of-School Time
Promising Practices
The Challenges and Strategies of Evaluating Community-Wide After School Programs
Drawing on their experiences from the Municipal Leadership for Expanded Learning Opportunities project, Mark Ouellette and Audrey Hutchinson of the National League of Cities reflect on the challenges of evaluating citywide after school programs and the strategies that cities have adopted to meet these challenges.
In a majority of the eight cities involved in the Municipal Leadership for Expanded Learning Opportunities project,¹ no one organization has authority over the after school programs run by the various providers in the city. Most of these communities have created collaboratives comprised of diverse stakeholders to provide some type of governance structure for after school programs. Representatives include businesses, school personnel, municipal officials, youth-serving organizations, and community and faith-based organizations. But the role of the collaboratives is limited to providing direction for after school programs, increasing awareness of the importance of out-of-school time, and raising new funds for local initiatives. The governing bodies do not usually have any direct control over program staff, curriculum, or services offered.
The collaborative nature of these municipal efforts poses two main challenges for evaluation:
1. Identifying evaluation goals. The principal challenge with evaluating this type of collaborative is identifying the goals of the evaluation. Certainly different audiences—researchers, service providers, and legislators—require different standards of evidence, and what satisfies one will frequently be unsatisfactory or confusing to another.
2. Financing the evaluation. Financing an evaluation is another major obstacle with new funds typically being used to expand or create new after school programs rather than to support new research efforts to examine already established programs.
Related ResourceNational League of Cities. Expanding Afterschool Opportunities Action Kit. www.nlc. |
Strategies for Addressing the Challenges
Many of the collaboratives, however, know they need local evidence to demonstrate to the public that after school programs are a worthwhile investment and are having a positive impact on the children and youth in their communities. In the absence a “logic-model-driven” evaluation,² seven of the eight cities in the Municipal Leadership for Expanded Learning Opportunities project are documenting the effects of their collaboratives.³ They are using three strategies:
1. Present data to make the case for the collaborative. Several communities have collected data on program availability as measured by the number of slots available for children and youth. In Indianapolis, Indiana data collected by the United Way showed that in one year, program availability increased by 71 percent as a result of three new collaborative projects: (1) a new 21st Century Community Learning Centers grant, (2) the first-time provision of services by the Indianapolis Parks and Recreation Department, and (3) other providers expanding their programs into new areas as a result of a needs assessment conducted by the collaborative.
2. Use an evaluation from one of the partners to demonstrate the value to the community. All eight of the Municipal Leadership for Expanded Learning Opportunities cities have on their own received a 21st Century Community Learning Centers grant from the U.S. Department of Education. The cities use the data collected from these grants to demonstrate to the public the positive impact that after school programs across the collaboratives have on children and youth in their communities.
3. Partner with local organizations that can evaluate after school programs. Several communities have sought assistance from local nonprofits for help in evaluating their after school and summer school programs. In Washington, D.C. the collaborative secured the assistance of the Urban Institute to evaluate programs operated within the District of Columbia.
It is never easy to establish the causal link between the activities of the after school program and positive outcomes for children and youth. Evaluating community-wide systems of after school programs is even more challenging because the evaluations must focus on a variety of after school providers operating different programs, different times of the day, and providing different dosages, and different services (e.g., academic, enrichment, recreational) for children and youth. However, the strategies that the Municipal Leadership sites have developed can help other city collaboratives in their efforts to assess the impact of their programs.
Audrey Hutchinson,
Program Director, Education and Afterschool Initiatives
Mark Ouellette,
Senior Program Associate, Afterschool Initiatives
National League of Cities
1301 Pennsylvania Avenue, N.W., Ste. 550
Washington, DC 20004
Tel: 202-626-3000
Fax: 202-626-3043
Email (for Mark Ouellette): ouellette@nlc.org
¹ The aim of the Municipal Leadership for Expanded Learning Opportunities project is to help city leaders increase the availability, and improve the quality, of expanded learning opportunities during the nonschool hours for children and youth. The eight cities involved in this 30-month project are: Charlotte, North Carolina; Fort Worth, Texas; Fresno, California; Grand Rapids, Michigan; Indianapolis, Indiana; Lincoln, Nebraska; Spokane, Washington; and Washington, District of Columbia. The project began in June 2001 through support from the Charles Stewart Mott Foundation.
² The authors use this phrase to refer to an evaluation that has clearly articulated goals, indicators, and outcomes. For more information on using logic models for evaluation, see Logic Models in Out-of-School Time.
³ Fort Worth, Texas is an exception and has been engaged in an evaluation of their after school programs since the programs’ origin. The city of Fort Worth and the Fort Worth Independent School District jointly funded 52 after school programs (which operate under the umbrella of Fort Worth After School or FWAS), and hired an independent evaluator to demonstrate FWAS’ impact on children and youth in the community. See the related article, Learning From Evaluation: Lessons From After School Evaluation in Texas.