The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume IX, Number 1, Spring 2003
Issue Topic: Evaluating Out-of-School Time
Promising Practices
Learning From Evaluation: Lessons From After School Evaluation in Texas
Dr. Peter A. Witt, the Elda K. Bradberry Recreation and Youth Development Chair at Texas A&M University, reflects on seven years of evaluating city after school programs in Texas.
Over the last seven years, the external project team that I lead has been involved in evaluations of after school programs at various school campuses in urban areas in Texas, including Austin, Corpus Christi, Dallas, Fort Worth, and Houston.¹ Reflecting on the project team’s evaluation experiences across these sites, a number of key lessons emerge as central to conducting after school evaluation.
Meet the Information Needs of Multiple Stakeholders
Most programs have multiple stakeholders, many of whom emphasize different goals in their support of, and expectations for, program outcomes. In these cases, means have to be found to collect data honoring the differing stakeholders’ perspectives. For example, the Fort Worth After School project has to answer to its funders, the Fort Worth Independent School District and the Fort Worth Crime Prevention District, its manager, a Coordinating Board made up of city and school district representatives, and the staff who manage the day-to-day administration and oversight of the program. Other interested parties include the 11 service providers used by the program, the principal at each school who chooses the service provider for their site, and, of course, the parents and students.
As with any project, these various stakeholders have different expectations for the program. Parents, for example, want their children to be in a safe environment, to complete their homework, and to undertake activities beyond watching television; children want to have fun and be around program staff and other participants they like. The two major funders are interested in ensuring that children are in a safe environment, that they increase their educational performance, that they increase their social and physical skills, and that crime in the community is reduced. Thus the evaluation process has to include variables and information that enables each of the stakeholders to see if their goals are being met.
Use an Organizing Framework to Set Program Goals
Using an organizing theory or framework helps all stakeholders agree to a set of program goals, which serves as the basis for determining the targeted outcomes for the evaluation. Before the after school program began in Fort Worth, the city was already invested in a Developmental Assets Model² so framing the evaluation in these terms provided useful ties between the after school program and other projects for children in the community. In addition, the Assets framework helped avoid two particularly sticky issues that plague a number of after school program evaluations: demonstrating increases in educational test scores and reducing crime. Using the Assets framework enabled measurement of the precursors to academic improvement and reduced crime, without having to provide direct measurements in these two areas.
Related ResourcesTo read Dr. Peter Witt’s full evaluations of after school programs in Austin, Dallas, and Fort Worth see: www.rpts.tamu. A summary of the evaluation from the Austin after school program is available in HFRP’s Out-of-School Time Program Evaluation Database. A summary for Fort Worth is coming soon. |
Use a Participatory Approach to Evaluation
After school programs are best served by a participatory, ongoing evaluation process. While the evaluators need to retain their impartiality and objectivity to gain access to insider information, it is best if stakeholders see the evaluators as an integral part of the overall project team. In addition, including an evaluation component as a part of the program for multiple years as well as maintaining consistency in the evaluation team helps identify longitudinal changes in the program and its outcomes.
Help Program Organizers Match Goals to Outcomes
One of the most interesting challenges is to help after school program organizers understand that the program they are providing must match the goals they are seeking to achieve. In other words, the content of the program must be consistent with what stakeholders want the project to accomplish. Unfortunately this is often not the case and there is a mismatch between a program’s marketing description and what really occurs in the program. An external evaluator can help bring these discrepancies to the attention of the program organizers.
Tailor Reports to Meet Stakeholder Needs
Evaluators often want to demonstrate that they have earned their money by presenting an overly detailed and complicated report. However, most stakeholders want the results boiled down to a couple of pages, with bulleted findings and recommendations. In the case of the Fort Worth After School program, we produced a fully developed executive summary as well as an easy to read and accessible “report card.” In this way even the final evaluation report served multiple stakeholders.
Dr. Peter Witt
Elda K. Bradberry Recreation and Youth Development Chair
Department of Recreation, Park and Tourism Sciences
Texas A&M University
2261 TAMU
College Station, TX 77843
Tel: 409-845-7325
Fax: 409-845-0446
Email: pwitt@tamu.edu
¹ The programs had a variety of different sponsors, including the Parks and Recreation Departments, local school districts, and a County Commissioners Court. In most cases the evaluation process was guided by a risk/protective factors/resiliency framework or the Search Institute’s Developmental Assets Model (www.search-institute.org/assets).
² Developed by the Search Institute, this model identifies 40 critical factors for young people’s growth and development. For more information see www.search-institute.org/assets.