Volume VII, Number 2, Spring 2001

Issue Topic: Out-of-School Time Issue #2

Ask the Expert

Why evaluate after school programs?

Olatokunbo (Toks) Fashola, Ph.D., is an Associate Research Scientist at the Johns Hopkins University Center for Research on the Education of Students Placed at Risk (CRESPAR). CRESPAR researches, develops, and evaluates programs needed to improve schooling for students placed at risk. Dr. Fashola’s involvement in out-of-school time programs began in 1996 when the City of Baltimore, the Baltimore Empowerment Zones, and the Child First Authority received funding to establish out-of-school programs. Her most recent work includes research and evaluation of after school programs across the country. Her book titled “Building Effective After-School Programs: Research and Practice” will be out in September 2001.

Why evaluate after school programs?

With the recent increase in interest in children’s activities during out-of-school hours, large amounts of federal funding are being directed toward after school programs. Educators, researchers, practitioners, and policy makers are all interested in the impact of these dollars on the communities in which they are invested; evaluation is a necessary tool for the documentation of after school programs’ impacts. Future policies regarding after school programs will be influenced not only by national research and evaluation investments, local evaluation efforts will also provide important information to inform the field.

Evaluation serves multiple purposes for after school programs. In addition to helping programs be accountable to the multiple stakeholders of the program—funders, schools, parents, student participants, and communities—evaluation helps programs to strengthen themselves. Engaging in frequent, active evaluation is one way for after school programs to track progress toward goals and ensure that they are meeting the needs of communities.

New programs are often too busy getting the program off the ground to think forward to evaluation planning. What type of evaluation activities can new programs incorporate into their planning and implementation process?

In the first year of a program, it is not unrealistic for staff to spend a lot of their time developing the program’s goals and creating the program. Engaging in formative evaluation from the very beginning will help to shape the outcome expectations for the program. While this type of evaluation will not immediately give “evidence of effectiveness” or other statistical outcome measures, such as means, standard deviations, and effect sizes pertaining to the impact of the program, it will provide important evaluation information to stakeholders of the after school program. Such formative information—including feedback on program structure and activities, descriptions of participant demographics, documentation of parent and student satisfaction, attendance patterns of the students and staff, and descriptions of program successes and challenges—is essential to frame the outcomes evaluations to follow.

When incorporating evaluation into a program, the key to success is connecting the needs of the program to the goals of the program and eventually to the outcomes. In a logic model for the program, needs should drive goals and objectives, and those should be directly connected to expected outcomes for the program. In order to be able to make some claim of success, it is important to create an infrastructure for the evaluation, based on a solid logic model, that links activities in the program to outcomes expected for the participants.

The program planning, implementation, and evaluation process benefits from a strong data collection and evaluation infrastructure. Creating an evaluation infrastructure first and foremost requires the organization, its staff, and the evaluation team to commit to conducting a sound, rigorous evaluation. The evaluation infrastructure addresses not only the physical mechanisms that pertain to evaluation, such as data collection tools and evaluation personnel, but accounts for the non-tangible attitudes among stakeholders about evaluation. Points of accountability and responsibility from the participants in the program all the way to the evaluator and eventually the funders are highlighted in a comprehensive evaluation infrastructure. The evaluation infrastructure is a whole communication mechanism that exists between and among the various stakeholders involved in the program and its evaluation. For a strong infrastructure to exist, all of the stakeholders must be in agreement about the needs, goals, and intended outcomes of the program. Building a data collection and evaluation infrastructure into the program from the very start creates an environment that will support effective evaluation efforts in the future.

Olatokunbo S. Fashola, Ph.D.
Center for Research on the Education of Students Placed at Risk
The Johns Hopkins University
3003 N. Charles St., Suite 200
Baltimore, MD 21218-3888