Search Publications |  The Evaluation Exchange |  FINE Network |  OST Database

HFRP > Evaluation > The Evaluation Exchange > Issue Archive > Out-of-School Time Issue #1 > Building the After School Field: A Conversation With Evaluators, Policymakers, and Practitioners

Volume VI, Number 1, Fall 2000

Issue Topic: Out-of-School Time Issue #1

Theory & Practice

Building the After School Field: A Conversation With Evaluators, Policymakers, and Practitioners

Over the past year with support from the Charles Stewart Mott Foundation, Harvard Family Research Project (HFRP) “mapped” the after school field: marking the trends and issues, challenges, unifying themes, and the ways in which connections between policymaking, practice, and research and evaluation can be strengthened. As part of this work, HFRP hosted a series of round table conversations to gather perspectives. In fall, 1999, we conducted five focus groups with evaluators and researchers of after school programs. In June, 2000, we co-convened a meeting of after school stakeholders with the John D. and Catherine T. MacArthur Foundation Research Network on Successful Pathways Through Middle Childhood. These conversations sought to determine the salient policy, practice, and research issues in the out-of-school time field, to discuss the current state of research and evaluation in the field, and to identify the role research and evaluation play in supporting improved policy and practice. The following summarizes these discussions.

The after school field emerged from the national consensus that the out-of-school hours play an important role in the development of healthy children and youth and brings together a diversity of voices, experiences, and perspectives. In order to better understand existing needs, HFRP began by documenting the different “players” in the field, identifying three broad stakeholder groups: policymakers, practitioners, and evaluators and developmental researchers. We defined policymakers as elected officials and bureaucrats, as well as public and private sector staff, who develop, implement, and fund after school initiatives. Professional service practitioners, the staff and managers of out-of-school time programs, provide direct services in the field. Members of the research and evaluation communities include evaluators of local, state, large scale, and national initiatives and developmental researchers. These stakeholders represent a variety of programmatic emphases (school-aged child care, youth development, and education) and various institutions (schools, community-based organizations, universities, and businesses).

The diversity of the after school field gives it the strength of different perspectives, but also inherent and substantial challenges. “After school” and “out-of-school time” vary in meaning among stakeholders. But for those who participated in the meeting and focus groups, mutual concerns began to emerge, including the focus for after school programs, their funding sources, which institutions should provide after school services and what types of individuals should staff them.

The Role of Evaluation and Research in Supporting After School Programs
The discussions surfaced several other key questions to be addressed by researchers and evaluators:

• Who is participating in after school programs? Data on participation is needed to understand who is and is not being served, and which programs/activities are most popular and well attended.

• What are the key ingredients of a quality program? Correlating key elements of quality programs with positive developmental outcomes offers insight into programs that are having an impact on children’s lives.

• What do children want and what do parents need? It is essential to ensure that programs are locally relevant and well used.

• What are the long term benefits of participation in out-of-school time programs? There is little longitudinal information to date that examines this question; finding data to answer it will be critical to ensuring policymakers’ and communities’ commitment to these programs.

• What are the most appropriate and important outcomes to measure? Although research on outcomes of after school programs is increasing, there is still little known about program effects, particularly across the developmental continuum and over time.

In this era of accountability, evaluation and research that demonstrate program success will ultimately drum up financial support for programs and build credibility in the field. Although some feel that policy and practice need a greater selection of research to draw upon, the out-of-school time field now has enough credible knowledge to establish promising practices.

Afterschool Alliance On October 12, 2000, the Afterschool Alliance sponsored the LightsOn Afterschool! project as part of their efforts to build awareness and expand resources for after school programs. The Alliance is an emerging group of public, private, and nonprofit groups that grew out of a partnership between the Charles Stewart Mott Foundation and the U.S. Department of Education. The Afterschool Alliance hopes to raise national and local awareness about the importance of after school programs so that all children will have access to top quality, affordable after school programs by 2010. Contact the Afterschool Alliance, P.O. Box 65166, Washington, DC, 20035; 202-296-9378; www.afterschoolalliance.org

Connecting Evaluation and Research to Policy and Practice
If evaluation and research findings on after school programs are to affect practice and policy, it is critical that they become, as one participant noted, a “part of the water supply.” This requires work up front to ensure that evaluators and researchers are answering the important questions facing individual programs, policymakers, and the field at large. It also requires post-evaluation work to make sure that findings are accessible to policy and practice audiences. Many with whom we spoke noted that evaluation and research information is making its way into policy and practice discussions more now than in years past.

Research and evaluation have the potential to affect practice and influence legislative initiatives on both regional and national levels. Although policymaking in out-of-school time has historically been at the state and local levels, it is beginning to surface as a national priority and policymakers are looking for guidance as they invest dollars. Reports documenting effective programs that measure long term outcomes and demonstrate success in reaching goals are especially useful for policymakers. Models of successful collaborations among programs are of particular interest to practitioners because, more often than not, they are financed by multiple funders and held responsible for a range of outcomes. As is the case in the family and children’s services field, a new challenge facing out-of-school time programs and collaborative efforts is the documentation and demonstration of the value added by each collaborating partner program.

The diversity of the after school field makes connecting researchers and evaluators with those working in policy and practice difficult. At the local level, evaluators will need to work closely with program staff to identify the salient questions, issues and outcomes of greatest relevance to those working in after school programs.

Building Infrastructure to Improve and Share Evaluation Practices and Findings
To tackle these challenges, we must develop ways to support evaluation and research and their connections to policy and practice. Beyond that, we need to use those connections to continue to develop and improve programs. Examples include:

• The Center for the Study of Evaluation (CSE) at the University of California, Los Angeles, found, as they evaluated LA’s BEST initiative, that local capacity building is both an important focus and challenge. Through development of locally-relevant measures and data collection systems, CSE hopes to internalize evaluation in local sites, and use that information for the national evaluation. From the perspective of local evaluators, this connection and support is essential.

• Texas A&M developed a local university consortium to help programs improve their evaluation. A team of university-affiliated evaluators worked closely with program practitioners to define program goals, elements to achieve them, and indicators of success.

• The U.S. Department of Education developed a continuous improvement manual to assist 21st Century Community Learning Centers program grantees in collecting evaluative data. The Department is also conducting a longitudinal evaluation of the program and will report its performance to Congress.

• The US Department of Health and Human Services Child Care Bureau is building a child care research data archive that will serve as a repository for research and provide technical assistance to policymakers and practitioners as they locate and use information from the studies.

• Harvard Family Research Project is developing an evaluation data collection and dissemination strategy to centralize evaluation knowledge at the national and local level by creating an after school program/initiative evaluation database. The database is a compilation of evaluation designs, data collection methods, outcomes, and findings of both large- and small-scale after school initiatives. Information will be shared broadly via the Web, a series of research publications, and policy and practice briefs that will promote understanding and discussion of evaluation in after school programs.

Next Steps
In order to address the challenges raised, it is imperative to channel existing momentum around out-of-school time into the development of the field. This will require collective vision, strategic planning, and meaningful collaboration. The focus groups and meeting we conducted provided an important opportunity for researchers and evaluators to hear the information needs of policymakers and practitioners and learn about some important evaluation and research related infrastructure work currently being done. We hope that these discussions will mark the beginning of continued conversations to build the field of out-of-school time.

Karen Horsch, Senior Researcher, HFRP
Kathleen Hart, Research Analyst, HFRP