The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume X, Number 1, Spring 2004
Issue Topic: Evaluating Out-of-School Time Program Quality
Spotlight
Getting Inside the “Black Box” to Measure Program Quality
Nicole Yohalem, Karen Pittman, and Alicia Wilson-Ahlstrom from the Forum for Youth Investment provide an overview of program quality assessment tools.
As the landscape of out-of-school time programs for children and youth has expanded and matured, the issue of program quality has moved to the front burner across research, practice, and policy contexts. While the quality of individual programs hinges on the capabilities of local practitioners and the resources available to them, the research community plays several important roles in defining, measuring, and improving quality.
First, research helps demonstrate that quality matters by surfacing connections between levels of program quality and program outcomes. Second, researchers play a critical role, along with evaluators and practitioners, in demonstrating that quality is measurable and in developing tools for measurement. Finally, research helps demonstrate that quality is malleable, by showing that, once named and assessed, specific program features related to quality can be improved over time.¹ This article demonstrates how quality can be measured and offers an overview of the quality assessment tools that are, or will soon be, available to programs that serve young people.
An array of tools is surfacing within different areas of practice across the youth fields that will allow practitioners and researchers in any setting where youth are engaged, from a public park to an after school journalism club, to measure the quality of that setting and the activities available there. The Forum for Youth Investment has collected and reviewed several existing and forthcoming program quality assessment tools, including standards lists and observation and survey instruments used in large-scale evaluations (see Table 1, below). While these tools alone do not reflect a comprehensive scan of the field, they do represent a range of purposes, methodologies, and institutional perspectives.
Table 1 (13KB Acrobat file) shows a high degree of consensus regarding specific elements of program quality. Significantly, this consensus exists despite the fact that the developers of these instruments represent a range of perspectives, including youth employment, after school, camping, and alternative education.
When we looked beneath the scaffolding of each tool listed, we found that quality is generally being defined and measured in terms of three broad areas: youth opportunities, staff practices and supports, and administrative and management policies. Within those areas, we also found significant consistency in the specific quality features included in each instrument.
Table 2 summarizes common constructs across tools reviewed. The specific elements of quality listed align closely with the features of positive developmental settings described by the National Research Council in 2002.² These parallels support the consensus that is building across research and practice communities about what it takes to engage young people and what high quality, supportive settings look and feel like. This consensus signals an important developmental milestone for the youth development field, as discussions about the need to do something for youth shift to analyses of what to do and how to do it.
|
Youth opportunities for...
|
|
Staff practices and supports that promote...
|
|
Organizational policies and structures that promote...
|
Being able to define, measure, and improve youth program quality is critical, and progress made in both research and practice over the past several years is indeed promising. However, while mounting evidence shows that quality indeed matters, is measurable, and is malleable, the bottom line is that quality costs.
The effective development and use of quality assessment tools like those summarized in this article require significant time and resources on the part of the research community and programs themselves. Researchers are making tremendous efforts to refine, adapt, and make assessment tools more widely available; programs are implementing and fine-tuning continuous improvement systems. But, for further progress to be made, the policy and funding communities must recognize the importance of these efforts. Ensuring quality is costly; the costs of not doing so, in a relatively new and rapidly expanding field, are far greater.
¹ Forum for Youth Investment. (2003, July/August). Quality counts. Forum Focus, 1(1).
² Eccles, J., & Gootman, J. A. (Eds.). (2002). Community programs to promote youth development. Washington, DC: National Academy Press.
Nicole Yohalem
Email: nicole@forumforyouthinvestment.org
Karen Pittman
Email: karen@forumforyouthinvestment.org
Alicia Wilson-Ahlstrom
Email: alicia@forumforyouthinvestment.org
Forum for Youth Investment
The Cady-Lee House
7064 Eastern Avenue, NW
Washington, DC 20012
Tel: 202-207-3333
Website: www.forumforyouthinvestment.org