The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume X, Number 1, Spring 2004
Issue Topic: Evaluating Out-of-School Time Program Quality
Ask the Expert
Designing Quality Assessment Tools in Early Childhood Education and Youth Development
Ted Jurkiewicz and Charles Hohmann of the High/Scope Educational Research Foundation discuss the lessons from developing High/Scope’s Preschool Quality Assessment Instrument that brought to bear on the design of the new Youth Program Quality Assessment tool.
Our design for the new Youth Program Quality Assessment tool draws on what we learned from our experience developing High/Scope’s Preschool Quality Assessment Instrument. The new tool assesses the extent to which practices used in out-of-school time (OST) services achieve high standards of developmental appropriateness. Practices in the domains of youth opportunities and youth supports are assessed through observations of workshops, classes, meetings, and other forms of OST service as they are delivered by youth-serving agencies, school districts, and other organizations.
Organizational practices focusing on structure, policy, activity, and setting are assessed both by observing activities and by interviewing program staff. Each domain of developmentally appropriate practice—youth opportunities, youth supports, structure, policy, activity, and setting—is built up from indicators and items that collectively represent consensus about best practices in the youth development field.
Because they have a longer history, efforts to create acceptable program quality measures in the early childhood field offer valuable lessons for similar attempts in the youth development arena. Our experiences developing tools for assessing practices in both fields suggest that the following steps are key to success:
- Start by formally gauging consensus around what constitutes developmentally appropriate practices. The early childhood and youth development fields house different schools of thought about what practices are essential to growth and development. However, variation may be greater in the early childhood field because its thought is more entrenched, with a longer history and more evidence to draw on than concepts in the youth development field. However, striving to consensus is still productive in either field if that consensus is widely vetted among leaders.
- Employ a conceptual framework that is meaningful to practitioners, researchers, and policymakers. A well conceived framework helps developers and potential users organize and parcel out the wide array of behaviors, events, decisions, and other phenomena that comprise practices or directly support them. Such a framework clarifies whether an indicator is focused on a specific aspect of practice in an activity or classroom, such as a human relationship, or on a specific aspect of organizational practice, such as a policy. Interestingly, frameworks have been a little easier to develop in early childhood than in youth development, because preschool programs take place in more standardized settings than is typical of youth programs.
- Create an iterative process of testing and revising the instrument that includes data analyses and feedback from practitioners. Practitioners, who know what language will best resonate in a particular field, can use their knowledge of program practices to tell whether measures make sense and are usable. Data analyses will help determine if measurement goals are being met. Instrument developers need to take practitioner knowledge and use it to mold indicators that generate meaningful assessments. Multiple attempts should be made to get feedback from practitioners about findings to make instrument revisions efficient and meaningful.
- Instrument developers must meet the needs of practitioners, researchers, and policymakers to develop acceptable assessment tools. Policymakers are more likely to listen, understand, and respond appropriately to demands from youth development professionals if tools are valid, useful, usable, and supportive of discussion about what is being measured using a common language. To meet all of these goals, the development process must address the concerns of practitioners and researchers by integrating their perspectives, experiences, and expertise effectively.
Carrying out an instrument development process that integrates the perspectives, experiences, and expertise of researchers and practitioners does not come easily, or cheaply, if it is done well. However, the first step is to design such a process by building on lessons like those we have described.
Ted Jurkiewicz
Email: tjurkiewicz@highscope.org
Charles Hohmann
Email: chohmann@highscope.org
High/Scope Educational Research Foundation
600 North River Street
Ypsilianti, MI 48198
Tel: 734-485-2000
Website: www.highscope.org