The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume X, Number 1, Spring 2004
Issue Topic: Evaluating Out-of-School Time Program Quality
Beyond Basic Training
Promoting Quality Out-of-School Time Programs Through Professional Development
Suzanne Bouffard from HFRP explores the contribution that staff development initiatives and evaluations make to improving the quality of youth programming.
Staffing is a key component of quality in out-of-school time (OST) programs. Many programs attribute their success to skillful providers, and research shows the importance of positive staff-child relationships for youth outcomes (Eccles & Gootman, 2002). Professional development can enhance the skills of both new personnel and longtime staffers. While the concept of professional development is not new, its significance for those working in OST is still emerging. As the professional development field begins to have a larger impact in youth programming, evaluation will continue to play an important role in measuring success and bringing effective initiatives to scale.
What Is Professional Development?
Professional development programs have the common goal of increasing the knowledge and skills of staff to improve youth outcomes. Many programs have the secondary goal of improving the quality and sustainability of the youth workforce. Professional development activities may be conducted before the job application process or later in the career cycle. Examples include higher education, new-staff orientation, in-service training, external seminars and conferences, apprenticeship programs that combine college courses with job experience, and informal resources such as newsletters and online discussion boards.
Research from education and early child care demonstrates that professional development is related to positive social and cognitive outcomes for youth (National Institute of Child Health and Human Development, 2001; Westat & Policy Studies Associates, 2001). In the OST field, professional development activities are also related to providers’ confidence and satisfaction (Center for School and Community Services, 2002) and form the backbone of the “career development ladder,” a new system that ties increases in staff salaries and responsibilities to ongoing education and experience.
Related ResourcesThis article is based on an upcoming HFRP brief entitled Promoting Quality Out-of-School Time Programs Through Professional Development. Part of our Issues and Opportunities in Out-of-School Time Evaluation series, this brief examines professional development in OST and related fields and provides a framework for evaluation tailored to OST professional development initiatives. To be notified when this brief is available on our website, sign up for our out-of-school time updates email. The Out-of-School Time Resource Center is housed in the University of Pennsylvania’s Center for Research on Youth and Social Policy. The Center aims to help connect OST programs with resources, and features professional development tools and research. |
A Multilevel Framework for Evaluation
Evaluation is a critical part of the professional development process because it identifies successful program elements that can be replicated and ineffective elements that call for improvement. However, evaluations of professional development have been sparse. Multiple barriers could be responsible for this scarcity, including limited time and resources; for example, educators and other youth workers often report feeling that evaluation of professional development wastes valuable staff time (Guskey, 2000). However, one of the most significant barriers may be the need for an evaluation framework that is tailored for professional development.
An evaluation framework for professional development must demonstrate the effect of an initiative at multiple levels of outcomes. Staff development experts agree that evaluations can and should assess the following four levels: (1) feedback from providers about satisfaction, (2) providers’ knowledge of youth development and best program practices, (3) the practices employed by program providers, and (4) positive developmental outcomes for youth and other stakeholders, such as families and communities (Guskey, 2000; Killion, 1998; Mizell, 2003). Each level can provide valuable information; however, only the fourth level can establish whether the ultimate goal was achieved, that is, whether the initiative had a positive impact on youth.
Most current evaluations of professional development in OST focus on the first level, in the form of post-training satisfaction surveys. These surveys cannot demonstrate impact on providers’ practices or youth outcomes, but they can illuminate providers’ opinions. Awareness of such opinions can help professional development planners to revise their programs, thereby increasing participation and engagement in the future.
A few initiatives, such as Building Exemplary Systems for Training Youth Workers (Center for School and Community Services, 2002), Making the Most of Out-of-School Time (Halpern, Spielberger, & Robb, 2001), and the North Carolina Quality Enhancement Initiative (Hall & Cassidy, 2002), have reported outcomes on the second and third levels. This encouraging trend has helped to establish the positive role that professional development plays in the way providers interact with youth and lead programs.
To date, no studies have examined the fourth level of how professional development impacts youth. Positive youth outcomes, while most conclusive, are also the most difficult to attain. A multitude of complex factors, both within and outside of OST programs, operate to influence youth, which makes it difficult to demonstrate that outcomes observed in level four are due specifically to staff training.
When embarking on an evaluation of a program’s professional development efforts, it is critical to begin with level one evaluation and build toward level four. Combining multiple levels and working toward the highest level will help programs meet accountability requirements as well as generate valuable feedback for improvement. Such a system of continuous improvement can contribute to high quality programs that benefit providers, youth participants, and communities alike.
References
Center for School and Community Services Academy for Educational Development. (2002). BEST strengthens youth worker practice: An evaluation of building exemplary systems for training youth workers (BEST). New York: Academy for Education Development.
Eccles, J. S., & Gootman, J. A. (Eds.). (2002). Community programs to promote youth development. Washington, DC: National Academy Press.
Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press.
Hall, A. H., & Cassidy, D. J. (2002). An assessment of the North Carolina school-age child care accreditation project. Journal of Research in Childhood Education, 17, 84–96.
Halpern, R., Spielberger, J., & Robb, S. (2001). Evaluation of the MOST (Making the Most of Out-of-School Time) initiative: Final report and summary of findings. Chicago: Chapin Hall Center for Children at the University of Chicago.
Killion, J. (1998). Scaling the elusive summit. Journal of Staff Development, 19(4).
Mizell, H. (2003). Facilitator: 10, Refreshments: 8, Evaluation: 0. Journal of Staff Development, 24(4), 10–13.
National Institute of Child Health and Human Development Early Child Care Research Network. (2001). Nonmaternal care and family factors in early development: An overview of the NICHD Study of Early Child Care. Journal of Applied Developmental Psychology, 2, 457–492.
Westat & Policy Studies Associates. (2001). The longitudinal evaluation of school change and performance (LESCP) in Title 1 schools. Final report, volume 2: Technical report. Washington, DC: Author. www.policystudies.com/studies/school/lescp.html
Suzanne Bouffard, Consultant, HFRP
Email: suzanne.bouffard@duke.edu