The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume X, Number 1, Spring 2004
Issue Topic: Evaluating Out-of-School Time Program Quality
New & Noteworthy: Expanded Web Only Version
In this section HFRP offers a selected list of new tools for evaluating out-of-school time (OST) programs.
The After-School Corporation is sponsoring the Promising Practices Initiatives to document and disseminate effective practices of after school programs. The initiatives include a series of resource briefs and tool kits that focus on staffing and professional development. Also included is the Citigroup Success Fund for Promising Practices in After-School Programs, which provides awards for programs to write about their promising practices for a wide audience of after school practitioners. www.tascorp.org
American Youth Policy Forum. (2003). Shaping the future of American youth: Youth policy in the 21st century. Washington, DC: Author. This report brings together 14 leading authors to examine the future of American youth through the lenses of policy, practice, and research. The report addresses school reform, community service, strengths-based youth development, and the systems integration of youth development programs and resources. www.aypf.org/publications/shaping_future_youth.pdf
Bartko, W. T., & Eccles, J. S. (2003). Adolescent participation in structured and unstructured activities: A person-oriented analysis. Journal of Youth & Adolescence, 32, 233–241. This article extends previous research on how participating in individual activities affects youth outcomes by examining participation in patterns of multiple activities. It describes the most common patterns of participation, characteristics of children and families associated with these patterns, and the relationships between these patterns and children’s academic, emotional, and behavioral functioning.
Blyth, D. A., & Borden, L. M. (2003). Stimulating research, promoting youth development: Final report of the National Youth Development Research Response Initiative. Minneapolis, MN: University of Minnesota Center for 4-H Youth Development. This report reviews the need for multidirectional connections between youth developmental research and practice, as well as strategies for addressing this need. Five recommendations for promoting sustainable research-practice links are targeted toward a range of stakeholders, including practitioners, funders, policymakers, researchers, and universities. www.fourh.umn.edu/nyddri/NYDRR_exec_summary.pdf
Boston’s After-School for All Partnership. (2003). Strategies for success: Strengthening learning in out-of-school time. Boston: Author. This report summarizes the responses of seven organizations to the question of how best to promote learning in out-of-school time initiatives. Full-length papers from each organization are also available on the After-School for All Partnership’s website. www.afterschoolforall.org/news/research.html
Brown, E. G., McComb, E. M., & Scott-Little, C. (2003). Expanded learning opportunities programs: A review of research and evaluations on participant outcomes in school readiness and after-school programs. Greensboro, NC: SERVE. This report synthesizes research on school readiness initiatives and after school programs.
California Tomorrow. (2003). Pursuing the promise: Addressing equity, access, and diversity in after school and youth programs. Oakland, CA: Author. In addition to addressing access, equity, and diversity, this report also discusses trends and challenges in the after school field, youth and community perspectives on youth needs, promising approaches from around the nation, and recommendations for policy and practice. www.californiatomorrow.org/resources/publications/index.php?cat_id=3&pub_id=31
To help identify programs and practices that effectively serve youth, Child Trends offers the Guide to Effective Programs for Children and Youth. The guide presents the results of completed evaluations and research studies using a user-friendly graphic format that categorizes programs according to both age group and program type, and features both programs that have demonstrated success and those that have not. www.childtrends.org/Lifecourse/index.htm
Child Trends has also produced a series of resources for understanding and improving positive development among adolescents. The Research Tools to Improve Youth Development resource includes the Youth Outcomes Compendium and the series What Works: Programs for Teens. Together these resources employ a youth development framework to summarize current knowledge about the types of outcomes that matter for youth, the types of programs and practices that affect these outcomes, and strategies for measuring practices and outcomes. www.childtrends.org/what_works/clarkwww/clarkoverall_intro.asp
Chinman, M., Imm, P., & Wandersman, A. (2004). Getting to outcomes 2004: Promoting accountability through methods and tools for planning, implementation, and evaluation. Santa Monica, CA: RAND Corporation. Researchers at RAND designed this manual to aid practitioners in improving the quality of youth prevention programs. Its 10-step approach includes strategies for assessing needs, implementing programs, conducting evaluation, and promoting continuous improvement. The primary focus of the report is on substance abuse prevention, with implications for other types of prevention programs as well. www.rand.org/publications/TR/TR101
Dworkin, J. B., Larson, R., Hansen, D. (2003). Adolescents’ accounts of growth experiences in youth activities. Journal of Youth & Adolescence, 32, 17–26. This article presents the results of research conducted at the University of Illinois that used focus groups with high school students to better understand and describe how adolescents view out-of-school activities as contexts for development. Youth reported multiple developmental processes during activities, including both personal and interpersonal growth.
Fight Crime: Invest in Kids California. (2004). California’s next after-school challenge: Keeping high school teens off the street and on the right track. Washington, DC: Author. The authors of this report argue that while programs for teens reduce problem behaviors, insufficient funding has lead to a lack of available programming throughout California. www.fightcrime.org/ca/highschool/index.php
Fletcher, A. C., Nickerson, P., & Wright, K. L. (2003). Structured leisure activities in middle childhood: Links to well-being. Journal of Community Psychology, 31, 641–659. This article examines the outcomes associated with elementary students’ participation in clubs, sports, and church activities. As with studies of older youth, results suggest that different types of activities are associated with different outcomes.
Froschl, M., Sprung, B., Archer, E., & Fancsali, C. (2003). Science, gender, and afterschool: A research-action agenda. New York: Educational Equity Concepts and Academy for Educational Development. This report explores how after school programs can help increase girls’ pursuit of science and technology-related courses and careers. www.edequity.org/sgaagenda.pdf
The Forum for Youth Investment (FYI) offers several new publications related to quality in youth programming. The first issue of their latest newsletter, Forum Focus, includes articles and interviews on the role of research in defining and assessing quality, as well as portraits of quality programs. FYI’s Out-of-School Time Policy Commentary series also addresses key components of quality. See in particular Commentary #5, Inside the black box: Exploring the “content” of after-school, and Commentary #6, Participation during out-of-school time: Taking a closer look. www.forumforyouthinvestment.org
Geiger, E., & Britsch, B. (2003). Out-of-school time program evaluation: Tools for action. Portland, OR: Education, Career, and Community Program, Northwest Regional Educational Laboratory (NWREL). NWREL evaluators have produced this how-to guide for designing and implementing program evaluations. The guide provides useable surveys, focus group questions, and strategies for data analysis. www.nwrel.org/ecc/21century/publications/ost_tools.pdf
Guest, A., & Schneider, B. (2003). Adolescents’ extracurricular participation in context: The mediating effects of school, communities, and identity. Sociology of Education, 76, 89–109. This article examines how youth activity participation is related to outcomes and how this relationship varies according to social and community contexts.
Hall, G., Israel, L., & Shortt, J. (2004). It’s about time: A look at out-of-school time for urban teens. Wellesley, MA: National Institute on Out-of-School Time. This report addresses issues of capacity building, effective practices of quality programs for adolescents, and the challenges of eliciting and sustaining teen participation. www.niost.org/AOLTW.pdf
Hall, G., Yohalem, N., Tolman, J., & Wilson, A. (2003). How afterschool programs can most effectively promote positive youth development as a support to academic achievement. Wellesley, MA: National Institute on Out-of-School Time (NIOST). Co-authored by NIOST and the Forum for Youth Investment, this report reviews how a positive youth development framework can help promote academic achievement and focuses on the needs of and strategies for after school programming in Boston. www.niost.org/WCW3.pdf
Halpern, R. (2003). Supporting the literacy development of low-income children in afterschool programs: Challenges and exemplary practices. New York: The Robert Bowne Foundation. Halpern presents research from program observations and previous studies on how high quality after school programs can promote literacy development, and illustrates exemplary practices with real-world examples from Chicago, New York, and Seattle. The report also addresses what after school programs should and should not be expected to accomplish and advocates a literacy strategy distinct from that provided in most school contexts. www.robertbownefoundation.org/pdf_files/occasional_paper_01.pdf
Hansen, D., Larson, R., & Dworkin, J. (2003). What adolescents learn in organized youth activities: A survey of self-reported developmental experiences. Journal of Research on Adolescence, 13, 25–55. This article reports findings from a study that used the new Youth Experiences Survey to illuminate the processes at work in the “black box” of youth activity participation. Researchers measured the impact of participation in structured activities on several developmental processes, and youth reported more learning in organized activities than in other contexts.
Harrison, P. A., & Narayan, G. (2003). Differences in behavior, psychological factors, and environmental factors associated with participation in school sports and other activities in adolescence. Journal of School Health, 73(3), 113–120. This article examines the multiple outcomes—including health and developmental indicators—associated with youth participation in various combinations of activities (sports only, other activities only, sports and other activities, or neither).
Herrera, C., & Arbreton, A. J. A. (2003). Increasing opportunities for older youth in after-school programs: A report on the experiences of Boys & Girls Clubs in Boston and New York City. Philadelphia: Public/Private Ventures. This report describes an initiative to build the potential of youth-serving institutions to make reaching older, hard-to-reach youth a priority. www.ppv.org/ppv/youth/youth_publications.asp?section_id=8
Hipps, J., Ormsby, C., Diaz, M., & Heredia, A. (n.d.). Evaluating after-school programs: The program evaluator’s multiple challenges. Oakland, CA: WestEd. WestEd evaluators use their experiences evaluating 21st Century Community Learning Centers sites to describe the important role of, and challenges faced by, after school evaluators. The report highlights several key issues for evaluators, including potential organizational hazards to program implementation, tensions between schools and community-based agencies, and disconnections between those responsible for securing funding and those responsible for program implementation. education.ucdavis.edu/cress/ccsp/download/hipps.pdf
Huebner, A. J., & Mancini, J. A. (2003). Shaping structured out-of-school time use among youth: The effects of self, family, and friend systems. Journal of Youth and Adolescence, 32, 453–463. Moving beyond the question of whether participation in structured activities benefits youth, this study asks what factors predict youth participation. Researchers examine 12 different variables across the domains of self-system factors, family factors, and peer factors.
Innovation Center for Community and Youth Development. (2003). Lessons in leadership: How young people change their communities and themselves. An evaluation of the Youth Leadership for Development Initiative. Takoma Park, WA: Author. This report is one of several new resources on youth activism and engagement offered by the Innovation Center. www.theinnovationcenter.org
Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. (2004). The effectiveness of out-of-school time strategies in assisting low-achieving students in reading and mathematics: A research synthesis. Washington, DC: Institute of Education Sciences, U.S. Department of Education. This meta-analysis of 53 research studies that used comparison or control groups finds evidence that after school programs have positive effects on both reading and math achievement. The report also reviews potential moderating factors. www.mcrel.org/PDF/SchoolImprovementReform/5032RR_RSOSTeffectiveness.pdf
London, J. K., Zimmerman, K., & Erbstein, N. (2003). Youth-led research and evaluation: Tools for youth, community, and organizational development. New Directions for Evaluation, 98, 33–45. This article describes the Youth-Led Research, Evaluation, and Planning (Youth REP) model developed by Youth in Focus. The model includes a method for training youth in leadership and evaluation, and for training adults in sharing decision making and evaluation responsibilities with youth.
Mahoney, J. L., Cairns, B. D., & Farmer, T. W. (2003). Promoting interpersonal competence and educational success through extracurricular activity participation. Journal of Educational Psychology, 95, 409–418. Using data on activity participation throughout high school, this study examined the academic and social-skills outcomes associated with continued patterns of participation.
Mahoney, J. L., Larson, R. W., & Eccles, J. S. (Eds.). (in press). Organized activities as contexts of child development: Extracurricular activities, after-school, and community programs. Mahwah, NJ: Lawrence Erlbaum Associates. This upcoming book, which includes contributions from multidisciplinary experts on out-of-school activity participation, reviews existing research, with emphasis on multiple developmental periods from middle childhood to early adulthood. The book is geared toward a broad audience and includes implications for policy and practice.
Miller, B. M. (2003). Critical hours: Afterschool programs and educational success. Brookline, MA: Miller Midzik Research Associates. Focusing on academic achievement, this report synthesizes research on the relationship between after school programs and children’s development. Results suggest that after school programs have positive and significant effects on achievement and motivation. www.nmefdn.org/uploads/Critical_Hours.pdf
National Institute of Child Health and Human Development (NICHD) Early Child Care Research Network. (2004). Are child developmental outcomes related to before- and after-school care arrangements? Results from the NICHD Study of Early Child Care. Child Development, 75, 280–295. The most recent publication from the Early Child Care Research Network reports findings on the predictors and outcomes of OST care over the two-year longitudinal period from kindergarten to first grade. Results showed that children who were consistently enrolled in extracurricular activities during this period performed better on standardized tests, taking into account family characteristics and prior intellectual functioning.
National Institute on Out-of-School Time. (2004). Making the case: A fact sheet on children and youth in out-of-school time. Wellesley, MA: Author. NIOST has released its most recent fact sheet, which covers issues such as child and family demographic statistics, how youth spend their time, and how OST programs are serving the needs of children and families. www.niost.org/publications/Fact%20Sheet%202006%20Feb9%20.pdf
The National Mentoring Partnership sponsors a series of online Community Forums for program organizers, researchers, and others to share information on effective strategies and practices in mentoring. To reach the Forum page, go to www.mentoring.org. Once there, click first on “Run a Program” and then on “Community Forum.”
The National Partnership for Quality Afterschool Programming promotes academic learning by supporting capacity building for after school programs and for research. In the interest of helping to turn “promising” after school sites into “exemplary” ones, the partnership—consisting of the Southwest Educational Development Laboratory and seven other organizations—offers various services, including identification of best practices sites, quality site validation, and technical assistance. The initiative offers local and national training on academic learning in afterschool and plans to produce technical assistance resource kits for programs and providers. www.sedl.org/afterschool/welcome.html
Nicholson, H. J., Collins, C., & Holmer, H. (2004). Youth as people: The protective aspects of youth development in after-school settings. Annals of the American Academy of Political and Social Science, 591, 55–71. This article reviews the multiple ways after school programming can contribute to positive developmental outcomes for youth, including outcomes such as self-awareness and confidence, creative expression, career development, and physical health.
Partee, G. L. (2003). Lessons learned about effective policies and practices for out-of-school time programming. Washington, DC: American Youth Policy Forum. This report is the product of a three-year study aimed at understanding the practices, supports, and challenges for promoting quality youth development opportunities. It presents data and insights from program observations and “learning events” held with policymakers, legislative aides, youth development organizations, practitioners, and youth. www.aypf.org/pressreleases/pr25.htm
Roth, J. L., & Brooks-Gunn, J. (2003). Youth development programs: Risk, prevention, and policy. Journal of Adolescent Health, 32, 170–182. This article presents the results of a review of high quality, rigorously designed research and evaluation studies of youth development programs. It examines (1) how youth development programs can be defined (along the three dimensions of goals, atmosphere, and activities), (2) the characteristics of each of these dimensions and the extent to which they are present in existing programs, and (3) the ways in which these specific program characteristics affect youth.
Verma, S. & Larson, R. (Eds.). (2003). Examining adolescent leisure time across cultures: Developmental opportunities and risks. [Special issue]. New Directions for Child and Adolescent Development, 99. This special issue of the journal, edited by two experts on out-of-school time, includes articles that explore patterns in the way adolescents in Japan, Korea, India, Europe, and the U.S. use their time.
Weiss, A. R., & Brigham, R. A. (2003). The family participation in after-school study. Boston, MA: Institute for Responsive Education. This study documents the results of a nationwide survey of program coordinators of a cohort of 21st Century Community Learning Centers programs to learn about efforts to encourage family involvement. www.responsiveeducation.org/pdf/afterschoolReportOne.pdf
Wimer, C., Post, M., & Little, P. Leveraging resources to promote positive school-CBO relationships. Afterschool Matters, 3, 13–21. This study reviews evaluation material concerning ways schools and community-based organizations can build and leverage physical, financial, social, and intellectual resources to develop and implement effective after school programs.
Zaff, J. F., Moore, K. A., Papillo, A. R., & Williams, S. (2003). Implications of extracurricular activity participation during adolescence on positive outcomes. Journal of Adolescent Research, 18, 599–630. Using the National Education Longitudinal Study, this article analyzes outcomes associated with longitudinal patterns of activity participation throughout high school and includes indicators of civic engagement, such as voting and volunteering.
New Resources From HFRPThe third and fourth publications in our series, Out-of-School Time Evaluation Snapshots, are now available:
The Snapshot series draws on the program evaluations featured in our Out-of-School Time Program Evaluation Database. Snapshots can be read or downloaded online and some are also available in hard copy. |
Suzanne Bouffard, Consultant, HFRP
Because of limitations on space, we were unable to include as many new and noteworthy resources as we would have liked in the print version of this issue. This online version includes all citations listed in the print version along with many others. If you like, you can sign up to be notified by email when the latest issue of The Evaluation Exchange is available online.