Search Publications |  The Evaluation Exchange |  FINE Network |  OST Database

HFRP > Evaluation > The Evaluation Exchange > Issue Archive > Evaluating Out-of-School Time > Does Youth Participation in Out-of-School Time Activities Make a Difference?

Volume IX, Number 1, Spring 2003

Issue Topic: Evaluating Out-of-School Time

Theory & Practice

Does Youth Participation in Out-of-School Time Activities Make a Difference?

Sandra Simpkins, HFRP Consultant and Research Investigator at the University of Michigan, integrates findings from academic research and program evaluation to provide a comprehensive look at the relationship between participation in out-of-school time (OST) activities and positive youth outcomes, and points to new directions for OST research and evaluation.

In the last few decades, there has been a surge of public and research interest in the impact of youth’s participation in out-of-school time (OST) activities. Researchers and practitioners argue that high quality, structured OST programs are environments that have the potential to support and promote youth’s development because they: (a) situate youth in safe environments, (b) prevent youth from engaging in delinquent activities, (c) teach youth general and specific skills, beliefs, and behaviors, and (d) provide opportunities for youth to develop relationships with peers and mentors. In fact, there is increasing evidence that youth’s participation in quality OST activities influences their current outcomes, which, in turn, impact outcomes into adulthood (Gambone, Klem, & Connell, 2002).

However, documenting the associations between youth outcomes and activity participation is challenging due to the diversity of OST programs and youth’s experiences in those programs. First, youth participate in a wide range of OST programs, from large-scale mentoring programs like Big Brothers Big Sisters, to small-scale, single-site programs such as a school football team. Second, the amount of time youth spend in OST programs and the quality of these programs can vary dramatically within and across activities. Third, youth’s reasons to participate and the role youth have in OST program activities (e.g., leader vs. participant) differ. Despite these challenges, there is a growing collection of academic research and program evaluations that converge on the same conclusion—youth’s participation in OST activities does matter in important ways.

Related Resources Eccles, J., & Gootman, J. A. (Eds.). (2002). Community programs to promote youth development. Washington, DC: National Academy Press. This report explores the role of youth development programs and how best to design programs that enable youth to develop into healthy, happy, and productive adults. Policy, practice, and research recommendations to address the developmental needs of youth are included. www.nap.edu/catalog/10022.html Harvard Family Research Project. Out-of-School Time Program Evaluation Database.  Larson, R. W. (2001). How U.S. children and adolescents spend time: What it does (and doesn’t) tell us about their development. Current Directions in Psychological Science, 10(5), 160–164.

A Note on Methodology for This Review
Prior to presenting research and evaluation results that support the above statement, it is important to describe the set of studies included in this review. Over 75 published and unpublished papers, including program evaluations, were identified for possible inclusion. Screening for our first criteria, scientific merit, 25 studies were first selected because they either used complex designs (e.g., experimental, quasi-experimental, longitudinal) or measured an array of youth development or OST program indicators. From this set of 25 studies, 10 studies were selected to ensure that this review covers a range of youth development indicators, OST program and participation indicators, group differences (e.g., family income, grade levels), and evaluation and academic research.¹

Participation Matters for Academic Success
Youth’s participation in OST activities is often predictive of academic success as measured through test scores, absenteeism, school dropout rates, homework completion, and school grades (Eccles & Barber, 1999; Gore, Farrell, & Gordon, 2001; Marsh, 1992). For example, Posner and Vandell (1994) found that academic activities with adult OST staff predicted children’s school grades for conduct and most subject areas. Results from program evaluations also suggest that these relationships are particularly consistent if the program focuses solely on academics or if it has a central educational component, such as homework help or educational enrichment activities (Huang, Gribbons, Kim, Lee, & Baker, 2000).

Participation Matters for Social Development
Indicators of academic success have received more attention than indicators of social, moral, and physical development. However, results from studies that examine nonacademic youth outcomes suggest that youth’s participation in OST activities is related to multiple indicators of positive social development.

Several studies have noted that OST activity participation is associated with multiple aspects of youth’s friendships, including the number of friends, the quality of those friendships, and who those friends are (e.g., Eccles & Barber, 1999; Grossman, Resch, & Tierney, 2000; Rodriquez, Hirschl, Mead, & Groggin, 1999). In addition, participation is linked to fewer feelings of loneliness and depression and less problem behavior (Grossman et al.; Gore et al., 2001), although some results suggest that sports participation is associated with higher alcohol use in adolescence (Eccles & Barber).

There is also evidence that OST program participation is related to other indicators of positive social and moral development, such as communication skills and values, but the number of studies examining each indicator is quite small (e.g., Rodriquez et al., 1999). Even though many programs are under increasing pressure to demonstrate measurable academic achievement results due to the No Child Left Behind Act, it is important to continue research in these and other critical nonacademic areas of youth development.

Possible Factors
These studies also suggest that there are at least two factors which may affect the relationship between OST program participation and outcomes: age and socio-economic status (SES).

Age. The associations between activity participation and outcomes appear to be stronger for adolescents than elementary school children. For instance, Posner and Vandell (1994) found that third graders’ participation in non-sports-related lessons was not significantly associated with math and reading grades, but was positively associated with conduct grades, work habits, and grades in other subjects besides math and reading. In addition to these findings, researchers have found that elementary school children’s activity participation and academic achievement were positively related for children at some grade levels (e.g., third), but not others (e.g., second, fourth; e.g., Ross, Lewis, Smith, & Sterbin, 1996).

SES. The associations between youth’s participation and positive youth development outcomes appear to be stronger for youth in low-income versus middle-income households (Marsh, 1992). This is not to say that activities for middle-income youth do not matter. On the contrary, the participation of middle-income youth has been found to be positively associated with peer relationships and beliefs concerning particular domains, such as interest and self-concept of ability in academics (Eccles & Barber, 1999; Marsh). These outcomes, in turn, impact development through adolescence and into adulthood. Activities for youth in low-income households may have a larger impact because the alternative home and neighborhood environments are typically less enriching and more dangerous than for middle-income youth.

Directions for Future Research
Given what we know, what research would significantly contribute to our current knowledge about the impact of specific OST activities on positive youth development? There are numerous promising avenues that this research could follow. Four directions, in particular, have the potential to clarify and build on current findings.

Process. We need to use theory-driven hypotheses and analyses to examine the processes behind how activity participation and development influence each other. For example, do we expect activity participation to be related to youth’s school grades because program youth gain new knowledge and cognitive skills, or because they have a mentor that inspires their educational goals and renews their commitment to school, or other reasons? Although research in this area is in its early stages, the theoretical and empirical work by Larson (2000) on adolescents’ sense of initiative and emotional experiences and Eccles on identity and task beliefs (Eccles & Barber, 1999) have been very instructive.

Indicators of Participation. To date, many researchers have grouped youth into one of two categories: youth who participate in OST programs and youth who do not. Although these groups have been and will continue to be useful in our understanding of OST programs, they overlook many details concerning the quantity of youth’s participation. For instance, some researchers have found it essential to conceptualize the quantity of program participation in terms of weekly or yearly “dosage” of participation, i.e., the amount of time youth participate (Huang et al., 2000; Ross et al., 1996). In addition, researchers have questioned whether there is a threshold or particular amount of time that is necessary for programs to have an impact (Gambone et al., 2002).

Program Quality. OST programs include multiple activities and qualities. We have yet to understand which aspects of the program account for the relationship between participation and outcomes. Researchers have begun to look at which qualities are associated with youth’s experiences in activities (Rosenthal & Vandell, 1996) and youth development (Beckett, Hawken, & Jacknowitz, 2001). For example, some researchers have examined the effects of interventions on staff quality, such as coaches’ behavior. Findings suggest that interventions on coaches’ behavior change youth’s enjoyment of and persistence in sports activities (e.g., Smith, Smoll, & Christensen, 1996).

Research Design. Many studies of OST programs are cross-sectional and correlational. These types of studies and other non-experimental evaluations contribute to our understanding of the associations between program participation and outcomes, program quality, and continuous program improvement. However, more rigorous designs are essential to addressing selection effects and differentiating program effects from normal development (Beckett et al., 2001). Like other social science disciplines, in order to understand the impact of activities on outcomes, the field needs to continue to use rigorous designs, such as experimental, quasi-experimental, and longitudinal studies.

See also in this issue Doing What Works: Scientifically Based Research in Education and The Evaluation Exchange Special Report on the 21st Century Community Learning Centers National Evaluation. We plan to publish an expanded version of this paper, with a more comprehensive review of research and evaluation on the relationships between activity participation and youth outcomes, on our website in July 2003. To be notified when it is available, sign up for our OST notification email.

References

Beckett, M., Hawken, A., & Jacknowitz, A. (2001). Accountability for after-school care: Devising standards and measuring adherence to them. Santa Monica, CA: RAND Corporation.

*Eccles, J., & Barber, B. L. (1999). Student council, volunteering, basketball, or marching band: What kind of extracurricular participation matters? Journal of Adolescent Research, 14(1), 10–43.

Gambone, M. A., Klem, A. M., & Connell, J. P. (2002). Finding out what matters for youth: Testing key links in a community action framework for youth development. Unpublished manuscript, Philadelphia: Youth Development Strategies, Inc. and Institute for Research and Reform in Education.

*Gore, S., Farrell, F., & Gordon, J. (2001). Sports involvement as protection against depressed mood. Journal of Research on Adolescence, 11, 119–130.

*Grossman, J. B., Resch, N. L., & Tierney, J. P. (2000). Making a difference: An impact study of Big Brothers Big Sisters. Unpublished manuscript, Philadelphia: Public/Private Ventures.

*Huang, D., Gribbons, B., Kim, K. S., Lee, C., & Baker, E. L. (2000, June). A decade of results: The impact of the LA’s BEST after school enrichment initiative on subsequent student achievement and performance. Unpublished manuscript, Los Angeles: UCLA Center for the Study of Evaluation, Graduate School of Education & Information Studies, University of California.

Larson, R. W. (2000). Toward a psychology of positive youth development. American Psychologist, 55(1), 170–183.

*Marsh, H. W. (1992). Extracurricular activities: Beneficial extension of the traditional curriculum or subversion of academic goals? Journal of Educational Psychology, 84(4), 553–562.

*Posner, J. K., & Vandell, D. L. (1994). Low-income children’s after-school care: Are there beneficial effects of after-school programs? Child Development, 65(2), 440–456.

*Rodriguez, E., Hirschl, T. A., Mead, J. P., & Groggin, S. E. (1999). Understanding the difference 4-H clubs make in the lives of New York youth: How 4-H contributes to positive youth development. Unpublished manuscript, Ithaca, NY: Cornell University.

*Rosenthal, R., & Vandell, D. L. (1996, September/October). Quality of care at school-aged child-care programs: Regulatable features, observed experiences, child perspectives, and parent perspectives. Child Development, 67(5), 2434–2445.

*Ross, S. M., Lewis, T., Smith, L., & Sterbin, A. (1996). Evaluation of the extended-day tutoring program in Memphis city schools: Final report to CRESPAR. Unpublished manuscript, Memphis, TN: University of Memphis.

*Smith, R. E., Smoll, F. L., & Christensen, D. S. (1996). Behavioral assessment and interventions in youth sports. Behavior Modification, 20, 3–44.

* Studies included in this review.

¹ These 10 studies are denoted by an asterisk (*) in the reference list.

Sandra Simpkins, Ph.D., HFRP Consultant and Research Investigator at the University of Michigan
Email: sandisim@umich.edu