Volume XII, Number 1 & 2, Fall 2006

Issue Topic: Building and Evaluating Out-of-School Time Connections

Theory & Practice

Building and Evaluating Out-of-School Time Connections

Suzanne Bouffard, Priscilla Little, and Heather Weiss build a research-based case that a network of supports, with out-of-school time programs as a key component, are critical to positive learning and developmental outcomes for children and youth.

Out-of-school time (OST) programs are in the spotlight more than ever before due to calls for nonschool services to address achievement gaps, interest in promoting “21st century skills” such as critical thinking and technological fluency, and the upcoming reauthorization process for the No Child Left Behind Act, which includes the 21st Century Community Learning Centers program and other supplementary education services. Recent research studies have contributed to the growing evidence base about the benefits of OST programs for youth.1 Policymakers and the public see the value of OST programs, and parents report continued unmet demand for services.2 All of these trends make the case for continued investments in OST—not only in services for children and youth, but also in continued research and evaluation to support knowledge development and best practices.

Research and evaluation investments over the past several years have moved researchers, practitioners, and policymakers beyond the question of whether OST programs matter for youth to questions about why, how, and for whom these programs matter and matter most. As this issue of The Evaluation Exchange demonstrates, there has been a growing emphasis on understanding and promoting quality in OST programs. Increased investments in quality are evident in the growth of quality standards and assessment tools, professional development and resources for program staff, and new research and evaluation studies such as those featured in Evaluations to Watch (pages 28 through 34). Stakeholders of all backgrounds now acknowledge that youth need access to not just any programs, but to well-designed, high-quality programs. In fact, recent research shows that poor-quality programs potentially can do harm to participants.3

The emerging consensus on what constitutes quality includes positive staff–youth relationships; opportunities for skill-building and mastery; opportunities for youth engagement, voice, and decision making; and positive peer relationships. A cross-cutting theme among findings on quality is that connections matter. Relationships among staff, schools, families, youth, and communities are crucial. In addition, access to and equity of participation are critical issues and require the collaborative efforts of many stakeholders.4 Intentional linkages among the many settings and institutions in which youth learn and grow can improve recruitment and retention, contribute to program quality, and promote better youth outcomes.5

Complementary Learning
Such linkages are at the heart of what HFRP calls complementary learning. Complementary learning occurs when two or more institutions intentionally link with each other to improve learning and developmental outcomes for children and youth. These institutions include families, early childhood programs, schools, OST programs and activities, higher education, health and social service agencies, businesses, libraries, museums, and other community-based institutions.

Many after school programs link with schools by aligning curricula and sharing resources. Head Start and other early childhood programs connect with families through outreach and parenting workshops. Comprehensive initiatives like the Harlem Children's Zone in New York City link many services for children, youth, and families under one umbrella. Complementary learning can include linkages of many different types and scopes. Stakeholders may aim to achieve one strong linkage, several linkages of varying degrees, or a comprehensive system of supports. What is common to these approaches is that they all have an intentional strategy for linking multiple nonschool supports with each other and with schools in the service of shared goals for improving child and youth outcomes.

The complementary learning framework is part of a larger, more cohesive agenda for improving learning and development. For decades, researchers have acknowledged the importance of supports across multiple contexts in children's lives.6 Recent research has also begun to account for the constellation of settings and activities within the OST hours7 and the fact that different settings provide different developmental opportunities.8 Just as schools cannot do it alone,9 OST programs are necessary but not solely sufficient to support learning and development. Rather, they are one integral part of families' lives, of communities, and of education. Operating from a complementary learning framework can lead OST programs to be intentional about how they connect with these other institutions. Through complementary learning, OST programs can build shared missions and goals with other institutions, share resources and ideas, build stakeholder buy-in, and provide more coordinated services. They can build linkages—with schools, families, community-based institutions, and higher education—that are equal to more than the sum of their parts.

Complementary learning initiatives are growing—and so is the evidence that they have tangible benefits for youth, families, and communities. Recent research, much of it from the OST arena, points to the benefits of linking many different institutions and supports. Below, we describe recent research on linkages between OST programs and initiatives and other complementary learning agents. Each type of linkage has unique goals and implementation challenges. However, all types of linkages are associated with a common set of benefits, including recruitment and retention, improved program quality, and academic and social benefits. After reviewing the research on these benefits and on how to facilitate each type of linkage, we describe how multiple linkages can be integrated into a cohesive network of supports, which goes beyond two-way partnerships to a full vision of complementary learning.

OST–School Linkages
The Goals. In the context of a national dialogue about improving academic achievement and reducing achievement gaps, there is widespread interest in linking OST programs with schools. Such linkages can facilitate continuity of academic goals and approaches, provide remediation and enrichment, present academic content in nontraditional and experiential ways, and address implementation challenges by promoting resource-sharing. OST–school linkages come in many varieties. For example, programs share staff, resources, and curricula; encourage regular communication between program and school staff; align goals and curricula to state and school standards; and coordinate their academic content with school work.

The Benefits. Recent research supports the benefits of OST–school linkages. Several studies have found that such linkages are one of the common characteristics of high-quality, high-performing OST programs. Youth whose OST programs are linked with their schools demonstrate better academic and social outcomes.10 For example, one recent study found that when program staff had positive relationships with school principals, youth demonstrated greater improvements in homework completion, initiative-taking, peer relationships, and positive behaviors.11 OST–school linkages are also a successful strategy for recruitment.12 For example, school-based programs often eliminate transportation barriers, and many programs rely on school staff to identify eligible youth.

Tackling the Challenges. Despite the potential benefits of OST–school links, many programs find that linking with schools is challenging13 and some express concern about replicating the school day after school. However, recent initiatives aim to tackle implementation challenges (see, for example, the Evaluations to Watch article by Jennifer Maltby), and many new resources, as featured throughout this issue, share strategies for linking OST with school in ways that are engaging, effective, and sustainable.

OST–Family Linkages
The Goals. Facilitating positive relationships between families and OST programs is increasingly a focus of both practice and research. Many programs target a range of goals to engage and support families,14 and some new evaluation studies report rates of parent involvement (e.g., the Citizen Schools evaluation). One unique advantage is that OST–family linkages can provide an additional entry point for families to the many institutions in which their children learn. For example, OST staff can serve as liaisons between families and schools and between families and social service providers.

The Benefits. As with OST–school linkages, positive relationships between programs and families are one of the common features of high-quality programs15 and have multiple benefits. Families play an important role in determining whether youth participate in programs.16 As Suzanne Bouffard, Sandra Simpkins, and Carrie-Anne DeDeo describe in the Evaluations to Watch article on page 31 of this issue, a new study from HFRP shows that youth are more likely to participate when their parents are engaged in their lives and schooling, and are less likely to participate when their parents show low levels of support, involvement, and cognitive stimulation. OST–family linkages benefit not only youth but family members. They can increase family involvement in education, which is associated with improved academic achievement, and are associated with improved family relationships.17

Tackling the Challenges. In addition to providing evidence that connections with families matter, recent research and evaluation provide the following lessons and strategies for developing such connections: create program environments and events that welcome families;18 support families' needs as well as youth's; communicate and build trusting relationships with families; respect cultural diversity; hire and develop a family-focused staff reflective of the parent population; and build linkages between families, schools, and communities (e.g., by serving as a liaison between parents and school staff and by involving the PTA). This research demonstrates that OST–family linkages are feasible and need not require extensive resources. For example, one of the most common ways for staff to engage families is to initiate frequent informal conversations at pick-up time.19

OST–Community Linkages
The Goals. The category of “community” is broader than those of “school” and “family,” and we use the term inclusively. Community includes neighborhoods—both their organizations and individual members; youth-related services and agencies (e.g., the Department of Social Services, community health clinics, etc.); businesses; and cultural institutions, such as museums, libraries, and arts centers. By linking with these organizations, OST programs can avoid overlap of services, provide more choices for youth,20 and can leverage resources. For example, businesses provide financial support, volunteers, and apprenticeship opportunities; cultural institutions contribute innovative programming and field trip sites; and neighborhood organizations provide feedback and guidance on the support their members want and need. A unique goal of OST–community linkages is to promote better outcomes not just for youth, but also for other community members and institutions.

The Benefits. There is less research on the benefits of OST–community connections than on the other linkages described above. However, similar themes are emerging about quality and recruitment, and a few key studies have shaped our understanding of relationships between OST and communities. For example, we know that neighborhoods exert an influence on youth participation. Lower participation rates among low-income children are largely explained by the fact that they tend to live in disadvantaged, underresourced neighborhoods.21 They are also partly explained by the influence of neighborhood conditions on parenting behaviors and strategies: Parents in more dangerous neighborhoods are more restrictive of their children's activities and less likely to enroll them in OST activities.22 At the same time, community factors can have potentially positive effects on participation—for example, by serving as target recruitment areas for youth and staff.23 Some cities, like New York and Providence, are tracking the availability of OST experiences by neighborhood, in order to target resources and recruitment efforts accordingly, and to connect the most in-need youth with services (see the Spotlight article about Providence). Other potential benefits of OST–community linkages include helping community members to work together, and integrating OST programs into the fabric of the community.

Tackling the Challenges. Evaluation studies show that OST programs can link with community institutions in many ways and that thinking creatively about these linkages can leverage additional resources. For example, cultural institutions such as museums and libraries can provide innovative and efficient programming. An initial report from the Public Libraries as Partners in Youth Development initiative concluded that structured programs that train youth to work in libraries cannot only promote youth and workforce development, but can also benefit the community by increasing adults' awareness of, positive views about, and leadership around issues affecting youth.24 Research is also beginning to emerge on the positive role of linkages between OST programs and community businesses (see Ask the Expert). Although research on this topic is new, connections between OST and businesses are not, and they have long served as an important complementary learning linkage.25

OST–Higher Education Linkages
The Goals. Complementary learning means not only linking multiple supports at a single point in time, but linking over time. As such, one of the functions of complementary learning is to facilitate smooth transitions across developmental stages and contexts—from preschool to kindergarten, from elementary school to middle school, and from high school to college. Linkages between OST and universities can facilitate the postsecondary transition by educating youth about their options for the future and preparing them to apply to and succeed in college.

The Benefits. In a recent ACT survey of eighth through tenth graders, 78% of respondents reported that extracurricular activities were helpful in exploring postsecondary opportunities.26 A survey in Rhode Island found that adolescents were interested in OST opportunities that are focused on “future goals and aspirations.”27 Some OST programs help youth begin this exploration and preparation process early on, in middle school or even elementary school, which is critical for ensuring that college aspirations become a reality.28 OST–university linkages also provide other benefits to programs, including staff and volunteers, curriculum advice, and evaluation support.

Multiple Linkages: Putting It All Together
The growing research base cited above makes a strong case for many of the single-link components of complementary learning. However, complementary learning is broader than just bidirectional linkages between two institutions. It is a framework that includes a network of linkages. This is a unique contribution of complementary learning that distinguishes it from partnerships and other two-way connections. Understanding and acknowledging the constellation of possible complementary learning linkages can allow stakeholders to assess their current work within a larger vision and to help them reflect on and strategize about ways to build their linkages moving forward. Likewise, complementary learning linkages vary not only in their constellation, but also along dimensions such as depth or innovation. Consider a complementary learning example with one strong and innovative link with a focused objective, such as the Providence Children's Museum's family program (see box). Then consider a different example from New York City or New Jersey, where large-scale comprehensive initiatives are emerging that link OST programs with communities, cultural institutions, schools, families, and other contexts.29 All of these examples fit within the complementary learning framework.

Evaluating OST Connections
Amid these strides, the OST field faces a new challenge: building and evaluating OST linkages to create the connections that are necessary for turning knowledge and experience into quality and sustainability of OST programming. As the availability and sophistication of OST research and evaluation grows, there is more need than ever before to make connections across research and evaluation efforts, to connect the many stakeholders working to improve the nonschool-hours experiences for youth, and to connect research with policy and practice. These needs are reflected in two of the key tasks facing the OST field today: to build systems of support for OST and to promote effective connections between the many settings in which children and youth learn, including schools, OST programs, community institutions, and families. As we and others continue to document examples of complementary learning initiatives around the country, developing new evaluation approaches and conducting more assessment are critical to our understanding of how, when, and under what conditions linkages between OST programs and other institutions can add value to learning and development. Of particular interest and need are evaluations of multiple linkages—that is, initiatives that intentionally connect more than two institutions.

This work raises several challenging evaluation questions that require researchers, evaluators, and other stakeholders to think creatively about evaluation in order to learn important implementation lessons and to evaluate effectiveness and added value:

• With multiple components, how can the value-added of each institution and linkage be evaluated individually and collectively? In other words, is the whole greater than the sum of the parts?
• What would a theory of change for an evaluation of a multicomponent complementary learning effort look like?
• What would be the shared goals of a multicomponent complementary learning effort?
• How can existing self-assessment tools help programs to evaluate and improve their complementary learning efforts? How can they be modified to accommodate a complementary learning approach?
• Can OST programs and initiatives support school reform efforts? And, if so, how?

As the articles in this issue demonstrate, research and evaluation have played a critical part in the growth of OST programs and improvements in program quality. Over the past several years, the field has benefited from increasingly sophisticated research methods and knowledge. The value of research and evaluation for the field is clear, and the role of evaluation for both continuous program improvement and accountability is now well established. The field has learned valuable lessons from both large-scale rigorous studies and small-scale program evaluation and is actively applying this knowledge to policy and practice. As this article has documented, one clear lesson from a decade of research is that just as “schools can't do it alone,” neither can OST programs. However, there is growing consensus that a rich network of supports—one that includes OST programs along with families and community-based organizations—can make a difference. In the coming months, Harvard Family Research Project will continue to document places where OST connections, especially between after school programs and schools, are strong, effective, and replicable.

1 Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L. (2006). Out-of-school time programs: A meta-analysis of effects for at-risk students. Review of Educational Research, 76, 275–313; Mahoney, J. L., Lord, H., & Carryl, E. (2005). Afterschool program participation and the development of child obesity and peer acceptance. Applied Developmental Science, 9, 202–215; Zief, S. G., Lauver, S., & Maynard, R. A. (2006). Impacts of after-school programs on student outcomes: A systematic review for the Campbell Collaboration. Philadelphia: Campbell Collaboration.
2 Afterschool Alliance. (2004). America After 3 pm. Washington: Author; Duffet, A., Johnson, J., Farkas, S., Kung, S., & Ott, A. (2004). All work and no play? Listening to what kids and parents really want from out-of-school time. New York: Public Agenda.
3 Mahoney, J. L., Stattin, H., & Lord, H. (2004). Unstructured youth recreation centre participation and antisocial behaviour development: Selection influences and the moderating role of antisocial peers. International Journal of Behavioral Development, 28, 553–560.
4 Weiss, H. B., Little, P. M. D., & Bouffard, S. M. (2005). More than just being there: Balancing the participation equation. New Directions for Youth Development, 105, 15–32.
5 Lauver, S., Little, P. M. D., & Weiss, H. B. (2004). Moving beyond the barriers: Attracting and sustaining youth participation in out-of-school time programs. Cambridge, MA: Harvard Family Research Project; Vandell, D. L., Reisner, E. R., Brown, B. B., Dadisman, K., Pierce, K. M., Lee, D., et al. (2005). The study of promisnig after-school programs: Examination of intermediate outcomes in year 2. Madison, WI: Wisconsin Center for Education Research; Intercultural Center for Research in Education and the National Institute on Out-of-School Time. (2005). Pathways to success for youth: What counts in after-school: Massachusetts After-School Research Study (MARS) report. Boston, MA: United Way of Massachusetts Bay.
6 Bronfenbrenner, U. (1977). Toward an experimental ecology of human development. American Psychologist, 32, 513–531.
7 Vandell, Reisner, Brown, Dadisman, Pierce, Lee, et al., 2005.
8 Hansen, D., Larson, R, & Dworkin, J. (2003).What adolescents learn in organized youth activities: A survey of self-reported developmental experiences. Journal of Research on Adolescence, 13, 25–56.
9 Gordon, E. W., Brigdlall, B. L., & Meroe, A. S. (2004). Supplementary education: The hidden curriculum of high academic achievement. Lanham, MD: Rowman & Littlefield; Rothstein, R. (2005). Class and schools: Using social, economic, and educational reform to close the black–white achievement gap. Washington, DC: Economic Policy Institute.
10 Birmingham, J., Pechman, E. M., Russell, C. A., & Mielke, M. (2005). Shared features of high-performing after-school programs: A follow-up to the TASC evaluation. Washington, DC: Policy Studies Associates; Russell, C. A., & Reisner, E. R. (with Johnson, J. C., Rouk, Ü., & White, R. N.). (2005). Supporting social and cognitive growth among disadvantaged middle-grades students in TASC after-school projects. Washington DC: Policy Studies Associates.
11 Intercultural Center for Research in Education and the National Institute on Out-of-School Time, 2005.
12 Arbreton, A. J. A., Sheldon, J., & Herrera, C. (2005). Beyond safe havens: A synthesis of 20 years of research on the Boys & Girls Clubs. Philadelphia: Public/Private Ventures; Lauver, Little, & Weiss, 2004.
13 Miller, 2003.
14 Weiss, A. R., & Brigham, R. A. (2003). The family participation in after-school study. Boston, MA: Institute for Responsive Education.
15 Vandell, Reisner, Brown, Dadisman, Pierce, Lee, et al., 2005.
16 Lauver, Little, & Weiss, 2004.
17 U.S. Department of Education, Office of the Under Secretary. (2003). When schools stay open late: The national evaluation of the 21st-Century Learning Centers program, first year findings. Washington, DC: Author.
18 Kakli, Z., Kreider, H., & Little, P. M. D. (2006). Focus on families! How to build and support family-centered practices in after school. Cambridge, MA: Harvard Family Research Project, United Way of Massachusetts Bay, and BOSTnet; Weiss, A. R., & Brigham, R. A. (2003). The family participation in after-school study. Boston, MA: Institute for Responsive Education.
19 Weiss & Brigham, 2003.
20 Policy Studies Associates. (2006). Everyone plays: A review of research on the integration of sports and physical activity into out-of-school time programs. Washington, DC: Author.
21 Wimer, C., & Dearing, E. (2005). Selection into out-of-school time activities: The role of family contexts within and across neighborhoods. Poster session submitted to Society for Research on Adolescence 2006 Bienniel Meeting, San Francisco, CA.
22 Furstenberg, F. F., Jr., Cook, T. D., Eccles, J., & Elder, G. H. (1998). Managing to make it: Urban families and adolescent success. Chicago: University of Chicago Press.
23 Lauver, Little, & Weiss, 2004.
24 Spielberger, J., Horton, C., & Michels, L. (2004). New on the shelf: Teens in the library. Chicago: University of Chicago, Chapin Hall Center for Children.
25 Mitchell, L. (2005). Increasing the bottom line by supporting families. The Evaluation Exchange, 11(1), 15.
26 Wimberly, G. L., & Noeth, R. J. (2005). College readiness begins in middle school: ACT policy report. Iowa City, IA: ACT.
27 Forum for Youth Investment (2003). High school after-school: What is it? What might it be? Why is it important? Washington, DC: Author.
28 Wimberly & Noeth, 2005.
29 For information on the evaluations of the New York Department of Youth and Child Development's OST initiative and the New Jersey After 3 initiative, visit the Policy Studies Associates website at www.policystudies.com/studies/indexyouth.html.

Suzanne Bouffard, Research Analyst, HFRP

Priscilla Little, Associate Director, HFRP

Heather Weiss, Founder and Director, HFRP