Volume XII, Number 1 & 2, Fall 2006

Issue Topic: Building and Evaluating Out-of-School Time Connections

Spotlight

Using Quality Assessment Tools to Evaluate OST Linkages

Helen Westmoreland from HFRP discusses how OST programs are using quality assessment tools to evaluate and promote linkages with families, schools, and communities.

Linkages between out-of-school time (OST) programs, families, schools, and communities are increasingly common. As other articles in this issue illustrate, these linkages exist in a variety of forms and are core indicators of program quality. They are associated with a range of positive outcomes and play a critical role in creating a network of complementary learning supports. As the consensus about the importance of linkages with families, schools, and communities grows, OST stakeholders need resources for evaluating them and for guiding program improvement efforts. One strategy for evaluating and improving linkages is the process of quality assessment.

What is Quality Assessment?
Quality assessment is a process that measures a program or initiative's adherence to agreed-upon standards of quality. One of its main purposes is to guide continuous program improvement by clearly communicating expectations, outlining goals, and providing benchmarks to track progress. Quality assessment is also useful in identifying program needs (e.g., professional development), building buy-in and shared vision among stakeholders, and meeting accountability requirements.

Many OST organizations and other stakeholders—including researchers, foundations, city and state agencies, and intermediary organizations—have created tools that support the quality assessment process. These tools range from self-assessment checklists to observation instruments and include hierarchically organized elements of program quality. Key thematic areas are called categories and provide an overarching framework of important quality components. Each category is supported by a set of standards that describe conditions of quality for the program, its participants, and all stakeholders. A few quality assessment tools also include indicators, which are specific measures that quantify the attainment of quality standards.1

Quality assessment tools assess a wide range of program quality characteristics and reflect the growing consensus about the essential elements of quality (e.g., physical and psychological safety, positive relationships, and opportunities for youth voice). They increasingly include whole categories of standards related to linkages with other institutions, such as schools, and provide an opportunity for programs to assess and build these important elements of complementary learning.

Quality Assessment of Linkages
Harvard Family Research Project (HFRP) recently conducted a national scan of OST program quality assessment tools to uncover trends in the policies and practices that programs are using to define and assess quality. Through literature and Internet searches, surveys, and key informant interviews, we identified 42 quality assessment tools.2 We then conducted a content analysis to identify the kinds of standards assessed by each tool and to glean common themes across the tools. As part of the content analysis, we compiled and categorized the standards related to linkages with families, schools, and communities.

There was considerable variation in how quality assessment tools categorized and assessed linkages. Some tools designated specific categories for each type of linkage, while others included all linkages within one broad category, and still others nested linkages within other categories of standards, such as program administration. We focus our content analysis for this article on the first two types—because tools that designated a higher order category for linkages tended to include more specific standards and indicators to guide assessment.

We found that 17 of the 42 quality assessment tools included categories of standards that explicitly addressed linkages with families, schools, and communities (see Table 1). Within these 17 tools, we identified a total of 154 standards related to the three types of linkages. These standards varied in depth and breadth. For example, although fewer tools included standards about OST–school linkages, these tools often articulated more specific and varied policies and practices than those for OST–community linkages. Across these categories and despite their diversity, our review uncovered common examples of how OST programs are defining and assessing quality linkages with families, schools, and communities.

OST–Family Linkages

• The program has a plan for family involvement, which outlines roles and has been activated, reviewed, and updated for effectiveness.
• The program effectively communicates important program information across language groups to families participating in and connected to the program.
• The program offers opportunities for staff and families to discuss individual participants.
• The program welcomes and encourages family participation in the program, through:
  • Events, activities, and celebrations.
  • Decision making and program planning.
  • Program assessment and evaluation.
  • Informal conversations with staff and visits to the site.
• The program provides youth and their families with information about community resources and assists them in connecting with these resources.
• The program provides learning opportunities for the participants' families.

OST–School Linkages

• The program incorporates programming that integrates and complements school-day activities for a holistic approach to youth development, through:
  • An aligned vision.
  • Funds and resources.
  • Complementary, not competitive, scheduling, recruiting, and programming.
  • Staffing and professional development.
  • Curriculum and standards-based testing.
• Program staff communicate and connect with school staff regularly, regarding:
  • Resources (funding and in-kind contributions) and facilities.
  • Individual students and their needs.

OST–Community Linkages

• The program engages in community collaborations that enhance program activities and/or sustainability.
• The program builds links between youth in the program and the community, by:
  • Educating youth about their community.
  • Encouraging youth to give back to their community through service projects.
  • Encouraging youth representation in community and government organizations.
  • The program builds links with businesses, municipal government, and local education institutions for funding, volunteering, programming, and/or advocacy.
• Program staff reflect the ethnic, cultural, racial, and linguistic diversity of the community.

Assessing OST Linkages
HFRP's scan of quality assessment tools sheds light on the pertinent elements of quality policies and practices for building linkages with families, schools, and communities; it also documents ways in which OST programs are assessing these linkages. Using one of the many quality assessment tools that examine OST linkages can help programs identify their strong partners, uncover areas where they need to improve their connections to others, and develop a plan for program improvement. As such, they are a vital diagnostic tool on the road to developing and supporting OST linkages with families, schools, and communities and an important framework for understanding how after school programs can support complementary learning.

1 Adapted from Little, P., DuPree, S., & Deich, S. (2002). Documenting progress and demonstrating results: Evaluating local out-of-school time programs. Cambridge, MA: Harvard Family Research Project and The Finance Project.
2 Click here to view additional resources related to quality assessment, including summaries of scanned tools, on HFRP's website.

Helen Westmoreland, Research Analyst, HFRP