Search Publications |  The Evaluation Exchange |  FINE Network |  OST Database

HFRP > Evaluation > The Evaluation Exchange > Issue Archive > School-Linked Services > Evaluating School-Linked Services: Considerations and Best Practices

Volume lll, Number 2, 1997

Issue Topic: School-Linked Services

Theory & Practice

Evaluating School-Linked Services: Considerations and Best Practices

Karen Horsch of Harvard Family Research Project discusses forthcoming research on the evaluation of school-linked services based on the insights of nine evaluators.

Increasingly, the traditional organizational and political boundaries between education and social services are giving way to more integrated approaches that help children and their families. School-linked services are one program approach that seeks to address concerns about fragmented and duplicative services by offering a single point of entry—the school site. These efforts take many forms, and the goals of such programs are numerous. Most display a holistic approach to children; joint planning; shared service delivery; and collaboration and/or coordination.

Determining how school-linked services programs work, what their impact is, and whether they should be expanded, however, is difficult. We asked nine evaluators of a variety of school-linked services programs to identify some considerations and best practices related to evaluating outcomes, sustainability, and collaboration. We also asked them to provide some insights into evaluation design and data collection methods. Following is a summary of their responses:

Evaluating Outcomes

In this era of increased scrutiny of and calls for accountability in public programs, demonstrating the outcomes of school-linked services initiatives is vital.

The Evaluators Mary Geisz, Cornell University: Evaluated the Delaware Academy Student Health Program Robert Illback, R.E.A.C.H.: Evaluated the Kentucky Family Resource and Youth Services Center Program Scott Keir, the Hogg Foundation for Mental Health: Evaluated the School of the Future Initiative George Madaus and Mary Walsh, Boston College School of Education: Are evaluating the Gardner Extended Services School Ray Morley and James Veale, State of Iowa, Department of Education: Are evaluating the Iowa School-Based Youth Services Program Susan Philliber, Philliber Research Associates: Is evaluating the Missouri Caring Communities Initiative Mary Wagner and Shari Golan, SRI International: Evaluated the California Healthy Start School-Linked Services Initiative Constancia Warren and Anita Baker, the Academy for Educational Development: Are evaluating the New Jersey School-Based Youth Services Program Sam Whalen, The Center for Talent Development, Northwestern University: Is evaluating the Chicago Full Service Schools Initiative Harvard Family Research Project wishes to thank all who shared their insights with us.

Help to elucidate complex relationships and processes by using a theory of change approach. A theory of change approach requires programs to specify clearly their intended activities and the expected short- and longer-term outcomes of these activities. This approach is one way of beginning to clarify the “black box” of integrated social interventions. It enables evaluators and stakeholders to relate long-, intermediate-, and short-term goals with activities and processes and helps to clarify the linkages among them. A theory of change approach is best employed early in the process—ideally in the planning stage of the intervention. Evaluators employing such an approach must be willing to play the role of facilitator to articulate the program's theory of change.

Examine a variety of long-term outcomes. Evaluators of school-linked services programs must study a range of long-term outcomes to obtain a true picture of the impact of programs. Decisions about which outcomes to examine must be informed by the services and activities that programs actually provide as well as by consideration of the outcomes that stakeholders view as credible. In most cases, these include educational as well as social service results. Educational outcomes are critical to policymakers, school leaders, and parents but are often difficult to change. Consequently, evaluators are examining test scores, dropout rates, student behavior, and attendance and completion rates. They are also assessing important social service outcomes such as substance abuse, violence and sexual behavior, health status, and social skills.

Include credible intermediate outcomes and process measures. Most desired changes in social and educational well-being will take many years to achieve. Thus, evaluators point out the importance of identifying and measuring important shorter-term outcomes and processes. Evaluators of the Gardner Extended Services School (Boston, MA) note that other indicators which are or could be related to school success should also be used when examining education outcomes. These might include children's beliefs about the relationship of education to jobs, and school-related attitudes of parents, teachers, and community agency personnel. Evaluators of the Kentucky Family Resource and Youth Services Center Program have found that by moving from long-term social indicators to more proximal classroom indicators, they were able to relieve some of the demands on the program to demonstrate quickly reduction in dropout and substance abuse rates and enhanced school attendance and achievement.

Identify and understand the context for outcomes. Given the newness of multiservice initiatives such as those linked with schools, it is important that evaluators provide information that helps stakeholders understand the context for outcomes. Findings on outcomes should be carefully framed within a description of the program's actual operating context (including fiscal and philosophical support) and should take into account some measure of the actual level of services each participant received. Evaluators of the School of the Future (Texas) point out that, in some cases, indicators may get worse before they get better. For example, a successful program which retains students with the lowest academic standing in school may result in lower test scores school-wide, at least in the short term.

Evaluating Collaboration

By their nature, school-linked services require collaboration among the different entities providing services and between these entities and the school. While collaboration among different entities serving children and families is one of the most important factors in the success of integrating social and educational services (and is often a goal of these programs), it is also one of the most difficult to evaluate.

Examine the structure, nature, and image of collaboration. The structure of collaboration has to do with contractual arrangements and financing of service provision. The evaluators are examining the following: what services are provided, by whom, and when; the extent to which program staff and services are housed in school facilities; the availability of other school resources (personnel, expertise, in-kind support) for programs; the level of integration of school and program budgets; and the extent to which program and school staff members engage in collaborative strategic planning.

An assessment of the nature of collaboration entails an examination of how smoothly the different entities work together and to what extent cultures, goals, and interests are congruent. To assess this, some evaluators are examining the extent to which program staff participate in school-related committees, activities, and functions; school staff participate in program committees, activities, and functions; and program services are incorporated into the school's planned response to particular school problems. Through surveys and interviews, evaluators can also learn about how far school personnel are knowledgeable about and supportive of the school-linked programs.

Finally, surveys of program beneficiaries (students and parents) as well as other important stakeholders can help evaluators glean information about whether the program and its operation are viewed positively or negatively. Such information can also help evaluators to determine the extent to which school-linked services might be sustainable.

Investigate parent and family involvement as an additional aspect of collaboration. While collaboration is traditionally conceived as the relationship among formal agencies, parent and family involvement in the school-linked services effort is also an important component of collaboration. Evaluators of the Iowa School-Based Youth Services Program are evaluating family and parent collaboration via staff assessments which consider communication systems and opportunities for families to be involved in the programs offered. Communication systems include, among other things, letters, phone contacts, home visits, newsletters, and in-school conferences. Opportunities for family involvement include individual program planning, parent/family counseling, decision-making committee participation, and classes for parents.

Use evaluation of collaboration for self-assessment and improvement. The process of assessing the health of a collaboration can itself be useful to sites. As the evaluators of Missouri Caring Communities Initiative have found, if all partners to a collaboration are invited to complete an assessment instrument, points of agreement and disagreement can be fed back to the site, and important issues can be determined. When gathered across sites, such data can be used to compare collaborative characteristics with outcomes. At a given site, members of the collaborative can see, for example, to what extent they agree on their biggest barriers and their greatest strengths.

Evaluating Sustainability

School-linked services often begin as experimental programs, so their duration beyond initial funding and their possible replication and scale-up are important evaluation considerations.

Examine the institutional and individual relationships. The evaluators note that an examination of the multiple and reciprocal relationships at a number of different levels is an important aspect of evaluating the sustainability of these programs. This includes the extent and degree of engagement by parents, community agencies, organizations, churches, and other entities as well as the extent and kinds of formal support offered by local educational leaders. The evaluator of the Delaware Academy Student Health Program (New York) notes that ongoing support of constituencies is likely to depend on at least four factors: the satisfaction of beneficiaries that their own needs are being met; the belief of beneficiaries that the program is “making a difference”; the degree of controversy related to the program; and the confidence of stakeholders that the use of scarce resources for this purpose is justified relative to other potential uses. These factors can be evaluated through interview and survey data and by analysis of utilization data.

Consider the stability and adequacy of funding sources. Evaluation of financial sustainability requires the identification of all funding sources and their relative importance and an assessment of long-term potential. It is also important to examine how programs have applied or acted on their knowledge about potential funding sources.

Examine the sustainability of outcomes as well. While program durability is an important aspect for evaluation, evaluators also suggest that evaluation examine the sustainability of the outcomes achieved. The evaluators of the New Jersey School-Based Youth Services Program recommend that the durability of outcomes be examined at two levels. To evaluate the sustainability of student outcomes, evaluators should look at the persistence of positive outcomes or reduction of negative outcomes over time, ideally following students through adolescence. They should also examine evidence of developmental outcomes, identifying strengths and assets that the student can both apply broadly and use to address other issues in his or her life. At the school and community level, evaluators should look at the establishment and maintenance of a stable infrastructure of peer and adult support for positive youth development and the institutionalization of a coherent and well-integrated system of linked services to address a range of youth needs.

Consider the extent to which institutions become effective “learning organizations.” Evaluators of the Chicago Full Service Schools Initiative have found that the extent to which the program is becoming an effective “learning organization” is also an important facet of sustainability. The evaluators have begun to track the three sites included in the evaluation to assess how effectively the three planning groups “learn to learn,” that is, learn to conserve, analyze, and disseminate information. This involves striking a balance among planners between conceptual conversation and action-oriented discussion.

Evaluation Design

The nine evaluators provided a variety of insights on evaluation design:

Use a participatory approach. Evaluators of school-linked services programs note that a participatory approach to the evaluation of these programs is important. The incorporation of multiple perspectives ensures the development of instruments and strategies sensitive to and reflective of stakeholder understandings of the programs and brings their perspectives to understanding and using the data collected.

Begin early. Evaluators suggest that the evaluative endeavor should begin as early as possible. This allows time to build trust, understanding, and buy-in among participants as well as to understand basic parameters for on-site research (including potential constraints on data collection methods).

Incorporate multiple methods. Evaluators note that the evaluation design for school-linked services programs should employ both quantitative and qualitative methods. These methods can reinforce and inform one another. For example, intensive interviews, focus groups, and observations can be used to develop questions for survey instruments. Findings from surveys and reviews of program data can be illuminated through individual interview and case study data. Information from multiple methods additionally helps evaluators to communicate findings to a wide variety of audiences.

Maintain an ongoing information exchange. Evaluation should be seen as part of organizational learning—as integral to program design, implementation, and success. Therefore, evaluators of school-linked services programs suggest that adequate feedback loops be established to ensure timely reporting of findings to program staff and other stakeholders. This helps to develop buy-in into the evaluation while at the same time assuring that the evaluators have obtained an accurate picture of reality.

Develop local evaluation capacity. Evaluators also note that, to the extent possible, evaluators should try to develop the capacity of program staff and others to conduct ongoing evaluative activity. The evaluators of the California Healthy Start School-linked Services Initiative used “evaluation coaches,” who were trained in the data needs of the statewide evaluation and were given specific questions to examine at their sites. Coaches, the evaluators found, were key to translating local evaluation reports provided by the statewide evaluator into possible implications for program development. The coaches increased the flexibility and relevance of large statewide evaluation to individual communities and helped communities begin to use systematically-collected information to support their program improvement and sustainability efforts.

A longer version of this paper will be available from HFRP in fall 1997. Please contact HFRP's publications office at 617-496-4304 for more information.

Karen Horsch, Research Specialist, HFRP