The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume I, Number 2, Spring 1995
Issue Topic: Evaluating School-Linked Services
Theory & Practice
Evaluating School-Linked Services
The following summarizes a paper that was presented at the American Educational Research Association's annual conference on April 19, 1995. Due to space constraints here, it was not possible to include all the detail in the working paper, nor the charts and graphs that accompany that document.
Despite the growing prominence of school-linked initiatives, there is little information about how they function, and the degree to which they affect the children they serve. Most have arisen within the last decade; for example, the number of school-based or school-linked clinics has risen from 32 in 1985 to 574 in 1993 (Dryfoos, 1993, p. 179). As a result, evaluators have had little time to document changes in the service delivery patterns themselves, and even less time to document outcomes related to these changes. Yet as Kahn and Kelley suggest, all of the assumptions on which the school-linked services model are based must be examined empirically (in Adler and Gardner, 1993).
This examination takes place within the intense political climate that characterizes educational policy, and the fiscal and administrative constraints that are the reality of today's configuration of educational and social services for children. Given these conditions, evaluation stands to play a critical role in the development of school-linked services policy and program design. Yet, like school-linked service efforts themselves, evaluation also operates under political, fiscal, and administrative constraints.
School-Linked Services Are Characterized By:• A Holistic Approach: Physical and emotional well-being, family self-sufficiency, and family stability affect a child's ability to learn. • Joint Planning and Service Delivery: School-linked services involve a multiple-partner planning phase (Gardner, 1993), and locate social services at or near schools, or institute a school-based referral process to services outside the school. • Collaboration: At least one social service agency cooperates with the public school system. • A Focus on Child Outcomes: These efforts concentrate on the well-being of children and their families. • Shifts in Roles of Professional Staff: Teachers, case managers, and other staff are trained to view and treat children in an ecological, holistic manner. |
Debating School-Linked Services
School-linked service efforts vary considerably in terms of design and intent. Yet, they all share a holistic, ecological view of the child, and attempt to operationalize this philosophy by linking a wide array of preventive services for children and their families to the school setting (Harvard Family Research Project, 1993).
Proponents of school-linked services maintain that delivering social services to children via schools makes sense because, of all public institutions, schools provide the most sustained and non-stigmatizing contact with children (Koppich & Kirst, 1993). Advocates hold that locating services at or near schools capitalizes on the near-universality of public education and thereby increases children's access to other social services that have not been as accessible as education.
Not everyone agrees. Some critics of school-linked services argue that it is unfair and detrimental to the educational enterprise to ask schools to deliver services. While they do not question the worth of social services themselves, they argue that schools should not be turned into social service institutions. Others contend that linking services to schools may not work because of the historical distrust of schools among disenfranchised and minority populations; the high level of school dropout, especially in urban areas; and the assumption that schools are serving neighborhood residents (Chaskin & Richman, 1993). Instead, they propose community-based service delivery models as more appropriate means through which to distribute services to children.
Still other critics hold that government services lead to negative outcomes—e.g., that more school-based health clinics lead to more pregnant teens (Kasun, 1987; Bennett, 1987). Since this camp sees social services themselves as suspect, it finds the idea of linking them with schools even more so.
The broadest resistance to school-linked services may, however, be the well-established lines of power and authority found in both education and social service institutions. The highly bureaucratic, professionalized models of public education and social service agencies conflict with the collaboration and sharing that is the cornerstone of the school-linked services movement (Mitchell & Scott, 1993). Thus, while coordinating services among different agencies is always difficult, the interface between public schools and social service agencies is particularly challenging.
Evaluation Challenges
The characteristics of school-linked services present several evaluation challenges.
Documenting the “treatment.” Since school-linked services are complex and cross boundaries that have historically existed between schools, parents, social service agencies, and local and state governments, it is difficult to fully document all aspects of their “treatment.” In addition, since many span many schools within a single district or across an entire state, there is also a high degree of program variability. Program variability and complexity make it hard to identify a coherent set of program effects that are applicable across all sites.
Assessing collaboration. “Collaboration” must first be clearly defined by the program and documented before evaluators can determine whether and how it affects child and family outcomes.
Documenting implementation and service delivery. Since the stakes are high—dropout rates reach 50% in many major urban school districts, and many parents and children do not trust public schools—showing that services are reaching targeted populations is critical.
Satisfying demands for accountability. The demand for outcomes-based accountability requires evaluators and program personnel to show that their services affect child or family health and cognitive and emotional well-being.
Determining outcomes. While some researchers call for a moratorium on outcomes-focused evaluation until school-linked services can be adequately implemented (First et al., 1993), the demand for accountability necessitates identification and measurement of at least some outcomes early in a program's implementation.
Measuring impact. Since public education is a universal entitlement and access to services linked to schools is often also universal, determining impact is difficult because a natural control group is often impossible to identify.
Satisfying multiple stakeholders. School-linked services require the cooperation of teachers, school administrators, school districts, social service agency administrators, front-line workers, parents, and funders, among others. Thus, evaluators must ensure that the questions and concerns of each set of stakeholders are adequately addressed.
Methodology
Evaluation and program data from seventeen school-linked services initiatives (see table) was compared along a number of variables. To be included in this study, school-linked services efforts had to be in operation and have an evaluation plan or a completed evaluation. Both programs with school-based services as well as those that link schools to existing services were examined.
|
17 School-Linked Service Initiatives
|
|
| Program Name | Evaluators |
| Success by Six (Pinellas County, FL) | Clearwater Partnership of Business, Community, Education, Government and Social Service Representatives, Clearwater, FL |
| Healthy Start (40 sites in CA) | Wagner et al., SRI International, Menlo Park, CA |
| New Beginnings (San Diego, CA) | Philiber Research, Accord, NY |
| Family Resource and Youth Services Centers (KY) | Robert J. Illback, R.E.A.C.H. of Louisville, Inc., Louisville, KY |
| Focus on Youth (Los Angeles, CA) | Orr and Uman, Los Angeles Educational Partnership |
| Georgia's Pre-K Program | Quay and Kaufman-McMurrain, Dept. of Early Childhood Education, Georgia State University, Atlanta, GA |
| Healthy Caring (24 sites/various states) | Mathematica Policy Research, Inc. and Mathtech, Inc., Princeton, NJ |
| Family Preservation Integration Project (31 programs in MN) | Wattenberg et al., Center for Urban and Regional Affairs, Minneapolis, MN |
| Parents as Teachers (MO) | Harvard Family Research Project, Cambridge, MA |
| Family Resource School Program (Denver, CO Public Schools) | Cuciti, Appelbaum and Witwer, Center for the Improvement of Public Management, University of Colorado |
| Decker Family Development Center (Akron, OH) | Internal Evaluation Team |
| Community Schools Program (NY) | Holzman, The Bruner Foundation, NY, NY |
| New Jersey School-Based Youth Services Program | Godin et al., East Stroudsburg University, East Stroudsburg, PA |
| Project Learn (Phoenix, AZ) | Vicki Romero & Associates, Inc., Scottsdale, AZ |
| New Futures (Dayton, OH) | Center for the Study of Social Policy, Washington, DC |
| LEAP (Cleveland, OH) | Manpower Demonstration Research Corporation, NY, NY |
| Florida Full Service Schools | Institute for At-Risk Infants, Children and Youth and Their Families, University of South Florida |
Results
Complexity/program variability. Most evaluations in our sample employed research designs and data collection techniques that acknowledged the complexity and program variability inherent in school-linked services initiatives. Triangulation was common, as were multiple and wide-ranging evaluation goals addressing a variety of issues such as outcomes, formative feedback, and documentation of implementation and collaboration. Evaluations tended not to document external factors that might play a role in the success of such initiatives, such as community support, economic conditions, or existing boundaries between schools and social service agencies.
Further ReadingAdler, L., & Gardner, S., (Eds.). (1994). The politics of linking schools and social services. Washington, DC: Falmer. Bennett, W. J. (1987). Sex and the education of our children. In B. L. Mosbacker, (Ed.), School based clinics and other critical issues in public education. Westchester, IL: Crossway. Chaskin, R. J., & Richman, H. A. (1992). Concerns about school-linked services: Institution-based versus community-based models. In R. E. Behrman, (Ed.), The future of children: School-linked services (Vol. 2, No. 1, pp. 107-117). Los Angeles: The Center for the Future of Children, The David and Lucile Packard Foundation. Dryfoos, L. (1994). Full service schools. San Francisco: Jossey-Bass. First, P. F., Curcio, J. L., & Young, D. L. (1994). State full-service school initiatives: New notions of policy development. In L. Adler, & S. Gardner, (Eds.), The politics of linking schools and social services (pp. 63-72). Washington, DC: Falmer. Gardner, S. L. (1992). Key issues in developing school-linked, integrated services. In R. E. Behrman (Ed.), The Future of children: School-linked services (Vol. 2, No. 1, pp. 85-94). Los Angeles: The Center for the Future of Children, The David and Lucile Packard Foundation. Kahne, J. & Kelley, C. (1993). Assessing the coordination of children's services: Dilemmas facing program administrators, evaluators, and policy analysts. Education and Urban Society, 25(2), 187-200. Kasun, J. R. (1987). The truth about sex education: Turning children into sex experts. In B. L. Mosbacker (Ed.), School based clinics and other critical issues in public education. Westchester, IL: Crossway. Koppich, J. E., & Kirst, M. W. (1993). Editors' introduction. Education and Urban Society, 25(2), 123-128. Lopez, M. E., & Hochberg, M. R. (1993). Paths to school readiness: An in-depth look at three early childhood programs. Cambridge, MA: Harvard Family Research Project. Mitchell, D. E., & Scott, L. D. (1994). Professional and institutional perspectives on interagency collaboration. In L. Adler & S. Gardner (Eds.), The politics of linking schools and social services (pp. 75-92). Washington, DC: Falmer. |
Collaborating. The elements of collaboration that were most commonly documented tended to be lists of activities and the formal creation of collaborative partnerships; less common were measurements of the collaborative process, such as shifts in procedures, financing, service delivery structure, and changes in underlying attitudes or mission.
Documenting implementation. Nine evaluations named implementation as a goal, and six named documentation of service delivery as a purpose. Those that measured implementation tended to use multiple methods to document multiple aspects of administration and service delivery. However, no evaluation presented a model or theory of successful implementation against which to assess the initiative being studied. Instead, specific components of service delivery such as quantity and access were documented.
The same was true of aspects of administrative behavior—measures of such activities as planning and daily operations were most common. No evaluation presented or attempted to link aspects of implementation, or the implementation process as a whole, to specific outcomes. Ten evaluations functioned as part of the implementation process by issuing some form of formative feedback to the program. Two evaluations also conducted needs assessments, commonly thought to be an integral, early step in program planning and implementation.
Tracking outcomes. Despite the relative youth of most school-linked services, many were tracking outcomes. This tracking may have been a response to a policy climate that is increasingly focused on accountability and reporting requirements.
Most of the evaluations documented outcomes for children and several did so for families and parents as well. Most focused on changes in children's cognitive ability or academic achievement. Many also attempted to measure behavioral and/or attitudinal changes, as well as social or emotional well-being. Parent and family outcomes tended to focus on parent activities in relation to school personnel or the child's educational progress.
Measuring impact. Only one evaluation in our sample employed an experimental design by using control groups. Several others attempted to control statistically a wide range of variables using multivariate techniques such as LISREL or by utilizing longitudinal designs; some used comparison groups.
Participating in research. Eight evaluations included external participation in research activities, usually in collecting data; less common was external representation in the evaluation design and decision-making process. Only professional staff were asked to participate; the documentation available to us did not indicate participation in evaluation from parents, program participants, community leaders, and/or activists.
Summary
The complexity and variety of school-linked services pose many challenges for evaluators. They must generate information useful to a variety of stakeholders, many of whom come from well-entrenched bureaucracies with well-defined interests. Particularly important is disseminating findings back to program personnel, who need advice on how to improve their programs or how to achieve better outcomes. This demand for formative feedback is onerous since the complexity and variability across and between programs makes attribution of outcomes to specific program offerings difficult at best and sometimes impossible. To this end, a theory of the hypothesized relationships among program elements and sought-after outcomes may help focus evaluators on providing better formative feedback to these ambitious initiatives.
The paper on which this article is based is available from the Harvard Family Research Project. For information on ordering the working paper entitled, please go to Challenges in Evaluating Comprehensive School-Linked Service Initiatives.
Kathleen M. Shaw, Senior Researcher, HFRP
Elaine M. Replogle, Research Assistant, HFRP