You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

The complexity and flexibility of family support programs create challenges for evaluators. This is particularly true when, as in the five-year National Evaluation of Family Support Programs, the intention is to examine several different programs simultaneously.

Conducted by Abt Associates Inc. of Cambridge, Massachusetts and Yale University for the U.S. Department of Health and Human Services, the National Evaluation of Family Support Programs seeks to assess the effectiveness of these programs, particularly with respect to low-income families and their children. To address this goal and to supplement the existing research, a set of eight to ten prospective studies is being designed and implemented.

The prevalent approach to evaluation, particularly when it is undertaken at a national or state level, is first to select a study design that will provide the strongest evidence of impact. This usually means conducting an experiment, with randomized assignment of families or individuals to a treatment or control groups. Then, evaluators typically choose indicators of impact that reflect the desires of program funders and program rhetoric, and that are directly relevant to public policy.

While this approach to evaluation is problematic for many kinds of programs, it is especially so in the case of family support. Since family support programs are typically small, informal, and constantly evolving, they violate most of the conditions for an experiment. By design, they do not offer a uniform treatment, or specify a period of participation. Rather, they provide a menu of possibilities from which families can choose according to their needs. In addition, while programs may agree on a set of broadly defined outcomes, these are often long-term ones that will manifest themselves long after families have left the program (e.g., children's school success or a reduction in juvenile crime and teenage pregnancy rates). Such outcomes are generally not measurable within the brief time frames of most studies.

Due to the difficulties that traditional evaluation methodology poses for family support programs, the Abt and Yale evaluators have adopted a different approach that emphasizes:

  • choosing a design suited to the unique features of family support;
  • designing studies that will provide the maximum information about what happens in family support programs, and;
  • identifying realistic, short-term outcomes that reflect the goals of individual programs.

Their methodology requires active collaboration with program staff at all levels.

Using a theory-driven approach—long advocated by Carol Weiss and others (see Weiss, 1972; Chen & Rossi, 1987)—the Abt and Yale evaluators are working with family support program staff members to build detailed theories of change. They assume that programs are indeed based on explicit or implicit theories about how and why they work. They also assume that program staff members can articulate these theories.

The evaluators' first step, undertaken with 16 family support programs, was to ask the program staff to identify long-term goals that appear in program proposals and other literature, and then to move back from those to the shorter-term outcomes and the small interim steps that children, families, and communities take toward those outcomes. Staff members then discussed how their programs produce those first small steps. Evaluators asked the staff to list program activities and to draw arrows connecting each activity to one or more processes, interim steps, and short-term outcomes. Wherever possible, staff members specified time frames within which changes might be expected to occur. Staff members, of course, know that families begin at different points, have different goals, and move at different speeds towards those goals.

The result of this collaborative, model-building process was a set of theory-of-change models that had both unique and common elements. Common to all was the program goal of child well-being, defined in a variety of ways, and linked to prior positive outcomes for families and community institutions. This common goal was incorporated into a theoretical model that drew on sociological and psychological theory, research in the field of family support, and the evaluators' own judgments in order to identify as comprehensively as possible the necessary conditions for optimal child well-being.

Evaluators believe that identifying a theoretical model of change is essential for recognizing the limits of what any one program can do, compared with what societal changes are necessary for the achievement of long-term goals. To understand this point, consider the range of conditions identified as theoretically necessary for optimal child well-being:

  1. A safe and nurturing home environment
  2. Adequate family resources such as income, education, housing, transportation, and employment
  3. Strong social support networks
  4. A safe and nurturing community environment

Many programs can effect positive change in (1) and (3), but most generally have limited impact on many aspects of (2) and (4).

Further Reading


Knapp, M. S. (1995). How shall we study comprehensive collaborative services for children and families? Educational Researcher, 24(4), 5-16.

Coulton, C. J. (1992). Framework for evaluating family centers. Cleveland, OH: Case Western Reserve University, Center for Urban Poverty and Social Change.

In the next step in the model-building process, evaluators constructed a new model for each of the programs selected for further study. This model incorporated the common aspects of the theoretical model with the activities, processes, and interim steps identified by program staff members. Evaluators asked staff members to identify those mediating conditions that their program might affect (generally a subset of those listed). These final models give the evaluators a choice of measures, some of which are common to all the programs studied and others which are unique to individual programs. The models will also provide a roadmap for the initial studies of each program, which will profile individual participation and change.

The programs in the evaluation found it useful to identify theories of change. It forced them to formulate and articulate an honest program rationale, removing (at least temporarily) the need to rely on the rhetoric that funders often require. Beyond their immediate utility, these models provide programs a way to evaluate themselves and identify needed improvement. The next step in this evaluation will be to share with the staff the measures selected for the studies, both to assess their appropriateness and adequacy, and to consider them for possible use in self-evaluation efforts. Compiling and assessing the host of possible measures is a time-consuming effort which is often beyond the resources of a small program. Thus, discussing evaluation measures with program staff members is a key collaborative endeavor, helpful to both the evaluation and the program.

For further information on the National Evaluation of Family Support Programs, contact Abt Associates, Inc.: Jean I. Layzer, Project Director, 617-349-2817 or Barbara D. Goodson, Senior Scientist, 617-349-2811.

Jean I. Layzer
Project Director
Abt Associates, Inc.
617-349-2817

‹ Previous Article | Table of Contents | Next Article ›

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project