Volume X, Number 2, Summer 2004

Issue Topic: Early Childhood Programs and Evaluation

Spotlight

Rethinking the Evaluation of Family Strengthening Strategies: Beyond Traditional Program Evaluation Models

Charles Bruner, from the Child and Family Policy Center, argues for a reworking of traditional program evaluation methods in order to better assess the effectiveness of family strengthening efforts.

Both science and common sense tell us that families play a crucial role in a child's development. Particularly for very young children, parents are the first and most important teachers. We know that attachment to a consistent and nurturing adult is critical to a child's social and emotional development, which in turn are crucial to educational growth and success.

Case studies of a number of programs have demonstrated the potential for impacting child development through activities that support and strengthen families. Findings, although not based on randomized trials and comparison groups, have been so pronounced that they justify attribution of child outcomes to the programs themselves (Schorr & Schorr, 1988). These case studies show that it is possible to construct programs and strategies that strengthen the capacity of vulnerable families to nurture the learning, growth, and health of young children. While these programs defy neat categorization, they generally adhere to a set of attributes of effective practice put forward by researchers and by reform efforts across most education and human service disciplines.

Overall, however, evaluations of parenting education and family support programs have shown modest impacts, at best. Findings from the federal Comprehensive Child Development Program showed that case management failed to produce gains in strengthening families or improving child development (St. Pierre, Layzer, Goodson, & Bernstein, 1997). A meta-analysis of evaluations of family support programs commissioned by the federal government indicated very modest program results and suggested that successful child development interventions needed to be child centered as well as family centered (Layzer, Goodson, Bernstein, & Price, 2001).

An overview of the research literature on home visiting with young parents similarly found few programs that could boast significant impacts (Gomby, Culross, & Behrman, 1999). A few select programs that have been subject to experimental or quasi-experimental design have shown evidence of success at strengthening families and improving child development. These include the Infant Healthy Development Program, the Chicago Child Parent Center Program, and Early Head Start. Yet these, in many respects, represent exceptions to the rule.

In addition to producing inconclusive results, evaluations of family strengthening programs often fail to help programs improve themselves or understand where they are having impacts and where they are not. With today's current emphasis on results-based accountability, there has been pressure to establish outcome measures that may be only indirectly and distally related to realistic program impacts; these measures prove to be largely unhelpful in program self-evaluation and improvement (Bruner et al., 2002).

To both better evaluate the effectiveness of family strengthening efforts and help programs in their own continuous improvement, we need to re-examine our evaluation methodologies. We must conduct evaluations that will build a body of credible evidence of what factors contribute to success in strengthening families, under what conditions, for what types of families, and with what impacts. To do so, we need to rethink traditional program evaluations to recognize the following points.

1. Relationships and practices, not program structure or curriculum, are key to achieving success.

Particularly in the human services world, most efforts to impact child or family behavior and therefore prevent “rotten child outcomes”¹ are based on nurturing human growth and development. A long line of research makes clear the importance of relationships and trust building to produce this growth and change.

Most evaluations of family strengthening and support programs, however, do not examine the quality of relationships that are established or the way they are established. Marc Freedman, a leading scholar in the resiliency field, suggests that one reason for this is that the focus of much evaluation is wrong. “[The substance abuse prevention field] spent lots of money on program evaluations—and they never look at relationships, only program content” (Henderson, Bernard, & Sharp-Light, 1999). Freedman goes on to emphasize that how a program is conducted (its focus, curriculum, and content) matters less than the development of genuine relationships that help sustain and support people in their continued growth.

The pioneering work of Carl Dunst and Carol Trivette, two of a small group of researchers examining the relationship side of family strengthening programs, shows that while programs and centers adhering to relationship-based family support principles produce results, programs that don't put those principles into practice do not (Dunst & Trivette, 2001a, 2001b).

Evaluations that measure relationship building are not as simply or neatly conducted as those based on program participation. Measuring relationship building poses particular challenges to experimental design. Yet to evaluate fairly the effectiveness of family strengthening programs and strategies and to help foster continuous improvement, evaluations must focus on this area.

2. Impacting “rotten outcomes” requires a systemic, not simply programmatic, focus.

Policymakers, particularly legislators, often seek “silver bullets” for solutions to what they have identified as pressing social problems. They would like to find and then fund “the program” that can eliminate “the problem.” Increasingly, as results-based decision making is being employed, these policymakers also justify inaction by citing the absence of an identified “silver bullet.”

However, rotten child outcomes rarely emanate from a single incident or source. A confluence of factors, both risk and protective, interact to produce them. This does not mean that single actions or interventions cannot make a difference, and in some instances the difference, in children's lives. Without other supporting structures and institutions, however, individual interventions may be battling uphill.

Individual programs, however good they are at relationship building and hope giving, still work with participants who require health care, education, decent housing, safe neighborhoods, and jobs to succeed. A family strengthening program may succeed in instilling resiliency in the families and children it serves. But if a child then goes to a school that does not believe he is educable, remains in a neighborhood infested with drugs and gangs, lacks adult role models, and exists in a society that marginalizes him because of the color of his skin or a disability he possesses, many of his gains in the program's environment will be negated.

A family strengthening program represents only part of what families and children need to succeed. Other public and private systems also must treat families and children with respect and value building relationships with them, recognizing their backgrounds, experiences, and journeys. In short, other systems besides the program need to be family strengthening as well. Programs need to be accountable for their role in supporting pathways to success, but they should not be held accountable for the entire journey.

3. Effective family strengthening programs and strategies create social capital, which is not measurable through a subject-treatment-impact approach.

The recent publication of the Child at Risk Committee, Hard Wired to Connect, provides scientific evidence for the importance of “authoritative institutions” in improving results for children at risk (Institute for American Values, 2003). The defining characteristics of authoritative institutions are analogous to the attributes of effective services cited earlier but emphasize communality and peer (not primarily staff-to-participant) relationships. These authoritative institutions need to be sufficiently diverse in structure and character within a community to engage families and children with different interests, cultures, and needs for social engagement.

Program evaluations, however, usually only examine participant impacts, based on a subject-treatment-impact model, which limits the evaluation's scope to the impact of the program's treatment on the subject. Programs are not recognized for their contribution to the community as mediating institutions or social-capital builders. They may serve as community anchors, solidifying support for children and families. If they are evaluated only for their direct impacts on those participating in a particular activity or treatment, their broader community impact is missed.

Clearly, most social interventions designed to strengthen families occur through programs or centers, involving staff, physical space, and activities. At the same time, the best such programs and centers are more than a set of activities and curricula. They create opportunities for families to exercise leadership, and they serve as community-building anchors. In fact, case studies of exemplary programs consistently show these community-based impacts as among the most significant (Schorr, 1997). Evaluations of family strengthening efforts need to focus on these as well as individual program impacts. We know such authoritative institutions represent essential building blocks for improving children's growth, development, and success on a community scale.

The evidence is clear that too many very young children are at risk, and that some of this risk is a consequence of stressed, unprepared, or inattentive parenting. Such parenting itself is often the result of family or community poverty, lack of education or economic opportunity, or violence. At a policy level, if we are to remove young children from risk, we must develop and support strategies that are effective in strengthening these families, in the context of their communities and the systems that serve them. This places a major responsibility on the evaluation field and requires new conceptions of what constitutes credible and relevant evaluation of family strengthening efforts and strategies (Bruner, in press-a, in press-b).

References
Bruner, C. (in press). Family support: Developing an outcome evaluation framework for family support centers. Des Moines, IA: Child and Family Policy Center.

Bruner, C. (in press). Family support programs—what they produce and what makes them work: Conversations with leaders in the field. Des Moines, IA: Child and Family Policy Center.

Bruner, C., Greenberg, M., Guy, C., Little, M., Schorr, L., Weiss, H., et al. (2002). Funding what works: Exploring the role of research on effective programs and practices in government decision-making. Des Moines, IA: National Center for Service Integration Clearinghouse and Center for Schools and Communities.

Dunst, C., & Trivette, C. (2001). Benefits associated with family resource center practices. Asheville, NC: Winterberry Press.

Dunst, C., & Trivette, C. (2001). Parenting supports and resources, help-giving practices, and parenting confidences. Asheville, NC: Winterberry Press.

Gomby, D. S., Culross, P. L., & Behrman, R. E. (1999). Home visiting: Recent program evaluations—analysis and recommendations. Future of Children, 9(1), 4–26.

Henderson, N., Bernard, B., & Sharp-Light, N. (Eds.). (1999). Resiliency in action: Practical ideas for overcoming risks and building strengths in youth, families, and communities. San Diego, CA: Resiliency in Action, Inc.

Institute for American Values. (2003). Hard wired to connect: The new scientific case for authoritative communities. New York: Author.

Layzer, J., Goodson, B. D., Bernstein, L., & Price, C. (2001). National evaluation of family support programs final report. Volume A: The meta-analysis. Cambridge, MA: Abt Associates.

Schorr, L. (1997). Common purpose: Strengthening families and neighborhoods to rebuild America. New York: Anchor Books Doubleday.

Schorr, L., & Schorr, D. (1988). Within our reach: Breaking the cycle of disadvantage. New York: Anchor Press Doubleday.

St. Pierre, R. G., Layzer, J., Goodson, B. D., & Bernstein, L. S. (1997). National impact evaluation of the comprehensive child development program: Final report. Cambridge, MA: Abt Associates.

¹ The term “rotten outcomes” has been used to refer to a variety of child and adolescent problems, including infant mortality, low birth weight, child abuse, school dropout, juvenile delinquency, substance abuse, teen parenting, and youth violence. Lisbeth Schorr attributes the term “rotten outcomes” to Mary Jo Bane. Schorr, L., & Schorr, D. (1988). Within our reach: Breaking the cycle of disadvantage. New York: Anchor Press Doubleday.

Charles Bruner, Ph.D.
Executive Director
Child and Family Policy Center
1021 Fleming Building
218 6th Avenue
Des Moines, IA 50309
Tel: 515-280-9027
Email: cbruner@cfpciowa.org