The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume V, Number 2 & 3, 1999
Issue Topic: Methodology
Evaluations to Watch
Evaluating the National Replication of a Prevention Program for Youth and Their Families: Middle School Families and Schools Together
Laura Pinsoneault and James Sass from Alliance for Children and Families on their organization's replication and evaluation of the middle school adaptation of the Families and Schools Together (FAST) program.
Introduction
In 1997, the Alliance for Children and Families, with major support from the DeWitt Wallace–Reader’s Digest Fund, began replicating and evaluating the middle school adaptation of the Families and Schools Together (FAST) program. This program adaptation was developed in 1991 by staff of Family Service in Madison,Wisconsin, as part of a Center for Substance Abuse Prevention High Risk Youth Grant. Middle School FAST is a school-based, whole-family program intended to increase the likelihood of youth success in the home, in the school, and in the community. Program activities build positive, respectful, and supportive relationships for middle school youth, their parents, and other family members.
Middle school youth are recruited for the program and begin participation in a school-based youth group facilitated by a youth advocate. Four weeks into the youth group, the whole family begins attending 10 weekly, multiple-family meetings at which they engage in research-based activities designed to be fun, to strengthen the family unit, and to build support networks. At the end of the 14 weeks, families graduate into FASTWORKS, a parent and youth-led multiple-family group that meets monthly for the next two years to continue building on the bonds created during the weekly sessions of FAST.
As of May 1999, 29 Middle School FAST program cycles have been implemented and evaluated. Family- and child-serving agencies in eight states conducted these cycles in 23 middle schools. Ninety-one percent of the families who began the Middle School program attended eight or more sessions, and graduated. The average age of the youth participants is 12 years, with the largest proportion of youths in sixth (42%) or seventh (30%) grade. About 56% of the youths are male and 44% are female. Approximately half of the graduating youths are Caucasian, and one-quarter each are African-American or Hispanic. Two hundred three (203) youths and their families have completed evaluation data.
Evaluation Design
The evaluation design for Middle School FAST is based on the design for the elementary school FAST program (Billingham, 1993; McDonald & Billingham, 1992). It is a non-experimental pre-test/post-test design focusing on initial outcomes and protective factors. At the initial recruitment visit, the parent(s) and the youth give written consent to participate in the evaluation. The youth completes one pre-/post-instrument, and a qualitative questionnaire at the posttest The parent completes four pre-/post-instruments, and family demographics and qualitative questionnaires at the posttest (The youth and parent qualitative questionnaires assess the participants’ program experiences.) The evaluation procedures encourage collecting all pre-tests prior to the start of the youth group meetings, but permit parent forms to be collected before the start of the first multiple-family meeting. Post-tests are administered within two weeks after graduation. In addition to the measures collected from the youth and their parents, an academic information form is submitted to the school.
Instruments
As an early intervention/prevention program, Middle School FAST focuses on factors that exhibit high correlations with the onset of school failure, violence, delinquency, or substance abuse in later adolescence and adulthood. These factors are family environment, youth behavior, and parent-school involvement.
Family Environment
Families complete two forms that measure characteristics of healthy families. Both youths and their parents complete the Family Relationships Index of the Family Environment Scale (Moos & Moos, 1994) to assess the quality of family relationships. The Family Relationships Index (27 items) consists of three subscales: Cohesion, Expressiveness, and Conflict. Pretest/posttest comparisons show significant improvements on three subscales for the parents and on one subscale for the youths. Parents also complete the Isolation subscale of the Parenting Stress Index (Abidin, 1995). This subscale is a six-item measure of the parent’s sense of a lack of social support.
Youth Behavior Problems
To assess behavior problems, FAST uses the six scales of the Revised Behavior Problem Checklist (Quay & Peterson, 1987): conduct disorder, socialized aggression, attention problems, anxiety/withdrawal, psychotic behaviors, and motor excess.
Parent-School Involvement
Middle School FAST now uses 21 items of the Parent Survey (Witte, 1991) to assess changes in parent contact with the school, school contact with the parents, parental involvement in school organizations, and parental participation in school-related activities with their child.
Discussion
Evaluation has been central to the FAST process since the program’s inception. In order to facilitate the use of evaluation within FAST communities, standardized instruments that are easily administered and understood by families were selected to measure multiple outcome areas linked to risk and protective factors. Notwithstanding the usefulness of the current package, the middle school evaluation is still in the development phase. Thus far, from this design we have learned lessons in four areas: response rate, parent-school involvement, long-term outcomes, and asset-based measurement.
Response Rate
The current, more concise version has received a better response than did the original evaluation package, which contained five pre/post measures for the parents and three for the youth. The increased response rate is attributed to decreased resistance from team members in administering the evaluation instruments, and reduced fatigue from families. The overall response rate has been about 80 percent across the 29 program cycles. However, the response rate for academic data has been much lower. The 14-week FAST cycle does not easily fall within a typical semester’s grading period. Furthermore, there is no national set of academic and behavior standards that can be used to create a set of national FAST outcomes. Schools have provided academic data for less than 5 percent of the program youth. Therefore, FAST has been unable to track outcomes in academic achievement. This low figure indicates a need to establish improved forms and procedures for collecting academic data. The immediate solution has been to permit communities to create their own forms for assessing academic indicators, but the national consistency has been lost.
Parent-School Involvement
The Parent Survey (Witte, 1991) replaced an earlier measure that assessed parents’satisfaction with the middle school. The connection between satisfaction and program goals is tenuous at best. As program sites noted, a parent who becomes more actively involved with a school may also become less satisfied with that school. The Parent Survey, using items similar to those on the National Education Longitudinal Study, directly assesses the amount of parent contact and involvement, but does not measure the quality of contact between parents and schools.
Long-Term Outcomes
Evaluation reports based on the initial 14-week outcomes have been a vital tool for Middle School FAST program sites in securing funding and assessing program quality and effectiveness. Still, longitudinal data are necessary to evaluate Middle School FAST’s effectiveness as a prevention program for youth and their families. A longitudinal evaluation component addressing maintenance of the changes that occur at FAST is important to the future of the program. Due to the maturational changes that occur during adolescence, control and/or comparison groups will be critical for capturing the long-term effects that can be attributed to the program. Allliance members and Middle School FAST communities are working to seek out additional funding resources for more longitudinal research.
Asset-Based Measurement
Two of the evaluation measures explicitly focus on deficits or problems (Abidin, 1995; Quay & Peterson, 1987). Although the measurement of deficits is common in social service assessment, this approach is not consistent with FAST’s emphasis on building on the strengths already present in families. Thus, asset-based measures (e.g., Epstein, 1998; Lengyel, Thompson, & Niesl, 1997) deserve serious consideration as tools for evaluating Middle School FAST.
For more information about this evaluation, please contact Jim Sass at 414-359-1040, ext. 3612; jsass@alliance1.org.
References
Abidin, R. R. (1995). Parenting Stress Index: Professional manual (3rd ed). Odessa, FL: Psychological Assessment Resources.
Billingham, S. C. (1993). Evaluation research design for the Family and Schools Together program (FAST). Unpublished doctoral dissertation. DePaul University.
Epstein, M. H. (1998). Assessing the emotional and behavioral strengths of children. Reclaiming Children and Youth, 6, 250–252.
Lengyel, T. E., Thompson, C., & Niesl, P. J. (1997). Strength in adversity: The resourcefulness of America’s families in need. Milwaukee, WI: Family Service America.
McDonald, L., & Billingham, S. (1992). FAST evaluation: Report to the FAST statewide advisory board. Madison, WI: Family Service.
Moos, R. H., & Moos, B. S. (1994). Family Environment Scale manual (3rd ed). Palo Alto, CA: Consulting Psychologists Press.
Quay, H. C., & Peterson, D. R. (1987). Revised Behavior Problem Checklist: Professional manual. Odessa, FL: Psychological Assessment Resources.
Witte, J. F. (1991). First year report: Milwaukee Parental Choice Program. Madison, WI: Robert M. LaFollette Institute of Public Policy.
Laura Pinsoneault
FAST National Trainer/Resource Specialist
Alliance for Children and Families
James S. Sass
Research Analyst
Alliance for Children and Families