You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

John Kalafat from Rutgers University describes how he and his colleagues used Innovative Configuration Analysis to evaluate a statewide family resource initiative’s implementation and impact.1

The Kentucky Family Resource Center (FRC) program is a statewide initiative consisting of school-based centers whose mission is to ensure that children come to school healthy, safe, and prepared to learn. As required by the legislation enabling their creation, FRCs provide a set of core family services adapted to the needs and contexts of the local community. FRCs address the physical, psychological, and social needs of children and their families; increase family participation in the educational process and access to community services; and forge cooperation among families, schools, and communities.

Because the state’s mandate is flexible and ambitious, interventions vary from one FRC to another. Seeking to understand what FRCs were doing to meet the state’s mandate, we conducted a process evaluation involving extended visits to several FRCs. During these visits, we observed and interviewed FRC staff, school and community personnel, and family members. We also reviewed program materials. These visits yielded qualitative descriptions of what programs and services FRCs were providing and how they were providing these services to their schools, families, and communities.

From these descriptions, we were able to identify the “active ingredients” of FRCs—that is, the common cross-site services and approaches that seemed most likely to remove barriers to learning.2  For example, we found that, across multiple FRCs, coordinators had to win school principals and faculty over to the idea of increased family involvement in schools as a precondition to facilitating family involvement. Similarly, FRC coordinators frequently found it necessary to convince principals of the importance of the FRC working in the broader community with families, rather than working solely in the school with students.

As we examined the presence of these active ingredients in the FRCs, we quickly recognized that simply dichotomizing them as present or absent in an individual FRC would result in a substantial loss of information. These “ingredients” were in fact ongoing processes, and we realized that we needed to describe not presence or absence but rather the different levels of implementation of these components. For example, one FRC coordinator may be working closely with the school principal, while another may be just beginning to educate her principal about the role of FRCs in children’s school readiness.

In order to measure these ongoing processes and their varying levels across FRCs, we utilized Innovation Configuration Analysis.3  Innovation Configuration Analysis draws on detailed descriptions of program activities—such as relationships between schools and parents—and creates a scale on which each activity can be rated. To begin using this method in the FRC evaluation, a committee of evaluators who were familiar with FRCs, FRC coordinators, and state administrators held several meetings to generate descriptions and rating scales for program domains, or approaches to serving families. Ultimately, they generated the following domains: Needs Assessment, Relationship With School, Relationship With Community, Relationship With Families, Advisory Council (involvement of), and Mission Focus (degree to which activities addressed barriers to learning). The committee also generated descriptions of levels of implementation for each domain. Successive versions of these descriptions were submitted to other center coordinators for review and modification. This iterative process resulted in the Innovative Configuration Analysis measure.

As the next step in our evaluation, we assessed how the implementation of these domains related to students’ social and academic performance—which is, of course, the ultimate goal of the FRCs. For each domain, we assigned points to each implementation level and created total scores for each domain, representing the degree of implementation. We then created a percentage score by dividing the total score by the total possible score. Hall and Hord called this measure an Innovation Components Configuration (ICC) map.4  Finally, by averaging each of their domain scores, we developed an overall implementation score for each center in our study.

When examined together, the ICC maps create a picture of how intensively the FRCs implemented each of the activities, both as individual centers and across groups of centers. These overall implementation scores strongly relate to educational outcomes for the students who participate in the FRC programs. This procedure for carefully describing specific program activities yields practical information that can help FRC efforts to attenuate barriers to learning.

1 For more information about the topic discussed in this article, see Kalafat, J., Illback, R. J., & Sanders, D. (2007). The relationship between implementation fidelity and educational outcomes in a school-based family support program: Development of a model for evaluating multidimensional full-service programs. Evaluation and Program Planning, 30, 136–148.

2 Kalafat, J., & Illback, R. J. (1998). A qualitative evaluation of school-based family resource and youth service centers. American Journal of Community Psychology, 26, 573–604.

3 Hall, G. E., & Hord, S. M. (1987). Change in schools: Facilitating the process. Albany, NY: State University of New York Press.

4 Hall & Hord, 1987.

John Kalafat, Ph.D.
Associate Professor
Rutgers Graduate School of Applied and Professional Psychology
152 Frelinghuysen Road
Piscataway, NJ 08854
Tel: 732-445-2000 ext. 121

‹ Previous Article | Table of Contents | Next Article ›

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project