The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume 0, 1997
Issue Topic: Community-Based Initiatives
Evaluations to Watch
Evaluating the Kellogg Youth Initiative Partnerships
Ricardo Millett, Director of Evaluation at the W. K. Kellogg Foundation, shares information about the evaluation of the Kellogg Youth Initiative Partnerships.
In 1987, the W. K. Kellogg Foundation made a long-term commitment to work with three geographic areas of Michigan to help improve the lives of young people. The 20-year Kellogg Youth Initiative Partnerships Program (KYIP) is designed to work in partnership with these three areas, to provide opportunities for each to develop and implement programs uniquely suited to the needs of its residents. In 1995, the Foundation Board adopted a Framework for the Future, which established positive youth development (rather than a deficit-based approach) as KYIP's guiding philosophy, set forth five broad goals, and laid out six strategies for achieving them.
The Foundation has contracted with the Academy for Educational Development (AED) to conduct an evaluation of the initiative in the remaining ten years. The evaluation will produce both formative and summative findings. AED's work with KYIP will pursue levels of analysis that speak to the sites, the Foundation, and broader audiences to illuminate how KYIP has experienced successes and encountered challenges over a number of years. The evaluation is being conducted in partnership with the Foundation and the three sites—including their foundation staff members, their Site Advisory Councils (SACs), service providers, and residents—and other KYIP collaborators.
The basis of the evaluation approach for KYIP has been the articulation of a theory of change. Working over several months, KYIP staff members, the Sacs, and the evaluators articulated the theory of change for KYIP, including the longer and shorter-term outcomes to be achieved, and the indicators used to track progress. The theory of change serves as the basis for the outcome, process, and context evaluations of KYIP.
Outcome Evaluation: A relatively simple theory of change guides KYIP: If KYIP can create and sustain the conditions within a community that promote positive youth development, then increases in positive youth outcomes and decreases in problem behaviors are likely. In translating this theory into a usable program and evaluation model, KYIP staff, the Sacs, and evaluators agreed to a set of 11 outcomes in four broad domains: youth, adult, organizational, and community. Outcomes related to adults (for example, those related to increased adult involvement and interest in youth) and organizational change (for example, functioning and collaborative organizations to serve youth) are considered early and intermediate outcomes, necessary to achieve longer-term youth and community outcomes. For each of these 11 outcomes, a set of indicators has been identified to monitor progress. Data for these indicators will be collected through: a community organizational survey; an adult survey; a youth survey; focus group discussions; and extant data analysis. Recognizing that identifying and using comparison communities is not feasible for this evaluation, AED is using a pre and post-KYIP Framework evaluation design to assess KYIP sites' achievement of Framework goals.
Process Evaluation: The process evaluation will examine the degree to which the strategies and scope of work described in the Framework for the Future and each site's strategic plans are actually comparable to what is reported and observed in the sites. Using data from surveys, site visits, meetings, and correspondence, evaluators will examine what is occurring at the sites. This will be helpful to the Foundation in monitoring the overall progress of the KYIP partnerships and can be used to identify and then address program-wide technical assistance needs.
Context Evaluation: In addition to examining the implementation of KYIP and its outcomes, evaluators will also examine the context of the larger social and economic factors within which KYIP is operating to clarify the extent to which changes in outcome data are related to the initiative rather than other factors. This includes analysis of the demographic and socioeconomic factors that affect both participation in KYIP and its outcomes; unintended outcomes; and factors such as other public or private programs or economic fluctuations in a local industry or local economy that could affect KYIP outcomes.
Analysis of qualitative and quantitative data will occur at two levels. First, data will be interpreted at the site level within the context of each site's specific conditions. However, site-level analyses will not be conducted for comparative purposes. Analyses will also be conducted at the initiative level, serving the purpose of a cluster evaluation. These analyses will address the implementation and utility of the common framework and strategies of all three sites, as well as the suitability of the overall initiative to bring a new vision of youth development to the respective communities.
Ricardo A. Millett
Director of Evaluation
W. K. Kellogg Foundation