Volume XI, Number 2, Summer 2005

Issue Topic: Evaluation Methodology

Evaluations to Watch

The Knight Foundation's Approach to Cluster Evaluation

The John S. and James L. Knight Foundation and Wellsys Corporation describe how they plan to aggregate lessons learned across a “thematic cluster” of youth development investments.

In an effort to support adolescents during the critical middle school transition years, the John S. and James L. Knight Foundation, through its Community Partners Program, made significant investments in out-of-school-time programming in 6 of the 26 communities in which it works.¹ At each site, Knight's local Community Advisory Committees (CACs)—groups of community, business, and nonprofit leaders who help define funding priorities that fit the foundation's mission and strategically address issues of community concern—have focused the foundation's grantmaking on distinct but related efforts to improve outcomes for at-risk adolescents.

The Six Communities in the Cluster
The six communities vary considerably on a number of dimensions—size, location, demographics, and nonprofit capacity—and have found unique ways to target their grantmaking, often focusing on specific schools, neighborhoods, or ZIP codes. Additionally, although the six committees all based their funding decisions on similar concerns about the challenges facing young people, each approached grantmaking differently. In Milledgeville, Georgia, Knight is supporting the replication of a nationally recognized youth development and pregnancy prevention program. In Columbus, Georgia, Knight is helping existing youth-serving organizations develop new, collaborative models for service delivery. In Palm Beach, Florida, the foundation is focused on building the capacity of the youth development field and ensuring that community-based after school providers achieve the level of quality necessary for achieving positive youth outcomes.

With local funding strategies in place, Knight staff began investigating ways to aggregate lessons learned across this thematic cluster of investments. The foundation contracted with the Atlanta-based Wellsys Corporation, whose staff recently began working to develop a baseline view of the grant cluster as well as an ongoing evaluation plan.²

Developing the Baseline View
Wellsys, through its initial review of foundation grantmaking documents, has found several different approaches and strategies to help paint the baseline picture. Within each community with a youth development focus, Wellsys encountered many components, such as grant foci, community reach, theory of change, participants, service delivery systems, and embedded evaluations that needed to be unpacked and documented in order to really be able to assess the complex cluster. Additional components should emerge as conversations take place with foundation staff, local grantees, and evaluators. The cluster evaluation plan will rely on the local evaluators and established evaluation plans that many of the communities already have in place—a purposeful attempt to utilize local resources to the fullest possible extent.

Wellsys is also using several other strategies to develop the youth development cluster baseline view. The first was to interview Knight's Community Liaison Program Officers to document and understand progress to date and help determine next steps for engaging the identified communities. During the summer of 2005, the Wellsys team will visit each community to meet key grantees and their corresponding evaluators. Through these site visits, the team hopes to learn about the evaluation scope and methods being used in each setting; gain a deeper understanding of grantee activities; and obtain input on the cluster evaluation process. Grantee buy-in during the evaluation's early stages will be crucial, as the foundation hopes that grantees will benefit from the assessment though cross-community learning exchanges that may take the form, for example, of technical assistance and recommendations for improving programs and evaluation systems.

Following these community visits, Wellsys will develop a preliminary document that describes early evaluation findings and current status of evaluations, and outlines what the foundation may realistically glean about grantee effectiveness and impact at the end of the 2-year study period. In November 2005 this report will be shared with Knight Foundation's staff and board of trustees, as well as local evaluators and representatives from each community, to obtain feedback. Knight Foundation hopes that this cluster, or “systems,” view will lead to greater insight into the strengths and challenges of its various funding strategies.

Once the cluster theory of change has been developed, Wellsys, in concert with Knight Foundation, will identify the process and outcome evaluation questions and conduct an analysis to note any gaps in community data collection. Tools will be developed to capture identified information and processes established to support data collection. This aggregation of data will inform and support a higher level of foundation decision making and provide additional insight into the utility of various youth development programs in producing clearly identified outcomes and leveraging broader community change.

¹ The six communities are Akron, Ohio; Bradenton and Palm Beach County, Florida; Columbia, South Carolina; and Columbus and Milledgeville, Georgia.
² Both the baseline view and evaluation plan are slated for completion by December 2005, with the evaluation plan to be implemented over the next 2 years.

Julie K. Kohler, Ph.D.
Interim Director
National Venture Fund
Email: kohler@knightfdn.org

Lizabeth Sklaroff
President's Special Assistant for Evaluation
Email: sklaroff@knightfdn.org

The John S. and James L. Knight Foundation
Wachovia Financial Center, Suite 3300
200 South Biscayne Boulevard
Miami, FL 33131-2349

Denise Townsend, Ph.D.
President and Co-Founder
Email: dtownsend@wellsyscorp.com

Susan Boland Butts
Senior Consultant
Email: sbbutts@bellsouth.net

Wellsys Corporation
5045 Johns Creek Court
Alpharetta, GA 30022