The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume VII, Number 1, Winter 2001
Issue Topic: Strategic Communications
Evaluations to Watch
KIDS COUNT Self-Assessment: Bridging Evaluation With Strategic Communication of Data on Children & Families
Deborah Morgan of KIDS COUNT describes how the initiative used an integrated self-assessment evaluation design to incorporate strategic communications into its long-term vision to improve child well-being.
In 1990, the Annie E. Casey Foundation embarked on the KIDS COUNT initiative to measure, state by state, the educational, social, economic, and physical well-being of children and families using the best available data. KIDS COUNT began at the national level with the annual production of the KIDS COUNT Data Book. In 1991, the Foundation began supporting state-level projects. The Network of state level KIDS COUNT projects now includes all 50 states and the District of Columbia.
KIDS COUNT is more than numbers—it is about strategically communicating data to ensure that credible, reliable information provides the basis for informed decision-making about children and families. As such, state-level KIDS COUNT projects have made important contributions to the field of child well-being and to the evolution of data-based advocacy. Throughout the country, KIDS COUNT data are credited with influencing significant policy decisions to improve the condition of kids and families.
But how do we know the true impact that KIDS COUNT data have on decision-making
for children and families?
In order to answer this question, the KIDS COUNT Network embarked on a process
of Self-Assessment to review the internal functioning of each state-level project
and evaluate the impact of that work on improving the lives of children and
families. Underlying the entire effort is the principle of learning—if
we learn more about what we do, how we do it, and the effects of our work, then
we can become more effective and efficient communicators and better generate
our desired outcomes.
Self Assessment: Evaluation of choice for KIDSCOUNT
In 1999, a Self-Assessment working group was formed, comprised
of volunteers from the KIDS COUNT Network. Innovation Network,
Inc. (InnoNet), an evaluation consulting firm in Washington,
D.C., was hired to work with the group as it developed and
implemented the Self-Assessment process. Throughout the evolution
of this process, members of the KIDS COUNT Network have been
involved, advising InnoNet on development and serving as pilot
testers for each instrument and tool created. The full Network
has been consulted at various junctures to ensure buy-in from
the whole.
KIDS COUNT Self-Assessment includes two parts: a process and an outcome evaluation. This was important because, in order to understand and measure each project's outcomes and impact, the projects must better understand how they operate internally and how they might improve.
Assessing the capacity and internal structures of KIDSCOUNT
work
For the process evaluation, a tool was developed to assess
the capacity and internal structures of each project. The
tool examined standard KIDSCOUNT components: data collection
and analysis, communications and dissemination, policy analysis,
community and constituency mobilization, and fund development/sustainability.
Status indicators are used to examine the activities within
each component. Even though projects vary in structure, size
and budget, the tool was constructed to accommodate the diversity
within the Network.
The process evaluation tool is, in large part, a work plan document. As such, after each activity is assessed, projects use an Action Steps page to articulate the major challenges and solutions for items rated less than outstanding. By including the Action Steps pages in the tool, projects are able to plan for improvements and changes rather than merely acknowledging areas for growth. For many KIDS COUNT projects, this process tool allows them to easily report their work and future plans to funders.
The process tool was successfully pilot-tested with 11 KIDS COUNT projects in December, 1999, and was rolled out to the full Network in February, 2000. By the end of 2000, 40 projects completed the process evaluation tool, actively using their results to implement improvements. Each process tool was analyzed by InnoNet, given an individual report and provided with a variety of ways to understand project specific results. While individual project results are confidential, aggregate results are compiled and reviewed by the Network and the Annie E. Casey Foundation to determine areas for technical assistance and training.
The most innovative feature of the process tool is the database of “thinking points” which help guide, rather than dictate, the direction of KIDS COUNT projects as they improve activities identified in their Action Steps pages. The “thinking points,” phrased as questions, are derived from success stories within the Network and are intended to help frame potential improvements and solutions.
The responses to the process evaluation are encouraging. One KIDS COUNT project noted that the Self-Assessment helped identify weakness in the area of media communications, which was remedied by the hiring of two contractors with significant media relations experience who will develop a strategic communications plan for the organization and assist with the release of upcoming publications. Another project said, “The self-assessment tool has been enormously helpful to us. It has given us creative ideas that will improve our program. Our overall project is better integrated and more comprehensive as a result.”
Assessing outcomes and impacts of KIDSCOUNT work
Clearly, the answer to the overarching question—what is the impact that
KIDS COUNT data have on improving the lives of children and families?—is
largely dependent upon the outcomes part of the Self-Assessment, currently in
development. After a thorough search of the literature, a framework was designed
to measure the impact of data-based advocacy on various constituent groups.
When complete, projects will be able to measure and understand how the strategic
communication of data on children is influencing decision-making on behalf of
kids and families.
The outcomes evaluation will be structured differently from the process evaluation. Every KIDS COUNT project will be given a core module of evaluation questions, indicators, and a set of collection instruments with instructions on how to gather the information needed to measure the indicators. For the more well developed projects, there will be additional modules so they may obtain more in-depth information about the impact of their work. In the future, the entire Self-Assessment process will be Web-based.
Without a doubt, the outcomes evaluation is the more challenging aspect of the Self-Assessment. Unlike evaluations of direct service agencies, measuring the impact of an indirect service is a formidable task. And yet, KIDS COUNT projects recognize that the benefits are significant. If the projects can gain a concrete understanding of how the various users of KIDS COUNT data perceive and value their data and information, it can contribute significantly toward better targeting of their data and communications efforts.
The KIDS COUNT Network embarked on this Self-Assessment process to learn how to be more efficient and effective in their work. They continue to strive for excellence and to increase the ways in which data are used to advocate on behalf of children and families. In the end, the more KIDS COUNT data are utilized in policy making, community planning, and program delivery, the greater the likelihood that the lives of children and families will be improved. The Evaluation Exchange Cover Sheet Article author:
Deborah L. Morgan
KIDS COUNT Network Coordinator
National Association of Child Advocates
1522 K Street NW, suite 600
Washington, DC 20005
Tel: 202-289-0777
Email: morgan@childadvocacy.org