The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume ll, Number 3, 1996
Issue Topic: Results-Based Accountability
Theory & Practice
Key Stages and Common Themes in Outcome Measurement
Nancy McDaniel from the American Humane Association (AHA) outlines the AHA's work to help child welfare professionals clearly demonstrate the results of their work.
Background
In an era of shrinking resources and conflicting public expectations, child welfare professionals find it increasingly important to clearly demonstrate the results of their work. To support that goal, the American Humane Association (AHA) has worked extensively with child welfare agencies at the national, state and local level to develop and implement outcome measurement initiatives.
In addition, AHA has cosponsored four annual Roundtables on Outcome Measures in Child Welfare Services in partnership with the National Association of Public Child Welfare Administrators, an affiliate of the American Public Welfare Association. The AHA has also completed two national surveys of state child welfare outcome initiatives.
Despite the obvious and significant differences across states and localities, several key stages can be identified with the development and implementation of outcome measurement initiatives, including:
- Defining purpose and audience
- Identifying target outcomes
- Specifying measures
- Identifying sources of data
- Developing an implementation plan
- Changing the organization culture
Child welfare professionals have voiced similar challenges and strategies for successfully navigating each stage.
Key Stages
Defining Purpose and Audience—Why are we doing this and who wants to know? State and local agencies are interested in measuring outcomes in order to: justify budget and funding expenditures, ensure accountability, optimize the use of resources, establish performance baselines, and understand program effectiveness. Child welfare agencies usually have multiple purposes and audiences for outcome information. Audiences may be internal to the agency, such as administrators and frontline staff and/or external to the agency, such as other agencies, the legislature, a foundation or other funder, or interested community members. Clearly identifying these audiences up-front can help to define a clear direction, key partners, and needed resources.
Identifying Target Outcomes—What are the desired results for children and families? Child welfare agencies operate with multiple and sometimes conflicting missions. Thus, identifying the target outcomes can precipitate critical discussion regarding the fundamental mission and goals of the agency. Nationally, child protective service (CPS) agencies consistently name child safety and permanency as target outcomes because they are identified as central to the agency's mission. However, there is less consensus at the national level whether the target outcomes of child and family functioning fall within the scope of the CPS agency. Some CPS agencies view the target outcomes of child and family functioning as both relevant and essential goals in order to achieve child safety and permanency. In other agencies, these outcomes, although recognized as important, fall outside a more narrowly defined scope of mission.
|
Further ReadingMcDaniel, N., & Merkel-Holguin, L. (1995). National overview of child welfare outcome measures development efforts. In N. McDaniel & L. Merkel-Holguin (Eds.), Summary of proceedings: Third national roundtable on outcome measures in child welfare services. American Humane Association. Potter, C., & McCroskey, J. (in press). Measuring change: Selecting and utilizing assessment instruments. In N. McDaniel & L. Merkel-Holguin (Eds.), Summary of proceedings: Fourth annual roundtable on outcome measures in child welfare services. American Humane Association. |
Specifying Measures—How will we know it when we see it? An unwritten evaluation axiom repeatedly expressed to child welfare professionals is “don't measure it unless you plan to change it.” For example, if child safety is defined as the target outcome, the next stage is to specify measures that suggest progress toward achieving child safety. CPS agencies often measure child safety by, at minimum, the occurrence or reoccurrence of child maltreatment.
Identifying Sources of Data—Who has or is going to collect this information? An initial and ongoing process that occurs is determining the source of measurement information. For the target outcome of child safety, many states and CPS agencies have specified recidivism as one outcome measure because of the availability of central registry data. Agencies often struggle with how to assess progress in achieving the target outcomes related to child and family functioning, and how to collect outcome data at the client level. Fundamentally, the decision around the selection of assessment instruments will follow careful of what information is deemed to be critically important based on its relevance and usefulness, whether the information is already available, and what the agency can afford to collect.
Developing an Implementation Plan—We know where we want to go, so how do we get there? There can be substantial consensus, understanding, and commitment among key players during planning. However, we have all participated in well-intended initiatives that sank under the weight of their planning documents. Successful outcome initiatives allow time for necessary planning, but recognize that knowledge is always imperfect. Effective leaders know how and when to proceed with implementation by developing flexible plans with clear tasks and timelines.
A challenge is to ensure that not all of the resources are expended up-front for planning and to reserve adequate resources for implementation activities. Many initiatives pilot their plan on a limited basis before proceeding with agency- or state-wide implementation.
Changing the Organizational Culture—Are we doing business differently? Most professionals believe that focusing on outcomes will result in fundamental differences in the way that agencies view themselves and their partners. However, the challenge for leaders is to integrate that view into a cultural change in the organization. All staff must perceive the outcome initiative as central to their work, rather than as an added form to complete or an isolated activity. During the past four years, numerous state and county public or private agencies have initiated outcome-focused projects. Initiatives with the most longevity are just now able to share lessons about key stages involved and the consequences for organizational development.
Common Themes
- Outcome measurement development is a long-term process.
There are numerous philosophical and technical challenges associated with developing and implementing outcome-focused initiatives. In our work, we find that many states and local agencies are taking up to two years in the beginning stages of outcome development. The challenge is to balance the necessary process with the need to maintain momentum and interest.
- Involving frontline staff, community members, and clients increases the chances of commitment and successful implementation.
Many states and localities are using an inclusive process in the development and implementation of outcome initiatives. For example, they begin by organizing work groups or engaging a broad advisory committee. The more people involved, the longer the process will take. Conversely, the fewer the people, the more quickly the process may proceed. However, without strong commitment and understanding, the process may be derailed by lack of agreement.
- Outcome measurement initiatives require short- and long-term planning to build the capacity to use management information system technology.
For many agencies, the expectations for collecting and utilizing data are not matched by the capacity of their management information systems—either hardware, software, or staff. Successful outcome initiatives involve MIS, research, and program staff from the beginning so there are adequate feedback mechanisms to continually refine the questions generated by program staff and to improve the agency's capacity to answer those questions.
Conclusion
The field of child welfare and child protective services has been hampered by its inability to communicate a clear mission and purpose to the broader community. Building consensus with community partners about target outcomes can lead to greater agreement regarding the role of the CPS agency within the context of the broader community. Furthermore, focusing on outcomes can lead to greater knowledge about what works_what sets of interventions and services are associated with improved child safety, child and family functioning, and permanency.
The American Humane Association (AHA) is a national non-profit organization headquartered in Denver, Colorado. For more than a century, AHA has advocated for services and policies to support vulnerable children and families. AHA is a nationally recognized leader in the field of child abuse and neglect and conducts research, evaluation and training activities.
For more information about the national outcome measurement roundtables or to obtain the proceedings contact the American Humane Association.
Nancy C. McDaniel
Manager of Program Evaluation and Policy Analysis
American Humane Association
63 Inverness Drive East
Englewood, CO 80112
Tel: 303-792-9900