Search Publications |  The Evaluation Exchange |  FINE Network |  OST Database

HFRP > Evaluation > The Evaluation Exchange > Issue Archive > Results-Based Accountability #1 > Challenges to Data Capacity for Outcome-Based Accountability

Volume II, Number 1, Winter 1996

Issue Topic: Results-Based Accountability

Promising Practices

Challenges to Data Capacity for Outcome-Based Accountability

Indicators of children's well-being have been used for many years by state and local governments in a variety of policymaking, program planning, and management activities, e.g., needs assessments, site selection, funding allocation, and program monitoring. The emergence of both federal and state block grants and results-oriented program management will heighten the demand for state and, particularly, local data on children's well-being.

The State of New York has engaged in two major children's indicators projects. Modeled after the State of the Child report for New York City (Foundation for Child Development, 1976), the New York State Council on Children and Families published the State of the Child in New York State in 1985. More recently, New York has produced the state's Kids Count data books, under the sponsorship of the Annie E. Casey Foundation.

With over 15 years of experience in this field, we have encountered a variety of political, technical, and resource barriers to the production and utilization of social indicators. While the challenges of the future will undoubtedly be unique, the lessons learned in New York may prove relevant.

Political Challenges
New York's social services, including services for children and their families, are delivered through a state-supervised, locally-administered system. Despite the categorical nature of most funding streams, this situation is analogous in some ways to a block grant scenario. Local governments are given the responsibility for managing a funding stream, which in some instances is capped. While there certainly are federally- and state-designed programs, New York emphasizes the local design of community-based programs. The state, however, retains the responsibility for program monitoring and oversight.

In this situation, state policymakers and program administrators frequently regard the publication of social indicators profiles with some apprehension. These profiles have the effect of holding state officials publicly accountable for locally-run programs. This situation is exacerbated by the fact that some high profile programs are available only on a pilot basis, to small populations, and will not have demonstrable countywide impacts. For example, one community-based adolescent pregnancy prevention program in New York serves approximately 50,000 teens annually, including teens making one-time visits and teens participating in one-day workshops. At most, this represents contact with fewer than three percent of the teenage population. While the program may have an impact on “community” pregnancy rates, it is too small to affect the county or state pregnancy rates published in indicator profiles.

Concern over being held appropriately accountable is leading state officials to turn from social indicators towards performance monitoring, limiting their responsibility to the populations covered by programs. This change in measurement focus acknowledges that government is responsible for the efficient and effective utilization of tax dollars, but shares responsibility with other social institutions (businesses, religious and charitable organizations, and families themselves) for the well-being of children. Performance measures change the question from “How well are children in the general population doing?” to “How are the lives of children in our programs changed?”

With even more decision making devolving to localities under federal and state block grants, the need for county and sub-county data will be intensified. The federal, state, and local roles in this enhanced data capacity have yet to be identified, although there are several reasons for retaining this function at the federal and state levels. With reduced programmatic funding levels, the imposition of additional data production requirements could be seen by localities as an unfunded mandate. Moreover, there are two advantages to retaining this function at the federal and state levels. First, program oversight and monitoring are appropriately federal and state responsibilities. Second, federal and state leadership in setting uniform data collection and reporting standards would enhance the nation's ability to assess the outcomes of various program designs. The systematic collection of comparable data across all administrative geographies would provide the information infrastructure for future program planning and development.

A second unanswered question under the block grant scenario is what data will be collected—outcome indicators or performance measures. Although there is an incentive for governments to focus on performance measures, the extent of program coverage cannot be assessed without the collection of social indicator data.

Technical Challenges
Indicators of children's well-being come from a variety of data sources, including census data, vital statistics and other administrative data sets, federal, state, and local survey data, and estimates. Each source has strengths and weaknesses. Thus, it is important to specify meaningful measurement criteria before selecting data sources for children's indicators. Kristin Moore has developed a list of potential criteria that includes, among other items: complete population coverage, consistency over time, and rigorous data collection methods (Focus, 1995). (See article by Kristin Moore and Brett Brown in this issue.)

New York's experience in the production of social indicators illustrates the importance of having both clearly specified measurement criteria and a detailed knowledge of data sources. This is never more true than when using administrative data. While some administrative data are population-based and of very high quality (such as vital statistics), many administrative data files are of questionable quality or cover only portions of the population of interest.

Established to support program operations, many administrative data systems only cover a portion of the population that, theoretically, should be included in a social indicator. For example, there is no good source of data on the number of families receiving child support. Administrative data on child support collections are fairly complete for the AFDC population, but have limited coverage of the non-AFDC population—just 58 percent of those families counted in the Current Population Survey (CPS).

Moreover, administrative data may portray system operation, not the prevalence of certain characteristics. In an example that is illustrative to data issues with block grants, the New York State Department of Health collects data from public health clinics on the immunization status of two-year-olds. Examined over time, the data show large year-to-year fluctuations. This occurs because in some years counties allocate more funding to well-baby clinics, which keep good records on a child's immunization history, and in other years they allocate more funding to immunization clinics, which do not keep such complete records.

Further Reading Board on Children and Families and Committee on National Statistics, National Academy of Sciences. (1995). Integrating federal statistics on children: Report of a workshop. Washington, DC: National Academy Press. Institute for Research on Poverty. (1996, Spring). FOCUS. Madison, WI: University of Wisconsin. Institute for Research on Poverty. (1995, Spring). FOCUS. Madison, WI: University of Wisconsin.

National data sets, which are generally thought of as having population-wide coverage and high quality, may also prove problematic. For example, there is a sizable undercount in the CPS of the number of AFDC families receiving child support in New York State—the CPS count is just one third of that reported in the administrative count. In a second example, the self-enumeration data in the 1990 census found over 89,000 New York children and youth who “did not speak English well or at all,” while school district data from tests show that over 168,000 children were eligible for Limited English Proficiency instruction. Finally, even when coverage and quality are high, many national data sets do not provide information for state or substate areas.

Resource Challenges
Funding for social indicators work has always been limited and is likely to be more so in the future. To date, all of the children's indicators work in New York has been at least partially supported by funding from private foundations. If federal funds for program operation are reduced, there will be substantial public pressure for the maintenance of services. Funding to enhance or even maintain data systems will clearly be limited. Additionally, given the debate about program directions, any available data resources may be targeted to enhancing systems for fraud detection or for developing longitudinal data systems so that time limits can be enforced.

Future Directions
Several things can be done to improve the quality of indicators of children's well-being data and to prepare for the data needs of a block grant environment:

1. Enhance the national data infrastructure. It will be important to have a national frame of reference by which each state can assess itself as program responsibilities devolve to state and local areas. To that end, two improvements are badly needed. First, survey data sets should be structured so that state data can be reliably extracted. Second, estimates programs should be expanded to fill the need for substate demographic data for non-census years.
2. Establish national standards for social indicator data collection and reporting. This will keep a spotlight on societal issues, and help to focus investigation into the linkages between outcome indicators and performance measures.
3. Develop social indicator methodologies. There continue to be several life areas in which meaningful indicators or appropriate data are lacking, e.g., disability, mental health, positive youth development. Moreover, in order to enhance their policy and program utility, social indicators need to be empirically linked to multivariate models of social processes. The utility of indicators systems would be greatly enhanced if they illustrated paths to policy solutions, including leading indicators or intermediate outcomes.

Nancy Dunton
New York State Department of Social Services