The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume IV, Number 1, 1998
Issue Topic: Performance Measurement
Questions & Answers
Interview With Harry Hatry and Joe Wholey
Harry Hatry and Joe Wholey, who have been influential in the movement toward accountability at the federal level, discuss lessons learned and next steps for performance measurement.
In 1993, the United States Congress passed the Government Performance and Results Act (GPRA), mandating that each federal agency develop a strategic plan with outcome-related goals and objectives; develop annual performance plans with output- and outcome-related goals; and report annually on progress toward achievement of these. In the almost five years since the passage of this legislation, federal agencies have been working to develop planning and measurement systems to enable them to respond to the legislation and to help them achieve better results. The focus on performance measurement has not been limited to federal agencies; it has gained currency in state and local governments as well as foundation-funded programs and nonprofit organizations. Two individuals who have played key roles in the performance measurement movement are Joseph Wholey and Harry Hatry. Joe Wholey is Professor of Public Administration at the University of Southern California and Senior Advisor on Evaluation Methodology at the U.S. General Accounting Office. Harry Hatry is a Principal Research Associate and Program Director at the State Policy Center of the Urban Institute. We asked them to reflect on some of the lessons learned from the federal implementation of GPRA and the other results-based work they have seen.
What have been some of the key factors influencing the success of performance measurement systems?
Wholey: Executive leadership is key. You need people who believe in setting outcome-oriented goals and who are trying to aim their organizations in that direction. This requires that they consistently demonstrate their belief and commitment to it—not just one time. Some executives hold “how are we doing” sessions which examine what the data are and what they mean and what action to take. These continually reinforce that the executive is interested.
Hatry: At the federal level, one of the unique features has been support from both the Congress and the Executive, both political parties, and the Office of Management and Budget, and the General Accounting Office (GAO). This bipartisan support is remarkable and distinguishes the current work from previous budgetary and management reform efforts. We recognize that this is not the case in many results efforts. Where bipartisan support may not exist, a law on the books is helpful, although it is no guarantee.
Wholey: People who take this seriously also start looking hard at the way organization does business. This is the concept of “alignment.” Some people say that those organizations that really get into performance measurement find that the current way of doing business is not the right way to do business. They find they must reexamine the organization, the functions of the organization, the way the organization manages itself, and manages its systems, such as personnel and budgeting. Who takes responsibility for data is just one of the many things that is reexamined if people say they are going to succeed in achieving outcome goals.
Hatry: If you want to embed managing by results into an organization, you really need to decentralize it. The program manager has to be responsible for outcome information. This does not mean that they themselves collect it, but they have to be responsible for those data. However, you do not want to decentralize too much. For example, I think most organizations probably need some sort of central technical group who can provide technical assistance and some quality oversight to program people. This might be the evaluation people, those who are experts in measurement. For example, a federal or state agency may have twenty programs, and each wants customer surveys. The twenty should not do it all themselves. A central group could provide advice, check the questions, and maybe even manage the administration of the surveys.
Wholey: In some agencies “partnerships for results” are emerging. The Coast Guard is a good example of this. The agency is holding itself accountable for safety in the coastal waters. By examining data, the agency learned that the primary problem was with tugboats. Further analysis revealed that the problem with tugboats was not their equipment, but rather their operation. As a result, the Coast Guard has focused less on inspections and has formed an alliance with the tugboat industry to improve the training of tugboat operators.
Implementing the Government
|
What have been some of the key constraints that agencies have faced in designing and implementing performance measurement systems?
Hatry: Use of information has been, in general, a major weak spot. The focus to date has been on accountability and not managing for results. This is the issue Joe and I keep coming back to. Until that focus is changed, getting managers to collect information that is useful to them will not occur.
Wholey: To address this, you that need to look for opportunities to create incentives. Some United Ways are trying to do this by using multi-year grants. The hope is that funded agencies will do more reporting on results and less filling out of grant applications. Multi-year performance-based contracts with providers is another approach. In the United Way's National Learning Project Evaluation we are finding that some of the leading United Ways are focusing their training on helping people to use performance information to improve program quality and to communicate the value of the program to those outside the agency. In the public sector as well, being able to communicate the value of what programs do is a big deal, so it is important to show managers how to explain what their program accomplishes and to have actual data to show what the program accomplishes. Legislatures are properly skeptical—everyone has a wonderful program. Being able to have some evidence of what you produce is of value in the environment of tight resources we currently face.
Hatry: A major concern is what happens when the news is bad. Managers today at all levels fear the results of all of this. This is something I encounter at all levels—that program managers do not trust others to use the data properly. This is probably because they know that it is usually the negative people who don't like their program who will pick up any data they have. All of this is a two-edged sword, and somehow we have to get managers and legislators to be reasonable and constructive in the way that they handle information. That continues to emerge as one of the key problems that may occur. If the data are used only to batter the agencies, then we have a problem. To alleviate this problem, there should be an explicit provision stating that when a performance report is provided, explanatory information must accompany it (giving the agency managers the opportunity to tell their side of the story.
Wholey: I believe we should reorient our concept of performance measurement to include three things: 1) annual (or more frequent) numerical information; 2) annual (or more frequent) narrative assessment of what is being accomplished; 3) program evaluation studies to give a fuller picture of what the program produces and to address the knotty questions of whether the program makes a difference.
Six Challenges to Making GPRA WorkRemarks by Vice President Al Gore at the Government Performance and Results Act Conference On April 23, 1998, the Council for Excellence in Government sponsored a conference on the Government Performance and Results Act (GPRA) at which Vice President Gore spoke about the five years since the passage of GPRA. He noted that federal government agencies have prepared strategic plans and developed measures of progress and stressed that the next challenge is to “make the act work.” To do this, the Vice President suggested that: (1) There is a need to focus oversight and management attention on outcomes and asking customers what is important, and how well they think agencies are doing. (2) There is a need to develop a set of common visions, goals, and measures that reach across organizational boundaries and use the resources and energy of many agencies to solve complex problems. (3) Information technology is key. Technology can deliver services across agency boundaries, provide real-time information on performance, and allow more one-stop shopping. (4) In the long run, shared accountability means rethinking the traditional decision-making process that centers around the budget. This might mean moving to a biennial budget and focusing more of the policy debate on performance rather than budget. (5) There is a need to create incentives so leaders aren’t just required to do the right thing, they want to do it. This might mean the creation of performance-based organizations which create incentives for employees to be customer-focused and results-based and would grant agencies greater flexibility from rules in order to help them achieve results. (6) In the long run, there is a need to rethink how the government reorganizes to deliver its services. This would mean building agencies—and a congressional committee structure—that work more on horizontal than vertical lines. |
Now that the performance measurement building blocks are in place in many agencies, what are the next steps?
Hatry: Getting managers to use the information is very important although very difficult to achieve. You need to get regular feedback into the system so managers can make changes, and this is still one of the major uncertainties we have today.
Wholey: Some of the organizations that have been collecting outcome data for some time are taking the second step, which is to begin to understand these data in terms of degree of difficulty. This is designed to discourage “creaming.” One means by which they are doing this is to group programs or organizations with similar characteristics together when assessing performance. This has been done in education, where schools with similar characteristics (such as student socioeconomic status) are compared with one another. Another way to do this comparison is by using a regression model; this is done in job training where local economic conditions and client characteristics are taken into account. The point is that, for those working in a tougher situation, performance scores need to be treated more kindly, while those working in less difficult situations should be held to a higher standard of performance.
Hatry: Performance partnerships, where different organizations work together to achieve results, are a promising approach. This is almost inherently necessary in human services since upper levels of government typically do not deliver direct services. The indicators that really mean something are contributed to by many different parties. A family services program may have federal, state, and local policies, monies, and community entities. Through these partnerships, people start to realize that to improve outcomes is not the responsibility of one agency or one level of government—it is shared.
Wholey: Performance partnerships have many interesting features. But, at a more technical level, there is the issue of a common set of indicators. Many funders are enthusiastic about performance-based approaches but what each wants is different. It would be useful to have some national agreement on common indicators in the human services area. To do this would require a two-step process. First, you need agreement on what the goals are. It is a mistake to go to the measures first. At this point, you can also think about how to use performance information once you have it and what kind of incentive systems you want. Second, you need agreement on sensible measures of progress.
Hatry: Another issue that is going to come up is the quality control issue, and that has not yet been faced up to—once you have a performance measurement system in place, how do you retain and oversee the quality? Will Congress trust the agencies' numbers? With a grain of salt—they will rely on GAO. It is very important to try to address data quality at all levels (program level, upper management, and legislature) once you have the system in place.
Wholey: Typically, there is going to be more than one measurement system. Managers can use less valid information because they have other clues as to what is going on. Policymakers are further away, and they typically will want more validation of the information. I have spent my whole life working on utility of performance information as more important than validity and reliability issues—both sets of issues have to be faced and ought to be faced. In Canada, the Office of Auditor General has taken the position that it wants to see managers use the information to try to improve their programs and that a lot of the other things will get worked out. If managers use the information in their work, managers become concerned with the quality of the information.
Hatry: It is extremely important that people who will be involved in this sort of thing get acclimated to it as early as possible. This begins in the universities. We need to focus on the importance of the concept of government-by-results, and results-based performance measurement in universities and in-service training. This will help people understand the outcome focus and inculcate the values and understanding needed to create the culture change required of a results-based approach.
Wholey: We can also advance things if, at a central level, we create suitable incentive systems for better results. Bonuses on the basis of results achieved is one way—the Department of Health and Human Services is experimenting with this in the context of welfare reform. But financial bonuses are not always necessary. You can do a lot with partial deregulation or more onerous supervision as rewards and penalties. I think that getting measures and using the information to try to do better is important and that the creation of intangible incentive systems is an important way to get that to happen.
Karen Horsch, Research Associate, HFRP