The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume V, Number 2 & 3, 1999
Issue Topic: Methodology
Promising Practices
At What Price? Benefit-Cost Analysis and Cost-Effectiveness Analysis in Program Evaluation
James Edwin Kee, professor at George Washington University, discusses the purposes, strengths, and limitations of benefit-cost and cost-effectiveness analyses to determine the relative costs and benefits of the programs.
Introduction
In our current age of accountability, public and private sector funders are increasingly concerned with the relative costs and benefits of the programs they fund. Benefit-cost (or cost-benefit) and cost-effectiveness analyses can be useful quantitative tools to help address these concerns. However, they differ in their purposes, and each has strengths and limitations.
Benefit-cost analysis is an applied branch of economics that attempts to assess service programs by determining whether total societal welfare has increased (in the aggregate, people have been made better off) because of a given project or program. It can be used in evaluations of existing programs to assess their overall success or failure, to help determine whether the programs should be continued or modified, and to assess the probable results of proposed program changes. Benefit- cost analysis consists of three steps: (1) determine the benefits of a proposed or existing program and place a dollar value on those benefits; (2) calculate the total costs of the program; (3) compare the benefits and the costs.
Cost-effectiveness analysis is an alternative to benefit-cost analysis that relates the cost of a given alternative to specific measures of program objectives. A cost-effectiveness analysis helps to compare costs to units of program objectives and may be the first step in a benefit-cost analysis if the analyst then decides to attempt to place a dollar value on the benefits. Unlike benefit-cost analysis, cost-effectiveness analysis does not produce a “net benefit” number, with benefits exceeding costs or costs exceeding benefits. However, a cost-effectiveness analysis can determine that a program which costs $1 million produces ten units of outcome x, twelve units of outcome y, and twenty units of outcome z. Or, if the units are alike, it can determine the cost per unit of outcome.
An example of these two methods of analysis using a hypothetical dropout prevention program is presented in Box 2.
Box 2: Hypothetical Cost-Effectiveness and Benefit-Cost Results for Dropout Prevention StrategiesCost-Effectiveness The cost-effectiveness of each dropout prevention strategy is determined by dividing the cost for each strategy by its effectiveness (e.g., the percentage increase in the number of students graduating). The result is the cost for each percent increase in the number of students graduating.
Benefit-Cost The benefit-cost for each dropout prevention strategy is determined by calculating each strategy’s benefits (e.g., estimates of future earning increases of participants who stayed in school) and costs (e.g., personnel, materials, equipment) and then subtracting the benefits from the costs to get the net benefit for each strategy. The benefit-cost ratio can also be computed by dividing the dollar value of benefits by the costs (the higher the ratio, the more efficient the program in economic terms).
|
Challenges in Conducting Benefit-Cost and Cost-Effectiveness Evaluations
Identifying and Measuring Costs
Identifying and measuring costs, and in the case of benefit-cost analysis, quantifying and placing a dollar value on the benefits, is the biggest challenge to the evaluator trying to conduct these types of analyses. Direct costs (such as personnel, materials, and equipment) are often relatively easy to account for. Indirect costs (such as overhead, costs to other providers supporting the intervention, and costs to participants) as well as capital costs (such as buildings and computers) can be more difficult to calculate. Finally, intangible costs (such as the value of wilderness) are those for which the evaluator either cannot assign an explicit price or chooses not to. Lack of assigned price does not mean that intangible costs are unimportant; indeed, in presenting any results of these types of analyses, the evaluator should point out the intangible costs and benefits, thereby enabling the decision maker to consider these as he or she examines those benefits and costs that are quantified. When identifying any benefit or cost, it is important to state its nature clearly, to state how it is being measured, and to list any assumptions made in the calculation of the dollars involved.
Identifying and Measuring Benefits
Identifying benefits can also be tricky. As with costs, there are direct, indirect, and intangible benefits. In the case of benefit-cost analysis, placing a dollar value on the benefits is also a challenge. The evaluator might choose a market value, when one is available, or a surrogate such as willingness to pay. Because of the redistributional nature of government programs, public agencies and those who evaluate them must be concerned with who benefits as well as the amount of benefits in addition to the costs.
Where quantifying benefits is difficult, costly, or viewed as inappropriate, cost-effectiveness analysis can be used. Cost-effectiveness evaluation does not require that the evaluator place a dollar value on the benefits. This is particularly useful in cases where the benefit of a program is “lives saved.” While there are various ways to place a dollar value on a life saved or lost, each is controversial. In contrast to a benefit-cost analysis, a cost-effectiveness evaluation would calculate the cost of the program per life saved without making a judgment about the dollar value of that life. The evaluator would then present the results to the decision maker who must decide whether an outcome is worth the dollar cost when viewed in light of alternative uses for the funds.
A major challenge in cost-effectiveness analysis is the fact that programs frequently generate more than one type of benefit. For example, an education system might target more than one population group in the school system. When conducting a cost-effectiveness analysis comparing programs with multiple benefits, the evaluator may need to place weights on the relative benefits to assist the decision maker in making comparisons. If this is not done, the comparison becomes quite subjective. Yet assigning weights often becomes at least as problematic as assigning dollar values to each benefit: how do the benefits to one population group outweigh those to another, for example?
Boundaries
Another challenge in conducting benefit-cost and cost-effectiveness analyses is determining the geographic scope of an analysis. While the focus may be within a certain jurisdiction, such as a state, there may be benefits or costs that spill over to neighboring jurisdictions. It might be tempting to ignore spillover effects, but this can be unwise since spillovers often have political consequences. The question for the evaluator is whether to consider only those benefits and costs that accrue to the population within the jurisdiction for which the evaluator is doing the analysis.
Detail
One of the biggest dangers in these analyses, as in many other areas of evaluation, is the “black box” syndrome. Instead of laying out the relevant issues, assumptions, and concerns, the evaluator may be tempted to hide the messiness of the analysis from the decision maker, presenting a concise answer as to the net benefits or costs or cost-effectiveness. However, it is the detail—the assumptions involved and the sensitivity of the analysis to particular assumptions—which may be of most use to the decision makers in judging the value and usefulness of the evaluator’s work.
Deciding Between Cost-Effectiveness Analysis and Benefit-Cost Analysis
Those faced with deciding between the two types of analysis may find it helpful to keep three basic questions in mind:
1. How will you use the results? Benefit-cost analysis enables you to compare strategies that do not have the same outcomes, or to compare strategies across different areas of public expenditure (e.g., health, welfare, justice). Cost-effectiveness analysis is useful for comparing strategies that are trying to achieve the same objective (e.g., increased graduation rates).
2. What resources do you have? Benefit-cost analyses typically require more resources, because they take more time for analysis and involve significant methodological expertise (often in economics), such as the capacity for determining the discounted present value of a stream of benefits and costs.
3. How difficult are costs and benefits to value? While you may want to have as much information as possible on both benefits and costs, you must weigh the value of the increased accuracy gained from the accumulation of new data against the costs associated with the data collection. Thus, any analysis should begin by assimilating existing data to determine whether it is sufficient.
For Further ReadingKee, J. E. (1994). Benefit-cost analysis in program evaluation. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation. San Francisco, CA: Jossey-Bass. Levin, H. (1983). Cost-effectiveness: A primer. Thousand Oaks, CA: Sage Publications. (This is one of the volumes in the New Perspectives on Evaluation series.) Yates, B. (1996). Analyzing costs, procedures, processes, and outcomes in human services: An introduction. Thousand Oaks, CA: Sage Publications. |
The more intangible the benefit (for example saved wilderness), the more likely it is that a cost-effectiveness analysis will be of greater use to decision makers. This type of analysis can help them assess whether a cost is justifiable, when compared with other uses of the same funds.
It is important to note that benefit-cost analysis and cost-effectiveness analysis could lead to different conclusions about the same program, depending upon how benefits are valued in dollar terms. However, if the evaluation is concerned with a program with a single objective (or closely related objectives), programs or alternatives achieving the highest cost-effectiveness should also achieve the highest benefit-cost ratio.
Neither benefit-cost analysis nor cost-effectiveness analysis is a panacea. Both require judgments on measurement issues that should be brought to the attention of the decision maker. However, both techniques are useful to provide a format for analysis that can lead to better decisions.
James Edwin Kee
Giant Food, Inc. Professor of Public/Private Management
School of Business and Public Management
George Washington University