The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume XIII, Number 1, Spring 2007
Issue Topic: Advocacy and Policy Change
Evaluations to Watch
Strategies for Assessing Policy Change Efforts: A Prospective Approach
Justin Louie and Kendall Guthrie of Blueprint Research and Design outline the steps for advocacy and policy change evaluators to follow in using a prospective approach to evaluation.
Prospective evaluation defines a policy change project's short- and long-term goals up front and then emphasizes evaluating advocates' progress toward those goals throughout the life of the project. By more deeply integrating evaluation with program implementation, prospective evaluation provides funders with indicators of success long before policy change can be achieved. It also collects insights that advocates can use to refine strategies and document impact to funders and constituents.
Benefits of a prospective approach include:
- Insuring up-front alignment of expectations
- Providing a framework to assess impact and monitor progress
- Delivering feedback to refine strategy and implementation
- Encouraging advocate engagement in the evaluation process
- Promoting a learning culture
Prospective evaluation involves five basic steps:
1. Understand the context and policy environment. Grantees, funders, and evaluators involved in assessing progress in policy change need high-quality knowledge about the environment in which change occurs. Stakeholders should know the institutions, decision-making process, and culture of public policy, including:
- The decision makers (legislature, executive, administrative agency, court system, or general public)
- The potential tactics (education, organizing, litigation, and mobilization)
- The political context (Which party is in power? What external pressures on decision makers can help or hinder you?)
- Your potential opponents (Who is working against you?)
- Your allies (Who can you count on for help?)
- Your capacity to do the work (Should a window of opportunity open, what skills, networks, and organizational capabilities can you use to take advantage of it?)
2. Develop a theory of change. A theory of change explains how and why a project's activities are expected to lead to desired policy changes. It provides a road map to policy change, based on an assessment of the policy environment in which you are working. In many cases, policy change will be just one component of a larger social change strategy, and a theory of change can also define how specific targeted policy changes relate to larger social change goals.
Developing a theory of change need not be complex or time consuming. Often, answering some guiding questions can help jump-start the process:
- What is the problem you're trying to solve?
- What will be different if you're successful?
- What activities will you undertake to achieve your goal?
- What factors will accelerate or inhibit progress?
A theory of change must adapt to the evolving policy environment. If a strategy no longer looks like it will work and you decide to take another route, the theory should map that change.
Related ResourcesAvailable from The California Endowment at www.calendow.org: Guthrie, K., Louie, J., David, T., & Crystal Foster, C. (2005). The challenge of assessing policy and advocacy activities: Strategies for a prospective evaluation approach. Guthrie, K., Louie, J., & Crystal Foster, C. (2006). The challenge of assessing policy and advocacy activities: Part II—Moving from theory to practice. |
3. Define benchmarks. Generally, policy change is a long-term effort, demanding many years of work. Defining benchmarks to show progress along the way is vital to an effective and useful evaluation.
Benchmarks are milestones set in advance that indicate progress. They should include incremental progress in both achieving policy goals and building internal capacity for policy advocacy. Capacity-building benchmarks are especially important markers of long-term progress. These benchmarks indicate growth in an asset that can be applied to both current and future projects. Advocates have more control over and therefore can be held more accountable for capacity-building goals.
4. Collect data. Data collection requires time, energy, money, and organization. Often advocates feel that collecting data draws them away from their “real” work. By keeping data collection simple, building upon the data collection that organizations already conduct, focusing on data that is meaningful to the advocate as well as the funder, and emphasizing learning over accountability, you can ease the burden of advocates' data collection.
5. Use findings. Evaluations need to be useful to both funders and advocates. In particular, evaluations need to answer questions advocates have about their programs, and findings need to be relayed in time to improve their work. At the beginning, evaluation questions must include those that most interest advocates, and, throughout the evaluation, findings need to be presented as close to real time as possible. Interim reports are likely to be more valuable to advocates than a final report at project completion.
Justin Louie
Senior Analyst
Email: justin@blueprintrd.com
Kendall Guthrie
Consultant
Email: kkguthrie@comcast.net
Blueprint Research & Design, Inc.
720 Market Street, Suite 900
San Francisco, CA 94102
Tel: 415-677-9700.