The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume XIII, Number 1&2, Spring 2007
Issue Topic: Advocacy and Policy Change
Beyond Basic Training
A Guide to Measuring Advocacy and Policy
A Guide to Measuring Advocacy and Policy
Organizational Research Services identifies outcomes associated with advocacy and policy work based on its new resource, A Guide to Measuring Advocacy and Policy.
While many foundations and nonprofits are interested in measuring their advocacy and policy work, currently no commonly accepted evaluation approach or practice exists. To help remedy this, the Annie E. Casey Foundation commissioned Organizational Research Services (ORS), a Seattle-based evaluation consulting firm, to create a guide that would help both the Casey Foundation and other organizations better define and document the effectiveness of their advocacy and policy strategies.
A Guide to Measuring Advocacy and Policy gives foundations and nonprofits alike a way to identify and talk about the outcomes associated with advocacy and policy work. In addition, it offers evaluation design suggestions that include a broad range of methodologies, intensities, audiences, timeframes, and purposes. The guide aims for wider acknowledgement about how evaluation fits into the world of advocacy and policy, greater acceptance of evaluation's role in demonstrating success and learning about progress, and increased confidence among those undertaking evaluation in this area.
While outcome categories are fairly standardized and widely accepted in the service delivery arena, such standardization does not yet exist for advocacy and policy work. The guide highlights a core set of outcome categories and provides concrete direction for those searching for what to measure about their advocacy and policy strategies.
In developing the guide, ORS reviewed a broad range of advocacy and policy outcome categories and indicators of progress. Repeatedly, the same categories of outcomes emerged. Some represent the interim steps and infrastructure that create the conditions for social change; others reflect the end goal-policy adoption, funding, or implementation and enforcement.
ORS distilled these outcomes into six distinct categories, described below, that represent the essential changes in lives, community conditions, institutions, and systems that result from advocacy and policy work. The table at right also presents these categories and the specific outcomes, strategies, and units of analysis that relate to them. The order in which outcome categories appear does not represent their relative importance or sequence.
1. Shifts in social norms. Social norms are the knowledge, attitudes, values, and behaviors that comprise the normative structure of culture and society. Advocacy and policy work increasingly has focused on this area because of the importance of aligning advocacy and policy goals with core and enduring social values and behaviors.
2. Strengthened organizational capacity. Organizational capacity is another name for the skill set, staffing and leadership, organizational structure and systems, finances, and strategic planning of nonprofits and formal coalitions that do advocacy and policy work. Development of these core capacities is critical to advocacy and policy change efforts.
3. Strengthened alliances. Alliances among advocacy partners vary in levels of coordination, collaboration, and mission alignment and can include nontraditional alliances such as bipartisan alliances or relationships between unlikely allies. Alliances bring about structural changes in community and institutional relationships and are essential to presenting common messages, pursuing common goals, enforcing policy changes, and protecting policy “wins.”
4. Strengthened base of support. Nonprofits draw on grassroots, leadership, and institutional support in working for policy changes. The breadth, depth, and influence of support among the general public, interest groups, and opinion leaders for particular issues are a major structural condition for supporting policy changes. This outcome category spans many layers of culture and societal engagement including increases in civic participation and activism, “allied voices” among informal and formal groups, the coalescence of dissimilar interest groups, actions of opinion leader champions, and positive media attention.
5. Improved policies. Change in the public policy arena occurs in stages—including policy development, policy proposals, demonstration of support (e.g., cosponsorship), adoption, funding, and implementation. Advocacy and policy evaluation frequently focuses on this area as a measure of success. While and important focus, improved policies are rarely achieved without changes in the preconditions to policy change identified in other outcome categories.
6. Changes in impact. Changes in impact are the ultimate and long-term changes in social and physical lives and conditions (i.e., individuals, populations, and physical environments) that motivate policy change efforts. These changes are important to monitor and evaluate when grantmakers and advocacy organizations are partners in social change. Changes in impact are influenced by policy change but typically involve far more strategies, including direct interventions, community support, and personal and family behaviors.
A Guide to Measuring Advocacy and Policy is available on the Annie E. Casey (www.aecf.org) and ORS websites (www.organizationalresearch.com). In addition, The Innovation Network's website (www.innonet.org) offers an online supplement to the guide with sample measurement tools directly applicable to advocacy and policy work. We hope that the outcome categories described here and in the guide begin to provide foundations and nonprofits with a common approach to policy and advocacy evaluation.
Menu of Outcomes for Advocacy and Policy Work |
|
| 1. Shift in Social Norms | |
| Examples of outcomes | • Changes in awareness • Increased agreement about the definition of a problem (e.g., common language) • Changes in beliefs • Changes in attitudes • Changes in values • Changes in the salience of an issue • Increased alignment of campaign goal with core societal values • Changes in public behavior |
| Examples of strategies | • Framing issues • Media campaign • Message development (e.g., defining the problem, framing, naming) • Development of trusted messengers and champions |
| 2. Strengthened Organizational Capacity | |
| Examples of outcomes | • Improved management of organizational capacity of organizations involved with advocacy and policy work • Improved strategic abilities of organizations involved with advocacy and policy work • Improved capacity to communicate and promote advocacy messages of organizations involved with advocacy and policy work • Improved stability of organizations involved with advocacy and policy work |
| Examples of strategies | • Leadership development • Organizational capacity building • Communication skill building • Strategic planning |
| 3. Strengthened Alliances | |
| Examples of outcomes | • Increased number of partners supporting an issue • Increased level of collaboration (e.g., coordination) • Improved alignment of partnership efforts (e.g., shared priorities, shared goals, common accountability system) • Strategic alliances with important partners (e.g., stronger or more powerful relationships and alliances) • Increased ability of coalitions working toward policy change to identify policy change process (e.g., venue of policy change, steps of policy change based on strong understanding of the issue and barriers, jurisdiction of policy change) |
| Examples of strategies | • Partnership development • Coalition development • Cross-sector campaigns • Joint campaigns • Building alliances among unlikely allies |
| 4. Strengthened Base of Support | |
| Examples of outcomes | • Increased public involvement in an issue • Increased level of actions taken by champions of an issue • Increased voter registration • Changes in voting behavior • Increased breadth of partners supporting an issue (e.g., number of “unlikely allies” supporting an issue) • Increased media coverage (e.g., quantity, prioritization, extent of coverage, variety of media “beats,” message echoing) • Increased awareness of campaign principles and messages among selected groups (e.g., policymakers, general public, opinion leaders) • Increased visibility of the campaign message (e.g., engagement in debate, presence of campaign message in the media) • Changes in public will |
| Examples of strategies | • Community organizing • Media campaigns • Outreach • Public/grassroots engagement campaign • Voter registration campaign • Coalition development • Development of trusted messengers and champions • Policy analysis and debate • Policy impact statements |
| 5. Improved Policies | |
| Examples of outcomes | • Policy development • Policy adoption (e.g., ordinance, ballot measure, legislation, legally binding agreements) • Policy implementation (e.g., equity, adequate funding, other resources for implementing policy) • Policy enforcement (e.g., holding the line on bedrock legislation) |
| Examples of strategies | • Scientific research • Development of “white papers” • Development of policy proposals • Pilots/demonstration programs • Educational briefings of legislators • Watchdog function |
| 6. Changes in Impact | |
| Examples of outcomes | • Improved social and physical conditions (e.g., poverty, habitat diversity, health, equality, democracy) |
| Examples of strategies | • Combination of direct service and systems-changing strategies |
Jane Reisman, Ph.D.
President
Email: jreisman@organizationalresearch.com
Anne Gienapp
Associate
Email: agienapp@organizationalresearch.com
Sarah Stachowiak
Associate
Email: sstachowiak@organizationalresearch.com
Organizational Research Services
1932 First Avenue, Suite 400
Seattle, WA 98101