Volume XIII, Number 1&2, Spring 2007

Issue Topic: Advocacy and Policy Change

Evaluations to Watch

Working With Logic Models to Evaluate a Policy and Advocacy Program

Authors from the Institute for Health Policy Studies at the University of California, San Francisco describe how they used both macro-level and individual grantee logic models to drive the evaluation design of the Clinic Consortia Policy and Advocacy Program.

In response to challenges related to the shifting health care environment, community health centers in California have joined together to form regional consortia and statewide organizations. Consortia vary in size, staffing, scope, and age, but all provide a unified voice for increasing services to the uninsured, offering economies of scale for shared business and program services, and allowing clinics to partner on local health improvement programs to benefit clients. Activities often include coordinated policy advocacy efforts, group purchasing agreements, centralized HMO claims management, grant writing and management, joint managed-care contracting, and billing support. Collaborative efforts assist in reducing costs, improving efficiency, and enhancing the effectiveness of community health centers.

As part of its commitment to increasing access to high-quality and affordable health care for underserved Californians, The California Endowment (The Endowment) provided multiyear funding for the Clinic Consortia Policy and Advocacy Program. In early 2001, 15 California re-gional community clinic associations and 4 statewide clinic organizations (“consortia” or “grantees”) received 3 years of funding (totaling $9 million) to strengthen the role and capacity of consortia to support community clinic management, leadership development, policy, and systems integration needs. This funding supported activities related to policy advocacy, technical assistance, media advocacy, and shared services to increase the collective influence of clinics. In 2004, 18 grantees were re-funded for 3 years to undertake a similar set of activities.

Program Theory Drives Evaluation Design
In 2002, an evaluation team at the University of California, San Francisco (UCSF) commenced a multiyear evaluation to assess the Policy and Advocacy Program. The evaluation design was based on The Endowment's theory of change—that staffing and resources for policy advocacy and technical assistance would increase the collective influence of clinics and strengthen a broad base for long-term support of clinic policy issues. In addition, the theory hypothesized that policy advocacy activities, or activities that mobilize resources to support a policy issue or create a shift in public opinion, are critical for expanding local and state support for community clinic funding.

The UCSF team's macro-level program logic model (see figure below) is based on this theory of change. Specifically, increased grantee capacity in policy advocacy was expected to lead to increased policymaker awareness of safety-net and clinic policy issues and to increased policymaker support for clinic funding. Policy wins and increased funding were then expected to translate into strengthened clinic operations, in-creased services for the underserved and uninsured, and improved health outcomes for targeted communities and populations.

The model was based on four theoretical domains:

1. Political science. Representation, or how and what type of in-fluence interest groups bring to bear in influencing policy and the effectiveness of this influence
2. Partnerships. The role of relationships in extending an organization's reach and the gains to partner organizations
3. Organizational development. The gains from expanding capacity to ensure sustainability for the hub organization and its members
4. Media. The ability to increase decision maker and public awareness of policy issues and the potential for influencing policymaker support on a specific policy

The logic model organized the specific program objectives under three key goals:

1. To conduct policy and advocacy activities to increase awareness among policymakers and the public regarding the central role of community clinics and increase funding for comprehensive health services to underserved residents (through advocacy partnerships, policymaker education, media advocacy, and advocacy technical assistance)
2. To engage in specific quality of care improvement projects to improve patient outcomes
3. To provide clinics with technical assistance and resources to maintain or improve financial stability within clinics
4. Anticipated short- and intermediate-term outputs and outcomes were then listed for each objective.

Triangulation: Complementary Evaluation Methods
The UCSF team used this macro-level logic model as the framework for the evaluation design. To assess program outcomes as outlined in the logic model, the team administered both quantitative and qualitative tools, including longitudinal worksheets, open-ended interviews and surveys, focus groups, and financial data analysis. The worksheets tracked data on partnerships, policy advocacy activities, policy wins, and funding se-cured.

To describe the benefits, challenges, and impacts of grant-funded activities, qualitative data collection strategies included grantee interviews (annually), member clinic focus groups (2004, 2006), and nonmember clinics interviews (2006).

To assess decision makers' familiarity with consortia, clinic activities, and clinic policy issues, the team administered a policymaker awareness survey to policymakers and community leaders known to consortia in 2003 and those who were less familiar with consortia in 2004.

Finally, the UCSF team worked with The Endowment to develop an interim report template whereby information submitted to the foundation could be imported into individual grantee logic models and vice versa. The team analyzed grantee interim and final reports submitted to The Endowment from 2001 to 2005, noting the outcomes or successful passages of each policy.

Evaluation Tool Kit and Technical Assistance
Early in the evaluation, UCSF assessed grantee evaluation capacity and data systems, identifying outcomes and the logic model process as an area that needed strengthening. Consequently, the UCSF team incorporated grantee logic models into the evaluation tool kit and provided grantees with technical assistance on their development and use. UCSF worked with grantees to develop and annually update logic models to document progress in achieving their individual objectives. Each year, UCSF staff transferred information from the grantee interim reports to the grantee logic models and sent the models back to grantees to check for accuracy and completion.

Grantees and UCSF staff reviewed grantee logic models during a phone interview, noting the outputs (services, activities) completed and evidence of achievement of short-term outcomes in the previous year (qualitative and quantitative information). UCSF aggregated and analyzed this information, noting overall achievement of outputs and short-term outcomes. The logic models were also an excellent source for detailed information on particular policy advocacy activities, policy “wins,” partnerships, and programmatic expansions. (Most grantees have the same short-term outcomes.) In the past, UCSF gave grantees final versions of their models in an Excel matrix. In 2007, UCSF will develop a narrative describing grantee achievements of their respective outputs and outcomes and areas for strengthening.

Lessons Learned
In the beginning, there was limited grantee support for the logic model process, driven in part by requests for models from multiple funders and the plethora of logic model approaches. UCSF greatly simplified the process and assumed most of the responsibility for developing and updating grantee models. Additionally, UCSF continued to emphasize the link between the macro logic model and grantee models through inclusion of grantees and The Endowment in annual discussions on the evaluation's goals and focus. Last, the alignment between the interim reports required by The Endowment and the individual grantee logic models greatly reduced the administrative burden while yielding significant evidence of grantee progress and achievement on individual and overall program outcomes.

California's network model is being watched closely by other large states. The evaluation of the Clinic Consortia Policy and Advocacy Pro-gram has made significant headway in developing (or corroborating) the original hypotheses, most notably in detailing the ways in which consortia have maximized their relationships with decision makers and become a potent voice in the policy arena on behalf of clinics and their patients.

Annette Gardner, Ph.D., M.P.H.
Principal Investigator
Tel: 415-514-1543
Email: annette.gardner@ucsf.edu

Sara Geierstanger, M.P.H.
Senior Researcher
Email: sara.geierstanger@ucsf.edu

Institute for Health Policy Studies
University of California, San Francisco
3333 California Street, Suite 265
San Francisco, CA 94118