The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume V, Number 2 & 3, 1999
Issue Topic: Methodology
Promising Practices
Evaluating Collaboratives: Challenges and Practice
Ellen Taylor-Powell from the University of Wisconsin-Cooperative Extension examines the challenges of collaboratives, and how they stretch us to think about evaluation in new ways.
During the past years, the faculty and staff of University of Wisconsin-Cooperative Extension have increasingly found themselves engaged in collaborative work, playing unfamiliar roles or interacting in new situations with new players. The venue might be a community-based initiative, an interagency partnership, or an interdisciplinary work team in the area of agriculture, family living and nutrition education, 4-H and youth development, or community and economic development. Familiar evaluation practices—ones largely built on a discrete or distinct program delivered by one agency—do not fit this new context.
Collaboration is widely heralded as a mechanism for leveraging resources, dealing with scarcities, eliminating duplication, capitalizing on individual strengths, and building capacities. It offers the possibility for increasing participation and ownership, strengthened by the potential for synergy and greater impact. Yet, for all of us working in and with collaboratives, the challenges are numerous. Several, in particular, are stretching us to think about evaluation in new ways.
Evolving Nature
Collaboratives are dynamic and flexible, changing as they develop. They may look different from year to year. The membership or the roles of members may change, bringing new direction and emphasis. Some collaboratives have a clearly defined start and finish time, but many do not. They may not even start out being a collaborative. There is no grand plan. Rather, the work and direction of the group are invented as the members work together. Implementation may never be complete and is often difficult to track.
In this setting, process becomes particularly important. Conventional program evaluation and many current funders focus on activities that are delivered. We find, however, that special attention must be paid to the workings of the collaborative if it is to successfully deliver activities and programs. This includes the capacities, operations, and climate of the collaborative. In this outcomes era, we are finding it necessary to educate our constituents and funders about process and the linkage between process and outcomes.
Integrating process evaluation into a collaborative provides information for internal decision making, visibility, legitimization, and accountability. There are several ways in which this has been done. In one case, a community-based initiative guards 15 minutes at the end of every meeting to engage in a “how are we doing” process. Members facilitate a question-and-answer period on a rotating basis and write up and share the results. Accomplishments and issues being worked on are featured in a quarterly column in the local newspaper. In another case, an interdisciplinary work team has enlisted one member to evaluate its process and progress. This member summarizes the minutes of meetings and uses content analysis to keep track of decisions, actions, and achievements; observes team interactions and discussions; and conducts an annual survey of members to assess levels of satisfaction, capacity development, and operations. Ongoing feedback is provided to the team, and annual reports are submitted to administration. In yet another example, a community partnership is using the group member survey developed by Cooperative Extension to engage members in a process of self-assessment and learning. Members were informed of the evaluation opportunity and they committed to completing the survey. Teams of members volunteer to present and discuss the results at a series of meetings that engage the full group in interpretation and action related to the findings.
Broad Goals and Expected Outcomes
Collaboratives form for many reasons. Some seek to develop and sustain resilient families or communities. Others are initiated to provide a particular service, to leverage resources, to coordinate efforts, or to effect greater integration of activities or services. Other collaboratives instigate social activism, or seek to create consensus around politically charged issues such as land use or school improvements. Often groups may not have a clearly defined or single purpose or one that all agree to or understand.
Coming to a shared understanding of the collaborative’s goals and expected outcomes is critical. Time is needed to discuss and negotiate this vision as the collaborative forms and as the collaborative evolves, since initial expectations, context, or membership may change. Creating a logic model—mapping the collaborative journey—makes explicit the ideas members hold about what results are desired and how to achieve them. We have found that drawing the logic model, either individually or as a group, is a fun and useful process. We use newsprint and allow members to use any metaphor, design, or thought process to show the chain of events and desired final outcomes. Members then share their pictures; similarities and differences are noted and discussed as the beginning of building consensus on expected outcomes and the strategy for achieving them. Stakeholders are involved in the process to spread understanding and ownership.
Multiple Outcomes
Collaboratives often struggle with defining and measuring outcomes. As part of the logic modeling, differentiating among types of outcomes—which outcomes; for whom—helps members set realistic expectations. There may be outcomes for individuals that include changes in attitudes, knowledge, skills, behavior, actions, and/or lifestyles for clients, community residents, collaborative members themselves, and/or service providers. There may be outcomes for groups such as changes in interactions, values, or behaviors of families, the work group, the community group, or the collaborative itself. Often, collaboratives are focused on agency or organizational outcomes such as changes in service delivery, resource generation and use, practices, and policies. Some collaboratives are interested in system outcomes in which agencies, departments, or whole organizations work in new ways, behave differently, share resources, and provide services in an integrated fashion. Finally, collaboratives may be focused on outcomes for communities, including changes in norms, policies, or actions at a community-wide level. Collaboratives may lead to the institutionalization of change or the empowerment of individuals or groups. We find that collaboratives often have impact in more than one area, and the unanticipated outcomes are significant in terms of human and social capital development.
Asking questions about outcomes—what outcomes, for whom, when might we see them, how will we know it, what else is happening that we didn’t anticipate?—helps us detect and document results. One example of this is a family preservation and support initiative that collects longitudinal data through a series of interviews with participating families to document their evolving ability to deal with stress, access community resources, and reduce family isolation (family outcomes). In another case, a collaborative of youth-serving agencies, working to open up membership to youth, compiled membership data before and after the two-year effort to assess changes in membership (agency outcomes). In yet another example, a community gardening initiative collects data through observation, interviews with participants and leaders, a sample survey of residents, and logs kept by staff to assess interaction among neighbors and changes in attitudes, helping behaviors, and responsibilities concerning the neighborhood (community outcomes).
Individual Performance
Collaboratives are built on the premise that the whole is greater than the sum of the individual parts, but individuals make up the collaborative and many have performance appraisal systems that require evidence of individual performance. How can one assess individual performance without undermining the essence of the collaborative?
Collaborative members, who need to report to their supervisors or are concerned with personal accountability, are using a variety of techniques for defining their own contribution to the team effort and outcomes. These include using a log book or diary to track one’s inputs, activities, outcomes, and impact; using meeting minutes or other documentation to determine the role and influence of individual members; engaging members in evaluating each other in a nonjudgmental process; and using surveys, group discussions, or interviews with key stakeholders to collect data on member contributions and influence on outcomes. When individual members are responsible for a particular activity, it might also be possible to evaluate that event/activity and link the results to the member’s effort.
Collaboration as a Panacea
Collaboratives are being promoted, expected, or required everywhere. Yet, it seems that a critical initial question is often overlooked: Is a collaborative warranted—is it the most appropriate approach? Some groups and/or communities are not ready for collaboration; some problems do not need a collaborative approach; and sometimes an individual’s background and/or the agency’s mission do not fit that of the collaborative. As we look at this dimension, a variety of evaluative questions emerge that relate to the context and readiness for collaboration in the community and external environment, in the organization, and among stakeholders.
We think about this as feasibility evaluation that typically occurs at the beginning of an initiative and is often informal. Some collaboratives are using a readiness questionnaire to ascertain a number of factors, such as the willingness of the community and/or individuals to work together, levels of cooperation and trust, history of previous work, potential barriers to success, and the availability of leadership. Besides the initial feasibility evaluation, each time the collaborative changes, adds new members, or begins a new initiative, it seems important for the group to discuss the feasibility of each—the potential for success, the resources needed, and the compatibility. Again, we are finding it important to help partners and funders consider whether collaboration is the best approach and what technical assistance may be necessary.
Our approach as partners in collaboratives across Wisconsin is to view evaluation as learning and as a shared process among members and stakeholders. Evaluation in the collaborative context becomes a collaborative process itself. The purpose, direction, and expectations for evaluation are negotiated among the collaborative members. When evaluation becomes a part of the collaborative, it provides the focus, feedback, and learning to support continuous progress and growth.
Parts of this article are taken from Taylor-Powell, E., Rossing, B., & Geran, J. (1998). Evaluating collaboratives: Reaching the potential. Madison, WI: University of Wisconsin-Cooperative Extension. (Available at http://s142412519.onlinehome.us/uw/pdfs/G3658_8.PDF.)
¹ A collaborative is defined as a group working together to achieve a shared vision. Members engage in a process where they constructively explore their differences and search for (and implement) solutions that go beyond their own limited vision of what is possible. [Gray, B. (1989). Collaborating: Finding common ground for multi-party problems. San Francisco, CA: Jossey-Bass.]
Ellen Taylor-Powell
Evaluation Specialist
University of Wisconsin-Cooperative Extension