Volume XI, Number 3, Fall 2005

Issue Topic: Democratic Evaluation

Spotlight

Democratic Evaluation in Practice

Cheryl MacNeil, an evaluation consultant, describes the asymmetries of power in evaluation and her efforts to make her evaluation practice more democratic.

My evaluation practice is democratic. I pay attention to power—the power of stories, the power of values, the power of having a voice, the power that some people have over others, and the power of coming to better understand how we know what we know. As such, I choose to participate in practice-oriented evaluation, in which program stakeholders come together to consider the language they are using and how it reflects who has power and to offer their experience-based knowledge.1

Attending to the Storyline
In the mental health programs I evaluate, as is the case with other social programs, professional language frames conversation. The program stakeholders often talk about the field's need for oversight, regulation, medication, and specialized relationships and in doing so, create storylines about deficiencies. Approaching evaluation with a critical lens, I lead participants in questioning how powerful entities—like professional communities, insurance companies, the pharmaceutical industry, and the government—influence these storylines.2 I highlight the existence of the dominant discourses and structure evaluation questions guided by more than just the language and activities of expert-driven narratives. The problem is not that dominant discourse-driven storylines have nothing to offer; the problem rests in their dominance over other conversations.

In response to this problem, I practice inclusion. I create opportunities to shine light on shadowed narratives by using the frameworks of democratic evaluation.3 I do this by inviting into the conversation not just experts but those with practical experience—program participants, direct line workers, family members, friends, and neighbors. I bring them to the table to share their contextualized knowledge with professionals, and I encourage these different social actors to talk about their contrasting assumptions. In short, I promote dialogue and deliberation. Adding new voices to the mix inevitably creates a contrast between dominant and shadowed narratives and brings balance to the dominant storylines. When done well, this transformative process catalyzes larger conversations around issues of social justice.

On a practical level, I offer program participants opportunities to experience democratic evaluation in a variety of ways and at different points in time during the evaluation process. Often, the design phase is the most important time to be inclusive because this is when I negotiate what is important to know and how to go about learning it. Having different stakeholders deliberate about the focus of the evaluation helps to create a balance in whose information needs are being met and sets the tone for the forthcoming evaluation activities. I sometimes find it useful to establish forums for deliberation among program stakeholders to explore specific issues.4 At other times, I become a participant observer in the program community and raise reflective questions or bring attention to provocative points of view. In other instances, I simply encourage program participants to discuss their different values with one another. Always, the primary evaluation activities are dialogue and deliberation.

Relearning the Evaluation Process
Many of the program participants with whom I work have little lived experience with deliberative evaluation practices; most have encountered only minimally inclusive evaluation practices. Typically, decision makers unfamiliar with deliberative evaluation practices ask me to design an instrument that I can use to collect data and then write a report with prescriptive recommendations. I counter such suggestions by offering to be a steward of a deliberative process.5 I suggest an evaluation designed to bring together people from different walks of life to publicly share thoughts, collectively learn, and maybe even enter into an advanced moral conversation.

Within every program, there is always some relearning to do. I play the role of teacher and help participants understand that democratic evaluation is practical and practice-oriented and that it will probably look and feel very different from past evaluation experiences. It may even contradict some of the messages participants have received in their professional training. Democratic evaluation does not call on us to be detached during our inquiry processes, nor does it encourage aloof conversations among professionals. As a democratic evaluator, I seek to bring the conversation into the personal and to guide people into mutually responsible and authentic dialogue.

I experience most success with democratic evaluation when working with learning organizations that actively question the status quo and value critical examination of their partnerships. These organizations honor the opinions of the people they serve and make accommodations to elevate their voices into substantive conversations. They have genuine concern for the problems that their programs address and about the consequences of their actions. Bringing the art of deliberative democratic evaluation into the cultures of these programs is sometimes a struggle, but in the end, it is also a humanizing experience that helps to improve social programs.

1 Schwandt, T. A. (2005). The centrality of practice to evaluation. American Journal of Evaluation, 26(1), 95–105.
2 MacNeil, C. (2004). Critical theory evaluation. In S. Mathison (Ed.), Encyclopedia of evaluation. (pp. 92–94). Thousand Oaks: Sage.
3 Ryan, K. E., & DeStefano, L. (2000). (Eds.). Evaluation as a democratic process: Promoting inclusion, dialogue, and deliberation [Special issue]. New Directions for Evaluation, 85; House, E., & Howe, K. R. (1999). Values in evaluation and social research. Thousand Oaks, CA: Sage Publications.
4 MacNeil, C. (2000). Surfacing the realpolitik: Democratic evaluation in an antidemocratic climate. New Directions for Evaluation, 85, 51–62.
5 MacNeil, C. (2002). Evaluator as steward of citizen deliberation. American Journal of Evaluation, 23, 45–54.

Cheryl MacNeil, Ph.D.
Consultant
Cheryl MacNeil Consulting, Inc.
176 Fourth Street
Troy, NY 12180
Email: macred@capital.net