Volume lll, Number 3&4, 1997

Issue Topic: Community-Based Initiatives

Questions & Answers

Interview With Mercer Sullivan

Mercer Sullivan is Associate Professor of Criminal Justice at Rutgers University and Senior Research Fellow at the Vera Institute of Justice. He has worked and written extensively in the area of ethnography. We asked Dr. Sullivan to share with us insights and practices related to using ethnography to study community-based initiatives.

What does ethnography offer to the study of CBIs?

In the most general terms, ethnography is the best kind of research method to use for the purpose of exploring what the community element in CBIs is. Communities are not monolithic entities—there are different kinds of people in them and different social networks, even in seemingly homogenous groups–and understanding what is meant by community in practice is a notoriously difficult challenge. Ethnography, which grows out of the community studies traditions of anthropology and sociology, gives us a way to study community-level phenomena such as patterns of social networks, institutional infrastructure, the process of community change, and patterns of conflict and cooperation.

Traditional quantitative methods such as questionnaires and censuses can be used to gather information on communities and these methods can complement ethnography. However, it is difficult, in my opinion, to think about studying communities without going into them. You need to talk to the people in the community about the community and their lives in it.

It is important to recognize that ethnographic work is inductive. It is used for discovery and can help you to surface issues and questions about which you were not previously aware. If you already know the boxes that need to be filled in, you don't need ethnography.

What does one need to consider when incorporating ethnography into the evaluation of CBIs?

When thinking about how to use ethnography in studies of CBIs, several factors need to be considered:

• The level of intensity: Ethnography is an intensive and potentially expensive effort if done correctly. While you can spend some days at a site and get some information, to understand the social fabric in a community, you need a trained person who can spend several months to a year and a half there. This requires a lot of resources. While there are ways to do ethnography with people already involved in the CBI, they need training and have to be willing to commit the time necessary to write field notes, conduct interviews, and analyze the data.

• The relationships: Ethnographers form relationships with the people involved in the CBI and thus, there is a need to think about the match between the ethnographer and the people in the CBI. A good match between ethnographer and community is important for generating community commitment to the research process. A bad match does not mean a bad ethnographer or a bad community; there just may not be a match. The process of matching the ethnographer with the community can be difficult. Some assume that you have to match on race and ethnicity, but that is not always the case. The ethnographer's knowledge, past experience, and commitment are also very important.

• The feedback mechanisms: Ethnographers frequently struggle with the desire to “go native” (which can undercut the objectivity of their research) and the need to maintain distance (which can undercut rapport with a community's residents). One way you can clarify your role is to set up feedback mechanisms. Setting up a structure for communication between the you and stakeholders helps to make clear that you are not working for the CBI but at the same time are not detached from it. This structure should be set up in the beginning of the study and may be renegotiated throughout it. I recommend that regular meetings be set up but that these not be daily since this might put you in the position of providing day-to-day judgment of the effort.

How does ethnography fit with other approaches to evaluating CBIs?

Ethnography is frequently used to complement other research methods. One of the more common ways to integrate research traditions is to use ethnography on the front end to feed into the construction of quantitative measures. For example, an understanding of local experiences helps in designing surveys. It can work the other way as well. You might, for example, do a survey which shows broad patterns that the researcher just cannot figure out. The researcher can then go back into the field and ask the community's residents what they think the results mean.

There are, also, however, questions that really can only reasonably be looked at with one type of method. Questions of history, for example, cannot easily be done with a survey—qualitative methods are better for obtaining a historical profile of a community. Large quantitative surveys, on the other hand, can provide information on patterns that people may not even be aware of.

The best approach is to use a “tool bag” that includes qualitative and quantitative methods. These methods reinforce one another in both discovering and validating information. Using both approaches triangulates findings and increases confidence in them. As I mentioned earlier, the inductive nature of ethnography makes it tremendously helpful at the stage of data analysis—it can help the researcher to identify important questions and is an invaluable resource for interpretation.

What are the challenges in using ethnography to evaluate CBIs and how can these be addressed?

• Gathering comparable information across sites: In many cases, information on communities is not comparable across the sites in the study. The history of each community, for example, is unique. To address this challenge in the studies that I have conducted, I have tried to supervise and coordinate the activities of the ethnographers in each site such that the chances of getting comparable data are maximized. If I see that one story is emerging at a site, I try to see there are similar stories or different stories about the same thing at the other sites.

• Identifying comparison sites: Another challenge when evaluating community-based activities is the issue of comparison sites. Early on in the design of the evaluation, you need to think about whether you will look at a comparison site. As Hollister and Hill point out (New approaches to evaluating community initiatives. (1995). Washington, DC: Aspen Institute, pages 127–172), there are challenges to setting up even a quasi-experimental design. I have used a comparison site in one study. I used ethnography and quantitative methods to match sites. The ethnographic data came from interviews, expert opinion, and visual observation (such as housing stock). I complemented these data with census data on variables such as poverty levels, race, ethnicity, and housing type. However, due to resource constraints, I was not able to do a separate ethnography in the comparison sites. Thus, the qualitative information on the comparison sites had to come from the intervention sites.

• Establishing credibility: Ensuring that findings are credible, both internally and externally, is extremely challenging. Internal credibility is derived from the relationship that you have with people in the sites—do they believe that the information about their community is worthwhile? What might be missing? What distortions might exist? The issue of external credibility is very difficult. Although qualitative research can be laboriously and systematically executed, there is still skepticism about its credibility. It does not come with the same built-in scientific imprimatur as numeric results. This has to do, I think, with the fact that we are not used to seeing the same amount of qualitative research. If we did see more, people would begin to develop standards about what is good and credible. I think the answer is to support good work—the more good work that gets around, the more problems of external credibility will be reduced

What final thoughts can you share regarding ethnography as an approach to evaluating CBIs?

I think there is perhaps a far more fundamental issue that faces those who study and evaluate CBIs. I would describe this issue as the contradiction between the reasons why people are interested in community interventions and their conceptions of the standards by which those interventions are to be judged. In other words, we are interested in community interventions out of a conviction that each community is a unique place and that local involvement will lead to local solutions to local problems. But when we go in to study these things, we work with the mindset of finding the generalizable—the successful essence of things—so we can bottle it and send it to other places. We need to find ways to deal with the demands for judgment about CBI success while preserving the individuality of each community that make us think these are endeavors worth studying in the first place.

Karen Horsch, Research Specialist, HFRP