Volume IX, Number 3, Fall 2003

Issue Topic: Evaluating Community-Based Initiatives

Evaluations to Watch

Mixed Methods and Moving to Opportunity

Elisabeth Jacobs reflects on the value of mixed-methods research in a policy context, highlighting the example of Moving to Opportunity, the five-city demonstration program of the U.S. Department of Housing and Urban Development.

A clearer understanding of the links between communities and individual outcomes would benefit policymakers seeking to design and implement targeted, effective, and efficient programs. Embracing mixed-methods research is critical for clarifying our understanding of neighborhood effects. Detailing the causal relationship between communities and outcomes for the families and individuals in these communities requires both qualitative and quantitative methodologies.

Being Shamelessly Methodologically Eclectic
Policy evaluations traditionally collect quantitative data. Via broadly distributed surveys, institutional reporting, and other means, policy researchers garner statistical information on the outcomes of the individuals affected by a given policy. Unfortunately, the collection of qualitative data is far less routine. While quantitative data can provide a picture of a community’s opportunities and obstacles, qualitative analysis offers a unique opportunity to unearth how those effects occur. Moreover, qualitative research can enrich our sense of neighborhood effects by corroborating the aggregate trends in the quantitative survey-based research. Finally, qualitative data can enliven our understanding of the social issues at stake and provide a more human story for policymakers, funders, and the public.

Policy evaluators can think about qualitative research as methodological value added. As Rossman and Wilson describe in their aptly titled article, Numbers and Words Revisited: Being “Shamelessly Methodologically Eclectic” (1994), qualitative research can complement quantitative analyses in at least four ways:

1. Corroboration of patterns in the numbers. Do the qualitative and the quantitative results tell the same story?
2. Elaboration of the statistical evidence. The qualitative “enhances, clarifies, and illustrates” in ways that can be enormously important for extracting lessons.
3. Development—using the results of one method to shape the other method. For example, qualitative results can be used to shape later iterations of surveys used to collect quantitative data.
4. Initiation of additional and sometimes entirely new and even divergent conceptual directions and research angles beyond those suggested by the statistics or prior literature.

Not only can qualitative research be value added to survey-based evaluation projects, it can also elucidate aggregate-level community variables. It provides clearer answers to questions about patterns of social influence, development of micro-level patterns of social organization, nuances of job search behavior, and other multi-stage social processes.

Neither qualitative nor quantitative research can single-handedly provide comprehensive policy evaluation. Without corresponding broad-based survey research and quantitative analysis, qualitative analysis cannot reliably indicate the scale or representativeness of particular effects. Without the “thick description” qualitative analysis provides, quantitative survey data cannot reveal the nuances of the social processes they enumerate. The synergy of both allows for a comprehensive analysis that can balance a persuasive, generalizable analysis with nuance and complexity.

Methodological Diversity and Moving to Opportunity
Moving to Opportunity (MTO), a demonstration program sponsored by the U.S. Department of Housing and Urban Development (HUD), provides an excellent example of the rich substantive findings that mixed-methods approaches to policy evaluation can yield. The program targets very low-income families with children living in public housing or receiving project-based assistance under Section 8¹ in five cities—Boston, Baltimore, Chicago, New York, and Los Angeles—and aims to enable these families to move out of high-poverty neighborhoods into low-poverty neighborhoods in the same metropolitan area.

HUD has implemented an experimental design aimed to answer two critical questions:

1. What are the impacts of mobility counseling on families’ neighborhood choices and on their housing and neighborhood conditions?
2. What are the impacts of neighborhood conditions on the education, employment, income, and social well-being of MTO families?

HUD’s design randomly assigns participants in MTO into one of three groups. The experimental group receives Section 8 rental certificates useable in only low-poverty areas as well as counseling and assistance in finding a private unit to lease. The comparison group receives regular Section 8 rental certificates with no geographical restrictions, as well as typical benefits and assistance from HUD. Finally, the control group continues to receive its current project-based assistance.

HUD’s five-year evaluation of MTO is centered around a large-scale, “structured” survey of child and family well-being that draws on decades of research on neighborhood influences on family and child outcomes. Besides this core of quantitative research, MTO evaluation work has included several important qualitative elements.

Given the traditional adherence to quantitative methods amongst economists, it is perhaps a bit ironic that one of the most telling examples of the synergies made possible from mixed-method research comes from a team of economists. In 2001, Kling, Liebman, and Katz published Bullets Don’t Got No Name: Consequences of Fear in the Ghetto, which illuminated the importance of safety concerns in the mobility decisions of low-income parents in public housing projects in high-poverty neighborhoods. Based on a series of 12 in-depth interviews with MTO-participating families in Boston, the study emphasizes the need to focus research on health, child behavior, and employment in order to capture major effects of change in neighborhoods. Subsequent quantitative results showed that the largest effects were for the very indicators that the qualitative interviews suggested would change.

The authors discovered four main reasons for incorporating mixed-methods into their MTO analysis:

1. It refocused their quantitative data collection on a different set of outcomes.
2. It enabled them to develop an overall conceptual framework for thinking about the impact of high- and low-poverty neighborhoods on families.
3. It provided them with a deeper understanding of the institutional details of the MTO program, which helped them to more confidently interpret later findings.
4. It gave program participants a voice in shaping the researchers’ questions, supplying lessons with important implications for housing policy.

Taking the Next Steps: Future Research With MTO
While MTO’s five-year evaluation is winding down, HUD is gearing up for a second wave of evaluations, and many important avenues remain open for exploration. A June 2002 conference on qualitative research in the MTO evaluation generated an exciting agenda of new issues and concerns. The day-long discussion amongst policymakers, evaluators, HUD funders, and researchers revealed several areas critical for further study:

• Neighborhood institutions. A variety of perspectives could shed light on this category. A social integration perspective would shift the focus from families to community institutions as the organizing unit of social life; researchers could analyze schools, churches, and other neighborhood institutions to understand how these institutions serve to integrate (or segregate) communities. A service provision perspective would give policymakers a clearer sense of neighborhood service availability.
• Family interactions with (and in) neighborhoods. Increased research focus on the interaction between families and the local police, for example, might clarify quantitative findings on crime, and focusing on family-neighborhood interactions might indicate where low-income families are going for social supports.
• Economic opportunity. Low-income families’ moves to low-poverty neighborhoods provide a unique opportunity to further explore the relationship between geography and economic opportunity. For instance, qualitative research could explore the role of social relationships behind economic opportunity. Do low-poverty neighborhoods influence low-income movers’ social norms in such a way as to make them more likely to work or to work at higher paying jobs?

Mixed-methods research is invaluable not only for academics seeking to explain causal processes, but also for policymakers seeking to develop effective policy. Qualitative research provides a “story” to which policymakers can respond. As policy advocate Barbara Sard explains, “The most valuable aspect of the qualitative work from a policy perspective has been getting an understanding of the ‘why’ mechanisms. Fancy statistics are nice, but stories are better” (Briggs & Jacobs, 2002).

For more information on MTO and related research see www.mtoresearch.org.

¹ The Section 8 Housing Assistance Program was enacted by the Housing and Community Development Act of 1974. Section 8 housing rental certificates subsidize low-income families’ housing costs so that they can afford housing in the private market.

Elisabeth Jacobs
Ph.D. Candidate, Department of Sociology
Doctoral Fellow, Multidisciplinary Program in Inequality and Social Policy
Harvard University
37 Concord Avenue, #4
Cambridge, MA 02138
Tel: 617-497-1766
Email: esjacobs@fas.harvard.edu