You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼


Cami Anderson and Sybilla Dorros from Harvard Family Research Project describe four new approaches and innovations of established methods for evaluating CBIs with examples.

The complexity and importance of community-based initiatives (CBIs) cause programs and evaluators to rethink traditional methods of evaluation in tracking the diverse processes, outcomes, and goals of CBIs. Moreover, as CBIs aim to become learning organizations, designing tools that offer them useful information about how the organization is working and assessing its outcomes will require innovative approaches to evaluation methodology.

We highlight four methods frequently used in evaluating CBIs, discuss each method briefly, enumerate its usefulness in a CBI evaluation, provide an example of a CBI using this method, and then suggest follow-up readings. As we accrue more experience with these methods (alone and in mixed-method strategies), their creative uses and limitations in CBI evaluations will become increasingly clear.

1. Ethnography

The goal of ethnography is to create a vivid, first-hand description of communities in order to obtain a detailed understanding of their strengths. Ethnography can paint a clear, ground-level picture of poverty by studying how individuals, families, and communities experience it. Ethnography can also help researchers develop a comprehensive understanding of how individuals and families deal with larger factors (such as immigration patterns and the loss of factory jobs). Through the combined use of field observations, maps of ecological and institutional features, and measures of formal and informal relationships, ethnographers document the complex experiences, behaviors, and perceptions of small populations.

Ethnography is useful in evaluating CBIs because it:

  • Utilizes an exploratory process, without preconceived notions of community process or externally defined outcomes.

  • Engages the community in change—a goal of many CBIs.

  • Shares the goal of neighborhood-based initiatives to capitalize on indigenous, existing, and natural, but perhaps less well-known, strengths in the community.

Example of a CBI evaluation that is using the ethnography methodology:

The Annie E. Casey Urban Children's Mental Health Initiative is designed to reform systems of mental health services in four national sites. The ethnographic component of the evaluation (by Susan Greenbaum) covers four areas: a summary of neighborhood history (changes in population and housing quality); a topographic description of locations and significant places (internal boundaries, geographic distribution of risk areas); interviews with twenty key informants (about conflicts among subgroups, social cohesion, existence of community groups); and field observations of the project site (including socializing and cohesion).

Further reading:

Greenbaum, S. (in press). The role of ethnography in linkage with communities: Identifying and assessing neighborhoods needs and strengths. In M. Hernandez, & M. Schockley-Isaacs (Eds.), Promoting cultural competence in systems of care. Baltimore, MD: Paul Brooks Publishing. In this chapter, the author explores how and why ethnography can be useful in evaluating CBIs. She includes a theoretical overview and specific examples of ethnographic activities that can help evaluators identify community needs and strengths.

Jarrett, R. (in press). Community context and intrafamilial process, and social mobility outcomes: Ethnographic contributions to the study of African American families and children in poverty. In G. K. Brookings, & M. B. Spenser (Eds.), Ethnicity and diversity. In this chapter, the author discusses the critical role ethnography can play in understanding the effects of poverty. Specifically, she examines the contributions of past and present ethnographic research in identifying factors that produce social mobility in African American families.

Sullivan, M. (1996). Neighborhood social organization: A forgotten object of ethnographic study? In R. Jessor, A. Colby, & R. Shweder (Eds.), Ethnography and human development: Context and meaning in social inquiry. Chicago, IL: University of Chicago Press. This essay discusses the substantive and methodological contributions of ethnographic community studies and their relevance to the study of human development, poverty, and community change. The author refutes the notion that ethnography should be thought of as a marginal, impressionistic, or imprecise tool.

2. Theories of Change Evaluations

Theories of change methods are specifically designed to involve practitioners in the evaluation process from the very beginning by helping them articulate the logical relationships among services, outcomes, programs, and other variables. These methods identify implicit theories about how and why a program will work. They can then detail the implications for intervention design and outcomes. Evaluators and practitioners work together in defining these theories and developing methods of data collection that will test their merit. The goal is to determine where there is evidence that the theory is working and where there are gaps in the approach so that the initiatives can make useful adjustments.

Theories of change evaluation methodology is useful for evaluating CBIs because it:

  • Focuses the evaluator on what to measure, concentrating on key elements of the complex initiative.

  • Asks program practitioners to make explicit the assumptions underlying their actions.

  • Gives programs valuable, accessible, and non-threatening information about their progress—making the use of evaluation findings more likely.

  • Assists evaluators in determining whether program failure or success is the result of implicit program theories or implementation.

  • Provides the basis for a substantive partnership among CBI evaluators and the community—a benefit that is consistent with CBIs' emphasis on empowerment and community engagement in the change process.

Example of a CBI evaluation that is using theories of change evaluation methodology:

Claudia Coulton, at the Center on Urban Poverty and Social Change at Case Western Reserve University, is conducting a theories of change evaluation of the Cleveland Community-Building Initiative (CCBI). CCBI was set up in 1994 to address poverty in Cleveland comprehensively and, from the beginning, a theories of change approach has been a key component of the Initiative. From the planning phase to the implementation phase, evaluators and practitioners have worked together to articulate their theories about how and why they plan to address poverty comprehensively. They have worked closely with various stakeholders in the area to come to some consensus on the particulars of their comprehensive approach and to identify the process and outcome factors they will be tracking. The first phase of the evaluation has been completed and the findings will be published in Vol. II of The Aspen Institute's Roundtable. (See below.)

Further reading:

The Aspen Institute Roundtable on Comprehensive Community Initiatives for Children and Families. (in press). New approaches to evaluating community initiatives (Vol. 2). New York: Author. This collection of commissioned papers will present a theories of change approach to the evaluation of comprehensive community initiatives (CCIs) through a conceptual overview and several empirical examples. It will then examine related evaluation issues, including measurement, small areas data, and causal attribution challenges.

Chen, H.-T. (1990). Theory-driven evaluations. Newbury Park, CA: Sage Publications. This volume presents an overview of the history and justification for theory-driven evaluation. It also examines the advantages and disadvantages of using evaluations based on practitioner theories.

Weiss, C. (1995). Nothing as practical as a good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In J. Connell, A. Kubisch, L. Shorr, & C. Weiss (Eds.), New approaches to evaluating community initiatives. New York: The Aspen Institute. This chapter explores the reasons for using theory-based evaluation in examining CCIs. The author argues that theory-driven evaluations should be integrated with outcome indicators for accountability in order to capture the complexity of CCIs.

3. Census and Survey Data

Census and survey data can allow evaluators to look at broad patterns and trends across large and diverse populations in order to track individual and community-wide change over time. They can also give program managers useful information with which to make program decisions and target their intervention strategies. Useful census and survey data include indicators of socioeconomic status, household composition, and service use. CBI evaluators identify existing census and survey data and design and conduct new surveys to understand neighborhoods better, as well as track change.

Census and survey data are useful in evaluating CBIs because they:

  • Help complex initiatives determine priority areas for action.

  • Identify program responsiveness to the community by quantifying residents' perceptions and attitudes.

  • Track the unique experiences of population subgroups (e.g., youth, homeless, immigrant families), and provide information for more targeted interventions.

  • Indicate community service availability and usage, and optimism about change by tracking residents' perceptions of mobility, identity, and neighborhood quality.

Examples of CBI evaluations that are using census and survey data:

The evaluation framework developed by the OMG Center to evaluate the Annie E. Casey Rebuilding Communities Initiative includes three distinct forms of data collection (among other methodologies): a neighborhood inventory which includes indicators such as the use of child care, parent education classes, adult education classes, and number of neighborhood-based organizations; a neighborhood survey designed to identify residents' perceptions of indicators such as neighborhood quality, safety, and future; and agency databases to track indicators such as neighborhood stability (residential mobility), quality of housing, and rates of voter registration and turnout.

Further reading:

Coulton, C. J., Korbin, J. E., & Su, M. (1996). Measuring neighborhood context for young children in an urban area. American Journal of Community Psychology, 24(1), 5–32. This article explores a method of measuring neighborhood environments using the average perceptions of caregivers of young children. It details salient themes from the authors' previous ethnographic research comparing neighborhoods classified as high-, medium-, and low-risk for children.

Earls, F., McGuire, J., & Shay, S. (1994). Evaluating a community intervention to reduce the risk of child abuse: Methodological strategies in conducting neighborhood surveys. Child Abuse and Neglect, 18, 473–485. The article examines the use of attitude surveys in measuring neighborhood perceptions of change as a result of a comprehensive CBI in Dorchester, MA.

Sommer, T., Brown, P., Chaskin, R., Goerge, R., Richman, H., Slavitt, L., et al. (1996). Creation of a community information infrastructure: Capturing the breadth and depth necessary for the effective planning, implementation, and evaluation of comprehensive community change efforts. Chicago, IL: Chapin Hall Center for Children. This paper discusses the rationale and strategies for the development of a community information infrastructure. It describes an approach to mapping information in such a way that a detailed, complete picture can be drawn about circumstances, needs, and opportunities. It emphasizes ways to develop data with and for the community to help guide their work.

4. Computer Tracking Systems

Because CBIs are complex, spanning many services, program areas, and community goals, gathering information on all of these diverse components can be challenging. Computer tracking systems such as Management Information Systems (MIS) and Geographic Information Systems (GIS) are particularly useful to complex initiatives in tracking large amounts of information and making it accessible to programs. The purpose of an MIS is to enhance the ability of a CBI to deliver effective and coordinated services. It is analogous to a large data entry system with interface screens that allow managers to enter and track data on individuals, families, providers, staff, available resources, and the community in general. MIS systems are customized to the needs of each initiative. A GIs allows practitioners and policymakers to select a geographic area and physically map various types of data throughout the community (e.g., physical location of resources, schools, household incomes).

Computer tracking systems are useful for evaluating CBIs because they:

  • Manage and store large quantities of multilevel data, and make them accessible in a way that enhances programs and services.

  • Serve efficiently two purposes at once by providing information useful for both improving program processes and tracking program process.

  • Cut down on unnecessary paperwork and costs once the initial investment in training and equipment is complete.

  • Track clients across program components and meet data management challenges that arise from service integration.

Examples of CBI initiatives that are using computer tracking systems:

Project MATCH, a welfare-to-work program in Chicago, is a long-term, individualized, developmental program whose aim is economic self-sufficiency. The Project's sophisticated, customized MIS program allows them carefully to document their individual and collective successes and provides useful public policy advocacy data. A larger initiative, the Comprehensive Community Revitalization Program, aims to link five well respected Community Development Corporations in the South Bronx in an effort to effect total community change. A crucial component of their planning and implementation has been the use of both GIs and MIS strategies throughout the area.

Further reading:

Chow, J., & Coulton, C. Cleveland Area Network for Data and Organizing (CAN DO) project summary. Cleveland, OH: Center for Urban Poverty and Social Change, Mendell School of Applied Social Sciences, Case Western Reserve University. This paper summarizes the CAN DO project, whose main goal is to make a comprehensive database on the neighborhoods of Cleveland available to nonprofit and government agencies. Included is a timetable of implementation and examples of elements in the database.

Herr, T., & Halpern, R. (1991). Changing what counts: Re-thinking the journey out of welfare. Chicago, IL: Center for Urban Affairs and Policy Research, Northwestern University. This paper describes the philosophy of Project MATCH, including the program's approach to tracking individual progress toward economic self-sufficiency with the help of an MIS system.

Markze, C., Both, D., & Focht, J. (1994). Information systems to support comprehensive human services delivery. Washington, DC: National Center for Service Integration. This study examines the current practice of and future opportunities for using information systems to integrate service delivery. The authors conclude that while there is potential for using these systems to streamline services, reduce duplication, and track outcomes, few providers have implemented them effectively.

Pew Partnership for Civic Change. (1995, Winter). The online community: Smaller cities in the information age. Charlottesville, VA: Author. This report discusses the ways in which members of a community—citizens, nonprofit groups, and the local government—can use online technology to meet their individual and collective needs.

The HUD Office of Community Planning and Development offers software (Community Connections) that collects, analyzes, and studies trends in select neighborhoods. You receive mapping software, details about HUD-funded projects in the community you select, and census data (including unemployment and income levels). You can also enter your own customized information. Available from HUD (1-800-998-9999) or email at amcom@sys.com.

Cami Anderson, Research Assistant, HFRP

Sybilla Dorros, Editor-in-Chief, HFRP

‹ Previous Article | Table of Contents | Next Article ›

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project