The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
Volume VIII, Number 3, Winter 2002
Issue Topic: Public Communications Campaigns and Evaluation
Evaluations to Watch
International Communication Research and Evaluation at the Johns Hopkins University Center for Communication Programs
Maria Elena Figueroa from the Johns Hopkins University Center for Communication Programs reveals the Center’s methods for evaluating communication campaigns and offers five examples of their evaluations in progress.
The Johns Hopkins University Center for Communication Programs (JHU/CCP) was established in 1988 to focus attention on the important role of research-based strategic communications in health promotion and behavior change. The Center provides communication expertise worldwide on a range of issues, including family planning/reproductive health, HIV/AIDS/STD prevention, child survival, women’s health, public health advocacy, health care reform, and environmental protection.
JHU/CCP has representatives in more than 25 countries, and through its over 700 country-based projects and programs, has helped transform the theory and practice of public health communication.
The Center follows a theory-driven, research-based approach to its communication, which gives research and evaluation a critical role. Built into different project stages, research and evaluation contribute to the understanding of the processes through which communication affects behavior for purposes of continuous program improvement.
Evaluation designs and methods of analysis used by JHU/CCP include:
-
Pre and post data collection (often using surveys) with audience members exposed to the campaign and control or comparison groups of those not exposed
-
Panel surveys in which the same members of the campaign’s audience are interviewed before, during, and after the campaign
-
Interrupted time-series analysis of indicators (e.g., health service statistics) before and after the campaign
-
Analysis of intervening effects using statistical techniques such as multiple regression to rule out factors that may be confounding evaluation results, and path analysis to determine the effects of intervening variables and the causal pathways of communication
-
Interaction analysis to examine the synergistic effects of mass media, interpersonal communication, and participation in community-level activities
-
Cost-effectiveness analysis, which divides the total cost of the communication intervention by specific indicators of change attributed to the campaign
-
Communication (social) network analysis to assess how information diffuses within a group or community, and to assess the effects of social influence on the adoption and maintenance of new behaviors
-
Image (character or method) mapping to scale audience perceptions
JHU/CCP has many evaluation studies underway worldwide; five examples follow. Others include evaluations of community-based and mass media communication programs for safe motherhood in Guatemala and Indonesia, water and sanitation in Nicaragua, men’s involvement in reproductive health in Jordan, and HIV and AIDS in India.
Arab Women Speak Out (seven Arab countries)
Arab Women Speak Out (AWSO) is an empowerment training program to help Arab women overcome social, economic, educational, and political obstacles. It is centered on the belief that media images can offer role models that inspire action. It uses media materials, produced by Arab women researchers and filmmakers, that include video profiles of women who have made positive changes in their lives, a tool for monitoring the image of women in the media, and a case study publication of 30 women from Jordan, Yemen, Egypt, Lebanon, Tunisia, and Palestine.
Workshops give women the analytic tools to think critically about their obstacles and opportunities, and to propose their own courses of action. AWSO also helps Arab women develop the self-esteem necessary to discuss their opinions and concerns with their husbands, families, and communities. More than 80,000 women in seven Arab countries have participated.
In 2001, JHU/CCP evaluated AWSO using a posttest only design with a nonequivalent control group. Data showed that participants were more likely than nonparticipants to engage in entrepreneurship, advise other women in negotiation techniques, and participate in activities that further community welfare. Future evaluation plans include assessing AWSO’s community impact using social network analysis to determine the extent to which participants (1) talk to and take action with other participants, (2) are sought out by community members for advice, and (3) are perceived as community leaders. (Contact: Carol Underwood, email: cunderwood@jhuccp.org)
National Family Planning Communication Campaign (Philippines)
The Philippine Department of Health conducted a nationwide campaign in 2000 to increase family planning practice. The campaign broadcast four television spots with messages based on research that revealed women wanted a contraceptive method that was suitable for them or hiyang (a Tagalog word meaning a natural fit to one’s own body), and that a husband’s support for family planning was necessary.
The campaign’s 2001 evaluation used a one-group, retrospective panel survey design. Results from the national sample of 1,516 married women from 16-49 years of age revealed the campaign increased spousal communication and acceptance of family planning as a social norm, and encouraged couples to choose a method that was suitable (hiyang) for them. The campaign reached 79% percent of married women. Exposure was measured by unaided message recall, recognition of television spot photographs, and correct identification of 10 distinct messages conveyed. Overall, modern contraceptive use increased by 4.71 percentage points during the campaign’s one-year period, compared to an annual increase of 1.1 percent over the previous five years.
Final data analysis will control for variables that may have confounded results and will determine the time-order of effects (using structural equation modeling). It will also estimate how much of the observed increase can be attributed to the campaign (using propensity score analysis). The goal is to determine the cost-effectiveness of the campaign. Preliminary findings show that the cost of each new family planning adopter is between $2–$4. (Contact: Larry Kincaid, email: lkincaid@jhuccp.org)
Related ResourcesRecent JHU/CCP contributions on theory and methodology include: Figueroa, M. E., Kincaid, D. L., Rani, M., & Lewis, G. (2002). Communication for social change: An integrated model for measuring the process and its outcomes. New York: The Rockefeller Foundation. Available at www.communicationforsocialchange.org/ Figueroa, M. E., Bertrand, J. T., & Kincaid, D. L. (2002, August). Evaluating the impact of communication programs: Summary of an expert meeting organized by the MEASURE Evaluation Project and the Population Communication Services Project. Carolina Population Center, University of North Carolina at Chapel Hill: Chapel Hill, North Carolina. Available at www.cpc.unc.edu/measure/ Kincaid, D. L. (2002). Drama, emotion, and cultural convergence. Communication Theory, 12, 136–152. Sood, S. (2002). Audience involvement and entertainment-education. Communication Theory, 12, 153–172. |
The Life Choices Program (Ghana)
Low levels of contraceptive use in Ghana are fueled by women’s fears of contraceptive side effects and perceptions that others around them are opposed to family planning. Using facilitated discussions within existing women’s groups and complementary short radio broadcasts, the Life Choices communication program attempts to combat these barriers by increasing social support for family planning and addressing contraception misconceptions and concerns.
The evaluation is using a panel survey design, administering two surveys (n=509) at two different time points to participants in nine women’s groups. Three of these groups participated in Life Choices prior to the first survey; three groups participated between the two surveys; and three groups will participate after the second survey (control group).
In addition to assessing knowledge, attitudes, behaviors, and media exposure, surveys ask participants to name their social network and their perceptions regarding the attitudes and behaviors of these women. The network of relations among women in each group will be mapped to understand the direct and indirect flow of information about family planning and identify how the Life Choices program activities contribute to this information diffusion. (Contact: Marc Boulay, email: mboulay@jhuccp.org)
Safe Motherhood Communication Strategy (Nepal)
Maternal death is the largest cause of mortality for women of reproductive age in Nepal. The Nepal Maternal and Neonatal Health Program’s Safe Motherhood communication strategy aims to improve pregnancy outcomes by encouraging the use of appropriate maternal and neonatal health practices and services.
Messages are based on a creative Nepali language-based pneumonic memory aid that encourages audiences (especially husbands and mothers-in-law) to care for pregnant women, share love and affection, and prepare for childbirth. The multimedia strategy embeds messages in radio spots and dramas, a television drama, mobile video van shows, community-based street theater performances, and print materials.
The evaluation is using pre and post door-to-door surveys with currently pregnant women, their adult family members, community-based and government health workers, and select community influentials. A snowball sampling technique (asking those you survey to refer you to other respondents) is being used to reach a sample of 1,200 women and men. (Contact: Suruchi Sood, email: ssood@jhuccp.org)
HEART Campaign (Zambia)
The Helping Each Other Act Responsibly Together (HEART) Campaign, designed specifically for and by Zambian youth ages 13 to 19, uses television to inform young people about HIV/AIDS, discusses ways to avoid infection, and promotes abstinence and condom use.
The evaluation, which used pre and post surveys and a control group of nonviewers, found that viewers (those respondents who saw at least one televised message) were 1.5 times more likely to report primary or secondary abstinence (return to abstinence) than nonviewers. Viewers were also 1.7 times more likely to report using a condom during their last sexual encounter (holding background characteristics constant). (Contact: Carol Underwood, email: cunderwood@jhuccp.org)
Maria Elena Figueroa
Chief, Research and Evaluation Division
Johns Hopkins University Center for Communication Programs
111 Market Place, Suite 310
Baltimore, MD 21202
Tel: 410-659-6300
Email: mfiguero@jhuccp.org
Website: www.jhuccp.org