Volume X, Number 3, Fall 2004

Issue Topic: Harnessing Technology for Evaluation

Promising Practices

Making Voices Heard: Using Multimedia to Give Evaluation a Cutting Edge

Suresh Balakrishnan describes the use of multimedia to disseminate evaluation results in Bangalore, India.

When water taps run dry, doctors at public clinics play truant, garbage piles grow, poor households the world over usually have no option except to cope as best they can with the apathy and inefficiency of public services. Is there a real alternative? Although initiatives like the Millennium Development Goals¹ provide an opportunity for change, sustainable improvement depends on a real push from the people who value public services the most: the users themselves. This article describes an evaluation of public services using Citizen Report Cards, a tool developed by Public Affairs Centre (PAC), in Bangalore, India. Citizen Report Cards are independent and objective evaluations of the performance of public services based on direct feedback from the citizens who use them. The Report Cards make use of multimedia to provide policymakers with cutting-edge evaluation information and compel them to action.

The concept of Citizen Report Cards evolved in Bangalore in the early 1990s as a civil society response to poor urban services. Instead of waiting for reform to occur, a small group of citizens—led by Dr. Samuel Paul, who would later found PAC—organized a user-survey-based evaluation of urban services, such as drinking-water supply, bus transport, garbage disposal, and so on. The assessment was limited in scope, since it was based entirely on what users could observe and experience, for example, ease of availability and reliability of services, politeness of staff, and intensity of corruption. The impact of this experiment led to the creation of PAC, which has developed and facilitated such evaluations across India and other countries.

This initiative used multimedia (e.g., videos that included graphical data, photographs, and presentations of quantitative and qualitative data in computer slide shows) to share evaluation findings. The findings reflected the “voice”of citizens, who included in their report comparisons of different agencies on key dimensions, such as proportion of satisfied users and extent of bribery encountered. This comparison put agencies that were performing poorly to shame and created pressure for change.

Service providers responded quickly to this “voice” of evaluation. While some tried to ignore the initial round of feedback, many service agencies found the process useful for identifying quick wins and treated satisfaction ratings as an independent performance assessment. The comparative ratings injected a form of competition and incentive to change among these largely unmovable monopolies. By initiating and owning these assessments, civil society was able to level the playing field with reference to powerful state institutions and have their demands for better services heard for the first time.

The impact of these evaluations depended heavily on how the results were disseminated. Dynamic multimedia presentations that accentuated variations in quality and gaps in service were crafted specifically for public meetings and for in-house delivery to service agencies. Simple visuals worked best with large public audiences: Videos allowed audiences to hear and see the actual beneficiaries or victims of the service-delivery system present the evaluation evidence. This reporting method surfaced a cutting-edge tool for evaluation and a compelling case for change.

Simple linkages between managerial actions and outcomes experienced by the users of public services become powerful calls for public accountability. By helping to illuminate these linkages, multimedia extended the ownership and reach of evaluations, and generated wider attention and deeper discussions than had taken place before.

International development agencies have recognized the potential of delivering Citizen Report Cards via multimedia for benchmarking and monitoring service standards in a manner that empowers citizens. The experience in Bangalore is being shared across India and other countries, in Africa, Asia, Europe, and South America. Video has been used extensively to introduce Citizen Report Cards to administrators and communities with significantly greater impact than written documents.

Ironically, the success of this technology has also brought some serious challenges in its wake. Multimedia presentations and short videos helped make it much easier to show evaluation findings and set agendas for action. They are not as helpful, however, as a means of portraying enabling factors and providing guidance about the amount and type of hard work over time required to achieve these outcomes. To meet these challenges, PAC is now working on a set of web-based tools and learning products to better respond to the many calls for assistance to initiate Citizen Report Cards.

For more information about the Public Affairs Centre visit www.pacindia.org.

¹ The Millennium Development Goals is a set of goals, targets, and indicators established by the United Nations (UN) in 2000 for the new millennium. The goals and targets are based on the UN Millennium Declaration, and the UN General Assembly has approved them as part of the secretary general's road map toward implementing the declaration. The UN Development Program worked with other UN departments, funds, and programs; the World Bank; the International Monetary Fund; and the Organization for Economic Cooperation Development to identify over 40 quantifiable indicators to assess progress.

Suresh Balakrishnan, Ph.D.
Executive Director
Public Affairs Centre
422, 80 Feet Road
Koramangala VI Block
Bangalore 560095
India
Tel: 011-918-025-537-260
Email: sbalakrishnan@vsnl.net