Volume X, Number 3, Fall 2004

Issue Topic: Harnessing Technology for Evaluation

Theory & Practice

Taking the Next Step: Harnessing the Power of Technology for Evaluation

Arnold Love, an internationally recognized independent consultant with more than 20 years' experience in evaluation, is the guest editor of this issue of The Evaluation Exchange. In this edition of Theory & Practice, he provides a conceptual map of the issue's theme—the uses of technology in evaluation.

Technology offers evaluators the opportunity to design and implement more effective evaluations while reducing costs. Although many evaluators are already reaping the benefits of technology, most have not moved much beyond email, teleconferencing, and accessing websites. They have not made technology an integral part of the evaluation process. There is a pressing need for evaluators to become aware of advances in technology and to appreciate those advances relevant for the evaluation field. With technology changing so rapidly, a closely related challenge is developing a “mental map” of the technology landscape so that we can appraise and select the right tools to meet our evaluation needs.

What do I mean by technology? In the evaluation arena, the term refers to information and communications technology (ICT), also known by its street name, compunications, a trendy word that emphasizes the union between computers and telecommunications. ICT has two immediate benefits for evaluators:

1. Enhancing the use of familiar methods (surveys, interviews) to achieve better, less expensive evaluations
2. Accessing the wellspring of innovation (wireless handheld devices, cellphone camcorders, etc.) to fashion new tools and create new evaluative processes (such as real-time analysis and collaborative weblogs, or blogs)

What Are Evaluators Doing With Technology?
A survey of the landscape shows four major areas in which evaluators are using information and communications technologies successfully: data collection and analysis, collaboration, knowledge mobilization, and capacity building. ICT is rapidly expanding the tools for generating evaluative knowledge, broadening participation and collaboration, fashioning highly effective channels for mobilizing evaluation information into focused action and social change, and linking evaluators together into dynamic, worldwide communities of practice. To provide a context and to help draw a clear mental map of the evaluation applications described in this issue, the following paragraphs review how evaluators are using technology in each of these areas.

Data collection and analysis. ICT permits evaluators to collect quantitative and qualitative data¹ more accurately, rapidly, and inexpensively than ever before. Evaluators are using websites to download consent forms, post data collection protocols for program staff, supply online help, act as gateways to secure servers for data storage, and streamline data collection by presenting videos that explain the informed consent process to potential participants. To improve evaluation design, evaluators are conducting literature searches using online electronic resources, such as e-journals, e-books, and specialized databases. They are using email and the Web to survey target groups rapidly, enter data automatically, and generate nearly instantaneous feedback. Evaluators are conducting individual and group interviews by email, online discussions, real-time chat, and videoconferencing. The ease of incorporating digital graphics, streaming video, and sound files makes organizing, analyzing, and presenting multimedia data not only feasible but practical. Finally, by building data warehouses and compact data marts for speedy access to historical and current data, and by using automated analysis tools for program monitoring, evaluators are able to spend more time supporting users and making evaluations useful.

Collaboration. Tools for electronic collaboration provide the means to do effective group work. Collaboration tools enhance the entire evaluation process in several important ways. Contemporary evaluation practice values participation by diverse stakeholders, but stakeholder involvement can be time-consuming and expensive, especially if travel is required. Electronic collaboration tools allow stakeholders to participate without concern for time or travel. Discussion forums permit evaluators and stakeholders to collaborate more effectively by visually grouping and storing messages into threads that permit a discussion to be followed easily, and by supplying tools for conducting secret ballots on important issues.

Evaluations are becoming more complex, with multiple partners, funders, and program sites now the norm. Electronic collaboration tools, which include scheduling, calendaring, approval tracking, and other project management features, help evaluators manage complex evaluation projects while staying on time and on budget. Evaluators often must craft multiple versions of evaluation reports, with each version tailored to the unique needs of a specific audience. Collaboration tools reduce this burden and its potential for error with sophisticated document management functions, such as version control and document routing, so that documents are disseminated to the right audiences at the right time. In short, e-collaboration tools enable evaluators to lighten their workloads and reduce costs while improving the evaluation process by building trust and a truly inclusive participatory process.

Knowledge mobilization. One of the most important applications of ICT is knowledge mobilization—using evaluation findings to mobilize action and drive change. Knowledge mobilization moves beyond disseminating evaluation findings to the strategic use of evaluative knowledge to realize an organization's mission. Technology implementation usually follows a three-step path. The first is acquiring access to technology; for example, virtually all government and nonprofit organizations in the United States and Canada have Internet access. The second step is adopting basic technology (word processing, email, statistical packages, and static websites) and related skills, but only on a limited basis. The third is embracing the networked world and using ICT in a strategic way to promote change and strengthen communities. For evaluators, strategic use of technology means broadening participation, expanding evaluation audiences, forging new collaborations, and maximizing the impact of information in a way that will mobilize change.

Evaluation capacity building. Technology is helping to build the capacity for evaluation and to develop the infrastructure for supporting evaluators and contributing to their professional development. Perhaps nowhere is the power of technology for evaluation capacity building more evident than in the experiences in Africa and in Central and South America. When the development aid agenda began to change in the mid-nineties, evaluators in these two regions faced the daunting task of developing indigenous evaluation capacity with little time and even less money. Taking advantage of the Internet, evaluators created mailing lists, online forums, and discussion groups to foster cooperation and build an extensive infrastructure of regional and national evaluation networks. They then created online databases and filled them with articles, documents, and links to resources in the languages and traditions relevant to the region. They built “knowledge networks” that offered online meetings, conferences, and workshops; these knowledge networks are now themselves spawning “child networks” of their own.

In addition to its role in these efforts, technology has already had widespread impact on the very structure of these organizations. Instead of bricks-and-mortar hierarchical arrangements, organizations are choosing to develop as networks and virtual communities of practice, interacting with each other in highly collaborative ways that lend a strong and unified voice across each region to promote evaluation.

Challenges and Next Steps
History shows that technology is rarely adopted unless it meets a need. Therefore a key question is, what will technology do for evaluators and what aspects of evaluation practice will it facilitate or replace? Another challenge is the furious rate of technological change, coupled with the common practice of rushing products to market amid overheated sales rhetoric and exaggerated promises. Should evaluators use technology simply because it is available, or should use of technology be based on specific evaluation considerations? Finally, given the potential for misuse and the often high-stakes nature of evaluative information, should stricter standards and ethical guidelines apply to the use of technology for evaluation?

Harnessing the power of ICT for evaluation depends largely on appreciating the pros and cons of these tools and using this knowledge to shape technology to our own ends. The inherent flexibility of electronic tools should encourage us not to fixate on the hardware and software but to direct our attention to improving the evaluation process: making it more inclusive and transparent, building truly collaborative evaluation efforts, removing the drudgery from data collection and focusing more on data analysis and use, and vastly increasing the reach and impact of evaluative information for the betterment of all.

¹ Quantitative data are objective, numerical data that can be quantified. In contrast, qualitative data provide descriptive details, often collected from a purposive sample of interviews, focus groups, or observations.

Arnold Love, Ph.D.
40 St. Clair Avenue East, Suite 309
Toronto, ON M4T 1M9
Canada
Tel: 416-485-2159
Email: ajlove1@attglobal.net