Volume X, Number 3, Fall 2004

Issue Topic: Harnessing Technology for Evaluation

Spotlight

Wearable Computers and Evaluation

Edward Dieterle, of Harvard University's Handheld Devices for Ubiquitous Learning project,¹ discusses the potential of wireless handheld devices for evaluation and provides a brief glossary of technological tools and terms with which evaluators should be familiar.

Evaluation usually seeks to address questions involving the need, process, outcome, and efficiency (Posavac & Carey, 1992) of an innovation, treatment, or program. To answer these questions, evaluators must travel to program sites to collect data but then return to their desktop or laptop computers to begin analysis. They process information from a stationary, contextually independent position, typically geographically and temporally removed from the program they are evaluating. If evaluators could collect and analyze data in real time while on site, however, they could more rapidly assess needs, monitor the process of program delivery while participants were actually engaged in program activities, supply timely feedback for program improvement, and develop various program models for increasing efficiency and improving outcomes.

Wireless handheld devices (WHDs)—also known as handhelds—now offer evaluators the opportunity to collect, model, and analyze data on site and in real time. Until recently, handheld devices, such as personal digital assistants and cell phones, were little more than efficient personal productivity tools and peer-to-peer communication devices. Through recent technological breakthroughs, handheld computers have evolved into powerful wireless handheld devices with the following characteristics:

• Connectability – They connect to the Internet wirelessly via wireless fidelity, or WiFi.
• Wearability – They are wearable and therefore always at the fingertips of the user.
• Instant accessibility – They turn instantly on and off.
• Flexibility – They help collect data by accommodating a wide variety of peripheral extensions (e.g., digital cameras, scientific probeware, global positioning system devices, radio frequency identification, and barcode readers).
• Economic viability – They have much of the computing capability and expandable storage capacity of laptops at a fraction of the cost.

Motivated by these emerging capabilities, the Handheld Devices for Ubiquitous Learning project (HDUL) at Harvard University seeks to determine how wireless handheld devices can enhance learning and teaching (Dieterle, 2003). During the 2003–2004 academic year, HDUL integrated handhelds into various courses at the Harvard Graduate School of Education (HGSE) and the Harvard Extension School (HES). This research demonstrates that WHDs can be highly useful as (1) portable research assistants and (2) traveling conduits for online learning.

Wireless Handheld Devices as Portable Research Assistants
Here are some of the ways that HGSE and HES have made use of WHDs in their capacity as portable research assistants:

• Students in a qualitative interviewing class exploited the potential of WHDs for digitally recording interviews and capturing digital images.
• In a course on improving performance through online learning, students brought their WHDs into the field and conducted surveys of approximately 10 participants. Afterward, the data were aggregated and analyzed by the whole class.
• Students in technology-supported assessment and teaching methodology classes evaluated various educational software packages designed to capture what students know. (Examples include the Educational Testing Service's Discourse and the Center for Highly Interactive Computing in Education's PiCoMap and Sketchy).
• Students in a team-learning class completed Likert-scale² surveys and free-response questions using their handheld computers. Afterward, their results were displayed via a computer projector in real time to facilitate immediate class discussion of the findings.

Wireless Handheld Devices as Traveling Conduits for Online Learning
In a learning and media class at HGSE, students participated in the MIT Teacher Education Program's Environmental Detectives Simulation (Klopfer, 2004). Using wireless handheld devices, students physically explored the MIT campus in augmented reality (computerized information superimposed onto the real world), conducting virtual interviews and collecting simulated data of a virtual chemical spill. Through WHDs, such enactments allowed evaluators to record participants' actions and decisions while they were engaged in the simulation.

Conclusions and Next Steps
Funding for these studies will enable HDUL to continue this research through the 2004–2005 academic school year. Building from the lessons learned during our first year of implementation, next year's goals include continued implementation and integration of WHDs in the courses previously mentioned, as well as involvement of a greater number of faculty and students, especially those in evaluation and in HGSE's Teacher Education Program. In particular, we hope that graduate students in the Teacher Education Program will, as student teachers, be able to integrate wireless handheld devices into their classrooms.

References and Related Resources
Bannasch, S., & Tinker, R. (2002). Probeware takes a seat in the classroom: Educational impact of probes improves with time and innovation. Retrieved June 26, 2003, from www.concord.org/newsletter/2002winter/probeware.html

Dede, C. (2002). Vignettes about the future of learning technologies. In Technology Administration (Ed.), 2020 visions: Transforming education and training through advanced technologies (pp. 18–25). Washington, DC: U.S. Department of Education. [Available at www.technology.gov/reports/TechPolicy/2020Visions.pdf (Acrobat file)]

Dieterle, E. (2003). Handheld devices for ubiquitous learning project. Retrieved August 6, 2004, from gseacademic.harvard.edu/~hdul

Educational Testing Service. (2003). Monitor classroom education with Discourse computer software & technology. Retrieved November 3, 2003, from www.ets.org/discourse

Klopfer, E. (2004). Environmental detectives simulation. Retrieved June 15, 2004, from education.mit.edu/ED/EnvDet.mov

Posavac, E. J., & Carey, R. G. (1992). Program evaluation: Methods and case studies (4th ed.). Englewood Cliffs, NJ: Prentice Hall.

Staudt, C. (2001). Curriculum design principles for using probeware in a project-based learning setting: Learning science in context. In R. Tinker & J. Krajcik (Eds.), Portable technologies: Science learning in context. Innovations in science education and technology (pp. 87–102). New York: Kluwer Academic.

Tinker, R. (2000). A history of probeware. Retrieved June 26, 2003, from www.concord.org/research/probeware_history.pdf (Acrobat file)

The University of Michigan Center for Highly Interactive Computing in Education. (n.d.). HICE handhelds homepage. Retrieved October 29, 2003, from www.handheld.hice-dev.org

Want, R. (2004). RFID: A key to automating everything. Scientific American, 290(1), 56–65.

¹ The author would like to thank the Office of the Provost at Harvard University for funding the Handheld Devices for Ubiquitous Learning project, and Chris Dede, the grant's principal investigator and the Timothy E. Wirth Professor in Learning Technologies.
² A Likert scale is a rating scale that measures the degree of a person's agreement with a series of statements. Such scales are often composed of the following five points: 1 = strongly disagree, 2 = disagree, 3 = neutral/don't know, 4 = agree, and 5 = strongly agree.

Edward Dieterle
Handheld Devices for Ubiquitous Learning
Learning and Teaching Doctoral Candidate
Harvard Graduate School of Education
13 Appian Way
334 Longfellow Hall
Cambridge, MA 02138
Tel: 617-384-9290
Email: dietered@gse.harvard.edu