Volume X, Number 3, Fall 2004

Issue Topic: Harnessing Technology for Evaluation

Ask the Expert

Internet Surveys: Back to the Future

Don Dillman is recognized internationally as a major contributor to the development of modern mail, telephone, and Internet survey methods. He is Regents Professor in the Department of Sociology and the Department of Community and Rural Sociology at Washington State University. He also serves as deputy director for research and development in the university's Social and Economic Sciences Research Center.

Why should we use Internet surveys?

The major advantages of Internet surveys are speed and lower marginal costs. Speed is an advantage because some people respond quickly—hundreds of responses may come back within a few hours. Speed is also an advantage for data processing. We can design Internet surveys so responses go directly into a database, thereby eliminating data entry. We can also set up analysis templates during the survey design phase so we can almost immediately analyze the data and review the results.

Marginal costs are important because, after evaluators have invested in the people, hardware, software, and programming, Internet surveys have the potential to return large numbers of responses at relatively little cost beyond setup. This is in contrast to telephone surveys, for example, where each interview has a significant cost, and those costs accumulate as more responses are gathered.

What are the limitations of Internet surveys?

There are serious challenges to doing quality Internet surveys—both human and technical. Good survey fundamentals have to apply, regardless of how the survey is administered. Internet surveys are subject to the same sources of error as all others, including measurement, coverage, sampling, and nonresponse errors.¹ Measurement error can occur from inattention to how a survey will look on different computer screens or from survey design characteristics that may lead a respondent to either abandon the survey altogether or send incomplete information. In terms of coverage, if people do not have access to the Internet, they can't be surveyed. For many populations, lack of Internet access is still a huge problem. For sampling, we do not yet have good lists of people with Internet addresses and can't draw random samples as a general rule. This is a difficult issue to overcome and will keep the Internet, at least for now, from replacing most other types of surveying.

Regarding nonresponse error, I am disturbed to see groups sending Internet surveys to tens of thousands of potential respondents, getting a small percentage back (say 1,000), and then suggesting the results are representative of the larger population surveyed. Suppose 1,000 responses represent only 5% of possible respondents. We have no idea what the other 95% thought, or the characteristics of those nonrespondents.

How can we get acceptable response rates with Internet surveys?

An acceptable response rate is over 50%, with careful work done to make sure there is no significant nonresponse error. To get this kind of response, we need to draw careful samples and spend resources to get as many people as possible in that sample to respond. Additionally, Internet surveys require repeated contacts, usually by several modes (email, mail, and/or phone). Mixed-mode surveys tend to get higher response rates. For example, a print survey sent by regular mail with a cash incentive enclosed would then be followed up with an email; this email would include a web address where respondents could go to complete the survey online if desired.

Are there special considerations for designing Internet surveys?

Evidence shows that layout is important. We can't simply pick up a paper-and-pencil survey, type the questions into an online format, and assume the instrument is the same. In addition to wording and question order, visual survey design and layout can affect survey responses.² Graphics are significant because what one person sees on his or her computer screen may not be what another sees. Differences in browsers, screen configurations, and Internet connection speeds may affect how respondents see the survey. I suggest that most programmers work at a level less advanced than they would like in order to allow more people to view a given survey in a similar way. I encourage special efforts to be made to get the same images on as many respondents' screens as possible. We also need to test surveys using both fast and slow electronic connections.

Another consideration is the tendency for Internet surveys to ask one question “per page.” That approach may lead to respondents losing context as they answer questions. If a person gets interrupted while responding to a line of related questions (e.g., the same series of questions being asked for each of several jobs held) for example, that person may forget the question's context when he or she returns to it later. We can solve this problem by paying careful attention to the way our items are worded and by changing them from the way they might have been presented in a mail survey to help them stand alone.

What are some fundamentals of high quality Internet surveys?

For starters, keep surveys short. How short depends on the population. I've been concerned for many years that survey designers try to get too much detail from respondents. The result is survey abandonment, which the Internet makes relatively easy. We can't expect people to sit for 30 or 40 minutes, as we sometimes try to do on mail and telephone surveys. Also, we need to test and retest our surveys and administrative procedures. Look at each question to determine whether respondents are abandoning the survey because of a particular item. Examine the data to determine what went wrong when someone tried to respond, rather than just ignoring those issues as casualties of the process.

Finally, we must design our surveys well, regardless of whether we use telephone, face-to-face, mail, or Internet. Carrying out successful surveys takes some understanding of survey methodology, sensitivity to the group or population being surveyed, and good questions that make sense and can be answered.

What developments can we look forward to in this field?

We are relatively new at Internet surveying and do not know exactly what changes are ahead. To draw a parallel, when I started working on telephone surveys in 1969, I did not predict the impact of answering machines, cell phones, the near elimination of long-distance charges, and the personal computer. The technology changed and so did our culture. For example, when telephone surveying started it was difficult for people to hang up on someone because cultural norms did not permit it. Over the years answering the telephone and staying on the line became more voluntary. Cultural norms regarding Internet usage are still developing.

This is an exciting but developmental period in Internet surveying. Given the challenges I outlined, right now I recommend use of the Internet in combination with other survey modes. As computers become much more integral to our lives, this may change. We will only know in time.

¹ Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method. New York: Wiley and Sons.
² See also www.sesrc.wsu.edu/dillman.

Julia Coffman, Consultant, HFRP