Back to SySurvey Whitepapers |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Using E-mail To Survey Internet Users In The United States: Methodology And Assessment |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Author: Kim Sheehan, Kim Bartel Sheehan
Email: ksheehan@ballmer.uoregon.edu, mhoy@utk.edu Link: http://www.uoregon.edu, http://www.utk.edu |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Reprinted with permission from the Journal of Computer-Mediated Communication, Volume 4, Issue 3 (http://www.ascusc.org/jcmc)
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Table of Contents
AbstractThe Internet's potential for academic and applied research has recently begun to be acknowledged and assessed. To date, researchers have used Web page-based surveys to study large groups of on-line users and e-mail surveys to study smaller, more homogenous on-line user groups. A relatively untapped use for the Internet is to use e-mail to survey broader Internet populations on both a national and international basis. Our experience using e-mail to study a national sample of Internet users is presented, beginning with a discussion of how a sample of on-line users can be selected using a 'people finder' search engine. We include an evaluation of the demographic characteristics of the respondent pool compared to both a web page-based survey and a telephone survey of Internet users. Considerations for researchers who are evaluating this method for their own studies are provided. IntroductionThe rate of adoption of the Internet is eclipsing that of all other technologies preceding it (Department of Commerce, 1998). In its first four years of availability to the public, fifty million people logged on to the Internet worldwide. It took 38 years for radio and 13 years for television to achieve this level of acceptance (Department of Commerce, 1998). In the past four years, the number of Internet hosts has increased six fold, and the number of domain names has increased almost thirty fold (Department of Commerce, 1998). Internet traffic doubles every 100 days; by the year 2005, one billion people worldwide may be on-line (Department of Commerce, 1998). Given this astounding growth rate, the Internet presents enormous potential for interaction between on-line users and researchers. The Internet's potential for academic and applied research has recently begun to be acknowledged and assessed. To date, researchers have used Web page-based surveys to study large groups of on-line users (e.g. Kehoe, Pitkow and Morton, 1997) and e-mail surveys to study smaller, more homogenous on-line user groups (e.g. Parker, 1992; Smith, 1997; Tse et al, 1995). However, it appears that a relatively untapped use for the Internet is to use e-mail is to survey broader Internet populations on both a national and international basis. Our goal in this paper is to examine e-mail as a way to study Internet users throughout the United States. We begin with a presentation of evidence based on previous research that discusses the strengths and limitations of web page-based surveys and assesses the viability of using e-mail as a survey data collection method. Next, we present our own experience using e-mail to study a national sample of Internet users. We include an evaluation of the demographic characteristics of the respondent pool compared to both a web-page based survey and a telephone survey of Internet users. Finally, we conclude with suggestions for future research and considerations for researchers who are evaluating this method for their own studies. On-line Research MethodsWeb-based Surveys To date, the Internet offers both web page-based surveys and e-mail for prospective researchers to use for data collection. Web page-based surveys tend to collect broad-based data from individuals all over the world who self-select to respond to surveys that are posted on web sites. These web page-based polls can collect demographic information, as well as other types of purchase, psychographic and opinion data. Numerous benefits to web-based surveys have been noted. Design Flexibility A web page-based survey can take advantage of the graphic power available through programming languages such as HTML and JavaScript to create an attractive, interesting, and compelling survey that is inviting to respondents (Schillewaert, Langerak and Duhamel, 1998). The use of CGI scripts allow adaptive questioning, which means that the questions that a respondent is asked depend on his or her answers to previous questions (Kehoe and Pitkow, 1996). This allows for follow-up questions that can enrich responses as well as easier navigation for respondents. Reach Web page-based polls have been noted for their ability to generate a high number of responses (Kehoe and Pitkow, 1995): the GVU polls at Georgia Institute of Technology generate more than 10,000 responses per poll. The sheer number of responses suggests that the results represent a diverse set of users. For example, it was estimated that one out of every 100 on-line users responded to each of the GVU polls (Kehoe and Pitkow, 1996). Time This high volume of responses can be collected very quickly (Smith, 1997; McCullough, 1998). For example, studies have shown that several hundred responses can be generated over the course of a single weekend (McCullough, 1998). This time factor alone suggests huge benefits over traditional surveying techniques in terms of being able to collect and analyze data quickly, and implement decisions based on the findings. Cost The costs of both data collection and analysis can be minimized by the use of web-based surveys (McCullough, 1998). Outside of high start-up costs for equipment and web page design, the actual implementation of a survey can be almost free, with no costs for paper or postage. Data analysis can be simplified by a direct transfer from the form to the analysis software, where limited data cleaning would be necessary (McCullough, 1998). Anonymity Web page-based surveys allow for anonymity in responses, since the respondent can chose whether to provide his or her name or not. Previous research (Kiesler and Sproull, 1986) has indicated that anonymity may affect response rates positively, as respondents may be more willing to respond without fear that their answers may be identifiable to them. Minimized interviewer error Since respondents type in their answers directly to a form on a web page, there is no need for an interviewer to have contact with the respondents (Schillewaert, Langerak and Duhamel, 1998). Therefore, survey responses will be free from errors caused by interviewers, resulting in cleaner data (McCullough, 1998). Minimized interviewer bias Similarly, the lack of an interviewer eliminates any potential for bias that the interviewer brings to the survey. An interviewer's mood, prejudices or opinions will not be reflected in the data (McCullough, 1998). However, web-based surveys do present some limitations that researchers must recognize when they are considering this method. Generalizability Issues Web page-based surveys must attract respondents to the web page with messages posted in news groups, links on other web pages, banner ads, and other types of methods. As a result, all segments of a Web population may not be represented in the sample (Kehoe and Pitkow, 1996). All Internet users do not use the same browsers, and different browsers may not present images and text on web pages in the same manner. For example, some users (such as those subscribing to freenets) use only a text-based web browser (such as Lynx), and may not be able to respond to the survey. Some web based-polls are announced in Usenet newsgroups. Therefore, if potential respondents are not a frequent visitor to newsgroups, they may not be aware of the survey announcement posted in newsgroups, and thus may not have the opportunity to complete the survey. The self-select nature of web page-based surveys also may effect their generalizability. (Pitkow and Recker, 1994; Schillewaert, Langerak and Duhamel 1998; Zikmund, 1991). Multiple and/or inappropriate responses Web page-based polls generally allow for multiple responses from a single individual, as well as responses from individuals outside of the population of interest (e.g. persons in countries where a product or service is not available, or from persons who are younger or older than the population of interest). This could also bias the results. Comparability issues One way to validate a method is to compare it to other methods that are accepted within the research community. Since it is almost impossible to develop response rates to web page-based surveys (Kehoe and Pitkow, 1996), it is difficult to compare web page-based survey methods to traditional survey data collection methods such as postal mail and telephone surveys. This leads to another generalizability issue. Without an understanding of the size of the respondent pool in comparison to the size of the universe and the sampling pool, it is also difficult to generalize research findings beyond the universe of those responding to the survey. E-mail as a Data Collection MethodUsing e-mail as a survey data collection method comparable to postal mail may ameliorate some of the issues inherent in web page-based data collection. Previous research presents several reasons to support the idea the e-mail offers much promise as a means of administering surveys as well as pitfalls to be avoided. Penetration of E-mail Today, as many as 100 million people worldwide have access to e-mail (DOC 1998). Eighty per cent of all users use the Internet daily, with many reporting that 'surfing' replaces 'TV viewing' as entertainment (Kehoe, Pitkow and Morton, 1997). The sheer number of individuals using the medium coupled with the frequency and ease with which they could be contacted suggest that e-mail is a viable survey method. Accessibility of Names A lack of a national directory of e-mail addresses could be seen as a limitation to e-mail surveys. For example, Schuldt and Totten (1994) reported a problem with obtaining names for their sample. This situation has changed in recent years. Many content providers compile their own databases and should be able to access names quickly from these sources. Some organizations (such as universities and trade associations) publish directories, both paper and on-line, with e-mail addresses. On-line search engines such as Lycos provide "People Finders' for e-mail addresses. Anonymity and Confidentiality When respondents use the 'reply' function of their e-mail programs to return their completed surveys, their names and e-mail addresses can be automatically written on the electronic message (i.e. the survey) the researcher would receive. While previous research (Kiesler and Sproull, 1986) has indicated that anonymity may have affected response rates positively, other researchers (Couper, Blair and Triplett, 1997) suggest that the lack of anonymity may not have any effect on response rates. With e-mail surveys, anonymity could be guaranteed through the use of encryption technology, and confidentiality can be guaranteed through confidentiality assurances. This study chose to guarantee confidentiality. Assuring that responses will be confidential throughout the data collection process should help to build respondent trust and enhance response rates. Identifying Duplicate Responses and Non-Response An additional benefit to using e-mail is that duplicate responses can be eliminated. Steel, Schwendig and Kilpatrick (1992) suggested that duplicate responses can become problematic since researchers using postal mail often send out multiple copies of questionnaires to their entire sample in order to increase response rates. E-mail presents a benefit over postal mail, then, since e-mail responses can be tracked and previous respondents can be eliminated from follow-up e-mail. E-mail surveys may allow the researcher to develop a profile of non-respondents. Depending on the search engine used and the respondents' server, some demographic information about persons with e-mail accounts is available on-line and some demographic information such as gender and location may be compiled. It might also be possible to attempt to contact non-respondents using an alternative method (such as postal mail or telephone) to solicit responses that could be compared to the e-mail sample for similarities. It should be noted that demographic information about persons with e-mail accounts may not be completely accurate, as individuals may have changed locations or jobs since the information was provided. However, the availability of such data allows for options that the researcher can consider when assessing non-response. Cost Benefits As with web page-based surveys, there appears to be some cost savings inherent in using new technology. Parker (1992) indicates that cost savings from e-mail compared to traditional mail and telephone surveys are based on low transmission costs and elimination or reduction of paper costs. E-mail may also present cost savings over web page-based surveys, as costs for page design and posting to a server would not be incurred. However, some savings may be offset by the on-line server used (costs vary by Internet service provider) and time considerations (transmission costs may increase by the minute, which may impact the length of the survey). Ease and Flexibility of Responding When respondents perceive technology as easy to use, they seem more likely to respond (Parker, 1992). As more people become familiar with the Internet, these individuals should become comfortable using the technology to answer surveys. An additional advantage to e-mail is that respondents can return it in one of three ways: e-mail, fax or postal mail (Parker, 1992). This flexibility may enhance the perception of ease of use. Unless the respondent purposely deletes the survey, it cannot be accidentally tossed or misplaced like a mail survey. Yet, comparable to a mail survey, the respondents still have the benefits of completing the survey at their own pace and convenience. Response Rate Benefits There is not clear evidence that new technology produces a higher response than postal mail. In a review of nine studies that have used both postal mail and e-mail (see Table 1), four studies show postal mail achieving higher response rates than e-mail, three studies indicate that e-mail response rates are higher than postal mail, and two studies did not show significant differences in response rates. Researchers indicated that the lack of familiarity with the technology may have impacted some of the response rates. It is also important to note that many of these studies are from small, homogenous populations, and thus may not represent larger population groups' response tendencies. Response Time Benefits Past studies (see Table 1) found direct marketers can collect data more quickly using e-mail than with postal mail methods. In the five studies that reported response time results, e-mail responses were collected significantly faster than postal mail responses. The variety of populations used in these studies suggest that this rapid rate of response might be seen among larger Internet populations. E-mail LimitationsCurrent research has also identified two key limitations unique to e-mail that must be considered when planning an e-mail survey. First, researchers must recognize that unsolicited surveys may be considered aggressive by respondents, and not in keeping with Internet culture (Mehta and Sivadas, 1995). Minimizing a perception of intrusiveness should help to address this problem (Schillewaert, Langerak and Duhamel, 1998). Second, the changing nature of the Internet suggests that it is possible that e-mail addresses may become out-of-date fairly quickly (Smith, 1997). Addressing this issue early on can prepare the researcher for dealing with delivery failures. Executing an E-mail SurveyIt seems apparent that e-mail is a potentially useful tool for collecting data among Internet users nationwide. To assess this possibility, we report our experience executing a national e-mail survey which queried individuals with personal e-mail accounts about their on-line behaviors and their attitudes and opinions regarding privacy. This section outlines the steps taken to implement the e-mail survey. Step One: Survey Pretest The survey was pre-tested among members of the target population. The pretest was used for the traditional assessment of validity and comprehension. Additionally, the pre-test was used to determine if the directions for completing the survey were clear as well as assess response and non-deliverable rates. The pre-test assisted in addressing one of the key limitations of e-mail: the amount of undeliverable surveys. Acknowledging the response rate allows for the estimation of an adequate sample size to achieve response goals. Step Two: Sampling The sample for this study was drawn from a population of known Internet users in the United States who access the Internet for their personal use. Since the study focused on concerns of current on-line users, it was not necessary to address the concerns of those persons who do not use on-line services. The topic of the study was attitudes and opinions about privacy and marketing activities on-line. Individuals from outside the United States were excluded from the current survey administration since different cultural, social and political climates create highly different perceptions about privacy among users. The sampling frame consisted of on-line users from randomly selected Internet Service Providers (ISPs). E-mail addresses of on-line users were generated from the Four11 Directory Service (www.four11.com). (Since this study was undertaken, the Four11 service was bought by Yahoo, and is now accessible from the Yahoo home page: www.yahoo.com; differences between how the search engine operated at the time of the study and the current date are noted where applicable). The Four11 on-line search engine is a database directory of names, addresses, phone numbers and e-mail addresses of individuals around the world, and contains over 15 million e-mail addresses. These e-mail addresses were compiled from three different sources: voluntary registrations, public sources, and what Four11 termed AutoRegistations (Four11.com, 1997). Voluntary registrations accounted for about 12% of the addresses and were generated either through Four11's own free e-mail service, Rocketmail, or from registration directly at the Four11 home page. Public sources for e-mail addresses were primarily Usenet newsgroups. In this case, it is likely that Four11 used a search bot to collect names of individuals who posted messages to Usenet newsgroups. AutoRegistation was a program that collected addresses from ISPs and other organizations. These organizations voluntarily provided their directories to Four11. Individuals listed in the service are given the option to change any information about them or to have their information excluded from the service. Percentages of addresses collected from public sources and AutoRegistration were not available from Four11. The Four11 search engine allows a search of e-mail addresses based on individuals' names, the domain names of the ISPs accessed by individuals, and individuals' home city, state/province, or country. For example, if one wished to find the e-mail address of Jane Smith, the first and last names would be entered into the search form. The search engine would then provide the e-mail addresses of all Jane Smiths in the database. If more than 200 names in the database match 'Jane Smith', the directory provides a sample of 200 matches. For this study, the search field used to generate e-mail addresses was the 'domain name' field. Any domain name can be entered in this field. For example, entering 'abc.edu' should provide up to 200 names of all individuals with accounts that 'match' the 'abc.edu' domain. Fifty-five randomly selected domain names of Internet Service Providers were generated. The domain names were randomly selected from Network USA's Internet Service Provider Catalog (www.netusa.net/ISP/index.html). This is a comprehensive on-line list of commercial personal Internet Service Providers in the United States organized by state and area code. Other on-line services providing similar information include The List (http://thelist/internet.com) and the Internet Access Provider Meta-list (http://www.heribson.com/herbison/iap-meta-list.html). All three lists are developed through ISP submissions for inclusion on the list. An individual, therefore, could look up his or her home state and telephone area code and find a list of Internet Service Providers that provide service in his or her area. Every sixth service provider was selected for inclusion on the list; no duplicate service providers were selected. Initial searches using the first five domain names indicated that an average of 115 names could be selected from each domain. Each of these domain names was entered into the Four11 search engine, along with the "USA" country designation. The results of the search provided either all the e-mail addresses in the domain name (if there were less than 200 addresses in the domain) or a sample of 200 addresses (if there are more than 200 addresses in the domain). When this study was undertaken, the search engine identified the total number of matches. Although this service is no longer provided, we used this information to determine how many addresses would be selected from each domain group for the sample. If there were less than 1000 total matches, every fourth name of the full address list of the sample 200 was selected to receive the survey. If there were 1000 or more total matches, the entire sample of 200 was selected to receive the survey. This systematic selection helped to ensure that a variety of providers were included in the sample. These names were manually selected and entered by the researcher into a mail list database. Coomber (1997) regards this type of stratification as important for on-line surveys, since sampling from a large number of different groups would result in samples representing a wide range of persons and opinions, based on the central limit theorem. Even though the search engine no longer provides the total matches found, researchers using the search engine may wish to consider some random sampling from within the e-mail group listed. Additionally, researchers should be aware that Four11.com will not provide a sample if the number of matches exceeds 10,000. In this case, researchers may wish to include a randomly selected initial in either the first name or last name fields in order to limit the total number of matches available. As a final note, a check of other 'people finder' search engines available (through web search engines such as Lycos [http://www.lycos.com] and Hotbot [http://www.hotbot.com]) found that no other people finder search engine allows searching solely on domain name. These other engines require the inclusion of a surname in the search. Researchers considering the use of these other search engines should recognize this limitation. Two pretests of the procedure involving a total of 350 selected e-mail addresses revealed a 25% undeliverable e-mail rate, a 14% percent decline rate and a completed survey response rate of 23%. The traditional formula for calculating the necessary sample size to provide a representative sample of a population (Zikmund, 1991) was used and resulted in a target sample size of 864 with a 95% confidence level. Therefore, assuming a 23% response rate, 3756 viable e-mail addresses would need to be solicited. To compensate for estimated undeliverable e-mail solicitations and replacement of individuals who declined to participate, 5000 e-mail addresses were selected . Actual administration of the solicitation and survey showed an undeliverable rate of 26%, a decline rate of 12% and a completed survey rate of 24%. Step Three: Solicitation One week before the survey was sent out, each prospective sample member received a solicitation to participate via e-mail (see Figure 1). With postal mail surveys, notification of surveys with a request to participate has been shown to increase response (Kanuk and Berenson, 1975). With electronic mail surveys, notification is imperative: unsolicited e-mail surveys are "clearly unacceptable" (Mehta and Sivadas, 1995). The solicitation e-mail provided potential respondents with the chance to opt-out, that is, not receive the survey. Potential respondents who opted out were replaced by other names from the sample pool.
Step Four: Administration A week after the notification message, the actual survey was e-mailed to the sample (see Figure 2). Instructions on completing the survey gave the respondent the option of returning the survey either via e-mail or postal mail. Because e-mail messages contained the name of the respondent included on the message, respondents were guaranteed confidentiality but not anonymity. Although there is no significant evidence that the lack of anonymity causes a decrease in response rates (Kanuk and Berenson, 1975), the nature of the survey (information privacy) obliged that the option of returning the survey through postal mail should be made available to respondents. The questionnaire length was approximately nine computer screens. Despite the fact that common sense suggests that shorter questionnaires should obtain higher response rates than longer questionnaires, research evidence does not support this view (Kanuk and Berenson, 1975). The instructions indicated that the survey should take approximately fifteen minutes to complete, and a requested response of 'within the next few days' was given. A hard copy of each survey was printed upon receipt. The electronic copy was saved to a floppy disk, and was deleted six months after the study was completed. The receipt of each respondent's survey was logged, and then the name and other identifying characteristics (e.g. domain name) of the respondent were stripped from the survey. Step Five: Reminder Those in the sample who did not respond within a week received a reminder message via e-mail, which included an additional copy of the survey. These follow-ups have been widely used in postal mail surveys, and tend to increase response rates from eight to forty-eight percent for postal mail (Kanuk and Berenson, 1975). More recent research into electronic mail (Sheehan and Hoy, 1997) indicates that a reminder message increased response by 25%. Assessing the TechnologyOur study indicated that e-mail can be a viable method for administering national surveys. Our assessment of the success of our survey is examined in this section. Response Rates Overall response rate to the survey was 24%. Five thousand invitations to participate in the survey were sent via e-mail from the period from November 22, 1997 to January 10, 1998 (surveys were not mailed during the weeks containing the Christmas and New Year's holidays). Of these 5,000 invitations, 1,276 (25.5%) were undeliverable. Additionally, 433 individuals who received the invitation declined to participate. These individuals were replaced in the sample pool from the 500 'reserve' names selected prior to the start of data collection. In total, 3,724 surveys were sent via e-mail during the period from December 1, 1997 to January 17, 1998. A reminder message with an additional copy of the survey was sent to those recipients who had not replied one week after the initial mailing of the survey. In total, 895 responses were received, resulting in a 24% response rate. Of these 895 responses, six were unusable and eliminated from the study, resulting in a final sample size of 889. This small number of incomplete surveys did not affect the overall response rate of 24%. While the response rate did not achieve the levels of previous studies, the rate still seems comparable to postal mail surveys. Additionally, other studies utilized much smaller and more homogenous samples. Sampling a larger, more diverse population may have caused this lower response rate. Response Time The average response time for the survey was 3.65 days. This response time is remarkably faster than the previous e-mail survey studies have shown (see Table 1), and speaks to the power of e-mail for gathering data very quickly. More than half of the responses had been returned by the end of the second day that the survey was in the field, with two-thirds of the responses returned by day four of the collection. A reminder to complete the e-mail survey, sent out at the end of the fifth day of data collection, resulted in an additional 23% of the total surveys returned.
This rapid rate of return may be a reflection of the impact that e-mail has had on the lives of on-line users. Response data has shown that a majority of the respondents check their e-mail several times a day. Interacting with e-mail seems to be a habitual part of life, and in many cases, it appears that responding to the survey would not have taken any additional effort. Additionally, several respondents indicated that this was the first survey they had received via e-mail. The novelty factor may have caused them to open and respond to the survey mail message quickly. Cost Simple calculations indicate that e-mail is an extremely cost-efficient method to gather data. The cost to administer the current study's survey via postal paper mail is estimated at $6,500 (printing and postage of solicitation mailing, survey, and reminder mailing). Given that we had free access to an Internet system, the only cost incurred was the cost of computer paper to print out responses (approximately $30). Approximately 22 hours of computer time were required to download surveys for output printing. At a cost of $2 per hour (average for the area), an additional $440 would be added to the cost, increasing the total cost to $470, a fraction of the cost of postal mail surveys. Ease of Use and Flexibility Results of the survey indicated that the respondents were fairly familiar with on-line communication and the Internet. On average, respondents have been using computers for almost twelve years. Respondents reported accessing the World Wide Web for personal reasons during eleven of the past fourteen days. Two-thirds of respondents (66%) checked their e-mail several times a day, with 20% checking their e-mail once per day. Given this, it is not surprising that there were relatively few problems with individuals' being physically capable of completing the survey. However, instances of individuals' insecurity with using e-mail were evident. As noted by Parker (1992), the 'human factor' caused some respondents to feel uncomfortable and even incapable of completing e-mail surveys. Compounding this is the availability of different programs which read e-mail; different programs allow for different degrees of interactivity with the provided text. During our study, twelve respondents indicated that they did not understand how to manipulate their mail program to allow interactivity. Basically, these respondents could not indicate their response within the provided text while on-line, and thus chose to send the survey back through postal mail. We responded with whatever help we could to assist these users, and also offered reimbursement for postal costs incurred by those who returned the survey via postal mail. Anonymity and Confidentiality In our instructions, we promised confidentiality, and separated the respondent's name from the survey as soon as it was noted as arriving. We also stated that responses would be reported only in the aggregate. We received several comments from respondents regarding the privacy issue, most inquiring exactly how we could guarantee confidentiality. When we informed them of our method, all individuals who questioned the method before receiving the survey eventually responded to the survey. Only one respondent returned the survey through an anonymous re-mailer. Respondent Profile Respondents tended to be fairly young, male, and from households with fairly high household incomes. The respondents to this survey are comparable to Internet user profiles found in studies conducted through web pages (e.g. Kehoe, Pitkow and Morton, 1997) and through traditional telephone surveying (e.g. MediaMark Research [MRI], 1997), although some differences were seen. Specifically, the respondent profile of the current study reflected higher levels of male users and well-educated respondents than either the web page-based or telephone studies. Of the total respondents to the current study, 70.4% were men and 29.6% were women. The percentage of male respondents is higher than other studies: Kehoe, Pitkow and Morton, 1997 reported a male/female ratio of 61/39 and MRI (1997) had a ratio of 54/46. The respondents in the current study were well-educated, with more than sixty percent of respondents to the current survey earning a bachelor's degree or higher. More than twenty percent reported earning a master's degree or higher. Kehoe, Pitkow and Morton (1997) reported that 47% of their respondents earned at least a bachelor's degree and half of the respondents to MRI's (1997) study reported educational levels at the bachelor's level or higher. The higher education level in the current study is not surprising since the web page-based study allowed responses from individuals of all ages, and the MRI study allowed responses from anyone over 16. The current study only included responses from individuals over the age of eighteen. The gender differences are not as easily explained. These differences could reflect the inherent difficulty of measuring and categorizing the constantly changing on-line environment. Two areas of demographic similarity were apparent: age and household income levels were similar. Somewhat less than half of the respondents (42.2%) to the current study were between the ages of 18 and 34, with an additional 27.3% of respondents between the ages of 35 and 44. Twenty percent of respondents were between 45 and 54 years old, with the remaining 10% of respondents above the age of 55. This age distribution corresponds to Kehoe, Pitkow and Morton's (1997) study where the average age was 35.2; MRI (1997) reports age breakouts that almost mirror the current study. Approximately one-fourth of respondents (27%) to the current study reported household incomes less than $40,000. An additional 29% reported an annual household income between $40,000 and $59,999. Twenty-one percent reported an annual income between $60,000 and $79,999. Thirteen percent reported an annual household income between $80,000 and $99,999, and ten percent reported household incomes over $100,000. Kehoe, Pitkow and Morton (1997) indicated that the mean household income for respondents was $53,000. Responses to MRI (1997) indicate a similar pattern to the current study. The typical respondent household in the current study consisted of two adults and one child under the age of 18. Respondents came from thirty-nine different states in the United States. The two geographic areas of the United States with minimal representation were the upper Midwest and parts of New England. Investigation of other demographic studies of Web users has not provided data on state of residence. This information, however, supports the assertion that e-mail can be used to generate a national respondent pool. The ability to triangulate demographic information with other methods provides a richer picture of the demographic profile of respondents and suggests a critical examination of any type of survey that portends to provide a complete demographic picture of the Internet and its users. Considerations for Future ResearchBased on these results, it appears that e-mail can successfully be used to survey Internet users nationwide. An e-mail survey addresses some of the limitations of a web page-based survey in that the response rates and speeds can be compared to traditional methods to assess the appropriateness of the method. It is possible to generate a national sample of email users which is not biased to Web users. Additionally, using e-mail addresses sampling problems in terms of multiple responses. However, certain limitations apply to this method. While some of these issues are also applicable to a web page-based survey, some are unique to the use of e-mail. It would not be possible to generalize results to mass markets including both Internet users and non-Internet users based on knowledge attained solely from on-line respondents. This has also been shown as a limitation to web based polls (Coomber, 1997; Kehoe and Pitkow, 1996). However, depending on the research question, it is possible that sample information can be used to generalize to the on-line population. One of the most challenging limitations is the changing nature of the Internet. The composition of the Internet changes daily with new individuals logging on and others adding or switching Internet service providers. Thus, some directories may contain information that is out of date or incomplete. This study only included those ISPs that provided service to individual customers at their homes. Researchers wishing to add Internet users who use e-mail at work or at school would need to find lists of these servers, and then assess how up-to-date that information is also. The changing nature of the Internet is also seen in changes to how search engines operate. Any ownership changes of a search engine or other web content provider may result in unanticipated changes to this methodology. Additionally, the technology allows individuals to set up mail filters, which delete messages from those senders not on the receiver's 'approved' list. This deletion may or may not be reported to the sender. As use of mail filters grows, response rates may be affected. Researchers should anticipate these changes by testing search engines prior to address generation to make sure that the method is still appropriate and pre-testing the study with a random sample of names to determine and plan for non-deliverable mail, as discussed in the results section.
While response rates now appear promising, respondent distrust of data collection may influence response rates in the future. Several individuals receiving the solicitation e-mail censured the researchers for sending out unsolicited e-mails, and accused the researchers of 'spamming'. It should be noted that definitions of spamming differ by Internet Service Provider and also in the mind of the respondent. With our ISP, spamming was defined as unsolicited e-mail that was of a commercial nature, or of a political nature, or that in some way caused distress to the receiver. We believe our survey was not intended to do any of those three, but some recipients disagreed with us. Many of the individuals who complained about spam believed that we were 'sugging' or 'frugging', that is, selling or fund raising under the guise of research. Some individuals questioned the academic association of the researchers, even though the purpose of the study was clearly explained in the introduction to the survey (e.g. that the survey was for doctoral research). One respondent wrote, "if you are a student than I am the Emperor of Japan". The novelty of using e-mail to collect data may be partly to blame. Until this method becomes more ingrained with academics and popularized among on-line users, respondent concern and distrust is likely to continue. Additionally, individual ISPs have policies and procedures that may limit the success of e-mail surveys. We encountered one ISP that monitored the number of e-mails delivered to its users that originated from a single address. If the number was very large, the ISP assumed that the sender was 'spamming', and the system operator blocked the orginator from sending additional messages to the ISP's subscribers. The system operator then contacted his counterpart at our university. Although our university was supportive of our research endeavors, this same support is not guaranteed with all systems. In the future, a special tag may be able to be assigned to research e-mail to distinguish it from other types of mail. Therefore, such problems may soon be avoided. How government regulation will effect the promise of e-mail remains to be seen. Federal courts have barred specific companies from sending unsolicited e-mail advertisements to subscribers of CompuServe (Kanaley, 1997). The courts are ruling that ISPs have the right to restrict access by 'spammers', mostly for economic reasons. Users who pay hourly access rates complain about spending too much time and money reading messages they have no interest in. How this will effect mailing in the future is not yet clear. Options being discussed include charging mailers a fee for each piece of mail sent. Some believe this will cause companies to be more selective in the addresses to which they send mail. Obviously, this would increase the costs of e-mail surveying. Even with these issues, however, it is apparent that e-mail can be used to gather information about national on-line populations. Information can be gathered extremely quickly, and at a relatively low cost. Future research should continue to investigate the method and approach it as postal mail has been approached. Researchers could examine, for example, the impact of color in the survey, image quality, and incentives on response rate, response speed, and quality of response. Researchers could investigate whether topic content influences response rate and speed. The current study, as with many studies on-line, dealt specifically with Internet and communications issues. Research could compare how surveys examining non-Internet related questions perform relative to surveys with Internet-related topics. Additionally, attitudes and opinions toward Internet surveys could be examined to assess how respondents view their participation in on-line surveys (e.g. as entertainment, as part of a marketing exchange, or for some other reason). Research should examine the data gathered from web page-based surveys and national email surveys to determine if the two methods can be used together for optimum data collection. Finally, the appropriateness of the e-mail method in countries outside the United States should be assessed to determine if the method is appropriate for world-wide data collection. While e-mail surveying will probably never replace the broad-based data available via postal mail surveys, it will probably provide adequate data for the study of on-line populations, and given the propensity of 'hard to reach' individuals to respond, may provide richer data about on-line behavior than postal mail surveys. As on-line usage continues to grow, and as more and more consumers have access to e-mail, it is conceivable that this method may be eventually used in place of postal mail to gather information about broad-based consumer segments.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Back to SySurvey Whitepapers |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||