Back to SySurvey Whitepapers

Introduction to Survey Methodology and Design

Introduction

Research projects like the IQP that examine the interface between science and technology and society often require the collection and analysis of social data. The most common response from beginning researchers to this need for data is to conduct a survey. After all, everyone has had experience answering surveys, and it is usually a simple and straightforward procedure. Many people conclude from this experience that writing and administering their own survey will be a simple and straightforward matter as well.

However, in actuality, the intuitions people form about survey research from their own experience are often incorrect. To test your own intuitions, you might consider how you would answer the following true/false questions (answers appear at the end of the chapter just ahead of the References):

T/F 1. Determining the opinions of the population of a city of 10,000,000 people requires a much larger sample than an opinion survey of a city of 100,000 people.

T/F 2. Randomly choosing names from a telephone directory is the best way to choose a sample for a telephone survey.

T/F 3. Survey questions should appear in random order.

T/F 4. Posting a survey on a web site is a good way to reach large numbers of people and to increase sample size.

T/F 5. If too few people from the first survey sample chosen fail to respond, a second sample should be chosen to increase the number of respondents.

An understanding of the answers to these and many other questions is essential for conducting scientific surveys that yield accurate, unbiased, and generalizable results. Yet it is a rare person indeed who can explain the reasoning behind such questions correctly without having made an effort to study survey design and methodology. In fact, there really is little reason to expect success in survey research without formal study of the topic. Basic social science research methods are often more complicated, more difficult to learn, and more counterintuitive compared with basic methods in other sciences since the subjects of study, human beings, are more complicated. Atoms and chemicals, for example, don't try to figure out the goals of your research, don't have a bad day, and don't change their minds from one moment to the next!

Social scientists, by conducting countless studies and experiments over the past several decades, now have a good understanding of how to conduct a survey. From such obviously important questions as how to select a random sample to seemingly trivial details such as whether it is better to include a preprinted business reply envelope or a stamped envelope for people to use to return mail surveys, the answers are available in the published academic literature and in textbooks. There are even excellent books on the subject written especially for beginning researchers [see, e.g., Rosnow and Rosenthal (1996) and Salant and Dillman (1994)].

Thus it is now possible for students who have never conducted a survey before to learn about and implement the basic principles of scientific survey design and methodology as part of their IQP. The goal of the present work is to introduce you to these basic principles and to describe where you can go to learn more.

Alternative Social Science Methodologies

Surveys can be a powerful and useful tool for collecting data on human characteristics, attitudes, thoughts, and behavior. And, sometimes, conducting a survey is the only available option for acquiring the data necessary to answer an important research question. However, doing a survey is not the best approach for every project. Before committing to do a survey, for example, you should first consider whether or not your project team has or can obtain the appropriate background. Achieving a background in psychology, sociology, or another social science through WPI's Social Science Requirement is important for understanding what researchers have already learned about the behavior of human individuals, groups, and societies and how this knowledge was acquired. In addition, a familiarity with the basic principles and methods of statistical analysis is essential for analyzing survey data.

Conducting a thorough literature review of prior research on your topic is also an important prerequisite for conducting a survey. There are literally thousands of survey professionals and social scientists conducting and publicizing the results of surveys and studies on every conceivable topic. It is possible that your research questions have already been asked and answered by other researchers. Although students are often disappointed when this happens, it also opens up new opportunities: by learning from the experience of other researchers, your own project will improve in quality; and, after reading and analyzing prior research in your area of interest, you will be able to choose a more specific topic that is unique and important.

Survey studies also have several inherent limitations, including the following:.

1. A single survey can establish whether or not a relationship exists between two variables but is not sufficient to determine the direction of causality. For example, a survey on the topic of aggression might find a significant correlation between the number of times children argue or fight and how much time they spend playing video games. However, this would not constitute sufficient evidence to demonstrate that playing video games causes aggression, because the following explanation is also consistent with the data: children who are more aggressive to begin with are more likely to spend time playing video games.

2. Survey studies rely on "self-report" data, that is, they depend on participants to truthfully and accurately report on their attitudes and characteristics. This does not always happen. For example, some respondents may deliberately answer questions incorrectly or flippantly. However, if the survey is conducted in a professional manner, this occurs less often than you might think. A much greater concern is that subjects may simply commit "honest" errors of omission, confusion, or false memory.

3. Survey studies are subject to well-known types of bias. For example, since respondents know they are being studied, and have at least some idea why, they may change their answers, either consciously or unconsciously, to show themselves in a better light or to conform to the expectations of those who are studying them. It is also possible for experimenters to deliberately or inadvertently write survey questions that bias people to respond the way they want them to.

4. If conducted properly, surveys can accurately represent the opinions and judgments of a population of people. However, this doesn't mean that these opinions are correct. Although survey data can be used to inform decision making and public policy, they cannot substitute for expert judgment and analysis.

5. Finally, conducting a scientific survey is not a trivial undertaking. Scientific surveys require careful research and planning, are labor intensive, and can take weeks to implement and analyze. If your project team has less than a full 7-week term to devote to the survey portion of your IQP, you would probably be well-advised to try to answer your research questions using another method.

These limitations of survey studies do not mean you should not conduct a survey: all social science methodologies have their own unique set of limitations. However, before deciding on conducting a survey it is important to investigate available alternative methods and weigh their pros and cons in relation to the goals of your project. The following are some of the major alternatives to conducting survey research:

1. Naturalistic observation involves the systematic watching and recording of naturally occurring behavior. Since the subjects do not even know they are being studied, the researcher can be confident that the behaviors are natural but does not have much control over what happens. See Martin and Bateson (1986).

2. Content analysis is a technique for drawing inferences from existing records or documents (ranging from the Congressional Record to personal ads in the newspaper) in a systematic and unbiased way. Its advantages include an ability to study large populations and document naturally occurring trends over time; however, it is subject to biases of interpretation and the researcher cannot control the collection of data. See Weber (1985).

3. Formal experiments on human subjects follow the scientific method, randomly assigning subjects to alternate experimental conditions that are identical except for a single hypothesized causal variable. They allow the direction of causal relationships to be identified, but often achieve this by sacrificing "external validity," that is, applicability to real-life situations. See Chapter 4 of Judd et al. (1991) and Chapters 7 and 8 of Rosnow and Rosenthal (1996).

4. A case study is an in-depth analysis of one particular organization, such as a university, a business, or a community. Although a case study allows for the thorough examination of a particular situation, the results of such a study cannot be generalized beyond the single case. See Yin (1989).

5. Secondary data analysis is the reanalysis of existing survey data that were collected by someone else for a different purpose. Its major advantage is the enormous savings in time and effort gained by avoiding the collection of new data; its primary disadvantage is the lack of control over what information was collected and how it was collected. See Chapter 13 of Frankfort-Nachmias and Nachmias (1996).

6. In participant observation researchers become involved in the daily lives of their subjects and record detailed field notes of their observations and experiences. This allows the researcher to engage in an open-ended exploration that allows great flexibility; however, the observations are difficult to generalize to other situations and people and the researcher's presence can alter people's behavior. See Emerson (1983).

7. Personal interviews or focus groups involve the face-to-face questioning of people selected for their particular knowledge, interests, or availability rather than at random. Although they allow for a more exploratory approach, the results cannot be generalized beyond the individuals or groups. See Survey Research Center (1983) and Morgan (1988).

For a review of alternative social science methodologies, see Frankfort-Nachmias and Nachmias (1996) and McKenna (1995).

Sampling

The goal of virtually all surveys is to enable the researcher to predict accurately the characteristics or thoughts of a predefined group of people. Toward this end, it sometimes makes sense to attempt to survey the entire population of interest (for example, when this population is small, such as a company with fewer than 100 employees, or when it is important for reasons of fairness to allow every individual the opportunity to respond, as is the case with student course evaluations). However, in the great majority of cases, surveying the entire population is impractical and unnecessary. If chosen wisely, a relatively small sample or subset of a population can yield highly accurate predictions, so limited resources are best spent not by trying to survey everyone but by pursuing other goals such as obtaining a high response rate (see Section IV). Professional polling organizations, for example, are now consistently able to predict how more than 100 million people will vote in national elections with a margin of error of just a few percentage points by surveying fewer than 2000 individuals! They are able to do this by employing complicated sampling techniques that ensure that their survey participants are highly representative of the U. S. voting population as a whole. For most survey projects, however, the basic technique known as simple random sampling is sufficient. The goal of simple random sampling is to, insofar as possible, ensure that every member of a chosen population has an equal chance of being included in the sample. In order to choose a sample, a list of people from which a sample can be drawn (called a sampling frame) must be found or constructed. All such lists must be carefully evaluated to ensure that they are correct and complete. Some sources of lists are notoriously problematic: telephone directories, for example, do not contain people who do not have phones or people who have unlisted numbers, and do contain multiple listings of the same people and people who have relocated since the directory was published. And, even apparently excellent lists can contain errors or omissions. For example, a set of class lists obtained from the Registrar's Office can be a good source of names for a survey of WPI students, but they are not flawless: such lists, for example, contain undergraduates from other schools and high school students who are taking WPI classes and don't contain the names of WPI students who are away at Project Centers. Once a satisfactory list is obtained, the sample should be chosen randomly. This can be accomplished in a variety of ways, for example, by consulting a random number table in a statistics textbook, by using a computer program that generates random numbers, by selecting every nth person on the list after randomly choosing a starting point, or even by drawing thoroughly mixed names out of a hat. No matter how they are drawn, all samples can be expected to misrepresent the population to some degree. Although such "sampling error" cannot be avoided, it can be reduced by obtaining a sample of sufficient size. Beginning researchers often seek a simple rule of thumb for determining sample size. However, no such rule exists: the question of how large a sample to draw depends on how a researcher answers the following two questions: (a) How much sampling error is acceptable? (b) How much variation is there in the population on answers to the most important survey question? The answer to question (a) partly depends on the available resources: every increase in sample size will increase accuracy, but will also increase the amount of time and money necessary to complete the project. This trade-off between accuracy and cost is unavoidable. The answer also depends on the consequences associated with making an error. For some projects it is sufficient to estimate a population characteristic within 10 or 15%; in other situations, when the costs associated with making an error are higher, estimates within 5% or less may be desired. The answer to question (b) can only be estimated (if the population variance were known precisely there would be no need to conduct a survey!). This estimate might come from a literature review of similar studies or from the results of a survey pretest (see Section VIII). Once these questions are resolved, the necessary sample size n can be calculated from the following formula: n = (SD)2/(SE) 2, where SD is the estimated standard deviation (the square root of the mean squared error) of the variable in the population and SE is the size of the acceptable standard error (the standard deviation of the set of all possible sample means). Thus, for example, suppose that the most important variable for a given survey project is SAT score, and that a literature review uncovers that the standard deviation of SAT scores nationally is 150. Suppose further that the researchers would like to be 95% confident that they will be able to estimate the average SAT score of their target population within plus or minus 30 points. The phrase "95% confident" means that the sample mean will fall within a range of 2 standard errors 95% of the time, implying a standard error of 15. The required sample size would then be n = (150) 2/(15) 2 = 22,500/225 = 100 It should be noted that the results of this equation yield the number of completed, usable surveys that must be obtained. This number will typically be only a fraction of the surveys that are administered. To calculate the number of surveys that must be administered, divide n by the expected response rate of the survey (see Section IV). It should also be noted that in most survey projects, it is desirable to obtain reliable estimates for important subgroups of the target population. In such cases, the above formula should be used to calculate the required size of the subgroups, which can then be added together to obtain the overall required sample size. For example, in the SAT example, if the same level of reliability were required for subgroups of freshmen, sophomores, juniors, and seniors, the overall required sample size would be 400 instead of 100. In certain cases a simple random sample is insufficient. For example, imagine a project designed to compare the SAT scores of men and women at WPI. A simple random sample of 200 students would yield about 160 men and 40 women -- too few women to draw conclusions about with a high degree of confidence. In cases such as this when an important subgroup of a population is comparatively rare, it must be oversampled, that is, selected at a much higher rate than it occurs in the population. Thus, if comparing men and women is an important goal, it makes more sense to draw a sample containing 100 women and 100 men. Overall population estimates could still be obtained by weighting the results obtained from each group according to its true prevalence in the population; in the present example, the mean SAT score for men would be multiplied by .8 and that for women by .2 before adding them together. For further information on sampling see Chapter 5 of Salant and Dillman (1994); Henry (1990); and Chapter 8 of Frankfort-Nachmias and Nachmias (1996).

Response Rates

Selecting an unbiased sample of sufficient size is an important component of any scientific survey, but is not enough to ensure that the people who answer are representative of the larger population from which they were drawn. It is also important to obtain a high response rate. The response rate of a survey is simply the number of completed, usable surveys obtained divided by the number of people who were asked to complete a survey. If this fraction is too low, there is a strong possibility of "nonresponse error," that is, that estimates are biased because those who didn't respond to the survey have different characteristics or opinions than those who did respond. To illustrate the potential biasing effect of a low response rate, suppose that 500 surveys are sent to randomly selected WPI students asking which of two alternate food service plans they prefer. Suppose further that 150 students (30%) respond, and that 98 of them choose Plan A and 52 choose Plan B. Thus a clear majority of 65% are in favor of Plan A. However, this simple analysis assumes that nonrespondents have the same opinions as respondents. What if, for example, the 350 nonrespondents in fact have just a slight preference, say, 55% vs. 45%, for Plan B? The true percentage preferring Plan A in this case would in fact be just 0.3 x 65% + 0.7 x 45% = 51%, which is not significantly different from 50%. What if, instead, the 350 nonrespondents had a strong preference, say 70% vs. 30%, for Plan B? In this case the true percentage preferring Plan A would be only 0.3 x 65% + 0.7 X 30% = 40.5. Under this scenario, the results obtained from the 150 respondents completely misrepresent the preferences of the student body! Of course, a low response rate to a survey does not guarantee that results will be biased. In the hypothetical food service survey, for example, it is entirely possible that nonrespondents had the same preferences (65% in favor of Plan A) as those who did respond. The problem is there is simply no way of knowing for sure what nonrespondents are like or how they are thinking. The only sure way of reducing this uncertainty is to obtain a high response rate in the first place. For example, suppose that the food service survey which reported that 65% of students preferred Plan A had achieved a response rate of 70% rather than 30%. Under this scenario, even if nonrespondents strongly disliked Plan A (say, 70% preferred Plan B instead), Plan A, as suggested by the survey, would still be the preference of the student body: 0.7 x 65% + 0.3 x 30% = 54.5%. Obtaining a high response rate to a survey requires a substantial investment of time and effort. A single mailing or phone call to each potential respondent is completely inadequate and is likely to result in a response rate of 20% or lower. This happens not because most people aren't interested or don't want to help, but simply because they are busy and have many competing demands on their time. Thus, some mail surveys never get opened because they are confused with junk mail and get thrown out; those that are opened often end up under a large stack of correspondence that people never find the time to get around to. And, telephone interviewers, if they don't get a busy signal or an answering machine, are highly likely to get a person who refuses to talk because they have company, or are eating dinner, or are on their way out the door, or have to put the kids to bed. There are two things a researcher can do to increase the chances that people will take the time to respond: (1) Design a questionnaire (and accompanying material) or telephone introduction that makes it immediately clear to people that responding to the survey is both important and easily accomplished (see Section VII). (2) Design an implementation plan that includes multiple mailings or phone calls, if necessary, directed at each potential respondent (see Section IX). If these established techniques are employed, response rates of 60-70% or higher can be achieved. However, in some cases, in spite of a researcher's best efforts, the response rate turns out to be less than 60%. In such cases the researcher should investigate and attempt to identify, insofar as possible, how similar or different nonrespondents are from respondents on relevant variables. Sometimes this can be accomplished with available information, for example, in most cases the gender and geographical location of nonrespondents is known, and occasionally sample lists contain other variables that can be compared. It is also possible to randomly select and telephone a small sample of nonrespondents and launch an all-out effort to get them to answer at least a couple of brief questions that will allow them to be compared with respondents. Even people who refused to participate initially will often agree to an interview that takes just a minute or two. For further information on response rates see Chapter 2 of Salant and Dillman (1994) and Chapter 2 of Groves (1989).

Choosing the Right Survey Method

There are several different ways of administering a survey. The most common methods are sending written surveys through the mail, asking survey questions over the telephone, and conducting face-to-face interviews. Each of these methods can be effective and can yield a high response rate in certain situations. However, they each have a unique set of strengths and weaknesses, and can be ineffective if applied under the wrong circumstances. The choice of method for a particular project should therefore be made only after careful consideration of the following factors: 1. Available resources. Mail surveys require money to make copies and to buy stamps and envelopes but their implementation does not require much labor. Telephone surveys require a substantial time commitment from several people to conduct interviews but fewer monetary resources are needed (unless long distance charges are accrued). Face-to-face surveys require even more labor due to the travel time involved but again require little money unless large distances must be covered. Thus, the choice of which method to use often comes down to the relative availability of money and labor. When labor costs are high (e.g., when interviewers must be paid), mail surveys are sure to be the cheapest alternative. However, when money is tight and free labor is available, telephone or face-to-face surveys may be a better choice. Chapter 4 of Salant and Dillman (1994) contains detailed budget examples that can help you determine the probable cost of each method for your particular project. 2. Time pressure to produce results. Mail and face-to-face surveys usually take a substantial amount of time, at least a month and more typically two months, to complete since each contact involves mailing or travel time. However, in a telephone survey, multiple contact attempts can easily be made in a matter of a few days. Thus, if results are needed quickly, a telephone survey may be the only viable option. 3. Sensitivity of topic. Mail surveys are filled out in the privacy of respondents' homes and they never meet the researchers. They therefore offer a high degree of "anonymity." As a result, if a survey includes sensitive topics or information which people may be reluctant to divulge, mail surveys will generally produce higher response rates and more accurate responses. 4. Complexity of survey questions. In a mail survey, respondents have the written questions in front of them, whereas in a telephone survey, questions must be heard and remembered. If a survey contains complicated questions with many different response options or technical questions that need long explanations, it can be very difficult to administer over the telephone, and a mail survey is more appropriate. For surveys that include pictures or diagrams, a telephone survey is out of the question. For surveys that include especially complex or technical questions, a face-to-face survey where respondents can ask questions while reading the survey may be needed. 5. Probability of introducing error or bias. All survey methods are susceptible to error and bias, but in different ways. For example, telephone and face-to-face surveys can be biased by inconsistencies in the behavior of different interviewers or by interviewers inadvertently giving respondents verbal or nonverbal clues about what sort of answer is appropriate or expected. Mail surveys, in which no interviewer is present, are not subject to these problems. However, they are more susceptible to certain other kinds of error. For example, nonresponse error can be more problematic for mail surveys because respondents can look over the survey before deciding whether or not to participate. This greatly increases the chance that respondents and nonrespondents differ on important variables related to the survey topic. In addition, in a mail survey, the researcher cannot control who exactly in a household is filling out the survey and cannot verify that they are doing so conscientously and completely. In contrast, in telephone and face-to-face interviews, the interviewer can exercise greater control over the situation and catch errors and omissions as they occur. Overall, telephone and face-to-face interviews allow for the possibility of obtaining somewhat more complete and accurate results than mail surveys, but only if interviewers are well-trained and consistent. 6. Characteristics of respondents. Sometimes the appropriate survey method is determined by the type of people who are being studied. For example, if a researcher wants to survey the homeless or illegal immigrants, for whom address or phone lists are unavailable, a face-to-face survey is the only viable option. For further information on choosing a survey method, see Chapter 2 of Dillman (1978).

Question Wording

Taking the appropriate steps to minimize sampling error and nonresponse error, while necessary and important, is not sufficient to produce a scientific survey. The questionnaire itself must be written in such a way that the questions are valid (that is, the questions measure what the researcher intends them to measure), reliable (the questions would yield the same results if administered at different times or to different samples), and unbiased (the questions are written in such a way that people are willing and able to provide accurate answers). Writing good questions is perhaps the most difficult and complicated part of any survey project, yet it is also one of the most often ignored.

Beginning researchers tend to make four common mistakes when constructing a questionnaire. First, they simply don't ask enough questions. Surveys from beginning researchers tend to be just a page or two in length, containing a dozen or fewer questions. Such a short survey will rarely if ever capture the information needed to answer a research question definitively. For example, just the demographic questions necessary to describe who the respondents are (questions about gender, age, income, educational background, ethnic background, and so on) can fill a couple of pages. Also, many seemingly simple research questions require multiple survey questions. For example, recall the hypothetical SAT survey from Section III. Did all of the students even take the SAT? If not, what tests did they take? Did they take it multiple times, and, if so, which score or scores should they report? Should they report their combined score or report math and verbal separately? When did they take the SAT (the scoring system was changed a few years ago)? . . . thus even a question as simple as "What was your SAT score?" can require multiple survey questions to answer. In addition, in many cases it makes sense to measure the same variable in multiple ways, since a comprehensive set of questions will be subject to less measurement error than a single question. For example, a survey designed to assess pro environmental behavior might ask respondents about what, specifically, they consistently recycle, what they do, if anything, to conserve energy or water, how often they take public transportation or carpool, whether they buy and use recycled products, what environmental organizations they belong to or contribute to, what they set their thermostat to in the winter, what they did to celebrate Earth Day, and so on. Any one of the single questions could give a misleading picture of an individual; e.g., someone may have been sick on Earth Day; may have had to let their membership in Greenpeace lapse due to lack of funds; or may not have control over the thermostat in their room. However, the entire set of questions taken as a whole would yield a reasonably accurate picture of how much a person does or does not do to help the environment.

A second common mistake made by beginning researchers is to ask too many "open-ended" questions. In open-ended questions, no restrictions are placed on the type of answers that are allowable. The alternative to open-ended questions are "closed-ended" questions in which the possible responses are listed for the respondent. Thus, for example, the question "Where do you live?" is open-ended and the question "Do you live: (a) on-campus or (b) off-campus?" is closed-ended. Open-ended questions, as useful as they are in everyday life, generally make for poor survey questions because they allow a wide variety of possible answers that often stray from the original intent of the question. For example, consider the following possible answers to the question "Where do you live?": "on Park Street," "in an apartment," "in New England," "in the suburbs," "upstairs." Even when respondents do interpret an open-ended question the same way, it can be very difficult to compile and compare answers: for example, how would the following answers to the question "Do you like sushi?" be quantified and compared?: "I love it!," "Sort of," No, it's disgusting!," "Not really," "Sometimes," "Yes." Closed-ended questions are much easier to quantify and analyze than open-ended questions and place fewer demands on respondents, and therefore they are emphasized by most scientific surveys. They do, however, have one potential drawback: in order to compose a good closed-ended question, you must be able to anticipate the great majority of the possible different answers to the question from an often diverse set of respondents. When this is not possible, open-ended questions are preferred, and most surveys will include one or a few such questions to make sure respondents have some chance to convey information not revealed by the closed-ended questions.

A third common survey error is to place too much emphasis on questions about attitudes (e.g., Do you like Bill Clinton?) at the expense of questions about behaviors (e.g., Who did you vote for in the 1996 U. S. Presidential election?). Questions about people's personal actions will generally yield accurate answers. Attitudes, however, are much more difficult to measure because (a) people are sometimes simply not aware of their true attitudes; (b) weakly held attitudes are easily changed; (c) people tend to respond as if their attitudes are long-held and well-formed even when they were just made up on the spot; and (d) attitudes are very sensitive to minor variations in how questions are worded. In particular, attitude questions should never be used as a substitute for behavioral questions since the correlation between attitudes and behavior is often quite low. Attitude questions can play an important role in a survey -- but only if their limitations are understood and if questions about respondent's actual behavior are included as well.

A fourth common mistake made by beginning survey researchers is to fail to think carefully about how to word survey questions. Beginning researchers tend to worry exclusively about the content of questions, resulting in data that are biased by easily avoided problems with the details of question wording. There are literally dozens of issues related to the precise wording of questions that should be carefully considered when constructing a survey. Thus, all survey questions should be put through a "debugging procedure" in which several quality control questions are asked, including the following:

1. Is the question one which respondents can easily answer based on their experience?

2. Is the question simple enough, specific enough, and sufficiently well-defined that all of the respondents will interpret it in the same way?

3. Does the question contain any words or phrases which could bias respondents to answer one way over another?

4. Is it clear to respondents exactly what types of answers are appropriate?

5. Does the question focus on a single topic or does it contain multiple topics that should be broken up into multiple questions?

6. Are any listed response options mutually exclusive?

7. Are any assumptions implied by a question warranted?

This process of writing, debugging, and revising survey questions can't be done in a day or even a week. Identifying all of the flaws and weaknesses in the wording of survey questions typically takes even experienced research teams a couple of weeks or more of review, during which they create, critique, and revise a dozen or more drafts of their questionnaire. There are, however, a couple of strategies that can make this process a little easier. First, it helps considerably to review textbook examples of poorly written survey questions and suggested fixes, such as those available in Chapter 6 of Salant and Dillman (1994) and throughout Fowler (1995).

Second, it is often possible to borrow or adapt questions used and published by other researchers which have already been subjected to a careful review process. So long as the original source of the question is properly acknowledged, this practice is encouraged since it promotes replication of existing results by other researchers, a necessary part of research in any domain.

For further information on question wording, see Fowler (1995); Chapter 11 of Judd et al. (1991); Chapter 6 of Salant and Dillman (1994); and Chapter 11 of Frankfort-Nachmias and Nachmias (1996).

Questionnaire Design

Constructing valid, reliable, and unbiased questions is necessary but not sufficient for creating a good questionnaire: how the questions are organized and presented also deserves careful consideration. The look and feel of a questionnaire serves as an important cue to respondents as they think about how to react to a request to answer a survey. If it is apparent within the first minute or two that the survey is important and easy to complete, people are highly likely to participate; if instead they are not given compelling reasons to take the time away from other activities to answer the survey or if the questions appear to be too difficult, a lot of people will toss the questionnaire into the trash bin or put it on the bottom of their to-do list, resulting in a low response rate. If it is apparent from examining the survey that the researchers put in a lot of time and effort to produce a professional-looking and carefully crafted document, people will likely respond with carefully considered, honest answers; if instead, the survey seems to be poorly organized or contains typographical or other careless errors, respondents will be equally as careless when answering the survey.

Thus questionnaire designers should take several steps to ensure that their instruments make a good impression on potential respondents and to encourage people to respond conscientiously, including the following:

1. Mail surveys should be accompanied by a "cover letter" that briefly introduces the study and explains why it is important and useful. The cover letter should also include three messages that are known to be important for encouraging people to respond: (a) a promise that the respondent's answers will be kept confidential; (b) a statement that describes why their responses, specifically, are necessary for the success of the study; and (c) an accurate estimate of the time it will take to complete the survey (which should generally be no more than 10-15 minutes).

2. A good survey is not a random series of questions but is organized. Questions on related topics should be grouped together into sections and placed under descriptive headings. The sections should appear in order from most to least important or most to least closely related to the central topic of the survey.

3. Questionnaires should contain more than just questions. Introductions and transitional statements that briefly explain to respondents what kind of questions they are going to get and why are important to include in a survey because people find questions much easier to answer when the organization of the survey is made apparent to them.

4. The first few questions on a survey should be carefully chosen, since they must serve to grab respondents' attention and help motivate them to continue to fill out the survey. It is best to begin with a few questions that are easy to answer and that address the most important, central issue of the survey.

5. The format and presentation of the questionnaire must be designed to make it easy to complete without error. Thus, for example, the typeface, type size, and spacing should be easily readable by most anyone (some respondents may have poor eyesight!) and the printing should be of high quality. A particularly important feature of good questionnaires is standardization, that is, when possible, different questions should be presented in the same format in order to reduce the time and effort required from the respondent.

For further information on questionnaire design, see Chapter 4 of Dillman (1978) and Chapter 7 of Salant and Dillman (1994).

Pretesting

In spite of a research team's best efforts, final drafts of surveys often contain errors, omissions, typos, questions that are confusing, biased, or poorly worded, and other problems. This is to be expected, since writing a survey is a complicated undertaking which requires the consideration of many important issues, some of which conflict with each other. And, constructing closed-ended questions requires researchers to anticipate how respondents will answer the questions -- a task which cannot be accomplished without error (if it could, there would be no need to conduct a survey!). However, there is a simple and effective step that researchers can and should take to reduce the chance of survey errors: conducting a survey pretest.

In a pretest a small, but representative sample of respondents are asked to complete the survey and are also interviewed either after each question or at the end of the survey to find out what they were thinking while answering the questions. This gives researchers an opportunity to identify any problems people are having with a survey, such as terms or phrases they find confusing and questions they find too difficult to answer, and to verify that different respondents are interpreting the questions in the same way. Researchers can also test questions for biasedness by asking respondents to guess what the researchers are predicting or expecting the survey results to show. If substantially more respondents than would be expected by random chance can guess the researchers' hypothesis, it is highly likely that the survey contains biased or leading questions.

For more information on pretesting see Chapter 5 of Fowler (1995).

Survey Implementation

Even a perfectly prepared questionnaire is of no use to a researcher if large numbers of people fail to complete it. Thus the main goal of any survey implementation plan should be to obtain a high response rate. Beginning researchers tend to assume that a single attempt to contact each respondent will result in an adequate response rate. However, this greatly underestimates the necessary level of effort: research has shown time and again that the only effective way to achieve survey response rates of 50% or higher is to make repeated, personalized attempts to contact and encourage potential respondents to participate. For a mail survey, the need for multiple mailings results in substantially higher copying and postage expenses and adds weeks of time to the research schedule. For a telephone survey, the need to make multiple calls to most respondents can double the amount of time interviewers spend on the phones. And for a face-to-face survey, the additional travel time required to make multiple visits to households or businesses can become prohibitively expensive. It is therefore critically important for survey researchers to develop a realistic implementation plan that takes these costs and delays into account.

A typical implementation plan for a mail survey would include 4 separate mailings:

1. The first mailing is an introductory postcard or letter informing people that they will be asked to participate in a survey and explaining what it is about.

2. The second mailing typically includes a cover letter, the survey, and a stamped return envelope.

3. A week or so after the survey is sent out, a third mailing is sent to all potential respondents to remind people to fill out the survey, if they haven't already done so.

4. Finally, a couple of weeks later a fourth mailing, including another copy of the survey, is sent to those people who have not yet responded to the survey.

All of these communications should be professional in appearance and should show evidence of personal attention, such as the use of authentic rather than computer-generated signatures and stamps rather than preprinted envelopes.

Telephone and face-to-face surveys should also, when possible, begin with an initial mailing that informs people that they will receive a telephone call or visit from a researcher. Again the principle of multiple attempts to contact people and encourage them to respond should be applied; a researcher should not give up on a respondent unless 3 or more phone calls or visits have been attempted, at different times on different days. It is also important to keep a careful record of the results of each contact attempt, particularly if the attempts will be made, as they often are, by different interviewers. The bulk of the preparation necessary to implement telephone and face-to-face surveys, however, involves various efforts which are necessary to make sure that different interviewers are collecting data using the same procedures. For example, a "script" should be prepared and faithfully followed by each interviewer for the first couple of minutes of an interview, when the study is described and participation is requested. It is also necessary to try to anticipate, insofar as possible, the questions respondents are likely to ask during interviews, so that each interviewer can refer to a standard set of "stock" answers.

For further information on survey implementation, including examples of mail correspondence and telephone interviewer instructions, see Chapters 5 and 7 of Dillman (1978) and Chapter 8 of Salant and Dillman (1994).

Ethical Considerations

All social science researchers have an ethical obligation to protect the welfare of the people they study. Although survey studies tend to be relatively innocuous compared to some alternate methodologies, there are three ethical principles that all survey studies should follow:

1. Respondents should be informed that participation is voluntary and that they may omit answers to any particular questions if they choose. Certain steps may be taken to encourage participation (e. g., you might explain the importance of your research or how, as part of a carefully selected sample, their answers are needed to help ensure the validity of the study). However, in the final analysis, people have every right to refuse to participate and should not be coerced.

2. Adequate measures must be taken to protect the confidentiality of respondents. Although overall survey results may be presented publicly, individuals should never be publicly identified or associated with their individual responses.

3. Any promises made to the survey respondents (e.g., that you will send them a copy of the survey results when they are available) must be kept.

There are two situations in which further steps must be taken to protect the rights of survey participants:

1. If your survey includes sensitive questions about intimate relationships, personal habits, or illegal practices or any question that might induce embarrassment, anxiety, shame, psychological stress, or any other strong emotional reaction in your respondents.

2. If your survey is part of an experimental design that involves giving alternate versions of your survey to different groups of people. (This turns your survey project into a formal experiment on human subjects, for which stricter ethical guidelines apply. For example, researchers are required to document the informed consent of experimental subjects and to debrief them on the reasoning behind the experiment when it is completed.)

In either case you should receive approval from WPI's Human Subjects Research Committee before proceeding with your study.

For further information on the ethics of social science research see American Psychological Association (1982); Chapter 20 of Judd et al. (1991); and Chapter 4 of Frankfort-Nachmias and Nachmias (1996).

Reporting on Survey Methodology

Gaining a thorough understanding of the principles of scientific survey design and implementing them in your study is not by itself sufficient to gain acceptance of your findings and conclusions. It is also necessary to convince the readers and users of your study that your methodology was sound. This requires the inclusion in your project report of a methodology section which explains in detail what you did and how you went about doing it, so that readers can judge the quality of your work for themselves. The methodology section also provides an opportunity for researchers to explain the reasoning behind the methodological choices that they made and to anticipate and answer the questions that skeptical readers are likely to ask.

For survey projects, the methodology section of the project report typically contains the following five sections:

1. Respondents. The first section should describe the people who participated in the study and answer such questions as: How many respondents are there? What are their general demographic characteristics (e.g., age, education, income, ethnic background)? Why did you choose to study this group of people and not other possible groups? What was done to ensure that respondents were treated in an ethical manner?

2. Sampling Design. The second section should describe in detail the design of the sampling plan. For example, How was the sampling frame compiled? How was the size of the sample determined? Was simple random sampling employed or was a more complicated sampling design necessary?

3. Questionnaire Design. The questionnaire used in the study should appear exactly as seen by or read to respondents in an appendix to the project report, and this third methodology section should guide the reader through its design and construction, answering such questions as: Why is the survey organized the way it is? Why does it include the questions it does and not alternate possible questions? What was the "debugging" procedure used to make help make choices about question wording? What was done to make the questionnaire easy to fill out? Was the survey pretested? If so, how was the pretest conducted and what did it reveal about the survey?

4. Procedure. The fourth section should describe each step in the implementation of the survey, and address such questions as: What survey method was employed and why? When was the survey administered? What steps were taken to increase the response rate to the survey? What response rate was obtained? What was participation in the study like, from the respondents' point of view? If interviewers were used, how were they trained and how did you ensure that different interviewers used the same procedures? Did any unexpected problems occur? If so, what were they and how were they resolved?

5. Limitations. As discussed in Section II, all survey studies have certain methodological limitations in common. And, most surveys have additional limitations that are imposed by constraints on time and money and by other factors unique to a particular project. Thus researchers are not expected by readers and users of their work to have conducted the "perfect" survey study. However, researchers are expected to demonstrate that they have a thorough understanding of the limitations of their own work and that they have made reasonable judgments about how to spend their limited time and resources. The final section concerning methodology should therefore acknowledge the limitations of the study and explain how they may affect the interpretation of the results. For example: To what extent was the sampling frame representative of the population, and what are the potential impacts of any errors or omissions? To what extent was the study subject to sampling error? What was the response rate? What, if anything, is known about the nonrespondents? Which questions are more senstive to possible errors or biases than others?

Data Analysis (A Very Brief Introduction)

The primary purpose of conducting a survey, of course, is to produce data that will help answer important research questions. Once collected, the data must be collated, organized, summarized, and described. Most beginning survey researchers understand this, and faithfully go about calculating summary measures such as means, frequencies, standard deviations, and correlations and creating tables and graphs that illustrate important findings. Such activities are appropriate, necessary, and important. However, they are not sufficient to allow conclusions to be drawn from survey data.

Unless the entire population of interest was surveyed and the response rate was 100%, the data provided by surveys are estimates of population variables. This means they are almost surely wrong. The estimates may be off by just a small fraction of a percentage point or they may be off by 10 percentage points or more, but they are off by some unknown amount. The amount of error cannot in fact be determined with certainty. However, it is possible, through applying a type of statistics called "inferential statistics," to determine the likelihood of different sizes of errors and therefore how much confidence one can have in the sample estimates. This determination of the degree of confidence in the results depends on the sample size and the pattern of variation in the data, and thus cannot be done simply by examining summary measures, tables, or graphs: it requires additional statistical calculations, and it is irresponsible to report sample estimates without completing this additional step.

The logic involved in inferential statistics can be illustrated in the simple case of determining the appropriate degree of confidence in a sample mean of a single variable by returning to the SAT example described in Section III. In that section the following formula was presented for calculating the necessary sample size n for a survey project:

n = (SD)²/(SE)².

This formula can be turned around to yield the following formula for calculating SE, the standard error of the mean:

SE = SD/SQRT(n),

where is SD is the estimated standard deviation of the variable in the population (the standard deviation of the sample serves as the estimate) and SQRT(n) is the square root of the sample size (see also footnote 3 regarding the finite population correction). Thus, suppose in the SAT study a sample size of 100 was obtained, the mean SAT score in the sample was 500, and the standard deviation of the SAT score in the sample was 150. The standard error of the mean in this case would be

SE = 150/SQRT(100) = 15.

Assuming the SAT scores are normally distributed (i.e., form a symmetrical "bell-shaped" curve about the mean score), there is a 68% chance that the population mean SAT score is within plus or minus one standard error, or within plus or minus 15 points of the sample mean of 500. Such a range of possible population values, in this case ranging from 485 to 515, is called a "confidence interval." Many different confidence intervals could be constructed depending on how much confidence is desired. In fact, a 68% confidence interval does not represent a very high degree of confidence, since the population mean will fall outside the stated interval 32% of the time. Social scientists prefer a lower chance of error, and generally use a 95% confidence interval, which corresponds to a range of plus or minus two standard errors. Thus, the results of the SAT study might be best described as follows: "We are 95% confident that the population mean SAT score falls between 470 and 530."

Statistical inference calculations of the sort presented in this simple example are an essential component of any scientifically defensible analysis of survey data.
However, for most survey projects the necessary statistics quickly become much more complicated than in this simple case, and a detailed discussion of these more complicated statistical techniques is beyond the scope of the current work. Thus, in order to learn more about how to conduct an appropriate analysis of your data, you will have to consult other sources. In particular, your research team should take the following steps:

1. Develop a strong foundation in the basic logic and terminology of both descriptive and inferential statistics. This foundation may come from a formal course in statistics or from reading descriptions of data analysis written especially for beginning researchers (see Chapters 10-15 of Rosnow and Rosenthal, 1996; Chapters 15-17 of Judd et al., 1991).

2. Determine which specific statistical analysis techniques are appropriate for your data (e.g., regression, analysis of variance, chi-square analysis) and learn how to apply them and interpret the results. This is best accomplished by consulting comprehensive introductory textbooks on statistics for the social sciences, such as Agresti (1997), McCall (1998), McGrath (1997), or Shavelson (1996).

3. Learn how to code survey data into a data matrix that can be analyzed by a computer program. Chapter 15 of Judd et al. (1991) contains a good description of this coding process.

4. Become proficient in the use of an appropriate statistical analysis software package. One of the packages available to WPI students is called SAS, and there is a SAS tutorial available on the WWW at

http://www.math.wpi.edu/Course_Materials/SAS/tutorial96.html

At this point it should be fairly obvious that data analysis is not a trivial undertaking. It is not uncommon, in fact, for a project team to take a month or even an entire 7-week term to learn about data analysis techniques, to code, analyze, and interpret their data, and to write up the results, so this time should be planned for and allocated at the beginning of the project.

Conclusion

This document has provided an introduction to the basic principles of scientific survey design and outlined the steps that all beginning survey researchers should take, including:

1. Determining if a survey study is the best way to answer your research questions.

2. Obtaining a random or representative sample of sufficient size.

3. Making an informed choice of survey method.

4. Creating a questionnaire that is valid, reliable, and unbiased.

5. Designing a questionnaire and implementation plan that achieve a high response rate.

6. Developing procedures that ensure that people are treated ethically.

1. Conducting a scientifically defensible statistical analysis of the survey data.

Decades of research on survey design and methodology have demonstrated the importance of these basic steps. When they are followed, survey data can be a critically important source of information that supports decision making and policy formation. When they are ignored, survey data can be inaccurate and biased, and the decisions and policies based upon them ineffective or even harmful. Many students are surprised when they find out how complicated survey design and methodology is and how much time and effort it takes to produce a high-quality survey project. However, there is no substitute for the level of effort and attention to detail that is required to obtain high quality: social science research methods of all types are inherently complicated because people and societies are complicated. In the case of survey research, at least, the basic information is now available to undergraduate students in this document and other easily accessible and highly readable books, so high quality surveys should be achievable by the majority of IQP projects. Of course, in some cases quality must be sacrificed to some degree due to constraints on time and budget. This is understandable and such surveys are still worth doing: as long as the limitations of the project are understood and acknowledged, most surveys will yield useful information.

Finally, it should be emphasized that this document is intended to serve as an introduction to survey design and methodology and is therefore incomplete in several important ways. For example, only the simplest sampling techniques are described, many important issues related to question wording have been omitted, detailed examples of questionnaires, cover letters, and other documents have not been included since they are readily available from other sources, and data analysis is presented in a particularly abbreviated fashion. Given these limitations, it is imperative that students consult additional sources on survey design and methodology and data analysis when planning their projects. The recent books by Salant and Dillman (1994) and Rosenthal and Rosnow (1996) are probably the best places to start, but even these books are overly simplistic and incomplete. Most survey projects will also need to consult the wider literature on survey design and methodology and data analysis for more detailed information related to the specific concerns of their project.

Answers to True/False Questions

The answer to all of the questions is false:

1. Once a population reaches a certain size, the size of the sample necessary to estimate opinions within a few percentage points is fairly constant. A carefully selected sample of about 1200 people is sufficient to determine the opinions of the entire U. S. population with a sampling error of plus-or-minus 3%.

2. Since telephone directories don't include people who have unlisted numbers, people without phones, and people whose listings have been changed or added since the last publication, they cannot be relied upon to provide a random sample. Better choices for choosing a sample of telephone numbers are random digit dialing and add-a-digit dialing. See Chapter 5 of Salant and Dillman (1994).

3. Actually, it is essential for survey questions to be grouped categorically to minimize the burden on the respondent and to demonstrate that careful thought went into the design of the questionnaire. In addition, in some cases an alternate ordering of identical questions can change responses by as much as 30%, so question ordering requires careful consideration.

4. Although posting a survey on the web can give you access to a huge audience, this audience is not representative of the general population. Even if you wish to generalize only to Web users, this strategy does not allow you to determine the response rate to your survey, a step which is absolutely necessary to be able to interpret survey results. See Section IV.

5. It is not acceptable practice to keep adding names to your sample until you get a sufficient number of respondents, since this results in a very low response rate. The correct approach is to adopt methods and procedures that will ensure a large percentage of your original sample will respond.

References Cited

Agresti, A. (1997). Statistical Methods for the Social Sciences, 3rd. ed. Upper Saddle River, NJ: Prentice-Hall.
American Psychological Association (1982). Ethical Principles in the Conduct of Research with Human Participants. Washington, D.C.: American Psychological Association.
Clayton, P., Applebee, A., and Pascoe, C. (1996). Email surveys: Old problems with a new delivery medium. LASIE: Information Bulletin of the Library Automated Systems Information Exchange, 27(2), 30-39.
Dillman, D. A. (1978). Mail and Telephone Surveys: The Total Design Method. New York: Wiley.
Emerson, R. B. (1983). Contemporary Field Research. Boston: Little, Brown.
Fowler, F. J., Jr. (1995). Improving Survey Questions: Design and Evaluation. Thousand Oaks, CA: Sage Publications.
Frankfort-Nachmias, C., and Nachmias, D. (1996). Research Methods in the Social Sciences, 5th ed. London: Edward Arnold.
Groves, R. M. (1989). Survey Errors and Survey Costs. New York: Wiley.
Henry, G. T. (1990). Practical Sampling. Newbury Park, CA: Sage.
Judd, C. M., Smith, E. R., and Kidder, L. H. (1991). Research Methods in Social Relations, 6th ed. Fort Worth, TX: Holt, Rinehart, and Winston.
Labaw, P. J., (1980). Advanced Questionnaire Design. Cambridge, MA: Abt Books.
Martin, P., and Bateson, P. (1986). Measuring Behavior: An Introductory Guide. Cambridge: Cambridge Univ. Press.
McCall, R. B. (1998). Fundamental Statistics for the Behavioral Sciences, 7th ed. Pacific Grove, CA: Brooks/Cole.
McGrath, R. E. (1997). Understanding Statistics: A Research Perspective. New York: Longman.
McKenna, R. J. (1995). The Undergraduate Researcher's Handbook: Creative Experimentation in Social Psychology. Boston: Allyn and Bacon.
Morgan, D. L. (1988). Focus Groups as Qualitative Research. Newbury Park, CA: Sage Publications.
Rosnow, R. L., and Rosenthal, R. (1996). Beginning Behavioral Research: A Conceptual Primer, 2nd ed. Englewood Cliffs, NJ: Prentice-Hall.
Salant, P. A., and Dillman, D. A. (1994). How to Conduct Your Own Survey. New York: Wiley.
Shavelson, R. J. (1996). Statistical Reasoning for the Behavioral Sciences, 3rd ed. Boston, MA: Allyn and Bacon.
Survey Research Center (1983). General Interviewing Techniques: A Self-
Instructional Workbook for Telephone and Personal Interviewer Training. University of Michigan, Ann Arbor: Survey Research Center.
Weber, R. P. (1985). Basic Content Analysis. Beverly Hills, CA: Sage.
Yin, R. K. (1989). Case Study Research: Design and Methods. Newbury Park, CA: Sage Publications.

Back to SySurvey Whitepapers