Back to SySurvey Whitepapers

Questionnaire Design and Surveys Sampling
Author: Professor Hossein Arsham
Link: University of Baltimore
The contents of this paper are aimed at those who need to perform basic statistical analyses on data from sample surveys, especially those in marketing science. A basic knowledge of statistics, such as descriptive statistics and the concept of hypothesis testing, is useful.
Professor Hossein Arsham   

Contents
  • Questionnaire Design and Surveys Management
  • Sample Size in Surveys Sampling
  • Multilevel Statistical Models
  • Surveys Sampling Routines
  • Cronbach's Alpha (Coefficient Alpha)
  • Instrumentality Theory
  • Value Measurements Survey Instruments (Rokeach's Value Survey)
  • Interesting and Useful Sites


    		•

    Questionnaire Design and Surveys Management
    When the sampling units are human beings, the main methods of collecting information are:
  • face-to-face interviewing
  • postal surveys

  • telephone surveys

  • direct observation

  • Internet


    		•
    Objectives:
  • To enable students to understand the integrated processes of designing and conducting quantitative survey research projects.
  • To give students experience of grappling with problems in the design of survey samples, the construction of data collection instruments and the management of survey projects.
  • To make student aware of main sources of error in the survey process and ways of detecting, controlling and minimising such error.


    • Course Content:
  • The quantitative survey process from project formulation, statistical design and sampling, through instrument design and question formulation, to data processing.
  • Basic principles and practice of probability sample design for field surveys.
  • How to operationalise concepts, word questions and design, develop and test survey instruments, taking account of intended uses of the data collected.
  • Principles of manual coding and editing of survey data, computer editing and preparing data for analysis.
  • Sources of error in survey data, ways of assessing them and ways of minimising error.
  • Planning and management of large scale surveys, piloting and pre testing, relations with stakeholders in the sponsored survey process, issues in survey ethics.

    • The main questions are:
  • What is the purpose of the survey?
  • What kinds of questions the survey would be developed to answer?
  • What sorts of actions is the company considering based on the results of the survey?


    • Step 1: Planning Questionnaire Research Consider the advantages and disadvantages of using questionnaires.
    Prepare written objectives for the research.
    Have your objectives reviewed by others.
    Review the literature related to the objectives.
    Determine the feasibility of administering your questionnaire to the population of interest.
    Prepare a time-line.
    Step 2. Conducting Item Try-Outs and an Item Analysis
    Have your items reviewed by others.
    Conduct "think-aloud" with several people.
    Carefully select individuals for think-aloud.
    Consider asking about 10 individuals to write detailed responses on a draft of your questionnaire.
    Ask some respondents to respond to the questionnaire for an item analysis. In the first stage of an item analysis, tally the number of respondents who selected each choice.
    In the second stage of an item analysis, compare the responses of high and low groups on individual items.
    Step 3: Preparing a Questionnaire for Administration
    Write a descriptive title for the questionnaire.
    Write an introduction to the questionnaire.
    Group the items by content, and provide a subtitle for each group.
    Within each group of items, place items with the same format together.
    At the end of the questionnaire, indicate what respondents should do next.
    Prepare an informed consent form, if needed.
    If the questionnaire will be mailed to respondents, avoid having your correspondence look like junk mail.
    If the questionnaire will be mailed, consider including a token reward.
    If the questionnaire will be mailed, write a follow-up letter.
    If the questionnaire will be administered in person, consider preparing written instructions for the administrator.
    Step 4: Selecting a Sample of Respondents
    Identify the accessible population.
    Avoid using samples of convenience.
    Simple random sampling is a desirable method of sampling.
    Systematic sampling is an acceptable method of sampling.
    Stratification may reduce sampling errors.
    Consider using random cluster sampling when every member of a population belongs to a group.
    Consider using multistage sampling to select respondents from large populations.
    Consider the importance of getting precise results when determining sample size.
    Remember that using a large sample does not compensate for a bias in sampling.
    Consider sampling non respondents to get information on the nature of a bias.
    The bias in the mean is the difference of the population means for respondents and non respondents multiplied by the population nonresponse rate.
    Step 5: Preparing Statistical Tables and Figures
    Prepare a table of frequencies.
    Consider calculating percentages and arranging them in a table with the frequencies.
    For nominal data, consider constructing a bar graph.
    Consider preparing a histogram to display a distribution of scores.
    Consider preparing polygons if distributions of scores are to be compared.
    Step 6: Describing Averages and Variability
    Use the median as the average for ordinal data.
    Consider using the mean as the average for equal interval data.
    Use the median as the average for highly skewed, equal interval data.
    Use the range very sparingly as the measure of variability.
    If the median has been selected as the average, use the interquartile range as the measure of variability.
    If the mean has been selected as the average, use the standard deviation as the measure of variability.
    Keep in mind that the standard deviation has a special relationship to the normal curve that helps in its interpretation.
    For moderately asymmetrical distributions the mode, median and mean satisfy the formula: mode=3*median-2*mean.
    Step 7: Describing Relationships
    For the relationship between two nominal variables, prepare a contingency table.
    When groups have unequal numbers of respondents, include percentages in contingency tables.
    For the relationship between two equal interval variables, compute a correlation coefficient.
    Interpret a Pearson r using the coefficient of determination.
    For the relationship between a nominal variable and an equal interval variable, examine differences among averages.
    Step 8: Estimating Margins of Error
    It is extremely difficult, and often impossible, to evaluate the effects of a bias in sampling.
    When evaluating a percentage, consider the standard error of a percentage.
    When evaluating a mean, consider the standard error of the mean.
    When evaluating a median, consider the standard error of the median.
    Consider building confidence intervals, especially when comparing two or more groups
    Step 9: Writing Reports of Questionnaire Research
    In an informal report, variations in the organization of the report are permitted.
    Academic reports should begin with a formal introduction that cites literature.
    The second section of academic reports should describe the research methods.
    The third section of academic reports should describe the results.
    The last section of academic reports should be a discussion. Acknowledge any weakness in your research methodology.

    Missing Values on a Sensitive Topic
    A natural way to get answers is to, as much as possible, assure people that the surveys are anonymous, and to find a way to make the respondent at least minimally comfortable. So, according to US General Accounting Office book, "Developing and Using Questionnaires" (Oct 1983) chapter 9, you should do the following:
    1. explain to respondent the reasons for asking the questions,
    2. make response categories as broad as possible.
    3. word the question in a nonjudgemental style that avoids the appearance of censure, or, if possible, make the behavior in question appear to be socially acceptable.
    4. present the request as factual matter as possible.
    5. guarantee confidentiality or anonymity
    6. make sure the respondent knows the info will not be used in any threatening way.
    7. explain how the info will be handled
    8. avoid cross classification that will allow for pinpointing responses.

    Source of Errors

  • The use of an inadequate frame.
  • A poorly designed questionnaire.
  • Recording and measurement errors.
  • Non-response problems.

    		•
    For example consider the following question: "Over the last twelve months would you say your health has on the whole been : Good? / Fairly good? / Not good?" . The respondent is required to tick one of 3 thus-labeled boxes.
    What is wrong with the following:
    It is the ONLY question on the form, which asks about a matter of opinion rather than fact, but this distinction is not in any way represented in its layout or wording.
    Whereas for a question about opinion there should be a response option of 'Don't Know' this is not provided. In some cases, such as the Census Form and the Census advisory staff are adamant that the question must be answered. Thus a person with no opinion on the matter is in a quandary and threatened with possible legal action.
    This particular question is highly ambiguous as regards the qualitative nature of what is being asked about (your health). Is one to respond in terms of how one feels, how one can perform, comparisons with peer groups, comparisons with other periods of one's life, or what?

    Relatively recent innovations surrounding the Internet have spawned new ways for conducting surveys: most notably electronic mail (e-mail) surveys and WWW surveys. While still in its infancy, it is clear that the Internet is here to stay and this new medium is going to be used for survey data collection. The main question is how the Internet can be used for survey data collection by some effective and efficient design considerations.
    Visit also the following Web sites:
    Association for Survey Computing
    Write more effective survey questions
    Research Methods Knowledge Base
    Research Methods & Statistics Resources
    Sampling In Research
    Sampling, Questionnaire Distribution and Interviewing
    SRMSNET: An Electronic Bulletin Board for Survey Researchers

    Read also some of the articles in the Public Opinion Quarterly, This journal focuses on methodologies for survey research.

    Sample Size in Surveys Sampling

    People sometimes ask me, what fraction of the population do you need? I answer, "It's irrelevant; accuracy is determined by sample size alone" This answer has to be modified if the sample is a sizable fraction of the population.

    For an item scored 0/1 for no/yes, the standard deviation of the item scores is given by SD = (p(1-p)/N) 1/2 where p is the proportion obtaining a score of 1, and N is the sample size.

    The standard error of estimate SE (the standard deviation of the range of possible p values based on your sample estimate) is given by SE= SD/N. Thus, SE is at a maximum when p = 0.5. Thus the worst case scenario occurs when 50% agree, 50% disagree.

    The sample size, N, can then be expressed as largest integer less than or equal to 0.25/SE2

    Thus, for SE to be 0.01 (i.e. 1%), a sample size of 2500 would be needed; 2%, 625; 3%, 278; 4%, 156, 5%, 100.

    Note, incidentally, that as long as the sample is a small fraction of the total population, the actual size of the population is entirely irrelevant for the purposes of this calculation.

    Sample sizes with regard to binary data:

    n = [t2 N p(1-p)] / [t2 p(1-p) + a2 (N-1)]

    with N being the size of the total number of cases, n being the sample size, a the expected error, t being the value taken from the t distribution corresponding to a certain confidence interval, and p being the probability of an event.

    There are several formulas for the sample size needed for a t-test. The simplest one is

    n = 2(Za+Zb)2s2/D2

    which underestimates the sample size, but is reasonable for large sample sizes. A less inaccurate formula replaces the Z values with t values, and requires iteration, since the df for the t distribution depends on the sample size. The accurate formula uses a non-central t distribution and it also requires iteration.

    The simplest approximation in your case is to replace the first Z value in the above formula with the value from the studentized range statistic that is used to derive Tukey's follow-up test. If you don't have sufficiently detailed tables of the studentized range, you can approximate the Tukey follow-up test using a Bonferroni correction. That is, change the first Z value to Za where k is the number of comparisons.

    Neither of these solutions is exact. I suspect that the exact solution is a bit messy. But either of the above approaches is probably close enough, especially if the resulting sample size is larger than (say) 30.

    A better stopping rule for conventional statistical tests is as follows:
    Test some minimum (pre-determined) number of subjects.
    Stop if p-value is equal to or less than .01, or p-value equal to or greater than .36; otherwise, run more subjects.

    Obviously, another option is to stop if/when the number of subjects becomes too great for the effect to be of practical interest. This procedure maintains a about 0.05.

    We may categorized probability proportion to size (PPS) sampling, stratification, and ratio estimation (or any other form of model assisted estimation) as tools that protect one from the results of a very unlucky sample. The first two (PPS sampling and stratification) do this by manipulation of the sampling plan (with PPS sampling conceptually a limiting case of stratification). Model assisted estimation methods such as ratio estimation serve the same purpose by introduction of ancillary information into the estimation procedure. Which tools are preferable depends, as others have said, on costs, availability of information that allows use of these tools, and the potential payoffs (none of these will help much if the stratification/PPS/ratio estimation variable is not well correlated with the response variable of interest).

    Therefore, you must use whatever tools are at your disposal that would improve your estimates at feasible costs.

    There are also heuristic methods for determination of sample size. For example, in healthcare behavior and process measurement sampling criteria are designed for a 95% CI of 10 percentage points around a population mean of 0.50; There is a heuristic rule: "If the number of individuals in the target population is smaller than 50 per month, systems do not use sampling procedures but, attempt to collect data from all individuals in the target population.", visit e.g., The Joint Commission on Accreditation of Healthcare Organizations.

    Also visit: Probability Sampling

    Multilevel Statistical Models

    Many kinds of data, including observational data collected in the human and biological sciences, have a hierarchical or clustered structure. For example, animal and human studies of inheritance deal with a natural hierarchy where offspring are grouped within families. Offspring from the same parents tend to be more alike in their physical and mental characteristics than individuals chosen at random from the population at large.

    Many designed experiments also create data hierarchies, for example clinical trials carried out in several randomly chosen centers or groups of individuals. Multilevel models are concerned only with the fact of such hierarchies not their provenance. We refer to a hierarchy as consisting of units grouped at different levels. Thus offspring may be the level 1 units in a 2-level structure where the level 2 units are the families: students may be the level 1 units clustered within schools that are the level 2 units.

    The existence of such data hierarchies is not accidental and should not be ignored. Individual people differ as do individual animals and this necessary differentiation is mirrored in all kinds of social activity where the latter is often a direct result of the former, for example when students with similar motivations or aptitudes are grouped in highly selective schools or colleges. In other cases, the groupings may arise for reasons less strongly associated with the characteristics of individuals, such as the allocation of young children to elementary schools, or the allocation of patients to different clinics. Once groupings are established, even if their establishment is effectively random, they will tend to become differentiated, and this differentiation implies that the group' and its members both influence and are influenced by the group membership. To ignore this relationship risks overlooking the importance of group effects, and may also render invalid many of the traditional statistical analysis techniques used for studying data relationships.

    A simple example will show its importance. A well known and influential study of primary (elementary) school children carried out in the 1970's claimed that children exposed to so called 'formal' styles of teaching reading exhibited more progress than those who were not. The data were analyzed using traditional multiple regression techniques, which recognized only the individual children as the units of analysis and ignored their groupings within teachers and into classes. The results were statistically significant. Subsequently, it has been demonstrated that when the analysis accounted properly for the grouping of children into classes, the significant differences disappeared and the 'formally' taught children could not be shown to differ from the others.

    This re-analysis is the first important example of a multilevel analysis of social science data. In essence what was occurring here was that the children within any one classroom, because they were taught together, tended to be similar in their performance. As a result they provide rather less information than would have been the case if the same number of students had been taught separately by different teachers. In other words, the basic unit for purposes of comparison should have been the teacher not the student. The function of the students can be seen as providing, for each teacher, an estimate of that teacher's effectiveness. Increasing the number of students per teacher would increase the precision of those estimates but not change the number of teachers being compared. Beyond a certain point, simply increasing the numbers of students in this way hardly improves things at all. On the other hand, increasing the number of teachers to be compared, with the same or somewhat smaller number of students per teacher, considerably improves the precision of the comparisons.

    Researchers have long recognized this issue. In education, for example, there has been much debate about the so called 'unit of analysis' problem, which is the one just outlined. Before multilevel modelling became well developed as a research tool, the problems of ignoring hierarchical structures were reasonably well understood, but they were difficult to solve because powerful general purpose tools were unavailable. Special purpose software, for example for the analysis of genetic data, has been available longer but this was restricted to 'variance components' models and was not suitable for handling general linear models. Sample survey workers have recognized this issue in another form. When population surveys are carried out, the sample design typically mirrors the hierarchical population structure, in terms of geography and household membership. Elaborate procedures have been developed to take such structures into account when carrying out statistical analyses.

    For more details, visit Multilevel Models Project

    References and Further Readings:
    Goldstein H., Multilevel Statistical Models, Halstead Press, New York, 1995.
    Longford N., Random Coefficient Models, Clarendon Press, Oxford, 1993.
    These books cover a very wide range of applications and theory.

    Surveys Sampling Routines

    Note: The following programs are referred to the Practical Methods for Design and Analysis of Complex Surveys, by R. Lehtonen, and E. Pahkinen, Wiley, Chichester, 1995. See also, L.Lyberg et al., (Editors), Survey Measurement and Process Quality, New York, Wiley, 1997.
    Other software packages such as Le Sphinx, CENVAR, CLUSTERS, Epi Info, Generalized Estimation System, Super CARP, Stata, SUDAAN, VPLX, WesVarPC, and ORIRIS IV. For a detailed review visit Summary of Survey Analysis Software

    		 
    			TITLE Bernoulli sampling; PI=0.25, N=32
			GET FILE (input dataset)
			COMPUTE PI=0.25
			COMPUTE EPSN=UNIF(1)
			SELECT IF (EPSN LT PI)
			WRITE OUTPUT=(output dataset)
			

    
			TITLE Simple random sampling with replacement; n=8, N=32
			GET FILE (input dataset)
			COMPUTE L=L+ID
			LEAVE L
			COMPUTE E=L-ID
			NUMERIC W(f2)
			COMPUTE W=0
			DO REPEAT A=A1-A8
			IF (ID=1) A=UNIF(32)
			LEAVE A
			IF (E LT A AND A LE L) W=W + 1
			END REPEAT
			SELECT IF (W GT 0)
			WRITE OUTFILE = (output dataset)
			

    
			TITLE Simple random sampling without replacement; n=8, N=32
    
			GET FILE (input dataset)
			SAMPLE 8 FROM 32
			WRITE OUTFILE = (output dataset)
			

    
			TITLE Systematic sampling; n=8, sampling interval =4
			MATRIX
			COMPUTE RAND = TRANC (4*UNIFORM(1,1))
			COMPUTE INT=RAND*MAKE(32, 1, 1)
			SAVE INT/OUTPUT=*/VAR=INT
			END MATRIX
			MATCH FILES FILE = (input dataset)/FILE=*
			COMPUTE INDEX = MOD ($CASENUM, 4)
			SELECT IF (INDEX=INT)
			SAVE OUTPUTFILE = (output dataset)/DROP=INDEX INT
			

    
			The Following routines are for sampling 
    
			(selection with probability proportion to size)
			

    
			TITLE PPS Poisson sampling with expected size of 8
    
			GET FILE )input dataset)
			COMPUTE PI=8*HOU*%/91753
			COMPUTE EPSN=UNIF(1)
			SELECT IF (EPSN LE PI)
			WRITE OUTFILE (output dataset)
			

    
			TITLE PPS Sampling with replacement; n=8
			GET FILE (input dataset)
			COMPUT L=L+HOU85
			LEAVE L
			COMPUTE E=L-HOU85
			NUMERIC W(F2)
			COMPUTE W=0
			DO REPEAT A A1 TO A8
			IF (ID=1) A=INIF(91753)
			LEAVE A
			IF (E LT A AND ALE L) W=W+1
			END REPEAT 
			SELECT IF W GT 0
			WRITE OUTFILE = (output dataset)
			

    
			TITLE PPS Systematic sampling n=8
			GET FILE (input dataset)
			COMPUTE #C=#C + 1
			COMPUTE CASE = #C
			COMPUTE #SN=8
			COMPUTE #PN=01753
			COMPUTE #INT=TRUNC (#PN/#SN)
			COMPUTE #RAN= TRUNC (UNIFORM (#INT) +1)
			DO IF CASE = 1
			COMPUTE #COMP=#RAN
			COMPUTE RAN=#RAN
			END IF
			COMPUTE SAMIND=0
			LOOP IF +COMP LE CUMHOU*%
			+ COMPUTE SAMIND = SAMIND+1
			+ COMPUTE #COMP+#COMP+#INT
			END LOOP
			EXECUTE.
			WRITE OUTFILE= (output dataset)
    Visit also Survey Samplings.
    References and Further Readings:
    Bethel J., Sample allocation in multivariate surveys, Survey Methodology, 15, 1989, 47-57.
    Valliant R., and J. Gentle, An application of mathematical programming to a sample allocation problem, Computational Statistics and Data Analysis, 25, 1997, 337-360.

    Cronbach's Alpha (Coefficient a)

    Perhaps the best way to conceptualize Cronbach's Alpha is to think of it as the average of all possible split half reliabilities for a set of items. A split half reliability is simply the reliability between two parts of a test or instrument where those two parts are halves of the total instrument. In general, the reliabilities of these two halves should then be stepped up (Spearman Brown Prophesy Formula) to estimate the reliability for the full length test rather than the reliability between to half length tests. Assuming, for ease of interpretation, that a test has an even number of items (e.g, 10), then items 1-5 versus 6-10 would be one split, evens versus odds would be another and, in fact, with 10 items chosen 5 at a time, there are 10 chose 5 or 252 possible split halves for this test. If we compute each of these stepped up split half reliabilities and averaged them all, this average would be Cronbach's Alpha. Since some splits will be better than others in terms of creating two more closely parallel halves, and the reliability between parallel halves is probably the most appropriate estimate of an instrument's reliability, Cronbach's alpha is often considered a relatively conservative estimate of the internal consistency of a test.

    The following is a SAS program for computing coefficient alpha or Cronbach's Alpha. Note that, it is an option in the PROC CORR procedure. In SAS, for a WORK data set called ONE, suppose we want the internal consistency or coefficient alpha or Cronbach's alpha for x1-x10, the syntax is: 

    			PROC CORR DATA=WORK.ONE ALPHA;
			  VAR X1-X10;
			RUN;
    There are at least three important caveats to consider when computing coefficient alpha.

    Note 1: How to handle "missing" values. In achievement testing, a missing value or a not reached value is traditionally coded as 0 or wrong. The CORR procedure is SAS DOES NOT treating missing as wrong. It is not difficult to write code to force this to happen, but we must write the code. In the above example we could do so as follows: 
    			DATA WORK.ONE;SET WORK.ONE;
			ARRAY X {10} X1-X10;  /* DEFINING AN 
			       ARRAY FOR THE 10 ITEMS */
			DO I=1 TO 10;
			IF X(I) = . THEN X(I) = 0; /* FOR EACH ITEM
			       X1-X10 CHANGING MISSING VALUES (.) TO 0   */
			  END;
			RUN;
    Note 2: The use of the NOMISS option in the CORR procedure. This is related to Note 1 above. Another way to handling missing observations is to use the NOMISS option in the CORR procedure. The syntax is as follows: 
    			PROC CORR DATA=WORK.ONE ALPHA NOMISS;
			  VAR X1-X10;
    The effect of this is to remove all items X1-X10 from analysis for any record where a at least one of these items X1-X10 are missing. Obviously, for achievement testing, especially for speeded tests, where most examines might not be expected to complete all items, this would be a problem. The use of the NOMISS option would restrict the analysis to the subset of examines who did complete all items and this quite often would not be the population of interest when wishing to establish an internal consistency reliability estimate.

    One common approach to resolving this problem might be to define a number of items that must be attempted for the record to be included. Some health status measures, for example the SF-36, have scoring rules that require that at least 50% of the items must be answered for the scale to be defined. If less than half of the items are attempted, then the scale is not interpreted. If the scale is considered valid, by THEIR definition, then all missing values on that scale are replace by the average of the non-missing items on that scale. The SAS code to implement this scoring algorithm is summarized below under the assumption that the scale is has 10 items. 

    			DATA WORK.ONE;SET WORK.ONE;
			 ARRAY X {10} X1-X10;
			       IF NMISS(OF X1-X10) >  5 THEN DO I=1 TO 10;
			         X(I) = .;
			  END;
			  ELSE IF NMISS(OF X1-X10) < = 5 THEN DO I=1 TO 10;
			       IF X(I) =. THEN X(I) = MEAN(OF X1-X10);
			 END;
			RUN;
    Note that replacing all missing values with the average of the non-missing values in the cases where then number of missing values is not greater than half of the total number of items will result in an inflated Cronbach's alpha. A better approach would be to remove from consideration records where fewer than 50% of the records are completed and to leave the remaining records intact, with the missing values still in. In other words, to implement that first IF statement above, but to eliminate the ELSE IF clause and then to run the PROC CORR without the NOMISS option. The bottom line: The NOMISS option in PROC CORR in general, and with the ALPHA options in particular must be considered carefully.

    Note 3: Making sure that all items in the set are coded in the same direction. Although 0/1 (wrong/right) coding is rarely a problem with this, for Likert or other scales with more than 2 points on the scale, it is not uncommon for the scale to remain constant (e.g., Strongly Agree, Agree, Disagree, Strongly Disagree), but for the wording of the questions to reverse the appropriate interpretation of the scale. For example, 
    			Q1.  Social Security System Must be reformed	SA  A  D  SD
			Q2.  Social Security System Remain the Same	SA  A  D  SD
    Clearly, the two questions are on the same scale, but the meanings of the end points opposite.

    In SAS, the way to adjust for this problem is to pick the direction that we want the scale to be coded, that is, do we want SA to be a positive statement about the Social Security System or a negative one, and then reverse scale those items were SA reflects negatively (or positively) about Social Security System. In the above example, SA for Q1 is a negative position relative to the Social Security System and, therefore should be reverse scaled if the decision is to scale so the SA implies positive attitudes.

    If the coding of the 4-point Likert Scale was SA-0, A-1, D-2, SD-3, then the item will be reverse scaled as follows:
    Q1 = 3-Q1, in this way 0 becomes 3-0 = 1; 1 becomes 3-1 = 2; 2 becomes 3-2 = 1; and 3 becomes 3-3 = 0.

    If the coding of the 4-point Likert Scale was SA-1, A-2, D-3, SD-4, then the item will be reverse scaled as follows:
    Q1 = 5-Q1, in this way 1 becomes 5-1 = 4; 2 becomes 5-2 = 3; 3 becomes 5-3 = 2; and 4 becomes 5-4 = 1.

    From the earlier example, If items X1, X3, X5, X7, and X9 would need to be reverse scaled for before computing an internal consistency estimate, then the following SAS code would do the job, Assuming a the 4-point Likert scale illustrated above with 1-4 scoring. 

    			DATA WORK.ONE;SET WORK.ONE; 
			 ARRAY X {10} X1-X10;  
			      /* DEFINING AN ARRAY FOR THE 10 ITEMS */
			  DO I=1,3,5,7,9;           /* INDICATING WHICH ITEMS
			 IN THE ARRAY TO BE REVERSE SCALED  */
			    X(I) = 5-X(I);           /* REVERSE SCALING 
			  FOR 1-4 CODING OF 4-POINT LIKERT SCALE    */
			   END;
			RUN;
    It should be noted that some of the output from PROC CORR with the ALPHA option, such as the correlation of the item with the total and the internal consistency estimate for the scale with each individual item NOT part of the scale provides very useful diagnostics that should alert the researcher about either poorly functioning items or items that were missed when considering reverse scaling. An item that correlated negatively with the total usually needs to be reverse scaled or is poorly formed.

    References and Further Readings:
    Feldt L., and R. Brennan, Reliability, in Educational Measurement, Linn R. (Ed.), 105-146, 1989, Macmillian Publishing Company.
    Garson G., Reliability
    SAS: Useful Material

    Instrumentality Theory

    Suppose two corresponding items, one from the dimension being rated and its mate, the relative importance of that topic, called the "valence", are cross-multiplied, then added up across all such pairs, then divided by the number of such pairs. This procedure provides a weighted score, the sum of the items each weighted by its relative importance. The higher the average weighted score, the greater the overall importance and rating of the topic. The technique has been well-liked since two issues are being considered here, how satisfied or prepared or . . . someone is, and how important that topic is to them. The approach has been applied to multivariate issues such as factors affecting leaving an organization, job satisfaction, managerial behavior, etc.

    References and Further Readings:
    Korn, Graubard, Analysis of Health Surveys, Wiley, 1999.

    Value Measurements Survey Instruments: Rokeach's Value Survey

    Anthropologists have traditionally observed the behavior of members of a specific society and inferred from such behavior the dominant or underlying values of the society. In recent years, however, there has been a gradual shift to measuring values directly by means of survey questionnaire research. Researchers use data collection instruments called value instruments to ask people how they feel about such basic personal and social concepts as freedom, comfort, national security, and peace.

    Research into the relationship between peoples values and their actions as consumers is still in its infancy. However, it is an area that is destined to receive increased attention, for it taps a broad dimension of human behavior that could not be explored effectively before the availability of standardized value instruments.

    A popular value instrument that has been employed in consumer behavior studies in the Rokeach Value Survey (RVS). This self-administered value inventory is divided into two parts, with each part measuring different but complementary types of personal values. The first part consists of eighteen terminal value items, which are designed to measure the relative importance of end- states of existence (i.e. personal goals). The second part consists of eighteen instrumental value items, which measure basic approaches and individual might take to reach end-state values. Thus, the firs half of the measurement instrument deals with ends, while the second half considers means.

    If the items are not reworded to accommodate the Likert format; instead, respondents are asked to indicate the degree of personal importance each RVS value holds, from "very unimportant" to "very important," and then they're given the standard Likert scale next to each RVS value. Some applications use , for example, a 5-point scale and then features a rank-ordering of the top three RVS values after each list of has already been rated, to use in correcting for end-piling. It is show that in many cases, slightly, but not significantly, lower test-retest reliabilities for the Likert versus rank-ordered procedure.

    Since the common reason for preferring to use the RVS in a Likert format is to be able to perform normative statistical tests on the data, it is worthwhile to point out that there are good arguments in favor of using normative statistical tests on RVS data with the scale in its original, rank-ordered format, under some conditions.

    References and Further Readings:
    Braithwaite V., Beyond Rokeach's equality-freedom model: Two dimensional values in a one dimensional world, Journal of Social Issues , 50, 67-94, 1994.
    Gibbins K., and I. Walker, Multiple interpretations of the Rokeach value survey, Journal of Social Psychology, 133, 797-805, 1993.

    Some Interesting and Useful Sites:

    Search and General Resources

    | All In One Search | AAA Search | ASA: American Statistical Association | The International Biometric Society | Institute of Mathematical Statistics|
    | Lecture Hall | How to Study Statistics |Statistical Terms | Statistics Education|
    | All Topics on the Web | All Topics Periodicals | Statistical Resources on the Web| Statistical Resources on the Web|
    Teaching Resources | Statistics and OR Resources | Statistical Resources | Statistics Archive|
    | World Wide Resources | Virtual Library | Books and Journal | OnLine Text Books | ASA Washington D.C. Chapter|
    | Online Statistical Textbooks | Statistical Page Resources | Statistical Resources | Psychometrics | StatServ |Argus Clearinghouse|
    | Job Listing | Careers in Statistics | Conferences |
    | Statistical Computing Journals | Rainer's Home Page for Statisticians | Statistics Around the world | International Statistical Institute|

    Demo's and Interactive

    | Coin Flipping | Equation Solver by Newton's Method | Probability Lessons | Real Analysis | Java Applets | XploRe|
    | ANOVA Applet | Outliers and Regression Line | Confidence Interval | Let's Make a Deal|
    | Histogram | Central Limit Theorem | Power of T-test | Computer-based Learning Statistics|
    | Learning Statistics by Demo | Normal Approx. to binomial | Small Sample Size Effect | Statistics Demonstrations|
    |Normal Curve Area |Critical Values for the t-Distribution | Critical Values for the F-Distribution | Critical Values for the Chi- square Distribution|
    | Tests for Binomial | Chi-square Goodness-of-fit| Confidence Intervals on a Proportion| Confidence Intervals on a Correlation| Confidence Intervals on a Mean| Confidence Intervals on the Difference of Two Independent Means|
    | T-test Paired | Statistical Calculators| Test of Two Proportions| Mean| Two Independent Means| Two Dependent Means| Median Test| Sign Test| McNemar's Test| Wilcoxon Matched-Pairs Test| Wilcoxon Test |Randomization Plans|
    Rank Correlation Coefficient| Correlation Coefficient| Comparing Correlation Coefficients|

    Survey Analysis

    | Resources for Methods in Evaluation and Social Research | Social Research and Statistical Links | Survey Analysis Software | ASA Survey Research | American Association for Public Opinion Research | Association for Survey Computing|
    | Applied Social Surveys| General Social Survey | Bureau of the Census | Social Surveys Question Bank | National Mapping Information |On-line Survey Response|

    Statistical Software


    | StatiBot | Analyse-it |Autobox |BMDP |Data Desk |Lisp-Stat |Matlab |Minitab |SAS |SPSS |Stata | Biostatistical Software Routines | Ecological Data Analysis |Multiple Imputation |Statistica |ASSUME |The Spreadsheet page |Add-ins for Excel. | Data analysis and statistical solutions for Excel|
    | Choosing a Statistical Analysis Package | Statistical Software Review | Statistical Software Providers | Visual Statistics System | WebStat | QDStat | Statistical Calculators on Web | Modstat | The AssiStat | XLStatistics|

    Data and Data Analysis


    | Economic Data and Links | Agricultural Statistics | The Gallup Organization | The Data and Story Library | Dr. B's Data | Datasets | Research Resources|
    | Forum on Applied Data Analysis | Multivariate Analysis | Multivariate Statistics | Statistical data analysis FAQ| | Data Analysis | Cast Your Vote!|
    | National Center for Health Statistics (NCHS) | FEDSTAT: USA | Economic Data sources|
    |Computer Routines and Data Files | Globally Accessible Statistical Procedures| Data Mining and Knowledge Discovery | USDA Economics and Statistics System|
    | Brixton Books: Epi Info | Software FAQs| Experimental Interactive Statistics | Discrete Genetic Data Analysis | Software Providers | Software Metrics

    Statistics and Probability


    |Test for Randomness |Confidence Intervals |ANOVA in Detail | The Probability Web | Business Statistics | Against All Odds|
    | A New View of Statistics | Statistics Homepage |MathForum | SIIP | Introductory Statistics | Cases | HyperStat | Introduction to Statistics|
    | Descriptive Statistics | Statistics Lab | Linear Regression | K-12 Statistics | Statistics Sites|
    | Statistics Links | Use and abuse of statistics | Statistics on Web | Statistical Links | Statistics Handouts|
    | Basic Statistics | SurfStat | Design Research | Statistics Mailing Lists | Statistics Links | Introduction to Statistics | Topics in Statistics | Statistics Refresher | World Lecture Hall | Statistics Online | Lecture Notes | Conceptual Statistics|
    | SimStat Package | Ordination Methods | Glossary for Ordination Methods | Guide for Statisticians | Power Analysis: How many samples? | Statistics for Everyone | American Risk and Insurance Association | Multilevel Models| | International Association For Statistical Computing | Risk Theory Society | StatLib | Statistical Education through Problem Solving|
    | AllStat | Institute of Mathematical Statistics | Statistics and Social Sciences | An Introduction to Geostatistics | Annotated Bibliography of Articles for the Statistics User|
    | Bibliography for Computational Probability and Statistics
    | Probability Archive | Probability Resources|
    A selection of:
    | BUBL Catalogue| Computational Probability| Connected University| CTI Statistics| Links2Go|
    McGraw-Hill| |Math Forum Phone-soft Cyber-world| Physical Sciences| Probability and Statistics| Quantitative Usability Methods: Surveys|
    Search Engines Directory
    | AltaVista| AOL| Dogpile| Excite| HotBot| Looksmart| Lycos| Netscape| NetFirst| OpenDirectory| OpenHere| Webcrawler| Yahoo|
    | Second Moment| Smallshop| Suite101| World Lecture Hall|

    Back to SySurvey Whitepapers