Sampling For Surveys

Sampling is the key to survey research. No matter how well a study is done in other ways, if the sample has not been properly found, the results cannot be regarded as correct. Though this chapter may be more difficult than the others, but it is perhaps the most important chapter in this book. It applies mainly to surveys, but is also important for planning other types of research.

1. Populations

The first concept you need to understand is the difference between a population and a sample.

To make a sample, you first need a population. In non-technical language, population means "the number of people living in an area." This meaning of population is also used in survey research, but this is only one of many possible definitions of population. The word universe is sometimes used in survey research, and means exactly the same in this context as population.

The unit of population is whatever you are counting: there can be a population of people, a population of households, a population of events, institutions, transactions, and so forth. Anything you can count can be a population unit. But if you can’t get information from it, and you can’t measure it in some way, it’s not a unit of population that is suitable for survey research.

For a survey, various limits (geographical and otherwise) can be placed on a population. Some populations that could be covered by surveys are...

• All people living in Cambodia.
• All people aged 18 and over.
• All households in Hanoi.
• All schools in Australia.
• All instances of tuning in to a radio station in the last seven days

...and so on. If you can express it in a phrase beginning "All," and you can count it, it’s a population of some kind. The commonest kind of population used in survey research uses the formula:

• All people aged X years and over, who live in area Y.

The "X years and over" criterion usually rules out children below a certain age, both because of the difficulties involved in interviewing them and the lack of relevance of their answers to many research questions.

Even though some populations can’t be questioned directly, they’re still populations. For example, schools can’t fill in questionnaires, but somebody can do so on behalf of each school. The distinction is important when finding the answers to questions like "What proportion of Samoan schools have libraries?" You need only one questionnaire from each school - not one from each teacher, or one from each student.

Often, the population you end up surveying is not the population you really wanted, because some part of the population cannot be surveyed. For example, if you want to survey opinions among the whole population of an area, and choose to do the survey by telephoning people at home, the population you actually survey will be people with a telephone in their home. If the people with no telephone have different opinions, you will not discover this.

As long as the surveyed population is a high proportion of the wanted population, the results obtained should also be true for the larger population. For example, if 90% of homes have a telephone, the 10% without a phone would have to be very different, for the survey's results not to be true for the whole population.

2. Sampling frames

A sampling frame can be one of two things: either a list of all members of a population, or a method of selecting any member of the population. The term general population refers to everybody in a particular geographical area. Common sampling frames for the general population are electoral rolls, street directories, telephone directories, and customer lists from utilities which are used by almost all households: water, electricity, sewerage, and so on.

It is best to use the list that is most accurate, most complete, and most up to date, but this differs from country to country. Some of these lists are of households, others are of people. For most surveys, a list of households (specially if it is in street order) is more useful than a list of people. Another commonly used sampling frame (which I do not recommend) is a map.

3. Samples

A sample is a part of the population from which it was drawn. Survey research is based on sampling, which involves getting information from only some members of the population.

If information is obtained from the whole population, it’s not a sample, but a census. Some surveys, based on very small populations (such as all members of an organization) in fact are censuses and not sample surveys. When you do a census, the techniques given in this book still apply, but there is no sampling error - as long as the whole group participates in the census.

Samples can be drawn in several different ways, such as probability samples, quota samples, purposive samples, and volunteer samples.

Sometimes known as random samples, probability samples are the most accurate of all. It is only with a probability sample that it’s possible to accurately estimate how different the sample is from the whole population. With a probability sample, every member of the population has an equal (or known) chance of being included in the sample. In most professional surveys, each member of the population has the same chance of being included in the sample, but sometimes certain types of people are deliberately over-represented in the sample. Results are calculated compensate for the sample imbalance.

With a probability sample, the first step is usually to try to find a sampling frame: a list of all members of the population. Using this list, individuals or households are numbered, and some numbers are chosen at random to determine who is surveyed. If no population list is available, other methods are used to ensure that every population member has an equal (or other known) chance of inclusion in the survey.

In the early days of survey research, quota sampling was very common. No population list is used, but a quota, usually based on Census data, is drawn up.

For example, suppose the general population is being surveyed, and 50% of them are known to be male, and half of each sex is aged over 40. If each interviewer had to obtain 20 interviews, she or he would be told to interview 10 men and 10 women, 5 of each aged under 40, and 5 of each aged 40-plus. It is usually the interviewers who decide how where they find the respondents. In this case, age and sex are referred to as control variables.

A problem with quota samples is that some respondents are easier to find than others. The interviewer in the previous example may have quickly found 10 women, and 5 men over 40, but may then have taken a lot of time finding men under 40. If too many control variables are used, interviewers will waste a lot of time trying to find respondents to fit particular categories. For example (if interviews had been specified in terms of occupation and household size, as well as age and sex) "2 male butchers aged 40 to 44, living in households of 8 or more people".

It’s important with quota sampling to use appropriate control variables. If some people in a category are more likely to take part in the survey than others, and also likely to give different answers from those in another category, then that category should be a control variable.

For example, if women are more willing than men to be surveyed (which is often true) and if the two sexes’ patterns of answers are expected to be quite different, then the quota design should obtain balanced numbers from each sex. In fact, sex and age group are the two commonest control variables in quota surveys, but occasionally a different variable can be the most relevant. If you’re planning a quota sample, don’t assume that by getting the right proportion in each age group for each sex, everything else will be OK.

Pure quota samples are little used today, except for surveys done in public places, but sometimes partial quota sampling can be useful. A common example is when choosing one respondent from a household. The probability method begins by finding out how many people live in the household, then selecting an interviewee purely at random. There are practical problems with this approach (explained later in this chapter), so inside randomly selected households, quota sampling is often used.

Samples of volunteers should generally be treated with suspicion. However, as all survey research involves some element of volunteering, there is no fixed line between a volunteer sample and a probability sample. The main difference between a pure volunteer sample and a probability sample of volunteers is that in the former case, volunteers make all the effort; no sampling frame is used.

The main source of problems with volunteer samples is the proportion who volunteer. If too few of the population volunteer for the survey, you must wonder what was so special about them. There is usually no way of finding out how those who volunteered are different from those who didn’t. But if the whole population volunteer to take part in the survey, there’s no problem.

In some circumstances, volunteer samples can be useful. For example, the Australian Broadcasting Corporation used to survey panels of listeners to its two serious radio networks, Radio National and Classic FM. To recruit people for the panels, the networks broadcast advertisements, asking those interested to contact the ABC.

To check whether these volunteers were representative of all listeners to the networks, we carried out of random surveys of listeners to these networks, and compared the answers of panel members and randomly selected listeners. We found that the volunteers were representative in most ways, but listened to the station much more often.

As only about 5% of the population listen regularly to these networks, random surveys are very expensive to conduct (20 households must be contacted to find each listener), so using volunteer samples saves a lot of money. Not all people who volunteer for panels need be accepted. A quota system can be used, to ensure that various parts of the population are accurately represented.

When people who know nothing about sampling organize surveys, they often have a large number of questionnaires printed, and offer one to everybody who’s interested. Amateur researchers often seem to feel that if the number of questionnaires returned is large enough, the lack of a sample design isn’t important. Certainly, you will get some results, but you will have no way of knowing how representative the respondents are of the population. You may not even know what the population is, with this method. The less effort that goes into distributing questionnaires to particular individuals and convincing them that participation is worthwhile, the more likely it is that those who complete and return questionnaires will be a very small (and probably not typical) section of the population.

About the only way in which a volunteer sample can produce accurate results (without being checked against a probability sample), is if a high proportion of the population voluntarily returns questionnaires. I’ve known this to work a few times, usually in country areas with a small population, where up to about 50% of all households have returned questionnaires. Even so, if all the effort is left to the respondents, there’s no certainty that somebody who wants to distort the results has not filled in hundreds of questionnaires.

The same problems apply to drawing conclusions from unsolicited mail and phone calls. For example, politicians sometimes make claims like "My mail is running five to one in favour of the stand I made last week." There are all sorts of reasons why the letters sent in may not be representative of the population. The same applies to letters sent to broadcasting organizations: all these tell you is the opinions of the letter-writers. It is only when the majority of listeners write letters that the opinions expressed in these letters might be representative.

A purposive sample is one in which a surveyor tries to create a representative sample without sampling at random.

One of the commonest uses of purposive sampling is in selecting a group of geographical areas to represent a larger area. For example, door-to-door interviewing can become extremely expensive in rural areas with a low population density. In a country such as Cambodia, it is not feasible to do a door-to-door survey covering the whole country. Though areas could be picked purely at random, if the budget was small and only a small number of towns and cities could be included, you might choose these in a purposive way, perhaps ensuring that different types of town were included. However, there are better ways to do this. Read on…

A maximum-diversity sample is a special kind of purposive sample. Normally, a purposive sample is not representative, and does not claim to be. A maximum-diversity sample aims to be more representative than a random sample (which, despite what many people think, is not always the most representative, specially when the sample size is small).

Instead of seeking representativeness through equal probability, it’s sought by including a wide range of extremes. This is an extension of the statistical principle of regression towards the mean - in other words, if a group of people that is (on average) extreme in some way, it will contain some people who themselves are average. So if you sought a "minimum diversity" sample by only trying to cover the types of people who you thought were average, you’d be likely to miss out on a number of different groups which might make up quite a high proportion of the population. But by seeking maximum diversity, average people are automatically included.

When you are selecting a multi-stage sample (explained in more detail below) the first stage might be to draw a sample of districts in the whole country. If this number is less than about 30, it’s likely that the sample will be unrepresentative in some ways. Two solutions to this are stratification (also explained below) and maximum-diversity sampling. For both of these, some local knowledge is needed.

With maximum-diversity sampling, you try to include all the extremes in the population. This method is normally used to choose no more than about 30 units. For example, in a small village, you might decide to interview 10 people. If this was a radio audience survey, you could ask to interview

• the oldest person in the village who listens to radio
• the oldest who does not listen to radio
• the youngest who listens to radio
• a man who listens to radio all day
• a woman who listens to radio all day
• somebody who has never listened to radio in his or her life
• the person with the most radios (a repairman, perhaps)
• the person with the biggest aerial
• a person who is thought to be completely average in all ways

...and so on. The principle is that if you deliberately try to interview a very different selection of people, their aggregate answers will be close to the average. The method sounds odd, but works well in places where a random sample cannot be drawn. And of course it only works when information about the different kinds of sample unit (e.g. a person) is widely known.

When you are planning a door to door survey, it is tempting to use a map as the basis for sampling. To get 100 starting points for clusters, all you need to do is throw 100 darts at the map.

This method, if properly done, gives every unit of area on the map an equal chance of being surveyed. This would be valid only if your unit of measurement was a unit of land area — for example, if you were estimating the distribution of a plant species. If you are surveying people or households, this equal-area method will over-represent farmers and people living on large properties. People living in high-density urban areas will be greatly under-represented. Even within a small urban area, large differences in density can exist.

Slightly better, but still badly flawed, is a method used in the 1980s by a Sydney research company. This was based on a street directory, and gave every street an equal chance of being surveyed. The trouble was that streets ranged in size from the Pacific Highway (with thousands of addresses on it) to cul-de-sacs with only one or two dwellings. There would have been no problem, however, if there were no consistent difference between long streets and short streets. However, in Sydney, long streets tend to have many blocks of flats, while short streets tend to have single-unit houses. The people living on long streets tend to be poorer and more transient than others, and include fewer families with children.

Perhaps you have a list of names and addresses of some of your audience, collected for a marketing purpose. This is known as a found sample or convenience sample. It’s tempting to survey these people, because it seems so easy. But don’t do it! You have no way of knowing how representative such a sample is. You can certainly get a result, but you won’t know to what extent that result is true of people who were not included in the sample.

If you’re researching a rare population, sometimes the only feasible way to find its members is by asking others. First of all, you somehow have to find a few members of the population - by any method you can. That is the first round.

You now ask each of these first-round members if they know of any others. These names form the second round.

Then you go to each of those second-round people, and ask them for more names.

Keep repeating the process, for several more rounds. The important thing is knowing when to stop. For each round, keep a count of the number of names you get, and also the number of new names - people you haven’t heard about before. Calculate the number of new names as a percentage of the total number of names. For example, if one round gives you 50 names, but 20 are for people who were mentioned in earlier rounds, the percentage of new names for that round is 60%. You’ll probably find that the percentage of new names rises at first, then drops sharply. When you start hearing about the same people over and over again, it's time to stop - perhaps when the percentage of new names drops to around 10%. This is often at the fourth or fifth round.

You now have something close to a list of the whole population (and many of them will know that you’re planning some research). Using that list, you can now draw a systematic sample, as detailed in section 9 of this chapter.

Snowball sampling requires a lot of work, if the population is large, because you need to draw up an almost-complete list of the population. So this method works best when a population is very small. But if the population is small enough to list every member without a huge amount of work, you could do a census, rather than a sample: in other words, contact all of them.

Snowball sampling works well when members of a population know the other members. For example, if you are studying people who speak a minority language, or who share some disability, there’s a good chance that most of them know of each other. The biggest problem with snowball sampling is that isolated people, who are not known to other members of the population, will not be included in your study, because you’ll never find out about them. In the case of minorities, sometimes the more successful members will blend into the ruling culture, feeling no need to communicate with other members of that minority. So if you survey only the ones who know each other, you may get a false impression. A partial solution to this is to begin with a telephone directory or other population list. If people in that population have some distinctive family names, you can find them in the directory, and take those people for the first round.

The simplest type of sampling involves drawing one sample from the whole survey area. If the coverage area of a radio station is a large town and its surrounding countryside, there may be a population list that covers the whole area - an electoral roll, perhaps. If you want to select (say) 40 random addresses as starting points for a door-to-door cluster survey, you could simply pick 40 addresses from the population list.

This is simple, but there’s a slight danger that all 40 addresses may be in the same part of the coverage area. This happened to me once, when I planned a survey in Timaru in New Zealand. We drew 20 addresses as starting points, then plotted them on a map. Unfortunately, they were all in one quarter of the town. I considered throwing out the sample, and selecting another 20 addresses. But what if the same imbalance occurred again?

The solution was to stratify the sample. Using census data from small areas, Timaru was divided into four quarters, with almost exactly equal populations. We then selected 5 addresses in each quarter. This way, we were certain that the clusters would be spread evenly across the town.

Stratification is easy to do, and you should use it whenever possible. But for it to be possible, you need to have (a) census data about smaller parts of the whole survey area, and (b) some way of selecting the sample within each small area. For example, if you were using a telephone directory as a sampling frame, each residential listing might show the suburb where that number was. (It doesn’t matter if the person mentioned in the listing still lives there - you use a telephone directory as a list of addresses, not people.) In this case, you’d need census data on the number of households in each suburb, to be able to use stratification effectively.

The principle of stratification is simply that, if an area has X% of the population, it should also have X% of the interviews.

Here’s an example of a stratified sample design for the Amhara Region of Ethiopia, from a survey I helped to organize there. The Amhara Region is divided into 11 zones. For each zone, you find out the population, work out what percentage it is of the total, then make sure that the number of clusters is as close as possible to those proportions. Produce a table laid out like this. You begin with the numbers shown in bold type, and calculate the rest.

Calculate the table as follows. For each zone:

Figure in column B = Figure in column A / the total of A x 100.

E.g. Column B for North Gondar = 2089 / 13835 x 100 = 15.1

Column C = B x total of C

Column D = same as C, but rounded to nearest whole number

Column E = D / total of D

As North Gondar has 15.1% of the population, it should also have 15.1% of the clusters.

But 15.1% of 45 clusters is 6.8, and you can't have 0.8 of a cluster. So the number of clusters in North Gondar is rounded up to 7.

This process is repeated for each other zone. Sometimes, because of the rounding, the total number of clusters is 1 more or less than the total you planned for. To fix this, you can change the final number of clusters, adding or subtracting 1. Another solution is to cheat a little, by rounding an exact number of clusters in the wrong direction: in the above table you could round 1.48 up to 2, or 5.54 down to 5, with very little effect on the accuracy of the proportions. When you round a figure in the wrong direction, choose the number ending in the closest figure to .5

You could also add a column F: the difference between B and E. The maximum difference depends on the number of clusters, but should usually be less than 2%. If the difference is too large, you may need to have more clusters, with fewer interviews in each.

In the above table, there's a problem with the Bahir Dar zone. The population there was only 96,000, so this zone needed 0.32 of a cluster. This is rounded down to 0, so there are no clusters in that zone, and therefore no interviews in that zone.

However, this makes nonsense of stratification: what do you do about it?

There are three solutions:

(1) If an area would have no interviews, combine it with an adjoining area. As Bahir Dar is inside West Gojam, these two zones could be combined. The exact number of clusters would be 6.12 (5.80 + 0.32), which in this case still rounds down to 6. This is normally the best solution.

(2) Round the 0.32 upwards instead of downwards, and include one cluster in Bahir Dar. The total number of clusters would then be 46 - but people living in Bahir Dar would be over-represented in the survey: 2.2% of the clusters (1 in 46), but 0.7% of the population.

(3) Change the cluster size in Bahir Dar. Instead of using clusters of 8 households (as elsewhere), you could do a single cluster of 4 households. So now there would be 45.5 clusters, and Bahir Dar, with 0.7% of the population, would have 1.1% of the interviews.

That's close enough. However, having different cluster sizes often confuses the interviewers, so you'd need two slightly different sets of interviewer instructions.

With door-to-door surveys, sampling is done in several steps. Often, the first step is stratification. For example, census data can be used to select which districts in the survey area will be included. In the second step, random sampling could be used, but each district might need to be treated separately, depending on the information available there. This would decide which households would be surveyed. The third step would involve sampling individuals within households, perhaps using quota sampling.

4. Random sampling

Before we discuss random sampling, you need to be clear about the exact meaning of "random." In common speech, it means "anything will do", but the meaning used in statistics is much more precise: a person is chosen at random from a population when every member of that population has the same chance of being sampled. If some people have a higher chance than others, the selection is not random. To maximize accuracy, surveys conducted on scientific principles always use random samples.

Imagine a complete list of the population, with an entry for every member: for example, a list of 1500 members of an organization, numbered from 1 up to 1500. Suppose you want to survey 100 of them. To draw a simple random sample, choose 100 different random numbers, between 1 and 1500. Any member whose number is chosen will be surveyed. If the same number comes up twice, the second occurrence is ignored, as nobody will be surveyed more than once. So if the method for selecting random numbers can produce the same number twice, about 110 selections will need to be made to get 100 people.

Another type of random sampling, called systematic sampling, is more commonly used. This ensures that no number will come up twice. No matter how many thousands of people you will interview, you need only one random number for systematic sampling.

In the above example, you are surveying 1 member in 15. Think of the population as divided into 100 groups, each with 15 people. You need to choose one person from each group, so you choose a random number between 1 and 15. Let’s say this number is 7. You then choose the 7th person in each group. If the members were numbered 1-15 in the first group, 16-30 in the second, 31-45 in the third, and so on, you'd interview people with numbers 7, 22, and 37 - adding 15 each time. Exactly 100 members would be chosen for the survey, and their numbers would be evenly spread through the membership list.

The commonest source of random numbers in most countries is the serial numbers on banknotes. There can be no bias in using the last few digits of the first banknote you happen to pull out of your pocket, because there should be an equal chance of drawing each possible combination of numbers. Other source of unpredictable large numbers (from which you can use the last few digits) include lottery results, public transport tickets, even stock market indexes.

You can also cheat. With systematic sampling, only one random number is needed. Just think of a number, between 1 and the upper limit. Though statisticians would frown, it will probably make no difference to the results.

The essential principle in survey research is that everybody in the population to be surveyed should have an equal chance of being questioned. If you do a survey, and everybody had an equal chance of inclusion, you’re in a position to estimate the accuracy of your results.

Every survey has sampling variation. If you survey 100 people, and get a certain result, this result will be slightly different than if you had surveyed another group of 100 people. This is like tossing coins: if you toss a coin 100 times, you know that there should be 50 heads and 50 tails. But the chances are quite strong (92 in 100, to be exact) that you won’t get exactly 50 heads and 50 tails. However, the chances of getting 0 heads and 100 tails are practically nonexistent.

Using statistical techniques, it’s possible to work out the exact chances of every possible combination of heads and tails. For example, there are 680 chances in 1000 that you’ll get between 45 and 55 heads in 100 throws. (If you doubt this, find 100 coins, throw them 1000 times, and see the result for yourself!)

In the same way, even though you know the results from a survey are not exactly accurate, they are probably pretty close — but only if every member of the surveyed population had an equal chance of being included in the survey.

To estimate how much sampling error there is likely to be in a survey result, use the following table. "Standard error" means (roughly) the average difference between the true figure and each case.

When using the above table, think of each question as having two possible answers. Although a question may have more than two answers (e.g. age groups of under 25, 25 to 44, and 45 or over), the number can always be reduced to two, conceptually.

For example, suppose 20% of a sample is in the 25 to 44 group. Therefore, the other 80% is in the "not 25 to 44" age group. The margin of error on this 20/80 split is 4%, so the true population figure is likely to be anywhere between 16% and 24%. There is one chance in three that it will be outside this range, and 1 chance in 20 that it be outside twice this range: i.e. less than 12 or more than 28%.

If all that sounds too difficult, just assume that the margin of error is 5%, on any result. For example, if a survey finds that 25% of the population listen to your station, it's likely that the true figure will be somewhere between 20% and 30%. (Likely - but not certain - because there's a small chance that the true figure could be less than 20% or more than 30%. A well-known saying among statisticians is "statistics means never having to say you’re certain.")

Always remember that the above table shows only sampling error, which is fairly predictable. There could also be other, unpredictable, sources of error.

Note in the above table that the margin of error for 400 interviews is always half that for 100. This means that to halve the error in a survey, you must quadruple the sample size. So unless you have a huge budget, you must learn to tolerate sampling error.

5. Choosing a sample size

There are several ways to choose a sample size: you can either calculate it from a formula, or use a rough "rule of thumb."

The formula for calculating the sampling error to a survey question is:

n = p x q / SE²
where:
n is the sample size: the number of people interviewed.
p is the percentage answering Yes to the question.
q is the percentage not answering Yes to the question.
SE is the standard error as shown in the table above.

This formula (which I have over-simplified slightly), is useful in working out how big a sample size you need for a given survey. But to calculate the sample size you first have to know roughly how many people will answer Yes to the question, and also decide how large a standard error you can tolerate. For beginners, this is not simple. Another problem is that samples calculated in this way can be horrifyingly large. For example, if you changed the tolerance from 3% to 1% in the above example, you’d have to interview 1875 people. Yet another problem is that every question in a survey may require a different sample size.

In an ideal world, you’d calculate the sample size for a survey as shown above, and cost would never be a problem. However, as most surveys are done to a budget, your starting point in practice may not be how much error you can tolerate, but rather how little error you can get for a given cost.

To do this, you need to divide the cost of the survey into two parts:

• a fixed part, whose cost is not proportional to sample size, and
• a variable part, for which the cost is so much per member of the sample.

Once you have allocated a proportion of the total budget to the fixed cost, and estimated the cost of getting back each completed questionnaire, you can calculate the affordable sample size.

But what if you don’t know the survey cost, and have to recommend a sample size? This is where the rule-of-thumb is useful.

For the majority of surveys, the sample size is between 200 and 2000. A sample below 200 is useful only if you have a very low budget, and little or no information on what proportion of the population engages in the activity of most interest to you — or if the entire population is not much larger than that. A sample size over 2000 is probably a waste of time and money, unless there are subgroups of the population, which must be studied in detail.

If you don’t absolutely need such large numbers, and have more funds than you need, don’t spend it on increasing the sample size beyond the normal level. Instead, spend it on improving the quality of the work: more interviewer training, more detailed supervision, more verification, and more pre-testing. Better still, do two surveys: a small one first, to get some idea of the data, then a larger one. With the experience you gain on the first survey, the second one will be of higher quality.

The sample size also depends on how much you know about the subject in question. If you have no information at all on a subject, a sample of only 100 can be quite useful, though its standard error is large.

Are you confused about which sample size to choose? Try my rule of thumb:

Consider this question: if a survey in a town with 10,000 people needs a sample of 400 for a given level of accuracy, what sample size would you need for the same level of accuracy in the whole country, with a population of 10,000,000? (That's 1000 times the population of the town.)

Did you guess 400,000? Many people do. The correct answer is 400.4 - you might as well call it 400.

The formula I gave above isn't quite complete. The full version has what's called the finite population correction (or FPC) added to the end, so the full formula is:

n = p x q / SE² x (N-n)/N

where N is the population. Unless the sample size is more than about 5% of the population, the (N-n)/N bit (the FPC) makes almost no difference to the required sample size.

Is that too technical? Think of it another way. Imagine that you have a bowl of soup. You don’t know what flavour it is. So you stir the soup in the bowl, take a spoonful, and sip it. The bowl of soup is the population, and the spoonful is the sample. As long as the bowl is well-stirred (so that each spoonful is a random sample), the size of the bowl is irrelevant. If the bowl was twice the size, you wouldn’t need to take two spoonfuls to assess the flavour: one spoonful would still be fine. This is equally true for human populations.

6. Nonrandom sampling

Though random sampling is the ideal, sometimes it’s not possible. In some countries, census information is either not available, or so far out of date that it’s useless. Even when good census data exists, there may be no maps showing the boundaries of the areas to which the data applies. And even when there exist both good census data and related maps, there may be no sampling frames.

The good news (from a sampling point of view) is that these conditions usually apply in very poor and undeveloped countries with large rural populations. In my experience, there’s not a wide range of variation in these populations. This is a difficult thing to prove, but I suspect that the more developed a country, the more differences there are between its citizens. All this is a way of saying that where random sampling is not possible, perhaps it’s not so necessary.

The best solution I can think of is to use maximum diversity sampling, described briefly in section 3 of this chapter.

Maximum-diversity samples are normally drawn in several stages, so they are multi-stage samples. The first stage is to decide which parts of the population area will be surveyed. For example, if a survey is to represent a whole province, and it’s not feasible to survey every part of the province, you must decide which parts of the province will be included. Let’s assume that these parts are called counties, and you will need to select some of these.

Maximum-diversity sampling works like this:

Stage 1

1. Think of all the ways in which the counties differ from the province as a whole -specially ways relevant to the subject of the survey. If the survey is about FM radio, and some areas are hilly, reception may be poorer there. If the survey is about malaria, and some counties have large swamps with a lot of mosquitoes, that will be a factor. If the survey will be related to wealth or education levels (as many surveys are), try to find out which counties have the richest and best-educated people, and which have the poorest and least-educated. Try to think of about 10 factors, which are relevant to the survey.

2. Try to find objective data about these factors. Failing that, try to find experts on the topics, or people who have travelled around the whole province. Using this information, for each factor make a list of the counties which have a high level of the factor (e.g. lots of mountains, lots of swamps, wealthy) and counties which have a low level (e.g. all flat, no swamps, poor).

3. The counties mentioned most often in these lists of extremes should be included in the survey. Mark these counties on a map of the province. Has any large area been omitted? If so, add another county, which is as far as possible from all the others mentioned.

Stage 2

When the counties (or whatever they are called) have been chosen, the next stage is to work out where in each county the cluster should be chosen. Continue the maximum-diversity principle by using the same principle in each country as in stage 1. If a county was chosen for its swampiness and flatness, choose the flattest and swampiest area in the country. If it was chosen for its mountains and wealth, choose a wealthy mountainous area.

To find out where these areas are, you will probably need to travel to that county and speak to local officials. Sometimes you then find that there are local population lists - e.g. lists of all houses in the area. In that case, you might be able to use random sampling for the final stage. If there are no population lists you can use, the surveyed households will have to be chosen by block listing, aerial photographs, or radial sampling - see section ii for details of these methods.

Maximum diversity sampling can produce samples that are as representative as random samples. The problem is that you can never be sure of this.

7. Choosing the sampling unit

Now you need to choose your sampling unit: what will you sample? It seems obvious at first: your sample will be people, because only people can be interviewed.

In fact, it’s not that simple, specially with door to door surveys. Most door to door surveys begin by sampling dwellings. (A dwelling is the place where the household lives: households are people, dwellings are homes.) Dwellings are easier to find than people: they don’t move around. Even if you make your initial sample from a list of people, such as an electoral roll, you’ll find that some people have moved since the list was compiled. It’s much easier to sample dwellings, and then, as a second stage, interview the people who live in those selected dwellings.

Sometimes it’s more appropriate to sample households than people. For example, a few years ago I organized an Australia survey about media usage. Part of this survey asked about the types of media equipment that were available in households. In each household, the interviewers asked for the person who knew most about technology. This person was then asked questions such as "How many radios in this household can receive FM programs?" The average numbers reported in the survey were then applied to the whole population of Australian households. We were able to make statements such as "there are between 29 and 31 million FM radios in Australia."

When the sampling unit is people, some parts of the population are usually excluded. Usually, children below some minimum age are excluded - because they don’t do the activity the survey deals with (e.g. reading newspapers), and also because interviews with children must be done differently. Normal questionnaires are usually too difficult for them. Depending on the subject of the survey, the minimum age is usually between 10 and 18 - most commonly, 15. Children under 10 seldom listen to radio, or read newspapers, so there’s no problem excluding them if this is the subject of your survey. But children as young as 2 watch TV, so any TV survey that does not involve young viewers will be incomplete. The best solution is usually to survey only people aged 10 or over, acknowledge the lack of data from younger viewers, and to do a separate study among children aged under 10, using observation instead of questionnaires.

Door-to-door surveys usually exclude people who don’t live in private households: visitors in hotels, troops in barracks, homeless people, and so on. These people are usually only a few percent of the population, so excluding them makes very little difference to the survey results. For any proposed door-to-door survey, you should try to find out how many people you will not be able to reach, and whether these people are likely to give different answers from the others.

In the 1980s, an Australian government department did a telephone survey with teenagers, and found a surprisingly low rate of unemployment - because it mainly reached teenagers who were living with their parents, in households rich enough to have telephones. At the time, only 10% of households had no telephones - but these were the poorest households.

8. Selecting samples from lists

If you have a complete list of all people in the population, with addresses, you can use this to draw a sample.

When population lists are available, they are often for specific populations, such as all people who work in a particular organization. Even when a list is supposed to contain the entire population, it usually doesn’t: perhaps because it’s out of date, or because certain types of people are excluded. Here are some lists that usually claim to apply to an entire population. Some of these populations are people, some are households, and some can be used as both.

Though an electoral roll is designed as a sample of people, it can be used as a sample of households. People may come and go, but addresses stay the same.

In most countries, electoral rolls are not very complete. I made a study in South Australia, around 1990, and found that approximately 20% of people were not on the electoral rolls at their current address. Some were not citizens, some had not bothered to enrol, and some (e.g. police and judges) were excluded from published rolls. Also, many people had moved in the several years since the rolls were last updated. And of course, electoral rolls always exclude people below the minimum voting age - though children over 10 can usually answer survey questions well.

In some countries, such as Britain, the situation is much better, because electoral rolls are updated every year, and printed in a very convenient street order (unlike the alphabetical order of surname used in Australia).

Before you use an electoral roll to represent the entire public, I suggest you take a small geographical area - perhaps a few street blocks, or a few hundred dwellings - and check how many people in that area are on the rolls. If the figure is less than 90%, look for a better population list.

A good compromise with electoral rolls is to use them with multi-stage sampling: i.e. carry out a cluster survey, and use electoral rolls only to choose the starting point for each cluster. If the sample is stratified, and the number of clusters in each small area is proportional to the population of that area, this helps to ensure that people living in areas where many are not on the electoral roll will still be included in the survey.

These are lists of addresses, in street order. Typically, you first look up a locality, and find all the streets in that area, listed in alphabetical order. For each street, addresses are listed in numerical order. Where up-to-date street directories exist, they are an excellent source of addresses for door-to-door surveys. But they are often incomplete, omitting many addresses - particularly where several dwellings share one street address.

A few years ago in rich countries, telephone directories were an excellent population list with one (and only one) entry for every household. In Australia in the late 1980s, over 90% of households had a telephone, few households were unlisted with "silent numbers", business numbers were clearly separated from residential numbers, few households had more than one number or answering machines, and few people had mobile phones.

But now, it’s all a mess. A telephone directory is no longer a good population list. Many households have more than one phone number, and these are usually the wealthier households. Other households have only mobile phones, which are not usually listed in printed directories. Because mobile phones are carried around, any directory of mobile phone numbers will be a sampling frame of people, not of households.

See the chapter on telephone surveys for full instructions on how to draw samples from telephone directories.

The advantage of a telephone directory, in an area where nearly everybody has a phone at home, is that it is a readily available list which includes most dwellings. But other such lists often exist. In areas where nearly every household has electricity from the mains, you can sometimes get access to a list of electrical subscribers. Unlike telephone directories, which are becoming messier by the day - with households having multiple numbers, unlisted numbers, and home-business numbers, utility lists mention each household once and only once.

Other utilities which can be used are those for services which almost all households use - such as local government, water supply, sewerage, rubbish collection, and so on. These lists are usually up to date and accurate. However, they are not published, so the main problem with using these for survey sampling is getting access to them from the authorities that own them.

Many organizations have mailing lists of their subscribers or users. These can easily be used to draw samples for surveys, using systematic sampling. They are usually samples of people.

A lot of these lists are not representative of a population. For example, a radio network I once worked with held a competition with a very valuable prize (an overseas holiday). All competitors had their names and addresses entered into a database, possibly to be surveyed later. Though competitors weren’t meant to put in more than one entry, we found that some addresses had 20 entrants. Judging from the names, it seemed that some people had entered their pets!

Before using any population list, find out more about it. Analyse it closely, and consider:

1. Is it a sample of people, of households, or what? Is it suitable for your purpose?

2. How complete is it?

3. How up to date is it?

4. How accurate is it?

5. Does it contain duplication?

A simple way to check a list is to find 20 people who should be on it, and look them up on it. On a perfect list, all of them will be there once (and only once), and all information will be up to date. This is rare!

Before using any list to generate a sample, I suggest you bring it into a computer program with which you can view the data in a spreadsheet-like format (rows and columns, with one row for each person, and a column for each piece of information). Then sort it on every field in turn. Examine the first and last few entries in each field. If there are any problems, you are likely to find them at the top or bottom of a column.

Read carefully through the list, searching for duplicates. It’s often easier to see mistakes in print than on a computer screen. If possible, print it out, in several different sequences, and have a different person check each printout. You’ll probably be amazed how many obvious errors you find. You’ll find some people on it several times, maybe with different addresses or slightly different versions of their name.

The biggest problem with an incomplete list is that it’s likely to be biased in some way: in other words, it may not represent a typical cross-section of your audience. If this problem exists, random sampling cannot fix it: all you’ll get is a representative sample of an unrepresentative list.

It can be tempting to use these "found" lists. It can be cheap and easy to do a mail survey with such a list, but if you don’t know how representative it is, the results it produces can be extremely misleading. Never rely on the results from a single survey, unless you know it's random.

If you only have a map, and no information about where the population is spread across the map area, it’s difficult to achieve an accurate sample. But it’s more difficult still when you don’t even have a map. Unfortunately, this is common in developing countries. Even though census data at a district level may be available, if there are no detailed maps it can be difficult to relate the census place-names to areas on the ground.

So it’s vital to find or create a local map. Sometimes these are painted on walls at local government offices, and you can copy this by hand from the wall (or photograph it).

If you can’t find a map, hold a meeting some well-informed local people and create a hand-drawn map. This does not need to be exactly to scale. It should show all locality names and main roads in the district.

9. How to draw a sample from a population list

The simplest method of sampling from population lists is to use systematic sampling. This means that you divide the list into a number of equal groups, select one random number, and sample the same location in each group. Here’s how.

So that’s systematic sampling: the advantage is that you draw only one random number, and use it over and over again. You should look for two problems:

1. There is a tiny chance that the population list is arranged so that there’s a regular sequence in the entries. Maybe in a list of people, every alternate one is a man and every other one a woman. If you use systematic sampling, you would select either only men or only women. (It’s extremely unlikely that any list would be ordered like this, but it would badly spoil your sample.)

2. It’s easier to round off the sampling interval so each group comes from a whole number of pages. This means that there will be some unused pages. Don’t leave all of these at the end - scatter them throughout the list. If the list is in geographical order, this will ensure you don’t exclude a large area. Later, you may be able to use some addresses on these unused pages, to replace sampled addresses that turn out to no longer exist.

If you are designing a stratified sample (dividing the survey area into a number of smaller areas, and taking a separate sample from each smaller area) you should check any population list you want to use, to see if is already stratified in a suitable form.

As stratification is based on Census data, the population list you use must be divided into Census areas. If it is not already divided in this way (e.g. a telephone directory covering more than the whole survey area) many hours’ work will be needed to draw a properly stratified sample.

10. Choosing the place of interview

People can be interviewed in three main places: at their homes, at their workplaces, and in public places. In most surveys, people are interviewed at home. As almost everybody has one home, home-based sampling provides a better coverage of the population than samples based on workplace (because not everybody has a job) and public places (because some people spend very little time there).

With a probability sample, it's usual to interview people at home, because it's usually the homes that are sampled, rather than the people who live in them.

If you are using a quota sample, people can be interviewed anywhere you find them: at home, at work, or in a street. Though this seems easier, it's not as valid - see section 3 above.

With some types of sample, it's better to find people at some place other than their home. If you are surveying the workforce of an organization, it may be more convenient to interview them at work (as long as they'll tell the truth there). If you are surveying people about shopping - common with market research, but rare with audience research - it can be better to survey them in a shopping area (see section 15 of this chapter). And if you are surveying the audience to some kind of event, the obvious place to interview them is at the event: see the chapter on event surveys for more details of this.

11. Selecting starting points for door-to-door surveys

Door-to-door surveys nearly always use cluster sampling, because it is so much cheaper than choosing individual households at random. When you are using clusters, the sampling is done in at least three stages:

1. Choose the starting address (at random, from a list)

2. Choose a random route to take after finding the starting address: a route that gives every household in the cluster an equal chance of being selected for the survey.

3. Choose one or more persons in each selected household.

When people are surveyed in their homes, usually one person is selected at each address.

With 500 respondents, 500 separate addresses would be used. Unless the survey was confined to a small town, the chances are that these addresses would be widely scattered. This would provide wide geographical coverage, but much time would be wasted going from one dwelling to another. In a large area, more of the interviewers’ time would be taken up by travelling than by doing interviews.

To increase productivity, surveys normally use cluster samples. Instead of selecting 500 individual addresses at random, only 50 might be chosen, and a cluster of 10 neighbouring households surveyed at each point. So there would be 50 clusters each of 10 households.

You can see intuitively that taking only 50 separate parts of the city is not going to be as representative as taking 500, because neighbours tend to be similar in their habits and characteristics. To equal the accuracy of a simple random sample of 500, a cluster sample would need about 750 people. However, it is cheaper to interview 750 people in clusters than 500 individually.

Clustering saves most money when interviews are brief, and travel cost (from home or office to cluster) is high, and few or no extra trips to the cluster need be made, to interview the last few respondents. If your survey is in a large city, and you have few interviewers, and questionnaires are left for respondents to fill in and be collected on a return trip, clustering can save a tremendous amount of money, and clusters can be quite large.

In practical terms, cluster sizes usually range between 3 and 20 households. If clusters are too small, travel costs rise, but (for a fixed sample size) there will be more clusters, and the effective sample size will be larger. If clusters are too large, travel costs will be less, but the effective sample size will also be smaller. Another problem with large clusters is that interviewers can run out of households.

A good compromise in most situations is to have about 10 households per cluster.

Three ways of selecting cluster starting points

(1) Using a local population list

In many countries, local authority offices have a list showing all households. If the authorities co-operate, this can be used to draw a random number, to select the starting point for a cluster. Other alternatives, as discussed above, include electoral rolls, street directories, and telephone directories.

Use systematic sampling, as explained in more detail above. For example, if you want to select 5 clusters in a village with 600 households:

1. Find a list of all households.

2. Divide this list into 5 equal sections, each with 120 households.

3. Choose a random number between 1 and 120 (e.g. using the last digits of a serial number on a banknote). Say it's 57.

4. In each of the 5 sections of the household list, choose the 57th household.

If you can’t find a population list, what can you do? The answer: create one. This is called block listing. It is time consuming, and therefore expensive. But when accuracy is important, and labour is cheap, block listing is the ideal choice. It will also be more up to date than any official population list.

To start with, you’ll need a large map, because it will have to show every street or road in the district. If such a map doesn’t already exist, you’ll build it up as you go. It doesn’t need to be exactly to scale. For a large district, it’s best to have a number of partial maps, and assign one interviewer to work on each.

Interviewers are now sent out to walk the whole length of every road in their assigned area. Nobody is interviewed at this stage, but the interviewers note each dwelling, and write a brief description (enough to distinguish it from its neighbours). If a street is not already on the map, it must be marked there. Dwellings that are clearly unoccupied are noted as such.

One interviewer can list several hundred dwellings in a day’s work - but this depends on the distance between dwellings, the difficulty of counting separate dwellings, and the interviewer’s ability to fend off interruptions from curious residents.

When every house on every road is listed, you have created a street directory for the survey area. When the block listing is completed, count the total number of dwellings listed - ignoring unoccupied dwellings. Number each dwelling, from 1 upwards, giving adjacent dwellings adjacent numbers (where possible). You can take a systematic sample from this list to work out the starting points for clusters.

When no population list is available, and block listing is too expensive, the only other method of finding cluster starting points is to use area-based sampling -which is similar to sampling from maps. This type of sampling is not ideal, because it gives each area of land an equal chance of being surveyed, not each person. Therefore, people who live in thinly populated areas have a greater chance of being included in the survey. As towns are densely populated, people living in towns will have a lower chance of being included.

There are several solutions to this problem, but the best solution may be different in each cluster.

Separating areas of equal population density

One solution is to survey towns separately from rural areas. In some rural areas -for example, fertile plains - the rural population is distributed quite evenly, with most people living on small farms. As long as the area of each cluster has a consistent population density, area-based sampling (e.g. from a map) will be reasonably accurate.

Aerial photographs

The second solution uses aerial photos. If you can get an aerial photograph of the area where the cluster is, you can count the roofs, number them, and form a sampling frame that way. If the scale is no greater than 1:10,000, and the roofs are clearly visible on the photo, this works quite well. However, aerial photos are sometimes many years out of date. Professional aerial photography is very expensive, because special aircraft are used. But if you hire a small plane for an hour or so, and take photos from that, this is much cheaper than block listing. Choose a time of day when roofs are most visible - this varies with the roof materials, roof shape, and the weather. In some places, the middle of the day is best; in others, early morning or late afternoon. The best height is about 5,000 feet (1,500 metres): below this, the area of each photo is too small, and above it, individual roofs are too difficult to make out. If possible, use a high-wing plane, so that the wings don’t get in the way. If you are taking photos though the windows, avoid using an auto-focus camera; these often try to focus on the glass. To be safe, have two photographers (one on each side of the plane) and two different cameras. Though official aerial photos are usually in black-and-white, colour photos are easier to interpret.

When the photos are developed, there will be a lot of overlapping. It’s best to number the photos, and draw lines on each to show where other photos overlap -otherwise it’s too easy to count some roofs twice.

Radial sampling

The third solution, which I call radial sampling, works well in countries where most people live along roads, and there are not a lot of roads. This often applies in south-east Asia.

For example, in 1997 I was training people in Laos in survey methods. Our class, with 12 students, decided to do a survey in the town of Phonhong, about two hours’ drive north of Vientiane (the capital). We had no information about Phonhong except its total population. Our first stop was at the local authority office, where we found the only map of the town: it was painted on a wall. We found that the town was Y-shaped; it had grown around the intersection of three roads.

By driving though the town, we found that early everybody lived on these three roads. The number of houses on each road was approximately equal. There and then, we decided to divide the town into six strips: three roads, each with two sides. The 12 trainees were divided into 6 teams, with 2 people in each. Three teams started from the central junction and worked outwards. The other three started at the edge of town and worked inwards, from the opposite side of each road - as shown on the above map.

The result was probably a good sample of the Phonhong population. I say "probably" because we had no way of being certain. If this had been a real survey, not just a training exercise, I’d have done a block-listing first, because the town had only about 600 households and we surveyed about 120 of these.

This method, radial sampling, works in any town or district where a number of roads meet in a central place. Here’s a more systematic set of rules for radial sampling.

When most people live along the radial roads, this method will produce a representative sample. The exception is when densely populated areas (e.g. squatter settlements) are in areas between the main roads. Radial sampling will often miss these areas - and block listing or recent aerial photographs would be better.

12. Sampling inside clusters

When the starting point of a cluster has been chosen, how is the rest of the cluster then decided? The interviewer finds at the randomly selected address, then follows a set of rules to work out which addresses will be chosen for interviews. The important thing is to have some rules. Don't let the interviewer choose - because the houses where interviewers prefer to go are not typical. Interviewers always prefer to visit rich homes rather than poor ones, homes where somebody is there, and homes that are easy to reach.

Interviewers also prefer to talk to people who are similar to them. In developing countries, interviewers are usually well educated, and don't like speaking to people they see as ignorant. This causes a bias in many surveys, making it appear that populations are better educated and wealthier than in fact they are. Although cluster sequence rules are arbitrary, they must be followed.

In each cluster, the address selected at random is usually not surveyed. This may seem strange, but the chances are that the population list from which that address was drawn is incomplete. Excluding the address actually selected partially compensates for the under-representation of other addresses on the population list.

A common set of rules for making a cluster is:

In this map, the starting point (marked Start) is just before the dwelling marked 1. Every second dwelling is ignored. Those marked x were selected, but did not result in interviews (due to refusals, etc.) and had to be replaced. Note the route taken when the interviewers went around the block and reached the starting point again: they crossed the road, turned around, and kept going in the opposite direction - still turning left whenever necessary.

Why every second household, and not every household? (This is called a skip interval of 2 households.) Mainly because neighbouring households tend to be more similar to each other. So using a skip interval brings more variety into the cluster, while still keeping it reasonably compact. The fewer households in a cluster, the more addresses should be skipped. But when a cluster includes more than about 50 dwellings, including skipped ones, it becomes too large (specially in rural areas), and some of the cost savings disappear.

Why go around the block, and not continue in a straight line? Because this would favour households living on main roads - which are often richer than those living in side streets. Where everybody lives on a long road (as they do in some parts of the world) there are no street blocks: observing the above rules, the route will simply follow the road.

Why keep turning left, and not right? This is completely arbitrary; it's just a convention. Change it, if you like - but don't give interviewers a choice in each cluster.

Instructions to interviewers should make it clear exactly what you mean by cluster size. What happens when a household is visited and nobody is home, or the occupants refuse to take part in the survey? Are these households counted in the cluster, or not?

The simplest solution is to keep going, adding more households to the end of the route, until interviews have been done at the required number of households. However, substitutes are not usually taken until a dwelling has been visited at least three times, in an attempt to find somebody home.

13. How many respondents in each household?

Another factor to take into account when designing a sample for a door-to-door survey is the number of respondents in each participating household.

For a personal-interview survey, when each respondent is questioned directly by the interviewer, it’s easiest to interview only one person per household. If, as is common, others are present during an interview, those who have already heard all the questions may give different answers from the initial respondent. If most of the questions relate to facts which would be known by anybody in the household (e.g. "how many television sets are at this address?") having extra people present may produce more accurate results. But for questions asking about personal attitudes, it is best not to have anybody else present, so that the selected respondent will feel free to give his or her true opinion.

An exception to interviewing only one person occurs when the focus of the survey is on something that is not particularly common. A survey of computer users, for example, may begin with the interviewer asking "Does anybody in this household use a computer?" and interviewing all computer users, if the household had more than one.

A estimation problem which occurs when only one person in a household is interviewed: people in small households will be over-represented. Among all households contacted for a survey, people living alone will have a 100% probability of being interviewed. But in a household with four eligible persons, each of these people will have only a 25% chance.

In Australia, about 10% of adults live alone, but these make up 20% of all households, and their media use habits are quite different from those of larger households. In developing countries, which generally have more people per household, single-person households are rarer, so it will not distort results so much to interview one person in each household. The easiest way to compensate for an excess of small households in the survey is for the interviewer to find out how many adults live in each household visited. Then multiple interviews can be made at larger households.

By "larger" households, I mean 3 or more people in developed countries, and 4 or more in developing countries (where households tend to be larger).

If you interview more people in larger households, this can slightly increase the accuracy of the survey, but you will be unable to determine the exact sample size in advance. The simplest solution is to base your calculations on one person per household. Not many households will have two interviews, so the final sample size will be perhaps 5% to 10% larger than you planned.

If you survey all people in the household (except perhaps children), this solves one problem, but creates several others:

• Sometimes one person can affect the answers given by others. This applies specially to questions that measure knowledge, such as "Please name all the radio stations you can think of."
• The effective sample size is lowered, for some types of question. This is equivalent to increasing the sampling error. If you ask 100 people in 100 households how many radios their household has, this result will be based on more radios than if you ask 100 people in 50 households the same question.
• It is awkward to interview more than one person in the same household. Often, after hearing two interviews, prospective respondents will simply say "my answers are the same", and refuse to be fully interviewed.

Another approach is to interview all household members at the same time, using a single questionnaire. We used this in a survey of Aboriginal people in central Australia. In the evenings, they usually sat outdoors in small groups, listening to portable radios. The interviewers would approach one of these groups, play brief taped extracts from radio programs, and ask the respondents’ opinions of each program. But the questionnaire was different from a normal one: instead of ticking boxes for "like it", "dislike it", and "not sure", the interviewer would write in the number of people giving each possible answer.

In a survey where respondents fill in their own questionnaires, and these are collected later by the interviewer, it’s normal to give a questionnaire to each person in the household. This boosts the sample size at little extra cost, but also helps prevent people filling in questionnaires intended for another member of the household.

14. Choosing respondents within households

A common mistake in survey research is to interview the first person met in each household. This will produce a badly skewed sample, nullifying any care that has been taken in producing a representative sample of households. This is important for any survey, but particularly for surveys measuring radio and TV audiences.

What is the problem with interviewing the first person the interviewer meets? It's because the more time somebody spends at home, the more chance they have of being interviewed, with this method of choosing respondents. People who spend a lot of time at home have different habits from people who are out a lot. For example, most radio and TV viewing is done at home, so if the first person found in each household is interviewed, the survey will overestimate the amount of listening and viewing.

For the same reason, surveys carried out in streets and public places will usually underestimate radio and TV audiences.

In Australia, some types of people (e.g. women and younger people) are much more likely than others to answer the door (or the telephone) when an interviewer visits. In other countries, such as Western Samoa, it is normal for the oldest man in the household to greet any strangers.

The best approach is for the interviewer to speak to the first person met, work out who should be interviewed, then to interview the appropriate person. There are three main methods for choosing a respondent: the birthday method, the Kish Grid, and quotas.

Most market research books recommend asking for the person who last had a birthday (or who next will have one). In theory, everybody in a household has an equal chance of being selected by this last birthday or next birthday method, but my research has found this does not produce the correct balance of sexes and age groups. Also, it only works in households where everybody knows everybody else's birthday. In countries where birthdays are not celebrated, many people don't know their family's birthdays.

This is a table of numbers, named after the statistician who invented it. The number of people in the household is discovered, and a random number is chosen to select a particular person.

My research in Australia found that the Kish method can cause a high refusal rate: elderly women, in particular, are often suspicious when the first question in a survey is "How many people live in your household?" — particularly if they live alone. In developing countries, where few old people live alone, this may not be a problem. Here’s an example of a Kish grid, with instructions. This is based on 8 households per cluster, interviewing 1 person per household.

The reason for numbering the household members from the youngest upwards (instead of the seemingly more obvious oldest downwards) is that younger people are more difficult to find at home, so the above grid gives young people a slightly higher chance of being interviewed.

When selecting a respondent within a household, the most practical method is often a type of quota sampling. Though quota sampling was criticized earlier in this chapter, most of its problems do not apply when selecting a member of a household.

A common approach is to interview a woman in half of all households, and a man in the other half — in most parts of the world, where the sex balance is close to 50/50.

To ensure a good balance of old and young people, age-based quotas can also be applied. One of the simplest quota systems is to ask to interview the oldest person in the household (in half the households visited) and the youngest eligible person (in the other half of the households).

Household quotas can be based on other factors apart from sex or age group. It can be useful in radio and TV surveys to have separate quotas for people who stay home most of the time (housewives, retired and unemployed people), and those who spend less time at home: workers and students. Of course, such quotas must be based on known figures, usually on Census data. If 60% of the eligible population are workers and students, and 40% stay home, and the quotas reflect these percentages, 60% of respondents will be workers and students.

15. Sampling people in public places

If survey results are to be projected to the general population, a bad way of selecting a sample is to interview people in the street or at a shopping centre, particularly on a weekday. Workers and students are under-represented in such surveys, as are people who are too busy to be interviewed, and those who seldom walk around streets or shopping centres. About the only valid use of shopping-centre surveys is when the population of interest is shoppers. Market research companies do many surveys in shopping centres, usually about products bought in shops. Because their target audience is shoppers, these surveys are reasonably representative.

Another problem with surveys in public places is that they often greatly under-estimate broadcast audiences - because people who spend a lot of time in public places (and therefore less at home, where they might watch TV or listen to radio) are more likely to be interviewed.

A partial solution to this problem is to control for how much time a respondent is likely to spend away from home, by setting quotas based on employment status and age, as shown in Census data. For example, if 15% of the whole population are students aged under 25, then 15% of respondents should be in the same category. This method is far from perfect, but it produces more accurate results than not using quotas.

Occasionally, there’s no alternative to doing surveys in public places. In cities in Papua New Guinea, for example, the crime rate is horrendous. Houses are surrounded by high fences with locked gates, and guarded by fierce dogs. Interviewers cannot get access during the day, and it is dangerous to go to unknown places at night. It’s not possible to do a survey by telephone, as less than 1% of households have a phone. Nor is it possible to do a mail survey, because the literacy rate is less than 50%. So surveys in public places are the only feasible alternative - despite their problems.

16. Checklist of sampling decisions to be made

This checklist applies to a door-to-door survey, which uses the most complex sampling. For other types of survey, which do not use clusters, items 5, 6, and 7 do not apply.

Conclusion: is sample design really necessary?

"Is it really worthwhile to go to all this trouble, just to get a sample?" you may wonder. "Why not just interview anybody?" Occasionally, an informal method of sampling will give reasonably accurate answers. The problem is that if you do a survey that takes such shortcuts, you will never know how inaccurate your findings are.

Market research companies, by repeated testing and comparison of results from various surveys, may have found they can get away with statistically imperfect sampling, but it’s harder for inexperienced researchers to justify such shortcuts.

If you are doing a survey whose results are likely to encounter some opposition, people who do not like the results may challenge the survey’s validity. If you can demonstrate that the sample was drawn by correct probability methods, the survey’s results are more likely to withstand scrutiny.

Even if you intend to use the results only for your own purposes, there is little point in doing a survey unless the results are as accurate as possible.