Post 3 of 3: Is a sample of 500 or 1,000 really enough?

This is my final post in a series of three that explain how random samples of 500 or 1,000 can be useful (and this one has a chart!).

  1. What is sampling error? (click here)
  2. How do you interpret the margin of error? (click here)
  3. Is a sample of 500 or 1,000 really enough? (see below)

Is a sample of 500 or 1,000 really enough?

The answer to this question depends on what you want to do with the results. As I see it, there are three main advantages to larger sample sizes. When I’m asked to recommend a sample size, these are some of the factors I consider.

  1. With a larger sample your survey will provide a more precise estimate of the target population (assuming you’ve sampled randomly)
  2. With a larger sample, you have the ability to carry out more detailed and robust sub-group analyses.
  3. With larger sample sizes, your results over time are less volatile. With a smaller margin of error, you can be more certain that changes you observe over time are ‘real changes’ (ie, are not due to random sample variation).

Some surveys require an extremely large sample size. The Ministry of Social Development’s Living Standards Survey is a good example here. For this survey it was important to gain robust estimates of the living standards of New Zealand sub-groups. MSD randomly sampled 5,000 New Zealanders over a number of months for this survey. This allowed them to cut the data a number of ways and still gain fairly robust estimates for population sub-groups (eg, ethnic groups or people who live in particular regions).

Surveys of 500 to 1,000 people are fairly typical in New Zealand

The Living Standards Survey is outside the norm. It is more typical in New Zealand to carry out surveys of between 500 and 1,000 people. The chart below helps to explain why.

The chart shows that the maximum margin of error decreases exponentially as sample size increases. You can make substantial gains in precision by increasing your sample size up to about 400 or 500, but then the gains become much less substantial as you increase your sample size beyond 500.

MoE2

As you can see in the chart, if you double a sample of 500 (and double the survey cost!) the maximum margin of error only decreases by +/- 1.3 percentage points at the 95% confidence level. To decrease the margin of error a further 1.3 percentage points, you’d need to survey an additional 1,807 people (and pay an additional 181% for the fieldwork!).

So, unless you have plans to carry out fairly robust analyses of a particular population sub-group, you need to think carefully about whether it’s worth investing in more than 1,000 interviews.*

Why survey 500 people and not 1,000?

As I mentioned above, the ideal sample size depends on what you want to do with the results. As you can see in the chart, a sample size of 500 is still fairly robust at the overall sample level. The maximum margin of error at the 95% confidence level only increases by +/- 1.3 percentage points over a sample of 1,000.

The main disadvantages of a sample of 500 are:

  1. Relative to a sample of 1,000, your sub-group analyses will be less robust
  2. Relative to a sample of 1,000, results over time will fluctuate more due to random sample variation.

If you’re not comparing results over time, and it’s mainly the overall result you’re interested in (rather than the results for subgroups), then a sample size of 500 is a useful one. Relative to a sample of 1,000, a sample of 500 will also cost less (for a given methodology) and will probably take less time to collect.

Last thing: Sample size has very little to do sample quality

This is a very important point. A very large sample can be extremely skewed (eg, see the Campbell Live GCSB and support for marriage equality polls) and a relatively small sample can represent a target population quite well. Just because a sample is large does not mean it is robust. Just because a sample is small does not make it worthless. A larger sample size will give you a more precise estimate of your target population, assuming you have surveyed a random sample from that population. If you have not taken a random sample, the large number of respondents may tell you nothing useful at all.

****

*An exception can be if you’re wanting to ‘weight’ your data. Weighting attempts to correct for the probabilty of selection and non-response, but it also increases the variance (margin of error) in your survey estimates, reducing your ‘effective sample size’. Sometimes researchers will sample additional members of the target population to ensure the effective sample size reaches a pre-determined level. For example, you might survey 1,200 people to ensure your effective sample size after weighting is around 1,000.

Advertisements

Make a comment...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s