Is response rate an indicator of sample quality?

A comment was left in a previous post about response rates, and that it’s possible to have a very representative survey with a low response rate.

So is the response rate an indicator of sample quality?

Not always – it’s more important to have a sample that is not biased than to have a high response rate.

As an example, consider the following two hypothetical surveys about same-sex marriage, which is an emotionally charged issue for many New Zealanders.

1) A random paper-based postal survey achieves a 45% response rate through the use of reminders, a second questionnaire sent to non-respondents, and a prize draw incentive.

2) A random telephone survey, introduced as a survey about ‘current issues’, achieves a 25% response rate.

In this instance the telephone survey (assuming good fieldwork practices) is likely to deliver a higher quality sample. In a postal survey potential respondents are able to see all the questions before they decide to take part. Although the initial mail-out would be random, people who feel strongly about the issue are more likely to respond. For this reason the postal survey sample will not be as representative as the telephone survey sample.

You can’t weight the survey results to correct for this type of response bias. For that you would need to know the population’s views on same-sex marriage – and that’s the purpose of the survey!

Advertisements

2 thoughts on “Is response rate an indicator of sample quality?

Make a comment...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s