Landline survey population non-coverage
The 2013 Census tells us that 85.5% of NZ households have a landline telephone. This means that landline RDD (random digit dialing) sample frame non-coverage is up 6.1 percentage points, from 8.4% of households in 2006 to 14.5% of households in 2013.
This isn’t the percentage of individuals not covered by landline sampling frames. It’s the percentage of households.
Between 1% and 2% of households will not have access to either a cell phone or landline, so this means the proportion of cell phone only households has increased from somewhere around 6% in 2006 to somewhere around 12-13% in 2013.
Other than door-to-door surveys (which can take a couple of months to carry out and process), there’s still no other single sample frame that provides the coverage of a landline sample frame.
- Simply adding cells to the sample frame won’t increase coverage by much, because you’re after people who have a cell but not a landline (all the others are already covered by your landline sample frame).
- The Electoral Roll can be a useful sample frame, except the process of matching names/addresses to phone numbers reduces coverage down to around the 40% mark. Also, the phone records used for tele-matching can be based on home and vehicle ownership databases, so they introduce a socio-economic bias to your sample.
My view about conducting a good phone survey
This is something I’ve been working up to blogging about, because I know it will annoy a few people (it already has). I think this is partly because my views are seen as naive by some – not those of a ‘seasoned pollster’. Oh well, I’ll just have to live with that. Someone obviously believes I know what I’m doing. I hope they’re right!
My litmus test for blogging is ‘would I say this to a client?’ I would definitely say this.
As I’ve said in the past, I think non-response is a bigger issue than non-coverage in phone surveys, because landline RDD sample frames still cover a fairly broad population base, although not without bias. To some extent you can weight the data to correct for biases, such as by age, gender and ethnicity.
Survey researchers can reduce the problem of non-response by using a methodology that is principally designed to drive as high a response rate as possible. This means (among other things) paying careful attention to call-backs, and not simply calling ‘new numbers’ when you can’t get the targeted number of interviews in a particular location.
Basically, if you’re trying to achieve a good response rate, you’re after the people who are harder to survey. Simply ‘adding more sample’ as we call it (ie, calling new numbers), is the easiest way to get your poll done quickly (and cheaply), but you’ll end up with a whole bunch of people who tend to be home a lot, and are happy to take part in surveys. While their views are important, I don’t think these people are a good representation of the NZ population.
People sometimes talk about a weekend vs weekday bias in polls. In my view they’re oversimplifying things, and this is a red herring. Of course there are daily differences in who tends to be home and not home at particular times, but the key is not interviewing on certain days. The key is trying to contact the selected person in each and every household that is randomly selected (again, not calling new numbers!). This means carrying out fieldwork over multiple days and at different times, and making appointments to call back when the selected person is not available – it’s not just about calling on weekends.
In my view, and in the view of others who I consider a lot more expert than myself, quota surveys aren’t so great for achieving good response rates. This is because you end up screening out a lot of people once your quotas begin to fill up (who should technically be counted as non-respondents), and because even quota surveys that conduct multiple call-backs end up taking the first person who helps to complete a quota. Technically quota surveys are not random surveys anymore. Although the household selection is usually random, the selection of the respondent is not. This means it’s not really appropriate to apply the margin of error used for random surveys.
This is not to say that quota surveys aren’t useful. I think they are a legitimate approach to phone surveys and can be very useful, especially in situations where an extremely low response rate is inevitable, or for an omnibus survey where you need to make sure you’re consistently surveying a certain number of people by age x gender x region.