Polls and cell phones… again…

Why don’t you poll cellphones?

This question, or variations on it, is the one I’m asked most frequently. I’ve answered it before on this blog, but this time I thought I’d share some data to help explain my view.

Firstly, let me state that the company I work for does call cellphones. We just don’t randomly dial them for the political poll. As I’ve mentioned before this has very little to do with the actual cost of calling cells. For a polling company, the cost isn’t that much more than it is for landline calls.

I’d like to start by addressing the misconception that it is just low income or ‘young’ households (for lack of a better term) that don’t have a landline telephone.

Please look at the chart below, which I created using data from Statistics New Zealand’s 2012 Household Use of Information and Communications Technology Survey. This is a very robust door-to-door survey of New Zealand households. You can find out more about the methodology here. As you can see in the chart, relative to all NZ households there is a greater proportion of non-landline households in the lower income (and likely younger) groups. However, what’s also clear is that there are substantial proportions of non-landline households in higher income groups too.

1

In fact, you can see in the next chart that nearly half (47%) of non-landline households receive more then $40,000 per year, and a quarter (25%) receive more than $70,000 per year. So sure, absolutely, there is a skew toward lower income (and likely younger) households, but don’t assume that cell-only households all contain lower income people and young people.

2

To help illustrate my next point I’ve combined these data with what we know from the most recent Census – that 85.5% of households have a landline. This chart is not perfect because the survey was carried out at a different time to the Census, and things will have changed a bit in between the two. The chart breaks all New Zealand households down by income band and whether they are covered by an up-to-date RDD (random digit dialing) sample frame.

3

Most polling companies will weight their data to (EDIT: try to) correct for things like non-coverage of age and socioeconomic groups. The problem with weighting is it assumes the people you have surveyed (and are weighting) are similar to the people who you haven’t surveyed.

But here’s what you need to think about if you’re wondering how big the cellphone non-coverage issue really is:

  • Do voters in the under $40k income band who are not covered differ in party support from voters in the under $40k income band that are covered?
  • Do voters in the $40-70k income band who are not covered differ in party support from voters in the $40-70k income band that are covered?
  • Do voters in the $70k+ income band who are not covered differ in party support from voters in the $70k+ income band that are covered?

When I say differ, I’m not talking differ ‘just a little’. When you factor in the size of each non-covered income group (8%, 3%, and 4%), they would have to differ massively from each covered income group, when it comes to party support, for them to make much difference at all to a total poll result.

Now add to this the complexities of calling cells, such as cell response rates and the difficulties nailing down where people actually live (to make sure you’re covering all rural, provincial and urban areas in the correct proportions), and there’s a possibility that including a cell number sample frame won’t add to the robustness of your poll at all.

I’m not saying that polling companies shouldn’t call cell phones. I’m saying each polling company needs to do their own analysis of all the potential sources of error in a poll, and make their own decision about how best to address them. As I’ve said before, calling cells is not, and will never be, the magic bullet for opinion polling.

 

 

Advertisements

7 thoughts on “Polls and cell phones… again…

  1. In the under $40k group, I would say that the non-landline households are different to the landline households. The former are more likely to be very low income i.e. beneficeries and jobless.

    Anyway, the point is not to poll cellphones but to poll people without landlines.

  2. Couple of other angles may be worth exploring:

    The nature of landline as defined. I have one, but no phone plugged in. Why? Because a burp in the way rural loading is applied to the copper costs means that it is cheaper than naked DSL. So there may be a skew around rural. I also know this applies in a number of apartment configurations (they require a landline for the entry phone).

    Also, I have two SIP lines, which also connect to my mobile. No idea whether they are classed as business or residential on a database. That distinction is getting a bit dated too!

    1. Good points.

      The Census asks people if they have a ‘telephone’ rather than a landline, so those without an actual telephone are included in the ‘non-landline’ portion of my charts.

      Any landline ‘number’ is included in an RDD sample frame – even if it’s VOIP, etc.

      We don’t use a database. The numbers are randomly generated and then connection tested. All connected numbers are eligible to be called, unless they are in a published business directory.

  3. The other problem is the “leave a message” incidence. In California, polls are difficult because of this: some 70% of calls reach an automatic message. (And you can imagine how many people would actually phone a pollster back.)

    So call-backs (once you’ve got a random number – don’t give up on it) become critical to fieldwork design. The number of call-backs required to reach a chosen mobile number would, I’m sure, be higher than reaching a chosen landline number.

    The act of calling mobiles is not the entire panacea.

    Love your work Andrew.

    1. Thanks very much Duncan.

      Yeah I think people assume calling cells will definitely improve things. However given the importance of getting a good response rate, and how terrible response rates can be when calling cells, there is no guarantee of an improvement.

Make a comment...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s