The future of polling in New Zealand

(I wrote the below article for the current edition of Interview – the magazine of the New Zealand Research Association.)

Polling is a small part of what we do as an industry, but as the most publicly facing part it’s important our polls are considered robust measures of voter sentiment. The recent failing of UK polls has brought New Zealand polling under even closer scrutiny. Although polling in New Zealand has stacked up fairly well over recent elections, changes to technology and the way we communicate mean that eventually, approaches will need to change.

Telephone polling still works, for now

For the time being, telephone polling still works. At the last Census 85.5% of households reported having a landline telephone. Although landline ownership will have declined further since then, pollsters have so far had some success at tweaking their approaches or making other adjustments to reduce the impact of non-coverage (and/or non-response).  Landline non-coverage and non-response have not yet reached a critical level, but they probably will. So what’s next?

What about more cell phone polling?

Calling cell phones as well as landlines can help to reduce non-coverage error, but due to the number of New Zealanders with both a cell and landline it can be extremely inefficient at doing so. Also, while calling cell phones may reduce one source of error, it could introduce others. For example, generating a representative sample of random cell numbers is made difficult by a lack of accurate data on the structure of the New Zealand cell number system. If your random numbers aren’t representative to start with, your sample of voters won’t be either.

What about more online polling?

Online polling is carried out overseas with varying levels of success. In New Zealand though, I have my doubts that online panels can deliver better estimates than landline polls. We all know online panels suffer from various socio-economic and demographic skews. Pollsters could probably make up for these by setting careful quotas, but given the size of New Zealand panels and high polling frequencies, we’d be going back to some groups of people repeatedly, and in doing so could alter their interest in politics (making them different from similar New Zealanders who are not on the panel).

However the bigger problem with online panels for political polls is a combination of the high frequency of polling and being at the mercy of other projects being carried out using that panel at the same time. If you’re polling during a period of particularly high panel demand, your poll sample that month could well end up being skewed toward people who, say, don’t eat breakfast cereal or aren’t responsible for household shopping. This would ultimately lead to increased volatility of your party support estimates, and your changes between polls could appear odd and unreliable.

So what’s the way forward?

In my view the most promising avenue for forecasting elections lies in data analytics. This means stepping away from tried and true research approaches. It means instead of spending time and money on collecting data and generating representative samples, we spend it on adjusting and modelling unrepresentative data. As an industry we’ve been highly critical of self-selecting polls in the past, but there’s compelling evidence they can be useful, given the right combination of data and analytics skill. For example, a recent study published in the International Journal of Forecasting showed it was possible to forecast 2012 US election results using extremely large samples of highly unrepresentative data from opt-in polls taken over the Xbox gaming platform.[1]

Evidence in New Zealand suggests conventional polling approaches still work, but the range and quantity of data we can access is increasing all the time. This change presents enormous opportunities to our industry, but it may mean reviewing old assumptions and learning new skills.

[1] Wang, W., Rothschild, D., Goel, D., & Gelman, A. (2015). Forecasting elections with non-representative polls. International Journal of Forecasting, 31, 980-991.

Advertisements

5 thoughts on “The future of polling in New Zealand

  1. One of the other difficulties with the data-analytics approach is that it’s much harder for anyone else to tell if your poll is reliable — they need a lot more information, and they need to be much more expert. In the Good Old Days they just needed to know that you did random-digit dialling and that you did some sort of quota or weighting thing.

    1. Yeah that’s a fair call. Without traditional polls I imagine it would be near impossible to calculate reliable estimates. This is a good reason for analysts to get started now – while traditional polls still exist!

  2. I reckon the answer for public good research could be a Stalin-esc government data analytics study with approval from tel-cos and large $ to examine landline/cellphone allocation and usage patterns – and customer data- among ALL tel-co customers every five years or so (conducted at Census time). This then forms the basis for a good dual-sample frame which targets individual adults according to the number they use most. And all polls should be three minutes long with good cellphone protocols for engagement. There we go – totally feasible solution!

Make a comment...

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s