Polling error: Babies and bathwater (bah!)

20 01 2016

A colleague emailed me this article about inaccuracies in UK political polls.

This was an interesting quote.

Prof Patrick Sturgis, director of the National Centre for Research Methods at the University of Southampton and chair of the panel, told the BBC: “They don’t collect samples in the way the Office for National Statistics does by taking random samples and keeping knocking on doors until they have got enough people.

“What they do is get anyone they can and try and match them to the population… That approach is perfectly fine in many cases but sometimes it goes wrong.”

For the record, this is not the approach used by all New Zealand political polls. Two New Zealand polls that I know of do not target specific numbers of people (quotas) by age, gender or ethnic identification. Instead, they randomly generate phone numbers, randomly select the person in the each household to interview, and try repeatedly to get hold of that particular person until that person says whether they want to take part. Where possible, interviewers make appointments to reach that person, or will even call them person on a different number or outside of normal shift times if requested. No substitution is made within the household under any circumstances. This approach is what’s called ‘approximating a probability sample’.

The key benefit of this approach is it seeks to generate a sample that is representative by both the known and unknown factors that contribute to party support. Sure, age, gender and ethnicity might predict preference for some parties, but what about the infinite number of other factors that contribute to one’s political preference (eg, socialisation, personality, peer group, etc). A methodology that focuses solely on age, gender and ethnicity is one that may be ignoring these unknown factors.

Yes, response rates are certainly an issue. For that reason no poll can be said to be ‘truly random’ – that’s where weighting comes in, to (hopefully) adjust for non-response and non-coverage.

Why am I making this point?

Recently I’ve come across some very senior people within the research industry telling clients “weighting is bad”, and they’ve recommended changing a well thought out methodology to a non-random quota approach for no other reason. BAH!

I recommend quota sampling myself in some circumstances. It can be very useful. However the ‘weighting = bad’ idea is throwing the baby out with the bathwater.

Sure, if your weights are extreme, this is bad. This increases ‘variance’, and can make party support appear quite volatile. However if you’re carrying out high quality fieldwork in New Zealand among the general population, your weights should not need to be too extreme. If your weighting approach is logical and scientific, your poll results should be close to the mark. This has been borne out through poll results close to the last couple of New Zealand general elections.

Whenever I’m polling I’m looking carefully at my final survey weights. When the weights become extreme (perhaps due to non-coverage of landlines, etc), that’s one signal that it may be time to adjust the polling methodology.

If you’re telling people “weighting = bad” either a) you don’t know what you’re talking about, or b) you do know what you’re talking about but you don’t have the ability to weight your data (I’m guessing due to lack of software, or understanding of weighting).



Support for NZ becoming a republic: Improved poll question

2 12 2015

A couple of years ago I read a critique of the different poll questions that measure support or opposition for New Zealand becoming a republic. It was a fair and reasonable critique – but I can’t remember who wrote it or where I saw it (happy to link if someone can find it).

One of the points related to Colmar Brunton’s question: Do you believe New Zealand should become a republic, or stay as it is, with the Queen as head of state?

This question only mentions the possibility that a Queen would remain head of state. If I recall correctly, it was suggested the result may be different if the question mentioned the possibility that a King could be head of state (presumably because people may feel more or less positive toward Prince Charles than the Queen).

Fair point.

In the most recent Q+A Colmar Brunton poll, the question was revised to read: Do you believe New Zealand should become a republic, or stay as it is, with a King or Queen as head of state?

Interestingly, the result was no different from 2012.


Perhaps there actually has been a change, but it’s masked by the change in wording. For example, it’s possible support for the status quo has increased since 2012, but including ‘King’ has offset this. On the other hand, opposition to the status quo could have increased, but this has been offset by more positive feelings toward either Prince Charles or William.

Who knows? It’s still a better question.



How to improve pie charts (via Stats Chat)

3 06 2015

Just brilliant – link.

(Via Stats Chat)

Article in Nature about UK polling accuracy

10 05 2015

Really interesting article in Nature.

Don’t polling companies take these factors into account?
Not really, and it would be unfair to blame polling companies for the way things are done. They always state quite openly that they are only taking a snapshot of public opinion at a particular point in time.

Important point! A poll can only measure public sentiment at the time of the poll.

…But pollsters use quota samples, in which you try to create a representative mini-population based on a number of criteria: gender, age, group, region and social class. In that case it’s problematic to talk about margins of error.

When you do quota samples, you are making assumptions about what actually matters. From the point of view of the psychological behaviour of people, this isn’t correlated very much with gender, religion and so on. So the quota samples could be biased from the point of view of psychological behaviour.

Why don’t polls use a random sample?
Random samples are much more expensive.

Quota sampling has its uses, especially in situations where it’s not really possible to try to approximate a random sample.

Interestingly, things are not quite the same in New Zealand. The two public polls that have come closest to the New Zealand election result (for the last two elections) do not use quota sampling. They use the (more expensive) random sampling approach. (Well, I know one of them does, and I’m 90% sure the other does too.)

One of these polls did get things quite wrong in 2005, but that had very little to do with their sampling approach. It would be a mistake to assume polling accuracy all comes down to the sampling method.

A friend at Auckland University and I are working on a paper where we’re modelling the demographic vs psychological determinants of political preference in New Zealand. We’re using a year’s worth of polling data (with permission from the client) and data from the longitudinal New Zealand Attitudes and Values study. When it’s published, I’ll blog about it here.

EDITS: Because iPads suck for blogging.

Inquiry to be held into UK polling accuracy

9 05 2015

Report by  Sky News.

An independent inquiry is to be carried out into the accuracy of election polls after they consistently underestimated the Conservatives’ lead over Labour.

Predictions of a neck-and-neck race, a near-balanced parliament, and a potential constitutional crisis following the General Election put forward by all major pollsters during the campaign have been proved drastically wrong.

My impression is that, relative to NZ, there’s more information available about individual poll methodologies in the UK. This should assist the inquiry.

The British Polling Council (BPC), which acts as the association for opinion pollsters, will look into the causes of the “apparent bias” and make recommendations for future polls.

The BPC, which counts all major UK pollsters among its members, said in a statement: “The final opinion polls before the election were clearly not as accurate as we would like, and the fact that all the pollsters underestimated the Conservative lead over Labour suggests that the methods that were used should be subject to careful, independent investigation.

It would be wrong to assume political preference is stable during the week before an election, so ‘apparent bias’ is the correct term.

Survation said it conducted a telephone poll on Wednesday evening which showed the Tories on 37% and Labour on 31% but “chickened out” of publishing it as it appeared so out of line with other surveys.

Case in point.

Meanwhile, ICM director Martin Boon appeared to sum up the mood among Britain’s pollsters, tweeting “oh s**t” after the publication of the exit poll showing the Tories would be by far the largest party.

Heh… that’s what he said in public.

UPDATE: Article by Survation. (Thank you Matthew Beveridge)

Survation conducted a voting intention telephone poll the day before the election (Wednesday) with three specific attributes:

  • Naming candidates through a ballot prompt specific to the respondents’ constituency based on their postcode.
  • Carefully balancing our sample based on age, region, sex, and past vote prior to weighting, from a nationally representative sample frame
  • And crucially,
  • Speaking only to the named person from the dataset and calling mobile and landline telephone numbers to maximise the “reach” of the poll.

This was conducted over the afternoon and evening of Wednesday 6th May, as close as possible to the election to capture any late “swing” to any party – the same method we used in our telephone polls during the Independence Referendum that produced a 54% and a 53% figure for “no”.
This poll produced figures of:

Survation Telephone, Ballot Paper Prompt:
CON 37%
LAB 31%
LD 10
Others (including the SNP) 6%

Which would have been very close to the final result.


We had flagged that we were conducting this poll to the Daily Mirror as something we might share as an interesting check on our online vs our telephone methodology, but the results seemed so “out of line” with all the polling conducted by ourselves and our peers – what poll commentators would term an “outlier” – that I “chickened out” of publishing the figures – something I’m sure I’ll always regret.

So in short, there will be no internal review of polling methodology for Survation post this General Election result.

Really? Based on the result of a single poll that was a better predictor? Let’s hope their model holds true for the next election.

Online polling is very effective and useful for a wide range of reasons, however we’re now clearer than ever that this method of telephone polling as described above will be our “Gold Standard” for voting intention going forward.

I think online polling can be effective for political polling. However this is just one of an infinite number of things that contribute to a poll.

Annoying article about polling error

31 03 2015


The Royal Statistical Society has published a ‘report card‘ detailing various problems with various polling methods.

Actually, that’s all the article really does – points out all the problems with different survey methodologies, and suggests folks should keep all these in mind when interpreting results.

NEWS FLASH: The potential sources of error for any methodology are endless. All survey methodologies have advantages and disadvantages. I could write pages and pages on error sources within each of the methodologies they mention.

The article might give a reader the impression it’s actually possible to carry out a ‘perfect’ survey. It’s not possible. It has never been possible since the dawn of population surveys. The data are flawed!

As I’ve said numerous times on this blog, the pollster’s job is not to carry out the perfect survey. It’s their job to understand why they can’t.


Non-probability sampling and sampling error

29 03 2015

Just noticed this infographic on Twitter, and I’m blogging it mainly so I don’t forget where I saw it. It has a wee note at the bottom:

“This online survey is not based on a probability sample and therefore no estimate of theoretical sampling error can be calculated.”

I may use this note in future when I don’t use probability sampling.

A probability sample is one where every person in the target population has a chance of being selected for the survey, and you can accurately determine the probability of that happening. Due to things like poor response rates, it’s never actually possible to collect a true probability sample. However survey researchers can try to closely approximate it.

The margin of error that is often reported with survey results is based on the assumption that the survey used probability sampling. However quite a lot of the time the researchers have made no effort to do this, or it’s simply not practical to do this.

One of the reasons some political polls weight by household size is that people in larger households have a lower probability of being selected then people in smaller households. So, if they’re trying to approximate a probability sampe, pollsters will apply an inverse probability weight to adjust for this. In my experience, if the survey has been well conducted, this weight alone goes a fair way toward adjusting for Māori, Pacific and low-income under-representation.


Get every new post delivered to your Inbox.

Join 779 other followers