Annoying article about polling error

31 03 2015


The Royal Statistical Society has published a ‘report card‘ detailing various problems with various polling methods.

Actually, that’s all the article really does – points out all the problems with different survey methodologies, and suggests folks should keep all these in mind when interpreting results.

NEWS FLASH: The potential sources of error for any methodology are endless. All survey methodologies have advantages and disadvantages. I could write pages and pages on error sources within each of the methodologies they mention.

The article might give a reader the impression it’s actually possible to carry out a ‘perfect’ survey. It’s not possible. It has never been possible since the dawn of population surveys. The data are flawed!

As I’ve said numerous times on this blog, the pollster’s job is not to carry out the perfect survey. It’s their job to understand why they can’t.


Non-probability sampling and sampling error

29 03 2015

Just noticed this infographic on Twitter, and I’m blogging it mainly so I don’t forget where I saw it. It has a wee note at the bottom:

“This online survey is not based on a probability sample and therefore no estimate of theoretical sampling error can be calculated.”

I may use this note in future when I don’t use probability sampling.

A probability sample is one where every person in the target population has a chance of being selected for the survey, and you can accurately determine the probability of that happening. Due to things like poor response rates, it’s never actually possible to collect a true probability sample. However survey researchers can try to closely approximate it.

The margin of error that is often reported with survey results is based on the assumption that the survey used probability sampling. However quite a lot of the time the researchers have made no effort to do this, or it’s simply not practical to do this.

One of the reasons some political polls weight by household size is that people in larger households have a lower probability of being selected then people in smaller households. So, if they’re trying to approximate a probability sampe, pollsters will apply an inverse probability weight to adjust for this. In my experience, if the survey has been well conducted, this weight alone goes a fair way toward adjusting for Māori, Pacific and low-income under-representation.

Not about polls and surveys anymore

19 12 2014

I like to always have something to do.

One Sunday I was feeling bored, and I came across this tweet by Vaughn Davis (@vaughndavis).

After a brief discussion about his technique for those fantastic looking panel lines, I was off to Toy World in J-ville. My first build was this 1/72 scale Spitfire Mkla.

Photo 14-12-14 08 19 38

There’s a massive learning curve, and clearly I’ve got a way to go. I was never patient enough to build scale models when I was a kid. I’m very slightly more patient now, and I’m totally addicted to this. I’ve just bought a second hand compressor and airbrush, and am two thirds of the way through an ME109 (sticking with the WW2 theme for now). I’ve also bought the kit for a Douglas C-47 Skytrain – and will be painting it in D-Day colours.

So, as well as posting about polls and surveys, I’ll now post the occasional completed scale model. :)

How did the polls do? The final outcome.

4 10 2014

Now we have the final election result, I’ve updated the table from my previous post. In addition, I’ve included a similar table for the polls-of-polls, and a pretty graph!

UPDATE: I’ve revised the chart and first table with UMR’s pre-election poll result, published by Gavin White on SAYit Blog. I’ve checked all my numbers fairly carefully, but if any pollster, pundit, or media organisation spots any errors please let me know and I’ll update this post accordingly.

Final result chart

How I calculated the above results.


Poll of polls

The overall picture remains similar.

  1. Well done DigiPoll and DPF (Curia poll-of-polls)
  2. Still no evidence, this election, of the ‘National bias’ that some people talk about.
  3. If there is any poll bias, it appears to be toward the Green Party.
  4. The landline bias/non-coverage issue is a red herring – the polls that came closest only call landlines. It’s just one of many potential sources of error that pollsters need to consider. Here’s another post about this, if anyone is interested in finding out why it’s not such a big deal.

How good was my prediction? (updated)

21 09 2014

It was okay. Not happy with my Green Party prediction.

UPDATE: Now updated with final election result. Things are a little better when it comes to the Green Party result, so I’m a little happier.


How did the polls do?

21 09 2014

This table shows the provisional election result and all the final pre-election poll results.

Table 1

This table shows deviations from the final result, for each party and poll.

Table 2

A few initial thoughts:

  1. Well done DigiPoll.
  2. Looking at these results, I see no evidence of the ‘National bias’ that some people talk about.
  3. If there is any poll bias, it appears to be toward the Green Party.
  4. The landline bias/non-coverage issue is a red herring.


20 09 2014

We have more polls-of-polls now than we have actual polls. I thought it would be interesting to put all their predictions in one place, and also to calculate a basic average.

Please note that this is not my election prediction. I posted that yesterday.



Get every new post delivered to your Inbox.

Join 711 other followers