Just noticed this infographic on Twitter, and I’m blogging it mainly so I don’t forget where I saw it. It has a wee note at the bottom:
“This online survey is not based on a probability sample and therefore no estimate of theoretical sampling error can be calculated.”
I may use this note in future when I don’t use probability sampling.
A probability sample is one where every person in the target population has a chance of being selected for the survey, and you can accurately determine the probability of that happening. Due to things like poor response rates, it’s never actually possible to collect a true probability sample. However survey researchers can try to closely approximate it.
The margin of error that is often reported with survey results is based on the assumption that the survey used probability sampling. However quite a lot of the time the researchers have made no effort to do this, or it’s simply not practical to do this.
One of the reasons some political polls weight by household size is that people in larger households have a lower probability of being selected then people in smaller households. So, if they’re trying to approximate a probability sampe, pollsters will apply an inverse probability weight to adjust for this. In my experience, if the survey has been well conducted, this weight alone goes a fair way toward adjusting for Māori, Pacific and low-income under-representation.