This post is about writing questions for opinion polls, but I’ll begin by contrasting opinion polls with the work that I usually do.
Nearly all of the research I work on is social research, typically for government and not-for-profit organisations. I work within a team of social researchers who are all very passionate about doing research that contributes to social good.
Designing social research questionnaires
In my experience, designing questionnaires for social surveys can be challenging and sometime very daunting. Firstly, you need to understand the topic of the survey. The topic could be a specific behaviour a client is wanting to influence or change, an organisation’s processes, a possible policy or legislation change, or the public’s experiences, perceptions, beliefs, or knowledge of something.
Secondly, and equally as important, you need to put yourself in the shoes of the respondent. The questions you ask need to make sense to all respondents, and be interpreted in the same way by quite different people. From the respondent’s point of view, all the questions should be straight forward and relevant to them and their experiences. To achieve this though, the questionnaire scripts themselves sometimes need to be long and highly complex, with lots of tricky skips and question routing to ensure you’re asking the right questions of the right people. From what I’ve heard, the clever people who programme our social research scripts (into a telephone or online format) will sometimes open our questionnaire documents with a sense of trepidation – but most seem to enjoy the challenge in the end.
Finally, the terms and phrases that we and our clients use can sometimes be different from those used by everyday New Zealanders. Very occasionally I need to explain to clients that we need to drop some of the terms (and acronyms!) they use in their discipline or industry, in favour of terms and phrases that are familiar to the wider public.
Writing questions for polls
Questions for polls are also challenging. The points above still apply; however in social surveys you might have 10 or 15 minutes to explain and explore nearly every nuance of the survey topic. In an opinion poll you’ve got roughly 20 to 30 seconds to ask one question, or possibly two. The reality is that you could often write an entire questionnaire on the topic of a single poll question!
In a poll you don’t have much opportunity to tailor questions to specific groups, so the question needs to be meaningful to a wide range of people. Here are some of things I try to think about when writing questions for opinion polls:
- How will the results be used and reported? Will a simple yes/no response be enough, or do we need to understand strength of support or opposition to something (eg, how strongly do you agree or disagree that…)?
- What prior knowledge do the public have about the topic? Do we need to include a sentence or two to define or give context to the topic? How much can we explain before a respondent begins to switch off?
- Are we looking for a ‘gut feeling’ or a more considered opinion? Have the specifics of a suggested policy or legislative change even been laid out? Do we need to explain the possible consequences of a legislation change? Is it possible to do so in a balanced way?
- Do the public and the media commonly use accurate terms to discuss and understand the topic (eg, think: anti-smacking bill, asset sales, and bigger class sizes)? If the commonly used terms are not accurate, does it really matter? In some cases it definitely does.
- Does the question include any emotive or loaded terms? If it’s an emotionally charged and loaded topic, sometimes it’s not really possible to ask the question in a non-emotive way.
- Is the question biased or leading in some other way? I don’t really think it’s possible to design a question that will completely satisfy everyone on all sides of an issue, but do I think the question is fair to both sides? An interesting example – I’ve seen research showing that people often associate the term ‘homosexual’ with gay men, so if asking about ‘homosexual marriage’ you’re likely to be leading people to think of same-sex male couples rather than same-sex female couples (you can reach your own conclusions about how that might influence a poll result).
I could probably keep adding to this list. To make things even more challenging, we occasionally need to write or edit questions in the last possible moments before the script is due with the programmers (although they are usually very kind to us when we’re late).
I’ve found the best thing I can do though is to have someone else look over my questions. When you’re thinking through all of the above points you can sometimes lose the wood for the trees. A second pair of eyes is always helpful.