Good Practice: Common Question Pitfalls:

Leading Questions

It is very easy to inadvertently formulate questions that lead respondents to a particular answer. Leading questions will often make respondents feel as if there is an implicit ‘right’ answer – and this will introduce bias to the responses.

"How enjoyable did you find the activity?" – This question implies that all respondents must have found the activity enjoyable – and might discourage them to express a negative opinion in the event that they had not enjoyed it.

Possible Solution: Aim to be as neutral as possible and avoid questions which have an implicit suggestion in them.

"What was your overall experience of the activity?" - This may not result in responses that talk about the enjoyability or otherwise of the activity. However, this might tell you what was uppermost in respondents minds.

If the ‘enjoyability’ or otherwise of the activity is an important aspect that you need to measure, think about including a rating scale which asks respondents to rate their experience from unenjoyable to enjoyable. Again, you would need to ask yourself what you want to get from the question and how you’re going to use the resulting data.

(See below for more suggestions about using ratings scales).

Loaded words or phrases

Some words carry specific values or implied meanings and may encourage respondents to give particular responses.

"Should student ambassadors be satisfied with such a simple response from outreach participants? Yes, No, Not Sure" – The phrase ‘such a simple response’ is loaded in that it implies a negative attitude on the part of the questioner and might encourage respondents to feel obliged to answer negatively too.

Also think about whether any other parts of the question encourage certain responses – perhaps because of particular feelings or emotions that they may provoke. For example, references to particular politicians or policies can encourage particular responses that may mask more nuanced consideration.

"Michael Gove was responsible for accelerating the process of school Academisation. Do you think that students from Academies are likely to be more engaged in our activities?"

Including references to Michael Gove and/or Academisation (unnecessarily here!) may provoke strong and/or unintentionally emotive responses from some recipients.

Some phrases can encourage bias.

"Given the busy schedule of the day, do you feel we could drop the advice session in future?" – By including the phrase ‘busy schedule’, we’re encouraging a positive answer here. Well, if it’s already a busy schedule, maybe one less session would be a good thing…

Possible Solution: Read your questions carefully trying to put yourself in a neutral position to assess whether the language used subtly or implicitly leads respondents to certain kinds of responses. Again, ask someone neutral to test the questionnaire and see if they feel themselves being led.

Ambiguity

When you are formulating your questions, be as specific as possible. Make sure that you have removed any room for ambiguity.

"Do you use the library for recreation?" - Which library? (Not to mention - what do you mean by recreation?)

Possible Solution: Ask other people to check your questions for clear unambiguous meaning. Preferably ask colleagues who are not involved in your work, but who can see things from your intended respondents’ perspective.

The use of words, phrases or concepts that may have different meanings for different respondents

Some words or concepts may have different meanings for different respondents. Unfortunately, this is particularly the case for evaluative words.

"Was the food good?" - This is relative and may depend on whether you are used to eating pot noodles or haute cuisine. Without knowing respondents’ baselines, it’s difficult to make any comparisons between responses.

This is equally the case for relational words.

Possible Solution: Read through each question looking for words or concepts that could be ambiguous. Even better, get someone else to do this too!

"How often?"

The use of the term ‘often’ is a common example of this kind of ambiguity.

"How often do you skip school?" - This question suggests that respondents may be able to easily quantify this. But in practice, their skipping school maybe very irregular. They may have skipped school twice in the last fortnight, but prior to that not at all in the previous year – in which case they might struggle to answer this question. (This question also assumes that respondents might be skipping school, which may not be the case).

Possible Solution: Again how you resolve this depends on what you want to find out.

This question could work better as:

"Have you skipped school in the past?"

"If you have, how often have you skipped school in the last three months?"

Variable meanings

Try to avoid using categories that are subjective and could be differently interpreted by each respondent.

"Do you regularly stay late in the Information Commons?" - Regularly can me different things to different people.

Possible Solution: Decide on a frame to standardise the response – e.g. "Do you stay late in the Information Commons more than once a week?"

Late can also be a relative term. One person’s “late” is another person’s “just crawled out of bed”!

Possible Solution: "Do you stay in the Information Commons beyond 7.30pm more than once a week?"

This question may also assume that all respondents will stay late in the Information Commons at certain times. It may never have occurred to some respondents to do so. Maybe they just start early….

Questions based on implicit assumptions

Are your questions framed in a way that makes particular assumptions about respondents?

"Where did you go for breakfast before the activity?" - This makes the assumption that the respondents went for breakfast before taking part in the activity.

Possible Solution: Experts suggest that you test each question to ensure an ‘I don’t’/didn’t response is possible. If so, make sure that your question allows for this.

"If you went somewhere for breakfast before the activity where did you go?"

Shared references

Are you taking it for granted that your respondents will have the same understandings of certain phrases, words or concepts as you or other respondents? This could lead to different respondents answering the same question in different ways – leading to differences in the implications you deduce from the data.

"What information do you need to do well in your school studies?" - Different respondents may understand different things by the term ‘information’. Do you mean written published information or advice or guidance from teachers? ‘Do well’ is also a relative term.

Possible Solution: Where words, phrases or concepts have multiple meanings or are imprecise consider providing definitions or explanations.

"What written guidance from teachers do you feel you need to help you reach your potential in your current school studies?"

Double or negative phrasing

The use of negatives – and especially double negatives – is a minefield when formulating questionnaire questions.

"Do you agree with the statement – I don’t like the seating in the Diamond?"

If you actually didn’t like the Diamond seating would you respond yes or no? Yes, I don’t like the Diamond seating. Or No, I don’t like the Diamond seating……?"

Again, complex or confusing questions are likely to put respondents off answering the question and possibly give up on the whole questionnaire.

Possible Solution: Remove the negative or double negative.

"Do you agree with the statement – I like the seating in the Diamond?" - A yes or no answer is much clearer here. Or if you wanted to see how strongly respondents felt about the Diamond seating arrangements you could ask them to rank their responses on a scale.

Double (or even triple) questions

Make sure each question is only focusing on one clear piece of information at a time. Don’t try to save time and space by collapsing two or more issues together or asking two questions at once – even if they appear to be closely related.

"Were the Student Ambassadors and the Outreach Staff helpful and supportive during the activity?" – This collapses together the Student Ambassadors and Outreach Staff into the same answer. If participants had different responses for each they wouldn’t be able to answer this question.

The question also combines helpful and supportive together – which may not always be the case – you can be one without the other. Do you need to know about both aspects? What will each separate aspect tell you?

Possible Solution: Think carefully about what you want to know and why and if you really need separate answers, ask separate questions. Again look back at your research questions or evaluation objectives to be clear about what you need to know.

"Were the Student Ambassadors helpful during the activity?"

"Were Outreach Staff helpful during the activity?"

'Dual Thought' questions

If you ask respondents for a range of opinions in the same question, they may concentrate more on one aspect at the expense of the other.

"What, if anything, did you most like and dislike about the session?" - Depending on what they felt about the session, respondents might concentrate on what they liked and not give much thought to what they didn’t or vice versa. (Notice, though, the way in which the question neatly avoids being leading!)

Possible Solution: It might be more effective to ask about likes and dislikes separately (without being leading). Or, if appropriate, to opt for a more open frame – "What, if anything, would you like to tell us about the session?" -  (Although bear in mind what you want to know and whether the resulting data will support this).

Questions that rely on memory or recall

Some questions can require of respondents superhuman powers of recall by asking them for very detailed responses.

"What is the title of the text book you found most useful when preparing for this activity?"

Offensive or insensitive questions

Some questions, for example, those about personal finance, politics, religion or demographic background could be uncomfortable for some of your respondent group to answer.

Again, it’s helpful to put yourself in the place of respondents – or ask someone else to do so - to check that you would be comfortable answering the questions.

Sometimes, however, you definitely do need answers to these kinds of questions. Expert advice suggests that it’s better to put these kinds of questions as the end of the questionnaire when the respondent is less likely to give up.