Question design

General question categories

Questions are likely to be about

  • background/demographics – asking respondents to provide information about themselves
  • experience and behaviour – asking respondents to describe what they have done in the learning and teaching (L&T) context
  • sensory – asking respondents to describe what they have seen, heard etc. e.g. whether students have seen or been told particular information relating to a course
  • opinions - asking respondents to make judgements about aspects of L&T, e.g. whether students have found particular activities and resources effective for supporting their learning
  • feelings – eliciting emotions, e.g. whether students enjoyed a learning activity or using a learning resource
  • knowledge – finding out what respondents know, e.g. whether students understand what support is available to help them with their learning

Question formats

Closed questions

  • Dichotomous questions force respondents to make a choice, e.g. yes/no questions; male/female etc.
  • Multiple choice questions provide a range of discrete responses.
  • Rank ordering questions provide a range of possible responses (like multiple choice questions) but in addition, respondents are asked to identify priorities.
  • Rating scale questions achieve the opposite effect to dichotomous questions. Rather than forcing individuals to make a choice, this type of question enables a degree of sensitivity and differentiation but still allows the evaluator to generate quantitative data. There are two main types.

1. A Likert scale, which provides a range of responses to a question or statement. For example:

"I enjoyed using the new learning resources."

1 = Strongly disagree
2 = Disagree
3 = Neither agree nor disagree
4 = Agree
5 = Strongly agree

2. A semantic differential, which operates between an adjective and its opposite. E.g.
"How useful did you find the new learning resources?"

Useful 1 2 3 4 5 6 7 8 9 10 Useless

Open questions

These allow respondents to answer in their own words, adding explanation and qualification as appropriate. E.g. How do you feel about…..? What did you do…..? What´s your opinion of….?

When to use different types of questions

Dichotomous questions are useful in situations where you want to force respondents to express a clear opinion between opposing perspectives or as a filter for determining which subsequent questions are appropriate. For example, in an interview you may ask a student whether they have used a particular learning resource. Only if their answer is positive will you need to ask about their experience of using it. If their answer is negative your line of questioning might concern the reasons they have not used it.

Multiple choice questions are useful when there is more complexity in the range of possible responses in discrete categories, but the range of expected responses is still fairly limited. Rank ordering can be useful when the relative preference of respondents is sought from a relatively limited range of options.

Rating scales are useful for seeking a measure of perceptions and attitudes of respondents.
Open questions should be used when rich qualitative data is needed that describes the respondent´s perception of their own experience.

Phrasing questions

Design considerations for phrasing questions are that they should be

  • answerable. For example asking students whether they think they have achieved the learning outcomes presents two problems.
  1. They may not be able to recall the learning outcomes at the time of answering the question.
  2. Feedback on assessment provides them with information about their progress with respect to achieving learning outcomes, and they may not have received this feedback at the time of answering the questionnaire.

easy to understand and unambiguous. For example asking students if they received ‘prompt’ feedback, may generate confusion about what is meant by ‘prompt’. Should they make a judgement about whether this means ‘prompt’ from their own subjective perspective, or from an objective perspective in relation to previously published dates for release of results or feedback? The use of negatives or double negatives may be confusing and easy to misunderstand.

  • succinct and address a single issue at a time
  • in language appropriate for the respondents, avoiding jargon or terms that might be unfamiliar to them or unnecessarily intellectual
  • neutral in tone, not leading respondents to think that a particular answer or view might be expected, more appropriate or acceptable

Multiple choice categories should be discrete, complete, and clear to respondents about whether or not more than one option can be selected. This involves eliminating overlap, as in age groups 21-30, 31-40 and providing an 'other' category if not all possible categories are known, or if including all categories will take up a disproportionate amount of space and time for the number of respondents this will affect.

Using presupposition questions avoids the need to use dichotomous questions. For example “What did you find most useful about the learning resource, if anything?” avoids the need to lead with the opening question, “Did you use the resource?” Adding ‘if anything’ aims to keep the question neutral in tone.

‘Why?’ questions may not provide the focus you are hoping for in the response. For example “Why did you enjoy the learning activity?” could produce a wide range of perceived reasons. Any one student may have a number of different reasons (to do with their personality, motivation, social influences, learning needs etc.), and they may only choose to give you one or two of these. This may result in a wide range of responses that, although interesting, may not inform the evaluation issue you were intending to explore. A more focused question of “What did you enjoy about the learning activity?” is more likely to provide information about what it is that makes a L&T approach engaging for students

Strengths and limitations of different types of questions

Strengths

  • Closed questions can be easily coded and lend themselves to statistical analysis.
  • Open questions allow for issues to emerge from the respondents.

Limitations

  • Closed questions in questionnaires do not allow respondents to qualify or explain their answers, which may lead to bias in the interpretation due to the categories imposed by the questionnaire designer.
  • Statistical analysis of closed questions can be useful only as a measure of perception or satisfaction when ambiguous qualifiers such as ‘appropriate’, ‘useful’ etc are used which are open to subjective interpretation.
  • Rank ordering questions can be difficult for respondents who struggle to prioritise the options, particularly if there are more than five options.
  • Open questions can be difficult to code or classify in the analysis stage.
  • Open questions are more demanding of a respondent’s time, so there is a limit to the amount of open questioning you can achieve with a questionnaire.

Further reading

Cohen, L., Manion, L. and Morrison, K. (2000) Research Methods in Education. 5th Ed. Oxon: RoutledgeFalmer. See `Types of questionnaire items'. pp 248 - 258. [Available from University Main Library].

Fowler, F.J. Jr. (1995) Improving Survey Questions: Design and Evaluation, Thousand Oaks, Ca.:Sage Publications, Inc. [Available from University St George's Library and SOLAR].