Questionnaire Design
Local Government Data Unit ~ Wales
A guide to...
...Questionnaire Design
A
Questionnaire Design
Local Government Data Unit ~ Wales
This guide provides advice and guidance on designing questionnaires. It focuses mainly on points to consider when writing questions and how to format self-completion questionnaires. You may also want to look at:
•
Survey Design and Analysis
We have produced a series of guides which covers four broad areas:
•
•
•
•
Surveys
Performance management
Collecting and using data
Statistical processes
The series of guides is available electronically on our website, www.dataunitwales.gov.uk/guides We also provide advice, guidance and training support relating to each of the documents in the series. For further information contact:
Local Government Data Unit ~ Wales
029 2090 9500 or visit our website: www.dataunitwales.gov.uk
ISBN – 978-1-907073-02-1
Published 2009
Published by:
Local Government Data Unit ~ Wales
3–7 Columbus Walk, Cardiff, CF10 4SD
Telephone: 029 2090 9500
Email: enquiries@dataunitwales.gov.uk
Web: www.dataunitwales.gov.uk
© Local Government Data Unit ~ Wales, 2009 (Data Unit)
This material can be reproduced as long as the Data Unit is acknowledged as the original author. Any queries should be sent to the Data Unit at the above address or via e-mail: enquiries@dataunitwales.gov.uk B
Questionnaire Design
Contents
1.
Introduction
1
2.
Types of surveys
1
3.
Types of questions
2
4.
General question design guidelines
3
5.
Factual question design guidelines
8
6.
Non-factual question design guidelines
10
7.
Sensitive question guidelines
13
8.
General questionnaire design guidelines
14
9.
Self-completion questionnaire format guidelines
15
10.
Question testing
18
Appendix 1 – Question design checklist
20
Appendix 2 – Self-completion questionnaire checklist
21
Appendix 3 – Example self-completion questionnaire
22
Appendix 4 – Harmonised questions and question banks
24
Local Government Data Unit ~ Wales
C
Questionnaire Design
Local Government Data Unit ~ Wales
D
Questionnaire Design
1.
Introduction
This guide provides advice and guidance on designing a questionnaire. It builds upon the material included in the Survey Design and Analysis guide. In particular, this guide focuses on designing questions and formatting self-completion questionnaires. The guidelines included are also summarised in two checklists (see Appendices 1 and 2).
This guide is aimed at public body staff who design and use questionnaires to evidence service improvement and who often have little or no training in this field.
2.
Types of surveys
In this guide, we will refer to three particular types of survey: postal (or selfcompletion); face-to-face interviews; and telephone surveys. A questionnaire designer needs to recognise that there are specific implications for each of these types of survey. For example, the survey type determines how complex the questions, questionnaire layout and routing can be.
Apart from the simplest surveys, it is rare for all respondents to have to answer all of the questions. Routing is the term for instructing the interviewer or respondent to skip questions depending on the answers to previous questions. For example:
Q1. Do you attend community council meetings?
Tick one box only
Yes
No
Q2. Why do you not attend these meetings?
Tick one box only
Lack of time
Not interested
Other
Local Government Data Unit ~ Wales
For self-completion surveys, both the questions and questionnaire layout must be straightforward and easy to complete. If the respondent has difficulty filling in a self-completion questionnaire, they are likely to: leave some questions unanswered; answer some questions when they shouldn’t; or abandon the questionnaire altogether.
When an interviewer is involved (face-to-face or telephone surveys), the questions and questionnaire can be more complicated. The interviewer will be familiar with the questionnaire and will be able to ensure that the respondent understands the meaning of the questions and only answers the appropriate questions.
1
Questionnaire Design
Local Government Data Unit ~ Wales
Obviously, it is only relevant to ask Question 2 if the answer to Question 1 was ‘No’.
Many face-to-face and most telephone surveys now benefit from being computer assisted. This is of considerable help to the interviewer because any routing will be done automatically. The responses given can be checked for credibility at the time of interview and corrected if the data has been wrongly entered or if the response is unrealistic. Also, in the example overleaf, the computer programme would only display Question 2 if the answer to Question 1 was ‘No’. However, when the survey is paper-based, it is necessary to add routing instructions after the various answers which direct either the interviewer or respondent to the next appropriate question. For example, ‘Go to Q3’.
There are advantages and disadvantages in using the different types of survey. Postal or self-completion is best when asking sensitive questions, although a self-completion section can be included within a face-to-face interview. All answer option lists must be kept short for telephone surveys. Further information on advantages and disadvantages of various types of surveys can be found in Survey Design and Analysis.
Implications for using a particular survey type are commented on throughout this guide.
3.
Types of questions
There are two distinct question types: open-ended and closed. The types of question are defined within Survey Design and Analysis. While the nature of the question you want to ask usually provides an obvious, common-sense format, there are still a few issues to consider.
3.1
Open-ended questions
These allow respondents to answer questions in their own words. However, you will then need to spend time analysing the responses to derive results. If most of the questions in your questionnaire are open-ended, then a survey may not be the best method of conducting this research.
3.2
Closed questions
Although a subject may not appear to be immediately suited to a closed question, an open-ended question can often be transformed into a series of closed questions. This can reduce the survey burden on respondents despite increasing the number of questions asked. There are two types of closed questions: dichotomous and multi-choice.
Dichotomous
This is a question that has only two possible answers for example a Yes/No response. Where appropriate a ‘Don’t know’ option could be included as an additional category. Within self-completion questionnaires a dichotomous question is the most effective type of question to introduce simple and clear routing. Multi-choice
This type of question should offer a mutually exclusive and exhaustive list of answer options (that is, the items in the list should not overlap and should cover every possibility). This type of question does sometimes allow more than one answer in response to the question.
When an exhaustive list of answer options cannot be provided, for example when the list would be too long, there are two choices available. Either add an
‘Other’ or ‘Other – please specify’ option, which would introduce some of the advantages of an open-ended question or alternatively, use an open-ended question. 2
Questionnaire Design
4.
General question design guidelines
A number of useful guidelines to help you to develop effective questions or to review your questionnaires are listed below. The quality of the question that you ask will play a large part in determining the quality of the responses you will receive, and will increase the likelihood that respondents will complete your whole survey.
Ideally a question’s design should lessen the risk of any two respondents interpreting questions in different ways. Therefore, the different answers received should only reflect the genuine differences in respondents’ characteristics, behaviour or opinions.
Appendix 1 summarises these guidelines in the form of an easy to use checklist.
Avoid ambiguous terms or phrasing
Example – Have you visited a dentist recently?
The term recently is ambiguous and different respondents may interpret it in different ways. A regular visitor to the dentist may assume that recently refers to the last few months; while a less frequent visitor might consider that it refers to the last two years. A better question would be to ask the respondent if they had visited the dentist in the last six months, or whatever period you define as recent.
Avoid technical terms
Example – How do you rate the street furniture in your local area?
The expression street furniture is a technical term used by those involved in highway maintenance. When composing a question, it is important to keep your audience in mind.
If a question is intended for the public or a non-specialist audience then technical terms should either be changed or defined separately beforehand.
Do not use unnecessary negatives
Example – In the last 12 months, have you not had any missed refuse collections?
Do not ask leading questions
Example – It is recommended by the World Health Organisation that people should do 30 minutes of gentle exercise each day to sustain a minimum level of fitness. How long have you spent exercising in the last week?
Preceding statements or the wording of the question itself could inadvertently influence the respondent. The purpose of any such statements should always be obvious and they should not be phrased in a way that could affect the respondent’s answer. In this example, the opening statement may encourage respondents to overestimate the time they spent exercising.
Do not ask more than one question at a time
Example – How satisfied were you with the opening hours and access to your local civic amenity site?
This question effectively asks two questions: how satisfied the respondent was with the opening hours; and how satisfied they were with the access to the site. So, the question should be split into two separate questions to avoid any ambiguity and confusion for the respondent. If only one general question is intended, then a more suitable term should be used instead of opening hours and access to.
Local Government Data Unit ~ Wales
The not in this question is completely unnecessary and confusing. Double negatives in questions make them a lot more difficult to understand.
3
Questionnaire Design
Local Government Data Unit ~ Wales
Avoid overly long and complex questions
Example – Do you support the local authorities’ initiative to improve cleanliness and prevent littering in Wales by imposing ‘on-the-spot’ fines of £75 on those guilty of littering the streets, including the disposal of cigarette butts?
This question could be improved by providing a separate statement defining the initiative before asking whether the respondent agreed with it. A better example would be: In an effort to improve cleanliness and prevent littering, local authorities in Wales can now impose an ‘on-the-spot’ fine of £75 to those guilty of littering streets. This includes disposing cigarette butts. Do you support this initiative?
Overly long or complicated definitions should ideally be avoided as well.
Ensure that the you in any question is clear about who it is referring to
Example – How long have you lived at this address?
The you in this sentence could refer to the respondent or the respondent’s household or family. Simply substituting you with you personally or your household would introduce the appropriate context and provide clarity.
Avoid asking questions where respondents need to do mental arithmetic
Example - What proportion of your week do you usually spend at work?
Although the respondent is likely to know their usual working hours each week, to answer this question they are required to perform some tricky mental arithmetic. This information could be obtained from a simpler question about usual working hours instead. Avoid asking questions of proxies
Example - How easy or difficult is it for your child to get to and from their school or college? Asking the respondent questions about another person, regardless of whether the questions are factual or non-factual, is not ideal. Including a filter question beforehand, to ask if the respondent feels that they are in a position to comment, might help. Don’t know and Not applicable answer options are also important to try and ensure that only those who genuinely can answer on behalf of someone else respond.
Provide exhaustive answer options
Example – How do you usually travel to work?
Train
Car
Motor-cycle, scooter or moped
Taxi
Bicycle
On foot
An obvious omission from this example’s answer option list is bus. It might also be worthwhile including an Other – please specify option for respondents to record any answers not included in the list.
4
Questionnaire Design
Avoid using more answer options than necessary
Example – What is the main reason that you have not used public transport in the last seven days?
Takes too long
Inconvenient
No direct route
Use my own car/ get lifts
Need a car for or at work
Too expensive
Work unsocial or unusual hours
Public transport is unreliable
Lack of service
Too infrequent
Health reasons
Difficult access or on & off steps/Disability
Too much to carry
Uncomfortable
Prefer to walk
Prefer to cycle
Dislike waiting
Long walk to bus-stop
Live centrally or within walking distance
Safety
Have been away
No need
The number of answer options for each question depends on its purpose, the type of question and the type of survey.
Long lists of answer options, as in the example above, can create primacy (visual) or recency (aural) effects. These can cause a bias in the survey results and so must be avoided. Within telephone or face-to-face interviews, the last few options read out may be more memorable to the respondent than earlier answer options and would be selected more regularly. This is called a recency effect. For the previous example, in a telephone or face-to-face interview, the respondent may be more likely to select the no need answer option. The primacy and recency effects are increased as the number of answer options increase, although it is possible to keep all options on the list and reduce bias. Different versions of the questionnaire would be used within the survey – with the answer option lists reversed or randomised. Another option is to ask separate Yes/No questions for each answer option, although this approach does increase the survey burden.
In the previous example, the number of answer options could be reduced to five or six of the most likely answers and an Other – please specify option added to capture any answers that are not listed. In an interview survey, it would also be possible to ask this question without prompting the respondent with answer options, as an open question.
A long list of answer options could be provided for the interviewer only. They could then code the responses themselves. However, caution should be exercised when asking interviewers to become coders, particularly with long option lists.
Local Government Data Unit ~ Wales
For self-completion questionnaires, the answer options at the top of a long list may be selected more regularly than the alternatives listed afterwards. This is called a primacy effect. For the example above, in a self-completion context, the respondent may be more likely to select the takes too long answer option at the expense of the other options.
5
Questionnaire Design
Local Government Data Unit ~ Wales
If a very large number of answer options is needed, the question may be too open and it may be necessary to rephrase the question to make it more specific. Alternatively, the question could be split into a number of separate questions with fewer options listed for each. For the previous example, the answer options could be split into two categories: for personal reasons and service reasons. In the question below - a generic poor public transport service reason is included within a list of personal reasons; this particular option could then provide routing to a second question asking respondents specifically about why public transport is unsuitable.
Example - What is the main reason that you have not used public transport in the last seven days?
Poor public transport service
Use my own car/get lifts
Need a car for or at work
Work unsocial or unusual hours
Health reasons
Disability
Too much to carry
Prefer to walk
Prefer to cycle
Live centrally or within walking distance
Have been away
No need
(If the respondent answers poor public transport service, they get asked the following question.) What is/are the main reason/s why you consider public transport to be poor?
Inconvenient
Too expensive
Lack of service
Too infrequent
Public transport is unreliable
No direct route
Takes too long
Difficult access or on and off steps
Uncomfortable
Safety
Long walk to bus-stop
Using more than three or four answer options in telephone interviews should be avoided.
For face-to-face interviews the number of answer options can be increased by using a showcard. For both showcards and self-completion questionnaires, lists of more than 10 answer options should be avoided wherever possible.
6
Questionnaire Design
Ensure that the answer options create a suitable frame of reference for the respondent Example - How long did you spend watching television yesterday?
Answer option list 1:
Did not watch any television yesterday
Less than 10 minutes
At least 10 minutes but less than 30 minutes
At least 30 minutes but less than 1 hour
At least 1 hour
Answer option list 2:
Did not watch any television yesterday
Less than 2 hours
At least 2 hours but less than 4 hours
At least 4 hours but less than 6 hours
At least 6 hours
Answer option lists can provide an important source of information for the respondent, so great care should be used to determine which answer options are appropriate. In this example, the answer option lists might be interpreted by the respondent as representing everyone else’s television watching habits. And so, the respondent’s answer choice may be influenced by its position in the list.
The answer options can affect the respondent’s interpretation of ambiguous terms in the question. Using Answer option list 1, a respondent may be more inclined to exclude time spent watching DVDs or any time spent watching television while doing something else at the same time.
Ensure appropriate use of don’t know, no opinion and not applicable answer options Yes
No
Don’t know
Don’t know answer options can sometimes provide an easy (and unnecessary) option for the respondent, as in the previous example.
However, it is important that the respondent is always able to answer the question given to them. This is achieved by providing answer options that adequately represent their responses and, if necessary, including don’t know or no opinion options. The worst possible outcome is that respondents have to provide false answers just to continue through the questionnaire.
Although it is very unlikely that a question would include both don’t know and no opinion answer options, it is important to note that they are not equivalent. Both don’t know and no opinion options can be replaced by a preceding filter question if necessary.
Do not reinvent questions
Question banks are valuable tools used to identify possible questions to be included within a questionnaire. Use these to see whether other surveys have already developed and tested questions that meet your objectives. See Appendix 4 for further details.
Local Government Data Unit ~ Wales
Example – Do you have use of a car for activities such as visiting local shops or going to the doctor?
7
Questionnaire Design
Local Government Data Unit ~ Wales
5.
Factual question design guidelines
Questions which ask about facts, past behaviour or events are factual questions; obvious examples are questions on age, sex or where people work. In addition to using the general question guidelines, you should consider the respondent’s ability to answer the question effectively, and whether their memory is a reliable source of information.
Appendix 1 summarises these guidelines in the form of an easy to use checklist.
Provide definitions for unfamiliar concepts
Example - Have you used or contacted anyone about your local recycling facilities during the last 12 months? By local recycling facilities we mean your kerbside collection and other sites such as bottle banks and civic amenities (e.g. local tip).
When asking a question about an unfamiliar concept, it is important to remind the respondent what that concept means by providing a clear, short definition.
If recalling an answer could be difficult, use answer option lists to help the memory Example - Which of the following sources, if any, did you last use to obtain information on train services in Wales?
Train operating company
Traveline
Timetable at train station
Train station staff
National Rail Enquiries
Train Times
Other
Can’t remember
None of these
In this example, the answer option list helps to prompt the respondent to remember what source they used. During face-to-face interviews the respondent should be shown these lists on showcards.
When measuring frequency, ensure that the behaviour or event is regular
Example – About how often have you visited a GP surgery in the last 12 months?
Have
Once
Once
Once
Once
Once
not visited every two weeks a month a quarter every six months a year
In this example the respondent may have visited a GP surgery a few times during the year but all within the same month. There is no appropriate answer option listed as the behaviour is not regular.
8
Questionnaire Design
Ensure that the factual information being recalled is within a realistic time period Example - When did you last use a public footpath?
Within the last week
At least a week ago, but within the last month
At least a month ago, but within the last three months
At least three months ago, but within the last six months
At least six months ago, but within the last year
At least a year ago
Can’t remember
A respondent who is not a frequent user of public footpaths is unlikely to give a meaningful answer to this question. Whether the respondent can effectively recall an event depends upon: how significant the event was; how long ago it took place; and its relationship to other events. Recalling precisely when some events occurred, particularly insignificant ones, is difficult for all – so don’t expect accuracy.
Local Government Data Unit ~ Wales
9
Questionnaire Design
Local Government Data Unit ~ Wales
6.
Non-factual question design guidelines
Questions asking respondents’ opinions are considered to be non-factual questions.
There is greater freedom when designing these questions than with factual questions.
However, it is more difficult to ensure that you receive a meaningful response.
Appendix 1 summarises these guidelines in an easy to use checklist.
Ensure that agree/disagree statements are extreme/strong statements
Example - To what extent do you agree with the following statement?
Statement 1: I would have some reservations about having a gypsy/traveller site located near to my home
Statement 2: I would have no objection to a gypsy/traveller site being located near to my home
Strongly agree
Tend to agree
Neither agree nor disagree
Tend to disagree
Strongly disagree
Less extreme statements, such as Statement 1, do not make it clear what people who disagree strongly with the statement are thinking. For example, someone could disagree strongly as they have absolutely no reservations or because they have major reservations. The second statement is more extreme, and it is therefore easier to interpret that people who ‘disagree strongly’ with the statement do have an objection to a gypsy/traveller site being located near their homes.
Less extreme statements can also cause the respondent to merely comply; where they simply agree with reasonable sounding statements to complete the question quickly. A provocative statement encourages the respondent to contemplate the statement more carefully. Use both positive and negative statements when including more than one agree/disagree statement
Example - To what extent do you agree with the following statements?
Statement 1: This neighbourhood has a strong sense of community
Statement 2: I don’t consider myself similar to the people who live in this neighbourhood
Strongly agree
Tend to agree
Neither agree nor disagree
Tend to disagree
Strongly disagree
Using both positive and negative statements prevents the respondent from getting into a habit of simply answering each one positively. It helps to ensure that the respondent evaluates each statement on its own merits.
10
Questionnaire Design
Ensure that questions with ranking do not increase survey burden excessively
Example – Please rank the following aspects of life in their order of importance to you.
Order them from 1 to 8, where 1 is the most important and 8 the least.
Your health
Having a lot of money
Having children
Having a fulfilling job
Being independent
Owning your own home
Having a good marriage or partnership
Having good friends
Asking the respondent to rank statements can provide useful information, but also increases the effort required from the respondent. It is better to use a short list of only four or five items, unlike the example above. Alternatively, using only partial ranking
- asking the respondent to rank their top three only, for example, can also reduce the burden – as long as the list to be considered is not too long.
In a face-to-face interview, it is possible to give the respondent a set of cards with a statement on each card. When the respondent is asked to order them, it is much easier than on a self-completion survey. For a self-completion survey, the respondent needs to consider all statements on the list before deciding on every position.
Note that satisfaction questions can often lead to a high proportion of satisfied answers Example – Overall, how satisfied have you been with the time you have had to wait for a dental appointment?
Respondents can often be easily satisfied, either because they want to complete the question quickly, or they already have low expectations. An alternative approach is to replace satisfaction questions with two sets of questions: the first set asking how important a subject is to the respondent; and a second set asking about their experience of it. This could provide more meaningful and specific information.
Avoid asking hypothetical questions
Example – If the frequency of buses at your local bus stop was increased, would you use the bus more often?
Good responses are not usually achieved by asking hypothetical questions because of the limited experience respondents have for their answer. From the example question, it would only be possible to comment on what people say rather than the actual relationship between bus frequency and usage. Such questions may also lead to the respondent seeking more clarification: increased to what frequency?
Local Government Data Unit ~ Wales
In this example, if the respondent expects to have waited a long time for an appointment they may be tempted to say that they are satisfied. So this answer may be given in spite of the fact that they would really approve of shorter waiting times. Obtaining factual information might be preferable here: How long was the interval between seeking an appointment at the dentist and your appointment?
11
Questionnaire Design
Local Government Data Unit ~ Wales
Think carefully about whether to include a middle answer option category
Example – How satisfied were you with the outcome of your complaint?
Answer option list 1
Very satisfied
Fairly satisfied
Neither satisfied nor dissatisfied
Fairly dissatisfied
Very dissatisfied
Answer option list 2
Very satisfied
Fairly satisfied
Fairly dissatisfied
Very dissatisfied
If the question’s purpose is to discover where the majority of opinion falls then whether you use a four or five point scale is not important. Research has shown that those who choose the middle answer option category are typically drawn from both sides of the scale. That is, both Answer option list 1 and Answer option list 2 are equally valid in the previous example.
When neither a positive nor a negative response applies to the respondent, excluding a middle answer option category can increase the survey burden. Don’t know, no opinion and not applicable answer options should not be considered part of the scale.
Use consistent and balanced answer option scales
Example – How would you rate your local neighbourhood as a place to bring up children?
Answer option list 1
Answer option list 3
Very good
Fairly good
Neither good nor poor
Fairly poor
Very poor
Good
Neither good nor poor
Poor
Answer option list 2
Answer option list 4
Very good
Good
Neither good nor poor
Quite poor
Truly terrible
Very poor
Fairly poor
Neither good nor poor
Fairly good
Very good
When asking respondents to give a rating you should use a balanced answer option scale; Answer option list 1 is a good example of this. If the scale is not balanced then it is difficult to interpret and more of a burden. See Answer option list 2.
You should use a consistent answer option scale throughout a questionnaire, that is use either Answer option list 1 or Answer option list 3 but not both. You must also be careful to ensure that every occurrence of a similar scale is in the same order within a questionnaire. Using Answer option list 1 in one question and then Answer option list 4 in another would just confuse the respondent.
12
Questionnaire Design
7.
Sensitive question guidelines
A sensitive question is a question that the respondent may feel uncomfortable answering. Typical examples include: questions on income; savings; or sexual habits.
These guidelines give advice on how to create conditions where the respondent may feel more comfortable answering sensitive questions. The aim is to minimise the occasions when the respondent refuses to answer the question.
Appendix 1 summarises these guidelines in an easy to use checklist.
Reassure the respondent of confidentiality and remind them of the purpose of the sensitive questions before asking them
Example – The answers you give to these questions, as with the rest of the survey, will be treated in the strictest confidence, and used for research purposes only.
Before a series of sensitive questions, it is advisable to remind respondents that their answers are confidential, to encourage honest and complete responses. The example statement is quite generic and perhaps the specific purpose of the research should also be made clear.
Ensure sensitive questions are placed towards the end of questionnaire
If sensitive questions are placed at the start of the questionnaire, then any refusal to answer these questions is likely to affect the completion of subsequent questions.
Consider using showcards within face-to-face interviews
Example – From this card, please read out the letter alongside the total amount of your savings and money invested?
Showcard 1
For face-to-face interviews, a showcard can be referenced (as in the example above) so that the respondent only has to state a letter or number, rather than the exact amount or relevant banding.
Consider using a self-completion element within face-to-face interviews
As part of a face-to-face interview, the respondent could be given a self-completion form.
This could be completed and then sealed in an envelope before handing it to the interviewer. Alternatively it could be left to be completed later in private, with a freepost or pre-paid reply envelope. This has the effect of reducing the interviewer’s role and any influence their presence has on the responses given.
Local Government Data Unit ~ Wales
A Under £1,000
B £1,000 - £4,999
C £5,000 - £9,999
D £10,000 - £19,999
E £20,000 - £49,999
F £50,000 or over
13
Questionnaire Design
Local Government Data Unit ~ Wales
8.
General questionnaire design guidelines
These guidelines are common to each type of survey: postal (or self-completion), faceto-face interviews and telephone surveys. They are guidelines to help you think of the questionnaire as a whole, and not just as a series of questions.
Appendix 2 summarises these guidelines in an easy to use checklist.
Introduce the questionnaire/interview
For all surveys, the introduction should:
•
•
•
state the name of the organisation responsible for the research; explain the purpose and importance of the research; and emphasise confidentiality.
In addition, for postal (or self-completion) surveys, the introduction should:
•
•
•
give instructions on how to complete the questionnaire; give instructions on how to return the questionnaire; and thank the respondent for their time.
Ensure that the first question is appropriate
The first question should be:
•
•
•
relevant to all respondents; easy to answer; and clearly related to the questionnaire subject.
Ensure that the opening set of questions is easy to answer
The opening questions should encourage the respondent to complete the rest of the questionnaire. In principle they should be easy, short, interesting and applicable to every respondent.
Ensure that questions have a meaningful order
It is important that questions on a common subject are grouped together. In theory, questions of the same format should also be grouped, but this is a secondary consideration. The subject of self-completion questionnaires should be quite focused, but telephone or face-to-face interviews can cover a wide range of subjects.
Provide clear section separation
Example – Now I would like to ask you some questions about the environment.
Where more than one subject is addressed in a questionnaire, the transition between sections should be made obvious to the respondent. This can help to give the respondent a sense of progress as they work through the questions. In long questionnaires, you can also include encouragement towards the end, such as We are nearly at the end now, just a couple more questions.
For self-completion questionnaires the section separation can be achieved with a simple heading. Within face-to-face interviews or telephone surveys a sentence should be read out, such as the previous example statement, before asking questions on a new subject.
14
Questionnaire Design
9.
Self-completion questionnaire format guidelines
The self-completion questionnaire’s format is very important. An effective format reduces: survey burden for the respondent; the number of mistakes a respondent makes while completing the form; and the number of incomplete responses. It also encourages respondents to complete the questionnaire. There is little merit in designing a series of insightful questions only for the respondent to struggle to complete the questionnaire because of the format in which they are presented.
Appendix 2 summarises these guidelines in an easy to use checklist.
Appendix 3 shows an example of a formatted self-completion questionnaire.
Number each question
The questions should be numbered consecutively starting from 1. It is best to avoid using prefacing letters, such as A1, A2, B1. Use a letter suffix to denote related questions, such as 1a or 1b. Question numbers should be included to help the respondent navigate around the questionnaire rather than for the researcher’s purposes.
Apply a common format to each question
The format of each question should:
•
•
•
visually distinguish between question wording and other instructions; have a logical ordering of instructions, definitions and answer options; and have a consistent method for marking the selection of answer options.
Example –
1
What is your sex?
Tick one box only
Male
Female
Include clear completion instructions
Examples -
Tick one box only
Tick as many as apply
Circle the appropriate number
Write in the number
Enter the date in the format dd/mm/yyyy
It needs to be clear to the respondent how they should mark their answers and what type of response is expected. This is true for each individual question regardless of whether or not similar types of questions are grouped. It is very important to make this clear where multiple responses are needed. Failure to give appropriate completion instructions can considerably increase the survey burden.
Local Government Data Unit ~ Wales
In this example the question is in bold and is a larger size text than the completion instructions below. The tick boxes for the answer options are aligned and they are of equal size. The question number is highlighted. The question would be clearly separated from the previous and next question by the surrounding lines.
15
Questionnaire Design
Local Government Data Unit ~ Wales
Include clear routing instructions
Example –
1
Have you used the local bus service in your area within the last 12 months?
Tick one box only
Yes
Go to
2
No
Go to
3
The routing included in any self-completion questionnaire must be relatively simple and the routing instructions themselves need to be easy to follow. Including arrows as well as text to visually prompt the respondent can also be beneficial.
Separate answer option scales from don’t know, no opinion and not applicable answer options
Example –
2
How easy is it to obtain information about local bus services in your area?
Tick one box only
Answer option list 1
Very easy
Fairly easy
Fairly difficult
Very difficult
Don’t know
Answer option list 2
Very easy
Fairly easy
Fairly difficult
Very difficult
Don’t know
In this example, the stack of five answer options in Answer option list 1 creates the impression that the middle answer is Fairly difficult, however, there is no conceptual middle for the scale. A simple way to remedy the problem is to include a space or a line between the scale and any Don’t know or Not applicable answer options, as shown in
Answer option list 2.
16
Questionnaire Design
Do not split questions and answer options across pages
If questions or answer options are split across pages then it is possible that the respondent will not read the whole question before answering. If a large amount of white space remains on a page it may be worth including an instruction to continue to the next sheet, to preserve the flow of the questionnaire.
Provide appropriate space for ‘write in’ answers
The area where respondents write answers to open-ended questions or Other – please specify options should be big enough for the length of desired response. The size of the space will indicate to the respondent the researcher’s expectations and will influence their response accordingly.
At the end of the questionnaire, thank the respondent and include instructions on how to return the completed questionnaire
Although return instructions should have been included in the questionnaire’s introduction, it is usually worth repeating them at the end. Even if there is a prepaid envelope, it is always sensible to include the postal address (preferably Freepost) at the end of the questionnaire. This ensures that if the envelope is lost or not included, the respondent can still submit their questionnaire.
Local Government Data Unit ~ Wales
17
Questionnaire Design
Local Government Data Unit ~ Wales
10.
Question testing
When you have developed your questions and questionnaire using the good practice guidelines, you should test both of these. There are various methods that can be used to test questions and questionnaires. Some are quick and inexpensive, such as reading the questions out loud or asking your colleagues to complete the questionnaire. Other methods, for example conducting a pilot study, are more expensive and time consuming
– but an essential stage of questionnaire development.
A few methods that are used to test questions and questionnaires are summarised below. The nature of your survey and resources available to you will dictate which of the testing methods you should adopt. Note that each method has its own advantages and disadvantages, and large-scale surveys typically use more than one method.
A peer review
This involves asking a number people who are involved with the survey subject, or with questionnaires in general, to review it. A peer review can range from sending the questionnaire informally to colleagues, to asking a panel of experts to review it. Typically this is an inexpensive method for testing the questionnaire, which can highlight issues with it and offer solutions. The disadvantage of using a peer review is that you are not testing the questionnaire on people who are similar to those who will ultimately complete the survey.
Cognitive interviewing
Cognitive testing is a method of interviewing used to understand how people answer survey questions. It can be used to find out if a question is working as intended and whether respondents can answer it correctly. The questionnaire is completed by a testrespondent and then an interviewer questions them on how or why they have answered the questions in this particular way. A number of respondents should be interviewed and they should be representative of the survey population.
Cognitive interviewing can uncover problems that would go unnoticed when the survey was in the field, and offer solutions. This method only needs a small number of respondents and it is relatively cheap. If a problem is observed by one person, the survey can be changed in time for the next cognitive interview. However, disadvantages of this approach are that: the interpretation of responses by the interviewer can be subjective; and it does require some specialist training to undertake the test properly.
Focus group discussions
A focus group test is similar to a cognitive interview, but in a group situation, with a discussion leader or moderator and dedicated note-taker. This would normally involve around five to eight participants who can adequately represent the survey population.
They would complete the questionnaire, while the moderator observes any difficulties encountered. Once the forms are completed, any mistakes or missed questions can be analysed. This is followed by debriefing the respondents, involving pre-planned questions and those based on observation. Typically, debriefing questions ask about the participants’ interpretation of terms and how they came up with their answers.
Like cognitive interviewing, focus group discussions can uncover hidden problems and offer solutions. Again, it only needs a small number of respondents and is relatively cheap. The main disadvantage of using focus group discussions is that it is not easy to know if an observed problem was a significant one or not. Sometimes vocal individuals in the group can unduly influence others; problems encountered by only one individual are noticed and commented on by everyone in the group.
18
Questionnaire Design
Field/Pilot testing
A field test or pilot survey is a very small-scale survey which should replicate the exact conditions of the full survey to follow. The advantages of this are that it will immediately highlight any issues with: the routing in the questionnaire; if there are a lot of unanswered questions; and the don’t know and no opinion answer options can be reviewed. However, it can be very expensive. If it is a pilot face-to face or telephone survey, an interviewer and possibly a respondent debriefing session should follow. Advice on how to pilot a survey can be found in Survey Design and Analysis.
Interview debrief
The debriefing session normally involves the interviewers/participants reading through the questionnaire slowly and the researcher asking interviewers/participants if they encountered any problems with each question.
A debrief can identify which questions were difficult to read or understand – from both the interviewer’s and respondent’s point of view. For any problems that are highlighted, proposed solutions can also be suggested and discussed.
Behaviour coding
Behaviour coding is conducted through observing the respondent’s actions. This method is usually undertaken by a coder and mainly used for face-to face surveys, where the coder can more easily observe the interviewer and respondent and see how they interact.
They can note whether an interviewer asks the questions exactly as worded, whether the respondent asks for clarification or takes a long time to answer. Behaviour coding can detect problems in questions, is relatively cheap, but it does not suggest ways to solve the problems. For telephone surveys, the average delay between the interviewer asking the question and entering an answer can be calculated: those questions with longer time gaps should be examined carefully to see if they are complicated or difficult to understand, with a view to simplifying them.
Local Government Data Unit ~ Wales
Regardless of which question testing method you use, it is necessary to repeat the review process each time changes are made to the questionnaire. This is necessary to ensure that the changes made result in genuine improvements, and do not introduce any new problems. 19
Questionnaire Design
Local Government Data Unit ~ Wales
Appendix 1 – Question design checklist
General question guidelines
Questions do not include: ambiguous terms or phrasing technical terms unnecessary negative phrasing overly long lists of answer options any ambiguity with reference to you as a singular or a collective
Questions do not require: respondents to do mental arithmetic
Questions are not: leading overly long
Questions do not ask: more than one question at a time the respondent about someone else or someone else’s opinions
Questions provide: exhaustive answer options an appropriate frame of reference for the respondent appropriate Don’t know, No opinion and Not applicable answer options
Factual question guidelines
Questions include: definitions for unfamiliar concepts answer option lists when recall is difficult
Questions do not: make unreasonable assumptions that behaviour or events are regular require unrealistic factual recall
Non-factual question guidelines
Questions include: extreme or strong agree/disagree statements both positive and negative agree/disagree statements
Questions do not: require ranking that excessively increases the survey burden ask about hypothetical situations
Sensitive question guidelines
Questions are: preceded by a reminder of confidentiality preceded by a reminder of the purpose of the research located towards the end of the survey
20
Questionnaire Design
Appendix 2 – Self-completion questionnaire checklist
General questionnaire guidelines
Questionnaire has: an introduction an appropriate first question an opening set of questions that are easy to answer a meaningful order to the questions a clear section separation
Self-completion questionnaire format guidelines
Questions have: a number a common format clear completion instructions clear routing instructions appropriate space for ‘write in’ answers a format that distinguishes Don’t know, No opinion and Not applicable options from lists of scale answer options
Questions are not: split across pages
Questionnaire has at the end: a thank you statement instructions on how to return the completed form
Local Government Data Unit ~ Wales
21
Questionnaire Design
Local Government Data Unit ~ Wales
Appendix 3 – Example self-completion questionnaire
Library Services
We are interested in finding out how you feel about the service you received from us today. The information you give will be used to help us to improve the way we deliver our services.
All the information you provide will be kept completely confidential. No identifiable information about you, your home or your household will be passed on to any other bodies, members of the public or press.
Please return the completed survey to reception or any member of staff. Thank you for your help.
About the library:
4 Did you use a computer during your visit to the library today?
1 Did you visit the library
Tick one box only
today to borrow books?
Yes
Tick one box only
Yes
Go to
2
No
Go to
Go to
5
No
Go to
6
4
5 How do you rate the computer
2 How do you rate the choice of books at this library?
Tick one box only
facilities at this library?
Tick one box only
Very good
Very good
Good
Good
Adequate
Adequate
Poor
Poor
Very poor
Very poor
6 In the last three months, which
3 How do you rate the condition of books at this library? Tick one box only
of these library facilities have you used? Tick as many boxes as apply
Audio books
Very good
Newspapers
Good
Photocopier
Adequate
Other, write in below
Poor
Very poor
22
Questionnaire Design
7 Please rate the following aspects of this library
Tick one box in each row
Very good Good
Adequate
Poor
Very poor Provision of seating and tables
Staff helpfulness
Opening hours
The library overall
About you:
10 Can you understand, speak, read
8 What is your sex?
or write Welsh?
Tick one box in each row
Tick one box only
Yes
Male
No
Understand spoken Welsh
Female
Speak Welsh
Read Welsh
9 Which age group applies to
Write Welsh
you?
Tick one box only
0-24 years
25-44 years
65-74 years
75+ years
Thank you for completing this questionnaire.
Please return it to reception or any member of staff.
Local Government Data Unit ~ Wales
45-64 years
23
Questionnaire Design
Local Government Data Unit ~ Wales
Appendix 4 – Harmonised questions and question banks
The Survey Question Bank http://surveynet.ac.uk/sqb/about/introduction.asp The Survey Question Bank (SQB) is co-ordinated by the UK Data Archive (UKDA) at the
University of Essex. It provides a suite of online research resources with a specific focus on survey methods and can be used to locate, and view in context, survey questions as they were used in the data collection process.
This website holds the questionnaires for large-scale surveys which have a nationally representative sample and are generally conducted by a large and professional survey organisation. All the surveys that are held by the SQB are listed by survey name or topic and include:
•
•
•
an overview of the survey; methodology; and main topics that have been covered.
The SQB also contains links to relevant datasets.
Harmonised Concepts and Questions www.statistics.gov.uk/harmonisation The Harmonised Concepts and Questions booklet, which is available through the Office for National Statistics website is a very good source of questions and associated definitions that are used in government surveys. In addition to the questions, some information is given on the output categories for some variables, such as age, which will then allow comparison of information from one survey to another.
24
Questionnaire Design
Local Government Data Unit ~ Wales
25
Questionnaire Design
Local Government Data Unit ~ Wales
Local Government Data Unit ~ Wales
3-7 Columbus Walk, Cardiff, CF10 4SD
Telephone: 029 2090 9500
Email: enquiries@dataunitwales.gov.uk
Web: www.dataunitwales.gov.uk
26 