Tips for Writing Questionnaire Items
Office of Educational Assessment Last Updated: October 2006
This handout is provided by the UW Office of Educational Assessment (OEA) as a resource for WebQ workshop attendees who want some basic tips for writing questionnaire items. It does not address broader aspects of survey and research design, but additional resources can be found on our web site: http://www.washington.edu/oea. OEA also provides a variety of consulting and research support services; please feel free to contact us if you are interested in assistance with a research, assessment, or evaluation project (baytam@u.washington.edu, 206‐543‐5190).

TIP 1: BE RELEVANT
Before starting your survey, you should specify one or several over‐arching questions you wish to address (i.e., research questions or evaluation questions). Your questionnaire items should be focused around these questions.

Different Domains of Questions: Behavior, Beliefs, and Evaluation
Does providing optional, online practice quizzes help students learn the material in my course? Behavior Of the ten optional online practice quizzes, how many did you complete? Completing the online practice quizzes helped me learn the material in this course. How satisfied were you with the online quizzes?
0 1–2 3– 5 6–7 8– 9 10

Beliefs

Strongly Disagree Extremely Dissatisfied

Disagree

Neutral

Agree

Strongly Agree Extremely Satisfied

Evaluation

Somewhat Dissatisfied

Somewhat Satisfied

The three items above have different foci. The first could help answer your research question if you were able to correlate responses to the item with actual grades.

Copyright © 2006 University of Washington Office of Educational Assessment

Different Types of Questions: Open- and Closed-Ended
How do transfer students benefit from going to “Transfer Thursday” sessions in the Gateway center? Closed‐ended: Which of the following do you see as benefits of attending Transfer Thursday sessions? (check all that apply) o o o o Open‐ended: Learning about major requirements Meeting with a UW adviser Meeting other transfer students Other (specify): __________________

What do you see as the benefits of attending Transfer Thursday sessions?

Data from an open‐ended question can help you create “check all that apply” lists or other closed‐format items. The primary drawback of open‐ended questions is the time it takes to analyze the data; if resources are limited, use open‐ended questions sparingly.

TIP 2: BE SPECIFIC Avoid abstract terms and jargon
BAD GOOD

The in-class activities in this class facilitate active learning. The in-class activities (e.g., illusory correlation demonstration, eye-witness testimony activity) help keep me engaged in class.

Strongly Disagree Strongly Disagree

Disagree

Neutral

Agree

Strongly Agree Strongly Agree

Disagree

Neutral

Agree

In the revision, the terms “in‐class activities” and “active learning” have been specified.

Provide clarifying details
BAD GOOD

How many times did you visit CLUE? How many times did you visit the Center for Learning and Undergraduate Enrichment (CLUE) -evening tutoring in Mary Gates Hall?

Write out acronyms or abbreviations the first time. Provide further descriptors if appropriate.

Frequency estimates
BAD GOOD

How many times do you visit CLUE? During a typical quarter, how many times do you visit CLUE?

Never Never

Not often 1-3 times

Fairly Often About every other week

All the time About once per week 2 or more times per week

The less definite the response options, the more ʹnoiseʹ or error in the data (i.e., participants will vary in their interpretation of the options).

UW OEA: Tips for Writing Questionnaire Items

2

TIP 3: AVOID CONFUSION Avoid double-barreled questions
BAD GOOD

The TAs in this course were helpful and responsive.

Strongly Disagree

Disagree

Neutral

Agree

Strongly Agree

Rate the performance of the TAs on the following dimensions. Helping you understand the material Responding to inquires about your grade
Poor 1 Poor 1 Excellent 5 Excellent 5

2 2

3 3

4 4

Avoid double-negative wording
BAD GOOD

The instructor for this course did not respond to students' comments during class. The instructor for this course was response to students' comments during class.

Strongly Disagree Strongly Disagree

Disagree Disagree

Neutral Neutral

Agree Agree

Strongly Agree Strongly Agree

In the first case, students who thought the instructor was responsive (a positive trait) would have to choose strongly disagree (a negative answer). Watch out for such mental puzzles.

TIP 4: USE APPROPRIATE SCALES Provide a spectrum of options
BAD GOOD

Did you enjoy the interactive online activity about Newton's 3rd Law? To what extent did you enjoy the interactive online activity about Newton's 3rd Law?

No Not at all

Yes A little Somewhat Very much

Typical scales
Agreement Satisfaction Evaluation
Strongly Disagree Extremely Dissatisfied Poor Very unfair Not at all important Disagree Dissatisfied Fair Unfair Not too important Not too familiar Neutral Somewhat Dissatisfied Good Fair Somewhat important Somewhat familiar Agree Somewhat Satisfied Very good Very fair Very important Very familiar Strongly Agree Satisfied Excellent Extremely Satisfied

Knowledge

Not at all familiar

UW OEA: Tips for Writing Questionnaire Items

3

Unipolar vs. Bipolar scales
Unipolar Bipolar
None Extremely negative Some Very negative Very Somewhat negative Extremely Somewhat positive Very positive Extremely positive

Often, the presence of one attribute does not necessitate the absence of another. For example, one can enjoy both arts and sciences, so they should not be presented with a scale that forces them to choose one or the other. Beware of imposing a structure or model that may not be valid.

Providing a (neutral) midpoint
With midpoint Without midpoint
Extremely negative Extremely negative Very negative Very negative Neither negative nor positive Somewhat negative Somewhat positive Very positive Very positive Extremely positive Extremely positive

Opinions vary regarding the appropriateness or usefulness of a midpoint, particularly for agreement scales. Research suggests a forced response will yield roughly the same proportion of agree/disagree as a scale with a midpoint. For example: Results with a midpoint: Results without a midpoint: 30% con, 50% neutral, 20% pro 60% con, 40% pro

Number of response anchors
Should one label all of the response options? In general, the more specific you are about the response options, the less you will have to worry about idiosyncratic interpretations of those options by the respondents. In some instances (e.g., frequency estimates) it is imperative that you label all options. In other cases, you could increase error by trying to ʹforceʹ a label. Whatever system you choose, make sure that your labeling is symmetrical (e.g., do not label only ʺ1ʺ and ʺ2ʺ and ʺ5ʺ on a 5‐point scale).

N/A: Use sparingly!
NO

I would have completed the online practice quizzes, even if they did not count towards my grade. The optional online practice quizzes improved my grade on the midterm.

Strongly Disagree Strongly Disagree

Disagree

Agree

Strongly Agree Strongly Agree

N/A

OK

Disagree

Agree

N/A

Another alternative would be to direct respondents to skip a set of items if they had not completed any of the optional online practice questions, making the N/A option unnecessary. ʺNot applicableʺ and ʺdonʹt knowʺ should be avoided for opinion and attitude items.

"Check all that apply" lists
Make lists as exhaustive as possible. Use previously collected open‐ended data if available. Unless all options are known, it is almost always appropriate to provide a space for a write‐in ʺotherʺ response.

UW OEA: Tips for Writing Questionnaire Items

4

TIP 5: AVOID BIAS Avoid leading questions
BAD IFFY GOOD BEST

The UW is considering having wireless capacity for the entire campus within 3 years. Given the potential impact on students' behavior in the classroom, what do you think of this proposal? How do you think all-campus wireless capacity will affect students' behavior in the classroom? How, if at all, do you think all-campus wireless capacity will affect students' behavior in the classroom? Do you think all-campus wireless capacity will affect students' behavior in the classroom? Definitely not Possibly Definitely yes If yes or possibly, in what ways do you think it will affect students' behavior?

In the first example, there is an implicit assumption that expanded capacity will affect students’ behavior in class. The second wording is better, but it still does not make clear that respondents are free to deny any effect; the third wording provides that option. In the last example, the addition of a closed‐format introductory question allows for easy tabulation of overall opinions from the open‐ended question.

Beware hidden assumptions and contingencies
BAD GOOD

During the past quarter, how often have you skipped evening social events to study? In the course of a typical seven-day week, how many hours do you spend doing course-related work outside of class? In the course of a typical seven-day week, how many hours do you spend socializing with friends for non-academic purposes?

The overall question or objective is to find out how students in different majors balance their social lives and study time. The first example assumes that the respondent would normally take part in social events. Furthermore, there is an implication that all social events take place in the evening, that only evening social events ʹcountʹ, or that respondents typically only study during the evening. Finally, the respondent would be obliged to think back and average over an entire quarter.

REFERENCES/RESOURCES
Bradburn, N. M., Sudburn, S. & Wansink, B. (2004). Asking questions: The definitive guide to questionnaire design. San Francisco, CA: Jossey‐Bass. Czaja, R. & Blair, J. (2005). Designing surveys: A guide to decisions and procedures. Thousand Oaks, CA: Sage Publications. Fowler, F. J. (2002). Survey research methods, 3rd Ed. Thousand Oaks, CA: Sage Publications. Fowler, F. J. (1995). Improving survey questions: Design and evaluation. Thousand Oaks, CA: Sage Publications. Peterson, R. (2000). Constructing effective questionnaires. Thousand Oaks, CA: Sage Publications

UW OEA: Tips for Writing Questionnaire Items

5