Should a multiple choice poll contain a neutral response?

Question

I want to conduct a poll on the quality of a product containing several questions with five possible answers:

1. Very poor
2. Poor
3. OK / No opinion
4. Good
5. Very good

A colleague has advised me to ditch option 3 (OK / No opinion) to force people to choose.

Which will produce the most reliable / useful data? Is there a preferred option or is it dependent on other factors (if so, what)?

I understand that Gallup usually has five options, hence why I choose five.

Answer 1

Neutral points can mean many different things to many people. The way you labeled the middle choice yourself reflects this uncertainty. Some reasons for choosing the neutral point from the perspective of a participant:

• I don't care to really think about my answer to this question (I just want to get paid and leave)
• I have no strong opinion on this question
• I don't understand the question, but don't want to ask (I just want to get paid and leave)
• with regards to the given aspect, the product is truly medium in quality, i.e., it neither excels nor falls short of my expectations
• with regards to the given aspect, the product has some high-quality features, and some low-quality features

Without further qualification, the people who choose the middle category can thus represent a very heterogeneous collection of attitudes / cognitions. With good labeling, some of this confusion can be avoided.

You can also present a separate "no answer" category. However, participants often interpret such a category as a signal to only provide an answer if they feel very confident in their choice. In other words, participants then tend to choose "no answer" because they feel they're not well-informed enough to make a choice that meets the questionnaire-designers quality standards.

IMHO there's no right answer to your question. You have to be very careful in labeling some or all of the presented choices, do lots of pre-testing with additional free interviews of participants on how they perceived the options. If you're really pragmatic, you just choose a standard label-set for which you can cite an article that everybody else always cites and be done with it.

Answer 2

I think this whole "force people to choose" thing is just a complete red herring. People say it to me all the time. To me it sounds like "force people to state the capital of Uzbekistan". They don't know, and forcing them won't make them know any better.

With that mini-rant over, my only sensible contribution is to say that you should always pilot surveys whenever you can. Pilot both versions, see who uses the "don't know" category in the one where it's included, and look at the distribution of responses. And talk to the people who filled it out. "Were you sure of your answer?" "What made you say 'don't know' here"- that kind of thing.

Answer 3

I try to avoid questions with more than two answers, as it is impossible to compare them between users. (good vs. very good can be very subjective). I rephrase most questions into binary type (giving though the possibility to be indiffernt): "Would you use the product everyday?"

Yes No Indifferent

"Would you recommend the product to your friends?"

etc.

I found results obtained with this method to be way more consistent with the feeling I got from later interviews and performance tests. However, my work so far focuses on Human Computer Interaction questionnaires. The best is anyways to conduct real person interviews, as you learn more from them. Of course, they are also very time consuming :(

Answer 4

The fact that you are forcing the respondent for a positive answer or negative answer, in this situation, is not correct; the respondent may be undecided and it is a possibility; it is more when it is a new product. If you are developing an instrument to measure quality it is better that you use 5 options as given by you.

Answer 5

If you wish to detect overt opinions then put in a neutral option. If you wish to detect any potential positive or negative bias then leave it out. As caracal said, label things as unambiguously as possible with respect to what you wish the options to reflect.

I've seen studies where only the form of response was changed. When there were only two options, like / dislike, then two stimuli were rated as very strongly liked in roughly equal proportions. When subjects were subsequently given an infinite rating scale with neither like nor dislike in the middle the rating differences between the two stimuli were vast (75% of the scale vs. 4%). This suggests that with a limited scale and no neutral option you can detect very small biases as large effects so you should be careful in interpreting such scales and use them judiciously.