I want to conduct a poll on the quality of a product containing several questions with five possible answers:
A colleague has advised me to ditch option 3 (OK / No opinion) to force people to choose.
Which will produce the most reliable / useful data? Is there a preferred option or is it dependent on other factors (if so, what)?
I understand that Gallup usually has five options, hence why I choose five.
Neutral points can mean many different things to many people. The way you labeled the middle choice yourself reflects this uncertainty. Some reasons for choosing the neutral point from the perspective of a participant:
Without further qualification, the people who choose the middle category can thus represent a very heterogeneous collection of attitudes / cognitions. With good labeling, some of this confusion can be avoided.
You can also present a separate "no answer" category. However, participants often interpret such a category as a signal to only provide an answer if they feel very confident in their choice. In other words, participants then tend to choose "no answer" because they feel they're not well-informed enough to make a choice that meets the questionnaire-designers quality standards.
IMHO there's no right answer to your question. You have to be very careful in labeling some or all of the presented choices, do lots of pre-testing with additional free interviews of participants on how they perceived the options. If you're really pragmatic, you just choose a standard label-set for which you can cite an article that everybody else always cites and be done with it.
I think this whole "force people to choose" thing is just a complete red herring. People say it to me all the time. To me it sounds like "force people to state the capital of Uzbekistan". They don't know, and forcing them won't make them know any better.
With that mini-rant over, my only sensible contribution is to say that you should always pilot surveys whenever you can. Pilot both versions, see who uses the "don't know" category in the one where it's included, and look at the distribution of responses. And talk to the people who filled it out. "Were you sure of your answer?" "What made you say 'don't know' here"- that kind of thing.
I try to avoid questions with more than two answers, as it is impossible to compare them between users. (good vs. very good can be very subjective). I rephrase most questions into binary type (giving though the possibility to be indiffernt): "Would you use the product everyday?"
Yes No Indifferent
"Would you recommend the product to your friends?"
etc.
I found results obtained with this method to be way more consistent with the feeling I got from later interviews and performance tests. However, my work so far focuses on Human Computer Interaction questionnaires. The best is anyways to conduct real person interviews, as you learn more from them. Of course, they are also very time consuming :(
I would suggest that when trying to judge people's like or dislike of something, there are a few relevant scales of measure:
Since it's possible for strong negative feelings to be generated by problems which are easily fixed (and in some cases, the fact that a problem should be easily fixable may increase the extent of negative feelings), a company should want to know if there are many people who have both strong positive feelings and strong negative feelings, since addressing those people's complaints could generate massive goodwill relatively cheaply.
Additionally, a company may benefit from knowing if there are many people who discover things they don't like about a product before they delve deeply into it, and thus never go beyond those complaints. Further, if a product is designed to be useful at both a "shallow" level and a "deep" level, and only 10% of users go deep but 90% of those people have trouble, that would represent a very different picture from what one would get merely knowing that 9% of users had trouble.
A common problem with many surveys that try to restrict users' options is that they often don't provide choices that reflect what users want to say. Asking about positive and negative feelings separately will help users who think a product has some value but also have some major peeves with it; further, asking how well users have explored a product will help distinguish those who haven't found problems because there really aren't any, from those who haven't found problems because they've not used the product much.
If you wish to detect overt opinions then put in a neutral option. If you wish to detect any potential positive or negative bias then leave it out. As caracal said, label things as unambiguously as possible with respect to what you wish the options to reflect.
I've seen studies where only the form of response was changed. When there were only two options, like / dislike, then two stimuli were rated as very strongly liked in roughly equal proportions. When subjects were subsequently given an infinite rating scale with neither like nor dislike in the middle the rating differences between the two stimuli were vast (75% of the scale vs. 4%). This suggests that with a limited scale and no neutral option you can detect very small biases as large effects so you should be careful in interpreting such scales and use them judiciously.
The fact that you are forcing the respondent for a positive answer or negative answer, in this situation, is not correct; the respondent may be undecided and it is a possibility; it is more when it is a new product. If you are developing an instrument to measure quality it is better that you use 5 options as given by you.
