7
$\begingroup$

I want to conduct a poll on the quality of a product containing several questions with five possible answers:

  1. Very poor
  2. Poor
  3. OK / No opinion
  4. Good
  5. Very good

A colleague has advised me to ditch option 3 (OK / No opinion) to force people to choose.

Which will produce the most reliable / useful data? Is there a preferred option or is it dependent on other factors (if so, what)?

I understand that Gallup usually has five options, hence why I choose five.

$\endgroup$
  • 6
    $\begingroup$ This is not a professional opinion, so I'm putting it in a comment. As a responder to questionnaires, I always wonder about the validity of results based on questions where no answer corresponds to my opinion. If you force me to answer arbitrarily, you will get arbitrary results. If many respondents have the same problem with a question, the collective response will be arbitrary, too. (This can lead to awful decision making if you have no independent way of checking that these results are meaningless.) $\endgroup$ – whuber Jan 26 '11 at 15:16
  • $\begingroup$ As is I don't think this is a question of statistics. If the question were changed to how do you assess the reliability between the two separate options, that will be more within the domain of the site. Any statement treating one or the other as correct will be based on domain specific knowledge. $\endgroup$ – Andy W Jan 26 '11 at 16:01
  • 4
    $\begingroup$ I think OK is very different from No opinion in most situations. If you want to include "no opinion" then put it as a 6th option (or remove the OK part) $\endgroup$ – nico Jan 26 '11 at 16:19
10
$\begingroup$

Neutral points can mean many different things to many people. The way you labeled the middle choice yourself reflects this uncertainty. Some reasons for choosing the neutral point from the perspective of a participant:

  • I don't care to really think about my answer to this question (I just want to get paid and leave)
  • I have no strong opinion on this question
  • I don't understand the question, but don't want to ask (I just want to get paid and leave)
  • with regards to the given aspect, the product is truly medium in quality, i.e., it neither excels nor falls short of my expectations
  • with regards to the given aspect, the product has some high-quality features, and some low-quality features

Without further qualification, the people who choose the middle category can thus represent a very heterogeneous collection of attitudes / cognitions. With good labeling, some of this confusion can be avoided.

You can also present a separate "no answer" category. However, participants often interpret such a category as a signal to only provide an answer if they feel very confident in their choice. In other words, participants then tend to choose "no answer" because they feel they're not well-informed enough to make a choice that meets the questionnaire-designers quality standards.

IMHO there's no right answer to your question. You have to be very careful in labeling some or all of the presented choices, do lots of pre-testing with additional free interviews of participants on how they perceived the options. If you're really pragmatic, you just choose a standard label-set for which you can cite an article that everybody else always cites and be done with it.

$\endgroup$
7
$\begingroup$

I think this whole "force people to choose" thing is just a complete red herring. People say it to me all the time. To me it sounds like "force people to state the capital of Uzbekistan". They don't know, and forcing them won't make them know any better.

With that mini-rant over, my only sensible contribution is to say that you should always pilot surveys whenever you can. Pilot both versions, see who uses the "don't know" category in the one where it's included, and look at the distribution of responses. And talk to the people who filled it out. "Were you sure of your answer?" "What made you say 'don't know' here"- that kind of thing.

$\endgroup$
2
$\begingroup$

I try to avoid questions with more than two answers, as it is impossible to compare them between users. (good vs. very good can be very subjective). I rephrase most questions into binary type (giving though the possibility to be indiffernt): "Would you use the product everyday?"

Yes No Indifferent

"Would you recommend the product to your friends?"

etc.

I found results obtained with this method to be way more consistent with the feeling I got from later interviews and performance tests. However, my work so far focuses on Human Computer Interaction questionnaires. The best is anyways to conduct real person interviews, as you learn more from them. Of course, they are also very time consuming :(

$\endgroup$
  • 1
    $\begingroup$ Many people will have both good and bad feelings about a product; I think one of the biggest problems users have answering satisfaction surveys is that they don't really know how the good and bad feelings should be balanced when expressing an opinion. Following up a question like "Are there any little annoyances you'd like to see fixed" and "If such annoyances were fixed, would you recommend the product to your friends" could help separate out good and bad feelings. $\endgroup$ – supercat Dec 22 '14 at 18:17
2
$\begingroup$

I would suggest that when trying to judge people's like or dislike of something, there are a few relevant scales of measure:

  1. How strong are any positive feelings for the product.
  2. How strong are any negative feelings for the product.
  3. How thoroughly have things that would cause positive or negative feelings been explored.

Since it's possible for strong negative feelings to be generated by problems which are easily fixed (and in some cases, the fact that a problem should be easily fixable may increase the extent of negative feelings), a company should want to know if there are many people who have both strong positive feelings and strong negative feelings, since addressing those people's complaints could generate massive goodwill relatively cheaply.

Additionally, a company may benefit from knowing if there are many people who discover things they don't like about a product before they delve deeply into it, and thus never go beyond those complaints. Further, if a product is designed to be useful at both a "shallow" level and a "deep" level, and only 10% of users go deep but 90% of those people have trouble, that would represent a very different picture from what one would get merely knowing that 9% of users had trouble.

A common problem with many surveys that try to restrict users' options is that they often don't provide choices that reflect what users want to say. Asking about positive and negative feelings separately will help users who think a product has some value but also have some major peeves with it; further, asking how well users have explored a product will help distinguish those who haven't found problems because there really aren't any, from those who haven't found problems because they've not used the product much.

$\endgroup$
1
$\begingroup$

If you wish to detect overt opinions then put in a neutral option. If you wish to detect any potential positive or negative bias then leave it out. As caracal said, label things as unambiguously as possible with respect to what you wish the options to reflect.

I've seen studies where only the form of response was changed. When there were only two options, like / dislike, then two stimuli were rated as very strongly liked in roughly equal proportions. When subjects were subsequently given an infinite rating scale with neither like nor dislike in the middle the rating differences between the two stimuli were vast (75% of the scale vs. 4%). This suggests that with a limited scale and no neutral option you can detect very small biases as large effects so you should be careful in interpreting such scales and use them judiciously.

$\endgroup$
0
$\begingroup$

The fact that you are forcing the respondent for a positive answer or negative answer, in this situation, is not correct; the respondent may be undecided and it is a possibility; it is more when it is a new product. If you are developing an instrument to measure quality it is better that you use 5 options as given by you.

$\endgroup$
  • $\begingroup$ are you saying that four options is better for a new product. Can you elaborate - why do you say that? $\endgroup$ – Reinstate Monica - Goodbye SE Jan 26 '11 at 15:37
  • $\begingroup$ The new product is in its initial stage and so there may not be time enough to form an opinion; four options are not good for this case. $\endgroup$ – KrishKalyan Jan 26 '11 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.