17
$\begingroup$

I need a little bit of help salvaging the data from a questionnaire.

One of my colleagues applied a questionnaire, but inadvertently, instead of using the original 5-point Likert scale (strongly disagree to strongly agree), he inserted a 6th answer into the scale. And, to make the matter worse, the 6th response option is … “I don’t know”.

The problem is the big proportion of respondents who, at one point or another, chose “I don’t know”. If they were a reasonably small percentage, I’d have just excluded them from the database. However, the core of the research rests on a conceptual model, and excluding so many records would create a problem for the model.

Could someone point me in the right direction here? Are there any ‘good practices’, or can I do anything to use (transform, convert, etc.) those “I don’t know” responses?

Also, if I do any manipulation of the data in question (i.e., if I convert the “I don’t know” responses, by substitution, imputation, etc.), what kind of ‘disclaimer’, ‘warning’, annotation, should I use?

I know it is a long shot, but I confess, besides salvaging the responses, I am also curious what is the agreed practice (if there is one), in these type of cases.

PS: I know it sounds childish, but no, the ‘colleague’ isn’t me :)

Improve this question
$\endgroup$
5
  • 22
    $\begingroup$ Imagine your colleague had not included the I don't know category. What would those persons have answered? Maybe they would have ticked some random category, maybe they would have left it blank. Now you know that your variables are problematic and make an informed decision. Seen from this perspective, the colleague did you a favor. $\endgroup$ – Maarten Buis Apr 12 '16 at 7:14
  • $\begingroup$ Well, as far as scratching my head goes, he did indeed do me a favor... You're right about the randomness of the responses, and this could be an approach for imputation, however, I was wondering if there are any good practices, or at least some similar experiences $\endgroup$ – streamline Apr 12 '16 at 7:25
  • 4
    $\begingroup$ I find questionnaires that don't have such a "I don't know" option particularly frustrating (especially when you can't submit without picking something). Let's just say you throw a question like "The new features on that space rocket are an improvement", and then you're forced to be neutral at best, some people can then interpret that sort of surveys and conclude "We've asked 1000 people, and 100% didn't mind...", whereas in practice, that feature could be a complete mistake for those who use it. In those cases, it almost feel like silent endorsement for something I haven't used. $\endgroup$ – Bruno Apr 12 '16 at 16:13
  • 3
    $\begingroup$ To be more clear: if the model assumes that the 5 point scale can represent the opinions of the respondents, then your data proves that the model is inadequate. If it were correct then you'd have a negligible number of "I don't knows", because people would have been able to answer 1-5. So this data "would create a problem for the model" like the orbit of Mercury creates a problem for Newtonian physics. I'd have thought your only way out is if the middle point is "neither agree nor disagree" and you can make the case that "I don't know" is redundant with the same meaning as this. $\endgroup$ – Steve Jessop Apr 12 '16 at 21:52
  • $\begingroup$ Hi, @SteveJessop, while in theory/in principle, you're right about the number of DK answers as being indicative of a problem with the questionnaire (not the model, mind you, the model is not CFA for the scale, but regression for a bunch of variables, including the scale), it is still a bit strong to call it inadequate. I will do the scale analyses and see, and it is true, the shear number of DK answers may prove problematic. However, they can also mean that the respondent didn't feel like answering (incidentally, the vast majority of DK answers come from a group of students after an exam...) $\endgroup$ – streamline Apr 15 '16 at 7:36
28
$\begingroup$

Why try to force a calibration on something which is not true? As Maarten said, this is not a loss of data but a gain of information. If the magical pill you are looking for exists, it would mean that there are some assumptions about your population that are made, for example, a bias in favor of one particular label even though users say "I don't know".

I totally understand your frustration but the proper way to approach the problem is to modify the model to suit your needs based on the true existing data, and not the other way around (modifying the data).

Improve this answer
$\endgroup$
17
  • $\begingroup$ Hi, Hatim, I don't believe it is possible to modify the model. The scale is ordinal, and assumes a reasoned and explicit choice, whereas the 6th answer, "i don't know", can be interpreted in any number of ways. It may mean that "i never encountered this situation/i don't recall", it may stand for an 'intermediary' choice of some sort. Any such interpretation/assumption on my behalf, would be presumptuous and unfounded. Maarten's answer referred to a 'gain' of information, which I take it that a specific 'randomized' imputation can be used, but this is not what you say - "modify the model". $\endgroup$ – streamline Apr 12 '16 at 7:56
  • $\begingroup$ continuation... However, although I was and still am, tempted to look further into such 'randomized' imputation, the large volume of "i don't know" answers make me fear that the true (authentic) relations between variables will be altered. $\endgroup$ – streamline Apr 12 '16 at 8:01
  • 8
    $\begingroup$ +1. I know it's uncomfortable, but you [the OP] have a choice between finding a different dataset if you want to test that model or modifying the planned analysis. You've asked the question hoping for different answers but there are not, in my view, any that are defensible. If I were a respondent to such a questionnaire I would feel offended at the distortion and lack of trust in trying to treat my Don't knows as anything else. In fact as an occasional consumer of social research I am disconcerted too. $\endgroup$ – Nick Cox Apr 12 '16 at 8:24
  • 1
    $\begingroup$ You are naturally right that this has happened before, and many times. That's why those with some experience of projects that were diverted or complicated by unforeseen problems can say, so, the analysis will be different and the paper won't be as imagined. Or even, sometimes projects just don't work out, so there you go. (If somehow you are under instruction or compulsion to follow through, that is especially unfortunate, but it doesn't affect my advice on how to think about it.) $\endgroup$ – Nick Cox Apr 12 '16 at 8:53
  • 8
    $\begingroup$ @user2836366 I don't understand your assertions that the model cannot be modified. Certainly "Don't know" isn't part of the ordinal collection of responses, but that's entirely to the point; "Don't know" implies that for whatever reason (including actually know knowing) the person didn't choose one of the ordinal responses. So one such modification is you could have some model for that process (chose "don't know" vs "chose one of the other options") and then the usual model for the cases in the second category. Such models may be somewhat similar to hurdle models or zero-inflated models. $\endgroup$ – Glen_b Apr 12 '16 at 9:44
11
$\begingroup$

If this was a standardized questionnaire that has been validated independently, then you cannot claim that the new questionnaire is equivalent, and the data is no longer comparable. You could try to validate and examine the questionnaire in separate experiments (very time- and effort-consuming, especially if you also want to show comparability to the old version) or just accept that you are dealing with lower evidence quality (since your data comes from a non-validated questionnaire).

When you use your data, you will have to take the change into consideration. When faced with an attitude question, people don't give you a somehow "objectively true" answer, they give you the answer they feel to be true - and this is certainly influenced both by the answer options available (they "norm" their answers to the scale) and to the knowledge they have about the subject (there are known biases which work differently, sometimes in different directions(!) depending on whether the participant has much or little knowledge about the subject matter).

So, if we are dealing with an established quesitonnaire, you have the nice option for a comparison between your version of the questionnaire and the original one. If the original assumed that people know what they are selecting, and it turns out they don't, you can discuss how the old model is based on wrong assumptions, and what are the consequences of that. Note that this is a "side" discovery, which makes a nice new research question, but brings you away from the original one, and indeed shows that answering the original one is much more difficult than thought, so it certainly multiplies your work.

If you are not dealing with an established questionnaire, you can roll with the flow and pretend that your ad-hoc questionnaire was planned that way, and evaluate the results accordingly. Again, it might mean that the results you were hoping for are unobtainable with this method, but this is also an important thing to know.

For a good understanding of how wording and options influence the way questionnaires are answered, I suggest reading Tourangeau et al.'s "Psychology of the survey response". It is a great read for everybody who ever creates a questionnaire.

Improve this answer
$\endgroup$
2
$\begingroup$

If you ask how many children has the respondent given birth to, the answers "zero" and "not applicable" would not mean strictly the same thing, since men cannot give birth.

For some contexts, equating "I don't know" to the neutral response could be, likewise, a conceptual mistake.

Actually, you have two questions: a dichotomous "Do you have an opinion?" and an ordinal "What is it?", just as, above, you have an implicit "Are you a female?" beyond your explicit question.

Of course, you can introduce some assumptions (sometimes correctly, sometimes just for convenience, sometimes forcedly) to enable you some modeling, but I can see no universally applicable strategy without entering the realm of the specifics of your phenomenon.

As a last point to be thought of, it would not make sense to try and infer to male population anything from female fecundity answers.

Improve this answer
$\endgroup$
2
$\begingroup$

The dilemma whether one should include or not the Don't know response option into a questionnaire consisting of rating scales of Likert type is eternal. Often, when the items ask about opinion, the DK is included because having no opinion is an important status on its own and the option as such is expected by respondents. In personal trait inventories where people ascribe qualities to a target DK option is typically dropped because a respondent normally is expected to be able to assess the extent of affinity of a characteristic (i.e. respondent is always seen qualified); and when he occasionally finds difficulty he is allowed (by instruction) to skip that item. In personal trait inventories where people describe a target (behavioural items) DK (or don't remember) could be incorporated or dropped depending on the scale design and the specific question of the study.

@Hatim in his answer, @Maarten and some other commentators of the OP question have sensibly put up that a large amount of DK responses observed in the current study indicate problems (content validity or face valitity) in the items or that the subjects don't fit in with the questionnaire ordered to them.

But you can never tell the story, ultimately the interpretation of the impediment is on you (unless you address it in a separate investigation). One could claim, for example, that the inclusion of DK option to the likerts in that questionnaire (say, it is a trait ascription inventory) serves bad, not good. It didn't give you information (of which the commentators say, that it proves that the [rating] model is inadequate) but rather distracted/seduced a respondent. Be it not supplied the rating decision guided by the implicit cognitive trait schema could have been elicited; but seeing the cooling option precludes the schema and makes one hastily to withdraw.

If you further admit - on your risk, but why not? - that an easily distracted or lazy subject is the one whose potential, held back view is valid but tends to be weakly differentiated - that is, he would easily invoke conventional das Man, in place of personal Erlebnis, schema - then you may tentatively speculate that his missing response is around the sample's or population's mean for that item. If so, why not do mean (+noise) substitution of the missing responses? Or you might do EM or regressional (+noise) imputation to take correlations into account.

To repeat: the imputation decision is possible but risky, and is unlikely, given the large amount of missing data, to restore "truly" the absent data. As @rumtscho said, surely that the new questionnaire with DK is not equivalent to the original one without DK, and the data is no longer comparable.

These were speculations. But first of all, you ought to attempt to investigate the observed patterns of missingness. Who are those subjects who selected DK? Do they cluster together in subtypes? How they are different on the rest of the items from the "okay" subsample? Some software have Missing Value Analysis package. Then you could decide whether to drop the people entirely or partly, or to impute, or to analyze them as a separate subsample.

P.S. Also note that respondents are "stupid". They often just mix up with the scale grades. For example, if the DK point was placed close to one pole of the scale it would often get confused by inattention with that pole. I'm not joking.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi, @ttnphns, Well, your answer is too long to take it point by point (I've already commented on the other answers about some of the points that you've raised here). To be honest, I'll have to read it again a couple of times. However, it addresses everything I wanted . $\endgroup$ – streamline Apr 15 '16 at 7:42
0
$\begingroup$

You now have respondents self-selected for having an opinion on the matter. Whatever you conclude will solely be about those people. This might be OK, as polling those "don't knows" is by definition less useful.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Droll, but too pessimistic. Those people offered arguments on what one can, cannot, should, should not do, and it's the arguments that should be weighed. The same answer could be given on any thread here, but those who do not speak up have no say. $\endgroup$ – Nick Cox Apr 13 '16 at 8:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.