A 6th response option ("I don't know") was added to a 5-point Likert scale. Is the data lost?

Question

I need a little bit of help salvaging the data from a questionnaire.

One of my colleagues applied a questionnaire, but inadvertently, instead of using the original 5-point Likert scale (strongly disagree to strongly agree), he inserted a 6th answer into the scale. And, to make the matter worse, the 6th response option is … “I don’t know”.

The problem is the big proportion of respondents who, at one point or another, chose “I don’t know”. If they were a reasonably small percentage, I’d have just excluded them from the database. However, the core of the research rests on a conceptual model, and excluding so many records would create a problem for the model.

Could someone point me in the right direction here? Are there any ‘good practices’, or can I do anything to use (transform, convert, etc.) those “I don’t know” responses?

Also, if I do any manipulation of the data in question (i.e., if I convert the “I don’t know” responses, by substitution, imputation, etc.), what kind of ‘disclaimer’, ‘warning’, annotation, should I use?

I know it is a long shot, but I confess, besides salvaging the responses, I am also curious what is the agreed practice (if there is one), in these type of cases.

PS: I know it sounds childish, but no, the ‘colleague’ isn’t me :)

Answer 1

Why try to force a calibration on something which is not true? As Maarten said, this is not a loss of data but a gain of information. If the magical pill you are looking for exists, it would mean that there are some assumptions about your population that are made, for example, a bias in favor of one particular label even though users say "I don't know".

I totally understand your frustration but the proper way to approach the problem is to modify the model to suit your needs based on the true existing data, and not the other way around (modifying the data).

Answer 2

If this was a standardized questionnaire that has been validated independently, then you cannot claim that the new questionnaire is equivalent, and the data is no longer comparable. You could try to validate and examine the questionnaire in separate experiments (very time- and effort-consuming, especially if you also want to show comparability to the old version) or just accept that you are dealing with lower evidence quality (since your data comes from a non-validated questionnaire).

When you use your data, you will have to take the change into consideration. When faced with an attitude question, people don't give you a somehow "objectively true" answer, they give you the answer they feel to be true - and this is certainly influenced both by the answer options available (they "norm" their answers to the scale) and to the knowledge they have about the subject (there are known biases which work differently, sometimes in different directions(!) depending on whether the participant has much or little knowledge about the subject matter).

So, if we are dealing with an established quesitonnaire, you have the nice option for a comparison between your version of the questionnaire and the original one. If the original assumed that people know what they are selecting, and it turns out they don't, you can discuss how the old model is based on wrong assumptions, and what are the consequences of that. Note that this is a "side" discovery, which makes a nice new research question, but brings you away from the original one, and indeed shows that answering the original one is much more difficult than thought, so it certainly multiplies your work.

If you are not dealing with an established questionnaire, you can roll with the flow and pretend that your ad-hoc questionnaire was planned that way, and evaluate the results accordingly. Again, it might mean that the results you were hoping for are unobtainable with this method, but this is also an important thing to know.

For a good understanding of how wording and options influence the way questionnaires are answered, I suggest reading Tourangeau et al.'s "Psychology of the survey response". It is a great read for everybody who ever creates a questionnaire.

Answer 3

If you ask how many children has the respondent given birth to, the answers "zero" and "not applicable" would not mean strictly the same thing, since men cannot give birth.

For some contexts, equating "I don't know" to the neutral response could be, likewise, a conceptual mistake.

Actually, you have two questions: a dichotomous "Do you have an opinion?" and an ordinal "What is it?", just as, above, you have an implicit "Are you a female?" beyond your explicit question.

Of course, you can introduce some assumptions (sometimes correctly, sometimes just for convenience, sometimes forcedly) to enable you some modeling, but I can see no universally applicable strategy without entering the realm of the specifics of your phenomenon.

As a last point to be thought of, it would not make sense to try and infer to male population anything from female fecundity answers.

Answer 4

The dilemma whether one should include or not the Don't know response option into a questionnaire consisting of rating scales of Likert type is eternal. Often, when the items ask about opinion, the DK is included because having no opinion is an important status on its own and the option as such is expected by respondents. In personal trait inventories where people ascribe qualities to a target DK option is typically dropped because a respondent normally is expected to be able to assess the extent of affinity of a characteristic (i.e. respondent is always seen qualified); and when he occasionally finds difficulty he is allowed (by instruction) to skip that item. In personal trait inventories where people describe a target (behavioural items) DK (or don't remember) could be incorporated or dropped depending on the scale design and the specific question of the study.

@Hatim in his answer, @Maarten and some other commentators of the OP question have sensibly put up that a large amount of DK responses observed in the current study indicate problems (content validity or face valitity) in the items or that the subjects don't fit in with the questionnaire ordered to them.

But you can never tell the story, ultimately the interpretation of the impediment is on you (unless you address it in a separate investigation). One could claim, for example, that the inclusion of DK option to the likerts in that questionnaire (say, it is a trait ascription inventory) serves bad, not good. It didn't give you information (of which the commentators say, that it proves that the [rating] model is inadequate) but rather distracted/seduced a respondent. Be it not supplied the rating decision guided by the implicit cognitive trait schema could have been elicited; but seeing the cooling option precludes the schema and makes one hastily to withdraw.

If you further admit - on your risk, but why not? - that an easily distracted or lazy subject is the one whose potential, held back view is valid but tends to be weakly differentiated - that is, he would easily invoke conventional das Man, in place of personal Erlebnis, schema - then you may tentatively speculate that his missing response is around the sample's or population's mean for that item. If so, why not do mean (+noise) substitution of the missing responses? Or you might do EM or regressional (+noise) imputation to take correlations into account.

To repeat: the imputation decision is possible but risky, and is unlikely, given the large amount of missing data, to restore "truly" the absent data. As @rumtscho said, surely that the new questionnaire with DK is not equivalent to the original one without DK, and the data is no longer comparable.

These were speculations. But first of all, you ought to attempt to investigate the observed patterns of missingness. Who are those subjects who selected DK? Do they cluster together in subtypes? How they are different on the rest of the items from the "okay" subsample? Some software have Missing Value Analysis package. Then you could decide whether to drop the people entirely or partly, or to impute, or to analyze them as a separate subsample.

P.S. Also note that respondents are "stupid". They often just mix up with the scale grades. For example, if the DK point was placed close to one pole of the scale it would often get confused by inattention with that pole. I'm not joking.

Answer 5

You now have respondents self-selected for having an opinion on the matter. Whatever you conclude will solely be about those people. This might be OK, as polling those "don't knows" is by definition less useful.