Edit: While I do appreciate Peter Flom's suggestions and the subsequent discussion (and upvoted his answer), I am opening a bounty to solicit an answer which offers a specific, formal statistical approach to handling these inconsistencies while minimizing the loss of data. I strongly suspect this issue has inspired statistical research along the lines of my bullet points below (or maybe completely different!) but I am not aware of it and am hoping to rouse the attention of someone who is. I would consider an answer that is little more than a reference and a short description to be worthy of the bounty; there's no need to derive or implement the approach for me. I'd also accept an answer that gives a clever approach, with some details, without a reference. I think @Momo may be onto something with the comments below, so maybe this bounty will draw forth more details/background info :)

Edit: For the purposes of building a tractable model, I am willing to assume that the inconsistencies arise "randomly", e.g. due to randomly mis-reading the question or mis-clicking an answer on a tablet/computer, so that the errors could be conceived of as independent of any auxiliary variables, in contrast to the example given by @whuber in the comments.

I was recently approach with a statistical question about having inconsistent answers to questions. I think this is the same basic question as when you have multiple raters that disagree on how to rate a particular item, and I thought this language may be more familiar, which is why I chose this title.

A toy example:

Q1: Do you smoke?
A1: No
Q2: How many cigarettes a day do you smoke?
A2: 5

Of course, a "skip pattern" in the questionnaire would have prevented this but that ship has sailed. The real situation is more complex than this and involves more questions with more subtle inconsistencies but the basic issue is the same. The query of the person who asked me this can be simply stated as:

Are there methods for eliminating some degree of inconsistency while still preserving as many samples as possible?

I am very aware that, if you want to be "safe" then the only thing to do is throw out any samples with inconsistent replies, but that's not the answer I'm looking for. In particular, there are some cases where there is good evidence that a particular response is a mistake and I'm looking for principled ways to use that evidence - as an extreme example, suppose ten questions measure the same overall construct and nine of the ten agree - then it is quite likely that the one that disagrees was an error (e.g. a mis-read of the question or a mis-click on the computer/tablet used to administer the survey)

The basic thoughts I have on the subject bring to mind two general ideas:

  • Try to build a model that estimates the probability that a particular item is a mistake and switch responses whose "mistake probability" is very high. My concerns are that a) this is not tractable without making wild assumptions about the "error rate" and b) if there are only a few (say 3 or 4) questions for each construct, this approach would be basically useless.

  • Try to select some "reliable subset" of the questions (i.e. try to determine whether the disagreement regularly arises from a particular subset of the questions). This way I can get away with deleting columns from the data set rather than rows. This seems reasonable but would be more of an ad hoc procedure that I'm not sure how to formalize.

I'm not at all familiar with this field, and it seems like this kind of issue would come up occasionally in statistical consulting, so I wanted to know how this is handled (other than throwing the data out) before I try to "reinvent the wheel".

  • 4
    $\begingroup$ fwiw, having encountered some of these situations, I think reinventing the wheel will probably be necessary in order to fit a solution to the idiosyncracies of your situation. But based on your posts here that I've noticed over the past 2 years I am confident that your solution will fit better than any that we "onlookers" might devise for you. $\endgroup$
    – rolando2
    Jan 16 '13 at 16:22
  • $\begingroup$ Perhaps Latent Class Analysis en.wikipedia.org/wiki/Latent_class_model is what you are looking for. It basically formalizes/extends your two ideas from above. I also saw it being used for the problem you describe (sorry for not having references handy right now). In light of your question, LCA has one additional advantage: You can assign the respondents with inconsistent answers to the class for which they have the highest posterior and treat all those people "the same". $\endgroup$
    – Momo
    Jan 16 '13 at 22:40
  • $\begingroup$ Hi @Momo, I am very familiar with LCA/mixture modeling, but I'm having some trouble following the logic of this approach - maybe you can help. My basic understanding is that LCA uses the observed data to identify unobserved strata in the population, whatever that may mean in the context of the model. In this case, the observations that are the basis for the stratification are subject to an unknown (possibly substantial) amount of measurement error, so it's unclear what the identified classes would even mean or why it would provide a reliable basis for "smoothing over" the inconsistencies. $\endgroup$
    – Macro
    Jan 17 '13 at 1:18
  • $\begingroup$ The idea is that LCA clusters people based on response pattern including inconsistent patterns. One can then for example use covariates for the class probabilities to explain the soft group assignment. With different restricted LCA one can control how the inconsistencies are handled. Analysis of the people's response can then happen stratified for each class or similar. $\endgroup$
    – Momo
    Jan 17 '13 at 11:43
  • $\begingroup$ Thanks @Momo - it seems like the model would just identify one (or more) latent classes that are comprised of inconsistent measurements, but I don't see how that would remedy the problem. E.g., suppose there are four Yes/No questions that should all agree, and the data set has four basic response types: {Y,Y,Y,Y}, {N,N,N,N}, {Y,Y,N,N}, {N,N,Y,Y}. Then, it seems like four types would be identified and the latter two classes wouldn't be useful for subsequent analysis. Perhaps you suggesting only looking at the posterior probs for the first two classes? Am I missing something else? $\endgroup$
    – Macro
    Jan 17 '13 at 12:22

In a previous job I ran into this a lot. All sorts of inconsistencies. Like @rolando2 I don't think any general solution is going to be nearly as good as what you can come up with on your own; you would then just have to justify it to whoever your audience is.

However, one thing you can do is a series of sensitivity analyses, treating the data different ways. that is, if two questions have answers that are inconsistent you could first run the analysis assuming everyone answered the first question correctly and then as if everyone answered the second question correctly.

For some specific inconsistencies, there are known results. For example, it is known that if you ask people "how old are you?" and "When were you born?" the latter answers will be more accurate.

In general, if the questions are such that stigma is attached to one answer, the more stigmatized answer is likely to be correct.

  • $\begingroup$ +1, I was hoping you'd answer Peter, since I know you have a background in Psychometrics, which seems closely related - thanks. Perhaps I can pick your brain a bit more about these "specific inconsistencies" -- the general domain here are substance use, crime and other "bad behavior" type questions in adults -- would frequency type questions tend to be less accurate than Yes/No questions? E.g. "Do you drink regularly?" vs. "How many drinks per week do you have?". Also, there seems to be an incredibly large number of different sensitivity analyses that are possible - any specific ideas? $\endgroup$
    – Macro
    Jan 16 '13 at 21:20
  • $\begingroup$ Also, if you have any suggested authors/reading regarding what types of instruments are known to be less reliable, that would be terrific so I can forward it to the questioner. $\endgroup$
    – Macro
    Jan 16 '13 at 21:24
  • $\begingroup$ Measuring substance use has a lot of literature; it's a field I worked in for 10 years, but that was 10 years ago. Measuring alcohol use, for example, is fraught with complexities: What is a "drink" exactly? How to measure "binge" vs. frequent drinking. Illicit substance use runs into other problems. There are methods like ACASI, which are better at keeping things anonymous. If you are after just estimate rates in populations, there are methods that have half the sample answer a 'null' question. Unfortunately, it is very hard to get a gold standard. $\endgroup$
    – Peter Flom
    Jan 16 '13 at 22:32
  • $\begingroup$ Journal of Substance Abuse covers the field of substance use, it will have some articles on methods There's also Journal of Substance Abuse Treatment Journal of Alcohol and Drug Education and lots of others. But finding a few good articles and then looking at citation lists in them is probably the best way. It's a challenging field! $\endgroup$
    – Peter Flom
    Jan 16 '13 at 22:39
  • 2
    $\begingroup$ Peter, I think that's good advice. Plus, if the results aren't different then I think that would give evidence for the errors being more "at random" than systematically biased. If you get a chance, making that more explicit in your answer would probably help future readers. Cheers $\endgroup$
    – Macro
    Jan 17 '13 at 2:51

Structural equation modeling may help here, particularly confirmatory factor analysis, which allows you to test whether your hypothesized measurement model fits the data and if not, how you might adjust it to do so, for example by dropping items.

Like exploratory factor analysis, CFA models survey responses (or other items measuring some latent construct) as arising from one or more underlying dimensions and random error. Generally you need at least three items per construct to identify a measurement model though with multiple dimensions you can sometimes get away with only two.

Once you've specified your latent constructs and the items hypothesized to measure each construct, you estimate the model and you will get a number of fit indexes that you can use to judge whether your model is adequate. You can compare different models by means of various criteria as well. You can inspect factor loadings and error variances for each item to see whether it seems to be a good item for measuring the underlying construct or not. This gives you guidance as to which items (i.e. columns in your data set) may be dropped from your model.

Beyond just creating a measurement model, you can further specify directional relations between latent constructs using a structural equation model (SEM). This is preferable to using, for example, traditional linear regression because measurement error is explicitly modeled whereas in the usual approach only random error in the outcome variable is assumed.

You can use the sem package in R to build CFA and SEM models. I have not used that package however. I have used Mplus and Amos and prefer Mplus for the great variety of structural equation models it can handle including those with binary indicators.

For a reference, I like Kline's book on SEM.

  • 1
    $\begingroup$ Hi Anne, could you elaborate on your answer to contextualise it in OP's situation? I too am interested in how you would implement the problem in a CFA framework. $\endgroup$
    – R J
    Feb 26 '13 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.