Handling poor inter-rater reliability while minimizing the loss of data Edit: While I do appreciate Peter Flom's suggestions and the subsequent discussion (and upvoted his answer), I am opening a bounty to solicit an answer which offers a specific, formal statistical approach to handling these inconsistencies while minimizing the loss of data. I strongly suspect this issue has inspired statistical research along the lines of my bullet points below (or maybe completely different!) but I am not aware of it and am hoping to rouse the attention of someone who is. I would consider an answer that is little more than a reference and a short description to be worthy of the bounty; there's no need to derive or implement the approach for me. I'd also accept an answer that gives a clever approach, with some details, without a reference. I think @Momo may be onto something with the comments below, so maybe this bounty will draw forth more details/background info :)
Edit: For the purposes of building a tractable model, I am willing to assume that the inconsistencies arise "randomly", e.g. due to randomly mis-reading the question or mis-clicking an answer on a tablet/computer, so that the errors could be conceived of as independent of any auxiliary variables, in contrast to the example given by @whuber in the comments.  

I was recently approach with a statistical question about having inconsistent answers to questions. I think this is the same basic question as when you have multiple raters that disagree on how to rate a particular item, and I thought this language may be more familiar, which is why I chose this title. 
A toy example:  

Q1: Do you smoke?
  A1: No
  Q2: How many cigarettes a day do you smoke?
  A2: 5 

Of course, a "skip pattern" in the questionnaire would have prevented this but that ship has sailed. The real situation is more complex than this and involves more questions with more subtle inconsistencies but the basic issue is the same. The query of the person who asked me this can be simply stated as: 
Are there methods for eliminating some degree of inconsistency while still preserving as many samples as possible? 
I am very aware that, if you want to be "safe" then the only thing to do is throw out any samples with inconsistent replies, but that's not the answer I'm looking for. In particular, there are some cases where there is good evidence that a particular response is a mistake and I'm looking for principled ways to use that evidence - as an extreme example, suppose ten questions measure the same overall construct and nine of the ten agree - then it is quite likely that the one that disagrees was an error (e.g. a mis-read of the question or a mis-click on the computer/tablet used to administer the survey)
The basic thoughts I have on the subject bring to mind two general ideas: 


*

*Try to build a model that estimates the probability that a particular item is a mistake and switch responses whose "mistake probability" is very high. My concerns are that a) this is not tractable without making wild assumptions about the "error rate" and b) if there are only a few (say 3 or 4) questions for each construct, this approach would be basically useless. 

*Try to select some "reliable subset" of the questions (i.e. try to determine whether the disagreement regularly arises from a particular subset of the questions). This way I can get away with deleting columns from the data set rather than rows. This seems reasonable but would be more of an ad hoc procedure that I'm not sure how to formalize. 
I'm not at all familiar with this field, and it seems like this kind of issue would come up occasionally in statistical consulting, so I wanted to know how this is handled (other than throwing the data out) before I try to "reinvent the wheel". 
 A: In a previous job I ran into this a lot. All sorts of inconsistencies. Like @rolando2 I don't think any general solution is going to be nearly as good as what you can come up with on your own; you would then just have to justify it to whoever your audience is.
However, one thing you can do is a series of sensitivity analyses, treating the data different ways.  that is, if two questions have answers that are inconsistent you could first run the analysis assuming everyone answered the first question correctly and then as if everyone answered the second question correctly.
For some specific inconsistencies, there are known results. For example, it is known that if you ask people "how old are you?" and "When were you born?" the latter answers will be more accurate. 
In general, if the questions are such that stigma is attached to one answer, the more stigmatized answer is likely to be correct.
A: Structural equation modeling may help here, particularly confirmatory factor analysis, which allows you to test whether your hypothesized measurement model fits the data and if not, how you might adjust it to do so, for example by dropping items.
Like exploratory factor analysis, CFA models survey responses (or other items measuring some latent construct) as arising from one or more underlying dimensions and random error. Generally you need at least three items per construct to identify a measurement model though with multiple dimensions you can sometimes get away with only two. 
Once you've specified your latent constructs and the items hypothesized to measure each construct, you estimate the model and you will get a number of fit indexes that you can use to judge whether your model is adequate. You can compare different models by means of various criteria as well. You can inspect factor loadings and error variances for each item to see whether it seems to be a good item for measuring the underlying construct or not. This gives you guidance as to which items (i.e. columns in your data set) may be dropped from your model. 
Beyond just creating a measurement model, you can further specify directional relations between latent constructs using a structural equation model (SEM). This is preferable to using, for example, traditional linear regression because measurement error is explicitly modeled whereas in the usual approach only random error in the outcome variable is assumed. 
You can use the sem package in R to build CFA and SEM models. I have not used that package however. I have used Mplus and Amos and prefer Mplus for the great variety of structural equation models it can handle including those with binary indicators. 
For a reference, I like Kline's book on SEM. 
