What are good resources/criteria for judging human bias in data collection? I've just been given a stack of polling data to analyse. Some of the questions are obviously leading or present subtle incentives (for the poller or polled) for specific answers. Of other questions I'm not so sure but I have some doubts. I'm also starting to question other factors about how the poll was conducted (environment, privacy, etc.). I'd like to be able to present the analysis with the unreliable data discounted according to some recognised standard.
What are actionable rules or methods for assessing whether results are junk based on human factors in data collection?
I'd gladly accept specific criteria, or links to resources such as books or web pages on the subject.
As I'm not a professional statistician, even being told what terms are used for this field of study and it's concerns would help a lot (though maybe best added as comments).
 A: In regards to the leading questions, here are several options of how I would attempt to investigate if your suspicions are true;
1 - Conduct your own experiment. One of your conditions will be to mimic the leading questions in the prior surveys, the other condition will be a survey constructed with functionally similar questions and answers but without the leading question(s). Randomly allocate surveys, and differences in answer distributions can be attributable to the different questions (+ sampling error).
2 - Determine if any other surveys have functionally similar questions but are not leading, and look at the distribution of answers for those surveys. The only thing you have to worry about here is the differences in the sample characteristics between your unfair surveys and the fair surveys that could account for some of the observed differences in the answers.
3 - Identify constructs between questions in your unfair survey. If leading questions within that construct have low correlations with other fair items it could be taken as evidence that the question is not returning the information it should.
I'm sure their is some psychological/psychometrics literature on how to ask questions (or about other items such as "environment"). I'm sure it would do you some good to review some of that work.
Good luck
A: Possibly Benford's Law might help. Check the Application section on the wiki
