Filtering random guessers from daily surveys

Question

Assume we're polling or surveying people about their predictions regarding some future event. This survey is done every day, and the event they are asked to predict is the same every time. For example, each morning they are asked "Do you think it will be sunny this afternoon?" Assume for the purposes of this question that the respondents are being compensated to give an answer, but there is no incentive to be accurate and there is no downside/penalty for being wrong.

Some of these people will just guess to get the polling process over with each day and thus earn their compensation. Others will make a legitimate attempt. Some people will vary between these two. Assume that you can compare predictions to the actual outcome after the event has happened. That is, that you can measure each respondent's accuracy.

How could you probabilistically tell those that were just guessing from those that were giving their true opinion, keeping in mind that we don't want penalize poor performance. We wish to maintain as much diversity of opinion as possible. But we do wish to eliminate random guessers.

Answer 1

This is a difficult and widely studied problem. One of the big reasons people are interested in it is that people like to use Amazon Mechanical Turk (AMT) for experiments, but many of the turkers on AMT just guess.

The simplest strategy to avoid guessers is to manually establish the correct answer to some questions, and then weed out those people who answer too many of these questions wrong. In your case, you could plant some 'obvious' prediction problems in your survey ("do you think sun will rise tomorrow"), and then eliminate the people who do badly on those. Doing something like this is probably your best bet.

There is some literature on trying to detect guessers in a fully automatic way (i.e. without planting any questions with known answers), but it is usually quite complicated and the methods don't always work. Here's my own attempt at it from a couple years back: http://cs229.stanford.edu/proj2010/ChenHsuWager-HumanAccuracyAnalysisOnAmazonMechanicalTurk.pdf. Of course, the problem addressed there is easier than yours, because we assumed that there was a `true' answer to each question, while you allow for diversity of opinions.