4
$\begingroup$

Assume we're polling or surveying people about their predictions regarding some future event. This survey is done every day, and the event they are asked to predict is the same every time. For example, each morning they are asked "Do you think it will be sunny this afternoon?" Assume for the purposes of this question that the respondents are being compensated to give an answer, but there is no incentive to be accurate and there is no downside/penalty for being wrong.

Some of these people will just guess to get the polling process over with each day and thus earn their compensation. Others will make a legitimate attempt. Some people will vary between these two. Assume that you can compare predictions to the actual outcome after the event has happened. That is, that you can measure each respondent's accuracy.

How could you probabilistically tell those that were just guessing from those that were giving their true opinion, keeping in mind that we don't want penalize poor performance. We wish to maintain as much diversity of opinion as possible. But we do wish to eliminate random guessers.

Improve this question
$\endgroup$
5
$\begingroup$

This is a difficult and widely studied problem. One of the big reasons people are interested in it is that people like to use Amazon Mechanical Turk (AMT) for experiments, but many of the turkers on AMT just guess.

The simplest strategy to avoid guessers is to manually establish the correct answer to some questions, and then weed out those people who answer too many of these questions wrong. In your case, you could plant some 'obvious' prediction problems in your survey ("do you think sun will rise tomorrow"), and then eliminate the people who do badly on those. Doing something like this is probably your best bet.

There is some literature on trying to detect guessers in a fully automatic way (i.e. without planting any questions with known answers), but it is usually quite complicated and the methods don't always work. Here's my own attempt at it from a couple years back: http://cs229.stanford.edu/proj2010/ChenHsuWager-HumanAccuracyAnalysisOnAmazonMechanicalTurk.pdf. Of course, the problem addressed there is easier than yours, because we assumed that there was a `true' answer to each question, while you allow for diversity of opinions.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your reply. The machine learning approach that your paper outlines is pretty much what I had in mind. I will have a detailed look through it. In your opinion how successful was your paper's approach? $\endgroup$
    – Chris Tarn
    May 28 '13 at 10:59
  • $\begingroup$ I read through your paper and found it to be quite informative. I think that the supervised approach seems to be the way to go when establising a base line with which to evaluate and filter poor performers. Also, I want to point out that my scenario does have a 'true' after the fact answer. i.e. we can evaluate respondents' predictions against what actually happend. $\endgroup$
    – Chris Tarn
    May 28 '13 at 12:41
  • $\begingroup$ I also found one of the papers you reference "Learning from Crowds" to be very helpful umiacs.umd.edu/labs/cvl/pirl/vikas/publications/… $\endgroup$
    – Chris Tarn
    May 28 '13 at 12:44
  • $\begingroup$ I'm glad you found it helpful! All these papers are a few years old now; someone may have come up with better methods in the mean time since I think this is a fairly active research area. They would all probably cite the "learning from crowds" paper though. $\endgroup$
    – Stefan Wager
    May 28 '13 at 16:29
  • $\begingroup$ Thanks for the suggestion. Indeed, Google Scholar shows this paper has been cited by 140 other papers. scholar.google.com/… many of which are focused on precisely this area of research $\endgroup$
    – Chris Tarn
    May 29 '13 at 1:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.