Assume we're polling or surveying people about their predictions regarding some future event. This survey is done every day, and the event they are asked to predict is the same every time. For example, each morning they are asked "Do you think it will be sunny this afternoon?" Assume for the purposes of this question that the respondents are being compensated to give an answer, but there is no incentive to be accurate and there is no downside/penalty for being wrong.
Some of these people will just guess to get the polling process over with each day and thus earn their compensation. Others will make a legitimate attempt. Some people will vary between these two. Assume that you can compare predictions to the actual outcome after the event has happened. That is, that you can measure each respondent's accuracy.
How could you probabilistically tell those that were just guessing from those that were giving their true opinion, keeping in mind that we don't want penalize poor performance. We wish to maintain as much diversity of opinion as possible. But we do wish to eliminate random guessers.