5
$\begingroup$

I have data in which about 150 subjects each separately estimate 6 different quantities. The quantities are the answers to general knowledge questions like "How many far apart in kilometres are Milan and Minsk?", for which it's very unlikely anyone will know the exact value.

I have the true value for all 6 quantities.

I can see how confident (on a scale of 1 to 5) each subject was in each estimate that they made.

I'm interested in the accuracy of aggregations of estimates, and with that in mind I can calculate the mean or median estimate across everyone. This produces rather accurate results, with the median doing particularly well (the mean gets affected by wild outliers, which in this context are possible). This is often known as the wisdom of the crowd.

However, I wonder if it might work better to weight the estimates by confidence, with a confident estimate being weighted more highly than an unconfident estimate.

How can I decide what the optimal weighting scheme is? As per the comments below, it may be best to assume that minimizing the MSE of the crowd's estimate is the goal, although minimising median error, or any other appropriate measure of error, would also be fine.

I investigated the data a bit further, and have made it accessible here.

I looked at the correlation between confidence and MSE on the questions, and it's -0.06. This is a modest correlation, but it's in the right direction, since that indicates that higher confidence was associated with lower MSE.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Interesting question. You are asking for an objective rule for determining an "optimal" weighting. "Optimal" in this case seems to be a function of, first, whether or not you "weight" and, second, how you weight. If your criterion is accuracy, then developing a matrix of all of the possible combinations (e.g., none, various types of weights, using different metrics -- mean, median, etc. -- and different error metrics -- SD, MAD,etc.) to see where accuracy wrt "truth" is optimized. Another approach to "confidence" is the accuracy or closeness of a unique subject's estimates to the true value $\endgroup$
    – Mike Hunter
    Nov 12, 2015 at 13:48
  • $\begingroup$ I have an old and unanswered question at stats.stackexchange.com/questions/45845/… that asks about the various accuracy criteria that could be applied. It seems that people mostly use Mean Square Error in this context, so that's a reasonable way to go. Is your answer that working out the combinations exhaustively is likely to be the best way to go? $\endgroup$
    – user1205901 - Слава Україні
    Nov 12, 2015 at 22:05
  • $\begingroup$ That's one way to go, but only if there aren't too many combinations. It would certainly help figure out the sequences and the blanks in those sequences. $\endgroup$
    – Mike Hunter
    Nov 12, 2015 at 22:09
  • 2
    $\begingroup$ How "reliable" are the confidence estimates? Have you considered that? Some, not all, people are very confident even when they are (very) wrong. $\endgroup$
    – Mark L. Stone
    Dec 10, 2015 at 2:41
  • $\begingroup$ That's a good point. I am not sure how reliable there are. Is there some calculation I could perform that would satisfy your enquiry? $\endgroup$
    – user1205901 - Слава Україні
    Dec 10, 2015 at 6:12

1 Answer 1

Reset to default
1
$\begingroup$

I do not have any solution for you, but if I were you I would look for inspiration on how to think about the problem in Item Response Theory. Among IRT models, one of the simplest ones is Rasch model for binary data. It is used for modeling student responses for test items. Assume that your data consists of $i = 1,...,n$ students that answered $j = 1,...,k$ test items $X_{ij}$ marked $1$ for correct answer and $0$ for incorrect. The data can be modeled using the following model

$$ P(X_{ij} = 1) = \frac{\exp(\theta_i - \beta_j)}{1-\exp(\theta_i - \beta_j)} $$

where $\theta_j$ is students "ability" and $\beta_i$ is items "difficulty".

As far as I understand, your data is different, but some insights from IRT can be generalized to your problem. You have $k$ questions that (possibly) differ in how difficult they are. Among $n$ persons that answered the questions you can possibly find such persons that were confident about both easy and hard questions -- unfortunately you don't know if they are confident because they are right or if they are overconfident. Similar with the persons who are unconfident about everything. Since, as you say, the questions are frames in such way that it is unlikely that anyone knows the exact answer, then you assume that people ranking high on confidence would be rather overconfident than right. If is it so, then you should probably assign greater weights to answers by people with average ratings of confidence.

As I said, I don't have the ready answer and if you find this idea convincing, then you should think about ways of adapting it to your needs. Rasch model was made for different data and different purpose but the take-away message is to think of your problem in terms of two factors for questions' difficulty and people trait confidence.

Improve this answer
$\endgroup$
3

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.