I have data in which about 150 subjects each separately estimate 6 different quantities. The quantities are the answers to general knowledge questions like "How many far apart in kilometres are Milan and Minsk?", for which it's very unlikely anyone will know the exact value.
I have the true value for all 6 quantities.
I can see how confident (on a scale of 1 to 5) each subject was in each estimate that they made.
I'm interested in the accuracy of aggregations of estimates, and with that in mind I can calculate the mean or median estimate across everyone. This produces rather accurate results, with the median doing particularly well (the mean gets affected by wild outliers, which in this context are possible). This is often known as the wisdom of the crowd.
However, I wonder if it might work better to weight the estimates by confidence, with a confident estimate being weighted more highly than an unconfident estimate.
How can I decide what the optimal weighting scheme is? As per the comments below, it may be best to assume that minimizing the MSE of the crowd's estimate is the goal, although minimising median error, or any other appropriate measure of error, would also be fine.
I investigated the data a bit further, and have made it accessible here.
I looked at the correlation between confidence and MSE on the questions, and it's -0.06. This is a modest correlation, but it's in the right direction, since that indicates that higher confidence was associated with lower MSE.