How can I find the optimal weighting scheme to aggregate these estimates? I have data in which about 150 subjects each separately estimate 6 different quantities. The quantities are the answers to general knowledge questions like "How many far apart in kilometres are Milan and Minsk?", for which it's very unlikely anyone will know the exact value.
I have the true value for all 6 quantities. 
I can see how confident (on a scale of 1 to 5) each subject was in each estimate that they made.
I'm interested in the accuracy of aggregations of estimates, and with that in mind I can calculate the mean or median estimate across everyone. This produces rather accurate results, with the median doing particularly well (the mean gets affected by wild outliers, which in this context are possible). This is often known as the wisdom of the crowd.
However, I wonder if it might work better to weight the estimates by confidence, with a confident estimate being weighted more highly than an unconfident estimate. 
How can I decide what the optimal weighting scheme is? As per the comments below, it may be best to assume that minimizing the MSE of the crowd's estimate is the goal, although minimising median error, or any other appropriate measure of error, would also be fine.
I investigated the data a bit further, and have made it accessible here.
I looked at the correlation between confidence and MSE on the questions, and it's -0.06. This is a modest correlation, but it's in the right direction, since that indicates that higher confidence was associated with lower MSE.
 A: I do not have any solution for you, but if I were you I would look for inspiration on how to think about the problem in Item Response Theory. Among IRT models, one of the simplest ones is Rasch model for binary data. It is used for modeling student responses for test items. Assume that your data consists of $i = 1,...,n$ students that answered $j = 1,...,k$ test items $X_{ij}$ marked $1$ for correct answer and $0$ for incorrect. The data can be modeled using the following model
$$ P(X_{ij} = 1) = \frac{\exp(\theta_i - \beta_j)}{1-\exp(\theta_i - \beta_j)} $$
where $\theta_j$ is students "ability" and $\beta_i$ is items "difficulty".
As far as I understand, your data is different, but some insights from IRT can be generalized to your problem. You have $k$ questions that (possibly) differ in how difficult they are. Among $n$ persons that answered the questions you can possibly find such persons that were confident about both easy and hard questions -- unfortunately you don't know if they are confident because they are right or if they are overconfident. Similar with the persons who are unconfident about everything. Since, as you say, the questions are frames in such way that it is unlikely that anyone knows the exact answer, then you assume that people ranking high on confidence would be rather overconfident than right. If is it so, then you should probably assign greater weights to answers by people with average ratings of confidence.
As I said, I don't have the ready answer and if you find this idea convincing, then you should think about ways of adapting it to your needs. Rasch model was made for different data and different purpose but the take-away message is to think of your problem in terms of two factors for questions' difficulty and people trait confidence.
