Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Suppose I have $k$ 'experts', from whom I would like to elicit a prior distribution on some variable $X$. I would like to motivate them with real money. The idea is to elicit the priors, observe $n$ realizations of the random variable $X$, then divvy up some predetermined 'purse' among the experts based on how well their priors match the evidence. What are suggested methods for this last part, mapping the priors and evidence onto a payout vector?

share|improve this question
    
Since there is probably no right answer, we might want to CW this one. I leave that to moderator's discretion. –  shabbychef May 9 '12 at 0:44
1  
There may be a single objectively valid good answer to this question, so I hesitate to turn it into CW. –  whuber May 9 '12 at 0:47
1  
This is similar to the idea of prediction markets. PredictionBook is a decent place to look. –  EMS May 9 '12 at 5:47

3 Answers 3

In the spirit of my comment above, I think the right thing to consider is a prediction market. You should sell securities that have some fixed payoff for accuracy of predictions. You can use standard measures of probabilistic distance, such as those mentioned by Daniel Johnson in his answer. But the point is to fix the payouts in the form of securities and fix standards of measure ahead of time (preferably just use binary events, such as $A$ happened or it did not). That way, if someone is willing to pay $\$$X for a security that pays $1.00 if the event it covers does actually happen, you know they assign probability X to the event that the security covers. Market liquidity will take care of how the securities are distributed among the experts.

I think this is superior to having a fixed payout vector such as you might have for a golf tournament. The reason is that in a golf tournament, all that matters is how well you do against competitors, not your overall score. When you want to incentivize the most accurate prior beliefs possible, you don't want people thinking they only have to outdo one another to get the prize... you want them to be willing to wager their own money to get payouts because then they must themselves believe in their prior assessment, not just that their prior assessment is better than someone else's.

share|improve this answer
    
It's also worthwhile to note that effects of market manipulation have been experimentally studied in prediction markets (see here and here), and while more work needs to be done it appears that participants can easily compensate for malicious manipulators. The empirical results suggest that it would be extremely difficult to 'game' the system, as you mentioned in your other comment –  EMS May 15 '12 at 19:11

The keyword to look for is scoring rules: these are functions for evaluating and rewarding probabilistic predictions, and there has been quite a bit of work on the topic, going back to the 50s. The main thing you need to check is that it is proper, that is, that the expert from whom you're eliciting the prior has the incentive to be honest.

There are quite a lot of possible proper scoring rules: one of the simplest is the logarithmic scoring rule: you reward the expert with a (linear function of) the log-probability that they assigned to the event.

share|improve this answer
    
Thanks! I was leaning towards something like this. In particular, I wanted it to be difficult to 'game' the system by an agent without any information. –  shabbychef May 15 '12 at 18:41
    
Check the comment I added to my answer above (link), because there's some promising research about how prediction markets are specifically robust against manipulators and others trying to 'game' the system. This is genuinely superior to simple scoring rules that offer payouts only for getting better accuracy than peers. –  EMS May 15 '12 at 19:12
    
@EMS: What makes prediction markets superior? The whole point of a scoring rule is that the score IS independent of the competitors (though admittedly they are not often implemented that way in practice: i.e. all the money is given to the person with the highest score) –  Simon Byrne May 17 '12 at 10:52

If the true distribution is known by the one paying the money, a natural statistic to look at would be the relative entropy of the given prior and the true distribution. Then the payout could just be some monotone decreasing function of the relative entropy.

However, I am guessing that you are interested in the case where the true distribution is unknown and payouts must be decided using only the $n$ data points. One way you could do this is to consider the sum of the likelihood of the data points under each prior distribution. So, more formally, $\text{score}(\text{prior } j) = \sum^n_{i=1}P_j(X = x_i)$.

Another method would be very similar to the first where I assumed we knew the distribution of $X$. Since we have $n$ data points, we can use this information to approximate the true distribution using kernel density estimation. The relative entropy can then be computed between the estimated distribution and each of the priors provided by the experts.

share|improve this answer
    
Surely, though, the experts would take all these things into account before giving you their "prior". –  EMS May 9 '12 at 5:46
2  
Do you mean if the experts were allowed to view the $n$ data points? I was under the impression that their priors were created before the samples were drawn and could not be functions of this data. –  Daniel Johnson May 9 '12 at 6:06
    
It wouldn't be the same $n$ data points, but I assume they would try to collect data somehow, or connect the problem to something for which they have data. Otherwise, I don't know how a human could just verbally say a prior belief. How would you know it corresponded to their internal beliefs, and wasn't influenced like price fixing or something by the huge $\$$3.99 sign they saw at the gas station on the way to the experiment? Also... meow. –  EMS May 9 '12 at 6:10
1  
What's neat is that w.r.t. physical quantities, like estimating distances or the number of M&M's in a jar, humans are unbiased estimators: take the average of a large number of guesses and it's usually very close. But w.r.t. non-physical quantities, like what the price of gas will be next month, humans (even experts) are terrible, even in average. The literature on planning fallacy is scary, especially the examples of professional urban planners consistently poorly estimating the costs of municipal projects, much like how stats grad students do poorly with conjunction fallacy bias. –  EMS May 9 '12 at 6:21
1  
The main one I know offhand was some stuff discussed in an old book on vision, "The Ecological Approach to Visual Perception" by James Gibson. He mentioned some experiments having people estimate distances on a football field between two people standing far away, and other similar things. I can't recall where I heard the M&M's thing, but I'll try to find some sources on it. –  EMS May 9 '12 at 6:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.