Suppose I have $k$ 'experts', from whom I would like to elicit a prior distribution on some variable $X$. I would like to motivate them with real money. The idea is to elicit the priors, observe $n$ realizations of the random variable $X$, then divvy up some predetermined 'purse' among the experts based on how well their priors match the evidence. What are suggested methods for this last part, mapping the priors and evidence onto a payout vector?
Tell me more
×
Cross Validated is a question and answer site for
statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
|
|
In the spirit of my comment above, I think the right thing to consider is a prediction market. You should sell securities that have some fixed payoff for accuracy of predictions. You can use standard measures of probabilistic distance, such as those mentioned by Daniel Johnson in his answer. But the point is to fix the payouts in the form of securities and fix standards of measure ahead of time (preferably just use binary events, such as $A$ happened or it did not). That way, if someone is willing to pay $\$$X for a security that pays $1.00 if the event it covers does actually happen, you know they assign probability X to the event that the security covers. Market liquidity will take care of how the securities are distributed among the experts. I think this is superior to having a fixed payout vector such as you might have for a golf tournament. The reason is that in a golf tournament, all that matters is how well you do against competitors, not your overall score. When you want to incentivize the most accurate prior beliefs possible, you don't want people thinking they only have to outdo one another to get the prize... you want them to be willing to wager their own money to get payouts because then they must themselves believe in their prior assessment, not just that their prior assessment is better than someone else's. |
||||
|
|
The keyword to look for is scoring rules: these are functions for evaluating and rewarding probabilistic predictions, and there has been quite a bit of work on the topic, going back to the 50s. The main thing you need to check is that it is proper, that is, that the expert from whom you're eliciting the prior has the incentive to be honest. There are quite a lot of possible proper scoring rules: one of the simplest is the logarithmic scoring rule: you reward the expert with a (linear function of) the log-probability that they assigned to the event. |
|||||||
|
|
If the true distribution is known by the one paying the money, a natural statistic to look at would be the relative entropy of the given prior and the true distribution. Then the payout could just be some monotone decreasing function of the relative entropy. However, I am guessing that you are interested in the case where the true distribution is unknown and payouts must be decided using only the $n$ data points. One way you could do this is to consider the sum of the likelihood of the data points under each prior distribution. So, more formally, $\text{score}(\text{prior } j) = \sum^n_{i=1}P_j(X = x_i)$. Another method would be very similar to the first where I assumed we knew the distribution of $X$. Since we have $n$ data points, we can use this information to approximate the true distribution using kernel density estimation. The relative entropy can then be computed between the estimated distribution and each of the priors provided by the experts. |
|||||||||||||||||
|
