How to evaluate quality of probability estimator for Bernoulli experiments?

Question

Given that I have a set of bernoulli experiments, each with a different and unkown probability $p_i$ and an outcome $x_i$, and an estimator that for each experiment gives a prediction of the probability of the event, I want to measure the prediction quality of the estimator.

Example: I have a stack of n "unfair" coins, each with a different probability $p_i$ for heads and $1-p_i$ for tails. The probabilities are unknown and I can flip each coin only once. Assume that there is a "coin flipping expert" which can have a close look at each coin before flipping them and make an estimate for the probabilities, based on form, size, width, regularity and so on. After the expert makes his prediction, the coin is flipped and the result is noted.

After all coins are flipped, I want to measure how good the expert was, for example on a scale between 0 and 1, where 1 means perfect prediction and 0 means pure randomness. I would also be interested in bias / variance of the predictor.

Answer 1

You can quantify the quality of the estimator by calculating the total surprisal of all of the coin flips.

Suppose that your expert makes predictions $q_i$ for each coin. Then, given indicator variables for the coins coming up heads $x_i$, the total surprisal is:

\begin{align} \sum_i\left[ -x_i\log q_i - (1-x_i)\log (1-q_i)\right]. \end{align}

The expected value of the surprisal given the true values $\{p_i\}$ is the cross-entropy: \begin{align} \sum_i \left[-p_i\log q_i -(1-p_i)\log (1-q_i)\right]. \end{align} It is nonnegative, and achieves its minimum value (the entropy of $\{p_i\}$) if and only if $p_i = q_i \forall i$.

If you subtract the entropy from the cross-entropy, you get the relative entropy (whose minimum value is zero). If you take $e^{-x}$ of that, you have a number in $[0, 1]$ as you wanted with a reasonable probabilistic interpretation.

Answer 2

If I understand your question correctly, you might want to check out this question. As I explained there, one way of assessing the calibration of probability predictions is with a scoring rule. A common example of a scoring rule is the Brier score: $$BS = \frac{1}{N}\sum\limits _{t=1}^{N}(f_t-o_t)^2$$ where $f_t$ is the forecasted probability of the event happening and $o_t$ is 1 if the event did happen and 0 if it did not.

Of course the type of scoring rule you choose might depend on what type of event you are trying to predict. However, this should give you some ideas to research further.

Perfect prediction with the Brier score would actually be 0 though, so you could take $1 - BS$ if that quality is important to you. Note though that the other extreme score (0 or 1 depending upon whether you decide to flip the Brier score) actually would not be pure randomness but rather would represent getting the wrong answer every time.

Answer 3

The above answer may be useful but is complicated and I am not sure how to apply it. Thinking in simple terms this is a special case of testing whether or not a person has psychic powers (really no different from a scientific evaluation of the mechanics and physics of the coin flip). Obviously have psychic capability would have to be defined as doing something much better than by random chance. I would first want to define what is enough. The difficulty is to decide how much better than random guessing is and what random guessing will do. If all the coins were fair random choice would be 0.5 and so maybe saying anything over 0.75 is Then testing the hypothesis that the person has psychic powers is doing a one-sided hypothesis that a binomial parameter p is <= to 0.75 versus the alternative that it is greater. As an estimator I have chosen the binomial parameter for successfullu calling heads or tails and the variance of my estimator is p(1-p)/n. The added difficulty is that the coins are not fair and the individual pis are unknown. I would still define chance as random guessing of heads or tails and 0.5 is what I would test against. However with unfair coins there may be statistical strategies that would lead to a better than chance success rate but neither indicate skill with the individual coin flips or psychic powers. To illustrate suppose the average for the pis is 0.80. Then after seing heads come up much more frequently than tails we could switch to an all heads strategy and tend to be correct close to 80% of the time. This assumes randomguessing until we are convinced that head occurs much more often than tails and at that point we switch to all heads. So without knowing the pis or at least their averages I cannot tell what success rate would indicate skill. Comparing against random guessing is not the standard to beat in this case. Note that my argument only makes sense if the stack of coins is very large.

Answer 4

Another way to evaluate the quality is to look at the Reliability Diagram. This does not immediately yield a value between 0 and 1, but can be used for a first visual evaluation.