# How to evaluate quality of probability estimator for Bernoulli experiments?

Given that I have a set of bernoulli experiments, each with a different and unkown probability $p_i$ and an outcome $x_i$, and an estimator that for each experiment gives a prediction of the probability of the event, I want to measure the prediction quality of the estimator.

Example: I have a stack of n "unfair" coins, each with a different probability $p_i$ for heads and $1-p_i$ for tails. The probabilities are unknown and I can flip each coin only once. Assume that there is a "coin flipping expert" which can have a close look at each coin before flipping them and make an estimate for the probabilities, based on form, size, width, regularity and so on. After the expert makes his prediction, the coin is flipped and the result is noted.

After all coins are flipped, I want to measure how good the expert was, for example on a scale between 0 and 1, where 1 means perfect prediction and 0 means pure randomness. I would also be interested in bias / variance of the predictor.

-
Related -perhaps- math.stackexchange.com/questions/16455/… –  leonbloy May 13 '12 at 23:47

You can quantify the quality of the estimator by calculating the total surprisal of all of the coin flips.

Suppose that your expert makes predictions $q_i$ for each coin. Then, given indicator variables for the coins coming up heads $x_i$, the total surprisal is:

\begin{align} \sum_i\left[ -x_i\log q_i - (1-x_i)\log (1-q_i)\right]. \end{align}

The expected value of the surprisal given the true values $\{p_i\}$ is the cross-entropy: \begin{align} \sum_i \left[-p_i\log q_i -(1-p_i)\log (1-q_i)\right]. \end{align} It is nonnegative, and achieves its minimum value (the entropy of $\{p_i\}$) if and only if $p_i = q_i \forall i$.

If you subtract the entropy from the cross-entropy, you get the relative entropy (whose minimum value is zero). If you take $e^{-x}$ of that, you have a number in $[0, 1]$ as you wanted with a reasonable probabilistic interpretation.

-
One question - for $q_i \in {0,1}$ the formula does not really work - what to do with these values? –  Julian May 13 '12 at 6:27
@Julian: If you guess $q_i=0$ and $x_i=0$, then that is not surprising at all. If $x_i=1$, then that is infinitely surprising. Is that not what you want? (If you're coding this make sure that zero times log of zero is zero.) –  Neil G May 13 '12 at 15:17
I just don't think it's good to sum over values where some of the values can be infinite. I mean it IS bad to say you're 100% sure and you're wrong, but is it infinetly worse than saying you're 99% sure? –  Julian May 15 '12 at 16:10
@Julian: I wish I had more time to provide a longer explanation, but maybe it would be helpful to read the wikipedia entry on entropy under "Characterization — additivity". Remember that an infinite surprisal corresponds to a zero probability. Are you okay with a product of terms some of which may be zero? After studying the characterization of entropy, I wonder if your intuition about what you "think is not good" will shift. –  Neil G May 17 '12 at 8:31

If I understand your question correctly, you might want to check out this question. As I explained there, one way of assessing the calibration of probability predictions is with a scoring rule. A common example of a scoring rule is the Brier score: $$BS = \frac{1}{N}\sum\limits _{t=1}^{N}(f_t-o_t)^2$$ where $f_t$ is the forecasted probability of the event happening and $o_t$ is 1 if the event did happen and 0 if it did not.

Of course the type of scoring rule you choose might depend on what type of event you are trying to predict. However, this should give you some ideas to research further.

Perfect prediction with the Brier score would actually be 0 though, so you could take $1 - BS$ if that quality is important to you. Note though that the other extreme score (0 or 1 depending upon whether you decide to flip the Brier score) actually would not be pure randomness but rather would represent getting the wrong answer every time.

-