What metric should be used to compare models when the prediction is a probability of an event occurring? Let's say that I am trying to predict whether or not it will rain today, and I build a model that gives me a % chance of the event occurring.
Now let's say I build another model, and I want to see which one is better...
model1output <- c( 0.25 , 0.50 , 0.50 , 0.75 , 0.50 , 0.50 )
model2output <- c( 0.00 , 0.25 , 0.50 , 0.25 , 0.50 , 0.00 )
model3output <- c( 0.50 , 0.50 , 0.50 , 0.50 , 0.50 , 0.50 )
actual       <- c( 0.00 , 0.00 , 1.00 , 1.00 , 1.00 , 0.00 )

residualModel1 <- model1output - actual
residualModel2 <- model2output - actual
residualModel3 <- model3output - actual

mseModel1 <- mean((residualModel1)^2)
mseModel2 <- mean((residualModel2)^2)
mseModel3 <- mean((residualModel3)^2)

avgErrorModel1 <- mean(residualModel1)
avgErrorModel2 <- mean(residualModel2)
avgErrorModel3 <- mean(residualModel3)

errVarianceModel1 <- mean((residualModel1 - avgErrorModel1)^2)
errVarianceModel2 <- mean((residualModel2 - avgErrorModel2)^2)
errVarianceModel3 <- mean((residualModel3 - avgErrorModel3)^2)

These metrics give me the following...
> mseModel1
[1] 0.1875
> mseModel2
[1] 0.1875
> mseModel3
[1] 0.25
> 
> avgErrorModel1
[1] 0
> avgErrorModel2
[1] -0.25
> avgErrorModel3
[1] 0
> 
> errVarianceModel1
[1] 0.1875
> errVarianceModel2
[1] 0.125
> errVarianceModel3
[1] 0.25
> 

Model 1 has a 50% error on 4 of 6 cases.  It has a 25% error on 2 of 6 cases.
Model 3 has a 50% error on 6 of 6 cases.  Thus... at least model 1 is better than model 3.
But where to go from here?
 A: The main metrics for this are crossentropy, Brier score (squared error), and area under the curve.
Crossentropy, also known as log loss and negative log likelihood, is related to the maximum likelihood estimation technique classically used in logistic regression and other generalized linear models. Maximum likelihood estimation is an extremely standard technique throughout statistics.
$$Crossentropy(y,p)=-\dfrac{1}{n}\sum_{i=1}^n \left[y_i\log(p_i)+(1-y_i)\log(1-p_i)\right]\\y_i\in\{0,1\}$$
Frank Harrell, the founder of Vanderbilt University’s biostatistics department and a frequent poster on here, often calls this the “gold standard” when it comes to probability estimation.
Brier score is square loss, calculated exactly the same way you would calculate it for linear regression.
$$
BrierScore(y,p)=\dfrac{1}{n}\sum_{i=1}^n (y_i-p_i)^2
\\y_i\in\{0,1\}
$$
An advantage of these two metrics is that both are examples of so-called strictly proper scoring rules. The technical definition of that term is that strictly proper scoring rules are uniquely optimized in expected value by the true probability values; for intuition, you can think of them as seeking out the true probability values and only the true probability values.
Area under the curve is related to the receiver-operator characteristic curve, which is created by considering all of your predicted probability values at every threshold for making a hard classification (so below $0.01$ is $0$, otherwise $1$; then above $0.02$ is $0$, otherwise $1$, etc); and plotting the sensitivity on the vertical axis and $1$ minus specificity on the horizontal axis. Unlike crossentropy or Brier score, this is not a strictly proper scoring rule. Some people like this one because it gives some absolute sense of performance that is bounded between $0$ and $1$. Crossentropy and Brier score both can be modified, like squared error in linear regression being related to $R^2$, to have that same property of being between $0$ and $1$ (McFadden’s and Efron’s $R^2$, respectively), and I have my doubts about how useful AUC is (a doubt shared by the aforementioned Frank Harrell). Nonetheless, many people will use this measure.
UCLA has a webpage about many other metrics for predicted probabilities.
A: I believe the answer is entropy.
http://www.cs.rochester.edu/u/james/CSC248/Lec6.pdf
