RMSE (Root Mean Squared Error) for logistic models

Question

I have a question regarding the validity of using RMSE (Root Mean Squared Error) to compare different logistic models. The response is either 0 or 1 and the predictions are probabilities between 0-1?

Is the way applied below valid with the binary responses also?

##### Using glmnet
require(glmnet)
load(url("https://github.com/cran/glmnet/raw/master/data/BinomialExample.RData"))
cvfit = cv.glmnet(x, y, family="binomial", type.measure="mse")
A     = predict(cvfit, newx=x, s="lambda.min", type="response")
RMSE1 = mean((y - A)^2)
# 0.05816881

##### glm
mydata      = read.csv("https://stats.idre.ucla.edu/stat/data/binary.csv")
mydata$rank = factor(mydata$rank)
mylogit     = glm(admit~gre+gpa+rank, data=mydata, family="binomial")
AAA         = predict(mylogit, newdata=mydata, type="response")
RMSE2       = mean((mydata$admit - AAA)^2)
# 0.194714

Answer 1

To understand why "[t]he response is either 0 or 1 [but] the predictions are probabilities between 0-1", you need to understand the type of model you are working with. Strip away the penalization methods and the cross validation, and you are running a basic logistic regression. The parameters are fit on the log odds / logistic scale. This is called the "linear predictor". (For more on this, it may help you to read my answer here: Difference between logit and probit models.) If you plugged in an x value and simplified, the value would be the model's predicted natural logarithm of the odds of 'success' (1). If you exponentiated that value, you would have the model's predicted odds of 'success'. To get a predicted probability, you would need to convert the odds into a probability via odds/(1+odds). (For more on this, it may help you to read my answer here: Interpretation of simple predictions to odds ratios in logistic regression.) This still doesn't get you to a predicted class. To get that, you would need to compare your predicted probability to some threshold, and if it is less than the threshold predict 'failure' (0), else 'success' (1). The most common default threshold is .5, but this is often not optimal. R's predict.glm() function will allow you to use type="link", which outputs predictions on the scale of the linear predictor (i.e., before all those transformations above), but that won't help you in this context. Using type="response" gives you the predicted probabilities.

---

When you are trying to assess how well a binary (e.g., logistic) regression model predicts a response, you have several options:

1. The first, and most intuitive is to compare the predicted class with the observed class and compute the percent correct. Although it is intuitive, it has problems. It is contingent, in part, on the threshold being optimal in addition to the other aspects of the model being fit appropriately. It also throws a lot of information away (i.e., how far from the threshold the predicted probability is), which isn't a good thing to do.
2. Your next option is to use the area under the Receiver Operating Characteristic (ROC) curve. Most people use this option; it is considerably better than using the percent correct. The thing about the AUC that most people don't realize, though, is that it is actually measuring the appropriate ordering of your predictions, not their actual accuracy. That is, if you had predicted probabilities for four observations of .2, .4, .6, .8, and you added .01 to all of them (.21, .41, .61, .81), the AUC would be the same even though both sets of predicted probabilities cannot be equally accurate.
3. The third way to assess your model would be to use a proper score function. Perhaps the most popular score function in your context is the Brier score. As @fcoppens notes, the method in your code is the Brier score. It will assess not only if the model appropriately predicts that one observation is more likely to be a 'success' than another observation, but if the model's predicted probability is actually correct.

It is unfortunate that the Brier score is less well known and used. Although the relative intuitiveness of these methods is: percent correct > AUC > Brier score, their true informativeness is the opposite: Brier score > AUC > percent correct. If you only want a simple measure of the performance of your model, the distinction between these methods is less important, but if you want to use these methods to optimize a model or to select a model, using an inferior method will lead to worse performance.

Answer 2

Using RMSE makes a lot more sense if your error term is distributed normally with mean zero, but in binary outcomes it is not. Therefore, I think you should use different approaches that lend themselves to binary outcomes, like comparing differences in likelihood.