Calculating misclassification rate for k-fold cross validation (logistic regression)

Question

I am trying to manually write code to perform a k-fold cross validation for a logistic regression model for the first time. Unfortunately I am getting stuck trying to implement the following formula from An Introduction to Statistical Learning with Applications in R:

In this setting, cross-validation works just as described earlier in this chapter, except that rather than using MSE to quantify test error, we instead use the number of misclassified observations. For instance, in the classification setting, the LOOCV error rate takes the form:

$$ CV_{(n)} = \frac{1}{n} \sum\limits_{i=1}^n Err_i $$ where $ Err_i = I(y_i \neq \hat{y_i}). $

What I'm struggling with is how to find out if a specific observation was misclassified. As I understand it, the y-hat values from a logistic regression could be any value between 0 and 1. They can essentially be interpreted as the predicted probability that an outcome = 1 based on that specific combination of predictors.

The y values can only be either 0 or 1. So it seems to me that y and y-hat will never (or almost never) be equal.

So I am left with 2 questions:

1) Do I have some conceptual misunderstanding here?

2) How can I find out how many observations my model has classified correctly?

Answer 1

Your conceptual misunderstanding may come from one of the few complaints I have about ISLR, its overemphasis on misclassification rates as a measure of model quality. If you look at page 158 of ISLR you will see that the authors propose the same solution as @BrentFerrier in another answer here, which is to use a cutoff of 0.5 in the predicted probability for the classification. Although I appreciate the simplicity of this measure from an early-stage pedagogical purpose, this measure has the implicit assumption that all types of classification errors are equally important, which is seldom true in practice.

There are, however, much better possibilities for overall evaluation of logistic regression models than misclassification based on a cutoff of 0.5 probability. This page provides some entry into this area, and this page briefly compares 2 simple measures. For example, the Brier score is the mean-square difference between the predicted probabilities and the actual 0/1 outcomes. Measures of the calibration curve, as provided in the rms package in R, can be even better.

Answer 2

Logistic regression models output a probability of belonging to the positive class. So all you need to do to translate from these probabilities to classes is decide a cut off value. Many default algorithms use the value of .5. Therefore if you have a probability value greater than .5 it is predicting it to be a 1. to find out how many observations you have classified correctly you just compare the original to the predicted class, not the predict probability.

EDIT: The cutoff value can definitely be different per data set. There isn't really a definitive answer to what is best. I would experiment with different cutoff values and try and optimize the AUC value. Since this is a rare event classification problem I would definitely check your model on training and testing sets. If you see a drastic decrease in performance from your training set to your testing set you know that your model is not generalizing the data well.

For more information check out this post: Obtaining predicted values (Y=1 or 0) from a logistic regression model fit