2
$\begingroup$

Having fit a logistic regression, I want to do cross-validation.

How can I do this? The usual method of computing predictions and calculating the accuracy doesn't work ... because ... there's nothing to predict. My logistic regression predicts a probability, which I can't compare to anything, as my data is binary 1/0 values.

You could try and predict 1 if the predicted probability is greater than 0.5 and vice-versa, but that only works for trivial cases: in my case, the probabilities are always less than 0.5, so this method would always predict y = 0.

$\endgroup$
1
  • 1
    $\begingroup$ You could use the logloss on validation set predictions as your validation metric $\endgroup$
    – David
    Oct 1, 2019 at 13:14

1 Answer 1

5
$\begingroup$

there's nothing to predict

Certainly there is. And your logistic regression outputs probabilistic predictions. Which is a Very Good Thing. So what you need to do is to assess the quality of probabilistic predictions. (Out-of-bag, as always in cross-validation.)

The tool to do so is , which map probabilistic predictions and corresponding outcomes to scores. In your case, the simplest approach would be to use the logarithmic scoring rule.

Suppose your prediction for a particular instance $i$ to be of class 1 is $\hat{p}_i$. If the instance in fact is of class 1, the score is $s_i:=\log\hat{p}_i$. If it is of class 0, $s_i:=\log(1-\hat{p}_i)$. (It's always the log of the predicted probability for the actual class.)

The you just sum over the scores, $S:=\sum_i s_i$. A higher total score is better. (Some people use the opposite convention and minimize the score, then they work with negative logs.)

You may be interested in my answer to Why is accuracy not the best measure for assessing classification models? Or in Classification probability threshold.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.