I run a binary logistic regression, with a binary dependent variable and a continuous independent one.

Now I want to evaluate the out-of-sample performance of the classification algorithm so obtained. For instance, given a probability threshold, I want to compute out-of-sample accuracy or sensitivity.

One solution would be resampling-based estimators: for instance, the method of validation set. I would split the dataset into a training set and a test set, train the model on the training set, and evaluate the performance (e.g. accuracy or sensitivity) on the test set.

This is a generic approach, which can be used with any estimator. For example, I can use this approach also with SVM or neural networks.

But logistic regression is different in many ways from a (deep) neural network: in particular, we know a lot about the properties of the estimator. For instance, we know the confidence interval for the estimated coefficients, and we know an estimate for the irreducible error in the data.

My question is: are there any analytical solutions (not re-sampling based like validation set or cross-validation) to compute out-of-sample performance metrics like accuracy or sensitivity?

For instance, with least squares linear or polynomial regression, I could train the model on the entire dataset (without splitting train/test) and then get the Leave-One-Out Cross-Validation MSE with the following formula: $$ \text{CV} = \dfrac{1}{n} \sum\limits_{i=1}^n \left( \dfrac{y_i - \hat{y}_i}{1-h_i} \right)^2 $$

(see James, Introduction to Statistical Learning, pag. 180)

Is there a similar formula for the accuracy of logistic regression?

  • 1
    $\begingroup$ How is the linked question an answer to my question? I am asking for a formula to compute accuracy/sensitivity/etc. from the properties of the estimator, not "the bootstrap" $\endgroup$ Apr 12, 2021 at 5:49
  • $\begingroup$ I suggest you edit your question to clarify that you're looking for formulae - analytical solutions - rather than resampling-based estimators. $\endgroup$ Apr 12, 2021 at 8:27
  • $\begingroup$ @Scortchi-ReinstateMonica I have modified the question. I hope it is now clear. I have also removed the code which was misleading $\endgroup$ Apr 12, 2021 at 9:30
  • $\begingroup$ Thanks - I think it's clear now. Might be worth specifying it's out-of-sample performance you want to estimate. And that the analytical solution for linear regression is adjusted $R^2$. Note also that the threshold on predicted probability of class membership needs to be stipulated for accuracy. $\endgroup$ Apr 12, 2021 at 10:37
  • 1
    $\begingroup$ @Scortchi-ReinstateMonica I forgot! In case of linear or polynomial regression, we have an analytical solution for the (out-of-sample) LOOCV. A very simple one that allow to compute LOOCV MSE without actually retraining the model. That would be the best comparison I would have. I have modified the question again! $\endgroup$ Apr 13, 2021 at 12:36


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.