2
$\begingroup$

I'm training the logistic regression for binary classification on a labeled data set. Now I'm using the same entries and predict their scores using the model.

For example, I have an entry with label 0 and predicted score is 0.1 and another entry with known label 0 and predicted score 0.2. So basically I'm using model to get probability for seen (as opposite to unseen) data.

And I'm trying to argue whether the predicted probability shows the ordering/ranking of the entries - second entry from the example is closer to class 1 than first entry? Or do they just show the performance of my model?

This contradicts to common approach when trained model is used on unseen data, and I feel that comparing the scores of training data has no sense, but I can't understand why

$\endgroup$
  • $\begingroup$ Is there a particular reason you would not use bootstrap or repeated cross-validation in your work? (There are certain metrics that can be helpful for what you want, eg. AUCROC, Brier Score but if you looking into in-sample performance it is very hard to argue it is not overly optimistic.) $\endgroup$ – usεr11852 Jan 12 at 23:02
2
$\begingroup$

Logistic regression models, though used for categorical Y, are not classification methods. They are instead used to estimate tendencies, i.e., probabilities. Briefly, a classifier is appropriate when the probabilities of class membership hover around 0.0 and 1.0, i.e., when the signal:noise ratio is exceedingly high. Otherwise it is not often useful to force probabilities to be converted to labels. At any rate that would require a utility/loss/cost function because it is a decision, not a prediction. Details may be found here.

$\endgroup$
  • 1
    $\begingroup$ Maybe I've used wrong terminology. I'm not trying to use logistic regression as classifier. All my data is classified and has labels already. And I won't be having any more data I need to classify. I need to know whether the probabilities of logistic regression can be compared with each other. In particular, I'd like to look at the entries that have 0 label and see which of them are more close to 1 than the others. $\endgroup$ – Alexey Smirnov Jan 3 at 14:56
  • $\begingroup$ Then don't use the classifier terminology but rather describe this as probability estimation on a continuous scale. But to your question, you will learn a lot from plotting a high-resolution histogram (with a lot of bins) of the predicted probabilities, and by computing measures of predictive discrimination such as the c-index (concordance probability, equal to the area under an ROC curve but a more intuitive way to think about it). $\endgroup$ – Frank Harrell Jan 3 at 17:20
  • $\begingroup$ still this doesn't answer the question if probabilities of the training data can be compared $\endgroup$ – Alexey Smirnov Jan 4 at 11:02
  • 1
    $\begingroup$ You're correct to assume that examining the relationship between estimated probabilities that Y=1 and the observed values of Y (0,1) can show too much separation if overtraining (overfitting) has occurred. Still you might show the scatterplot of probability vs. Y. But a calibration curve is also useful, and you should adjust the cal. curve for overfitting by resampling, repeating the whole model estimation process many times using repeated cross-validation or the bootstrap. Details are in my RMS book and course notes. $\endgroup$ – Frank Harrell Jan 4 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.