2
$\begingroup$

Say you have a binary classification problem, but you'd like to have a sense of how confident the classifier is by using a numerical score and then using a threshold for the binarization.

This can be done by reaching into the model and using it's scoring function, but that's not always available in library implementations.

Many ML libraries have regression equivalents to classifiers, like SVR of Random Forest Regressors.

Is it ok to use regression on labels as a proxy for a classifier confidence score? Are they different?

By this I mean, say I have $n$ samples $X_1, ..., X_n$ and class labels $y_1,...,y_n$ with $y_i \in \{0,1\}$. I could train a binary classifier on these, or I could pretend that the $y_i \in \mathbb{R}$ and the labels happen to be 1.0 or 0.0.

If I train a regressor on this formulation, what is wrong with using the outputs as surrogates for classifier scores, and then evaluating the model with ROC and AUC?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ It is not clear what do you want to do - could you edit and clarify? What classifiers are you using? Why wouldn't you just use bootstrap in here? $\endgroup$
    – Tim
    Commented Nov 29, 2015 at 20:58
  • $\begingroup$ You're wrong. Confidence scores are available. randomForest(R) and RandomForestClassifier(Sklearn/python) inplements confidence scoring based on votes and/or content of terminal nodes. And svm for R and python has a confidence measure also. Try google 'svm probability' $\endgroup$
    – Soren Havelund Welling
    Commented Nov 30, 2015 at 14:03
  • $\begingroup$ It's true that some of these implementations do allow you to see the decision function, and I acknowledge that. I still don't think this takes away from the point of the question, which is "what is wrong about using regression for classification?" Logistic regression already does this, why can't other regressions be used for it? $\endgroup$
    – Taaam
    Commented Nov 30, 2015 at 17:50
  • $\begingroup$ What do you mean logistic regression does that? Logistic regression estimates a conditional probability, just like random forests, gradient boosting, neural networks, and pretty much all other useful ml algorithms. $\endgroup$
    – Matthew Drury
    Commented Jan 10, 2018 at 1:46

2 Answers 2

Reset to default
1
$\begingroup$

In machine learning, you can get away with many approximations if you can show they are useful. There are some questions and answers on this site stating that in some cases, linear regression can do for classification without using the extensions and adaptations that logistic regression made to it for that purpose.

In the case of random forest, know that they already come with approximate label confidences defined by the proportion of trees that classify the record with that label.

But who knows, perhaps you are on to something. You would just need to prove it empirically. When you want to show that your way provides better label confidences, you are saying that across the spectrum of possible classification cutoffs, they should reach better classification results than with the other way of estimating label confidences. You could compare classification performance with ROC AUC because that is what a ROC curve represents.

Improve this answer
$\endgroup$
0
$\begingroup$

No, regression typically has asymmetrical penalty for 'mistakes' which would cause various problems in modeling your data. i.e. if your model predicts scores greater than one, that is the same type of error as if it would predict less than one, and classifiers would flatten results greater than one.

You can look into logistic regression if you want something in that direction.

If you want actual confidence scores you can look into Bayesian logistic regression.

Improve this answer
$\endgroup$
1
  • $\begingroup$ How is the penalty asymmetric? In the same way that it may predict >1 for some instance, it may also predict <0 for some other instance. $\endgroup$
    – Taaam
    Commented Nov 30, 2015 at 17:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.