Given a binary classification problem, is there any inherent difference (or advantage) to using a classifier (say a logistic regression) and a regression, where the classes are denoted by 0 and 1 (or any two numbers)? Further, say, after running the regression, one can learn to optimize the 'cutoff' point (maybe not 0.5 in this example, but it turns out better to cut at 0.45).
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
7
Intriguing question, I had this question for a while,. Here is my findings Short Answer
You can create any number of classifier you want, but the point is, you can only prove a few of them to be Bayes/universally-consistent! ( Bayes consistency means that classifier is asymptotically optimal, i.e. with infinite data its risk limits Bayes risk, which is optimal risk)
The consistency of a classifier, depends on loss function and (inverse)-link function (i.e. mapping from [0 1] probability space to $\mathbb{R}$, and vice versa.)
Long answer
First, according to Tong's great paper all the (consistent) classifiers are equivalent! except in that they are minimizing different loss functions, and almost every difference between classifiers is a consequence of their loss functions. In fact, he showed that minimizing every loss function leads to optimal decision function (technically, inverse-link function), which is completely function of probabilities (even for SVMs!). His result is summarized in this table (by Hamed):
Despite of this unified view over all the classifiers, they are different in their outputs:
- Probability-Calibrated: for these class the classifiers (e.g. Logistic Regression), output is DIRECTLY within a probability measure, which this in turn not only answers yes/no question of the classifier, but also gives confidence of the of the decision.
- Not-probability-Calibrated: Other classifiers (e.g. SVM) are real-valued-output classifiers, which you can use some link functions to calibrate the to enforce outputs to be probabilities.
Conclusion
It really depend on loss-function, link-function, calibration. For example, first line of the table says that, least-squares regression and classification are the same,(if your classifier output is calibrated-probabilities $\eta$, and using the corresponding inverse link function)
-
1$\begingroup$ The equivalence of loss functions was earlier established by this paper M Saerens, P Latinne, C Decaestecker "Any reasonable cost function can be used for a posteriori probability approximation", Neural Networks, IEEE Transactions on 13 (5), 1204-1210. Which perhaps ought to have attracted rather more attention than it has. $\endgroup$ Commented Sep 20, 2014 at 8:57
-
$\begingroup$ I would also point out that asymptotic consistency is not necessarily important in practice as classifiers are often used in situations that do not approach any where near the infinite data limit (if not, there would be no need for regularisation etc.). In practice, the ability to estimate parameters based on limited data is the key issue, which is where Vapnik's dictum, discussed in my answer, becomes important. $\endgroup$ Commented Sep 20, 2014 at 9:00
-
$\begingroup$ You're right, I haven't seen that paper which is published earlier, but Tong's paper as well as Peter's paper (Convexity, Classification, and Risk Bounds.) are standard classic papers in the machine learning community. $\endgroup$– AryaCommented Sep 22, 2014 at 17:59
-
$\begingroup$ I disagree, consistency is an important aspect, Also, I should say that, regularization is a general technique for dealing with ill-posed problems, (i.e. in this case problem has many solutions). Regularization, ties objective function in "a favorable" solution among all solutions. But we can always learn biased estimators, e.g. shrinkage, provided that we have a justification for doing that. For example all the Bayesian estimators are biased! but since we are using a plausible prior it will have a smaller MSE (bias+variance) $\endgroup$– AryaCommented Sep 22, 2014 at 18:08
-
1$\begingroup$ The point I am making is that in practice the error is composed of both bias and variance. If you have an infinite amount of data, then generally the model variance will tend to zero, but in reality we often approach now where near that point, so the best you can do is to reach a compromise between bias and variance. The classic example of this is that LOOCV is often said to be good as it is asymptotically unbiased, but in practice it is often not very good because of its high variance. Asymptotic properties are not that important unless you approach the asymptote. $\endgroup$ Commented Sep 23, 2014 at 8:19
$\begingroup$
$\endgroup$
You begin with a misunderstanding. It is important to get the terminology right at the beginning. Logistic regression is not a classifier. It is a direct probability model.
You didn't explain why your problem is an all-or-nothing classification problem vs. a risk estimation problem.
I have to disagree with the answer above. You will get more efficient/powerful/precise estimates by using maximum likelihood estimation to fit a probability model such as the logistic model, then applying your utilities/cost/loss function to the predicted probabilities to make optimum decisions. If you cannot come up with a utility/loss function it's hard to argue that you should be doing classification in the first place, but you could make the ridiculous assumption that utilities are the same for every observation and make a classification based on predicted probabilities. You will quickly see that classification is arbitrary when you proceed that way.
Note that proportion "classified" correct is an improper accuracy scoring rule that is optimized by a bogus model.
answered Sep 19, 2014 at 13:21
$\begingroup$
$\endgroup$
The key question is whether you are likely to need estimates of the probability of class membership, a ranking, or whether you genuinely only interested in a binary classification. In my experience, you often do want the probabilities as the class frequencies, or the misclassification costs are unknown or variable in operation. If you have a probabilistic classifier you can compensate for these problems after training, if you have a discrete yes/no classifier you can't.
One of the guiding principles behind the support vector machine was Prof. Vapnik's idea that in solving a particular problem, you should not solve a more general problem and then simplify the answer. In classification this would mean that if you are only interested in binary classification, then we should not estimate probabilities and then threshold them, because modelling efforts and resources are wasted estimating the changes in probability away from the decision boundary, where they are of no interest. This is a very reasonable idea, and I fully agree, provided you really are only interested in a discrete yes/no classification.
As it happens, if you perform least-squares regression on 0/1 targets, you will asymptotically end up with estimates of the probabilities anyway. This is because least-squares results in the output being an estimate of the conditional mean of the target variable. If this is coded as 0/1 then the conditional mean is just the conditional probability of a 1, given the input vector.
In short, which method you use depends on the needs of the application, if you need the probabilities or a ranking of the test data, use a probabilistic method (or least-squares etc. for ranking). If you only want the hard classification into discrete classes, use something designed especially for that problem, such as the SVM.
answered Sep 19, 2014 at 10:55