14
$\begingroup$

I'm currently teaching myself how to do classification, and specifically I'm looking at three methods: support vector machines, neural networks, and logistic regression. What I am trying to understand is why logistic regression would ever perform better than the other two.

From my understanding of logistic regression, the idea is to fit a logistic function to the entire data. So if my data is binary, all my data with label 0 should be mapped to the value 0 (or close to it), and all my data with value 1 should be mapped to value 1 (or close to it). Now, because the logistic function is continuous and smooth, performing this regression requires all my data to fit the curve; there is no greater importance applied to data points near the decision boundary, and all data points contribute to the loss by different amounts.

However, with support vector machines and neural networks, only those data points near the decision boundary are important; as long as a data point remains on the same side of the decision boundary, it will contribute the same loss.

Therefore, why would logistic regression ever outperform support vector machines or neural networks, given that it "wastes resources" on trying to fit a curve to lots of unimportant (easily-classifiable) data, rather than focussing only on the difficult data around the decision boundary?

Improve this question
$\endgroup$
2
  • 5
    $\begingroup$ LR will give you probability estimates while SVM gives binary estimates. That also makes LR useful when there is no separating hyperplane between the classes. Also, you have to take into consideration the complexity of the algorithms and other characteristics like number of parameters and sensitivity. $\endgroup$
    – Bar
    Commented Feb 23, 2016 at 12:03
  • 1
    $\begingroup$ Related: stats.stackexchange.com/questions/127042/… $\endgroup$
    – Sycorax ♦
    Commented Feb 23, 2016 at 22:28

2 Answers 2

Reset to default
31
$\begingroup$

The resources you consider to be "wasted" are, in fact, information gains provided by logistic regression. You started out with the wrong premise. Logistic regression is not a classifier. It is a probability/risk estimator. Unlike SVM, it allows for and expects "close calls". It will lead to optimum decision making because it does not try to trick the predictive signal into incorporating a utility function that is implicit whenever you classify observations. The goal of logistic regression using maximum likelihood estimation is to provide optimum estimates of Prob$(Y=1|X)$. The result is used in many ways, e.g. lift curves, credit risk scoring, etc. See Nate Silver's book Signal and the Noise for compelling arguments in favor of probabilistic reasoning.

Note that the dependent variable $Y$ in logistic regression can be coded any way you want: 0/1, A/B, yes/no, etc.

The primary assumption of logistic regression is that $Y$ is truly binary, e.g. it was not contrived from an underlying ordinal or continuous response variable. It, like classification methods, is for truly all-or-nothing phenomena.

Some analysts think that logistic regression assumes linearity of predictor effects on the log odds scale. That was only true when DR Cox invented the logistic model in 1958 at a time when computing wasn't available to extend the model using tools such as regression splines. The only real weakness in logistic regression is that you need to specify which interactions you want to allow in the model. For most datasets this is turned into a strength because the additive main effects are generally much stronger predictors than interactions, and machine learning methods that give equal priority to interactions can be unstable, hard to interpret, and require larger sample sizes than logistic regression to predict well.

Improve this answer
$\endgroup$
2
  • 8
    $\begingroup$ +1. To be honest, I've never found SVMs to be useful. They're sexy but they're slow to train and score -- in my experience -- and have lots of choices you need to fiddle with (including kernel). Neural networks I've found to be useful, but also lots of options and adjustments. Logistic regression is simple and gives reasonably-well-calibrated results out of the box. Calibration is important for real-world use. Of course, the downside is that it's linear, so can't fit cluster-ish, lumpy data as well as other methods like Random Forest. $\endgroup$
    – Wayne
    Commented Feb 24, 2016 at 0:57
  • 1
    $\begingroup$ Great answer. By the way, you may be interested to know that recently the machine-learners have come around to fitting their fancy methods into traditional frameworks such as penalized maximum likelihood - and it turns out the fancy methods work way better when this is done. Consider XGBoost, arguably the most effective tree ensemble boosting algorithm in existence. The math is here: xgboost.readthedocs.io/en/latest/model.html. It should look quite familiar to a traditional statistician, and you can fit models for many common statistical purposes with the usual loss functions. $\endgroup$
    – Paul
    Commented Jun 25, 2017 at 21:48
5
$\begingroup$

You are right, oftentimes logistic regression does poorly as a classifier (especially when compared to other algorithms). However, this doesn't mean logistic regression should be forgotten and never studied as it has two big advantages:

  1. Probabilistic results. Frank Harrell (+1) explained this very well in his answer.

  2. It allows us to understand the impact an independent variable has on the dependent variable while controlling for other independent variables. For example, it provides estimates and standard errors for conditional odds ratios (how many times larger are the odds of $Y=1$ when $X_1 = 1$ instead of $2$ while holding $X_2,...X_p$ constant).

Improve this answer
$\endgroup$
5
  • 5
    $\begingroup$ And the apparent poor performance as a classifier is a result of using an improper accuracy score, not a problem inherent to logistic regression. $\endgroup$
    – Frank Harrell
    Commented Feb 23, 2016 at 22:23
  • $\begingroup$ @FrankHarrell: I've been doing some experiments lately and I'd say that Logistic Regression fits data with a lot less freedom than other methods. You need to add interactions and do more feature engineering to match, say, a Random Forest's or GAM's flexibility. (Of course flexibility is the tightrope that crosses the abyss of overfitting.) $\endgroup$
    – Wayne
    Commented Feb 24, 2016 at 1:00
  • 3
    $\begingroup$ @wayne This less freedom, as you state it, is very helpful in many cases, because it provides stability $\endgroup$
    – rapaio
    Commented Feb 24, 2016 at 2:21
  • 3
    $\begingroup$ Not only does assuming interaction terms are less important than additive terms add flexibility but you can relax the assumptions in many ways. I'm adding more about this in my original answer. $\endgroup$
    – Frank Harrell
    Commented Feb 24, 2016 at 12:26
  • 2
    $\begingroup$ @rapaio: Yes, flexibility is dangerous, both in terms of overfitting, but also in other ways. It's a domain/use issue: is your data noisy, or is it truly "lumpy/cluster-ish" if I may use that term? $\endgroup$
    – Wayne
    Commented Feb 24, 2016 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.