I need to find a situation in which logistic regression does not work well. Furthermore, I would like to know when a random forest might perform better than a logistic regression model.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ try to make clusters of data in different locations (e.g. class 1 around (-1,-1) and (1, 1) and class 2 around (-1, 1) and (1, -1))- works good with dicision tress but not with logistic regression. $\endgroup$– DreyNov 30, 2016 at 11:49
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
4
Consider these data (copied from @Sycorax's answer here: Can Random Forest be used for Feature Selection in Multiple Linear Regression?):
There are two aspects to the data in this figure. First, the relationship is non-linear. That isn't actually a problem for logistic regression properly specified. In some cases, a logistic regression might fair better than a standard decision tree (cf., my answer here: How to use boxplots to find the point where values are more likely to come from different conditions?, although vis-a-vie a random forest is more ambiguous). The bigger problem is that there is complete separation at the decision boundary. There are ways of trying to deal with that (see @Scortchi's answer here: How to deal with perfect separation in logistic regression?), but it adds complexity and requires considerable sophistication to address well. I think a random forest would handle this as a matter of course.
answered Nov 30, 2016 at 19:35
-
2$\begingroup$ +1 good point. But after feature transformation, e.g., polynomial expansion or kernel PCA, logistic regression will work well. $\endgroup$ Nov 30, 2016 at 19:38
-
$\begingroup$ Hah! I came here to write exactly this. +1 $\endgroup$– Sycorax ♦Nov 30, 2016 at 19:50
-
3$\begingroup$ @hxd1011: This is all fine in the above example given the trivial transformation you can apply. However in higher dimensions, especially when your clusters of classes are considerably more complex and have different separation widths, this becomes considerably more complex. $\endgroup$– Alex R.Nov 30, 2016 at 19:53
-
1$\begingroup$ @Sycorax, beat you to it ;-)! $\endgroup$ Nov 30, 2016 at 20:01
$\begingroup$
$\endgroup$
1
The answers so far emphasize the predictive failure of logistic regression. However there's also issues of poor feature importance/inference. For example, when your classes are highly correlate or highly nonlinear, the coefficients of your logistic regression will not correctly predict the gain/loss from each individual feature. In gung's example, if you were to train a logistic regression on the picture of points shown, it will likely create a linear split somewhere in the middle of the red region (for example a vertical line), implying that an increase of say, $x_1$, will lead to a higher probability of being a red class, which is true for $x_1$ starting to the left of the prediction boundary, and false for $x_1$ starting to the right.
-
1$\begingroup$ FWIW, my point was that you needn't fit a LR model composed of linear (in the log odds) functions of X1 & X2. $\endgroup$ Nov 30, 2016 at 20:07
$\begingroup$
$\endgroup$
2
@gung had a good answer. Logistic regression is a linear model, so it may not work well on non-linear cases. But as I mentioned in the comment, it might be some ways to transform data into another space, where logistic regression will be good again, but finding the basis expansion / feature transformation may be not trivial.
Essentially, certain model will work well when the data satisfy the assumption of the model, e.g., if the decision boundary is linear, then logistic regression will work well.
On the other hand, I would strongly recommend you to review bias variance trade off.
In terms of model complexity, logistic regression has high bias and low variance. And random forest is opposite. Which means, in general, logistic regression will preform with "less accurate" but "more stable", and random forest is opposite.
answered Nov 30, 2016 at 19:42
-
2$\begingroup$ You should read the full discussion in my linked answer that @gung linked to. If you know exactly what the functional form is, basis expansion is viable. But the form of the expansion is not automatically suggested. Moreover, the stability of random forest can be improved by increasing the number of trees and enforcing minimum sizes on terminal nodes/splits. However, RF is very weak when it has to extrapolate. $\endgroup$– Sycorax ♦Nov 30, 2016 at 20:01
-
$\begingroup$ @Sycorax thanks for Alex R. and your comments. I revised my answer. $\endgroup$ Nov 30, 2016 at 20:06
$\begingroup$
$\endgroup$
1
A simple example for a case when logistic regression can’t work properly: https://towardsdatascience.com/when-logistic-regression-simply-doesnt-work-8cd8f2f9d997
-
1$\begingroup$ Welcome to the site. We are trying to build a permanent repository of high-quality statistical information in the form of questions & answers. Thus, we're wary of link-only answers, due to linkrot. Can you post a full citation & a summary of the information at the link, in case it goes dead? $\endgroup$ Feb 20, 2021 at 13:09
$\begingroup$
$\endgroup$
Situations where features combined effect on response are not linear.
Imagine you try classifying documents as pet related or not per related.
You have two features -
* Number of words in the document(X1).
* Number of times the word Dog appears in the document(X2).
Intuitively, X2/X1 is a good way determining a document's class.
This is not a linear relation, so where tree like models can use a:
If X1>10:
if X2>5:
Pet related
else:
Not Pet related
A logistic regression will have no such option, and will result in model described by
if aX1 + bX2 > Z:
Pet related
else:
Not Pet related
For some Z.
