2
$\begingroup$

I used both techniques on a rather unbalance data set (90%/10%) with 2500 observations and about 20 features. Firstly, I compared the default models. In a second step, I weighed the misclassification costs differently to correct for the skewed distribution. However, classification trees kept to perform worse. Before weighing the costs differently, the trees kept splitting only once (after pruning). How should I conclude here? A few points which crossed my mind are the following:

  • the problem might be kind of linearly separable and this is why logistic regression is performing better
  • the features do not interact with each other
  • the features can simply not explain the response (but this brings up the question of why the variables are significant in the logistic model)
Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ "90/10" means that you have only 100 observations? Then a tree model will be very shallow, while a logistic regression can retain more variables before getting into overestimation. So I am not very surprised that logistic regression is better! $\endgroup$
    – kjetil b halvorsen ♦
    Jan 7, 2016 at 0:06
  • 1
    $\begingroup$ Don't be mislead by classification error, which is an improper accuracy scoring rule. But you would expect logistic regression to almost always beat single trees. That's why random forests were invented. $\endgroup$
    – Frank Harrell
    Jan 7, 2016 at 3:36
  • $\begingroup$ Thank you for the fast replies! However, I nean 90%/10%! I'm sorry for being not so clear. $\endgroup$
    – Patrick Balada
    Jan 7, 2016 at 8:00

1 Answer 1

Reset to default
3
$\begingroup$

If you literally mean classification trees (as opposed to a random forest), it shouldn't be surprising that a logistic model will outperform a tree in some cases (especially if the dataset is small). As an extreme example, suppose that the data actually come from a logistic model. Then, the logistic regression only has to "learn" a few coefficients, while the tree method will have to "learn," from scratch, an approximation of the logistic curve by a piecewise-constant function made out of cuts. The tree is more flexible in a way, at the cost of requiring more samples.

I would guess that you happen have a problem where the $f(x)=Pr[Y=1\ \|\ X=x]$ is very smooth in x, and furthermore approximately obeys the logistic curve. The fact that the trees only split once seems to support this; there aren't sharp jumps and dips in $f(x)$ to make cuts on.

If you'd like to test this, one easy step would be to round your x values (perhaps by binning into deciles). The performance of logistic regression should degrade fairly quickly to the level of the decision trees (and eventually worse, of course, if you keep reducing the number of bins).

Random forests, being a "fuzzy" ensemble method, should perform a bit better than a raw tree, but certainly might not match LR in this case.

IMHO, just abandon the idea of linear separability as an intuitive explanation for anything but SVMs.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.