How to interpret when logistic regression is performing better than classification trees? I used both techniques on a rather unbalance data set (90%/10%) with 2500 observations and about 20 features. Firstly, I compared the default models. In a second step, I weighed the misclassification costs differently to correct for the skewed distribution. However, classification trees kept to perform worse. Before weighing the costs differently, the trees kept splitting only once (after pruning). How should I conclude here? A few points which crossed my mind are the following:


*

*the problem might be kind of linearly separable and this is why logistic regression is performing better

*the features do not interact with each other

*the features can simply not explain the response (but this brings up the question of why the variables are significant in the logistic model)

 A: If you literally mean classification trees (as opposed to a random forest), it shouldn't be surprising that a logistic model will outperform a tree in some cases (especially if the dataset is small). As an extreme example, suppose that the data actually come from a logistic model. Then, the logistic regression only has to "learn" a few coefficients, while the tree method will have to "learn," from scratch, an approximation of the logistic curve by a piecewise-constant function made out of cuts. The tree is more flexible in a way, at the cost of requiring more samples.
I would guess that you happen have a problem where the $f(x)=Pr[Y=1\ \|\ X=x]$ is very smooth in x, and furthermore approximately obeys the logistic curve. The fact that the trees only split once seems to support this; there aren't sharp jumps and dips in $f(x)$ to make cuts on.
If you'd like to test this, one easy step would be to round your x values (perhaps by binning into deciles). The performance of logistic regression should degrade fairly quickly to the level of the decision trees (and eventually worse, of course, if you keep reducing the number of bins).
Random forests, being a "fuzzy" ensemble method, should perform a bit better than a raw tree, but certainly might not match LR in this case.
IMHO, just abandon the idea of linear separability as an intuitive explanation for anything but SVMs.
