regression tree vs linear regression

Question

I'm using one explanatory variable in a regression tree and in a linear regression. The tree finds a split (with variance reduction splitting rule), though R2 is pretty small (0.2). On the validation data the model is confirmed. On the other hand the linear regression shows no relation (not even with 2nd order polynomial reg): coef and R2 are almost 0s. (Outliers are handled by truncation.) How can you explain this? Can it be that the tree finds/creates a "non existing" pattern only because of the splitting rule?

Answer 1

Just because there is no linear relationship between the variables does not mean that the pattern is "non-existing". Here is a simple example (using R).

set.seed(1)
x = runif(100, 0, 5)
y = ifelse(x<1, 0, ifelse(x<2, 1, ifelse(x<3,0, ifelse(x<4,1,0))))
plot(x,y, pch=20)

Because there is no linear relationship, fitting linear model shows a very poor fit.

LM = lm(y~x)
summary(LM)          ## Output simplified

Residuals:
    Min      1Q  Median      3Q     Max 
-0.5214 -0.5106  0.4829  0.4921  0.4963 

Residual standard error: 0.5049 on 98 degrees of freedom
Multiple R-squared:  0.0001441, Adjusted R-squared:  -0.01006 
F-statistic: 0.01413 on 1 and 98 DF,  p-value: 0.9056

Even with a quadratic term we get a poor fit.

LM2 = lm(y~poly(x,2))
summary(LM2)      ## Output simplified

Residuals:
    Min      1Q  Median      3Q     Max 
-0.7255 -0.3478  0.3062  0.4062  0.5495 

Residual standard error: 0.4659 on 97 degrees of freedom
Multiple R-squared:  0.1576,    Adjusted R-squared:  0.1402
F-statistic:  9.07 on 2 and 97 DF,  p-value: 0.0002448

But a regression tree gives a perfect fit to the data.

library(rpart)
RP = rpart(y ~ x)
summary(predict(RP) - y)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
      0       0       0       0       0       0

There is a very real relationship between x and y, just not a linear relationship.

Answer 2

Let $y=x^2$

A linear model will not be able to capture anything and will just return $\beta_0$ as the mean and $\beta_1=0$

However, a regression tree will find a split based on the value of $x$

My point is, linear models are restricted to the detection of linear patterns.