I'm using one explanatory variable in a regression tree and in a linear regression. The tree finds a split (with variance reduction splitting rule), though R2 is pretty small (0.2). On the validation data the model is confirmed. On the other hand the linear regression shows no relation (not even with 2nd order polynomial reg): coef and R2 are almost 0s. (Outliers are handled by truncation.) How can you explain this? Can it be that the tree finds/creates a "non existing" pattern only because of the splitting rule?


2 Answers 2


Just because there is no linear relationship between the variables does not mean that the pattern is "non-existing". Here is a simple example (using R).

x = runif(100, 0, 5)
y = ifelse(x<1, 0, ifelse(x<2, 1, ifelse(x<3,0, ifelse(x<4,1,0))))
plot(x,y, pch=20)

Example data

Because there is no linear relationship, fitting linear model shows a very poor fit.

LM = lm(y~x)
summary(LM)          ## Output simplified

    Min      1Q  Median      3Q     Max 
-0.5214 -0.5106  0.4829  0.4921  0.4963 

Residual standard error: 0.5049 on 98 degrees of freedom
Multiple R-squared:  0.0001441, Adjusted R-squared:  -0.01006 
F-statistic: 0.01413 on 1 and 98 DF,  p-value: 0.9056

Even with a quadratic term we get a poor fit.

LM2 = lm(y~poly(x,2))
summary(LM2)      ## Output simplified

    Min      1Q  Median      3Q     Max 
-0.7255 -0.3478  0.3062  0.4062  0.5495 

Residual standard error: 0.4659 on 97 degrees of freedom
Multiple R-squared:  0.1576,    Adjusted R-squared:  0.1402
F-statistic:  9.07 on 2 and 97 DF,  p-value: 0.0002448

But a regression tree gives a perfect fit to the data.

RP = rpart(y ~ x)
summary(predict(RP) - y)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
      0       0       0       0       0       0

There is a very real relationship between x and y, just not a linear relationship.


Let $y=x^2$

A linear model will not be able to capture anything and will just return $\beta_0$ as the mean and $\beta_1=0$

However, a regression tree will find a split based on the value of $x$

My point is, linear models are restricted to the detection of linear patterns.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.