4
$\begingroup$

If a dataset contains a perfect predictor a linear regression is able to identify this variable. Why is it that a tree model cannot do the same?

In the R code below, the dataframe df contains an independent variable IndPerfect that is identical to the dependent variable Target. For the illustration only the last row is used for out of sample forecasting. A linear regression gives a perfect forecast and the parameters of the other parameters are set to (almost) zero:

set.seed(1)
df <- data.frame(Ind1 = rnorm(100),Ind2 = rnorm(100),Ind3 = rnorm(100),Target = rnorm(100)) 
df$IndPerfect<- df$Target #generate perfect independent variable

testindex <- 100
indizes_train <- 1:90
indizes_validation <-  91:99

#Lin. Regression
model <- lm(Target~.,data = df[indizes_train,])
df$Target[testindex] - predict(model,newdata = df[testindex,])  #Perfect Forecasts by the lin. Regression
#-3.330669e-16

This result is not surprising. However, I was not able to generate a perfect forecast by using a tree method such as in R's extraTrees or xgboost. The following code uses xgboost including the validation sample of 10 rows from the code above:

library(xgboost)
#Gradient Booster
name <- c("Ind1","Ind2","Ind3","IndPerfect")

set.seed(1)
dtrain <- xgb.DMatrix(data.matrix(df[indizes_train,name]), label=(df[indizes_train,"Target"]))  
dval <- xgb.DMatrix(data.matrix(df[indizes_validation,name]), label=(df[indizes_validation,"Target"])) 
watchlist <- list( train = dtrain,eval = dval)

param <- list(  objective           = "reg:linear", 
    eta                 = .01,
    subsample           = 1,
    colsample_bytree    =1,
    eval_metric         = "rmse"
)

xgbmodel <- xgb.train(   params              = param, 
        data                = dtrain, 
        nrounds             = 1000, 
        verbose             = 1, 
        early.stopping.round     = 50,
        print_every_n = 5,
        watchlist           = watchlist,
        maximize            = FALSE,
        nthread = 2)

dtest <- xgb.DMatrix(data.matrix(df[testindex,name]))  
df$Target[testindex] -predict(xgbmodel, dtest, ntreelimit = xgbmodel$bestInd)
# -0.02821238

It does not matter what parameters I choose - no tree model (xgb or et) ever gives a perfect forecast. Is there a theoretical reason for a tree model to not be able to make perfect forecasts?

$\endgroup$
  • 3
    $\begingroup$ Good question. I reckon its because trees are splitting data by variables and then fit regression to each node. So inherently are losing some information along the way $\endgroup$ – Jan Sila Mar 19 '17 at 16:42
  • $\begingroup$ @JanSila whilst some regression trees do what you say, xgboost doesn't fit a regression in the leaf nodes. Each leaf is assigned a weight which is returned as the prediction. For regression with squared error the leaf weight is the mean of the y vaues associated with the leaf. $\endgroup$ – David Waterworth Oct 10 at 0:43
0
$\begingroup$

For linear model, you could try gblinear in xgboost. Why it is linear model, you can check the model summary:

summary(model)

Gblinear example:

library(xgboost)
#Gradient Booster
name <- c("Ind1","Ind2","Ind3","IndPerfect")

set.seed(1)
dtrain <- xgb.DMatrix(data.matrix(df[indizes_train,name]), label=    (df[indizes_train,"Target"]))  
dval <- xgb.DMatrix(data.matrix(df[indizes_validation,name]), label=(df[indizes_validation,"Target"])) 
watchlist <- list( train = dtrain,eval = dval)

param <- list(  objective           = "reg:linear", 
eta                 = .01,
subsample           = 1,
colsample_bytree    =1,
eval_metric         = "rmse"
)

xgbmodel <- xgb.train(   params              = param, 
    data                = dtrain, 
    nrounds             = 1000, 
    verbose             = 1, 
    early.stopping.round     = 50,
    print_every_n = 5,
    watchlist           = watchlist,
    maximize            = FALSE,
    nthread = 2)

dtest <- xgb.DMatrix(data.matrix(df[testindex,name]))  
df$Target[testindex] -predict(xgbmodel, dtest, ntreelimit = xgbmodel$bestInd)

#  1.536799e-05
$\endgroup$
0
$\begingroup$

Because each tree grown by xgboost splits y into leaf nodes conditional on X. The prediction (or leaf weight) is the value which minimises the loss function for the split y's - so for example if the loss function is squared error, the leaf weight is the mean of the y's.

So say you have 1 tree of depth 1 (i.e. a single split on 1 feature). All this model can return is two discrete values, the mean of the y's for either the left or right leaf. So even a simple linear response takes a large number of splits / trees and will still never produce a smooth function.

There are regression trees which apply a linear regression to the X,y in each leaf node but `modern' implementations like xgboost, catboost, lightgbm etc don't do this.

Edit: also because of this, trees cannot extrapolate outside the data they were trained with

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.