4
$\begingroup$

If a dataset contains a perfect predictor a linear regression is able to identify this variable. Why is it that a tree model cannot do the same?

In the R code below, the dataframe df contains an independent variable IndPerfect that is identical to the dependent variable Target. For the illustration only the last row is used for out of sample forecasting. A linear regression gives a perfect forecast and the parameters of the other parameters are set to (almost) zero:

set.seed(1)
df <- data.frame(Ind1 = rnorm(100),Ind2 = rnorm(100),Ind3 = rnorm(100),Target = rnorm(100)) 
df$IndPerfect<- df$Target #generate perfect independent variable

testindex <- 100
indizes_train <- 1:90
indizes_validation <-  91:99

#Lin. Regression
model <- lm(Target~.,data = df[indizes_train,])
df$Target[testindex] - predict(model,newdata = df[testindex,])  #Perfect Forecasts by the lin. Regression
#-3.330669e-16

This result is not surprising. However, I was not able to generate a perfect forecast by using a tree method such as in R's extraTrees or xgboost. The following code uses xgboost including the validation sample of 10 rows from the code above:

library(xgboost)
#Gradient Booster
name <- c("Ind1","Ind2","Ind3","IndPerfect")

set.seed(1)
dtrain <- xgb.DMatrix(data.matrix(df[indizes_train,name]), label=(df[indizes_train,"Target"]))  
dval <- xgb.DMatrix(data.matrix(df[indizes_validation,name]), label=(df[indizes_validation,"Target"])) 
watchlist <- list( train = dtrain,eval = dval)

param <- list(  objective           = "reg:linear", 
    eta                 = .01,
    subsample           = 1,
    colsample_bytree    =1,
    eval_metric         = "rmse"
)

xgbmodel <- xgb.train(   params              = param, 
        data                = dtrain, 
        nrounds             = 1000, 
        verbose             = 1, 
        early.stopping.round     = 50,
        print_every_n = 5,
        watchlist           = watchlist,
        maximize            = FALSE,
        nthread = 2)

dtest <- xgb.DMatrix(data.matrix(df[testindex,name]))  
df$Target[testindex] -predict(xgbmodel, dtest, ntreelimit = xgbmodel$bestInd)
# -0.02821238

It does not matter what parameters I choose - no tree model (xgb or et) ever gives a perfect forecast. Is there a theoretical reason for a tree model to not be able to make perfect forecasts?

$\endgroup$
  • 3
    $\begingroup$ Good question. I reckon its because trees are splitting data by variables and then fit regression to each node. So inherently are losing some information along the way $\endgroup$ – Jan Sila Mar 19 '17 at 16:42
0
$\begingroup$

For linear model, you could try gblinear in xgboost. Why it is linear model, you can check the model summary:

summary(model)

Gblinear example:

library(xgboost)
#Gradient Booster
name <- c("Ind1","Ind2","Ind3","IndPerfect")

set.seed(1)
dtrain <- xgb.DMatrix(data.matrix(df[indizes_train,name]), label=    (df[indizes_train,"Target"]))  
dval <- xgb.DMatrix(data.matrix(df[indizes_validation,name]), label=(df[indizes_validation,"Target"])) 
watchlist <- list( train = dtrain,eval = dval)

param <- list(  objective           = "reg:linear", 
eta                 = .01,
subsample           = 1,
colsample_bytree    =1,
eval_metric         = "rmse"
)

xgbmodel <- xgb.train(   params              = param, 
    data                = dtrain, 
    nrounds             = 1000, 
    verbose             = 1, 
    early.stopping.round     = 50,
    print_every_n = 5,
    watchlist           = watchlist,
    maximize            = FALSE,
    nthread = 2)

dtest <- xgb.DMatrix(data.matrix(df[testindex,name]))  
df$Target[testindex] -predict(xgbmodel, dtest, ntreelimit = xgbmodel$bestInd)

#  1.536799e-05
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.