If a dataset contains a perfect predictor a linear regression is able to identify this variable. Why is it that a tree model cannot do the same?
In the R code below, the dataframe df contains an independent variable IndPerfect that is identical to the dependent variable Target. For the illustration only the last row is used for out of sample forecasting. A linear regression gives a perfect forecast and the parameters of the other parameters are set to (almost) zero:
set.seed(1) df <- data.frame(Ind1 = rnorm(100),Ind2 = rnorm(100),Ind3 = rnorm(100),Target = rnorm(100)) df$IndPerfect<- df$Target #generate perfect independent variable testindex <- 100 indizes_train <- 1:90 indizes_validation <- 91:99 #Lin. Regression model <- lm(Target~.,data = df[indizes_train,]) df$Target[testindex] - predict(model,newdata = df[testindex,]) #Perfect Forecasts by the lin. Regression #-3.330669e-16
This result is not surprising. However, I was not able to generate a perfect forecast by using a tree method such as in R's extraTrees or xgboost. The following code uses xgboost including the validation sample of 10 rows from the code above:
library(xgboost) #Gradient Booster name <- c("Ind1","Ind2","Ind3","IndPerfect") set.seed(1) dtrain <- xgb.DMatrix(data.matrix(df[indizes_train,name]), label=(df[indizes_train,"Target"])) dval <- xgb.DMatrix(data.matrix(df[indizes_validation,name]), label=(df[indizes_validation,"Target"])) watchlist <- list( train = dtrain,eval = dval) param <- list( objective = "reg:linear", eta = .01, subsample = 1, colsample_bytree =1, eval_metric = "rmse" ) xgbmodel <- xgb.train( params = param, data = dtrain, nrounds = 1000, verbose = 1, early.stopping.round = 50, print_every_n = 5, watchlist = watchlist, maximize = FALSE, nthread = 2) dtest <- xgb.DMatrix(data.matrix(df[testindex,name])) df$Target[testindex] -predict(xgbmodel, dtest, ntreelimit = xgbmodel$bestInd) # -0.02821238
It does not matter what parameters I choose - no tree model (xgb or et) ever gives a perfect forecast. Is there a theoretical reason for a tree model to not be able to make perfect forecasts?